* Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta. [not found] ` <e9aeeef80712141211v7cec29fdt7e2ec76bf877053f@mail.gmail.com> @ 2007-12-14 21:33 ` Denis 2007-12-17 15:32 ` Jan Kara 0 siblings, 1 reply; 4+ messages in thread From: Denis @ 2007-12-14 21:33 UTC (permalink / raw) To: linux-kernel Dears, I've got a double dual xeon wich sometimes is getting too slow. While i was observing one of these slowly times, I realized the time wait of processors come close to 100% and nfsd and kjournald process become to D status (uniterruptible sleep) .Does someone know why this can be happening or how can I troubleshoot it? Below is a 'draw' of my top at one of these moments. [root@cromo src]# top top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00 Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached 3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd 4283 root 16 0 20936 8136 1276 S 0 0.2 0:44.47 hald 21289 denismpa 16 0 6280 1092 776 R 0 0.0 0:00.12 top 1 root 16 0 4752 552 460 S 0 0.0 0:01.28 init 2 root RT 0 0 0 0 S 0 0.0 0:00.21 migration/0 3 root 34 19 0 0 0 S 0 0.0 0:00.05 ksoftirqd/0 4 root RT 0 0 0 0 S 0 0.0 0:00.28 migration/1 5 root 34 19 0 0 0 S 0 0.0 0:00.09 ksoftirqd/1 6 root RT 0 0 0 0 S 0 0.0 0:00.15 migration/2 7 root 34 19 0 0 0 S 0 0.0 0:00.05 ksoftirqd/2 8 root RT 0 0 0 0 S 0 0.0 0:00.15 migration/3 9 root 34 19 0 0 0 S 0 0.0 0:00.06 ksoftirqd/3 10 root 5 -10 0 0 0 S 0 0.0 0:01.01 events/0 11 root 5 -10 0 0 0 S 0 0.0 0:00.11 events/1 12 root 5 -10 0 0 0 S 0 0.0 0:00.06 events/2 13 root 5 -10 0 0 0 S 0 0.0 0:00.12 events/3 14 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper 15 root 15 -10 0 0 0 S 0 0.0 0:00.00 kacpid 63 root 5 -10 0 0 0 S 0 0.0 0: 00.00 kblockd/0 64 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1 65 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/2 66 root 5 -10 0 0 0 S 0 0.0 0: 00.00 kblockd/3 67 root 15 0 0 0 0 S 0 0.0 0:00.06 khubd 97 root 12 -10 0 0 0 S 0 0.0 0:00.00 aio/0 98 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1 99 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/2 100 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/3 244 root 23 0 0 0 0 S 0 0.0 0:00.00 kseriod 359 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0 360 root 15 0 0 0 0 S 0 0.0 0:00.00 aacraid 379 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/0 380 root 5 -10 0 0 0 S 0 0.0 0: 00.00 ata/1 381 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/2 382 root 8 -10 0 0 0 S 0 0.0 0:00.00 ata/3 388 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1 389 root 21 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2 403 root 15 0 0 0 0 D 0 0.0 0:03.87 kjournald 1708 root 6 -10 3604 444 364 S 0 0.0 0:00.00 udevd Thanks and regards -- Denis Anjos. Cisco Certified Network Associate. Universidade Federal do ABC Santo André - SP - BR ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta. 2007-12-14 21:33 ` Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta Denis @ 2007-12-17 15:32 ` Jan Kara 2007-12-17 16:41 ` Denis 0 siblings, 1 reply; 4+ messages in thread From: Jan Kara @ 2007-12-17 15:32 UTC (permalink / raw) To: Denis; +Cc: linux-kernel > Dears, I've got a double dual xeon wich sometimes is getting too slow. > > > While i was observing one of these slowly times, I realized the time > wait of processors come close to 100% and nfsd and kjournald process > become to D status (uniterruptible sleep) .Does someone know why this > can be happening or how can I troubleshoot it? > > Below is a 'draw' of my top at one of these moments. > > [root@cromo src]# top > top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00 > Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie > Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si > Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si > Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si > Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si All processors are in iowait - i.e. waiting for IO. > Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers > Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached > > 3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd > 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd > 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald > 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd > 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 > 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd > 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd > 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd It seems like heavy disk/network activity. You can try running vmstat 1 It will print every second the situation of your memory / IO subsystem. You should especially see the ---io--- column ('bi' means blocks in, 'bo' blocks outputted). I guess they'd contain some non-trivial values... BTW: I guess the processes wake up eventually, don't they? Honza -- Jan Kara <jack@suse.cz> SuSE CR Labs ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta. 2007-12-17 15:32 ` Jan Kara @ 2007-12-17 16:41 ` Denis 2007-12-17 17:49 ` Jan Kara 0 siblings, 1 reply; 4+ messages in thread From: Denis @ 2007-12-17 16:41 UTC (permalink / raw) To: Jan Kara; +Cc: linux-kernel 2007/12/17, Jan Kara <jack@suse.cz>: > > Dears, I've got a double dual xeon wich sometimes is getting too slow. > > > > > > While i was observing one of these slowly times, I realized the time > > wait of processors come close to 100% and nfsd and kjournald process > > become to D status (uniterruptible sleep) .Does someone know why this > > can be happening or how can I troubleshoot it? > > > > Below is a 'draw' of my top at one of these moments. > > > > [root@cromo src]# top > > top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00 > > Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie > > Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si > > Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si > > Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si > > Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si > All processors are in iowait - i.e. waiting for IO. > > > Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers > > Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached > > > > 3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd > > 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd > > 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald > > 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd > > 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 > > 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd > > 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd > > 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd > It seems like heavy disk/network activity. You can try running > vmstat 1 > It will print every second the situation of your memory / IO > subsystem. You should especially see the ---io--- column ('bi' means > blocks in, 'bo' blocks outputted). I guess they'd contain some > non-trivial values... > > Honza > -- > Jan Kara <jack@suse.cz> > SuSE CR Labs > Hello, Jan. The output of my vmstat look like this: procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 3612 239600 2828 3364300 0 0 1494 1300 10 4 0 3 80 17 0 1 3612 187756 2872 3413420 0 0 44 24608 4274 10679 0 12 72 17 0 1 3612 153196 2900 3446100 0 0 28 17248 3349 6018 0 7 72 20 0 4 3612 134804 2908 3453980 0 0 8 61140 1795 4528 0 3 31 65 1 0 3612 144592 2920 3458796 0 0 4 18972 1689 2102 0 2 37 62 0 2 3612 86936 2980 3512048 0 0 56 26656 4554 14563 0 13 69 18 0 2 3612 45424 3024 3552532 0 0 40 19076 3774 8148 0 9 71 20 0 1 3612 15792 3048 3579368 0 0 44 25696 4430 11954 0 12 72 16 0 4 3612 7892 3064 3576972 0 0 16 74808 2430 12205 0 6 57 37 0 1 3612 17828 3080 3581512 0 0 8 18832 2423 6117 0 5 50 45 1 0 3612 16944 3068 3581048 0 0 40 26816 4126 15803 0 12 66 22 0 0 3612 16032 3128 3578200 0 0 60 32640 5137 23987 0 15 68 17 1 0 3612 15664 3144 3576620 0 0 56 33200 5178 17395 0 15 71 14 0 5 3612 7956 3160 3573272 0 0 20 74092 2077 5504 0 4 54 41 0 6 3612 12212 3168 3578568 0 0 8 30936 1964 3848 0 3 42 55 1 1 3612 15752 3200 3579760 0 0 24 22352 2975 9692 0 8 60 32 0 1 3612 16696 3240 3577340 0 0 40 22496 3954 15224 0 12 71 17 2 0 3612 15472 3212 3576824 0 0 40 26624 4466 14382 0 13 71 16 1 3 3612 5908 3228 3575924 0 0 16 72304 2328 8965 0 5 39 55 0 1 3612 14188 3244 3579512 0 0 8 17892 2169 4147 0 4 57 39 0 2 3612 17136 3292 3575520 0 0 56 28100 4672 16158 0 15 69 16 So, the bo, appears to be the problem. > BTW: I guess the processes wake up eventually, don't they? Yes, they eventually wake up for a while and then get in initerruptible sleep again Considering that the BO is the problem, what can I try to fix it? Thanks, -- Denis Anjos. Cisco Certified Network Associate. Universidade Federal do ABC Santo André - SP - BR ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta. 2007-12-17 16:41 ` Denis @ 2007-12-17 17:49 ` Jan Kara 0 siblings, 0 replies; 4+ messages in thread From: Jan Kara @ 2007-12-17 17:49 UTC (permalink / raw) To: Denis; +Cc: linux-kernel On Mon 17-12-07 14:41:18, Denis wrote: > 2007/12/17, Jan Kara <jack@suse.cz>: > > > Dears, I've got a double dual xeon wich sometimes is getting too slow. > > > > > > > > > While i was observing one of these slowly times, I realized the time > > > wait of processors come close to 100% and nfsd and kjournald process > > > become to D status (uniterruptible sleep) .Does someone know why this > > > can be happening or how can I troubleshoot it? > > > > > > Below is a 'draw' of my top at one of these moments. > > > > > > [root@cromo src]# top > > > top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00 > > > Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie > > > Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si > > > Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si > > > Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si > > > Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si > > All processors are in iowait - i.e. waiting for IO. > > > > > Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers > > > Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached > > > > > > 3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd > > > 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd > > > 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald > > > 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd > > > 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 > > > 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd > > > 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd > > > 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd > > It seems like heavy disk/network activity. You can try running > > vmstat 1 > > It will print every second the situation of your memory / IO > > subsystem. You should especially see the ---io--- column ('bi' means > > blocks in, 'bo' blocks outputted). I guess they'd contain some > > non-trivial values... > > The output of my vmstat look like this: > > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 1 3612 239600 2828 3364300 0 0 1494 1300 10 4 0 3 80 17 > 0 1 3612 187756 2872 3413420 0 0 44 24608 4274 10679 0 12 72 17 > 0 1 3612 153196 2900 3446100 0 0 28 17248 3349 6018 0 7 72 20 > 0 4 3612 134804 2908 3453980 0 0 8 61140 1795 4528 0 3 31 65 > 1 0 3612 144592 2920 3458796 0 0 4 18972 1689 2102 0 2 37 62 > 0 2 3612 86936 2980 3512048 0 0 56 26656 4554 14563 0 13 69 18 > 0 2 3612 45424 3024 3552532 0 0 40 19076 3774 8148 0 9 71 20 > 0 1 3612 15792 3048 3579368 0 0 44 25696 4430 11954 0 12 72 16 > 0 4 3612 7892 3064 3576972 0 0 16 74808 2430 12205 0 6 57 37 > 0 1 3612 17828 3080 3581512 0 0 8 18832 2423 6117 0 5 50 45 > 1 0 3612 16944 3068 3581048 0 0 40 26816 4126 15803 0 12 66 22 > 0 0 3612 16032 3128 3578200 0 0 60 32640 5137 23987 0 15 68 17 > 1 0 3612 15664 3144 3576620 0 0 56 33200 5178 17395 0 15 71 14 > 0 5 3612 7956 3160 3573272 0 0 20 74092 2077 5504 0 4 54 41 > 0 6 3612 12212 3168 3578568 0 0 8 30936 1964 3848 0 3 42 55 > 1 1 3612 15752 3200 3579760 0 0 24 22352 2975 9692 0 8 60 32 > 0 1 3612 16696 3240 3577340 0 0 40 22496 3954 15224 0 12 71 17 > 2 0 3612 15472 3212 3576824 0 0 40 26624 4466 14382 0 13 71 16 > 1 3 3612 5908 3228 3575924 0 0 16 72304 2328 8965 0 5 39 55 > 0 1 3612 14188 3244 3579512 0 0 8 17892 2169 4147 0 4 57 39 > 0 2 3612 17136 3292 3575520 0 0 56 28100 4672 16158 0 15 69 16 > > > So, the bo, appears to be the problem. Yes, that means that your system is constantly writing about 30 MB/s to disk. That's not too much but in case those writes are not sequential it can quite easily saturate your disk (depends on what disks you have)... > > BTW: I guess the processes wake up eventually, don't they? > Yes, they eventually wake up for a while and then get in > initerruptible sleep again > > Considering that the BO is the problem, what can I try to fix it? First, you could find out what is writing those data to disk (I guess the load is comming over NFS since nfsds are waiting for IO too). If you'd expect higher throughput from your disk subsystem, you can also verify that the disk is seeking a lot (for example by blktrace). Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2007-12-17 17:49 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <e9aeeef80712141011i139e6ec2pe04250bd56f338d8@mail.gmail.com>
[not found] ` <e9aeeef80712141211v7cec29fdt7e2ec76bf877053f@mail.gmail.com>
2007-12-14 21:33 ` Fwd: NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta Denis
2007-12-17 15:32 ` Jan Kara
2007-12-17 16:41 ` Denis
2007-12-17 17:49 ` Jan Kara
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®