'D' processes on a healthy system?

'D' processes on a healthy system?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2447 bytes --]

[please CC me on replies]

Hi folks,

I just pulled up a fresh, powerful, and errorfree server on Debian
woody, with some packages from testing. It's running a vanilla 2.4.19
kernel with the grsecurity 1.9.7d and freeswan 1.99 patches. This is
an AMD Athlon Duron 1.2 GHz with 512 Mb of SD-RAM and a 7,200 UPM HDD
by Maxtor. Reiserfs is used as filesystem.

I also have a machine that's about a year old, still running a vanilla
2.4.12 kernel without any patches. It's an AMD K6-2 500 MHz with 192
Mb of RAM and a 5,400 UPM Western Digital drive. ext2 is the
filesystem here.

When either machine becomes loaded, certain processes become unusable
for a couple of seconds. top shows me this:

19268 madduck   18   0  2208 2208  1076 D     1.3  0.4   0:00 sanitizer
    6 root       9   0     0    0     0 DW    0.3  0.0   1:10 kupdated
 8457 root      10   0  1156 1156   820 R     0.3  0.2   0:02 top
10843 postfix    9   0  1364 1364  1016 D     0.3  0.2   0:00 cleanup
 3156 madduck    0   0   636  636   540 D     0.1  0.1   0:00 procmail
28356 root       9   0   292  288   240 R     0.0  0.0   0:01 supervise
28706 root       9   0  2060 2036  1724 D     0.0  0.4   0:00 sshd
21395 root      15   0  1944 1944  1876 D     0.0  0.3   0:00 zsh

notice the number of processes in       ^
uninterruptible sleep mode in this column.

This was after i did something like:

  while true; do echo test | sendmail madduck; mailq; done

When this state is reached, programs like mutt take 7 minutes to open
a mailbox of 100 messages. With the server specs, this should not
happen.

I have memtest86'd the RAM and ran badblocks over all partitions
without finding anything.

My laptop, which is running Debian testing/unstable is not showing
this behaviour, and its load goes far higher at times. I also run
various other servers, partially on P5-120 systems, vanilla 2.4.xx
kernels and Debian testing, and there are no such problems there.

What is this an indication of? Hardware problems? Software problems?
Have you heard of this before? How can I fix it?

Thanks,

[Please CC me on replies]

-- 
 .''`.     martin f. krafft <madduck@debian.org>
: :'  :    proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
NOTE: The public PGP keyservers are broken!
Get my key here: http://people.debian.org/~madduck/gpg/330c4a75.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 'D' processes on a healthy system?

[Alan Cox]

On Thu, 2002-12-19 at 12:40, martin f krafft wrote:
> [please CC me on replies]
> 19268 madduck   18   0  2208 2208  1076 D     1.3  0.4   0:00 sanitizer
>     6 root       9   0     0    0     0 DW    0.3  0.0   1:10 kupdated
>  8457 root      10   0  1156 1156   820 R     0.3  0.2   0:02 top
> 10843 postfix    9   0  1364 1364  1016 D     0.3  0.2   0:00 cleanup
>  3156 madduck    0   0   636  636   540 D     0.1  0.1   0:00 procmail
> 28356 root       9   0   292  288   240 R     0.0  0.0   0:01 supervise
> 28706 root       9   0  2060 2036  1724 D     0.0  0.4   0:00 sshd
> 21395 root      15   0  1944 1944  1876 D     0.0  0.3   0:00 zsh
> 
> notice the number of processes in       ^
> uninterruptible sleep mode in this column.

Your disk is too slow for the work being asked of it, thats all.
Eventually it'll get there

> My laptop, which is running Debian testing/unstable is not showing
> this behaviour, and its load goes far higher at times. I also run
> various other servers, partially on P5-120 systems, vanilla 2.4.xx
> kernels and Debian testing, and there are no such problems there.

sendmail tuning ?

Re: 'D' processes on a healthy system?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2820 bytes --]

[please continue to CC me]

Thank you for your reply:

also sprach Alan Cox <alan@lxorguk.ukuu.org.uk> [2002.12.19.1843 +0100]:
> Your disk is too slow for the work being asked of it, thats all.
> Eventually it'll get there

Alan, I am in no position to doubt what you say, but I can't imagine
that. Sure, maybe the 5,400 RPM one, but not the 7,200 RPM one.

The reason why I am saying this is twofold and empirical:

  - When the above occurs, the system in question might not be doing
    anything. My example with /usr/sbin/sendmail in a while loop is
    hardcore stresstesting. I have had the problem with no users on
    the system, no requests being served by the servers (ifconfig
    down), just two ssh connections, one displaying top, the other
    opening a Maildir folder of 1,000 messages with mutt. I really
    don't (want to) believe that a system with these specs can't
    handle that.

  - I have another system with exactly the same specs (AMD K6 Duron
    1.2 GHz, 512 MB, 7,200 HDD) that is happy doing all of the
    following at the same time
      * compiling a kernel
      * streaming local MP3s to three other computers
      * being used intensely through X (it's my main computer).
    In fact, to verify this, I told the system to also check and
    update tripwire while I was additionally running the slocate
    updater. Other than the interactive use, these activities are
    very tough on the disk, and yet I see no 'D' processes.

In any case, loading up mutt on a Maildir folder of 1,000 messages
should not take seven Minutes. If it does, there must be heavy usage
of the disk from another source. If top doesn't show anything, what
other tool could I use to see what processes are accessing the
harddrive? Is there something like a disk monitor for Linux, which
registers every request to the HDD like there is for Windoze
(http://www.sysinternals.com/ntw2k/freeware/diskmon.shtml)?

> > My laptop, which is running Debian testing/unstable is not showing
> > this behaviour, and its load goes far higher at times. I also run
> > various other servers, partially on P5-120 systems, vanilla 2.4.xx
> > kernels and Debian testing, and there are no such problems there.
> 
> sendmail tuning ?

postfix... but no. All my machines have identical postfix
configurations, and, as mentioned above, the problem is not only
triggered when postfix is active...

Thank you for your time!

[please continue to CC me]

-- 
 .''`.     martin f. krafft <madduck@debian.org>
: :'  :    proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
NOTE: The pgp.net keyservers and their mirrors are broken!
Get my key here: http://people.debian.org/~madduck/gpg/330c4a75.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 'D' processes on a healthy system?

[Alan Cox]

On Thu, 2002-12-19 at 18:23, martin f krafft wrote:
> [please continue to CC me]
> 
> Thank you for your reply:
> 
> also sprach Alan Cox <alan@lxorguk.ukuu.org.uk> [2002.12.19.1843 +0100]:
> > Your disk is too slow for the work being asked of it, thats all.
> > Eventually it'll get there
> 
> Alan, I am in no position to doubt what you say, but I can't imagine
> that. Sure, maybe the 5,400 RPM one, but not the 7,200 RPM one.

Its more to do with the controller and configuration. Eg if your disk
isnt in DMA mode it'll certainly show up

Re: 'D' processes on a healthy system?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

also sprach Alan Cox <alan@lxorguk.ukuu.org.uk> [2002.12.19.2027 +0100]:
> Its more to do with the controller and configuration. Eg if your disk
> isnt in DMA mode it'll certainly show up

Interesting. The 500 MHz system wasn't in DMA mode (and I though I had
it there). I'll continue monitoring it now that I turned it on.

Thank you for your help so far, Alan!

-- 
 .''`.     martin f. krafft <madduck@debian.org>
: :'  :    proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
NOTE: The pgp.net keyservers and their mirrors are broken!
Get my key here: http://people.debian.org/~madduck/gpg/330c4a75.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 'D' processes on a healthy system?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2081 bytes --]

also sprach Alan Cox <alan@lxorguk.ukuu.org.uk> [2002.12.19.2027 +0100]:
> Its more to do with the controller and configuration. Eg if your disk
> isnt in DMA mode it'll certainly show up

i know took the system offline to test a little more. the harddrive is
operating fine without errors. i defined runlevel 4 to be single user
mode + sshd, now the system is running 14 processes including the
kernel processes.

hdparm shows this for /dev/hda:

/dev/hda:
 multcount    = 16 (on)
 I/O support  =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 2491/255/63, sectors = 40021632, start = 0
 busstate     =  1 (on)

correct me if i am wrong, but it is properly tweaked. moreover, lspci
shows that there is a VT82C598 [Apollo MVP3] VIA Chipset in there, and
my kernel config is optimized for that:

  CONFIG_BLK_DEV_IDEPCI=y
  CONFIG_IDEPCI_SHARE_IRQ=y
  CONFIG_BLK_DEV_IDEDMA_PCI=y
  CONFIG_IDEDMA_PCI_AUTO=y
  CONFIG_BLK_DEV_IDEDMA=y
  CONFIG_BLK_DEV_ADMA=y
  CONFIG_BLK_DEV_VIA82CXXX=y
  CONFIG_IDEDMA_AUTO=y
  CONFIG_BLK_DEV_IDE_MODES=y

Nevertheless, with 14 processes running and none of them accessing
the disk, i started an rsync process over ssh for the home partition.
and performance is ridiculous. rsync will transfer about 40k before
the rsync process enters 'D' state as shown by top. this takes about
10 seconds, then rsync gets to transfer another 40k.

this is on an AMD K6-2 500 MHz machine with 160 Mb RAM, 256Mb of swap
and a Maxtor 10Gb drive spinning at 5,400 I believe.

What's the problem?

-- 
Please do not CC me! Mutt (www.mutt.org) can handle this automatically.
 
 .''`.     martin f. krafft <madduck@debian.org>
: :'  :    proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
NOTE: The pgp.net keyservers and their mirrors are broken!
Get my key here: http://people.debian.org/~madduck/gpg/330c4a75.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 'D' processes on a healthy system?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

also sprach martin f krafft <madduck@debian.org> [2003.01.07.0901 +0100]:
> this is on an AMD K6-2 500 MHz machine with 160 Mb RAM, 256Mb of swap

sorry, 512 Mb of swap.

the kernel is 2.4.19 with the grsecurity patches. the problem did also
occur with 2.4.12 without the grsecurity patches.

-- 
Please do not CC me! Mutt (www.mutt.org) can handle this automatically.
 
 .''`.     martin f. krafft <madduck@debian.org>
: :'  :    proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
NOTE: The pgp.net keyservers and their mirrors are broken!
Get my key here: http://people.debian.org/~madduck/gpg/330c4a75.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 'D' processes on a healthy system?

[Alan Cox]

On Tue, 2003-01-07 at 08:01, martin f krafft wrote:
> 
> correct me if i am wrong, but it is properly tweaked. moreover, lspci
> shows that there is a VT82C598 [Apollo MVP3] VIA Chipset in there, and
> my kernel config is optimized for that:

Looks good yes.

	hdparm -t will give you the raw disk speed 

> and performance is ridiculous. rsync will transfer about 40k before
> the rsync process enters 'D' state as shown by top. this takes about
> 10 seconds, then rsync gets to transfer another 40k.
> 
> this is on an AMD K6-2 500 MHz machine with 160 Mb RAM, 256Mb of swap
> and a Maxtor 10Gb drive spinning at 5,400 I believe.
> 
> What's the problem?

No idea. strace the rsync see what its spending its time stuck doing 
(eg read from disk, or write to net etc). It'll probably show read from
disk if this is a disk problem