Status field in ps output for threaded programs in 2.6 kernel

["Nikita V. Youshchenko" <yoush@cs.msu.su>]

Hello.

Today I faced a situation that confused me.

One moment I found that on our server there is almost 100% CPU load in 
system state. However, ps and top and other tools could not locate any 
process eating CPU.

The situation was permanent - several minutes passed, still almost 100% CPU 
in system state, still no visible running processes.

I executed chkrootkit and it found lots of "hidden processes" - ones for 
which /proc/N/ exists (i.e. it is possible to cd there), but N is not in 
ps output (and even not in ls /proc output).

I checked /proc/N/cmdline for some of such "hidden" pids; it looks like 
they represent threads. 
Well, maybe it is how NPTL threads work: for each thread a hidden /proc 
entry exists.
And it is chkrootkit's problem that is reports this (normal) case as 
possible LKM installed.

However, in my case, CPU-eating entities were found among thise hidden 
pids! They belonged to the same process (locally developed app, probably 
hanged), and this process was listed in ps output with status "S". After 
process was killed, CPU load went to 0%.

Looks like kernel bug: displayed status of threaded process should not be 
"S" if some threads are active. And CPU usage statistics should correspond 
to all threads.

This happened on a Debian testing/unstable system, running kernel package 
kernel-image-2.6.5-1-k7-smp 2.6.5-4.