apache threads not balanced over all CPU cores, schedular problem or apache?

["kyle" <kylewong@southa.com>]

Hi,

Please cc me if possible.

I have a machine with Intel Q9300 4-core CPU, 8GB RAM. It runs apache 2.0.63
with traditional prefork MPM.
It's a high traffic site, most of the time there's over 100 httpd processes
running and at peak hours around 200.

I found that most of the time, all httpd processes are running at 2 CPU
cores only, sometime it's cpu0 and cpu1, sometime it's cpu1 and cpu3,
sometime it's cpu0 and cpu2 ... etc. It may take several hours the processes
"migrate" from 2 cores to another 2. To make it clear:

top - 12:44:42 up 8 days, 15:45,  7 users,  load average: 2.07, 1.86, 2.02
Tasks: 251 total,   1 running, 249 sleeping,   0 stopped,   1 zombie
Cpu0  :  0.5%us,  0.5%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu1  : 14.6%us,  4.4%sy,  0.0%ni, 62.6%id, 17.0%wa,  0.0%hi,  1.5%si,
0.0%st
Cpu2  :  4.3%us,  1.4%sy,  0.0%ni, 93.8%id,  0.0%wa,  0.5%hi,  0.0%si,
0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni, 98.0%id,  2.0%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   8117172k total,  8060936k used,    56236k free,   357488k buffers
Swap:   208824k total,     8436k used,   200388k free,  5765168k cached

All httpd are running on cpu1 and cpu2 only for several hours, after that,
all httpd may run on cpu1 and cpu3, or cpu2 and cpu3, etc. again for several
hours.

Since the apache is in prefork mode, every httpd instance should be quite
independant, I suppose the CPU schedular should balance the processes over
the avaiable cores. However never seen httpd processes evenly distributed to
3 cores, or 4 cores.

Any help?

Regards,
Kyle