Poor scheduling when not loaded at 100% (Was: [PATCH] sched.c: Be a bit more conservative in SMP)

[Ludovic Drolez <ldrolez@linbox.com>]

Ludovic Drolez <ldrolez <at> linbox.com> writes:
> In fact, I tested the 1st patch on our cluster (Finite elements computing on 8 
> CPUs):
> - Under Windows: 875 seconds
> - Linux 2.6.16 : 1019 s
> - Linux 2.6.16 + manual taskset : 842 s
> - Linux 2.6.16 + Vincent's patch : 1373 s  

Anyone has an idea why the scheduling is poor when processes don't use all CPU ?

In the above example, we have 4 processes on 4 processors which use about 40% of
the CPU (computing and waiting for network packets).
1- If taskset is not used : CPU0 is used at 80%, and the 3 others at 30%. The
tasks are constantly migrated between cores -> poor performance (1019s), Windows
does better :-(
2- If taskset is used : All CPUs have 1 process and are used at 40%. No
migration -> high performance (842s), better than Windows :-)

I tried to play with the 'migration_cost' kernel parameter but it did not help.
By default, on the Bi-Xeon Dual Core MB (Dell 1855), migration_cost=1600, and
trying values up to 200000, did not improve performance...

Any Ideas ?

  Ludovic Drolez.