I tested fair group scheduler on my hyper-threading x86_64 box(2 CPU * 2 HT) and found the deviation of the groups' CPU usage was larger than 2.6.26 when *offline* a CPU or do hotplug frequently. It is less than 1% On 2.6.26，but On current kernel, it is often greater than 4%, even than 10% by accident. A test program which reproduces the problem on current kernel is attached. This program forks a lot of child tasks and attachs these child tasks into each CPU controller group, then the parent task gets and checks the CPU usage of every group every 5 seconds.(All of the child tasks do the same work - repeat doing sqrt) Child Task 1 while(!end) sqrt(f); Group 1 ...... Child Task m while(!end) sqrt(f); ----------------------------- Parent Task Group 2 ...... get and check the CPU ...... ...... usage of every group ----------------------------- Child Task m*n-m+1 while(!end) sqrt(f); Group n ...... Child Task m*n while(!end) sqrt(f); Steps to reproduce: # mkdir /dev/cpuctl # mount -t cgroup -o cpu,noprefix xxx /dev/cpuctl # ./cpuctl output on current kernel: ------------------------- Group Shares Actual(%) Expect(%) 0 1024 41.68 50.00 1 1024 58.32 50.00 0th group's usage is out of range(deviation = 8.32) 1th group's usage is out of range(deviation = 8.32) -------------------------- output on 2.6.26 ------------------------- Group Shares Actual(%) Expect(%) 0 1024 50.03 50.00 1 1024 49.97 50.00 -------------------------- On 2.6.26, the deviation may be greater than 4% at the beginning, but the scheduler adjusts soon, and don't occur the large deviation for ever. Bisect located below patch: commit c09595f63bb1909c5dc4dca288f4fe818561b5f3 Author: Peter Zijlstra Date: Fri Jun 27 13:41:14 2008 +0200 sched: revert revert of: fair-group: SMP-nice for group scheduling Try again.. Initial commit: 18d95a2832c1392a2d63227a7a6d433cb9f2037e Revert: 6363ca57c76b7b83639ca8c83fc285fa26a7880e Signed-off-by: Peter Zijlstra Cc: Srivatsa Vaddagiri Cc: Mike Galbraith Signed-off-by: Ingo Molnar Besides that, We found other problems by the attached program. 1. some tasks are hungry in the fair group. Steps to reproduce: # mkdir /dev/cpuctl # mount -t cgroup -o cpu,noprefix xxx /dev/cpuctl # ./cpuctl -g 1 -v -------------------- 1th Check Result: Group Shares Actual(%) Expect(%) 0 1024 100.00 100.00 Each task's usage: Task in Group 0: Task Usage(%) 5395 0.000000 5396 0.000000 5397 0.000000 5398 16.677785 5399 16.677785 5400 16.744496 5401 16.611074 5402 33.288859 2. some groups broke the limit of the fair group and get more CPU time When the groups is hiberarchy. Such as: top group | group 1 / \ task1 group 2 | task 2 Steps to reproduce: # mkdir /dev/cpuctl # mount -t cgroup -o cpu,noprefix xxx /dev/cpuctl # ./cpuctl -H ------------------------- Group Shares Actual(%) Expect(%) 0 1024 60.17 88.89 1 1024 39.83 11.11