Hi Peter, On Wed, Jan 04, 2006 at 12:25:40PM +1100, Peter Williams wrote: > Peter Williams wrote: > >Helge Hafting wrote: > > > >>On Wed, Dec 21, 2005 at 05:32:52PM +1100, Peter Williams wrote: > >> > >>>Trond Myklebust wrote: > >> > >> > >>[...] > >> > >>>>Sorry. That theory is just plain wrong. ALL of those case _ARE_ > >>>>interactive sleeps. > >>> > >>> > >>>It's not a theory. It's a result of observing a -j 16 build with the > >>>sources on an NFS mounted file system with top with and without the > >>>patches and comparing that with the same builds with the sources on a > >>>local file system. Without the patches the tasks in the kernel build > >>>all get the same dynamic priority as the X server and other > >>>interactive programs when the sources are on an NFS mounted file > >>>system. With the patches they generally have dynamic priorities > >>>between 6 to 10 higher than the X server and other interactive programs. > >>> > >> > >>A process waiting for NFS data looses cpu time, which is spent on > >>running something else. Therefore, it gains some priority so it won't be > >>forever behind when it wakes up. Same as for any other io waiting. > > > > > >That's more or less independent of this issue as the distribution of CPU > >to tasks is largely determined by the time slice mechanism and the > >dynamic priority is primarily about latency. (This distinction is a > >little distorted by the fact that, under some circumstances, > >"interactive" tasks don't get moved to the expired list at the end of > >their time slice but this usually won't matter as genuine interactive > >tasks aren't generally CPU hogs.) In other words, the issue that you > >raised is largely solved by the time tasks spend on the active queue > >before moving to the expired queue rather than the order in which they > >run when on the active queue. > > > >This problem is all about those tasks getting an inappropriate boost to > >improve their latency because they are mistakenly believed to be > >interactive. > > One of the unfortunate side effects of this is that it can effect > scheduler fairness because if these tasks get sufficient bonus points > the TASK_INTERACTIVE() macro will return true for them and they will be > rescheduled on the active queue instead of the expired queue at the end > of the time slice (provided EXPIRED_STARVING()) doesn't prevent this). > This will have an adverse effect on scheduling fairness. > > The ideal design of the scheduler would be for the fairness mechanism > and the interactive responsiveness mechanism to be independent but this > is not the case due to the fact that requeueing interactive tasks on the > expired array could add unacceptably to their latency. As I said above > this slight divergence from the ideal of perfect independence shouldn't > matter as genuine interactive processes aren't very CPU intensive. > > In summary, inappropriate identification of CPU intensive tasks as > interactive has two bad effects: 1) responsiveness problems for genuine > interactive tasks due to the extra competition at their dynamic priority > and 2) a degradation of scheduling fairness; not just one. > > For an example of the effect of inappropriate identification of CPU hogs > as interactive tasks see the thread "[SCHED] Totally WRONG priority > calculation with specific test-case (since 2.6.10-bk12)" in this list. And another real-life example of the issue you describe above. >From marcelo.tosatti@cyclades.com Fri Dec 2 18:51:59 2005 Date: Fri, 2 Dec 2005 18:51:59 -0200 From: Marcelo Tosatti To: Ingo Molnar , Nick Piggin Cc: Regina Kodato , Wanda Rosalino , Edson Seabra Subject: scheduler starvation with v2.6.11 on embedded PPC appliance We are experiencing what seems to be a scheduler starvation issue on our application, running v2.6.11. The same load works as expected on v2.4. We would like to know if v2.6.14 could possibly fix this problem. Hardware is a PowerPC 8xx at 48Mhz (embedded SoC) with 128MB RAM, handling remote access to its own 48 serial ports running at 9600bps each (8N1, HW flow control). Access to the ports is performed via SSH (one sshd instance for each port), and there are two different configurations: 1) slim socket mode: Each SSH process is responsible for handling IO to its own serial port. 2) buffering mode: Where a single process handles IO on the 48 tty's, copying data to a shared memory region and signalling the respective ssh daemon with SIGIO once a certain amount of data is ready. The test transfers a 78k file via each serial port (total = 48*78k = 3.7MB) from an x86 Linux box, usually taking: 78110 bytes after 81 seconds, 964 cps (+-9640 bps). Time varies from 77 sec upto 85 sec. Problem description: Using slim socket mode, where each SSH process handles IO to its own port, the scheduler starves a certain number of processes, causing their connections to timeout. Further investigation with schedstats allowed us to notice that "wait_ticks" is much higher using this mode. Follows the output of "latency" and "vmstat 2" with buffering mode (low wait_ticks, high number of context switches): 913 (cy_buffering) 25(25) 1077(1077) 843(843) 0.03 1.28 1166 (sshd) 220(220) 143(143) 1276(1276) 0.17 0.11 913 (cy_buffering) 36(11) 1078(1) 952(109) 0.10 0.01 1166 (sshd) 231(11) 191(48) 1883(607) 0.02 0.08 913 (cy_buffering) 242(206) 1131(53) 3200(2248) 0.09 0.02 1166 (sshd) 294(63) 383(192) 2523(640) 0.10 0.30 913 (cy_buffering) 440(198) 1172(41) 5637(2437) 0.08 0.02 1166 (sshd) 353(59) 574(191) 3160(637) 0.09 0.30 913 (cy_buffering) 644(204) 1199(27) 7918(2281) 0.09 0.01 1166 (sshd) 372(19) 678(104) 3771(611) 0.03 0.17 913 (cy_buffering) 644(0) 1201(2) 7978(60) 0.00 0.03 1166 (sshd) 372(0) 681(3) 4372(601) 0.00 0.00 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy wa id 0 0 0 159752 51200 9960 0 0 0 0 23 1171 1 11 0 88 0 0 0 159752 51200 9960 0 0 0 0 10 1111 0 5 0 94 1 0 0 159752 51200 9964 0 0 2 0 311 1226 35 55 0 10 1 0 0 159752 51200 9964 0 0 0 0 934 1718 50 50 0 0 1 0 0 159752 51200 9964 0 0 0 0 874 1519 52 48 0 0 11 0 0 159752 51200 9964 0 0 0 0 800 1358 47 53 0 0 7 0 0 159752 51200 9964 0 0 0 0 527 1235 44 56 0 0 1 0 0 159752 51200 9964 0 0 0 0 301 1144 47 53 0 0 1 0 0 159752 51200 9964 0 0 0 0 363 1241 43 57 0 0 2 0 0 159752 51200 9964 0 0 0 1 428 1194 45 55 0 0 1 0 0 159752 51200 9964 0 0 0 0 428 1141 42 58 0 0 1 0 0 159752 51200 9964 0 0 0 0 433 1255 44 56 0 0 2 0 0 159752 51200 9964 0 0 0 0 444 1067 46 54 0 0 1 0 0 159752 51200 9964 0 0 0 0 465 1071 55 45 0 0 1 0 0 159752 51200 9964 0 0 0 0 510 1101 42 58 0 0 1 0 0 159752 51200 9964 0 0 0 0 409 1082 47 53 0 0 1 0 0 159752 51200 9964 0 0 0 0 401 1075 40 60 0 0 1 0 0 159752 51200 9964 0 0 0 0 409 1081 44 56 0 0 And with slim socket mode (very high wait_ticks, low number of context switches): 1200 (sshd) 382(0) 3891(0) 1879(30) 0.00 0.00 1216 (sshd) 479(0) 7216(0) 2387(30) 0.00 0.00 1241 (sshd) 802(0) 6869(2) 4069(31) 0.00 0.06 1276 (sshd) 499(2) 8807(42) 3204(34) 0.06 1.24 1301 (sshd) 601(2) 8319(38) 2752(32) 0.06 1.19 1200 (sshd) 388(6) 4184(293) 1909(30) 0.20 9.77 1216 (sshd) 487(8) 7516(300) 2413(26) 0.31 11.54 1241 (sshd) 866(64) 7575(706) 4427(358) 0.18 1.97 1276 (sshd) 656(157) 9824(1017) 3756(552) 0.28 1.84 1301 (sshd) 610(9) 8422(103) 2761(9) 1.00 11.44 1200 (sshd) 415(27) 7132(2948) 1982(73) 0.37 40.38 1216 (sshd) 511(24) 10537(3021) 2496(83) 0.29 36.40 1241 (sshd) 943(77) 8537(962) 4875(448) 0.17 2.15 1276 (sshd) 776(120) 10892(1068) 4336(580) 0.21 1.84 1301 (sshd) 620(10) 11034(2612) 2771(10) 1.00 261.20 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy wa id 5 0 0 159816 51200 9916 0 0 0 0 18 113 0 1 0 99 0 0 0 159816 51200 9916 0 0 0 0 19 112 0 2 0 98 0 0 0 159816 51200 9916 0 0 0 0 166 176 1 6 0 93 37 0 0 159880 51200 9916 0 0 0 0 2857 1219 46 50 0 4 38 0 0 159880 51200 9916 0 0 0 0 2662 1059 58 42 0 0 33 0 0 159880 51200 9916 0 0 0 0 1058 496 72 28 0 0 33 0 0 159880 51200 9916 0 0 0 0 1593 743 70 30 0 0 33 0 0 159880 51200 9916 0 0 0 0 1519 706 71 29 0 0 34 0 0 159880 51200 9916 0 0 0 0 1073 520 74 26 0 0 35 0 0 159880 51200 9916 0 0 0 0 1047 493 67 33 0 0 49 0 0 159880 51200 9916 0 0 0 0 1130 543 70 30 0 0 34 0 0 159880 51200 9916 0 0 0 0 1239 612 70 30 0 0 46 0 0 159880 51200 9916 0 0 0 0 1427 737 69 31 0 0 34 0 0 159880 51200 9916 0 0 0 0 835 423 73 27 0 0 36 0 0 159880 51200 9916 0 0 0 1 1036 414 69 31 0 0 37 0 0 159880 51200 9916 0 0 0 0 917 379 73 27 0 0 44 0 0 159880 51200 9916 0 0 0 0 3401 1311 65 35 0 0 Another noticeable difference on schedstat output is that slim mode causes the scheduler to switch the active/expired queues 4 times during the total run, while buffering mode switches the queues 38 times. Attached you can find schedstats-buffering.txt and schedstats-slim.txt. On v2.4.17 both modes work fine, with a high context-switch number. We suspected that the TASK_INTERACTIVE() logic in kernel/sched.c would be moving some processes directly to the active list, thus starving some others. So we set the nice value of all 48 processes to "nice +19" to disable TASK_INTERACTIVE() and the starvation is gone. However with +19 it becomes impossible to use the box interactively while the test runs, which is the case with the default "0" nice value. Are there significant changes between v2.6.11 -> v2.6.14 aimed at fixing this problem?