Re: [report] renicing X, cfs-v5 vs sd-0.46

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Niel Lambrechts]

Gene Heskett wrote:
> This message prompted me to do some checking in re context switches myself, 
> and I've come to the conclusion that there could be a bug in vmstat itself.

Perhaps. perhaps not. :)

> Run singly the context switching is reasonable even for a -19 niceness of x, 
> its only showing about 200 or so on the first loop of vmstat.  But throw in 
> the -n 1 arguments and it goes crazy on the second and subsequent loops.

man vmstat:
"The first report produced gives averages since the last reboot.
Additional reports  give information on a sampling period of length delay."

Regards,
Niel

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Gene Heskett]

On Monday 23 April 2007, Niel Lambrechts wrote:
>Gene Heskett wrote:
>> This message prompted me to do some checking in re context switches
>> myself, and I've come to the conclusion that there could be a bug in
>> vmstat itself.
>
>Perhaps. perhaps not. :)
>
>> Run singly the context switching is reasonable even for a -19 niceness of
>> x, its only showing about 200 or so on the first loop of vmstat.  But
>> throw in the -n 1 arguments and it goes crazy on the second and subsequent
>> loops.
>
>man vmstat:
>"The first report produced gives averages since the last reboot.
>Additional reports  give information on a sampling period of length delay."

I missed that, concentrating on finding the method of telling it the delay I 
guess.

So then the next question is, over what period is that obviously lower figure 
being averaged over?  Certainly not over a 1 second period else it would then 
be much higher, as seen by the figures after the initial delay.  The time 
slice spec'd in /proc/sys/kernel/sched_granularity_ns, which here is 
currently 5000000 or 5 milliseconds?  If that was the case, the first answer 
would be in the area of 15, not 200.

So educate me, off list if you would like and have the time.

Thanks Niel.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Sweet sixteen is beautiful Bess,
And her voice is changing -- from "No" to "Yes".

[patch] CFS scheduler, v4

[Ingo Molnar]

i'm pleased to announce release -v4 of the CFS patchset. The patch 
against v2.6.21-rc7 can be downloaded from:

    http://redhat.com/~mingo/cfs-scheduler/
 
this CFS release too is mainly about fixing regressions and improving 
interactivity, so the rate of change is relatively low:

    11 files changed, 136 insertions(+), 72 deletions(-)

in particular the preemption fix could resolve the 'desktop slows down 
under IO load' reports and the 'firefox does not switch tabs fast 
enough' reports as well. The suspend2 crash and the yield related 
Kaffeine hangs should be resolved as well.

Changes since -v3:

 - usability fix: automatic renicing of kernel threads such as keventd, 
   OOM tasks and tasks doing privileged hardware access (such as Xorg). 
   (This is a substitute for group scheduling until the group scheduling 
    details have been worked out.)

 - bugfix: buggy yield() caused suspend2 problems

 - preemption fix: it caused desktop app latencies

As usual, any sort of feedback, bugreport, fix and suggestion is more 
than welcome,

	Ingo

Re: [REPORT] cfs-v4 vs sd-0.44

[Con Kolivas]

On Saturday 21 April 2007 22:12, Willy Tarreau wrote:
> I promised to perform some tests on your code. I'm short in time right now,
> but I observed behaviours that should be commented on.

>   Feels even better, mouse movements are very smooth even under high load.
>   I noticed that X gets reniced to -19 with this scheduler. I've not looked
>   at the code yet but this looked suspicious to me.

Looks like this code does it:

+int sysctl_sched_privileged_nice_level __read_mostly = -19;

allows anything that sets sched_privileged_task one way or another gets 
nice -19, and this is enabled by default.

--- linux-cfs-2.6.20.7.q.orig/arch/i386/kernel/ioport.c
+++ linux-cfs-2.6.20.7.q/arch/i386/kernel/ioport.c

+	if (turn_on) {
+		if (!capable(CAP_SYS_RAWIO))
+			return -EPERM;
+		/*
+		 * Task will be accessing hardware IO ports,
+		 * mark it as special with the scheduler too:
+		 */
+		sched_privileged_task(current);
+	}

presumably that selects out X as a privileged task... and sets it to nice -19 
by default.

-- 
-ck

Re: [REPORT] cfs-v4 vs sd-0.44

[Ingo Molnar]

* Con Kolivas <kernel@kolivas.org> wrote:

> >   Feels even better, mouse movements are very smooth even under high 
> >   load. I noticed that X gets reniced to -19 with this scheduler. 
> >   I've not looked at the code yet but this looked suspicious to me. 
> >   I've reniced it to 0 and it did not change any behaviour. Still 
> >   very good.
> 
> Looks like this code does it:
> 
> +int sysctl_sched_privileged_nice_level __read_mostly = -19;

correct. Note that Willy reniced X back to 0 so it had no relevance on 
his test. Also note that i pointed this change out in the -v4 CFS 
announcement:

|| Changes since -v3:
||
||  - usability fix: automatic renicing of kernel threads such as 
||    keventd, OOM tasks and tasks doing privileged hardware access
||    (such as Xorg).

i've attached it below in a standalone form, feel free to put it into 
SD! :)

	Ingo

---
 arch/i386/kernel/ioport.c   |   13 ++++++++++---
 arch/x86_64/kernel/ioport.c |    8 ++++++--
 drivers/block/loop.c        |    5 ++++-
 include/linux/sched.h       |    7 +++++++
 kernel/sched.c              |   40 ++++++++++++++++++++++++++++++++++++++++
 kernel/workqueue.c          |    2 +-
 mm/oom_kill.c               |    4 +++-
 7 files changed, 71 insertions(+), 8 deletions(-)

Index: linux/arch/i386/kernel/ioport.c
===================================================================
--- linux.orig/arch/i386/kernel/ioport.c
+++ linux/arch/i386/kernel/ioport.c
@@ -64,9 +64,15 @@ asmlinkage long sys_ioperm(unsigned long
 
 	if ((from + num <= from) || (from + num > IO_BITMAP_BITS))
 		return -EINVAL;
-	if (turn_on && !capable(CAP_SYS_RAWIO))
-		return -EPERM;
-
+	if (turn_on) {
+		if (!capable(CAP_SYS_RAWIO))
+			return -EPERM;
+		/*
+		 * Task will be accessing hardware IO ports,
+		 * mark it as special with the scheduler too:
+		 */
+		sched_privileged_task(current);
+	}
 	/*
 	 * If it's the first ioperm() call in this thread's lifetime, set the
 	 * IO bitmap up. ioperm() is much less timing critical than clone(),
@@ -145,6 +151,7 @@ asmlinkage long sys_iopl(unsigned long u
 	if (level > old) {
 		if (!capable(CAP_SYS_RAWIO))
 			return -EPERM;
+		sched_privileged_task(current);
 	}
 	t->iopl = level << 12;
 	regs->eflags = (regs->eflags & ~X86_EFLAGS_IOPL) | t->iopl;
Index: linux/arch/x86_64/kernel/ioport.c
===================================================================
--- linux.orig/arch/x86_64/kernel/ioport.c
+++ linux/arch/x86_64/kernel/ioport.c
@@ -41,8 +41,11 @@ asmlinkage long sys_ioperm(unsigned long
 
 	if ((from + num <= from) || (from + num > IO_BITMAP_BITS))
 		return -EINVAL;
-	if (turn_on && !capable(CAP_SYS_RAWIO))
-		return -EPERM;
+	if (turn_on) {
+		if (!capable(CAP_SYS_RAWIO))
+			return -EPERM;
+		sched_privileged_task(current);
+	}
 
 	/*
 	 * If it's the first ioperm() call in this thread's lifetime, set the
@@ -113,6 +116,7 @@ asmlinkage long sys_iopl(unsigned int le
 	if (level > old) {
 		if (!capable(CAP_SYS_RAWIO))
 			return -EPERM;
+		sched_privileged_task(current);
 	}
 	regs->eflags = (regs->eflags &~ X86_EFLAGS_IOPL) | (level << 12);
 	return 0;
Index: linux/drivers/block/loop.c
===================================================================
--- linux.orig/drivers/block/loop.c
+++ linux/drivers/block/loop.c
@@ -588,7 +588,10 @@ static int loop_thread(void *data)
 	 */
 	current->flags |= PF_NOFREEZE;
 
-	set_user_nice(current, -20);
+	/*
+	 * The loop thread is important enough to be given a boost:
+	 */
+	sched_privileged_task(current);
 
 	while (!kthread_should_stop() || lo->lo_bio) {
 
Index: linux/include/linux/sched.h
===================================================================
--- linux.orig/include/linux/sched.h
+++ linux/include/linux/sched.h
@@ -1256,6 +1256,13 @@ static inline int rt_mutex_getprio(struc
 #endif
 
 extern void set_user_nice(struct task_struct *p, long nice);
+/*
+ * Task has special privileges, give it more CPU power:
+ */
+extern void sched_privileged_task(struct task_struct *p);
+
+extern int sysctl_sched_privileged_nice_level;
+
 extern int task_prio(const struct task_struct *p);
 extern int task_nice(const struct task_struct *p);
 extern int can_nice(const struct task_struct *p, const int nice);
Index: linux/kernel/sched.c
===================================================================
--- linux.orig/kernel/sched.c
+++ linux/kernel/sched.c
@@ -3251,6 +3251,46 @@ out_unlock:
 EXPORT_SYMBOL(set_user_nice);
 
 /*
+ * Nice level for privileged tasks. (can be set to 0 for this
+ * to be turned off)
+ */
+int sysctl_sched_privileged_nice_level __read_mostly = -19;
+
+static int __init privileged_nice_level_setup(char *str)
+{
+	sysctl_sched_privileged_nice_level = simple_strtoul(str, NULL, 0);
+	return 1;
+}
+__setup("privileged_nice_level=", privileged_nice_level_setup);
+
+/*
+ * Tasks with special privileges call this and gain extra nice
+ * levels:
+ */
+void sched_privileged_task(struct task_struct *p)
+{
+	long new_nice = sysctl_sched_privileged_nice_level;
+	long old_nice = TASK_NICE(p);
+
+	if (new_nice >= old_nice)
+		return;
+	/*
+	 * Setting the sysctl to 0 turns off the boosting:
+	 */
+	if (unlikely(!new_nice))
+		return;
+
+	if (new_nice < -20)
+		new_nice = -20;
+	else if (new_nice > 19)
+		new_nice = 19;
+
+	set_user_nice(p, new_nice);
+}
+
+EXPORT_SYMBOL(sched_privileged_task);
+
+/*
  * can_nice - check if a task can reduce its nice value
  * @p: task
  * @nice: nice value
Index: linux/kernel/workqueue.c
===================================================================
--- linux.orig/kernel/workqueue.c
+++ linux/kernel/workqueue.c
@@ -355,7 +355,7 @@ static int worker_thread(void *__cwq)
 	if (!cwq->freezeable)
 		current->flags |= PF_NOFREEZE;
 
-	set_user_nice(current, -5);
+	sched_privileged_task(current);
 
 	/* Block and flush all signals */
 	sigfillset(&blocked);
Index: linux/mm/oom_kill.c
===================================================================
--- linux.orig/mm/oom_kill.c
+++ linux/mm/oom_kill.c
@@ -291,7 +291,9 @@ static void __oom_kill_task(struct task_
 	 * all the memory it needs. That way it should be able to
 	 * exit() and clear out its resources quickly...
 	 */
-	p->time_slice = HZ;
+	if (p->policy == SCHED_NORMAL || p->policy == SCHED_BATCH)
+		sched_privileged_task(p);
+
 	set_tsk_thread_flag(p, TIF_MEMDIE);
 
 	force_sig(SIGKILL, p);

Re: [REPORT] cfs-v4 vs sd-0.44

[Con Kolivas]

On Sunday 22 April 2007 02:00, Ingo Molnar wrote:
> * Con Kolivas <kernel@kolivas.org> wrote:
> > >   Feels even better, mouse movements are very smooth even under high
> > >   load. I noticed that X gets reniced to -19 with this scheduler.
> > >   I've not looked at the code yet but this looked suspicious to me.
> > >   I've reniced it to 0 and it did not change any behaviour. Still
> > >   very good.
> >
> > Looks like this code does it:
> >
> > +int sysctl_sched_privileged_nice_level __read_mostly = -19;
>
> correct. 

Oh I definitely was not advocating against renicing X, I just suspect that 
virtually all the users who gave glowing reports to CFS comparing it to SD 
had no idea it had reniced X to -19 behind their back and that they were 
comparing it to SD running X at nice 0. I think had they been comparing CFS 
with X nice -19 to SD running nice -10 in this interactivity soft and squishy 
comparison land their thoughts might have been different. I missed it in the 
announcement and had to go looking in the code since Willy just kinda tripped 
over it unwittingly as well.

> Note that Willy reniced X back to 0 so it had no relevance on 
> his test.

Oh yes I did notice that, but since the array swap is the remaining longest 
deadline in SD which would cause noticeable jerks, renicing X on SD by 
default would make the experience very different since reniced tasks do much 
better over array swaps compared to non niced tasks. I really should go and 
make the whole thing one circular list and blow away the array swap (if I can 
figure out how to do it). 

> Also note that i pointed this change out in the -v4 CFS 
>
> announcement:
> || Changes since -v3:
> ||
> ||  - usability fix: automatic renicing of kernel threads such as
> ||    keventd, OOM tasks and tasks doing privileged hardware access
> ||    (such as Xorg).

Reading the changelog in the gloss-over fashion that I unfortunately did, even 
I missed it. 

> i've attached it below in a standalone form, feel free to put it into
> SD! :)

Hmm well I have tried my very best to do all the changes without 
changing "policy" as much as possible since that trips over so many emotive 
issues that noone can agree on, and I don't have a strong opinion on this as 
I thought it would be better for it to be a config option for X in userspace 
instead. Either way it needs to be turned on/off by admin and doing it by 
default in the kernel is... not universally accepted as good. What else 
accesses ioports that can get privileged nice levels? Does this make it 
relatively exploitable just by poking an ioport?

> 	Ingo
>
> ---
>  arch/i386/kernel/ioport.c   |   13 ++++++++++---
>  arch/x86_64/kernel/ioport.c |    8 ++++++--
>  drivers/block/loop.c        |    5 ++++-
>  include/linux/sched.h       |    7 +++++++
>  kernel/sched.c              |   40

Thanks for the patch. I'll consider it. Since end users are testing this in 
fuzzy interactivity land I may simply be forced to do this just for 
comparisons to be meaningful between CFS and SD otherwise they're not really 
comparing them on a level playing field. I had almost given up SD for dead 
meat with all the momentum CFS had gained... until recently.

-- 
-ck

Re: [REPORT] cfs-v4 vs sd-0.44

[Juliusz Chroboczek]

> Oh I definitely was not advocating against renicing X,

Why not do it in the X server itself?  This will avoid controversial
policy in the kernel, and have the added advantage of working with
X servers that don't directly access hardware.

Con, if you tell me ``if you're running under Linux and such and such
/sys variable has value so-and-so, then it's definitely a good idea to
call nice(42) at the X server's start up'', then I'll commit it into
X.Org.  (Please CC both me the list, so I can point any people
complaining to the archives.)

                                        Juliusz

Re: [REPORT] cfs-v4 vs sd-0.44

[Linus Torvalds]

On Sun, 22 Apr 2007, Juliusz Chroboczek wrote:
> 
> Why not do it in the X server itself?  This will avoid controversial
> policy in the kernel, and have the added advantage of working with
> X servers that don't directly access hardware.

It's wrong *wherever* you do it.

The X server should not be re-niced. It was done in the past, and it was 
wrogn then (and caused problems - we had to tell people to undo it, 
because some distros had started doing it by default).

If you have a single client, the X server is *not* more important than the 
client, and indeed, renicing the X server causes bad patterns: just 
because the client sends a request does not mean that the X server should 
immediately be given the CPU as being "more important". 

In other words, the things that make it important that the X server _can_ 
get CPU time if needed are all totally different from the X server being 
"more important". The X server is more important only in the presense of 
multiple clients, not on its own! Needing to renice it is a hack for a bad 
scheduler, and shows that somebody doesn't understand the problem!

		Linus

[report] renicing X, cfs-v5 vs sd-0.46

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> The X server should not be re-niced. It was done in the past, and it 
> was wrogn then (and caused problems - we had to tell people to undo 
> it, because some distros had started doing it by default).
> 
> If you have a single client, the X server is *not* more important than 
> the client, and indeed, renicing the X server causes bad patterns: 
> just because the client sends a request does not mean that the X 
> server should immediately be given the CPU as being "more important".

You are completely right in the case of traditional schedulers.

Note that this is not the case for CFS though. CFS has natural, built-in 
buffering against high-rate preemptions from lower nice-level 
SCHED_OTHER tasks. So while X will indeed get more CPU time (and that i 
think is fully justified), it wont get nearly as high of a 
context-switch rate as under priority/runqueue-based schedulers.

To demonstrate this i have done the following simple experiment: i 
started 4 xterms on a single-CPU box, then i started the 'yes' utility 
in each xterm and resized all of the xterms to just 2 lines vertical. 
This generates a _lot_ of screen refresh events. Naturally, such a 
workload utilizes the whole CPU.

Using CFS-v5, with Xorg at nice 0, the context-switch rate is low:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 472132  13712 178604    0    0     0    32  113  170 83 17  0  0  0
 2  0      0 472172  13712 178604    0    0     0     0  112  184 85 15  0  0  0
 2  0      0 472196  13712 178604    0    0     0     0  108  162 83 17  0  0  0
 1  0      0 472076  13712 178604    0    0     0     0  115  189 86 14  0  0  0

X's CPU utilization is 49%, xterm's go to 12% each. Userspace 
utilization is 85%, system utilization is 15%.

Renicing X to -10 increases context-switching, but not dramatically so, 
because it is throttled by CFS:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0      0 475752  13492 176320    0    0     0    64  116 1498 85 15  0  0  0
 4  0      0 475752  13492 176320    0    0     0     0  107 1488 84 16  0  0  0
 4  0      0 475752  13492 176320    0    0     0     0  140 1514 86 14  0  0  0
 4  0      0 475752  13492 176320    0    0     0     0  107 1477 85 15  0  0  0
 4  0      0 475752  13492 176320    0    0     0     0  122 1498 84 16  0  0  0

The system is still usable, Xorg is 44% busy, each xterm is 14% busy. 
User utilization 85%, system utilization is 15% - just like in the first 
case.

"Performance of scrolling" is exactly the same in both cases (i have 
tested this by inserting periodic beeps after every 10,000 lines of text 
scrolled) - but the screen refresh rate is alot more eye-pleasing in the 
nice -10 case. (screen refresh it happens at ~500 Hz, while in the nice 
0 case it happens at ~40 Hz and visibly flickers. This is especially 
noticeable if the xterms have full size.)

I have tested the same workload on vanilla v2.6.21-rc7 and on SD-0.46
too, and they give roughly the same xterm scheduling behavior when Xorg 
is at nice 0:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0      0 450564  14844 194976    0    0     0     0  287  594 58 10 32  0  0
 4  0      0 450704  14844 194976    0    0     0     0  108  370 89 11  0  0  0
 0  0      0 449588  14844 194976    0    0     0     0  175  434 85 13  2  0  0
 3  0      0 450688  14852 194976    0    0     0    32  242  315 62  9 29  0  0

but when Xorg is reniced to -10 on the vanilla or SD schedulers, it 
indeed gives the markedly higher context-switching behavior you 
predicted:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  0      0 452272  13936 194896    0    0     0     0  126 14147 78 22  0  0  0
 4  0      0 452252  13944 194896    0    0     0    64  155 14143 80 20  0  0  0
 5  0      0 452612  13944 194896    0    0     0     0  187 14031 79 21  0  0  0
 4  0      0 452624  13944 194896    0    0     0     0  121 14300 82 18  0  0  0

User time drops to 78%, system time increases to 22%. "Scrolling 
performance" clearly decreases.

so i agree that renicing X can be a very bad idea, but it very much 
depends on the scheduler implementation too.

	Ingo

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Linus Torvalds]

On Mon, 23 Apr 2007, Ingo Molnar wrote:
> 
> You are completely right in the case of traditional schedulers.

And apparently I'm completely right with CFS too.

> Using CFS-v5, with Xorg at nice 0, the context-switch rate is low:
> 
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>  2  0      0 472132  13712 178604    0    0     0    32  113  170 83 17  0  0  0
>  2  0      0 472172  13712 178604    0    0     0     0  112  184 85 15  0  0  0
>  2  0      0 472196  13712 178604    0    0     0     0  108  162 83 17  0  0  0
>  1  0      0 472076  13712 178604    0    0     0     0  115  189 86 14  0  0  0

Around 170 context switches per second.

> Renicing X to -10 increases context-switching, but not dramatically so, 
> because it is throttled by CFS:
> 
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>  4  0      0 475752  13492 176320    0    0     0    64  116 1498 85 15  0  0  0
>  4  0      0 475752  13492 176320    0    0     0     0  107 1488 84 16  0  0  0
>  4  0      0 475752  13492 176320    0    0     0     0  140 1514 86 14  0  0  0
>  4  0      0 475752  13492 176320    0    0     0     0  107 1477 85 15  0  0  0
>  4  0      0 475752  13492 176320    0    0     0     0  122 1498 84 16  0  0  0

Did you even *look* at your own numbers? Maybe you looked at "interrpts". 
The context switch numbers go from 170 per second, to 1500 per second!

If that's not "dramatically so", I don't know what is! Just how many 
orders of magnitude worse does it have to be, to be "dramatic"? Apparently 
one order of magnitude isn't "dramatic"?

So you were wrong. The fact that it was still "usable" is a good 
indication, but how about just admitting that you were wrong, and that 
renicing X is the *WRONG*THING*TO*DO*.

Just don't do it. It's wrong. It was wrong with the old schedulers, it's 
wrong with the new scheduler, it's just WRONG.

It was a hack, and it's a failed hack. And the fact that you don't seem to 
realize that it's a failure, even when your OWN numbers clearly show that 
it's failed, is a bit scary.

		Linus

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Gene Heskett]

On Monday 23 April 2007, Linus Torvalds wrote:
>On Mon, 23 Apr 2007, Ingo Molnar wrote:
>> You are completely right in the case of traditional schedulers.
>
>And apparently I'm completely right with CFS too.
>
>> Using CFS-v5, with Xorg at nice 0, the context-switch rate is low:
>>
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in
>>   cs us sy id wa st 2  0      0 472132  13712 178604    0    0     0    32
>>  113  170 83 17  0  0  0 2  0      0 472172  13712 178604    0    0     0 
>>    0  112  184 85 15  0  0  0 2  0      0 472196  13712 178604    0    0  
>>   0     0  108  162 83 17  0  0  0 1  0      0 472076  13712 178604    0  
>>  0     0     0  115  189 86 14  0  0  0
>
>Around 170 context switches per second.
>
>> Renicing X to -10 increases context-switching, but not dramatically so,
>> because it is throttled by CFS:
>>
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in
>>   cs us sy id wa st 4  0      0 475752  13492 176320    0    0     0    64
>>  116 1498 85 15  0  0  0 4  0      0 475752  13492 176320    0    0     0 
>>    0  107 1488 84 16  0  0  0 4  0      0 475752  13492 176320    0    0  
>>   0     0  140 1514 86 14  0  0  0 4  0      0 475752  13492 176320    0  
>>  0     0     0  107 1477 85 15  0  0  0 4  0      0 475752  13492 176320  
>>  0    0     0     0  122 1498 84 16  0  0  0
>
>Did you even *look* at your own numbers? Maybe you looked at "interrpts".
>The context switch numbers go from 170 per second, to 1500 per second!
>
>If that's not "dramatically so", I don't know what is! Just how many
>orders of magnitude worse does it have to be, to be "dramatic"? Apparently
>one order of magnitude isn't "dramatic"?
>
>So you were wrong. The fact that it was still "usable" is a good
>indication, but how about just admitting that you were wrong, and that
>renicing X is the *WRONG*THING*TO*DO*.
>
>Just don't do it. It's wrong. It was wrong with the old schedulers, it's
>wrong with the new scheduler, it's just WRONG.
>
>It was a hack, and it's a failed hack. And the fact that you don't seem to
>realize that it's a failure, even when your OWN numbers clearly show that
>it's failed, is a bit scary.
>
>		Linus

This message prompted me to do some checking in re context switches myself, 
and I've come to the conclusion that there could be a bug in vmstat itself.

Run singly the context switching is reasonable even for a -19 niceness of x, 
its only showing about 200 or so on the first loop of vmstat.  But throw in 
the -n 1 arguments and it goes crazy on the second and subsequent loops.

X nice=0
[root@coyote ~]# vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 3  0    324  62836  37952 518080    0    0   786   446  474  201 10  4 82  4  
0
 0  0    324  62712  37952 518080    0    0     0     0 1309 2361  2  5 93  0  
0
 2  0    324  62712  37952 518080    0    0     0     0 1275 2203  2  4 94  0  
0
 0  0    324  62744  37952 518080    0    0     0     0 1305 2224  1  2 97  0  
0
 0  0    324  62744  37952 518080    0    0     0     0 1291 2232  0  1 99  0  
0

X nice=-10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 3  0    324  62432  38052 518080    0    0   784   445  476  205 10  4 82  4  
0
 0  0    324  62432  38052 518080    0    0     0     0 1190 3223  1  1 98  0  
0
 2  0    324  62440  38052 518080    0    0     0     0 1209 3210  2  3 95  0  
0
 0  0    324  62316  38060 518080    0    0     0   232 1201 3355  3  4 92  1  
0
 2  0    324  62316  38060 518080    0    0     0     0 1207 2794  1  2 97  0  
0

X nice=10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 4  0    324  62372  38184 518132    0    0   783   445  477  209 10  4 82  4  
0
 0  0    324  62372  38192 518132    0    0     0   272 1318 2262  0  3 97  0  
0
 0  0    324  62372  38192 518132    0    0     0     0 1293 2249  1  4 95  0  
0
 0  0    324  62248  38192 518132    0    0     0     0 1280 2443  4  2 94  0  
0
 0  0    324  62248  38192 518132    0    0     0     4 1294 2272  0  3 97  0  
0

Now, I have NDI which set of figures is the true set, but please note that in 
all 3 cases the reported values for cs didn't scale up and down all that much 
if separated out into 1st pass, and subsequent passes.

And, even with X nice=10, the system is still fairly smooth and usable.

This is with 2.6.21-rc7-CFS-v5 I built late last evening.  At Xnice=10 I just 
played a game of patience to watch the card animations and they were 
absolutely acceptably smooth. (and I won it in about 112 moves :)

>From this users viewpoint, it (cfs-v5) works, and works very well indeed, and 
it deserves a place as one of 3 selectable options in mainline.  The other 2 
being the existing mainline way, & Con K's sd-0.45 or later.  Both of these 
seem to be very large enhancements to the user experience over current 
mainline, which I'd discuss in terms borrowed from Joanne Dow.  Comparatively 
speaking, mainline has a very high vacuum.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Jayne: "Let's move this conversation in a not-Jayne's-fault direction."
				--Episode #14, "Objects in Space"

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Gene Heskett]

On Monday 23 April 2007, Linus Torvalds wrote:
>On Mon, 23 Apr 2007, Ingo Molnar wrote:
>> You are completely right in the case of traditional schedulers.
>
>And apparently I'm completely right with CFS too.
>
>> Using CFS-v5, with Xorg at nice 0, the context-switch rate is low:
>>
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in
>>   cs us sy id wa st 2  0      0 472132  13712 178604    0    0     0    32
>>  113  170 83 17  0  0  0 2  0      0 472172  13712 178604    0    0     0 
>>    0  112  184 85 15  0  0  0 2  0      0 472196  13712 178604    0    0  
>>   0     0  108  162 83 17  0  0  0 1  0      0 472076  13712 178604    0  
>>  0     0     0  115  189 86 14  0  0  0
>
>Around 170 context switches per second.
>
>> Renicing X to -10 increases context-switching, but not dramatically so,
>> because it is throttled by CFS:
>>
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in
>>   cs us sy id wa st 4  0      0 475752  13492 176320    0    0     0    64
>>  116 1498 85 15  0  0  0 4  0      0 475752  13492 176320    0    0     0 
>>    0  107 1488 84 16  0  0  0 4  0      0 475752  13492 176320    0    0  
>>   0     0  140 1514 86 14  0  0  0 4  0      0 475752  13492 176320    0  
>>  0     0     0  107 1477 85 15  0  0  0 4  0      0 475752  13492 176320  
>>  0    0     0     0  122 1498 84 16  0  0  0
>
>Did you even *look* at your own numbers? Maybe you looked at "interrpts".
>The context switch numbers go from 170 per second, to 1500 per second!
>
>If that's not "dramatically so", I don't know what is! Just how many
>orders of magnitude worse does it have to be, to be "dramatic"? Apparently
>one order of magnitude isn't "dramatic"?
>
>So you were wrong. The fact that it was still "usable" is a good
>indication, but how about just admitting that you were wrong, and that
>renicing X is the *WRONG*THING*TO*DO*.
>
>Just don't do it. It's wrong. It was wrong with the old schedulers, it's
>wrong with the new scheduler, it's just WRONG.
>
>It was a hack, and it's a failed hack. And the fact that you don't seem to
>realize that it's a failure, even when your OWN numbers clearly show that
>it's failed, is a bit scary.
>
>		Linus

This message prompted me to do some checking in re context switches myself, 
and I've come to the conclusion that there could be a bug in vmstat itself.

Run singly the context switching is reasonable even for a -19 niceness of x, 
its only showing about 200 or so on the first loop of vmstat.  But throw in 
the -n 1 arguments and it goes crazy on the second and subsequent loops.

X nice=0
[root@coyote ~]# vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 3  0    324  62836  37952 518080    0    0   786   446  474  201 10  4 82  4  
0
 0  0    324  62712  37952 518080    0    0     0     0 1309 2361  2  5 93  0  
0
 2  0    324  62712  37952 518080    0    0     0     0 1275 2203  2  4 94  0  
0
 0  0    324  62744  37952 518080    0    0     0     0 1305 2224  1  2 97  0  
0
 0  0    324  62744  37952 518080    0    0     0     0 1291 2232  0  1 99  0  
0

X nice=-10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 3  0    324  62432  38052 518080    0    0   784   445  476  205 10  4 82  4  
0
 0  0    324  62432  38052 518080    0    0     0     0 1190 3223  1  1 98  0  
0
 2  0    324  62440  38052 518080    0    0     0     0 1209 3210  2  3 95  0  
0
 0  0    324  62316  38060 518080    0    0     0   232 1201 3355  3  4 92  1  
0
 2  0    324  62316  38060 518080    0    0     0     0 1207 2794  1  2 97  0  
0

X nice=10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
st
 4  0    324  62372  38184 518132    0    0   783   445  477  209 10  4 82  4  
0
 0  0    324  62372  38192 518132    0    0     0   272 1318 2262  0  3 97  0  
0
 0  0    324  62372  38192 518132    0    0     0     0 1293 2249  1  4 95  0  
0
 0  0    324  62248  38192 518132    0    0     0     0 1280 2443  4  2 94  0  
0
 0  0    324  62248  38192 518132    0    0     0     4 1294 2272  0  3 97  0  
0

Now, I have NDI which set of figures is the true set, but please note that in 
all 3 cases the reported values for cs didn't scale up and down all that much 
if separated out into 1st pass, and subsequent passes.

And, even with X nice=10, the system is still fairly smooth and usable.

This is with 2.6.21-rc7-CFS-v5 I built late last evening.  At Xnice=10 I just 
played a game of patience to watch the card animations and they were 
absolutely acceptably smooth. (and I won it in about 112 moves :)

>From this users viewpoint, it (cfs-v5) works, and works very well indeed, and 
it deserves a place as one of 3 selectable options in mainline.  The other 2 
being the existing mainline way, & Con K's sd-0.45 or later.  Both of these 
seem to be very large enhancements to the user experience over current 
mainline, which I'd discuss in terms borrowed from Joanne Dow.  Comparatively 
speaking, mainline has a very high vacuum.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Jayne: "Let's move this conversation in a not-Jayne's-fault direction."
				--Episode #14, "Objects in Space"

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> >  4  0      0 475752  13492 176320    0    0     0     0  107 1477 85 15  0  0  0
> >  4  0      0 475752  13492 176320    0    0     0     0  122 1498 84 16  0  0  0
> 
> Did you even *look* at your own numbers? Maybe you looked at 
> "interrpts". The context switch numbers go from 170 per second, to 
> 1500 per second!

i think i managed to look at the correct column :) 1500 per second is 
the absolute ceiling under CFS.

but, even though this utterly ugly hack of renicing (Arjan immediately 
slapped me for it when i mentioned it to him and he correctly predicted 
that lkml would go amok on anything like this) undeniably behaves better 
under CFS and gives a _visually better_ desktop at a 1500 context 
switches per second, i share your unease about it on architectural and 
policy grounds. Doing this hack upstream could easily hinder the 
efficient creation of a healthy economy for "scheduler money", by 
forcibly hacking X out of the picture - while X could be such a nice 
(and important) prototype for a cool and useful new scheduling 
infrastructure.

Basically this hack is bad on policy grounds because it is giving X an 
"legislated, unfair monopoly" on the system. It's the equivalent of a 
state-guaranteed monopoly in certain 'strategic industries'. It has some 
advantages but it is very much net harmful. Most of the time the 
"strategic importance" of any industry can be cleanly driven by the 
normal mechanics of supply and demand: anything important is recognized 
by 'people' as important via actual actions of giving it 'money'. (This 
approach also gives formerly-strategic industries the boot quickly, were 
they to become less strategic to people as things evolve.)

still, recognizing all the very real advantages of a cleaner approach, 
my primary present goal with CFS is to reach "maximum interactivity" 
here and today on a maximimally broad set of workloads, whatever it 
takes, and then to look back and figure out cleaner ways while still 
carefully keeping that maximum interactivity propertly of CFS.

For this particular auto-renicing hack here are the observed objective 
advantages to the user:

 1) while it's still an ugly hack, the increased context-switching rate
    (surprisingly to me!) still has actual, objective, undeniable 
    positive effects even in this totally X-centric worst-case messaging 
    scenario i tried to trigger:

        - visibly better eye-pleasing X behavior under the same
          "performance of scrolling"

        - no hung mouse pointer. Ever. I'd not go as far as Windows to 
          put the mouse refresh code into the kernel, but now having 
          experienced under CFS the 'mouse never hangs under any load' 
          phenomenon for a longer time, i have to admit i got addicted 
          to it. It give instant emotionally positive feedback about 
          "yes, your system is still fine, just overworked a bit", and 
          it also gives a "you caused something to happen on this box, 
          cool boy!" reassurance to the impatient human who is waiting 
          on it - be it that such a minimal thing as a moving mouse 
          pointer.

    There's a new argument as well, not amongst the issues i raised 
    before: people are happily spending 40-50% of their CPU's
    power on Beryl just to get a more ergonomic desktop via 3D effects, 
    so why not allow them to achieve another type of visual ergonomy by 
    allowing an increased, maximum-throttled X context-switch rate,
    without any measurable drop in performance, to a tunable maximum? 
    I can see no easy way for X itself to control this context-switching
    "refresh" rate in a sane way, as its workload is largely detached
    from client workloads and there's no communication between clients.

 2) it's the absolute worst maximum rate you'll ever see under CFS, and
    i definitely concentrated on triggering the worst-case. On other
    schedulers i easily got to 14K context-switches per second or worse,
    depending on the X workload, which hurts performance and makes it 
    behave visually worse. On CFS the 1400 context-switches is the 
    _ceiling_, did not measurably hurt performance and it is tunable 
    ceiling.

 3) this behavior was totally uncontrollable on other schedulers i tried
    and indeed has hurt performance there. On CFS this is still totally
    tunable and controllable on several levels.

i'm not saying that any of this reduces the ugliness of the hack, or 
that any of this makes the strategic disadvantages of this hack 
disappear, i simply tried to point out that despite the existing 
conventional wisdom it's apparently much more useful in practice on CFS 
than on other schedulers.

And if the "economy of scheduling" experiment fails in practice for some 
presently unknown technological reason, we might as well have to go back 
to ugly tricks like this one. With its 5 lines and limited scope i think 
it still beats 500 lines of convoluted scheduling heuristics :-/ Right 
now i'm very positive about the "economy of scheduling" angle, i think 
we have a realistic chance to pull it off.

	Ingo

Re: [report] renicing X, cfs-v5 vs sd-0.46

[Michael K. Edwards]

On 4/23/07, Ingo Molnar <mingo@elte.hu> wrote:
> Basically this hack is bad on policy grounds because it is giving X an
> "legislated, unfair monopoly" on the system. It's the equivalent of a
> state-guaranteed monopoly in certain 'strategic industries'. It has some
> advantages but it is very much net harmful. Most of the time the
> "strategic importance" of any industry can be cleanly driven by the
> normal mechanics of supply and demand: anything important is recognized
> by 'people' as important via actual actions of giving it 'money'. (This
> approach also gives formerly-strategic industries the boot quickly, were
> they to become less strategic to people as things evolve.)

If you're going to drag free-market economics into it, why not
actually use the techniques of free-market economics?  Design a
bidding system in which agents (tasks) earn "money" by getting things
done, and can use that "money" to bid on "resources".  You will of
course need accurate cost accounting in order to decide which bids are
most "profitable" for the scheduler to accept, and accurate transfer
accounting to design price structures for contracts between agents in
which one agrees to accomplish work on behalf on another.  Actual
revenues come from doing the work that the consumer wants done and is
willing to pay for.  Etc., etc.  Has your horsepucky filter kicked in
yet?

If your system doesn't work this way -- perhaps because you think as I
do that scheduler design is principally an engineering problem, not an
economics problem -- then analogies from economics are probably worth
zip.  Yes, I wrote earlier about "economic dispatch" -- that's an
operations problem, a control theory problem, an _engineering_
problem, that happens to have a set of engineering goals and
constraints that take profitability into account.  I think you might
be able to design a better Linux scheduler anchored in the techniques
and literature of control theory, perhaps specifically with reference
to electric-utility economic dispatch, because the systems under
control and the goals of control are similar.

But there's a good reason not to treat X as special.  Namely, that it
_isn't_.  It may be the only program on many people's Linux desktops
with an opaque control structure -- a separate class of interactive
activities hidden inside an oversubscribed push-model pipeline stage
-- but it's hardly the only program designed this way.  Treat the X
server as a easily instrumented exemplar of a event-loop-centric
design whose thread structure doesn't distinguish between fast-twitch
and best-effort activity patterns.  I wrote earlier about what one
might do about this (attach urgency to to the work in the queue
instead of the worker being asked to do it).

Cheers,
- Michael