[PATCH RT] Fix RT balancing tasks pulling

[PATCH RT] Fix RT balancing tasks pulling

[Sébastien Dugué]

  Hi Ingo, Thomas,

  this one-liner fixes a bug in balance_rt_tasks() which sometimes manifests by
having a lower prio task being scheduled while a higher prio task is sitting
waiting on another runqueue.

  This is pretty hard to reproduce on low cpu count machines, for example, I
had to have sched_football run in a loop for ~38h before it failed on a dual HT
Xeon box.

  Sébastien.

---


  In the CPU loop in balance_rt_tasks(), the 'next' task pointer is only ever
updated if this_rq->lock has been dropped in double_lock_balance().

  This sometimes lead to 'this_cpu' pulling tasks that are only garanteed to
have a higher priority than the 'next' picked before the loop.

  Fix this to update 'next' to the last picked RT task.


Signed-off-by: Sébastien Dugué <sebastien.dugue@bull.net>

---
 kernel/sched.c |    8 ++++++++
 1 file changed, 8 insertions(+)

Index: linux-2.6.22.1-rt6/kernel/sched.c
===================================================================
--- linux-2.6.22.1-rt6.orig/kernel/sched.c	2007-07-24 11:34:18.000000000 +0200
+++ linux-2.6.22.1-rt6/kernel/sched.c	2007-07-24 11:34:35.000000000 +0200
@@ -1499,6 +1499,14 @@ static void balance_rt_tasks(struct rq *
 			 * in another runqueue. (low likelyhood
 			 * but possible)
 			 */
+
+			/*
+			 * Update next so that we won't pick a task
+			 * on another cpu with a priority lower (or equal)
+			 * than the one we just picked.
+			 */
+			next = p;
+
 		}
 		spin_unlock(&src_rq->lock);
 	}

Re: [PATCH RT] Fix RT balancing tasks pulling

[Ingo Molnar]

* Sébastien Dugué <sebastien.dugue@bull.net> wrote:

>   Hi Ingo, Thomas,
> 
>   this one-liner fixes a bug in balance_rt_tasks() which sometimes 
> manifests by having a lower prio task being scheduled while a higher 
> prio task is sitting waiting on another runqueue.
> 
>   This is pretty hard to reproduce on low cpu count machines, for 
> example, I had to have sched_football run in a loop for ~38h before it 
> failed on a dual HT Xeon box.

really nice catch! Applied.

	Ingo

Re: [PATCH RT] Fix RT balancing tasks pulling

[Josh Triplett]

On Tue, 2007-07-24 at 15:42 +0200, Sébastien Dugué wrote:
>   this one-liner fixes a bug in balance_rt_tasks() which sometimes manifests by
> having a lower prio task being scheduled while a higher prio task is sitting
> waiting on another runqueue.
> 
>   This is pretty hard to reproduce on low cpu count machines, for example, I
> had to have sched_football run in a loop for ~38h before it failed on a dual HT
> Xeon box.

Very impressive find, and this does seem to improve things in some
cases.

However, with 2.6.22.1-rt8, which includes this patch, I still managed
to get some failures after a few hours on an 8-way (quad dual-core) box.
Preliminary results so far:
    420 Final ball position: 0
      2 Final ball position: 1

- Josh Triplett

Re: [PATCH RT] Fix RT balancing tasks pulling

[Sébastien Dugué]

  Hi Josh,

On Mon, 30 Jul 2007 14:50:34 -0700 Josh Triplett <josht@linux.vnet.ibm.com> wrote:

> On Tue, 2007-07-24 at 15:42 +0200, Sébastien Dugué wrote:
> >   this one-liner fixes a bug in balance_rt_tasks() which sometimes manifests by
> > having a lower prio task being scheduled while a higher prio task is sitting
> > waiting on another runqueue.
> > 
> >   This is pretty hard to reproduce on low cpu count machines, for example, I
> > had to have sched_football run in a loop for ~38h before it failed on a dual HT
> > Xeon box.
> 
> Very impressive find, and this does seem to improve things in some
> cases.
> 
> However, with 2.6.22.1-rt8, which includes this patch, I still managed
> to get some failures after a few hours on an 8-way (quad dual-core) box.
> Preliminary results so far:
>     420 Final ball position: 0
>       2 Final ball position: 1
> 

  Darn, I thought it was it :(

  I had a ~60h run on my box with the very same kernel without a single
failure.

  Do you have any other user RT load running when those failures pop up?

  Anyway, back to scheduler code dissection now.

  Sébastien.

Re: [PATCH RT] Fix RT balancing tasks pulling

[Josh Triplett]

On Tue, 2007-07-31 at 13:48 +0200, Sébastien Dugué wrote:
> On Mon, 30 Jul 2007 14:50:34 -0700 Josh Triplett <josht@linux.vnet.ibm.com> wrote:
> > On Tue, 2007-07-24 at 15:42 +0200, Sébastien Dugué wrote:
> > >   this one-liner fixes a bug in balance_rt_tasks() which sometimes manifests by
> > > having a lower prio task being scheduled while a higher prio task is sitting
> > > waiting on another runqueue.
> > > 
> > >   This is pretty hard to reproduce on low cpu count machines, for example, I
> > > had to have sched_football run in a loop for ~38h before it failed on a dual HT
> > > Xeon box.
> > 
> > Very impressive find, and this does seem to improve things in some
> > cases.
> > 
> > However, with 2.6.22.1-rt8, which includes this patch, I still managed
> > to get some failures after a few hours on an 8-way (quad dual-core) box.
> > Preliminary results so far:
> >     420 Final ball position: 0
> >       2 Final ball position: 1
> > 
> 
>   Darn, I thought it was it :(
> 
>   I had a ~60h run on my box with the very same kernel without a single
> failure.

I think you really need a box with more processors to reproduce these
problems in a reasonable timeframe.  Happy to test any patches you have
to fix RT balancing; please let me know.

>   Do you have any other user RT load running when those failures pop up?

Other than sched_football, I had sshd, bash, and GNU screen,
theoretically all 100% idle (no input to read and no output to display).

- Josh Triplett