Re: [patch 1/2] sched: use resched IPI to kick off the nohz idle balance

[Suresh Siddha <suresh.b.siddha@intel.com>]

On Mon, 2011-10-03 at 12:36 -0700, Peter Zijlstra wrote:
> On Thu, 2011-09-29 at 15:30 -0700, Suresh Siddha wrote:
> 
> > ---
> >  kernel/sched.c      |   14 +++++++++++---
> >  kernel/sched_fair.c |   27 +++++++--------------------
> >  2 files changed, 18 insertions(+), 23 deletions(-)
> > 
> > Index: linux-2.6-tip/kernel/sched.c
> > ===================================================================
> > --- linux-2.6-tip.orig/kernel/sched.c
> > +++ linux-2.6-tip/kernel/sched.c
> > @@ -2733,7 +2733,7 @@ void scheduler_ipi(void)
> >  	struct rq *rq = this_rq();
> >  	struct task_struct *list = xchg(&rq->wake_list, NULL);
> >  
> > -	if (!list)
> > +	if (!list && !idle_cpu(cpu_of(rq)))
> >  		return;
> 
> Why not make that !rq->nohz_balance_kick? (wrapped in a helper for !
> CONFIG_NO_HZ)

If a rq gets busy before we do nohz_idle_balance() which does the
nohz_balance_kick reset, we will have a busy rq with nohz_balance_kick
set. And wanted to bail out sooner by checking for idle cpu and minimize
the impact for a busy rq having the nohz_idle_balance set.

I can probably rename your got_nohz_kick() as got_nohz_idle_kick() and
fix it.

> > tself as idle load_balancer, while
> > @@ -4450,11 +4434,14 @@ static void nohz_balancer_kick(int cpu)
> >  	}
> >  
> >  	if (!cpu_rq(ilb_cpu)->nohz_balance_kick) {
> > -		struct call_single_data *cp;
> > -
> >  		cpu_rq(ilb_cpu)->nohz_balance_kick = 1;
> > -		cp = &per_cpu(remote_sched_softirq_cb, cpu);
> > -		__smp_call_function_single(ilb_cpu, cp, 0);
> > +		/*
> > +		 * Use kick_process instead of resched_cpu.
> > +		 * This way we generate a sched IPI on the target cpu which
> > +		 * is idle. And the softirq performing nohz idle load balance
> > +		 * will be run before returning from the IPI.
> > +		 */
> 
> Shouldn't we have a memory barrier of sorts before sending the IPI?
> 
> > +		kick_process(idle_task(ilb_cpu));

Correct and also I think we can use smp_send_reschedule() directly
instead of kick process. Will fix it.

thanks,
suresh