find_new_ilb

find_new_ilb

[Julia Lawall]

Starting from the following commit:

commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
Author: Peter Zijlstra <peterz@infradead.org>
Date:   Tue Aug 18 10:48:17 2020 +0200

    sched/fair: Exclude the current CPU from find_new_ilb()

up through Linux 5.12, I observed that often when most of the machine was
idle, there could be many (thousands) of sched_wake_idle_without_ipi
events, typically between cores 0 and 1.  I don't see this any more in
Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
subsequent to v5.12, and I didn't see anything that explicitly addresses
this issue.  Before I plunge into another set of rounds of bisecting, I
wonder if anyone knows whether and how this problem was resolved?

thanks,
julia

Re: find_new_ilb

[Peter Zijlstra]

On Tue, Jun 08, 2021 at 09:51:30PM +0200, Julia Lawall wrote:
> Starting from the following commit:
> 
> commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
> Author: Peter Zijlstra <peterz@infradead.org>
> Date:   Tue Aug 18 10:48:17 2020 +0200
> 
>     sched/fair: Exclude the current CPU from find_new_ilb()
> 
> up through Linux 5.12, I observed that often when most of the machine was
> idle, there could be many (thousands) of sched_wake_idle_without_ipi
> events, typically between cores 0 and 1.  I don't see this any more in
> Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
> subsequent to v5.12, and I didn't see anything that explicitly addresses
> this issue.  Before I plunge into another set of rounds of bisecting, I
> wonder if anyone knows whether and how this problem was resolved?

Hurmph.. that patch was preparation for a later change that never seems
to have happened. If it is causing trouble for you, I think you can
savely revert it.

At the time I thought it was very strange that new_idle would select
itself as idle-balancer, doubly so, because the only way to get there
would be with NEED_RESCHED already set, so the IPI wouldn't in fact do
anything.

Looking again, the difference is ofcourse that previously we'd select
self and NO-OP, but now we'll potentially select another CPU and
actually do something.

This is arguably an improvement, because we did want to do something.

 I can't quite remember what would've change here since, Vincent, can
 you remember?

Anyway, is this actually causing you trouble, or are you just going on
the increased number of events? 

Re: find_new_ilb

[Vincent Guittot]

On Wed, 9 Jun 2021 at 09:12, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, Jun 08, 2021 at 09:51:30PM +0200, Julia Lawall wrote:
> > Starting from the following commit:
> >
> > commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
> > Author: Peter Zijlstra <peterz@infradead.org>
> > Date:   Tue Aug 18 10:48:17 2020 +0200
> >
> >     sched/fair: Exclude the current CPU from find_new_ilb()
> >
> > up through Linux 5.12, I observed that often when most of the machine was
> > idle, there could be many (thousands) of sched_wake_idle_without_ipi
> > events, typically between cores 0 and 1.  I don't see this any more in
> > Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
> > subsequent to v5.12, and I didn't see anything that explicitly addresses
> > this issue.  Before I plunge into another set of rounds of bisecting, I
> > wonder if anyone knows whether and how this problem was resolved?
>
> Hurmph.. that patch was preparation for a later change that never seems
> to have happened. If it is causing trouble for you, I think you can
> savely revert it.
>
> At the time I thought it was very strange that new_idle would select
> itself as idle-balancer, doubly so, because the only way to get there
> would be with NEED_RESCHED already set, so the IPI wouldn't in fact do
> anything.
>
> Looking again, the difference is ofcourse that previously we'd select
> self and NO-OP, but now we'll potentially select another CPU and
> actually do something.
>
> This is arguably an improvement, because we did want to do something.
>
>  I can't quite remember what would've change here since, Vincent, can
>  you remember?

c6f886546cb8 ("sched/fair: Trigger the update of blocked load on newly
idle cpu") could be the one which fixes it.
We don't kick_ilb from newilde_balance() since this commit

>
> Anyway, is this actually causing you trouble, or are you just going on
> the increased number of events?

Re: find_new_ilb

[Julia Lawall]

On Wed, 9 Jun 2021, Vincent Guittot wrote:

> On Wed, 9 Jun 2021 at 09:12, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Tue, Jun 08, 2021 at 09:51:30PM +0200, Julia Lawall wrote:
> > > Starting from the following commit:
> > >
> > > commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
> > > Author: Peter Zijlstra <peterz@infradead.org>
> > > Date:   Tue Aug 18 10:48:17 2020 +0200
> > >
> > >     sched/fair: Exclude the current CPU from find_new_ilb()
> > >
> > > up through Linux 5.12, I observed that often when most of the machine was
> > > idle, there could be many (thousands) of sched_wake_idle_without_ipi
> > > events, typically between cores 0 and 1.  I don't see this any more in
> > > Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
> > > subsequent to v5.12, and I didn't see anything that explicitly addresses
> > > this issue.  Before I plunge into another set of rounds of bisecting, I
> > > wonder if anyone knows whether and how this problem was resolved?
> >
> > Hurmph.. that patch was preparation for a later change that never seems
> > to have happened. If it is causing trouble for you, I think you can
> > savely revert it.
> >
> > At the time I thought it was very strange that new_idle would select
> > itself as idle-balancer, doubly so, because the only way to get there
> > would be with NEED_RESCHED already set, so the IPI wouldn't in fact do
> > anything.
> >
> > Looking again, the difference is ofcourse that previously we'd select
> > self and NO-OP, but now we'll potentially select another CPU and
> > actually do something.
> >
> > This is arguably an improvement, because we did want to do something.
> >
> >  I can't quite remember what would've change here since, Vincent, can
> >  you remember?
>
> c6f886546cb8 ("sched/fair: Trigger the update of blocked load on newly
> idle cpu") could be the one which fixes it.
> We don't kick_ilb from newilde_balance() since this commit

Thanks!  I'll check on that.

julia

>
> >
> > Anyway, is this actually causing you trouble, or are you just going on
> > the increased number of events?
>

Re: find_new_ilb

[Julia Lawall]

On Wed, 9 Jun 2021, Vincent Guittot wrote:

> On Wed, 9 Jun 2021 at 09:12, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Tue, Jun 08, 2021 at 09:51:30PM +0200, Julia Lawall wrote:
> > > Starting from the following commit:
> > >
> > > commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
> > > Author: Peter Zijlstra <peterz@infradead.org>
> > > Date:   Tue Aug 18 10:48:17 2020 +0200
> > >
> > >     sched/fair: Exclude the current CPU from find_new_ilb()
> > >
> > > up through Linux 5.12, I observed that often when most of the machine was
> > > idle, there could be many (thousands) of sched_wake_idle_without_ipi
> > > events, typically between cores 0 and 1.  I don't see this any more in
> > > Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
> > > subsequent to v5.12, and I didn't see anything that explicitly addresses
> > > this issue.  Before I plunge into another set of rounds of bisecting, I
> > > wonder if anyone knows whether and how this problem was resolved?
> >
> > Hurmph.. that patch was preparation for a later change that never seems
> > to have happened. If it is causing trouble for you, I think you can
> > savely revert it.
> >
> > At the time I thought it was very strange that new_idle would select
> > itself as idle-balancer, doubly so, because the only way to get there
> > would be with NEED_RESCHED already set, so the IPI wouldn't in fact do
> > anything.
> >
> > Looking again, the difference is ofcourse that previously we'd select
> > self and NO-OP, but now we'll potentially select another CPU and
> > actually do something.
> >
> > This is arguably an improvement, because we did want to do something.
> >
> >  I can't quite remember what would've change here since, Vincent, can
> >  you remember?
>
> c6f886546cb8 ("sched/fair: Trigger the update of blocked load on newly
> idle cpu") could be the one which fixes it.
> We don't kick_ilb from newilde_balance() since this commit

This is indeed the commit that fixes the problem.  Increasing the blocked
load did seem to be what was not getting done.

thanks,
julia

>
> >
> > Anyway, is this actually causing you trouble, or are you just going on
> > the increased number of events?
>

Re: find_new_ilb

[Julia Lawall]

On Wed, 9 Jun 2021, Peter Zijlstra wrote:

> On Tue, Jun 08, 2021 at 09:51:30PM +0200, Julia Lawall wrote:
> > Starting from the following commit:
> >
> > commit 45da7a2b0af8fa29dff2e6ba8926322068350fce
> > Author: Peter Zijlstra <peterz@infradead.org>
> > Date:   Tue Aug 18 10:48:17 2020 +0200
> >
> >     sched/fair: Exclude the current CPU from find_new_ilb()
> >
> > up through Linux 5.12, I observed that often when most of the machine was
> > idle, there could be many (thousands) of sched_wake_idle_without_ipi
> > events, typically between cores 0 and 1.  I don't see this any more in
> > Linux v5.13-rc1.  I looked through the patches to fair.c and core.c
> > subsequent to v5.12, and I didn't see anything that explicitly addresses
> > this issue.  Before I plunge into another set of rounds of bisecting, I
> > wonder if anyone knows whether and how this problem was resolved?
>
> Hurmph.. that patch was preparation for a later change that never seems
> to have happened. If it is causing trouble for you, I think you can
> savely revert it.
>
> At the time I thought it was very strange that new_idle would select
> itself as idle-balancer, doubly so, because the only way to get there
> would be with NEED_RESCHED already set, so the IPI wouldn't in fact do
> anything.
>
> Looking again, the difference is ofcourse that previously we'd select
> self and NO-OP, but now we'll potentially select another CPU and
> actually do something.
>
> This is arguably an improvement, because we did want to do something.
>
>  I can't quite remember what would've change here since, Vincent, can
>  you remember?
>
> Anyway, is this actually causing you trouble, or are you just going on
> the increased number of events?

Thanks for the feedback.  The scenarios I saw wouldn't have caused trouble
for anyone, because the machine was highly idle (maybe a couple of cores
in use).  And the problem seems to be gone as well in v5.13.

julia