From: "Paul E. McKenney" <paulmck@kernel.org>
To: Mukesh Ojha <quic_mojha@quicinc.com>
Cc: kernel-team@fb.com, linux-kernel@vger.kernel.org,
rcu@vger.kernel.org, rostedt@goodmis.org, tj@kernel.org,
Peter Zijlstra <peterz@infradead.org>
Subject: Re: [PATCH rcu 3/3] rcu: Allow expedited RCU grace periods on incoming CPUs
Date: Fri, 18 Feb 2022 10:09:26 -0800 [thread overview]
Message-ID: <20220218180926.GX4285@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <ca0cac53-68ec-7df4-e617-2a4cd9710491@quicinc.com>
On Fri, Feb 18, 2022 at 11:03:03PM +0530, Mukesh Ojha wrote:
>
> On 2/15/2022 11:09 PM, Paul E. McKenney wrote:
> > On Tue, Feb 15, 2022 at 07:53:10PM +0530, Mukesh Ojha wrote:
> > > On 2/14/2022 10:14 PM, Paul E. McKenney wrote:
> > > > On Thu, Feb 10, 2022 at 12:38:11AM +0100, Frederic Weisbecker wrote:
> > > > > On Fri, Feb 04, 2022 at 02:55:07PM -0800, Paul E. McKenney wrote:
> > > > > > Although it is usually safe to invoke synchronize_rcu_expedited() from a
> > > > > > preemption-enabled CPU-hotplug notifier, if it is invoked from a notifier
> > > > > > between CPUHP_AP_RCUTREE_ONLINE and CPUHP_AP_ACTIVE, its attempts to
> > > > > > invoke a workqueue handler will hang due to RCU waiting on a CPU that
> > > > > > the scheduler is not paying attention to. This commit therefore expands
> > > > > > use of the existing workqueue-independent synchronize_rcu_expedited()
> > > > > > from early boot to also include CPUs that are being hotplugged.
> > > > > >
> > > > > > Link:https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@quicinc.com/
> > > > > > Reported-by: Mukesh Ojha<quic_mojha@quicinc.com>
> > > > > > Cc: Tejun Heo<tj@kernel.org>
> > > > > > Signed-off-by: Paul E. McKenney<paulmck@kernel.org>
> > > > > I'm surprised by this scheduler behaviour.
> > > > >
> > > > > Since sched_cpu_activate() hasn't been called yet,
> > > > > rq->balance_callback = balance_push_callback. As a result, balance_push() should
> > > > > be called at the end of schedule() when the workqueue is picked as the next task.
> > > > > Then eventually the workqueue should be immediately preempted by the stop task to
> > > > > be migrated elsewhere.
> > > > >
> > > > > So I must be missing something. For the fun, I booted the following and it
> > > > > didn't produce any issue:
> > > > >
> > > > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > > > index 80faf2273ce9..b1e74a508881 100644
> > > > > --- a/kernel/rcu/tree.c
> > > > > +++ b/kernel/rcu/tree.c
> > > > > @@ -4234,6 +4234,8 @@ int rcutree_online_cpu(unsigned int cpu)
> > > > > // Stop-machine done, so allow nohz_full to disable tick.
> > > > > tick_dep_clear(TICK_DEP_BIT_RCU);
> > > > > + if (cpu != 0)
> > > > > + synchronize_rcu_expedited();
> > > > > return 0;
> > > > > }
> > > > That does seem compelling. And others have argued that the workqueue
> > > > system's handling of offline CPUs should deal with this.
> > > >
> > > > Mukesh, was this a theoretical bug, or did you actually make it happen?
> > > > If you made it happen, as seems to have been the case given your original
> > > > email [1], could you please post your reproducer?
> > > No, it was not theoretical one. We saw this issue only once in our testing
> > > and i don't think it is easy to reproduce otherwise
> > > it would been fixed by now.
> > >
> > > When one of thread calling synchronize_expedite_rcu with timer of 20s but it
> > > did not get the exp funnel
> > > lock for 20s and there we crash it with panic() on timeout.
> > >
> > > The other thread cpuhp which was having the lock got stuck at the point
> > > mentioned at the below link.
> > >
> > > https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@quicinc.com/
> > OK. Are you able to create an in-kernel reproducer, perhaps similar to
> > Frederic's change above?
> >
> > I am worried that the patch that I am carrying might be fixing some
> > other bug by accident...
>
> Just for information, we are running on 5.10 kernel and after numerous
> attempt, i was not able to reproduce the issue:-)
Thank you for checking!
I will drop this commit from -rcu's "dev" branch, but leave a tag
"exponl.2022.02.18a" should it ever prove necessary.
Thanx, Paul
> Thanks,
> -Mukesh
>
> >
> > Thanx, Paul
> >
> > > e.g Below sample test in combination of many other test in parallel
> > >
> > > :loop
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu0/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu0/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu1/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu1/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu2/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu2/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu3/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu3/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu4/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu4/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu5/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu5/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu6/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu6/online"
> > >
> > > adb shell "echo 0 > /sys/devices/system/cpu/cpu7/online"
> > >
> > > adb shell "echo 1 > /sys/devices/system/cpu/cpu7/online"
> > >
> > > goto loop
> > >
> > >
> > >
> > > Thanks, Mukesh
> > >
> > > > Thanx, Paul
> > > >
> > > > [1]https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@quicinc.com/
next prev parent reply other threads:[~2022-02-18 18:09 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-09 23:38 Frederic Weisbecker
2022-02-14 16:44 ` Paul E. McKenney
2022-02-15 14:28 ` Mukesh Ojha
[not found] ` <f8cff19c-5e8f-a7ed-c2ff-49a264b4e342@quicinc.com>
2022-02-15 17:39 ` Paul E. McKenney
2022-02-17 16:13 ` Mukesh Ojha
2022-02-18 17:33 ` Mukesh Ojha
2022-02-18 18:09 ` Paul E. McKenney [this message]
-- strict thread matches above, loose matches on Subject: below --
2022-02-04 22:54 [PATCH rcu 0/3] Expedited-grace-period updates for v5.18 Paul E. McKenney
2022-02-04 22:55 ` [PATCH rcu 3/3] rcu: Allow expedited RCU grace periods on incoming CPUs Paul E. McKenney
2022-02-08 18:56 ` Tejun Heo
2022-02-09 18:23 ` Mukesh Ojha
2022-02-09 22:06 ` Paul E. McKenney
2022-02-11 18:44 ` Mukesh Ojha
2022-02-11 22:14 ` Paul E. McKenney
2022-02-12 8:47 ` Mukesh Ojha
2022-02-12 11:28 ` Neeraj Upadhyay
2022-02-12 13:56 ` Mukesh Ojha
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220218180926.GX4285@paulmck-ThinkPad-P17-Gen-1 \
--to=paulmck@kernel.org \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=quic_mojha@quicinc.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®