mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Paul E. McKenney" <paulmck@kernel.org>
To: Joel Fernandes <joel@joelfernandes.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Boqun Feng <boqun.feng@gmail.com>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	Thomas Gleixner <tglx@linutronix.de>,
	LKML <linux-kernel@vger.kernel.org>,
	Steven Rostedt <rostedt@goodmis.org>
Subject: Re: RCU vs NOHZ
Date: Sat, 17 Sep 2022 07:28:23 -0700	[thread overview]
Message-ID: <20220917142823.GG246308@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <82a58d5c-73e7-6e78-e72e-3e46a1a3afbc@joelfernandes.org>

On Sat, Sep 17, 2022 at 09:52:49AM -0400, Joel Fernandes wrote:
> On 9/17/2022 9:35 AM, Peter Zijlstra wrote:
> > On Fri, Sep 16, 2022 at 02:11:10PM -0400, Joel Fernandes wrote:
> >> Hi Peter,
> >>
> >> On Fri, Sep 16, 2022 at 5:20 AM Peter Zijlstra <peterz@infradead.org> wrote:
> >> [...]
> >>>> It wasn't enabled for ChromeOS.
> >>>>
> >>>> When fully enabled, it gave them the energy-efficiency advantages Joel
> >>>> described.  And then Joel described some additional call_rcu_lazy()
> >>>> changes that provided even better energy efficiency.  Though I believe
> >>>> that the application should also be changed to avoid incessantly opening
> >>>> and closing that file while the device is idle, as this would remove
> >>>> -all- RCU work when nearly idle.  But some of the other call_rcu_lazy()
> >>>> use cases would likely remain.
> >>>
> >>> So I'm thinking the scheme I outlined gets you most if not all of what
> >>> lazy would get you without having to add the lazy thing. A CPU is never
> >>> refused deep idle when it passes off the callbacks.
> >>>
> >>> The NOHZ thing is a nice hook for 'this-cpu-wants-to-go-idle-long-term'
> >>> and do our utmost bestest to move work away from it. You *want* to break
> >>> affinity at this point.
> >>>
> >>> If you hate on the global, push it to a per rcu_node offload list until
> >>> the whole node is idle and then push it up the next rcu_node level until
> >>> you reach the top.
> >>>
> >>> Then when the top rcu_node is full idle; you can insta progress the QS
> >>> state and run the callbacks and go idle.
> >>
> >> In my opinion the speed brakes have to be applied before the GP and
> >> other threads are even awakened. The issue Android and ChromeOS
> >> observe is that even a single CB queued every few jiffies can cause
> >> work that can be otherwise delayed / batched, to be scheduled in. I am
> >> not sure if your suggestions above address that. Does it?
> > 
> > Scheduled how? Is this callbacks doing queue_work() or something?
> 
> Way before the callback is even ready to execute, you can rcuog, rcuop,
> rcu_preempt threads running to go through the grace period state machine.
> 
> > Anyway; the thinking is that by passing off the callbacks on NOHZ, the
> > idle CPUs stay idle. By running the callbacks before going full idle,
> > all work is done and you can stay idle longer.
> 
> But all CPUs idle does not mean grace period is over, you can have a task (at
> least on PREEMPT_RT) block in the middle of an RCU read-side critical section
> and then all CPUs go idle.
> 
> Other than that, a typical flow could look like:
> 
> 1. CPU queues a callback.
> 2. CPU then goes idle.
> 3. Another CPU is running the RCU threads waking up otherwise idle CPUs.
> 4. Grace period completes and an RCU thread runs a callback.
> 
> >> Try this experiment on your ADL system (for fun). Boot to the login
> >> screen on any distro,
> > 
> > All my dev boxes are headless :-) I don't thinkt he ADL even has X or
> > wayland installed.
> 
> Ah, ok. Maybe what you have (like daemons) are already requesting RCU for
> something. Android folks had some logger requesting RCU all the time.
> 
> >> and before logging in, run turbostat over ssh
> >> and observe PC8 percent residencies. Now increase
> >> jiffies_till_first_fqs boot parameter value to 64 or so and try again.
> >> You may be surprised how much PC8 percent increases by delaying RCU
> >> and batching callbacks (via jiffies boot option) Admittedly this is
> >> more amplified on ADL because of package-C-states, firmware and what
> >> not, and isn’t as much a problem on Android; but still gives a nice
> >> power improvement there.
> > 
> > I can try; but as of now turbostat doesn't seem to work on that thing at
> > all. I think localyesconfig might've stripped a required bit. I'll poke
> > at it later.
> 
> Cool! I believe Len Brown can help on that , or maybe there is another way you
> can read the counters to figure out the PC8% and RAPL power.

Whatever the evaluation scheme, it absolutely -must- measure real power
consumed by real hardware running some real-world workload compared to
Joel et al.'s scheme, or I will cheerfully ignore it.  ;-)

							Thanx, Paul

  reply	other threads:[~2022-09-17 14:28 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-15  8:39 Peter Zijlstra
2022-09-15 15:37 ` Joel Fernandes
2022-09-15 16:06 ` Paul E. McKenney
2022-09-15 18:50   ` Peter Zijlstra
2022-09-15 19:14     ` Paul E. McKenney
2022-09-15 22:30       ` Peter Zijlstra
2022-09-16  3:40         ` Joel Fernandes
2022-09-16  7:58         ` Paul E. McKenney
2022-09-16  9:20           ` Peter Zijlstra
2022-09-16 18:11             ` Joel Fernandes
2022-09-17 13:35               ` Peter Zijlstra
2022-09-17 13:52                 ` Joel Fernandes
2022-09-17 14:28                   ` Paul E. McKenney [this message]
2022-09-17 14:25             ` Paul E. McKenney
2022-09-21 21:36               ` Paul E. McKenney
2022-09-23 15:12                 ` Joel Fernandes
2022-09-23 16:01                   ` Paul E. McKenney
2022-09-23 17:47                     ` Joel Fernandes
2022-09-23 18:15                       ` Paul E. McKenney
2022-09-29 11:06                 ` Peter Zijlstra
2022-09-29 10:55               ` Peter Zijlstra
2022-09-29 15:20                 ` Paul E. McKenney
2022-09-29 15:46                   ` Paul E. McKenney
2022-09-29 16:05                     ` Joel Fernandes
2022-09-29 16:23                     ` Peter Zijlstra
2022-09-29 16:42                       ` Paul E. McKenney
2022-09-29 19:01                         ` Peter Zijlstra
2022-09-29 19:52                           ` Paul E. McKenney
2022-09-29 16:18                   ` Peter Zijlstra
2022-09-29 16:36                     ` Paul E. McKenney
2022-09-29 19:08                       ` Peter Zijlstra
2022-09-29 19:56                         ` Paul E. McKenney
2022-09-30 14:48                           ` Paul E. McKenney
2022-09-23 19:47 Joel Fernandes

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220917142823.GG246308@paulmck-ThinkPad-P17-Gen-1 \
    --to=paulmck@kernel.org \
    --cc=boqun.feng@gmail.com \
    --cc=fweisbec@gmail.com \
    --cc=joel@joelfernandes.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=rjw@rjwysocki.net \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®