Re: [PATCH tip/core/rcu 6/7] rcu: Drive quiescent-state-forcing delay from HZ

[Peter Zijlstra <peterz@infradead.org>]

On Tue, May 14, 2013 at 08:47:28AM -0700, Paul E. McKenney wrote:
> On Tue, May 14, 2013 at 04:51:20PM +0200, Peter Zijlstra wrote:
> > > In theory, yes.  In practice, this requires lots of lock acquisitions
> > > and releases on large systems, including some global locks.  The weight
> > > could be reduced, but...
> > > 
> > > What I would like to do instead would be to specify expedited grace
> > > periods during boot.
> > 
> > But why, surely going idle without any RCU callbacks isn't completely unheard
> > of, even outside of the boot process?
> 
> Yep, and RCU has special-cased that for quite some time.
> 
> > Being able to quickly drop out of the RCU state machinery would be a good thing IMO.
> 
> And this is currently possible -- this is the job of rcu_idle_enter()
> and friends.  And it works well, at least when I get my "if" statements
> set up correctly (hence the earlier patch).
> 
> Or are you seeing a slowdown even with that earlier patch applied?  If so,
> please let me know what you are seeing.

I'm not running anything in particular, except maybe a broken mental
model of RCU ;-)

So what I'm talking about is the !rcu_cpu_has_callbacks() case, where
there's absolutely nothing for RCU to do except tell the state machine
its no longer participating.

Your patch to rcu_needs_cpu() frobbing the lazy condition is after that
and thus irrelevant for this AFAICT.

Now as far as I can see, rcu_needs_cpu() will return false in this case;
allowing the cpu to enter NO_HZ state. We then call rcu_idle_enter()
which would call rcu_eqs_enter(). Which should put the CPU in extended
quiescent state.

However, you're still running into these FQSs delaying boot. Why is
that? Is that because rcu_eqs_enter() doesn't really do enough?

The thing is, if all other CPUs are idle, detecting the end of a grace
period should be rather trivial and not involve FQSs and thus be tons
faster.

Clearly I'm missing something obvious and not communicating right or so.