Re: RCU related performance regression in 3.3

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Fri, Apr 27, 2012 at 02:15:20PM +0200, Pascal Chapperon wrote:
> Le 18/04/2012 17:23, Paul E. McKenney a écrit :
> > On Wed, Apr 18, 2012 at 05:00:14PM +0200, Pascal Chapperon wrote:
> >> Le 18/04/2012 16:01, Paul E. McKenney a écrit :
> >>> On Wed, Apr 18, 2012 at 11:37:28AM +0200, Pascal Chapperon wrote:
> >>>> Mount and umount operations are not slower with RCU_FAST_NO_HZ during
> >>>> runtime; systemctl start and stop operations are also not slower. In
> >>>> fact, i couldn't find a single operation slower during runtime with
> >>>> RCU_FAST_NO_HZ.
> >>>
> >>> Your boot-time setup is such that all CPUs are online before the
> >>> boot-time mount operations take place, right?
> >> Yes :
> >> [ 0.242697] Brought up 8 CPUs
> >> [ 0.242699] Total of 8 processors activated (35118.33 BogoMIPS).
> >>
> >>> Struggling to understand
> >>> how RCU can tell the difference between post-CPU-bringup boot time
> >>> and run time...
> >>>
> >> systemd is controlling the whole boot process including mount
> >> operation (apart root filesystem) and as I can see, uses heavily
> >> sockets to do it (not to mention cpu-affinity). It controls also the
> >> major part of umount operations. Is it possible that your patch hits
> >> a systemd bug ?
> > 
> > Is it possible that systemd is using network operations that include
> > synchronize_rcu()? Then if you did the same operation from the
> > command line at runtime, you might not see the slowdown.
> > 
> > Is it possible for you to convince systemd to collect RCU event tracing
> > during the slow operation? RCU event tracing is available under
> > /sys/kernel/debug/tracing/rcu.
> >
> .
> I have collected the RCU event tracing during a slow boot with
> FAST_NO_HZ (and the same without FAST_NO_HZ, same kernel config).
> The full logs and associated "systemd-analyze plot" can be found
> (in comment 32) at :
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=806548
> 
> With FAST_NO_HZ, almost each rcu_prep_idle is followed by ksoftirqd
> (75000 ksoftirqd lines with FAST_NO_HZ, 4000 without).
> 
> Sorry, the logs are very huge, but I can't figure where are the 
> plots of some interest. 

Thank you for collecting them!  I clearly will need to do some scripting.  ;-)

							Thanx, Paul