[PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[Mukesh Kumar Chaurasiya (IBM)]

ct_kernel_exit() unconditionally calls rcu_preempt_deferred_qs(current)
on every return to userspace.  On the common fast path nothing is
actually deferred, so this is a needless write to
current->rcu_read_unlock_special -- a word that lives on the task_struct
and is therefore subject to cross-CPU cache-line traffic.

On weakly-ordered architectures such as ppc64le, rcu_read_lock() and
rcu_read_unlock() already issue lwsync/isync barriers and touch that
same cache line in the syscall body.  Bouncing it again at syscall exit
adds measurable overhead, particularly on workloads with a high syscall
rate (e.g. SELinux-heavy workloads where every AVC check issues a
system call).

Introduce rcu_ct_kernel_exit_qs() which wraps the deferred-QS call with
a rcu_preempt_need_deferred_qs() guard, matching the pattern already
used in rcu_flavor_sched_clock_irq():

  notrace void rcu_ct_kernel_exit_qs(void)
  {
          if (rcu_preempt_need_deferred_qs(current))
                  rcu_preempt_deferred_qs(current);
  }

The declaration is added to <linux/rcutree.h> and a stub no-op is added
to <linux/rcutiny.h> so that TINY_RCU builds are unaffected.

ct_kernel_exit() is updated to call rcu_ct_kernel_exit_qs() in place of
the direct rcu_preempt_deferred_qs() call.  Semantics are identical when
a deferred QS is actually pending; only the unnecessary write on the
fast path is eliminated.

Signed-off-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
---
 include/linux/rcutiny.h   |  1 +
 include/linux/rcutree.h   |  1 +
 kernel/context_tracking.c |  2 +-
 kernel/rcu/tree.c         | 19 +++++++++++++++++++
 4 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index e56ded733b1b..dcad641eb2c2 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -120,6 +120,7 @@ static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
 	return false;
 }
 static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
+static inline void rcu_ct_kernel_exit_qs(void) { }
 void rcu_scheduler_starting(void);
 static inline void rcu_end_inkernel_boot(void) { }
 static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 16a04202888b..d623f2a7d3fc 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -87,6 +87,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
 
 struct task_struct;
 void rcu_preempt_deferred_qs(struct task_struct *t);
+void rcu_ct_kernel_exit_qs(void);
 
 void exit_rcu(void);
 
diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
index a743e7ffa6c0..011018214c6d 100644
--- a/kernel/context_tracking.c
+++ b/kernel/context_tracking.c
@@ -118,7 +118,7 @@ static void noinstr ct_kernel_exit(bool user, int offset)
 	lockdep_assert_irqs_disabled();
 	trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching());
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
-	rcu_preempt_deferred_qs(current);
+	rcu_ct_kernel_exit_qs();
 
 	// instrumentation for the noinstr ct_kernel_exit_state()
 	instrument_atomic_write(&ct->state, sizeof(ct->state));
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 96848fc1f02b..c23478f70c17 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -368,6 +368,25 @@ notrace void rcu_momentary_eqs(void)
 }
 EXPORT_SYMBOL_GPL(rcu_momentary_eqs);
 
+/**
+ * rcu_ct_kernel_exit_qs - report deferred QS on syscall/exception exit if needed
+ *
+ * Called from ct_kernel_exit() on every return to userspace.  Guards the
+ * rcu_preempt_deferred_qs() call with rcu_preempt_need_deferred_qs() so that
+ * on the common fast path -- where nothing is deferred -- we avoid the
+ * cache-line traffic on current->rcu_read_unlock_special that the unconditional
+ * call causes.  This is particularly significant on weakly-ordered architectures
+ * (e.g. ppc64le) where rcu_read_lock/unlock issue lwsync/isync barriers and
+ * already touch that cache line in the syscall body.
+ *
+ * Follows the same pattern used by rcu_flavor_sched_clock_irq().
+ */
+notrace void rcu_ct_kernel_exit_qs(void)
+{
+	if (rcu_preempt_need_deferred_qs(current))
+		rcu_preempt_deferred_qs(current);
+}
+
 /**
  * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
-- 
2.55.0

Re: [PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[Frederic Weisbecker]

Le Tue, Sep 15, 2026 at 03:52:41PM +0530, Mukesh Kumar Chaurasiya (IBM) a écrit :
> ct_kernel_exit() unconditionally calls rcu_preempt_deferred_qs(current)
> on every return to userspace.

When nohz_full is active, right?

> On the common fast path nothing is
> actually deferred, so this is a needless write to
> current->rcu_read_unlock_special

When nothing is to be deferred, rcu_preempt_deferred_qs() does nothing, right?

And if some nohz_full workloads involve rare syscalls, they usually run a single
task, so no preemption that would trigger a deferred qs.


> -- a word that lives on the task_struct
> and is therefore subject to cross-CPU cache-line traffic.
> 
> On weakly-ordered architectures such as ppc64le, rcu_read_lock() and
> rcu_read_unlock() already issue lwsync/isync barriers and touch that
> same cache line in the syscall body.  Bouncing it again at syscall exit
> adds measurable overhead, particularly on workloads with a high syscall
> rate (e.g. SELinux-heavy workloads where every AVC check issues a
> system call).
> 
> Introduce rcu_ct_kernel_exit_qs() which wraps the deferred-QS call with
> a rcu_preempt_need_deferred_qs() guard, matching the pattern already
> used in rcu_flavor_sched_clock_irq():
> 
>   notrace void rcu_ct_kernel_exit_qs(void)
>   {
>           if (rcu_preempt_need_deferred_qs(current))
>                   rcu_preempt_deferred_qs(current);
>   }
> 
> The declaration is added to <linux/rcutree.h> and a stub no-op is added
> to <linux/rcutiny.h> so that TINY_RCU builds are unaffected.
> 
> ct_kernel_exit() is updated to call rcu_ct_kernel_exit_qs() in place of
> the direct rcu_preempt_deferred_qs() call.  Semantics are identical when
> a deferred QS is actually pending; only the unnecessary write on the
> fast path is eliminated.
> 
> Signed-off-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
> ---
>  include/linux/rcutiny.h   |  1 +
>  include/linux/rcutree.h   |  1 +
>  kernel/context_tracking.c |  2 +-
>  kernel/rcu/tree.c         | 19 +++++++++++++++++++
>  4 files changed, 22 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> index e56ded733b1b..dcad641eb2c2 100644
> --- a/include/linux/rcutiny.h
> +++ b/include/linux/rcutiny.h
> @@ -120,6 +120,7 @@ static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
>  	return false;
>  }
>  static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
> +static inline void rcu_ct_kernel_exit_qs(void) { }
>  void rcu_scheduler_starting(void);
>  static inline void rcu_end_inkernel_boot(void) { }
>  static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
> diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> index 16a04202888b..d623f2a7d3fc 100644
> --- a/include/linux/rcutree.h
> +++ b/include/linux/rcutree.h
> @@ -87,6 +87,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
>  
>  struct task_struct;
>  void rcu_preempt_deferred_qs(struct task_struct *t);
> +void rcu_ct_kernel_exit_qs(void);
>  
>  void exit_rcu(void);
>  
> diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
> index a743e7ffa6c0..011018214c6d 100644
> --- a/kernel/context_tracking.c
> +++ b/kernel/context_tracking.c
> @@ -118,7 +118,7 @@ static void noinstr ct_kernel_exit(bool user, int offset)
>  	lockdep_assert_irqs_disabled();
>  	trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching());
>  	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
> -	rcu_preempt_deferred_qs(current);
> +	rcu_ct_kernel_exit_qs();
>  
>  	// instrumentation for the noinstr ct_kernel_exit_state()
>  	instrument_atomic_write(&ct->state, sizeof(ct->state));
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 96848fc1f02b..c23478f70c17 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -368,6 +368,25 @@ notrace void rcu_momentary_eqs(void)
>  }
>  EXPORT_SYMBOL_GPL(rcu_momentary_eqs);
>  
> +/**
> + * rcu_ct_kernel_exit_qs - report deferred QS on syscall/exception exit if needed
> + *
> + * Called from ct_kernel_exit() on every return to userspace.  Guards the
> + * rcu_preempt_deferred_qs() call with rcu_preempt_need_deferred_qs() so that
> + * on the common fast path -- where nothing is deferred -- we avoid the
> + * cache-line traffic on current->rcu_read_unlock_special that the unconditional
> + * call causes.  This is particularly significant on weakly-ordered architectures
> + * (e.g. ppc64le) where rcu_read_lock/unlock issue lwsync/isync barriers and
> + * already touch that cache line in the syscall body.
> + *
> + * Follows the same pattern used by rcu_flavor_sched_clock_irq().
> + */
> +notrace void rcu_ct_kernel_exit_qs(void)
> +{
> +	if (rcu_preempt_need_deferred_qs(current))
> +		rcu_preempt_deferred_qs(current);
> +}
> +

rcu_preempt_deferred_qs() already has a rcu_preempt_need_deferred_qs() fast
path. Am I missing something?

Thanks.


>  /**
>   * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
>   *
> -- 
> 2.55.0
> 

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[Mukesh Kumar Chaurasiya]

On Tue, Sep 15, 2026 at 02:02:21PM +0200, Frederic Weisbecker wrote:
> Le Tue, Sep 15, 2026 at 03:52:41PM +0530, Mukesh Kumar Chaurasiya (IBM) a écrit :
> > ct_kernel_exit() unconditionally calls rcu_preempt_deferred_qs(current)
> > on every return to userspace.
> 
> When nohz_full is active, right?
> 
Yes, correct. I should have been more precise — this is only reached
on nohz_full CPUs under CONFIG_CONTEXT_TRACKING_IDLE.
> > On the common fast path nothing is
> > actually deferred, so this is a needless write to
> > current->rcu_read_unlock_special
> 
> When nothing is to be deferred, rcu_preempt_deferred_qs() does nothing, right?
> 
> And if some nohz_full workloads involve rare syscalls, they usually run a single
> task, so no preemption that would trigger a deferred qs.
> 
> 
You are right, I missed that. rcu_preempt_deferred_qs() already
bails out at line 619 of tree_plugin.h via the same
!rcu_preempt_need_deferred_qs() check, so the outer guard in
rcu_ct_kernel_exit_qs() is entirely redundant.

The overhead I was seeing on ppc64le with SELinux-heavy workloads is
real, but I clearly haven't identified the right call site yet. I'll
drop this patch and profile more carefully before sending a v2.

Thanks for catching this.

Regards
Mukesh
> > -- a word that lives on the task_struct
> > and is therefore subject to cross-CPU cache-line traffic.
> > 
> > On weakly-ordered architectures such as ppc64le, rcu_read_lock() and
> > rcu_read_unlock() already issue lwsync/isync barriers and touch that
> > same cache line in the syscall body.  Bouncing it again at syscall exit
> > adds measurable overhead, particularly on workloads with a high syscall
> > rate (e.g. SELinux-heavy workloads where every AVC check issues a
> > system call).
> > 
> > Introduce rcu_ct_kernel_exit_qs() which wraps the deferred-QS call with
> > a rcu_preempt_need_deferred_qs() guard, matching the pattern already
> > used in rcu_flavor_sched_clock_irq():
> > 
> >   notrace void rcu_ct_kernel_exit_qs(void)
> >   {
> >           if (rcu_preempt_need_deferred_qs(current))
> >                   rcu_preempt_deferred_qs(current);
> >   }
> > 
> > The declaration is added to <linux/rcutree.h> and a stub no-op is added
> > to <linux/rcutiny.h> so that TINY_RCU builds are unaffected.
> > 
> > ct_kernel_exit() is updated to call rcu_ct_kernel_exit_qs() in place of
> > the direct rcu_preempt_deferred_qs() call.  Semantics are identical when
> > a deferred QS is actually pending; only the unnecessary write on the
> > fast path is eliminated.
> > 
> > Signed-off-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
> > ---
> >  include/linux/rcutiny.h   |  1 +
> >  include/linux/rcutree.h   |  1 +
> >  kernel/context_tracking.c |  2 +-
> >  kernel/rcu/tree.c         | 19 +++++++++++++++++++
> >  4 files changed, 22 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > index e56ded733b1b..dcad641eb2c2 100644
> > --- a/include/linux/rcutiny.h
> > +++ b/include/linux/rcutiny.h
> > @@ -120,6 +120,7 @@ static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
> >  	return false;
> >  }
> >  static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
> > +static inline void rcu_ct_kernel_exit_qs(void) { }
> >  void rcu_scheduler_starting(void);
> >  static inline void rcu_end_inkernel_boot(void) { }
> >  static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
> > diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> > index 16a04202888b..d623f2a7d3fc 100644
> > --- a/include/linux/rcutree.h
> > +++ b/include/linux/rcutree.h
> > @@ -87,6 +87,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
> >  
> >  struct task_struct;
> >  void rcu_preempt_deferred_qs(struct task_struct *t);
> > +void rcu_ct_kernel_exit_qs(void);
> >  
> >  void exit_rcu(void);
> >  
> > diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
> > index a743e7ffa6c0..011018214c6d 100644
> > --- a/kernel/context_tracking.c
> > +++ b/kernel/context_tracking.c
> > @@ -118,7 +118,7 @@ static void noinstr ct_kernel_exit(bool user, int offset)
> >  	lockdep_assert_irqs_disabled();
> >  	trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching());
> >  	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
> > -	rcu_preempt_deferred_qs(current);
> > +	rcu_ct_kernel_exit_qs();
> >  
> >  	// instrumentation for the noinstr ct_kernel_exit_state()
> >  	instrument_atomic_write(&ct->state, sizeof(ct->state));
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 96848fc1f02b..c23478f70c17 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -368,6 +368,25 @@ notrace void rcu_momentary_eqs(void)
> >  }
> >  EXPORT_SYMBOL_GPL(rcu_momentary_eqs);
> >  
> > +/**
> > + * rcu_ct_kernel_exit_qs - report deferred QS on syscall/exception exit if needed
> > + *
> > + * Called from ct_kernel_exit() on every return to userspace.  Guards the
> > + * rcu_preempt_deferred_qs() call with rcu_preempt_need_deferred_qs() so that
> > + * on the common fast path -- where nothing is deferred -- we avoid the
> > + * cache-line traffic on current->rcu_read_unlock_special that the unconditional
> > + * call causes.  This is particularly significant on weakly-ordered architectures
> > + * (e.g. ppc64le) where rcu_read_lock/unlock issue lwsync/isync barriers and
> > + * already touch that cache line in the syscall body.
> > + *
> > + * Follows the same pattern used by rcu_flavor_sched_clock_irq().
> > + */
> > +notrace void rcu_ct_kernel_exit_qs(void)
> > +{
> > +	if (rcu_preempt_need_deferred_qs(current))
> > +		rcu_preempt_deferred_qs(current);
> > +}
> > +
> 
> rcu_preempt_deferred_qs() already has a rcu_preempt_need_deferred_qs() fast
> path. Am I missing something?
> 
> Thanks.
> 
> 
> >  /**
> >   * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
> >   *
> > -- 
> > 2.55.0
> > 
> 
> -- 
> Frederic Weisbecker
> SUSE Labs

Re: [PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[Paul E. McKenney]

On Tue, Sep 15, 2026 at 03:52:41PM +0530, Mukesh Kumar Chaurasiya (IBM) wrote:
> ct_kernel_exit() unconditionally calls rcu_preempt_deferred_qs(current)
> on every return to userspace.  On the common fast path nothing is
> actually deferred, so this is a needless write to
> current->rcu_read_unlock_special -- a word that lives on the task_struct
> and is therefore subject to cross-CPU cache-line traffic.
> 
> On weakly-ordered architectures such as ppc64le, rcu_read_lock() and
> rcu_read_unlock() already issue lwsync/isync barriers and touch that
> same cache line in the syscall body.

You lost me on this one.  The non-debugging operations in preemptible
rcu_read_lock() are:

	WRITE_ONCE(current->rcu_read_lock_nesting, READ_ONCE(current->rcu_read_lock_nesting) + 1);
	barrier(); // Empty as with the "memory" clobber.

I don't see how this emits lwsync or isync instructions.

Now I do agree that rcu_read_unlock() will emit all sorts of heavyweight
instructions in the case where there are deferred quiescent states,
which is the only case where rcu_preempt_need_deferred_qs() is important.
But I don't see how rcu_read_unlock() emits isync or lwsync in the normal
case where the preceding RCU read-side critical section was not preempted.

But on rcu_read_lock(), what am I missing?

>                                       Bouncing it again at syscall exit
> adds measurable overhead, particularly on workloads with a high syscall
> rate (e.g. SELinux-heavy workloads where every AVC check issues a
> system call).
> 
> Introduce rcu_ct_kernel_exit_qs() which wraps the deferred-QS call with
> a rcu_preempt_need_deferred_qs() guard, matching the pattern already
> used in rcu_flavor_sched_clock_irq():
> 
>   notrace void rcu_ct_kernel_exit_qs(void)
>   {
>           if (rcu_preempt_need_deferred_qs(current))
>                   rcu_preempt_deferred_qs(current);
>   }
> 
> The declaration is added to <linux/rcutree.h> and a stub no-op is added
> to <linux/rcutiny.h> so that TINY_RCU builds are unaffected.
> 
> ct_kernel_exit() is updated to call rcu_ct_kernel_exit_qs() in place of
> the direct rcu_preempt_deferred_qs() call.  Semantics are identical when
> a deferred QS is actually pending; only the unnecessary write on the
> fast path is eliminated.
> 
> Signed-off-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
> ---
>  include/linux/rcutiny.h   |  1 +
>  include/linux/rcutree.h   |  1 +
>  kernel/context_tracking.c |  2 +-
>  kernel/rcu/tree.c         | 19 +++++++++++++++++++
>  4 files changed, 22 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> index e56ded733b1b..dcad641eb2c2 100644
> --- a/include/linux/rcutiny.h
> +++ b/include/linux/rcutiny.h
> @@ -120,6 +120,7 @@ static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
>  	return false;
>  }
>  static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
> +static inline void rcu_ct_kernel_exit_qs(void) { }
>  void rcu_scheduler_starting(void);
>  static inline void rcu_end_inkernel_boot(void) { }
>  static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
> diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> index 16a04202888b..d623f2a7d3fc 100644
> --- a/include/linux/rcutree.h
> +++ b/include/linux/rcutree.h
> @@ -87,6 +87,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
>  
>  struct task_struct;
>  void rcu_preempt_deferred_qs(struct task_struct *t);
> +void rcu_ct_kernel_exit_qs(void);
>  
>  void exit_rcu(void);
>  
> diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
> index a743e7ffa6c0..011018214c6d 100644
> --- a/kernel/context_tracking.c
> +++ b/kernel/context_tracking.c
> @@ -118,7 +118,7 @@ static void noinstr ct_kernel_exit(bool user, int offset)
>  	lockdep_assert_irqs_disabled();
>  	trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching());
>  	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
> -	rcu_preempt_deferred_qs(current);
> +	rcu_ct_kernel_exit_qs();

You also lost me here.  The rcu_preempt_deferred_qs() function is as
follows:

notrace void rcu_preempt_deferred_qs(struct task_struct *t)
{
	unsigned long flags;

	if (!rcu_preempt_need_deferred_qs(t))
		return;
	local_irq_save(flags);
	rcu_preempt_deferred_qs_irqrestore(t, flags);
}

So the only savings is the local_irq_save(), and even then only in the
uncommon case where rcu_preempt_need_deferred_qs() returns true.

So again, what am I missing here?

							Thanx, Paul

>  	// instrumentation for the noinstr ct_kernel_exit_state()
>  	instrument_atomic_write(&ct->state, sizeof(ct->state));
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 96848fc1f02b..c23478f70c17 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -368,6 +368,25 @@ notrace void rcu_momentary_eqs(void)
>  }
>  EXPORT_SYMBOL_GPL(rcu_momentary_eqs);
>  
> +/**
> + * rcu_ct_kernel_exit_qs - report deferred QS on syscall/exception exit if needed
> + *
> + * Called from ct_kernel_exit() on every return to userspace.  Guards the
> + * rcu_preempt_deferred_qs() call with rcu_preempt_need_deferred_qs() so that
> + * on the common fast path -- where nothing is deferred -- we avoid the
> + * cache-line traffic on current->rcu_read_unlock_special that the unconditional
> + * call causes.  This is particularly significant on weakly-ordered architectures
> + * (e.g. ppc64le) where rcu_read_lock/unlock issue lwsync/isync barriers and
> + * already touch that cache line in the syscall body.
> + *
> + * Follows the same pattern used by rcu_flavor_sched_clock_irq().
> + */
> +notrace void rcu_ct_kernel_exit_qs(void)
> +{
> +	if (rcu_preempt_need_deferred_qs(current))
> +		rcu_preempt_deferred_qs(current);
> +}
> +
>  /**
>   * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
>   *
> -- 
> 2.55.0
> 

Re: [PATCH] rcu: Guard deferred QS on kernel exit behind need_deferred_qs() check

[Mukesh Kumar Chaurasiya]

On Tue, Sep 15, 2026 at 05:06:39PM -0700, Paul E. McKenney wrote:
> On Tue, Sep 15, 2026 at 03:52:41PM +0530, Mukesh Kumar Chaurasiya (IBM) wrote:
> > ct_kernel_exit() unconditionally calls rcu_preempt_deferred_qs(current)
> > on every return to userspace.  On the common fast path nothing is
> > actually deferred, so this is a needless write to
> > current->rcu_read_unlock_special -- a word that lives on the task_struct
> > and is therefore subject to cross-CPU cache-line traffic.
> > 
> > On weakly-ordered architectures such as ppc64le, rcu_read_lock() and
> > rcu_read_unlock() already issue lwsync/isync barriers and touch that
> > same cache line in the syscall body.
> 
> You lost me on this one.  The non-debugging operations in preemptible
> rcu_read_lock() are:
> 
> 	WRITE_ONCE(current->rcu_read_lock_nesting, READ_ONCE(current->rcu_read_lock_nesting) + 1);
> 	barrier(); // Empty as with the "memory" clobber.
> 
> I don't see how this emits lwsync or isync instructions.
> 
> Now I do agree that rcu_read_unlock() will emit all sorts of heavyweight
> instructions in the case where there are deferred quiescent states,
> which is the only case where rcu_preempt_need_deferred_qs() is important.
> But I don't see how rcu_read_unlock() emits isync or lwsync in the normal
> case where the preceding RCU read-side critical section was not preempted.
> 
> But on rcu_read_lock(), what am I missing?
> 
Hey Paul,
You are correct.  Looking at __rcu_read_lock() in tree_plugin.h, it is
just a WRITE_ONCE on ->rcu_read_lock_nesting and a compiler barrier().
No lwsync, no isync.  That claim in the commit message was wrong.

> >                                       Bouncing it again at syscall exit
> > adds measurable overhead, particularly on workloads with a high syscall
> > rate (e.g. SELinux-heavy workloads where every AVC check issues a
> > system call).
> > 
> > Introduce rcu_ct_kernel_exit_qs() which wraps the deferred-QS call with
> > a rcu_preempt_need_deferred_qs() guard, matching the pattern already
> > used in rcu_flavor_sched_clock_irq():
> > 
> >   notrace void rcu_ct_kernel_exit_qs(void)
> >   {
> >           if (rcu_preempt_need_deferred_qs(current))
> >                   rcu_preempt_deferred_qs(current);
> >   }
> > 
> > The declaration is added to <linux/rcutree.h> and a stub no-op is added
> > to <linux/rcutiny.h> so that TINY_RCU builds are unaffected.
> > 
> > ct_kernel_exit() is updated to call rcu_ct_kernel_exit_qs() in place of
> > the direct rcu_preempt_deferred_qs() call.  Semantics are identical when
> > a deferred QS is actually pending; only the unnecessary write on the
> > fast path is eliminated.
> > 
> > Signed-off-by: Mukesh Kumar Chaurasiya (IBM) <mkchauras@gmail.com>
> > ---
> >  include/linux/rcutiny.h   |  1 +
> >  include/linux/rcutree.h   |  1 +
> >  kernel/context_tracking.c |  2 +-
> >  kernel/rcu/tree.c         | 19 +++++++++++++++++++
> >  4 files changed, 22 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
> > index e56ded733b1b..dcad641eb2c2 100644
> > --- a/include/linux/rcutiny.h
> > +++ b/include/linux/rcutiny.h
> > @@ -120,6 +120,7 @@ static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
> >  	return false;
> >  }
> >  static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
> > +static inline void rcu_ct_kernel_exit_qs(void) { }
> >  void rcu_scheduler_starting(void);
> >  static inline void rcu_end_inkernel_boot(void) { }
> >  static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
> > diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
> > index 16a04202888b..d623f2a7d3fc 100644
> > --- a/include/linux/rcutree.h
> > +++ b/include/linux/rcutree.h
> > @@ -87,6 +87,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
> >  
> >  struct task_struct;
> >  void rcu_preempt_deferred_qs(struct task_struct *t);
> > +void rcu_ct_kernel_exit_qs(void);
> >  
> >  void exit_rcu(void);
> >  
> > diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
> > index a743e7ffa6c0..011018214c6d 100644
> > --- a/kernel/context_tracking.c
> > +++ b/kernel/context_tracking.c
> > @@ -118,7 +118,7 @@ static void noinstr ct_kernel_exit(bool user, int offset)
> >  	lockdep_assert_irqs_disabled();
> >  	trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching());
> >  	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
> > -	rcu_preempt_deferred_qs(current);
> > +	rcu_ct_kernel_exit_qs();
> 
> You also lost me here.  The rcu_preempt_deferred_qs() function is as
> follows:
> 
> notrace void rcu_preempt_deferred_qs(struct task_struct *t)
> {
> 	unsigned long flags;
> 
> 	if (!rcu_preempt_need_deferred_qs(t))
> 		return;
> 	local_irq_save(flags);
> 	rcu_preempt_deferred_qs_irqrestore(t, flags);
> }
> 
> So the only savings is the local_irq_save(), and even then only in the
> uncommon case where rcu_preempt_need_deferred_qs() returns true.
> 
> So again, what am I missing here?
> 
> 							Thanx, Paul
> 
Nothing you are not missing anything.  Looking at
rcu_preempt_deferred_qs() again, the fast path (need_deferred_qs
returns false) already returns immediately with no writes and no
irq save. My wrapper adds an outer check that is redundant with
the one already inside.  In the case where need_deferred_qs() *is*
true, my patch saves a local_irq_save() only to then call
rcu_preempt_deferred_qs() which immediately does local_irq_save()
again — so it saves nothing even there.

I will drop this patch and properly profile the regression before
sending anything further.

Sorry for the noise.

Regards,
Mukesh

> >  	// instrumentation for the noinstr ct_kernel_exit_state()
> >  	instrument_atomic_write(&ct->state, sizeof(ct->state));
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 96848fc1f02b..c23478f70c17 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -368,6 +368,25 @@ notrace void rcu_momentary_eqs(void)
> >  }
> >  EXPORT_SYMBOL_GPL(rcu_momentary_eqs);
> >  
> > +/**
> > + * rcu_ct_kernel_exit_qs - report deferred QS on syscall/exception exit if needed
> > + *
> > + * Called from ct_kernel_exit() on every return to userspace.  Guards the
> > + * rcu_preempt_deferred_qs() call with rcu_preempt_need_deferred_qs() so that
> > + * on the common fast path -- where nothing is deferred -- we avoid the
> > + * cache-line traffic on current->rcu_read_unlock_special that the unconditional
> > + * call causes.  This is particularly significant on weakly-ordered architectures
> > + * (e.g. ppc64le) where rcu_read_lock/unlock issue lwsync/isync barriers and
> > + * already touch that cache line in the syscall body.
> > + *
> > + * Follows the same pattern used by rcu_flavor_sched_clock_irq().
> > + */
> > +notrace void rcu_ct_kernel_exit_qs(void)
> > +{
> > +	if (rcu_preempt_need_deferred_qs(current))
> > +		rcu_preempt_deferred_qs(current);
> > +}
> > +
> >  /**
> >   * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
> >   *
> > -- 
> > 2.55.0
> >