[PATCH 0/4] srcu: NMI-safe atomic SRCU readers, and their irqsave forms

[PATCH 0/4] srcu: NMI-safe atomic SRCU readers, and their irqsave forms

[Mathieu Desnoyers]

Hi Paul,

This series adapts atomic SRCU to the needs of RCU pseudo-transactions
[1], as we discussed [2].  Their readers resolve a proxy within a short
section, in any context, NMI included.  A writer waits for these
sections without sleeping, sometimes with interrupts disabled.  Atomic
SRCU fits, with two additions:

- Patch 1 makes the atomic readers NMI-safe.  They use the counter
  increments of __srcu_read_lock_nmisafe(), which keep the full barriers
  that synchronize_srcu_atomic() pairs with.
- Patch 2 adds srcu_read_lock_atomic_irqsave() and
  srcu_read_unlock_atomic_irqrestore().  They disable interrupts instead
  of preemption, for a domain whose writers wait with interrupts
  disabled.
- Patch 3 makes rcutorture run readers from NMI, for the NMI-safe reader
  flavors.
- Patch 4 makes rcutorture use the irqsave forms for half of the atomic
  readers, with a new atomic_irqsave parameter.  It also adds an SRCU-P
  run of this mode to torture.sh.

On architectures without NMI-safe this_cpu_inc(), every atomic reader
now pays an atomic increment at lock and at unlock.  The update side is
unchanged.  On architectures with NMI-safe this_cpu_inc(), such as x86,
the readers use the same counter increments as before.  David agreed to
the change, as long as synchronize_srcu_atomic() stays within the 4-16us
it was designed for [3].

Testing was on x86_64 under QEMU/KVM, with
CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y:

- SRCU-P with atomic readers, half of them irqsave, for 5 minutes: no
  error.  It ran 32529 grace periods and 104114 readers in NMI.
- SRCU-P with the NMI-safe reader flavor, then with the fast one, for 2
  minutes each: no error.  Both ran readers in NMI.
- SRCU-T (Tiny SRCU) with atomic readers, half of them irqsave, for 2
  minutes: no error.

Each new check fails when what it checks is broken:

- A reader's counter update is split in two, with an NMI reader in
  between: the writer stalls in its grace period.
- The flavor check rejects atomic readers in NMI: it warns.
- An irqsave unlock restores interrupts while another irqsave reader of
  its CPU is held: the new check in rcutorture warns.

The readers in NMI need a PMU in the guest.  kvm.sh starts x86 guests
with -cpu kvm64, which has none.  So these runs added --qemu-args
"-cpu host,pmu=on".  Without it, rcutorture reports that NMIs went
untested.  This also holds for the existing test of ->call() from NMI.

This series was developed with assistance from an LLM.  I reviewed the
code and the tests.

The series is based on 8a36f53fd5ec of Linus' master (7.4 merge
window).

[1] https://lore.kernel.org/all/20261016133319.433617-1-mathieu.desnoyers@efficios.com/
[2] https://lore.kernel.org/all/853d8469-94a4-4b33-a0d5-d6d2138fe3ef@paulmck-laptop/
[3] https://lore.kernel.org/all/84fdca577207092bd16fc8f3949c9b7a776750ba.camel@infradead.org/

Thanks,

Mathieu

Mathieu Desnoyers (4):
  srcu: Make atomic SRCU readers NMI-safe
  srcu: Add irqsave variants of the atomic SRCU readers
  rcutorture: Run NMI-safe SRCU readers from NMI
  rcutorture: Test the irqsave forms of atomic SRCU readers

 include/linux/srcu.h                          | 108 ++++++++++----
 kernel/rcu/rcutorture.c                       | 134 ++++++++++++++++--
 kernel/rcu/srcutree.c                         |   3 +-
 .../selftests/rcutorture/bin/torture.sh       |   6 +
 4 files changed, 216 insertions(+), 35 deletions(-)


base-commit: 8a36f53fd5ec3eff01f1b637d42cbdc3821d0493
-- 
2.47.3

[PATCH 1/4] srcu: Make atomic SRCU readers NMI-safe

[Mathieu Desnoyers]

Atomic SRCU readers count themselves with __srcu_read_lock(), whose
this_cpu_inc() is not NMI-safe on every architecture. With
CONFIG_PROVE_RCU, which lockdep enables, __srcu_check_read_flavor() also
warns when an atomic reader runs in NMI. So an NMI handler cannot use
atomic SRCU, while a tracer or a BPF program may need to read a
structure that atomic SRCU protects.

Count atomic readers with __srcu_read_lock_nmisafe() and
__srcu_read_unlock_nmisafe() instead. They use NMI-safe increments on
the architectures that need them, and keep the full barriers that
synchronize_srcu_atomic() pairs with. The __srcu_read_lock_fast()
helpers have no such barriers: they rely on a grace period that calls
synchronize_rcu(), which synchronize_srcu_atomic() cannot do while it
spins.

Also accept the atomic flavor in NMI in __srcu_check_read_flavor().
Annotate an atomic reader in NMI as a trylock for lockdep, as
srcu_read_lock_nmisafe() does. An NMI can interrupt lockdep itself, and
lockdep then reports any other acquisition of a lock that a grace
period has used.

On the architectures that need NMI-safe counters, every atomic reader
now pays an atomic increment at lock and at unlock.

Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/srcu.h  | 18 ++++++++++++++----
 kernel/rcu/srcutree.c |  3 ++-
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 1a8a465a5650..7a338e3a63d3 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -92,7 +92,7 @@ int init_srcu_struct_atomic(struct srcu_struct *ssp);
 //					0x04		// SRCU-lite is no longer with us.
 #define SRCU_READ_FLAVOR_FAST		0x04		// srcu_read_lock_fast(), also NMI-safe.
 #define SRCU_READ_FLAVOR_FAST_UPDOWN	0x08		// srcu_read_lock_fast_updown().
-#define SRCU_READ_FLAVOR_ATOMIC		0x10		// srcu_read_lock_atomic().
+#define SRCU_READ_FLAVOR_ATOMIC		0x10		// srcu_read_lock_atomic(), also NMI-safe.
 #define SRCU_READ_FLAVOR_ALL		(SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \
 					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN | \
 					 SRCU_READ_FLAVOR_ATOMIC)
@@ -336,6 +336,9 @@ static inline int srcu_read_lock(struct srcu_struct *ssp)
  * The lock and matching srcu_read_unlock_atomic() must be invoked on
  * the same CPU, from the same context; passing the return value to
  * another task is not permitted for this flavor.
+ *
+ * This function is NMI-safe, in a manner similar to
+ * srcu_read_lock_nmisafe().
  */
 static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
 	__acquires_shared(ssp)
@@ -355,8 +358,15 @@ static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
 	if (in_task())
 		non_block_start();
 	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
-	retval = __srcu_read_lock(ssp);
-	srcu_lock_acquire(&ssp->dep_map);
+	retval = __srcu_read_lock_nmisafe(ssp);
+	/*
+	 * An NMI may interrupt lockdep itself.  Annotate it as a trylock
+	 * there, as srcu_read_lock_nmisafe() does.
+	 */
+	if (in_nmi())
+		rcu_try_lock_acquire(&ssp->dep_map);
+	else
+		srcu_lock_acquire(&ssp->dep_map);
 	return retval;
 }
 
@@ -563,7 +573,7 @@ static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
 	WARN_ON_ONCE(idx & ~0x1);
 	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
 	srcu_lock_release(&ssp->dep_map);
-	__srcu_read_unlock(ssp, idx);
+	__srcu_read_unlock_nmisafe(ssp, idx);
 	if (in_task())
 		non_block_end();
 	preempt_enable();
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 2b0e2b2d2c18..6aebd3e31654 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -828,7 +828,8 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
 
 	/* NMI-unsafe use in NMI is a bad sign, as is multi-bit read_flavor values. */
 	WARN_ON_ONCE(read_flavor != SRCU_READ_FLAVOR_NMI &&
-		     read_flavor != SRCU_READ_FLAVOR_FAST && in_nmi());
+		     read_flavor != SRCU_READ_FLAVOR_FAST &&
+		     read_flavor != SRCU_READ_FLAVOR_ATOMIC && in_nmi());
 	WARN_ON_ONCE(read_flavor & (read_flavor - 1));
 
 	sdp = raw_cpu_ptr(ssp->sda);
-- 
2.47.3

[PATCH 2/4] srcu: Add irqsave variants of the atomic SRCU readers

[Mathieu Desnoyers]

synchronize_srcu_atomic() may be invoked with interrupts disabled, for
example under a raw spinlock. Its kerneldoc then requires every reader
of that srcu_struct to disable interrupts too. srcu_read_lock_atomic()
only disables preemption, so such a reader has to disable interrupts
itself around it.

Add srcu_read_lock_atomic_irqsave() and
srcu_read_unlock_atomic_irqrestore(). They disable interrupts instead
of preemption, and keep the interrupt state in a flags argument, so
they are macros. Both forms share the rest of the reader through
__srcu_read_lock_atomic() and __srcu_read_unlock_atomic().

Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/linux/srcu.h | 100 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 75 insertions(+), 25 deletions(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 7a338e3a63d3..78f8b3c14819 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -321,31 +321,15 @@ static inline int srcu_read_lock(struct srcu_struct *ssp)
 	return retval;
 }
 
-/**
- * srcu_read_lock_atomic - register a new reader promising an atomic section
- * @ssp: srcu_struct in which to register the new reader.
- *
- * As srcu_read_lock(), but the caller promises that the read-side
- * critical section never sleeps and never blocks on anything which
- * may itself depend on memory allocation to make progress. Preemption
- * is disabled for the duration, which both enforces that promise (any
- * sleepable call in the section will splat on every configuration)
- * and bounds the section so that the update side may spin rather
- * than sleep when waiting for readers: see synchronize_srcu_atomic().
- *
- * The lock and matching srcu_read_unlock_atomic() must be invoked on
- * the same CPU, from the same context; passing the return value to
- * another task is not permitted for this flavor.
- *
- * This function is NMI-safe, in a manner similar to
- * srcu_read_lock_nmisafe().
+/*
+ * Register an atomic-section reader.  The caller has disabled preemption,
+ * or interrupts.
  */
-static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
+static inline int __srcu_read_lock_atomic(struct srcu_struct *ssp)
 	__acquires_shared(ssp)
 {
 	int retval;
 
-	preempt_disable();
 	/*
 	 * Arm might_sleep() to catch even a *potentially* sleeping call
 	 * in the section, not just an actual schedule: the atomic-domain
@@ -370,6 +354,50 @@ static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
 	return retval;
 }
 
+/**
+ * srcu_read_lock_atomic - register a new reader promising an atomic section
+ * @ssp: srcu_struct in which to register the new reader.
+ *
+ * As srcu_read_lock(), but the caller promises that the read-side
+ * critical section never sleeps and never blocks on anything which
+ * may itself depend on memory allocation to make progress. Preemption
+ * is disabled for the duration, which both enforces that promise (any
+ * sleepable call in the section will splat on every configuration)
+ * and bounds the section so that the update side may spin rather
+ * than sleep when waiting for readers: see synchronize_srcu_atomic().
+ *
+ * The lock and matching srcu_read_unlock_atomic() must be invoked on
+ * the same CPU, from the same context; passing the return value to
+ * another task is not permitted for this flavor.
+ *
+ * This function is NMI-safe, in a manner similar to
+ * srcu_read_lock_nmisafe().
+ */
+static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
+	__acquires_shared(ssp)
+{
+	preempt_disable();
+	return __srcu_read_lock_atomic(ssp);
+}
+
+/**
+ * srcu_read_lock_atomic_irqsave - register an atomic reader, interrupts off
+ * @ssp: srcu_struct in which to register the new reader.
+ * @flags: interrupt state to restore at srcu_read_unlock_atomic_irqrestore().
+ *
+ * As srcu_read_lock_atomic(), but with interrupts disabled for the
+ * duration instead of preemption.  Use it when synchronize_srcu_atomic()
+ * may be invoked with interrupts disabled for this srcu_struct: its
+ * readers must then disable interrupts too.
+ *
+ * Returns the index to pass to srcu_read_unlock_atomic_irqrestore().
+ */
+#define srcu_read_lock_atomic_irqsave(ssp, flags)			\
+({									\
+	local_irq_save(flags);						\
+	__srcu_read_lock_atomic((ssp));					\
+})
+
 /**
  * srcu_read_lock_fast - register a new reader for an SRCU-protected structure.
  * @ssp: srcu_struct in which to register the new reader.
@@ -562,12 +590,11 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)
 	__srcu_read_unlock(ssp, idx);
 }
 
-/**
- * srcu_read_unlock_atomic - unregister an atomic-section reader
- * @ssp: srcu_struct from which to unregister the old reader.
- * @idx: return value from corresponding srcu_read_lock_atomic().
+/*
+ * Unregister an atomic-section reader.  The caller then enables preemption,
+ * or interrupts, again.
  */
-static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
+static inline void __srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
 	__releases_shared(ssp)
 {
 	WARN_ON_ONCE(idx & ~0x1);
@@ -576,9 +603,32 @@ static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
 	__srcu_read_unlock_nmisafe(ssp, idx);
 	if (in_task())
 		non_block_end();
+}
+
+/**
+ * srcu_read_unlock_atomic - unregister an atomic-section reader
+ * @ssp: srcu_struct from which to unregister the old reader.
+ * @idx: return value from corresponding srcu_read_lock_atomic().
+ */
+static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
+	__releases_shared(ssp)
+{
+	__srcu_read_unlock_atomic(ssp, idx);
 	preempt_enable();
 }
 
+/**
+ * srcu_read_unlock_atomic_irqrestore - unregister an irqsave atomic reader
+ * @ssp: srcu_struct from which to unregister the old reader.
+ * @idx: return value from corresponding srcu_read_lock_atomic_irqsave().
+ * @flags: interrupt state saved by srcu_read_lock_atomic_irqsave().
+ */
+#define srcu_read_unlock_atomic_irqrestore(ssp, idx, flags)		\
+do {									\
+	__srcu_read_unlock_atomic((ssp), (idx));			\
+	local_irq_restore(flags);					\
+} while (0)
+
 /**
  * srcu_read_unlock_fast - unregister a old reader from an SRCU-protected structure.
  * @ssp: srcu_struct in which to unregister the old reader.
-- 
2.47.3

[PATCH 3/4] rcutorture: Run NMI-safe SRCU readers from NMI

[Mathieu Desnoyers]

rcutorture exercises ->call() from NMI, but never runs a reader there.
So nothing tests the NMI-safe reader flavors where it matters most: in
an NMI that interrupts another reader during its counter update.

Run a reader from the same per-CPU perf-counter NMI, when every reader
flavor in use is NMI-safe. These flavors are srcu_read_lock_nmisafe(),
srcu_read_lock_fast() and srcu_read_lock_atomic(). A new ops hook,
->nmi_reader_capable(), reports whether the reader flavors in use are
all NMI-safe. A new nmi_readers module parameter, on by default, enables
these NMI readers. The statistics count these readers as nmi-reads.

The reader loads rcu_torture_current and checks its element as the
other readers do. The element must not be freed before the reader ends.
It must not go through two grace periods either.

The NMI may interrupt lockdep, which then does not record the reader.
So the reader loads the pointer with rcu_dereference_raw().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/rcu/rcutorture.c | 60 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 55 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 968343be4e0d..3b9c08a094cd 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -117,6 +117,7 @@ torture_param(int, n_barrier_cbs, 0, "# of callbacks/kthreads for barrier testin
 torture_param(int, n_up_down, 32, "# of concurrent up/down hrtimer-based RCU readers");
 torture_param(int, nfakewriters, 4, "Number of RCU fake writer threads");
 torture_param(bool, nmi_calls, true, "Exercise ->call() from NMI on nmi_capable flavors");
+torture_param(bool, nmi_readers, true, "Run readers from NMI with NMI-safe reader flavors");
 torture_param(int, nreaders, -1, "Number of RCU reader threads");
 torture_param(bool, nwriters, 1, "Number of RCU writer threads (0 or 1)");
 torture_param(int, object_debug, 0, "Enable debug-object double call_rcu() testing");
@@ -220,6 +221,7 @@ static atomic_long_t n_rcu_torture_timers;
 static atomic_long_t n_rcu_torture_irqs;
 static atomic_long_t n_rcu_torture_nmi_call;
 static atomic_long_t n_rcu_torture_nmi_cb;
+static atomic_long_t n_rcu_torture_nmi_read;
 static long n_barrier_attempts;
 static long n_barrier_successes; /* did rcu_barrier test succeed? */
 static unsigned long n_read_exits;
@@ -438,6 +440,7 @@ struct rcu_torture_ops {
 	long cbflood_max;
 	int irq_capable;
 	int nmi_capable;
+	bool (*nmi_reader_capable)(void); // Are ->readlock() and ->readunlock() NMI-safe?
 	int can_boost;
 	int extendables;
 	int slow_gps;
@@ -790,6 +793,14 @@ static int srcu_torture_read_lock(void)
 	return ret;
 }
 
+// Only these reader flavors may be used from NMI.
+static bool srcu_torture_nmi_reader_capable(void)
+{
+	return reader_flavor &&
+	       !(reader_flavor & ~(SRCU_READ_FLAVOR_NMI | SRCU_READ_FLAVOR_FAST |
+				   SRCU_READ_FLAVOR_ATOMIC));
+}
+
 static void
 srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)
 {
@@ -981,6 +992,7 @@ static struct rcu_torture_ops srcu_ops = {
 	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
 				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN,
 	.nmi_capable	= 1,
+	.nmi_reader_capable = srcu_torture_nmi_reader_capable,
 	.name		= "srcu"
 };
 
@@ -1049,6 +1061,7 @@ static struct rcu_torture_ops srcud_ops = {
 	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
 				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN,
 	.nmi_capable	= 1,
+	.nmi_reader_capable = srcu_torture_nmi_reader_capable,
 	.name		= "srcud"
 };
 
@@ -2711,6 +2724,10 @@ static DEFINE_TORTURE_RANDOM_PERCPU(rcu_torture_timer_rand);
  * per-CPU hardware perf counter overflows into an NMI, and its handler submits
  * a preallocated callback via ->call().  One callback per CPU is in flight at a
  * time (guarded by an atomic) to avoid allocating in NMI.
+ *
+ * The same handler also runs a reader, for flavors whose ->nmi_reader_capable()
+ * says that their readers are NMI-safe.  The reader may interrupt another
+ * reader on its CPU, in the middle of its counter update.
  */
 #ifdef CONFIG_PERF_EVENTS
 static struct perf_event_attr rcu_torture_nmi_attr = {
@@ -2735,6 +2752,8 @@ struct rcu_torture_nmi_cb {
 
 static struct perf_event **rcu_torture_nmi_events;
 static int rcu_torture_nmi_hp_state;
+static bool rcu_torture_nmi_do_calls;
+static bool rcu_torture_nmi_do_reads;
 static DEFINE_PER_CPU(struct rcu_torture_nmi_cb, rcu_torture_nmi_cb);
 
 static void rcu_torture_nmi_invoked(struct rcu_head *rhp)
@@ -2745,6 +2764,28 @@ static void rcu_torture_nmi_invoked(struct rcu_head *rhp)
 	atomic_set(&rtncp->inuse, 0);
 }
 
+/*
+ * A reader in NMI.  The NMI may interrupt lockdep, which then does not record
+ * the reader, so the pointer is loaded without a lockdep check.  The element
+ * must not go through two grace periods before the reader ends.
+ */
+static void rcu_torture_nmi_read(void)
+{
+	struct rcu_torture *p;
+	int idx;
+
+	idx = cur_ops->readlock();
+	p = rcu_dereference_raw(rcu_torture_current);
+	if (p) {
+		if (!READ_ONCE(p->rtort_mbtest))
+			atomic_inc(&n_rcu_torture_mberror);
+		if (READ_ONCE(p->rtort_pipe_count) > 1)
+			atomic_inc(&n_rcu_torture_error);
+	}
+	cur_ops->readunlock(idx);
+	atomic_long_inc(&n_rcu_torture_nmi_read);
+}
+
 static void rcu_torture_nmi_overflow(struct perf_event *event,
 				     struct perf_sample_data *data,
 				     struct pt_regs *regs)
@@ -2753,10 +2794,12 @@ static void rcu_torture_nmi_overflow(struct perf_event *event,
 
 	if (!in_nmi())
 		return;
-	if (cur_ops->call && !atomic_xchg(&rtncp->inuse, 1)) {
+	if (rcu_torture_nmi_do_calls && !atomic_xchg(&rtncp->inuse, 1)) {
 		atomic_long_inc(&n_rcu_torture_nmi_call);
 		cur_ops->call(&rtncp->rh, rcu_torture_nmi_invoked);
 	}
+	if (rcu_torture_nmi_do_reads)
+		rcu_torture_nmi_read();
 }
 
 static int rcu_torture_nmi_online(unsigned int cpu)
@@ -2789,7 +2832,10 @@ static void rcu_torture_nmi_init(void)
 {
 	int ret;
 
-	if (!nmi_calls || !cur_ops->nmi_capable || !cur_ops->call)
+	rcu_torture_nmi_do_calls = nmi_calls && cur_ops->nmi_capable && cur_ops->call;
+	rcu_torture_nmi_do_reads = nmi_readers && cur_ops->nmi_reader_capable &&
+				   cur_ops->nmi_reader_capable();
+	if (!rcu_torture_nmi_do_calls && !rcu_torture_nmi_do_reads)
 		return;
 	rcu_torture_nmi_events = kcalloc(nr_cpu_ids, sizeof(*rcu_torture_nmi_events),
 					 GFP_KERNEL);
@@ -2815,9 +2861,12 @@ static void rcu_torture_nmi_cleanup(void)
 	}
 	kfree(rcu_torture_nmi_events);
 	rcu_torture_nmi_events = NULL;
-	if (!atomic_long_read(&n_rcu_torture_nmi_call))
+	if (rcu_torture_nmi_do_calls && !atomic_long_read(&n_rcu_torture_nmi_call))
 		pr_alert("%s: nmi_calls set but no ->call() ever issued from NMI, so NMI ->call() went untested (no PMU, or NMIs unavailable here).\n",
 			 __func__);
+	if (rcu_torture_nmi_do_reads && !atomic_long_read(&n_rcu_torture_nmi_read))
+		pr_alert("%s: nmi_readers set but no reader ever ran in NMI, so NMI readers went untested (no PMU, or NMIs unavailable here).\n",
+			 __func__);
 }
 #else /* #ifdef CONFIG_PERF_EVENTS */
 static void rcu_torture_nmi_init(void) { }
@@ -3212,9 +3261,10 @@ rcu_torture_stats_print(void)
 		data_race(n_barrier_attempts),
 		data_race(n_rcu_torture_barrier_error));
 	pr_cont("read-exits: %ld ", data_race(n_read_exits)); // Statistic.
-	pr_cont("nmi-calls: %ld nmi-cbs: %ld ",
+	pr_cont("nmi-calls: %ld nmi-cbs: %ld nmi-reads: %ld ",
 		atomic_long_read(&n_rcu_torture_nmi_call),
-		atomic_long_read(&n_rcu_torture_nmi_cb));
+		atomic_long_read(&n_rcu_torture_nmi_cb),
+		atomic_long_read(&n_rcu_torture_nmi_read));
 	pr_cont("nocb-toggles: %ld:%ld ",
 		atomic_long_read(&n_nocb_offload), atomic_long_read(&n_nocb_deoffload));
 	pr_cont("gpwraps: %ld\n", n_gpwraps);
-- 
2.47.3

[PATCH 4/4] rcutorture: Test the irqsave forms of atomic SRCU readers

[Mathieu Desnoyers]

Nothing tests srcu_read_lock_atomic_irqsave() and
srcu_read_unlock_atomic_irqrestore(). Testing them in the normal readers
has two obstacles. rcutorture_one_extend() ends the readers of a CPU in
any order, but each irqrestore restores the state its own irqsave saw.
It also enables interrupts and bottom halves within readers, which an
irqsave reader forbids.

Add an atomic_irqsave module parameter. With it, half the atomic
readers use the irqsave forms, alternating per CPU. A per-CPU count
tracks these readers: the first one saves the interrupt state, and only
the last one restores it. The others restore the current state, with
interrupts disabled. The parameter also removes the bottom-half and
interrupt reader extensions, and the reader end under ->pi_lock, which
would enable interrupts under a raw spinlock.

An irqsave reader checks that interrupts are still disabled when it
ends.

Add an atomic-srcu-irqsave run of SRCU-P to torture.sh.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/rcu/rcutorture.c                       | 74 +++++++++++++++++--
 .../selftests/rcutorture/bin/torture.sh       |  6 ++
 2 files changed, 75 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 3b9c08a094cd..99dde988f974 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -134,6 +134,7 @@ torture_param(int, preempt_interval, MSEC_PER_SEC, "Interval between preemptions
 torture_param(int, read_exit_delay, 13, "Delay between read-then-exit episodes (s)");
 torture_param(int, read_exit_burst, 16, "# of read-then-exit bursts per episode, zero to disable");
 torture_param(int, reader_flavor, SRCU_READ_FLAVOR_NORMAL, "Reader flavors to use, one per bit.");
+torture_param(bool, atomic_irqsave, false, "Half the atomic SRCU readers disable irqs");
 torture_param(int, shuffle_interval, 3, "Number of seconds between shuffles");
 torture_param(int, shutdown_secs, 0, "Shutdown time (s), <= zero to disable.");
 torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable.");
@@ -728,6 +729,66 @@ static void srcu_torture_init_forbidden_apis(void)
 	cur_ops->start_gp_poll = NULL;
 }
 
+/*
+ * With atomic_irqsave, half the atomic SRCU readers disable interrupts instead
+ * of preemption.  rcutorture_one_extend() may end the readers of a CPU in any
+ * order, so the first one saves the interrupt state and the last one restores
+ * it.  An extension must then neither enable interrupts nor enable bottom
+ * halves within such a reader, and no reader may end under ->pi_lock.
+ */
+#define SRCU_TORTURE_IRQSAVE	0x20	// The reader disabled interrupts.
+
+struct srcu_torture_irqsave {
+	unsigned int count;	// Picks the form of the next reader.
+	int depth;		// Readers of this CPU that disabled interrupts.
+	unsigned long flags;	// Interrupt state before the first one.
+};
+
+static DEFINE_PER_CPU(struct srcu_torture_irqsave, srcu_torture_irqsave);
+
+static void srcu_torture_init_irqsave(void)
+{
+	if (!atomic_irqsave || !(reader_flavor & SRCU_READ_FLAVOR_ATOMIC))
+		return;
+	cur_ops->extendables &= ~(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH |
+				  RCUTORTURE_RDR_IRQ);
+	cur_ops->no_pi_lock = 1;
+	VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU, irqsave readers");
+}
+
+static int srcu_torture_read_lock_atomic(void)
+{
+	struct srcu_torture_irqsave *stip;
+	unsigned long flags;
+	int idx;
+
+	if (!atomic_irqsave || in_nmi() || !(this_cpu_inc_return(srcu_torture_irqsave.count) & 0x1))
+		return srcu_read_lock_atomic(srcu_ctlp);
+	idx = srcu_read_lock_atomic_irqsave(srcu_ctlp, flags);
+	stip = this_cpu_ptr(&srcu_torture_irqsave);
+	if (!stip->depth++)
+		stip->flags = flags;
+	return idx | SRCU_TORTURE_IRQSAVE;
+}
+
+static void srcu_torture_read_unlock_atomic(int idx)
+{
+	struct srcu_torture_irqsave *stip;
+	unsigned long flags;
+
+	if (!(idx & SRCU_TORTURE_IRQSAVE)) {
+		srcu_read_unlock_atomic(srcu_ctlp, idx);
+		return;
+	}
+	idx &= ~SRCU_TORTURE_IRQSAVE;
+	WARN_ON_ONCE(!irqs_disabled());  // Its section must be interrupts-off.
+	stip = this_cpu_ptr(&srcu_torture_irqsave);
+	local_save_flags(flags);  // Disabled: keep it so for the other readers.
+	if (!WARN_ON_ONCE(stip->depth <= 0) && !--stip->depth)
+		flags = stip->flags;
+	srcu_read_unlock_atomic_irqrestore(srcu_ctlp, idx, flags);
+}
+
 static void srcu_torture_init(void)
 {
 	rcu_sync_torture_init();
@@ -747,6 +808,7 @@ static void srcu_torture_init(void)
 		srcu_ctlp = &srcu_ctla;
 		VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU");
 		srcu_torture_init_forbidden_apis();
+		srcu_torture_init_irqsave();
 	}
 }
 
@@ -786,9 +848,9 @@ static int srcu_torture_read_lock(void)
 		ret += idx << 3;
 	}
 	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC) {
-		idx = srcu_read_lock_atomic(srcu_ctlp);
-		WARN_ON_ONCE(idx & ~0x1);
-		ret += idx << 4;
+		idx = srcu_torture_read_lock_atomic();
+		WARN_ON_ONCE(idx & ~(0x1 | SRCU_TORTURE_IRQSAVE));
+		ret += ((idx & 0x1) << 4) | (idx & SRCU_TORTURE_IRQSAVE);
 	}
 	return ret;
 }
@@ -828,9 +890,10 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)
 
 static void srcu_torture_read_unlock(int idx)
 {
-	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
+	WARN_ON_ONCE((reader_flavor && (idx & ~(reader_flavor | SRCU_TORTURE_IRQSAVE))) ||
+		     (!reader_flavor && (idx & ~0x1)));
 	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC)
-		srcu_read_unlock_atomic(srcu_ctlp, (idx & 0x10) >> 4);
+		srcu_torture_read_unlock_atomic(((idx & 0x10) >> 4) | (idx & SRCU_TORTURE_IRQSAVE));
 	if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN)
 		srcu_read_unlock_fast_updown(srcu_ctlp,
 					     __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
@@ -1015,6 +1078,7 @@ static void srcud_torture_init(void)
 		WARN_ON(init_srcu_struct_atomic(&srcu_ctld));
 		VERBOSE_TOROUT_STRING("srcud_torture_init atomic SRCU");
 		srcu_torture_init_forbidden_apis();
+		srcu_torture_init_irqsave();
 	} else {
 		WARN_ON(init_srcu_struct(&srcu_ctld));
 	}
diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
index e7b578ec152f..18a03fad1038 100755
--- a/tools/testing/selftests/rcutorture/bin/torture.sh
+++ b/tools/testing/selftests/rcutorture/bin/torture.sh
@@ -519,6 +519,12 @@ then
 	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
 		--allcpus --duration "$duration_rcutorture" \
 		--configs "$configs_atomic_srcu" --trust-make
+
+	# Half the readers then use srcu_read_lock_atomic_irqsave().
+	torture_bootargs="rcutorture.reader_flavor=0x10 rcutorture.atomic_irqsave=1"
+	torture_set "atomic-srcu-irqsave" tools/testing/selftests/rcutorture/bin/kvm.sh \
+		--allcpus --duration "$duration_rcutorture" \
+		--configs "SRCU-P" --trust-make
 fi
 
 if test "$do_locktorture" = "yes"
-- 
2.47.3

Re: [PATCH 4/4] rcutorture: Test the irqsave forms of atomic SRCU readers

[Paul E. McKenney]

On Tue, Nov 10, 2026 at 02:28:44PM -0500, Mathieu Desnoyers wrote:
> Nothing tests srcu_read_lock_atomic_irqsave() and
> srcu_read_unlock_atomic_irqrestore(). Testing them in the normal readers
> has two obstacles. rcutorture_one_extend() ends the readers of a CPU in
> any order, but each irqrestore restores the state its own irqsave saw.
> It also enables interrupts and bottom halves within readers, which an
> irqsave reader forbids.
> 
> Add an atomic_irqsave module parameter. With it, half the atomic
> readers use the irqsave forms, alternating per CPU. A per-CPU count
> tracks these readers: the first one saves the interrupt state, and only
> the last one restores it. The others restore the current state, with
> interrupts disabled. The parameter also removes the bottom-half and
> interrupt reader extensions, and the reader end under ->pi_lock, which
> would enable interrupts under a raw spinlock.
> 
> An irqsave reader checks that interrupts are still disabled when it
> ends.
> 
> Add an atomic-srcu-irqsave run of SRCU-P to torture.sh.
> 
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Cute trick with the per-CPU variable for the flags value!  ;-)

A few questions and comments below.

I am pulling this series in for testing and further review, but I do
understand that it might need to be replaced by a later version.

							Thanx, Paul

> ---
>  kernel/rcu/rcutorture.c                       | 74 +++++++++++++++++--
>  .../selftests/rcutorture/bin/torture.sh       |  6 ++
>  2 files changed, 75 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 3b9c08a094cd..99dde988f974 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -134,6 +134,7 @@ torture_param(int, preempt_interval, MSEC_PER_SEC, "Interval between preemptions
>  torture_param(int, read_exit_delay, 13, "Delay between read-then-exit episodes (s)");
>  torture_param(int, read_exit_burst, 16, "# of read-then-exit bursts per episode, zero to disable");
>  torture_param(int, reader_flavor, SRCU_READ_FLAVOR_NORMAL, "Reader flavors to use, one per bit.");
> +torture_param(bool, atomic_irqsave, false, "Half the atomic SRCU readers disable irqs");

What happens if there is only one reader?  Hey, you knew that this
question was coming!  ;-)

(And I see the use of ->count below.)

>  torture_param(int, shuffle_interval, 3, "Number of seconds between shuffles");
>  torture_param(int, shutdown_secs, 0, "Shutdown time (s), <= zero to disable.");
>  torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable.");
> @@ -728,6 +729,66 @@ static void srcu_torture_init_forbidden_apis(void)
>  	cur_ops->start_gp_poll = NULL;
>  }
>  
> +/*
> + * With atomic_irqsave, half the atomic SRCU readers disable interrupts instead
> + * of preemption.  rcutorture_one_extend() may end the readers of a CPU in any
> + * order, so the first one saves the interrupt state and the last one restores
> + * it.  An extension must then neither enable interrupts nor enable bottom
> + * halves within such a reader, and no reader may end under ->pi_lock.
> + */

Good point:  SRCU does not flatten readers by default, but you really
need to for the _irqsave() case.

> +#define SRCU_TORTURE_IRQSAVE	0x20	// The reader disabled interrupts.

Is this supposed to slot into the RCUTORTURE_RDR_* bit positions?
If so, it is overwriting RCUTORTURE_RDR_RCU_1.  Or is this a trick
that you are using to make half of the readers disable interrupts?
If so, let's please make this more explicit and more obvious to someone
(like yourself) who might be reading the RCUTORTURE_RDR_* bit position
definitions some time in the future.  ;-)

Except...  The rule is that RCUTORTURE_RDR_RCU_2 cannot be set unless
RCUTORTURE_RDR_RCU_1 is also set.  So there needs to be some other way
to get the "half the readers doing _irqsave()" effect.  (And I now see
the use of ->count below.)

If this is independent of the RCUTORTURE_RDR_* bit positions, why
the 0x20?

> +struct srcu_torture_irqsave {
> +	unsigned int count;	// Picks the form of the next reader.

Like maybe this count being odd or even, for example.  ;-)

And I do see below that this is in fact the strategy.  Very good!

> +	int depth;		// Readers of this CPU that disabled interrupts.
> +	unsigned long flags;	// Interrupt state before the first one.
> +};
> +
> +static DEFINE_PER_CPU(struct srcu_torture_irqsave, srcu_torture_irqsave);
> +
> +static void srcu_torture_init_irqsave(void)
> +{
> +	if (!atomic_irqsave || !(reader_flavor & SRCU_READ_FLAVOR_ATOMIC))
> +		return;
> +	cur_ops->extendables &= ~(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH |
> +				  RCUTORTURE_RDR_IRQ);

All the "srcu*" torture types implicitly set ->extendables to zero, so
you shouldn't need this.  The exception is the "busted_srcud" torture
type, which is supposed to fail anyway.  Or is "busted_srcud" the case
that this code is trying to handle?

If so, an alternative approach would be to only do the _irqsave()
readers when ->extendables is equal to zero.  Which is admittedly still
a bit hacky.

> +	cur_ops->no_pi_lock = 1;
> +	VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU, irqsave readers");
> +}
> +
> +static int srcu_torture_read_lock_atomic(void)
> +{
> +	struct srcu_torture_irqsave *stip;
> +	unsigned long flags;
> +	int idx;
> +
> +	if (!atomic_irqsave || in_nmi() || !(this_cpu_inc_return(srcu_torture_irqsave.count) & 0x1))

The suggestion above would add "|| cur_ops->extendables" to the above
condition.  Atomic SRCU readers are a bit heavyweight, so the extra
runtime overhead should not be a concern.

> +		return srcu_read_lock_atomic(srcu_ctlp);
> +	idx = srcu_read_lock_atomic_irqsave(srcu_ctlp, flags);
> +	stip = this_cpu_ptr(&srcu_torture_irqsave);
> +	if (!stip->depth++)
> +		stip->flags = flags;
> +	return idx | SRCU_TORTURE_IRQSAVE;

OK, so the value of 0x20 instead goes into RCUTORTURE_RDR_MASK_1 and
maybe also into RCUTORTURE_RDR_MASK_2, correct?

If so, it would be good to define SRCU_TORTURE_IRQSAVE somewhere near
the definitions of RCUTORTURE_RDR_MASK_1 and RCUTORTURE_RDR_MASK_2.

/me scratches his head trying to think why 0x20 instead of 0x2, 0x10,
or whatever...

Ah, because there are five reader flavors, which consume bits 0x1 through
0x10.  But we have been adding reader flavors at a surprisingly fast clip.
Might you therefore be willing to instead consider the value 0x80?

> +}
> +
> +static void srcu_torture_read_unlock_atomic(int idx)
> +{
> +	struct srcu_torture_irqsave *stip;
> +	unsigned long flags;
> +
> +	if (!(idx & SRCU_TORTURE_IRQSAVE)) {
> +		srcu_read_unlock_atomic(srcu_ctlp, idx);
> +		return;
> +	}
> +	idx &= ~SRCU_TORTURE_IRQSAVE;
> +	WARN_ON_ONCE(!irqs_disabled());  // Its section must be interrupts-off.
> +	stip = this_cpu_ptr(&srcu_torture_irqsave);
> +	local_save_flags(flags);  // Disabled: keep it so for the other readers.
> +	if (!WARN_ON_ONCE(stip->depth <= 0) && !--stip->depth)
> +		flags = stip->flags;
> +	srcu_read_unlock_atomic_irqrestore(srcu_ctlp, idx, flags);
> +}
> +
>  static void srcu_torture_init(void)
>  {
>  	rcu_sync_torture_init();
> @@ -747,6 +808,7 @@ static void srcu_torture_init(void)
>  		srcu_ctlp = &srcu_ctla;
>  		VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU");
>  		srcu_torture_init_forbidden_apis();
> +		srcu_torture_init_irqsave();
>  	}
>  }
>  
> @@ -786,9 +848,9 @@ static int srcu_torture_read_lock(void)
>  		ret += idx << 3;
>  	}
>  	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC) {
> -		idx = srcu_read_lock_atomic(srcu_ctlp);
> -		WARN_ON_ONCE(idx & ~0x1);
> -		ret += idx << 4;
> +		idx = srcu_torture_read_lock_atomic();
> +		WARN_ON_ONCE(idx & ~(0x1 | SRCU_TORTURE_IRQSAVE));
> +		ret += ((idx & 0x1) << 4) | (idx & SRCU_TORTURE_IRQSAVE);

This bit manipulation is getting a bit ornate, but that is a separate
issue and can be dealt with later.  And who knows?  This might be optimal.

>  	}
>  	return ret;
>  }
> @@ -828,9 +890,10 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)
>  
>  static void srcu_torture_read_unlock(int idx)
>  {
> -	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
> +	WARN_ON_ONCE((reader_flavor && (idx & ~(reader_flavor | SRCU_TORTURE_IRQSAVE))) ||
> +		     (!reader_flavor && (idx & ~0x1)));
>  	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC)
> -		srcu_read_unlock_atomic(srcu_ctlp, (idx & 0x10) >> 4);
> +		srcu_torture_read_unlock_atomic(((idx & 0x10) >> 4) | (idx & SRCU_TORTURE_IRQSAVE));
>  	if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN)
>  		srcu_read_unlock_fast_updown(srcu_ctlp,
>  					     __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
> @@ -1015,6 +1078,7 @@ static void srcud_torture_init(void)
>  		WARN_ON(init_srcu_struct_atomic(&srcu_ctld));
>  		VERBOSE_TOROUT_STRING("srcud_torture_init atomic SRCU");
>  		srcu_torture_init_forbidden_apis();
> +		srcu_torture_init_irqsave();
>  	} else {
>  		WARN_ON(init_srcu_struct(&srcu_ctld));
>  	}
> diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> index e7b578ec152f..18a03fad1038 100755
> --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> @@ -519,6 +519,12 @@ then
>  	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
>  		--allcpus --duration "$duration_rcutorture" \
>  		--configs "$configs_atomic_srcu" --trust-make
> +
> +	# Half the readers then use srcu_read_lock_atomic_irqsave().
> +	torture_bootargs="rcutorture.reader_flavor=0x10 rcutorture.atomic_irqsave=1"
> +	torture_set "atomic-srcu-irqsave" tools/testing/selftests/rcutorture/bin/kvm.sh \
> +		--allcpus --duration "$duration_rcutorture" \
> +		--configs "SRCU-P" --trust-make
>  fi
>  
>  if test "$do_locktorture" = "yes"
> -- 
> 2.47.3
> 

Re: [PATCH 3/4] rcutorture: Run NMI-safe SRCU readers from NMI

[Paul E. McKenney]

On Tue, Nov 10, 2026 at 02:28:43PM -0500, Mathieu Desnoyers wrote:
> rcutorture exercises ->call() from NMI, but never runs a reader there.
> So nothing tests the NMI-safe reader flavors where it matters most: in
> an NMI that interrupts another reader during its counter update.
> 
> Run a reader from the same per-CPU perf-counter NMI, when every reader
> flavor in use is NMI-safe. These flavors are srcu_read_lock_nmisafe(),
> srcu_read_lock_fast() and srcu_read_lock_atomic(). A new ops hook,
> ->nmi_reader_capable(), reports whether the reader flavors in use are
> all NMI-safe. A new nmi_readers module parameter, on by default, enables
> these NMI readers. The statistics count these readers as nmi-reads.
> 
> The reader loads rcu_torture_current and checks its element as the
> other readers do. The element must not be freed before the reader ends.
> It must not go through two grace periods either.
> 
> The NMI may interrupt lockdep, which then does not record the reader.
> So the reader loads the pointer with rcu_dereference_raw().
> 
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

A good addition!  Vanilla RCU could benefit from this as well, but that
can be done later.

A few questions below.

						Thanx, Paul

> ---
>  kernel/rcu/rcutorture.c | 60 +++++++++++++++++++++++++++++++++++++----
>  1 file changed, 55 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index 968343be4e0d..3b9c08a094cd 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -117,6 +117,7 @@ torture_param(int, n_barrier_cbs, 0, "# of callbacks/kthreads for barrier testin
>  torture_param(int, n_up_down, 32, "# of concurrent up/down hrtimer-based RCU readers");
>  torture_param(int, nfakewriters, 4, "Number of RCU fake writer threads");
>  torture_param(bool, nmi_calls, true, "Exercise ->call() from NMI on nmi_capable flavors");
> +torture_param(bool, nmi_readers, true, "Run readers from NMI with NMI-safe reader flavors");
>  torture_param(int, nreaders, -1, "Number of RCU reader threads");
>  torture_param(bool, nwriters, 1, "Number of RCU writer threads (0 or 1)");
>  torture_param(int, object_debug, 0, "Enable debug-object double call_rcu() testing");
> @@ -220,6 +221,7 @@ static atomic_long_t n_rcu_torture_timers;
>  static atomic_long_t n_rcu_torture_irqs;
>  static atomic_long_t n_rcu_torture_nmi_call;
>  static atomic_long_t n_rcu_torture_nmi_cb;
> +static atomic_long_t n_rcu_torture_nmi_read;
>  static long n_barrier_attempts;
>  static long n_barrier_successes; /* did rcu_barrier test succeed? */
>  static unsigned long n_read_exits;
> @@ -438,6 +440,7 @@ struct rcu_torture_ops {
>  	long cbflood_max;
>  	int irq_capable;
>  	int nmi_capable;
> +	bool (*nmi_reader_capable)(void); // Are ->readlock() and ->readunlock() NMI-safe?
>  	int can_boost;
>  	int extendables;
>  	int slow_gps;
> @@ -790,6 +793,14 @@ static int srcu_torture_read_lock(void)
>  	return ret;
>  }
>  
> +// Only these reader flavors may be used from NMI.
> +static bool srcu_torture_nmi_reader_capable(void)
> +{
> +	return reader_flavor &&
> +	       !(reader_flavor & ~(SRCU_READ_FLAVOR_NMI | SRCU_READ_FLAVOR_FAST |
> +				   SRCU_READ_FLAVOR_ATOMIC));
> +}
> +
>  static void
>  srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)
>  {
> @@ -981,6 +992,7 @@ static struct rcu_torture_ops srcu_ops = {
>  	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
>  				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN,
>  	.nmi_capable	= 1,
> +	.nmi_reader_capable = srcu_torture_nmi_reader_capable,
>  	.name		= "srcu"
>  };
>  
> @@ -1049,6 +1061,7 @@ static struct rcu_torture_ops srcud_ops = {
>  	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
>  				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN,
>  	.nmi_capable	= 1,
> +	.nmi_reader_capable = srcu_torture_nmi_reader_capable,
>  	.name		= "srcud"
>  };
>  
> @@ -2711,6 +2724,10 @@ static DEFINE_TORTURE_RANDOM_PERCPU(rcu_torture_timer_rand);
>   * per-CPU hardware perf counter overflows into an NMI, and its handler submits
>   * a preallocated callback via ->call().  One callback per CPU is in flight at a
>   * time (guarded by an atomic) to avoid allocating in NMI.
> + *
> + * The same handler also runs a reader, for flavors whose ->nmi_reader_capable()
> + * says that their readers are NMI-safe.  The reader may interrupt another
> + * reader on its CPU, in the middle of its counter update.
>   */
>  #ifdef CONFIG_PERF_EVENTS
>  static struct perf_event_attr rcu_torture_nmi_attr = {
> @@ -2735,6 +2752,8 @@ struct rcu_torture_nmi_cb {
>  
>  static struct perf_event **rcu_torture_nmi_events;
>  static int rcu_torture_nmi_hp_state;
> +static bool rcu_torture_nmi_do_calls;
> +static bool rcu_torture_nmi_do_reads;
>  static DEFINE_PER_CPU(struct rcu_torture_nmi_cb, rcu_torture_nmi_cb);
>  
>  static void rcu_torture_nmi_invoked(struct rcu_head *rhp)
> @@ -2745,6 +2764,28 @@ static void rcu_torture_nmi_invoked(struct rcu_head *rhp)
>  	atomic_set(&rtncp->inuse, 0);
>  }
>  
> +/*
> + * A reader in NMI.  The NMI may interrupt lockdep, which then does not record
> + * the reader, so the pointer is loaded without a lockdep check.  The element
> + * must not go through two grace periods before the reader ends.
> + */
> +static void rcu_torture_nmi_read(void)
> +{
> +	struct rcu_torture *p;
> +	int idx;
> +
> +	idx = cur_ops->readlock();
> +	p = rcu_dereference_raw(rcu_torture_current);
> +	if (p) {
> +		if (!READ_ONCE(p->rtort_mbtest))
> +			atomic_inc(&n_rcu_torture_mberror);
> +		if (READ_ONCE(p->rtort_pipe_count) > 1)
> +			atomic_inc(&n_rcu_torture_error);

Doesn't this miss doing a splat on RCU too-short grace period?

One way to handle this is to have a separate n_rcu_torture_nmi_error
counter and then have rcu_torture_stats_print() sample this counter before
the "for (i = RCU_TORTURE_PIPE_LEN; i >= 0; i--)" loop and apply it there.

But that loses the visible count in the "Reader Pipe:" line.  Which
would be *really* confusing.  And we don't need atomic operations
slowing up non-NMI RCU readers, either.

Maybe a separate global array for NMIs, parallel to the "Reader Pipe:"
and "Reader Batch:" arrays, that is atomically updated and printed
separately, maybe "NMI Reader Pipe:" or similar?

Or is there a better way?

> +	}
> +	cur_ops->readunlock(idx);
> +	atomic_long_inc(&n_rcu_torture_nmi_read);
> +}
> +
>  static void rcu_torture_nmi_overflow(struct perf_event *event,
>  				     struct perf_sample_data *data,
>  				     struct pt_regs *regs)
> @@ -2753,10 +2794,12 @@ static void rcu_torture_nmi_overflow(struct perf_event *event,
>  
>  	if (!in_nmi())
>  		return;
> -	if (cur_ops->call && !atomic_xchg(&rtncp->inuse, 1)) {
> +	if (rcu_torture_nmi_do_calls && !atomic_xchg(&rtncp->inuse, 1)) {
>  		atomic_long_inc(&n_rcu_torture_nmi_call);
>  		cur_ops->call(&rtncp->rh, rcu_torture_nmi_invoked);

OK, the invocation of ->call() without testing ->call() is OK because
rcu_torture_nmi_do_calls never gets set if ->call() is NULL.

>  	}
> +	if (rcu_torture_nmi_do_reads)
> +		rcu_torture_nmi_read();
>  }
>  
>  static int rcu_torture_nmi_online(unsigned int cpu)
> @@ -2789,7 +2832,10 @@ static void rcu_torture_nmi_init(void)
>  {
>  	int ret;
>  
> -	if (!nmi_calls || !cur_ops->nmi_capable || !cur_ops->call)
> +	rcu_torture_nmi_do_calls = nmi_calls && cur_ops->nmi_capable && cur_ops->call;
> +	rcu_torture_nmi_do_reads = nmi_readers && cur_ops->nmi_reader_capable &&
> +				   cur_ops->nmi_reader_capable();
> +	if (!rcu_torture_nmi_do_calls && !rcu_torture_nmi_do_reads)
>  		return;
>  	rcu_torture_nmi_events = kcalloc(nr_cpu_ids, sizeof(*rcu_torture_nmi_events),
>  					 GFP_KERNEL);
> @@ -2815,9 +2861,12 @@ static void rcu_torture_nmi_cleanup(void)
>  	}
>  	kfree(rcu_torture_nmi_events);
>  	rcu_torture_nmi_events = NULL;
> -	if (!atomic_long_read(&n_rcu_torture_nmi_call))
> +	if (rcu_torture_nmi_do_calls && !atomic_long_read(&n_rcu_torture_nmi_call))
>  		pr_alert("%s: nmi_calls set but no ->call() ever issued from NMI, so NMI ->call() went untested (no PMU, or NMIs unavailable here).\n",
>  			 __func__);
> +	if (rcu_torture_nmi_do_reads && !atomic_long_read(&n_rcu_torture_nmi_read))
> +		pr_alert("%s: nmi_readers set but no reader ever ran in NMI, so NMI readers went untested (no PMU, or NMIs unavailable here).\n",
> +			 __func__);
>  }
>  #else /* #ifdef CONFIG_PERF_EVENTS */
>  static void rcu_torture_nmi_init(void) { }
> @@ -3212,9 +3261,10 @@ rcu_torture_stats_print(void)
>  		data_race(n_barrier_attempts),
>  		data_race(n_rcu_torture_barrier_error));
>  	pr_cont("read-exits: %ld ", data_race(n_read_exits)); // Statistic.
> -	pr_cont("nmi-calls: %ld nmi-cbs: %ld ",
> +	pr_cont("nmi-calls: %ld nmi-cbs: %ld nmi-reads: %ld ",
>  		atomic_long_read(&n_rcu_torture_nmi_call),
> -		atomic_long_read(&n_rcu_torture_nmi_cb));
> +		atomic_long_read(&n_rcu_torture_nmi_cb),
> +		atomic_long_read(&n_rcu_torture_nmi_read));
>  	pr_cont("nocb-toggles: %ld:%ld ",
>  		atomic_long_read(&n_nocb_offload), atomic_long_read(&n_nocb_deoffload));
>  	pr_cont("gpwraps: %ld\n", n_gpwraps);
> -- 
> 2.47.3
> 

Re: [PATCH 4/4] rcutorture: Test the irqsave forms of atomic SRCU readers

[Paul E. McKenney]

On Tue, Nov 10, 2026 at 02:33:20PM -0800, Paul E. McKenney wrote:
> On Tue, Nov 10, 2026 at 02:28:44PM -0500, Mathieu Desnoyers wrote:
> > Nothing tests srcu_read_lock_atomic_irqsave() and
> > srcu_read_unlock_atomic_irqrestore(). Testing them in the normal readers
> > has two obstacles. rcutorture_one_extend() ends the readers of a CPU in
> > any order, but each irqrestore restores the state its own irqsave saw.
> > It also enables interrupts and bottom halves within readers, which an
> > irqsave reader forbids.
> > 
> > Add an atomic_irqsave module parameter. With it, half the atomic
> > readers use the irqsave forms, alternating per CPU. A per-CPU count
> > tracks these readers: the first one saves the interrupt state, and only
> > the last one restores it. The others restore the current state, with
> > interrupts disabled. The parameter also removes the bottom-half and
> > interrupt reader extensions, and the reader end under ->pi_lock, which
> > would enable interrupts under a raw spinlock.
> > 
> > An irqsave reader checks that interrupts are still disabled when it
> > ends.
> > 
> > Add an atomic-srcu-irqsave run of SRCU-P to torture.sh.
> > 
> > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> 
> Cute trick with the per-CPU variable for the flags value!  ;-)
> 
> A few questions and comments below.
> 
> I am pulling this series in for testing and further review, but I do
> understand that it might need to be replaced by a later version.

And it passes initial "smoke" tests.  ;-)

							Thanx, Paul

> > ---
> >  kernel/rcu/rcutorture.c                       | 74 +++++++++++++++++--
> >  .../selftests/rcutorture/bin/torture.sh       |  6 ++
> >  2 files changed, 75 insertions(+), 5 deletions(-)
> > 
> > diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> > index 3b9c08a094cd..99dde988f974 100644
> > --- a/kernel/rcu/rcutorture.c
> > +++ b/kernel/rcu/rcutorture.c
> > @@ -134,6 +134,7 @@ torture_param(int, preempt_interval, MSEC_PER_SEC, "Interval between preemptions
> >  torture_param(int, read_exit_delay, 13, "Delay between read-then-exit episodes (s)");
> >  torture_param(int, read_exit_burst, 16, "# of read-then-exit bursts per episode, zero to disable");
> >  torture_param(int, reader_flavor, SRCU_READ_FLAVOR_NORMAL, "Reader flavors to use, one per bit.");
> > +torture_param(bool, atomic_irqsave, false, "Half the atomic SRCU readers disable irqs");
> 
> What happens if there is only one reader?  Hey, you knew that this
> question was coming!  ;-)
> 
> (And I see the use of ->count below.)
> 
> >  torture_param(int, shuffle_interval, 3, "Number of seconds between shuffles");
> >  torture_param(int, shutdown_secs, 0, "Shutdown time (s), <= zero to disable.");
> >  torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable.");
> > @@ -728,6 +729,66 @@ static void srcu_torture_init_forbidden_apis(void)
> >  	cur_ops->start_gp_poll = NULL;
> >  }
> >  
> > +/*
> > + * With atomic_irqsave, half the atomic SRCU readers disable interrupts instead
> > + * of preemption.  rcutorture_one_extend() may end the readers of a CPU in any
> > + * order, so the first one saves the interrupt state and the last one restores
> > + * it.  An extension must then neither enable interrupts nor enable bottom
> > + * halves within such a reader, and no reader may end under ->pi_lock.
> > + */
> 
> Good point:  SRCU does not flatten readers by default, but you really
> need to for the _irqsave() case.
> 
> > +#define SRCU_TORTURE_IRQSAVE	0x20	// The reader disabled interrupts.
> 
> Is this supposed to slot into the RCUTORTURE_RDR_* bit positions?
> If so, it is overwriting RCUTORTURE_RDR_RCU_1.  Or is this a trick
> that you are using to make half of the readers disable interrupts?
> If so, let's please make this more explicit and more obvious to someone
> (like yourself) who might be reading the RCUTORTURE_RDR_* bit position
> definitions some time in the future.  ;-)
> 
> Except...  The rule is that RCUTORTURE_RDR_RCU_2 cannot be set unless
> RCUTORTURE_RDR_RCU_1 is also set.  So there needs to be some other way
> to get the "half the readers doing _irqsave()" effect.  (And I now see
> the use of ->count below.)
> 
> If this is independent of the RCUTORTURE_RDR_* bit positions, why
> the 0x20?
> 
> > +struct srcu_torture_irqsave {
> > +	unsigned int count;	// Picks the form of the next reader.
> 
> Like maybe this count being odd or even, for example.  ;-)
> 
> And I do see below that this is in fact the strategy.  Very good!
> 
> > +	int depth;		// Readers of this CPU that disabled interrupts.
> > +	unsigned long flags;	// Interrupt state before the first one.
> > +};
> > +
> > +static DEFINE_PER_CPU(struct srcu_torture_irqsave, srcu_torture_irqsave);
> > +
> > +static void srcu_torture_init_irqsave(void)
> > +{
> > +	if (!atomic_irqsave || !(reader_flavor & SRCU_READ_FLAVOR_ATOMIC))
> > +		return;
> > +	cur_ops->extendables &= ~(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH |
> > +				  RCUTORTURE_RDR_IRQ);
> 
> All the "srcu*" torture types implicitly set ->extendables to zero, so
> you shouldn't need this.  The exception is the "busted_srcud" torture
> type, which is supposed to fail anyway.  Or is "busted_srcud" the case
> that this code is trying to handle?
> 
> If so, an alternative approach would be to only do the _irqsave()
> readers when ->extendables is equal to zero.  Which is admittedly still
> a bit hacky.
> 
> > +	cur_ops->no_pi_lock = 1;
> > +	VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU, irqsave readers");
> > +}
> > +
> > +static int srcu_torture_read_lock_atomic(void)
> > +{
> > +	struct srcu_torture_irqsave *stip;
> > +	unsigned long flags;
> > +	int idx;
> > +
> > +	if (!atomic_irqsave || in_nmi() || !(this_cpu_inc_return(srcu_torture_irqsave.count) & 0x1))
> 
> The suggestion above would add "|| cur_ops->extendables" to the above
> condition.  Atomic SRCU readers are a bit heavyweight, so the extra
> runtime overhead should not be a concern.
> 
> > +		return srcu_read_lock_atomic(srcu_ctlp);
> > +	idx = srcu_read_lock_atomic_irqsave(srcu_ctlp, flags);
> > +	stip = this_cpu_ptr(&srcu_torture_irqsave);
> > +	if (!stip->depth++)
> > +		stip->flags = flags;
> > +	return idx | SRCU_TORTURE_IRQSAVE;
> 
> OK, so the value of 0x20 instead goes into RCUTORTURE_RDR_MASK_1 and
> maybe also into RCUTORTURE_RDR_MASK_2, correct?
> 
> If so, it would be good to define SRCU_TORTURE_IRQSAVE somewhere near
> the definitions of RCUTORTURE_RDR_MASK_1 and RCUTORTURE_RDR_MASK_2.
> 
> /me scratches his head trying to think why 0x20 instead of 0x2, 0x10,
> or whatever...
> 
> Ah, because there are five reader flavors, which consume bits 0x1 through
> 0x10.  But we have been adding reader flavors at a surprisingly fast clip.
> Might you therefore be willing to instead consider the value 0x80?
> 
> > +}
> > +
> > +static void srcu_torture_read_unlock_atomic(int idx)
> > +{
> > +	struct srcu_torture_irqsave *stip;
> > +	unsigned long flags;
> > +
> > +	if (!(idx & SRCU_TORTURE_IRQSAVE)) {
> > +		srcu_read_unlock_atomic(srcu_ctlp, idx);
> > +		return;
> > +	}
> > +	idx &= ~SRCU_TORTURE_IRQSAVE;
> > +	WARN_ON_ONCE(!irqs_disabled());  // Its section must be interrupts-off.
> > +	stip = this_cpu_ptr(&srcu_torture_irqsave);
> > +	local_save_flags(flags);  // Disabled: keep it so for the other readers.
> > +	if (!WARN_ON_ONCE(stip->depth <= 0) && !--stip->depth)
> > +		flags = stip->flags;
> > +	srcu_read_unlock_atomic_irqrestore(srcu_ctlp, idx, flags);
> > +}
> > +
> >  static void srcu_torture_init(void)
> >  {
> >  	rcu_sync_torture_init();
> > @@ -747,6 +808,7 @@ static void srcu_torture_init(void)
> >  		srcu_ctlp = &srcu_ctla;
> >  		VERBOSE_TOROUT_STRING("srcu_torture_init atomic SRCU");
> >  		srcu_torture_init_forbidden_apis();
> > +		srcu_torture_init_irqsave();
> >  	}
> >  }
> >  
> > @@ -786,9 +848,9 @@ static int srcu_torture_read_lock(void)
> >  		ret += idx << 3;
> >  	}
> >  	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC) {
> > -		idx = srcu_read_lock_atomic(srcu_ctlp);
> > -		WARN_ON_ONCE(idx & ~0x1);
> > -		ret += idx << 4;
> > +		idx = srcu_torture_read_lock_atomic();
> > +		WARN_ON_ONCE(idx & ~(0x1 | SRCU_TORTURE_IRQSAVE));
> > +		ret += ((idx & 0x1) << 4) | (idx & SRCU_TORTURE_IRQSAVE);
> 
> This bit manipulation is getting a bit ornate, but that is a separate
> issue and can be dealt with later.  And who knows?  This might be optimal.
> 
> >  	}
> >  	return ret;
> >  }
> > @@ -828,9 +890,10 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)
> >  
> >  static void srcu_torture_read_unlock(int idx)
> >  {
> > -	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
> > +	WARN_ON_ONCE((reader_flavor && (idx & ~(reader_flavor | SRCU_TORTURE_IRQSAVE))) ||
> > +		     (!reader_flavor && (idx & ~0x1)));
> >  	if (reader_flavor & SRCU_READ_FLAVOR_ATOMIC)
> > -		srcu_read_unlock_atomic(srcu_ctlp, (idx & 0x10) >> 4);
> > +		srcu_torture_read_unlock_atomic(((idx & 0x10) >> 4) | (idx & SRCU_TORTURE_IRQSAVE));
> >  	if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN)
> >  		srcu_read_unlock_fast_updown(srcu_ctlp,
> >  					     __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
> > @@ -1015,6 +1078,7 @@ static void srcud_torture_init(void)
> >  		WARN_ON(init_srcu_struct_atomic(&srcu_ctld));
> >  		VERBOSE_TOROUT_STRING("srcud_torture_init atomic SRCU");
> >  		srcu_torture_init_forbidden_apis();
> > +		srcu_torture_init_irqsave();
> >  	} else {
> >  		WARN_ON(init_srcu_struct(&srcu_ctld));
> >  	}
> > diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> > index e7b578ec152f..18a03fad1038 100755
> > --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> > +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> > @@ -519,6 +519,12 @@ then
> >  	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> >  		--allcpus --duration "$duration_rcutorture" \
> >  		--configs "$configs_atomic_srcu" --trust-make
> > +
> > +	# Half the readers then use srcu_read_lock_atomic_irqsave().
> > +	torture_bootargs="rcutorture.reader_flavor=0x10 rcutorture.atomic_irqsave=1"
> > +	torture_set "atomic-srcu-irqsave" tools/testing/selftests/rcutorture/bin/kvm.sh \
> > +		--allcpus --duration "$duration_rcutorture" \
> > +		--configs "SRCU-P" --trust-make
> >  fi
> >  
> >  if test "$do_locktorture" = "yes"
> > -- 
> > 2.47.3
> > 
> 

Re: [PATCH 4/4] rcutorture: Test the irqsave forms of atomic SRCU readers

[Mathieu Desnoyers]

On 2026-11-10 17:33, Paul E. McKenney wrote:
> On Tue, Nov 10, 2026 at 02:28:44PM -0500, Mathieu Desnoyers wrote:
[...]
> Cute trick with the per-CPU variable for the flags value!  ;-)
>
> A few questions and comments below.
>
> I am pulling this series in for testing and further review, but I do
> understand that it might need to be replaced by a later version.

Thanks!  A v2 will follow, rebased on v7.4-rc2: patch 3 no longer
applies there, because of the kzalloc_objs() conversion in rcutorture.

[...]
>> +torture_param(bool, atomic_irqsave, false, "Half the atomic SRCU readers disable irqs");
>
> What happens if there is only one reader?  Hey, you knew that this
> question was coming!  ;-)

The choice alternates per CPU, from one reader to the next.  So even a
single reader thread alternates between the two forms.  The description
says "half", which suggests a split between threads.  v2 says
"Atomic SRCU readers alternate per CPU with irqsave readers".

[...]
>> +#define SRCU_TORTURE_IRQSAVE	0x20	// The reader disabled interrupts.
>
> Is this supposed to slot into the RCUTORTURE_RDR_* bit positions?
[...]
> If this is independent of the RCUTORTURE_RDR_* bit positions, why
> the 0x20?

It is independent of them.  It is a bit of the index that
srcu_torture_read_lock() returns, so it lands in RCUTORTURE_RDR_MASK_1
or _2.  0x20 was the first bit above the five reader flavors.

[...]
> If so, it would be good to define SRCU_TORTURE_IRQSAVE somewhere near
> the definitions of RCUTORTURE_RDR_MASK_1 and RCUTORTURE_RDR_MASK_2.
[...]
> Might you therefore be willing to instead consider the value 0x80?

Yes to both.  v2 defines it as 0x80, next to the masks, with a comment
that it is a bit of the SRCU reader index.

[...]
>> +	cur_ops->extendables &= ~(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH |
>> +				  RCUTORTURE_RDR_IRQ);
>
> All the "srcu*" torture types implicitly set ->extendables to zero, so
> you shouldn't need this.  The exception is the "busted_srcud" torture
> type, which is supposed to fail anyway.  Or is "busted_srcud" the case
> that this code is trying to handle?
>
> If so, an alternative approach would be to only do the _irqsave()
> readers when ->extendables is equal to zero.  Which is admittedly still
> a bit hacky.

It was only defensive.  I had not checked that every srcu type but
busted_srcud has no extendables.  v2 takes your approach: the irqsave
readers run only when ->extendables is zero.  It checks this once, at
init, and keeps the result in a bool that the reader tests.

[...]
>> +		ret += ((idx & 0x1) << 4) | (idx & SRCU_TORTURE_IRQSAVE);
>
> This bit manipulation is getting a bit ornate, but that is a separate
> issue and can be dealt with later.  And who knows?  This might be optimal.

Agreed, I left it as is.

> And it passes initial "smoke" tests.  ;-)

Good to hear!  Two more points for v2, both about synchronize_srcu_atomic()
rather than this patch.  I will raise them in its cover letter.

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH 3/4] rcutorture: Run NMI-safe SRCU readers from NMI

[Mathieu Desnoyers]

On 2026-11-10 17:46, Paul E. McKenney wrote:
> On Tue, Nov 10, 2026 at 02:28:43PM -0500, Mathieu Desnoyers wrote:
[...]
> A good addition!  Vanilla RCU could benefit from this as well, but that
> can be done later.

Thanks!

[...]
>> +            if (READ_ONCE(p->rtort_pipe_count) > 1)
>> +                    atomic_inc(&n_rcu_torture_error);
>
> Doesn't this miss doing a splat on RCU too-short grace period?
[...]
> Maybe a separate global array for NMIs, parallel to the "Reader Pipe:"
> and "Reader Batch:" arrays, that is atomically updated and printed
> separately, maybe "NMI Reader Pipe:" or similar?
>
> Or is there a better way?

It does miss it.  The test still ends in FAILURE.  But there is no
"!!!", no warning, and nothing in "Reader Pipe:".

v2 takes your approach.  The NMI reader counts its pipe and batch values
in two arrays of atomic_long_t.  rcu_torture_stats_print() prints them
as "NMI Reader Pipe:" and "NMI Reader Batch:", only when readers ran in
NMI.  A too-short grace period in NMI joins the "!!!" condition, with
its own WARN_ON_ONCE().  So the other readers keep their this_cpu_inc().

The NMI reader skips the trace event and rcu_ftrace_dump() of the other
readers.  The "!!!" and the warning then come from the statistics
kthread.

I checked it with a mutation that makes every NMI reader see a pipe
count of 2: the run splats as expected.

[...]
>>              cur_ops->call(&rtncp->rh, rcu_torture_nmi_invoked);
>
> OK, the invocation of ->call() without testing ->call() is OK because
> rcu_torture_nmi_do_calls never gets set if ->call() is NULL.

Yes.

Thanks!

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com