From: David Woodhouse <dwmw2@infradead.org>
To: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>,
Josh Triplett <josh@joshtriplett.org>,
Steven Rostedt <rostedt@goodmis.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Boqun Feng <boqun.feng@gmail.com>,
Neeraj Upadhyay <neeraj.upadhyay@kernel.org>,
Joel Fernandes <joel@joelfernandes.org>,
Uladzislau Rezki <urezki@gmail.com>,
Sean Christopherson <seanjc@kernel.org>,
Paolo Bonzini <pbonzini@redhat.com>,
rcu@vger.kernel.org, kvm@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH] srcu: Add try_synchronize_srcu() for callers which can prove readers absent
Date: Wed, 12 Aug 2026 13:32:48 +0100 [thread overview]
Message-ID: <0f4dea21bac43685d3286df329401177e4452b36.camel@infradead.org> (raw)
[-- Attachment #1: Type: text/plain, Size: 8693 bytes --]
From: David Woodhouse <dwmw@amazon.co.uk>
For a dedicated srcu_struct whose read-side critical sections are
short, atomic, and usually absent — such as one converted from a
spinning lock — the common case at synchronize time is that there are
no readers at all. Even synchronize_srcu_expedited() still costs the
caller an unconditional sleep and two trips through the SRCU workqueue
to discover that: tens of microseconds of machinery to wait for
nothing. And synchronize_srcu()'s auto-expedite heuristic explicitly
declines to expedite within exp_holdoff (25µs) of the previous grace
period, which is exactly the regime a burst of back-to-back
invalidations puts such a domain in.
Provide try_synchronize_srcu(), which proves the no-readers case
inline and returns true without sleeping, without the workqueue, and
without advancing the grace-period sequence (so cookies from
get_state_synchronize_srcu() and queued callbacks are unaffected). If
the proof fails, it returns false and the caller falls back:
if (!try_synchronize_srcu(ssp))
synchronize_srcu_expedited(ssp);
For Tree SRCU the proof sums both epochs' unlock counters, executes a
full barrier, then sums both epochs' lock counters. Equality proves a
moment within this function at which no reader existed: a reader
entering between the sums inflates only the lock sum (spurious
fallback, safe), and a reader whose increment is unobserved has not
yet returned from srcu_read_lock() — the barrier pairing with
__srcu_read_lock() guarantees such a reader sees every store the
caller made beforehand, so it is not a reader the caller is obliged
to wait for. Summing both epochs means no index flip is required, and
without a flip the counter-wrap concerns of
srcu_readers_active_idx_check() do not arise. Readers of the _fast()
flavors elide the read-side barrier this depends on; any sign of them
(in the unlock-side rdm mask, which is gathered unconditionally)
disqualifies the fast path.
For Tiny SRCU (!SMP) both nesting counts being zero already proves no
reader exists — a mid-section reader could only be preempted or in an
interrupt, either of which leaves its count visibly elevated — and
program order on the sole CPU provides all the required ordering.
The immediate motivation is a proposed conversion of KVM's
gfn_to_pfn_cache to use SRCU¹, where the mmu_notifier invalidation
path must drain readers of a cache (in the manner of a TLB shootdown)
before the primary MMU zaps the backing page. Those readers are short
non-sleeping fast paths, some in contexts which cannot sleep (hardirq
event channel delivery, the scheduler's sched-out hook); measurement
under a worst-case invalidation flood shows 98.8% of drains complete inline
in 4-16µs where the expedited grace period took 32-128µs, with the
wait dominated by workqueue round-trip latency, not by readers.
¹ https://lore.kernel.org/all/20260811132237.102400-1-dwmw2@infradead.org/
More potential use cases already exist in the tree with the same
no-readers-common-case profile: kvm->irq_srcu takes half a dozen
expedited grace periods in the irqfd and routing-update paths (bursts
of which, at VM boot, fall inside the exp_holdoff window), and mshv's
pt_irq_srcu is the same shape.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
---
Compile-tested for Tiny SRCU (tinyconfig); the Tree version has had
the KVM gfn_to_pfn_cache conversion soaking on it under an adversarial
invalidation flood.
include/linux/srcu.h | 1 +
kernel/rcu/srcutiny.c | 23 ++++++++++++++
kernel/rcu/srcutree.c | 70 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 94 insertions(+)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index a54ce9e808b9..0d7543c7becc 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -91,6 +91,7 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
void (*func)(struct rcu_head *head));
void cleanup_srcu_struct(struct srcu_struct *ssp);
void synchronize_srcu(struct srcu_struct *ssp);
+bool try_synchronize_srcu(struct srcu_struct *ssp);
#define SRCU_GET_STATE_COMPLETED 0x1
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
index a2e2d516e51b..f07159cae241 100644
--- a/kernel/rcu/srcutiny.c
+++ b/kernel/rcu/srcutiny.c
@@ -260,6 +260,29 @@ void synchronize_srcu(struct srcu_struct *ssp)
}
EXPORT_SYMBOL_GPL(synchronize_srcu);
+/**
+ * try_synchronize_srcu - inline grace period for a reader-free srcu_struct
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * If @ssp provably has no readers in either epoch, provide the
+ * synchronize_srcu() guarantee to the caller immediately, without
+ * sleeping. Returns true on success; on failure the caller must fall
+ * back to synchronize_srcu().
+ *
+ * On !SMP a reader can only be mid-critical-section if it was
+ * preempted (or is running in an interrupt which preempted us), in
+ * which case its nesting count is visibly non-zero. Both counts being
+ * zero therefore proves that no reader exists, and any reader which
+ * begins after this function returns will, by program order on this
+ * sole CPU, observe every store the caller made before calling it.
+ */
+bool try_synchronize_srcu(struct srcu_struct *ssp)
+{
+ return !READ_ONCE(ssp->srcu_lock_nesting[0]) &&
+ !READ_ONCE(ssp->srcu_lock_nesting[1]);
+}
+EXPORT_SYMBOL_GPL(try_synchronize_srcu);
+
/*
* get_state_synchronize_srcu - Provide an end-of-grace-period cookie
*/
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 2601566c254a..76a78336feaa 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1645,6 +1645,76 @@ void synchronize_srcu(struct srcu_struct *ssp)
}
EXPORT_SYMBOL_GPL(synchronize_srcu);
+/**
+ * try_synchronize_srcu - inline grace period for a reader-free srcu_struct
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * If @ssp provably has no readers in either epoch, provide the
+ * synchronize_srcu() guarantee to the caller immediately: without
+ * sleeping, without a trip through the SRCU workqueue, and without
+ * advancing the grace-period sequence. Returns true on success; on
+ * failure the caller must fall back to synchronize_srcu() or
+ * synchronize_srcu_expedited().
+ *
+ * This serves dedicated srcu_struct structures whose read-side critical
+ * sections are short, atomic, and usually absent — where even an
+ * expedited grace period costs two trips through the workqueue and an
+ * unconditional sleep of the caller, three orders of magnitude more
+ * than the check below.
+ *
+ * Only readers of the srcu_read_lock() and srcu_read_lock_nmisafe()
+ * flavors are compatible with this proof; if the _fast() flavors have
+ * ever been used on @ssp, this function always returns false.
+ */
+bool try_synchronize_srcu(struct srcu_struct *ssp)
+{
+ unsigned long unlocks0, unlocks1;
+ unsigned long rdm0, rdm1;
+
+ check_init_srcu_struct(ssp);
+
+ /*
+ * Order the caller's prior stores before the counter reads below.
+ * Pairs (store-buffering pattern) with the smp_mb() in
+ * __srcu_read_lock(): any reader whose lock increment is not
+ * observed by the sums below is guaranteed to observe, within its
+ * critical section, every store the caller made before calling
+ * this function.
+ */
+ smp_mb();
+
+ unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
+ unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
+
+ /*
+ * Reader flavors which elide the read-side smp_mb() that the
+ * pairing above depends on cannot be proven absent this way;
+ * they need a real grace period.
+ */
+ if ((rdm0 | rdm1) & SRCU_READ_FLAVOR_SLOWGP)
+ return false;
+
+ /*
+ * As in srcu_readers_active_idx_check(): ensure that a lock is
+ * always counted if the corresponding unlock is counted, so that
+ * a reader racing with these sums can only inflate the lock sum
+ * and force the (safe) fallback. Summing both epochs means no
+ * index flip is needed: a stable equality proves there was a
+ * moment in this function at which no readers existed at all.
+ */
+ smp_mb();
+
+ if (!srcu_readers_lock_idx(ssp, 0, false, unlocks0))
+ return false;
+ if (!srcu_readers_lock_idx(ssp, 1, false, unlocks1))
+ return false;
+
+ /* Order the caller's subsequent accesses after the proof. */
+ smp_mb();
+ return true;
+}
+EXPORT_SYMBOL_GPL(try_synchronize_srcu);
+
/**
* get_state_synchronize_srcu - Provide an end-of-grace-period cookie
* @ssp: srcu_struct to provide cookie for.
--
2.43.0
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]
next reply other threads:[~2026-08-12 12:33 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-12 12:32 David Woodhouse [this message]
2026-08-12 13:24 ` David Woodhouse
2026-08-21 3:22 ` Kunwu Chan
2026-08-21 11:53 ` David Woodhouse
2026-08-24 15:51 ` KunWu Chan
2026-08-25 23:49 ` Paul E. McKenney
2026-08-26 3:26 ` Kunwu Chan
2026-08-26 3:53 ` Paul E. McKenney
2026-08-26 4:19 ` KunWu Chan
2026-08-26 4:28 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0f4dea21bac43685d3286df329401177e4452b36.camel@infradead.org \
--to=dwmw2@infradead.org \
--cc=boqun.feng@gmail.com \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=neeraj.upadhyay@kernel.org \
--cc=paulmck@kernel.org \
--cc=pbonzini@redhat.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=seanjc@kernel.org \
--cc=urezki@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®