* [RFC PATCH 01/12] rcu_txn: Add single-writer RCU pseudo-transactions
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 02/12] rcu_txn: Add KUnit tests Mathieu Desnoyers
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
rcu_assign_pointer() publishes one pointer atomically. An update that
re-points several pointers exposes its intermediate states to RCU
readers: a reader that loads two of them may see the new value of one
and then the old value of the other. Moving an object between two hash
buckets and editing a doubly-linked list are such updates.
Add RCU pseudo-transactions, which switch a set of RCU-protected
pointers from their old to their new values with a single store, as
observed by readers. The writer records one {slot, old, new} triple per
pointer, then commits:
1. Install: each slot is made to point at its record, tagged so that
readers recognise a proxy. A proxy resolves through a selector
shared by the records of the transaction, to its old target.
2. Commit: one store to the selector makes every proxy resolve to its
new target.
3. Settle: each slot is overwritten with its new target, and the
records are freed after a grace period.
Readers use rcu_txn_dereference() in place of rcu_dereference(). On a
slot that holds no proxy, it adds a test of the tag bits.
Writers of a slot are excluded by the caller, as for any RCU update.
The transaction has no read set, no conflict detection and no abort, and
readers get no snapshot, hence "pseudo".
A transaction of a single record installs no proxy and commits with one
rcu_assign_pointer(). A writer that may write the same slot twice reads
it back with rcu_txn_load() and records with rcu_txn_record_chain(),
which keeps one record per slot. The lookup is a linear scan of the
records, which rcu_txn_declare_disjoint() turns off.
The records and the selector live in one descriptor, allocated with the
gfp_t given to rcu_txn_init() and freed with kfree_rcu() after a commit
that installed proxies. A descriptor that installed none, after a
commit of one record or a cancel, owes no grace period: it is freed with
kfree() or, on PREEMPT_RT, with kfree_rcu(): rcu_txn_commit() and
rcu_txn_cancel() may then be called under a raw or a bit spinlock on
every kernel, as a hash table with hlist_bl buckets does. An allocation
failure is sticky and reported by rcu_txn_commit(), which then publishes
nothing. Slots are __rcu pointers, handed over as void __rcu ** through
rcu_txn_slot() so that sparse checks the annotation.
The inline paths are in <linux/rcu_txn.h>. What allocates or frees a
descriptor, and the commit of two records or more, are in lib/rcu_txn.c,
built with the hidden CONFIG_RCU_TXN that users select; the header does
not include <linux/slab.h>.
Add a MAINTAINERS entry for the new files.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
MAINTAINERS | 7 +
include/linux/rcu_txn.h | 765 ++++++++++++++++++++++++++++++++++++++++
lib/Kconfig | 3 +
lib/Makefile | 2 +
lib/rcu_txn.c | 105 ++++++
lib/rcu_txn_internal.h | 50 +++
6 files changed, 932 insertions(+)
create mode 100644 include/linux/rcu_txn.h
create mode 100644 lib/rcu_txn.c
create mode 100644 lib/rcu_txn_internal.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f53047aa8d3c..a5dc2eda633f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -22853,6 +22853,13 @@ M: Patrick Lerda <patrick9876@free.fr>
S: Maintained
F: drivers/media/rc/ir-rcmm-decoder.c
+RCU PSEUDO-TRANSACTIONS
+M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+L: rcu@vger.kernel.org
+S: Maintained
+F: include/linux/rcu_txn*.h
+F: lib/rcu_txn*
+
RCUTORTURE TEST FRAMEWORK
M: "Paul E. McKenney" <paulmck@kernel.org>
M: Josh Triplett <josh@joshtriplett.org>
diff --git a/include/linux/rcu_txn.h b/include/linux/rcu_txn.h
new file mode 100644
index 000000000000..48e87ea2d90b
--- /dev/null
+++ b/include/linux/rcu_txn.h
@@ -0,0 +1,765 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+/*
+ * Single-writer RCU pseudo-transactions
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+#ifndef _LINUX_RCU_TXN_H
+#define _LINUX_RCU_TXN_H
+
+/*
+ * Single-writer RCU pseudo-transactions switch a set of RCU-protected pointers,
+ * the slots, from their old to their new targets with a single store, as
+ * observed by concurrent RCU readers: at any instant, every slot of the set
+ * resolves to its old target, or every slot to its new one.
+ *
+ * The slots are the edges of the data structure that an update re-points; the
+ * writer records each edge as its slot with the slot's old and new targets.
+ *
+ * Each slot is made to hold a tagged pointer to a record of the transaction, a
+ * proxy, which resolves to the slot's old target while the selector of the
+ * transaction's flip group is 0, and to its new target once it is 1.
+ * rcu_txn_commit() installs the proxies, stores 1 to the selector, then settles
+ * each slot to its new target and frees the records after a grace period.
+ *
+ * The embedding data structure is responsible for:
+ * - excluding concurrent writers from the slots a transaction commits;
+ * - choosing the tag that marks a proxy pointer so its readers recognise
+ * it (a spare low bit, or a reserved type code in an already-tagged
+ * pointer), and never storing a live value that carries it;
+ * - routing its readers' slot loads through rcu_txn_resolve(), within an
+ * RCU read-side critical section.
+ *
+ * Store-release / load-acquire pairs
+ * ----------------------------------
+ * What each of the writer's store-release orders, and against which load:
+ *
+ * - Install. The store-release of the tagged proxy into the slot
+ * (rcu_assign_pointer()) pairs with the reader's rcu_dereference() of
+ * that slot. It orders the writer's earlier stores before the reader's
+ * loads through the proxy: the record's {old, new, group}, the selector's
+ * initialization to 0, and the new data being published, which the
+ * embedder builds before the commit. A reader that finds the proxy thus
+ * sees the new data built, whatever the selector holds.
+ *
+ * - Commit. The store-release of the selector (0 -> 1) pairs with the
+ * load-acquire of the selector in __rcu_txn_proxy_resolve(). It orders
+ * every install of the transaction before the flip, so that a reader that
+ * resolves one proxy to its new target, then loads another slot of the
+ * transaction, ordered after that resolution, finds that slot's proxy or
+ * its new target, never its old one.
+ *
+ * - Settle. The store-release of the new target into the slot pairs with
+ * the reader's rcu_dereference() of that slot: the usual RCU publication
+ * guarantee, for a reader that finds the new target settled.
+ *
+ * A single-record commit installs no proxy: it is the settle store alone.
+ *
+ * Ordering / monotonicity
+ * ------------------------
+ * The selector is written exactly once (0 -> 1) and never back, so a
+ * reader resolving several proxy records over time observes a monotone
+ * old...old,new...new sequence -- never new-then-old. Embedders can rely
+ * on this together with a fixed read order to avoid a per-traversal
+ * snapshot: a reader that follows a forward edge of a transaction to its new
+ * target, then a back edge of the same transaction from that node, finds the
+ * back edge's new target, never its old one.
+ *
+ * Scope of "never new-then-old": Monotonicity of the selector is
+ * unconditional. It is one word, stored once. What a reader's sequence of
+ * resolutions inherits from it depends on how the reader got from one slot to
+ * the next. A dependency-chained traversal (each hop's address derived from
+ * the value the previous hop loaded) carries the order with rcu_dereference().
+ * A reader that hops without that chain -- re-reading a pointer it cached
+ * earlier, or walking prev-then-next between two independently-reached slots --
+ * has no such edge: on weakly-ordered hardware its two loads may be reordered,
+ * and it can observe the new selector's effect on the later slot and the old on
+ * the earlier one. Such a reader owes itself an explicit acquire (or a
+ * dependency).
+ *
+ * Readers and reclaim
+ * -------------------
+ * rcu_txn_commit() reclaims the transaction descriptor with kfree_rcu(). A
+ * reader must therefore load a slot and resolve what it loaded within one RCU
+ * read-side critical section.
+ *
+ * Allocation
+ * ----------
+ * The descriptor is allocated with the gfp_t given to rcu_txn_init(), by
+ * rcu_txn_reserve() or by an rcu_txn_record() that finds the handle full. A
+ * writer that records under a spinlock either initializes the handle with a
+ * non-sleeping gfp_t, or reserves its record bound before it takes the lock.
+ * Under a raw or a bit spinlock only the reserve will do: on PREEMPT_RT the
+ * allocator cannot be called there.
+ *
+ * rcu_txn_commit() and rcu_txn_cancel() allocate nothing, do not sleep, and
+ * may be called under a raw or a bit spinlock.
+ */
+
+#include <linux/bug.h>
+#include <linux/build_bug.h>
+#include <linux/compiler.h>
+#include <linux/errno.h>
+#include <linux/gfp_types.h>
+#include <linux/rcupdate.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+/*
+ * First capacity of an unreserved handle: with its header, such a descriptor
+ * fits a 256-byte allocation. A transaction that records more should
+ * rcu_txn_reserve() its bound rather than grow to it.
+ */
+#define RCU_TXN_CAP 4
+
+/*
+ * RCU pseudo-transaction (struct rcu_txn)
+ * =======================================
+ *
+ * A growable transaction over a set of slots. The handle is a small on-stack
+ * object the embedder declares and initializes with rcu_txn_init(). Nothing
+ * is allocated before the first record() or reserve(), so init cannot fail.
+ *
+ * init() --> PREPARE --commit() or cancel()--> DONE
+ * | ^
+ * | reserve() or record() |
+ * | cannot allocate |
+ * v |
+ * OOM ---commit() or cancel()------+
+ *
+ * record() appends a record {slot, old, new} to the transaction's descriptor
+ * and installs nothing, so the descriptor may still move as it grows. The
+ * record set is frozen once commit() installs the proxies. Records must target
+ * pairwise-distinct slots: see rcu_txn_record().
+ *
+ * A handle ends in commit(), which publishes the record set, or in cancel(),
+ * which drops it. Either consumes the handle, which must be initialized again
+ * before it is reused.
+ *
+ * OOM is sticky. Once a reserve() or a record() has failed to allocate, later
+ * ones do nothing and commit() publishes nothing and returns -ENOMEM. An
+ * embedder may therefore ignore their return values and test only commit()'s.
+ *
+ * Each record carries its own tag: the bits OR'd into the proxy value installed
+ * in its slot, by which the slot's readers recognise a proxy (a reserved type
+ * code, for instance). It must be within RCU_TXN_TAG_MASK.
+ */
+
+enum rcu_txn_state {
+ RCU_TXN_PREPARE = 0,
+ RCU_TXN_OOM, /* sticky: commit() returns -ENOMEM */
+ /*
+ * Terminal: commit() or cancel() consumed the handle. Using it again
+ * without initializing it warns.
+ */
+ RCU_TXN_DONE,
+};
+
+/*
+ * Shared selector for a flip group. selector == 0 -> records resolve to
+ * old_ptr; selector == 1 -> new_ptr. Written once (0 -> 1) with a
+ * store-release by __rcu_txn_group_commit(); read with a load-acquire by
+ * __rcu_txn_proxy_resolve().
+ */
+struct rcu_txn_group {
+ unsigned long selector;
+};
+
+/*
+ * Records are aligned on RCU_TXN_RECORD_ALIGN bytes, which leaves the low bits
+ * of a record's address, RCU_TXN_TAG_MASK, free for the tag that marks an
+ * installed proxy.
+ */
+#define RCU_TXN_RECORD_ALIGN 16
+#define RCU_TXN_TAG_MASK (RCU_TXN_RECORD_ALIGN - 1UL)
+
+struct rcu_txn_record {
+ void *ptr[2]; /* [0] old, [1] new; selector-indexed */
+ struct rcu_txn_group *group;
+ void __rcu **slot; /* install / settle target */
+ /*
+ * Tag bits OR'd into the proxy value installed in @slot. Per record,
+ * so that slots tagged differently (a 0xF type code, a list's bit 0)
+ * can share one transaction.
+ */
+ unsigned long tag;
+} __aligned(RCU_TXN_RECORD_ALIGN);
+
+/*
+ * Transaction descriptor: the single allocation that carries a committed
+ * transaction across its grace period. It holds the flip group every installed
+ * proxy reads, the rcu_head that defers its reclaim, and the records, which are
+ * the proxies. The first record() or reserve() allocates it, and it may move
+ * as it grows in PREPARE, no proxy being installed yet.
+ */
+struct rcu_txn_desc {
+ struct rcu_txn_group group; /* read by installed proxies */
+ struct rcu_head rcu_head; /* deferred-free handle */
+ unsigned int cap; /* records it can hold */
+ struct rcu_txn_record records[] __counted_by(cap);
+};
+
+/*
+ * A descriptor's size is then a multiple of a record's alignment, which is
+ * what makes kmalloc() return it aligned that much: see rcu_txn_desc_alloc().
+ */
+static_assert(!(offsetof(struct rcu_txn_desc, records) % RCU_TXN_RECORD_ALIGN));
+static_assert(!(sizeof(struct rcu_txn_record) % RCU_TXN_RECORD_ALIGN));
+
+/*
+ * Transaction handle: a small on-stack object, private to one writer.
+ */
+struct rcu_txn {
+ struct rcu_txn_record *records; /* @desc's, NULL until allocated */
+ struct rcu_txn_desc *desc; /* NULL until first allocated */
+ unsigned int nr;
+ unsigned int cap;
+ gfp_t gfp; /* for the descriptor */
+ enum rcu_txn_state state;
+ bool disjoint; /* declared slot-disjoint */
+};
+
+/*
+ * Reader side
+ * ===========
+ *
+ * A reader loads a transacted slot with rcu_dereference() and passes the value
+ * to rcu_txn_resolve(). rcu_txn_dereference() does both.
+ */
+
+/**
+ * rcu_txn_is_proxy() - is a slot's value an installed proxy?
+ * @v: value loaded from a transacted slot
+ * @tag: the slot's proxy tag
+ *
+ * No live value an embedder stores in a transacted slot may carry all of
+ * @tag's bits: see rcu_txn_record().
+ *
+ * Return: true if @v carries all of @tag's bits.
+ */
+static inline bool rcu_txn_is_proxy(const void *v, unsigned long tag)
+{
+ return ((unsigned long)v & tag) == tag;
+}
+
+/*
+ * Recover the record's address from a slot value that holds an installed proxy.
+ *
+ * Subtracting the tag is equivalent to masking it off, since
+ * rcu_txn_is_proxy() has shown every tag bit set in @v. It lets the compiler
+ * fold a constant @tag into the displacement of the loads that follow, where a
+ * mask would be one more operation in the reader's chain of dependent loads.
+ */
+static inline struct rcu_txn_record *__rcu_txn_untag(void *v, unsigned long tag)
+{
+ return (struct rcu_txn_record *)((unsigned long)v - tag);
+}
+
+/*
+ * Resolve a proxy to its current target.
+ *
+ * @proxy must come from an rcu_dereference() of the slot that holds it. What
+ * orders the loads below is described under "Store-release / load-acquire
+ * pairs" at the top of this file.
+ *
+ * The selector indexes proxy->ptr[] directly, so resolution is a data
+ * dependency rather than a conditional branch.
+ */
+static inline void *__rcu_txn_proxy_resolve(const struct rcu_txn_record *proxy)
+{
+ /*
+ * Load-acquire (A) pairs with the store-release (B) of
+ * __rcu_txn_group_commit().
+ */
+ return proxy->ptr[smp_load_acquire(&proxy->group->selector)];
+}
+
+/**
+ * rcu_txn_resolve() - resolve a value loaded from a transacted slot
+ * @v: value returned by rcu_dereference() on the slot
+ * @tag: the slot's proxy tag
+ *
+ * A plain value is returned as is. An installed proxy resolves through its
+ * flip group's selector, to the slot's old target until the transaction
+ * commits and to its new target from then on.
+ *
+ * Context: RCU read-side critical section, the one @v was loaded in.
+ * Return: the slot's target.
+ */
+static inline void *rcu_txn_resolve(void *v, unsigned long tag)
+{
+ if (likely(!rcu_txn_is_proxy(v, tag)))
+ return v;
+ return __rcu_txn_proxy_resolve(__rcu_txn_untag(v, tag));
+}
+
+/**
+ * rcu_txn_dereference_check() - load and resolve a transacted pointer
+ * @p: the RCU-protected pointer, a transacted slot
+ * @tag: the slot's proxy tag
+ * @c: condition under which the load is also legal, as for
+ * rcu_dereference_check()
+ *
+ * Return: the slot's target, with the type @p points to.
+ */
+#define rcu_txn_dereference_check(p, tag, c) \
+({ \
+ void *__rcu_txn_v = rcu_dereference_check(p, c); \
+ \
+ (typeof(*(p)) __force __kernel *)rcu_txn_resolve(__rcu_txn_v, tag); \
+})
+
+/**
+ * rcu_txn_dereference() - load and resolve a transacted pointer
+ * @p: the RCU-protected pointer, a transacted slot
+ * @tag: the slot's proxy tag
+ *
+ * rcu_dereference() followed by rcu_txn_resolve(). A writer that excludes
+ * the slot's other writers never finds a proxy there outside of its own
+ * rcu_txn_commit(), and may use rcu_dereference_protected() as is.
+ *
+ * Context: RCU read-side critical section.
+ * Return: the slot's target, with the type @p points to.
+ */
+#define rcu_txn_dereference(p, tag) rcu_txn_dereference_check(p, tag, 0)
+
+/**
+ * rcu_txn_slot() - the engine's view of an RCU-protected pointer's address
+ * @pp: address of an __rcu pointer, of any type
+ *
+ * The engine takes every slot as a ``void __rcu **``. This converts the
+ * address of an embedder's typed pointer, after sparse has checked that the
+ * pointer is __rcu.
+ *
+ * Return: @pp as a ``void __rcu **``.
+ */
+#define rcu_txn_slot(pp) \
+({ \
+ typeof(pp) __rcu_txn_pp = (pp); \
+ \
+ rcu_check_sparse(*__rcu_txn_pp, __rcu); \
+ (void __rcu **)__rcu_txn_pp; \
+})
+
+/*
+ * Record primitives
+ * =================
+ *
+ * What the inline paths need: record(), and the commit of a single record. A
+ * record is installed and a group committed in lib/rcu_txn.c: see
+ * __rcu_txn_commit_desc().
+ */
+
+/*
+ * Set a record's {old, new, slot, tag}; its group is bound at install.
+ */
+static inline void __rcu_txn_record_set(struct rcu_txn_record *r,
+ void __rcu **slot, void *old_ptr,
+ void *new_ptr, unsigned long tag)
+{
+ r->ptr[0] = old_ptr;
+ r->ptr[1] = new_ptr;
+ r->slot = slot;
+ r->tag = tag;
+}
+
+/*
+ * Settle record @r's slot: store its new target there, over the proxy. For a
+ * transaction of one record, which installs no proxy, this store is the whole
+ * commit.
+ */
+static inline void __rcu_txn_record_settle(struct rcu_txn_record *r)
+{
+ /* Store-release of slot pairs with rcu_dereference(). */
+ rcu_assign_pointer(*r->slot, r->ptr[1]);
+}
+
+/*
+ * Writer side
+ * ===========
+ *
+ * The handle's contract is stated under "RCU pseudo-transaction" above.
+ */
+
+/**
+ * rcu_txn_init() - initialize a transaction handle
+ * @t: the handle
+ * @gfp: allocation flags for the transaction's descriptor
+ *
+ * Allocates nothing, so it cannot fail. @gfp is used by rcu_txn_reserve()
+ * and by an rcu_txn_record() that finds the handle full.
+ */
+static inline void rcu_txn_init(struct rcu_txn *t, gfp_t gfp)
+{
+ memset(t, 0, sizeof(*t));
+ t->state = RCU_TXN_PREPARE;
+ t->gfp = gfp;
+}
+
+/*
+ * Move the handle's records to a new descriptor holding at least @cap of them.
+ * PREPARE only: no proxy is installed yet, so the records may move. Returns
+ * false if it cannot allocate, leaving the handle in the RCU_TXN_OOM state.
+ */
+bool __rcu_txn_resize(struct rcu_txn *t, unsigned int cap);
+
+/**
+ * rcu_txn_reserve() - give the handle a descriptor for a number of records
+ * @t: the handle
+ * @cap: number of records the descriptor must hold
+ *
+ * Optional: record() grows the descriptor on demand. An embedder whose record
+ * count is bounded can reserve that bound up front, where a failure is easy to
+ * handle and where it may sleep; every later record() then appends without
+ * allocating and cannot fail.
+ *
+ * RCU_TXN_CAP is not a floor: it is only where an unreserved handle starts.
+ * Reserving less than the transaction goes on to record is correct, and costs
+ * the grow that a reserve exists to avoid.
+ *
+ * Call after init and before commit. A handle that already buffers records
+ * grows to fit @cap.
+ *
+ * Context: May sleep if the handle's gfp_t allows it.
+ * Return: true, or false if it cannot allocate, which is sticky.
+ */
+static inline bool rcu_txn_reserve(struct rcu_txn *t, unsigned int cap)
+{
+ if (unlikely(t->state != RCU_TXN_PREPARE)) {
+ /* OOM is sticky. DONE: commit() consumed the handle. */
+ WARN_ON_ONCE(t->state != RCU_TXN_OOM);
+ return false;
+ }
+ if (t->records && cap <= t->cap)
+ return true; /* already large enough */
+ return __rcu_txn_resize(t, cap);
+}
+
+/* Free a descriptor that installed no proxy: see lib/rcu_txn.c. */
+void __rcu_txn_free_desc(struct rcu_txn_desc *desc);
+
+/*
+ * Release the handle's descriptor, if it has one. No proxy of it was
+ * installed, so no grace period is owed.
+ */
+static inline void __rcu_txn_release_desc(struct rcu_txn *t)
+{
+ if (t->desc)
+ __rcu_txn_free_desc(t->desc);
+}
+
+/*
+ * Mark the handle consumed, by commit() or by cancel(): it no longer owns a
+ * descriptor, and must be initialized again before it is reused.
+ */
+static inline void __rcu_txn_done(struct rcu_txn *t)
+{
+ t->desc = NULL;
+ t->records = NULL;
+ t->nr = 0;
+ t->state = RCU_TXN_DONE;
+}
+
+/*
+ * Scan this transaction's records for @slot, or NULL. Linear, so O(nr) per
+ * call and O(nr^2) to build a write set. The returned pointer is invalidated
+ * by the next record() (a grow may move the records), so use it before
+ * recording again.
+ */
+static inline struct rcu_txn_record *__rcu_txn_scan(const struct rcu_txn *t,
+ void __rcu **slot)
+{
+ unsigned int i;
+
+ for (i = 0; i < t->nr; i++)
+ if (t->records[i].slot == slot)
+ return &t->records[i];
+ return NULL;
+}
+
+/*
+ * Grow a full handle: double its capacity, or give an unreserved handle its
+ * first RCU_TXN_CAP records. Returns false if it cannot allocate.
+ *
+ * Out of line: with this path in its body, the compiler does not inline
+ * rcu_txn_record(), and every recorded edge pays a call for an append that is
+ * five stores.
+ */
+bool __rcu_txn_grow(struct rcu_txn *t);
+
+/**
+ * rcu_txn_record() - record one edge of a transaction
+ * @t: the handle
+ * @slot: the slot to switch
+ * @old_ptr: the slot's current target
+ * @new_ptr: the target the transaction gives it
+ * @tag: the slot's proxy tag
+ *
+ * Record that the transaction switches @slot from @old_ptr to @new_ptr. @tag
+ * is OR'd into the proxy value installed in @slot, so that the slot's readers
+ * recognise the proxy and route its resolution (a reserved type code, or a
+ * list's bit 0). Neither @old_ptr nor @new_ptr may carry all of @tag's bits:
+ * after settle the slot holds @new_ptr directly, and a reader would resolve it
+ * as a proxy.
+ *
+ * PREPARE only: the record set is frozen once proxies are installed, so this
+ * must run before commit(). Nothing is installed here.
+ *
+ * Records must target pairwise-distinct slots. record() appends without
+ * looking for an earlier record of the same slot; recording a slot twice
+ * installs two proxies on it and settles both in record order, so the earlier
+ * edit is lost. Use rcu_txn_record_chain() when the same slot may be written
+ * twice.
+ *
+ * Context: The caller excludes other writers of @slot. May sleep if the
+ * handle is full and its gfp_t allows it.
+ * Return: true, or false if it cannot allocate, which is sticky.
+ */
+static inline bool rcu_txn_record(struct rcu_txn *t, void __rcu **slot,
+ void *old_ptr, void *new_ptr,
+ unsigned long tag)
+{
+ struct rcu_txn_record *r;
+
+ if (unlikely(t->state != RCU_TXN_PREPARE)) {
+ /* OOM is sticky. DONE: commit() consumed the handle. */
+ WARN_ON_ONCE(t->state != RCU_TXN_OOM);
+ return false;
+ }
+ if (unlikely(t->nr == t->cap) && !__rcu_txn_grow(t))
+ return false; /* OOM, now sticky */
+ r = &t->records[t->nr++];
+ __rcu_txn_record_set(r, slot, old_ptr, new_ptr, tag);
+ return true;
+}
+
+/*
+ * Read-your-own-writes pair (rcu_txn_load() / rcu_txn_record_chain())
+ * ====================================================================
+ *
+ * rcu_txn_record() requires pairwise-distinct slots. This suits an embedder
+ * whose slots are distinct by construction, such as a list operation recording
+ * two edges: it pays no scan per store.
+ *
+ * An embedder whose write sites can coincide needs the opposite. Deleting two
+ * adjacent nodes of a list in one transaction writes the predecessor's next
+ * slot twice, first to skip the one node and then the other, and the second
+ * store must fuse with the pending one rather than duplicate it. This pair
+ * does that:
+ *
+ * old = rcu_txn_load(t, slot, TAG);
+ * rcu_txn_record_chain(t, slot, old, f(old), TAG);
+ *
+ * The transaction then keeps at most one record per slot. There is no read
+ * set: a load records nothing and is not validated, since a single writer has
+ * nothing to validate against.
+ *
+ * Scope. The pair fuses same-slot stores. It does not make every composition
+ * sound: an embedder whose new value is a neighbour's pointer (a list) can
+ * still build its write set from stale reads of distinct slots, unless its
+ * traversal also reads through rcu_txn_load().
+ *
+ * After an OOM the pair no longer reads its own writes: the failed record()
+ * appended nothing, so a later load() of that slot returns the committed value.
+ * commit() then publishes nothing, but an embedder that changes state outside
+ * the transaction as it records must not derive it from such a load.
+ */
+
+/**
+ * rcu_txn_declare_disjoint() - declare the write set slot-disjoint
+ * @t: the handle
+ *
+ * Declare that no slot is recorded twice. rcu_txn_load() then returns the
+ * committed value without looking for a pending one, and
+ * rcu_txn_record_chain() appends without looking for an earlier record.
+ *
+ * Distinctness is a property of the slot address, not of the nodes edited: two
+ * operations on distinct nodes, such as the deletion of two adjacent nodes of
+ * a list, may write the same slot.
+ *
+ * This skips the lookup among the records so far on every load and record, a
+ * scan: a small saving from the second record on.
+ *
+ * Recording a slot twice on such a handle breaks the rule of rcu_txn_record(),
+ * with the consequence stated there.
+ *
+ * Call after init and before the first record(). On a handle whose reserve()
+ * failed it does nothing: the failure is sticky, and commit() reports it.
+ */
+static inline void rcu_txn_declare_disjoint(struct rcu_txn *t)
+{
+ if (t->state == RCU_TXN_OOM)
+ return;
+ /* Later, the records so far would no longer be looked up. */
+ if (WARN_ON_ONCE(t->state != RCU_TXN_PREPARE || t->nr))
+ return;
+ t->disjoint = true;
+}
+
+/**
+ * rcu_txn_load() - load a slot as the transaction will leave it
+ * @t: the handle
+ * @slot: the slot
+ * @tag: the slot's proxy tag
+ *
+ * On a handle declared disjoint, this returns the committed value without
+ * looking for a pending one.
+ *
+ * Context: The caller excludes other writers of @slot.
+ * Return: the pending new value of the transaction's record for @slot if it
+ * has one, else the slot's current value.
+ */
+static inline void *rcu_txn_load(struct rcu_txn *t, void __rcu **slot,
+ unsigned long tag)
+{
+ struct rcu_txn_record *r;
+ void *v;
+
+ /* A consumed handle has no record left: it reads committed values. */
+ WARN_ON_ONCE(t->state == RCU_TXN_DONE);
+ if (!t->disjoint) {
+ r = __rcu_txn_scan(t, slot);
+ if (r)
+ return r->ptr[1]; /* our own pending write */
+ }
+ v = rcu_dereference_protected(*slot, 1);
+ return v;
+}
+
+/*
+ * The read-your-own-writes half of rcu_txn_record_chain(): chain onto this
+ * transaction's record for @slot, or append one if it has none.
+ */
+static inline bool __rcu_txn_chain(struct rcu_txn *t, void __rcu **slot,
+ void *old_ptr, void *new_ptr,
+ unsigned long tag)
+{
+ struct rcu_txn_record *r;
+
+ r = __rcu_txn_scan(t, slot);
+ if (!r)
+ return rcu_txn_record(t, slot, old_ptr, new_ptr, tag);
+ r->ptr[1] = new_ptr; /* committed old preserved */
+ return true;
+}
+
+/**
+ * rcu_txn_record_chain() - record an edge on a slot that may repeat
+ * @t: the handle
+ * @slot: the slot to switch
+ * @old_ptr: what rcu_txn_load() returned for @slot
+ * @new_ptr: the target the transaction gives it
+ * @tag: the slot's proxy tag
+ *
+ * Record that the transaction switches @slot from @old_ptr to @new_ptr,
+ * chaining onto this transaction's record for @slot if it has one: that record
+ * keeps its committed old and takes @new_ptr. Otherwise append, as
+ * rcu_txn_record() does. See the pair's contract above.
+ *
+ * On the chain path @old_ptr is the record's pending value, and a different
+ * @old_ptr means the caller computed @new_ptr from a stale read.
+ *
+ * Context: As rcu_txn_record().
+ * Return: As rcu_txn_record().
+ */
+static inline bool rcu_txn_record_chain(struct rcu_txn *t, void __rcu **slot,
+ void *old_ptr, void *new_ptr,
+ unsigned long tag)
+{
+ if (unlikely(t->state != RCU_TXN_PREPARE)) {
+ /* OOM is sticky. DONE: commit() consumed the handle. */
+ WARN_ON_ONCE(t->state != RCU_TXN_OOM);
+ return false;
+ }
+ if (t->disjoint) { /* no slot repeats: skip the lookup */
+ return rcu_txn_record(t, slot, old_ptr, new_ptr, tag);
+ }
+ return __rcu_txn_chain(t, slot, old_ptr, new_ptr, tag);
+}
+
+/**
+ * rcu_txn_cancel() - drop a transaction instead of committing it
+ * @t: the handle
+ *
+ * No proxy was installed, so nothing is published, and no grace period is
+ * owed: the descriptor is freed with kfree(), or with kfree_rcu() on
+ * PREEMPT_RT (see lib/rcu_txn.c). Like commit(), it consumes the handle, which
+ * must be initialized again before it is reused.
+ *
+ * A handle ends in commit() or in cancel(): one that is merely dropped leaks
+ * its descriptor. A handle in the OOM state may be cancelled as well.
+ *
+ * Only the records are dropped: what an embedder changed outside the
+ * transaction as it recorded is not rolled back.
+ */
+static inline void rcu_txn_cancel(struct rcu_txn *t)
+{
+ /* A consumed handle must be initialized again before reuse. */
+ if (WARN_ON_ONCE(t->state == RCU_TXN_DONE))
+ return;
+ __rcu_txn_release_desc(t); /* no proxy: free now */
+ __rcu_txn_done(t);
+}
+
+/*
+ * Install, flip, settle and retire the first @nr records of @desc: the commit
+ * of a transaction that installs proxies, which takes two records or more.
+ */
+void __rcu_txn_commit_desc(struct rcu_txn_desc *desc, unsigned int nr);
+
+/**
+ * rcu_txn_commit() - commit a transaction
+ * @t: the handle
+ *
+ * Install every record's proxy, flip the group so that every proxy resolves
+ * to new atomically, then settle each slot to its new value. commit() owns
+ * reclaim and consumes the handle, which must be initialized again before it
+ * is reused.
+ *
+ * With one record, a lone store-release to the slot is already an atomic
+ * commit. No proxy is installed, so no reader can hold one: no grace period is
+ * owed, and the descriptor is freed as by rcu_txn_cancel(). The common
+ * one-pointer publish thus needs no install and no settle. An empty
+ * transaction publishes nothing.
+ *
+ * With two or more records, the descriptor's reclaim is deferred with
+ * kfree_rcu(): a reader may still hold a proxy.
+ *
+ * Context: The caller excludes other writers of the recorded slots. Does not
+ * sleep, and may be called under a raw or a bit spinlock.
+ * Return: 0, or -ENOMEM if a reserve() or a record() of this transaction
+ * failed, in which case nothing is published. -EINVAL, with a warning,
+ * for a handle that was already consumed.
+ */
+static inline int rcu_txn_commit(struct rcu_txn *t)
+{
+ const unsigned int nr = t->nr;
+
+ /* A consumed handle must be initialized again before reuse. */
+ if (WARN_ON_ONCE(t->state == RCU_TXN_DONE))
+ return -EINVAL;
+ if (unlikely(t->state == RCU_TXN_OOM)) {
+ rcu_txn_cancel(t);
+ return -ENOMEM;
+ }
+ if (nr <= 1) {
+ if (nr == 1) {
+ struct rcu_txn_record *r = &t->records[0];
+
+ /* One record: its settle store is the whole commit. */
+ __rcu_txn_record_settle(r);
+ }
+ __rcu_txn_release_desc(t); /* no proxy: free now */
+ } else {
+ /*
+ * The proxies live in the descriptor, which is reclaimed after
+ * a grace period.
+ */
+ __rcu_txn_commit_desc(t->desc, nr);
+ }
+ __rcu_txn_done(t);
+ return 0;
+}
+
+#endif /* _LINUX_RCU_TXN_H */
diff --git a/lib/Kconfig b/lib/Kconfig
index 4e6b34c3346d..c03dff4b4ed6 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -366,6 +366,9 @@ config ASSOCIATIVE_ARRAY
config CLOSURES
bool
+config RCU_TXN
+ bool
+
config HAS_IOMEM
bool
depends on !NO_IOMEM
diff --git a/lib/Makefile b/lib/Makefile
index dfab958327c5..ef01dc3182e5 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -223,6 +223,8 @@ obj-$(CONFIG_CPU_RMAP) += cpu_rmap.o
obj-$(CONFIG_CLOSURES) += closure.o
+obj-$(CONFIG_RCU_TXN) += rcu_txn.o
+
obj-$(CONFIG_DQL) += dynamic_queue_limits.o
obj-$(CONFIG_GLOB) += glob.o
diff --git a/lib/rcu_txn.c b/lib/rcu_txn.c
new file mode 100644
index 000000000000..232ad715468a
--- /dev/null
+++ b/lib/rcu_txn.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Single-writer RCU pseudo-transactions: the out-of-line paths
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * The engine is inline in <linux/rcu_txn.h>. What lives here is what allocates
+ * a transaction's descriptor, and the commit that installs proxies.
+ */
+
+#include <linux/export.h>
+#include <linux/overflow.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+
+#include "rcu_txn_internal.h"
+
+/*
+ * Allocate a transaction descriptor for at least @cap records, with its flip
+ * group initialized. Its capacity is what the allocation holds, which may be
+ * more. Returns NULL if it cannot allocate.
+ *
+ * The records need RCU_TXN_RECORD_ALIGN. kmalloc() aligns an object on the
+ * largest power of two that divides its size, and the size of a descriptor is a
+ * multiple of that alignment, as is every size kmalloc_size_roundup() returns
+ * for it.
+ */
+static struct rcu_txn_desc *rcu_txn_desc_alloc(unsigned int cap, gfp_t gfp)
+{
+ struct rcu_txn_desc *desc;
+ size_t size;
+
+ size = kmalloc_size_roundup(struct_size(desc, records, cap));
+ desc = kmalloc(size, gfp);
+ if (!desc)
+ return NULL;
+ __rcu_txn_group_init(&desc->group);
+ desc->cap = (size - sizeof(*desc)) / sizeof(desc->records[0]);
+ return desc;
+}
+
+/*
+ * Free a descriptor that installed no proxy, which owes no grace period.
+ * commit() and cancel() may run under a raw or a bit spinlock, where
+ * PREEMPT_RT does not allow kfree(). It does allow kfree_rcu().
+ */
+void __rcu_txn_free_desc(struct rcu_txn_desc *desc)
+{
+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
+ kfree_rcu(desc, rcu_head);
+ else
+ kfree(desc);
+}
+EXPORT_SYMBOL_GPL(__rcu_txn_free_desc);
+
+/* See <linux/rcu_txn.h>. */
+bool __rcu_txn_resize(struct rcu_txn *t, unsigned int cap)
+{
+ struct rcu_txn_desc *desc;
+
+ desc = rcu_txn_desc_alloc(cap, t->gfp);
+ if (!desc)
+ goto fail;
+ if (t->nr) /* @records is NULL until first allocated */
+ memcpy(desc->records, t->records,
+ (size_t)t->nr * sizeof(*desc->records));
+ kfree(t->desc); /* NULL if it had none */
+ t->desc = desc;
+ t->records = desc->records;
+ t->cap = desc->cap;
+ return true;
+
+fail:
+ t->state = RCU_TXN_OOM; /* sticky */
+ return false;
+}
+EXPORT_SYMBOL_GPL(__rcu_txn_resize);
+
+/* See <linux/rcu_txn.h>. */
+bool __rcu_txn_grow(struct rcu_txn *t)
+{
+ return __rcu_txn_resize(t, t->cap ? t->cap * 2 : RCU_TXN_CAP);
+}
+EXPORT_SYMBOL_GPL(__rcu_txn_grow);
+
+/* See <linux/rcu_txn.h>. */
+void __rcu_txn_commit_desc(struct rcu_txn_desc *desc, unsigned int nr)
+{
+ struct rcu_txn_record *records = desc->records;
+ unsigned int i;
+
+ /* Install: readers still resolve to old, the selector being 0. */
+ for (i = 0; i < nr; i++)
+ __rcu_txn_record_install(&desc->group, &records[i]);
+ /* Flip: every proxy resolves to new. */
+ __rcu_txn_group_commit(&desc->group);
+ /* Settle. */
+ for (i = 0; i < nr; i++)
+ __rcu_txn_record_settle(&records[i]);
+ /* A reader may still hold a proxy, which lives in the descriptor. */
+ kfree_rcu(desc, rcu_head);
+}
+EXPORT_SYMBOL_GPL(__rcu_txn_commit_desc);
diff --git a/lib/rcu_txn_internal.h b/lib/rcu_txn_internal.h
new file mode 100644
index 000000000000..7e79a48697a8
--- /dev/null
+++ b/lib/rcu_txn_internal.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+/*
+ * Single-writer RCU pseudo-transactions: the steps of a commit
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+#ifndef _LIB_RCU_TXN_INTERNAL_H
+#define _LIB_RCU_TXN_INTERNAL_H
+
+/*
+ * The steps of a commit that installs proxies: see __rcu_txn_commit_desc() in
+ * rcu_txn.c. The KUnit tests include this as well, to stage a transaction
+ * between two steps.
+ */
+
+#include <linux/rcu_txn.h>
+
+static inline void __rcu_txn_group_init(struct rcu_txn_group *group)
+{
+ group->selector = 0;
+}
+
+/*
+ * Commit the flip group: switch every record in the transaction from old to new
+ * with a single store-release. Must be called once every record is installed:
+ * the store-release orders the installs before the flip.
+ */
+static inline void __rcu_txn_group_commit(struct rcu_txn_group *group)
+{
+ /*
+ * Store-release (B) pairs with the load-acquire (A) of
+ * __rcu_txn_proxy_resolve().
+ */
+ smp_store_release(&group->selector, 1);
+}
+
+/*
+ * Install record @r's tagged proxy into its slot, bound to flip group @group.
+ * Readers resolve it to old until the group commits. The installed value is
+ * @r's address OR'd with the record's tag.
+ */
+static inline void __rcu_txn_record_install(struct rcu_txn_group *group,
+ struct rcu_txn_record *r)
+{
+ r->group = group;
+ /* Store-release of slot pairs with rcu_dereference(). */
+ rcu_assign_pointer(*r->slot, (void *)((unsigned long)r | r->tag));
+}
+
+#endif /* _LIB_RCU_TXN_INTERNAL_H */
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 02/12] rcu_txn: Add KUnit tests
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 01/12] rcu_txn: Add single-writer " Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 03/12] rcu_txn: Skip the record lookup with a Bloom filter Mathieu Desnoyers
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Test the RCU pseudo-transactions: commits of one and of several records,
the three steps of a commit as a reader observes them, typed slots,
growth and reservation of a transaction, cancellation, the sticky
allocation failure, reading back one's own writes, and misuse of a
handle.
The steps of a commit are staged by hand through lib/rcu_txn_internal.h,
since rcu_txn_commit() runs them back to back.
A misuse reports through WARN_ON_ONCE(), which the tests suppress and
count. Where the architecture does not keep a WARN_ON_ONCE()'s once
state in its bug table (no __WARN_FLAGS), it fires once per boot whether
it is suppressed or not, so the count is only checked to be at most one
there.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
MAINTAINERS | 1 +
lib/Kconfig.debug | 13 ++
lib/tests/Makefile | 1 +
lib/tests/rcu_txn_kunit.c | 463 ++++++++++++++++++++++++++++++++++++++
4 files changed, 478 insertions(+)
create mode 100644 lib/tests/rcu_txn_kunit.c
diff --git a/MAINTAINERS b/MAINTAINERS
index a5dc2eda633f..5126f4ece4e7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -22859,6 +22859,7 @@ L: rcu@vger.kernel.org
S: Maintained
F: include/linux/rcu_txn*.h
F: lib/rcu_txn*
+F: lib/tests/rcu_txn_kunit.c
RCUTORTURE TEST FRAMEWORK
M: "Paul E. McKenney" <paulmck@kernel.org>
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 6b24a2c89b84..5a93d2afdd96 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2844,6 +2844,19 @@ config LIST_PRIVATE_KUNIT_TEST
If unsure, say N.
+config RCU_TXN_KUNIT_TEST
+ tristate "KUnit Test for single-writer RCU pseudo-transactions" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ select RCU_TXN
+ default KUNIT_ALL_TESTS
+ help
+ This builds the KUnit test suite of the single-writer RCU
+ pseudo-transactions defined in include/linux/rcu_txn.h. It tests
+ the commit of a transaction as readers observe it, and the
+ lifecycle of a transaction handle, including its misuse.
+
+ If unsure, say N.
+
config HASHTABLE_KUNIT_TEST
tristate "KUnit Test for Kernel Hashtable structures" if !KUNIT_ALL_TESTS
depends on KUNIT
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 3cac3b63a752..d9d41a5da87d 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -44,6 +44,7 @@ obj-$(CONFIG_OVERFLOW_KUNIT_TEST) += overflow_kunit.o
obj-$(CONFIG_PRANDOM_KUNIT_TEST) += random32_kunit.o
obj-$(CONFIG_PRINTF_KUNIT_TEST) += printf_kunit.o
obj-$(CONFIG_RANDSTRUCT_KUNIT_TEST) += randstruct_kunit.o
+obj-$(CONFIG_RCU_TXN_KUNIT_TEST) += rcu_txn_kunit.o
obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o
obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o
obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
new file mode 100644
index 000000000000..e5323e21157b
--- /dev/null
+++ b/lib/tests/rcu_txn_kunit.c
@@ -0,0 +1,463 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * KUnit tests of the single-writer RCU pseudo-transactions
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+
+#include <kunit/test.h>
+#include <linux/bug.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcupdate.h>
+#include <linux/string.h>
+
+#include "../rcu_txn_internal.h" /* the steps of a commit */
+
+/* Bit 0 of the proxy pointer: no value below carries it. */
+#define TAG 1UL
+
+/* Opaque slot values with the tag bit clear. */
+#define T(n) ((void *)(((unsigned long)(n) + 1) << 8))
+
+/*
+ * A WARN_ON_ONCE() of the engine reports once per boot, so a later run of the
+ * suite sees none. Where the architecture keeps the once state in its bug
+ * table, KUnit counts every suppressed occurrence instead.
+ */
+#ifdef __WARN_FLAGS
+# define EXPECT_ONE_WARNING(test) \
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1)
+#else
+# define EXPECT_ONE_WARNING(test) \
+ KUNIT_EXPECT_LE(test, KUNIT_SUPPRESSED_WARNING_COUNT(), 1)
+#endif
+
+/*
+ * Each test is the only writer of the structures it builds. It reads them
+ * as a writer, through the forms that take a lockdep condition, passing true,
+ * and outside of any RCU read-side critical section.
+ *
+ * A test that reads as a reader instead, as read_slot() below does, checks
+ * what it read only after rcu_read_unlock(): a failed KUNIT_EXPECT_*()
+ * allocates with GFP_KERNEL, which may sleep.
+ */
+
+/* Load @slot and resolve it, as a reader does. */
+static void *read_slot(void __rcu **slot)
+{
+ void *v;
+
+ rcu_read_lock();
+ v = rcu_txn_resolve(rcu_dereference(*slot), TAG);
+ rcu_read_unlock();
+ return v; /* a test value: never dereferenced */
+}
+
+/*
+ * One record: the commit stores the new target directly. No proxy is ever
+ * installed, so the descriptor is freed at once.
+ */
+static void rcu_txn_test_single_record(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_EXPECT_TRUE(test, rcu_txn_reserve(&txn, 4));
+ rcu_txn_record(&txn, &slot, T(0), T(1), TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slot), T(1));
+ KUNIT_EXPECT_FALSE(test,
+ rcu_txn_is_proxy(rcu_access_pointer(slot), TAG));
+}
+
+/*
+ * Two records or more: the commit installs every proxy, flips the group and
+ * settles every slot to new.
+ */
+static void rcu_txn_test_multi_record(struct kunit *test)
+{
+ void __rcu *s1 = RCU_INITIALIZER(T(0));
+ void __rcu *s2 = RCU_INITIALIZER(T(1));
+ void __rcu *s3 = RCU_INITIALIZER(T(2));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 4);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG);
+ rcu_txn_record(&txn, &s3, T(2), T(12), TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(10));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(11));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s3), T(12));
+}
+
+/*
+ * The three steps of a commit, by hand: an installed proxy resolves to old,
+ * the flip makes it resolve to new, and the settle leaves the new target in the
+ * slot.
+ */
+static void rcu_txn_test_commit_steps(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+
+ __rcu_txn_group_init(&group);
+ __rcu_txn_record_set(&rec, &slot, T(0), T(1), TAG);
+ __rcu_txn_record_install(&group, &rec);
+ KUNIT_EXPECT_TRUE(test,
+ rcu_txn_is_proxy(rcu_access_pointer(slot), TAG));
+ KUNIT_EXPECT_PTR_EQ(test, read_slot(&slot), T(0));
+ __rcu_txn_group_commit(&group);
+ KUNIT_EXPECT_TRUE(test,
+ rcu_txn_is_proxy(rcu_access_pointer(slot), TAG));
+ KUNIT_EXPECT_PTR_EQ(test, read_slot(&slot), T(1));
+ __rcu_txn_record_settle(&rec);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slot), T(1));
+}
+
+struct item {
+ struct item __rcu *next;
+ int key;
+};
+
+/*
+ * A typed __rcu pointer as a slot: rcu_txn_slot() hands its address to the
+ * engine and rcu_txn_dereference_check() reads it back with its type, through a
+ * proxy or not.
+ */
+static void rcu_txn_test_typed_slot(struct kunit *test)
+{
+ struct item a = { .key = 1 }, b = { .key = 2 }, c = { .key = 3 };
+ struct item __rcu *head = RCU_INITIALIZER(&a);
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+ struct rcu_txn txn;
+ struct item *it;
+
+ /* Through an installed proxy, before and after the flip. */
+ __rcu_txn_group_init(&group);
+ __rcu_txn_record_set(&rec, rcu_txn_slot(&head), &a, &b, TAG);
+ __rcu_txn_record_install(&group, &rec);
+ it = rcu_txn_dereference_check(head, TAG, true);
+ KUNIT_EXPECT_PTR_EQ(test, it, &a);
+ __rcu_txn_group_commit(&group);
+ it = rcu_txn_dereference_check(head, TAG, true);
+ KUNIT_EXPECT_PTR_EQ(test, it, &b);
+ __rcu_txn_record_settle(&rec);
+
+ /* Two typed slots in one transaction: head -> c -> b. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, rcu_txn_slot(&head), &b, &c, TAG);
+ rcu_txn_record(&txn, rcu_txn_slot(&c.next), NULL, &b, TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ it = rcu_txn_dereference_check(head, TAG, true);
+ KUNIT_EXPECT_EQ(test, it->key, 3);
+ it = rcu_txn_dereference_check(it->next, TAG, true);
+ KUNIT_EXPECT_EQ(test, it->key, 2);
+ KUNIT_EXPECT_NULL(test, rcu_txn_dereference_check(it->next, TAG, true));
+}
+
+/* Record @nr edges on distinct slots. Returns false if a record() failed. */
+static bool record_all(struct rcu_txn *txn, void __rcu **slots, unsigned int nr)
+{
+ bool all_recorded = true;
+ unsigned int i;
+
+ for (i = 0; i < nr; i++) {
+ RCU_INIT_POINTER(slots[i], T(i));
+ if (!rcu_txn_record(txn, &slots[i], T(i), T(100 + i), TAG))
+ all_recorded = false;
+ }
+ return all_recorded;
+}
+
+static bool all_settled(void __rcu **slots, unsigned int nr)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr; i++)
+ if (rcu_access_pointer(slots[i]) != T(100 + i))
+ return false;
+ return true;
+}
+
+/*
+ * Record more edges than the first capacity without a reserve(): the
+ * descriptor grows, and the commit settles the whole set.
+ */
+static void rcu_txn_test_grow(struct kunit *test)
+{
+ enum { NR = 3 * RCU_TXN_CAP + 1 };
+ void __rcu *slots[NR];
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_EXPECT_TRUE(test, record_all(&txn, slots, 1));
+ KUNIT_EXPECT_GE(test, txn.cap, RCU_TXN_CAP);
+ rcu_txn_cancel(&txn);
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_EXPECT_TRUE(test, record_all(&txn, slots, NR));
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
+}
+
+/*
+ * reserve() gives the handle room for its bound, and recording up to the bound
+ * does not move the records. Recording past it grows the descriptor.
+ */
+static void rcu_txn_test_reserve(struct kunit *test)
+{
+ enum { NR = 2 * RCU_TXN_CAP + 1 };
+ struct rcu_txn_record *records;
+ void __rcu *slots[NR];
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_EXPECT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ KUNIT_EXPECT_GE(test, txn.cap, 2);
+ records = txn.records;
+ KUNIT_EXPECT_TRUE(test, record_all(&txn, slots, 2));
+ KUNIT_EXPECT_PTR_EQ(test, txn.records, records);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, 2));
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 2);
+ KUNIT_EXPECT_TRUE(test, record_all(&txn, slots, NR));
+ KUNIT_EXPECT_GE(test, txn.cap, NR);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
+}
+
+/* A single-record transaction installs no proxy, and may use a zero tag. */
+static void rcu_txn_test_zero_tag(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ kunit_warning_suppress(test) {
+ rcu_txn_record(&txn, &slot, T(0), T(1), 0);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 0);
+ }
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slot), T(1));
+}
+
+/*
+ * A cancelled transaction publishes nothing and consumes the handle, which
+ * commits as usual once initialized again.
+ */
+static void rcu_txn_test_cancel(struct kunit *test)
+{
+ void __rcu *s1 = RCU_INITIALIZER(T(0));
+ void __rcu *s2 = RCU_INITIALIZER(T(1));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG);
+ rcu_txn_cancel(&txn);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(0));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(1));
+ KUNIT_EXPECT_NULL(test, txn.desc);
+ KUNIT_EXPECT_NULL(test, txn.records);
+ KUNIT_EXPECT_EQ(test, txn.state, RCU_TXN_DONE);
+
+ /* A handle that recorded nothing. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_cancel(&txn);
+ KUNIT_EXPECT_EQ(test, txn.state, RCU_TXN_DONE);
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(10));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(11));
+}
+
+/*
+ * The sticky OOM state, set by hand: no allocation of this size fails on
+ * demand. Later records do nothing, and commit() and cancel() both release
+ * the handle without publishing.
+ */
+static void rcu_txn_test_oom(struct kunit *test)
+{
+ void __rcu *s1 = RCU_INITIALIZER(T(0));
+ void __rcu *s2 = RCU_INITIALIZER(T(1));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ txn.state = RCU_TXN_OOM;
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_FALSE(test, rcu_txn_reserve(&txn, 8));
+ KUNIT_EXPECT_FALSE(test,
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG));
+ KUNIT_EXPECT_FALSE(test,
+ rcu_txn_record_chain(&txn, &s2, T(1), T(11),
+ TAG));
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), -ENOMEM);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 0);
+ }
+ KUNIT_EXPECT_NULL(test, txn.desc);
+ KUNIT_EXPECT_EQ(test, txn.state, RCU_TXN_DONE);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(0));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(1));
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG);
+ txn.state = RCU_TXN_OOM;
+ rcu_txn_cancel(&txn);
+ KUNIT_EXPECT_NULL(test, txn.desc);
+ KUNIT_EXPECT_EQ(test, txn.state, RCU_TXN_DONE);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(0));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(1));
+
+ /* declare_disjoint() leaves a handle whose reserve() failed as it is. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ txn.state = RCU_TXN_OOM;
+ kunit_warning_suppress(test) {
+ rcu_txn_declare_disjoint(&txn);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 0);
+ }
+ KUNIT_EXPECT_FALSE(test, txn.disjoint);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), -ENOMEM);
+}
+
+/*
+ * load() returns the transaction's pending value of a recorded slot, and
+ * record_chain() fuses a second store to it into the same record.
+ */
+static void rcu_txn_test_ryw_chain(struct kunit *test)
+{
+ enum { NR = 6 };
+ void __rcu *other = RCU_INITIALIZER(T(50));
+ void __rcu *slots[NR];
+ struct rcu_txn txn;
+ unsigned int i;
+
+ for (i = 0; i < NR; i++)
+ RCU_INIT_POINTER(slots[i], T(i));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ KUNIT_EXPECT_PTR_EQ(test, v, T(i));
+ rcu_txn_record_chain(&txn, &slots[i], v, T(20 + i), TAG);
+ }
+ /* Every slot is recorded: loads return the pending values. */
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ KUNIT_EXPECT_PTR_EQ(test, v, T(20 + i));
+ rcu_txn_record_chain(&txn, &slots[i], v, T(100 + i), TAG);
+ }
+ KUNIT_EXPECT_EQ(test, txn.nr, NR); /* fused */
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_load(&txn, &other, TAG), T(50));
+ for (i = 0; i < NR; i++) /* nothing is published yet */
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slots[i]), T(i));
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
+}
+
+/*
+ * A handle declared disjoint skips the lookup: record_chain() appends, and
+ * load() returns the committed value.
+ */
+static void rcu_txn_test_disjoint(struct kunit *test)
+{
+ void __rcu *a = RCU_INITIALIZER(T(0));
+ void __rcu *b = RCU_INITIALIZER(T(1));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_declare_disjoint(&txn);
+ rcu_txn_record_chain(&txn, &a, rcu_txn_load(&txn, &a, TAG), T(10), TAG);
+ rcu_txn_record_chain(&txn, &b, rcu_txn_load(&txn, &b, TAG), T(11), TAG);
+ KUNIT_EXPECT_EQ(test, txn.nr, 2);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(a), T(10));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(b), T(11));
+}
+
+/*
+ * A handle that commit() consumed must be initialized again. Each entry
+ * point warns and does nothing.
+ */
+static void rcu_txn_test_consumed_handle(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &slot, T(0), T(1), TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_FALSE(test, rcu_txn_reserve(&txn, 4));
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ bool recorded = rcu_txn_record(&txn, &slot, T(1), T(2), TAG);
+
+ KUNIT_EXPECT_FALSE(test, recorded);
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_FALSE(test,
+ rcu_txn_record_chain(&txn, &slot, T(1), T(2),
+ TAG));
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_load(&txn, &slot, TAG), T(1));
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ rcu_txn_declare_disjoint(&txn);
+ KUNIT_EXPECT_FALSE(test, txn.disjoint);
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ rcu_txn_cancel(&txn);
+ EXPECT_ONE_WARNING(test);
+ }
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), -EINVAL);
+ EXPECT_ONE_WARNING(test);
+ }
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slot), T(1));
+}
+
+static struct kunit_case rcu_txn_test_cases[] = {
+ KUNIT_CASE(rcu_txn_test_single_record),
+ KUNIT_CASE(rcu_txn_test_multi_record),
+ KUNIT_CASE(rcu_txn_test_commit_steps),
+ KUNIT_CASE(rcu_txn_test_typed_slot),
+ KUNIT_CASE(rcu_txn_test_grow),
+ KUNIT_CASE(rcu_txn_test_reserve),
+ KUNIT_CASE(rcu_txn_test_zero_tag),
+ KUNIT_CASE(rcu_txn_test_cancel),
+ KUNIT_CASE(rcu_txn_test_oom),
+ KUNIT_CASE(rcu_txn_test_ryw_chain),
+ KUNIT_CASE(rcu_txn_test_disjoint),
+ KUNIT_CASE(rcu_txn_test_consumed_handle),
+ {}
+};
+
+static struct kunit_suite rcu_txn_test_suite = {
+ .name = "rcu_txn",
+ .test_cases = rcu_txn_test_cases,
+};
+
+kunit_test_suite(rcu_txn_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests of the single-writer RCU pseudo-transactions");
+MODULE_LICENSE("GPL");
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 03/12] rcu_txn: Skip the record lookup with a Bloom filter
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 01/12] rcu_txn: Add single-writer " Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 02/12] rcu_txn: Add KUnit tests Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 04/12] rcu_txn: Add CONFIG_DEBUG_RCU_TXN Mathieu Desnoyers
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
rcu_txn_load() and rcu_txn_record_chain() look the slot up among the
records of the transaction with a linear scan, so a transaction of n
records costs O(n) per access and O(n^2) to build. Most lookups miss.
Answer the miss from a Bloom filter: a clear bit means the slot is not
recorded, and the scan is skipped. A hit still goes to the scan, so the
filter never changes an answer.
The filter is armed by the first lookup that faces RCU_TXN_BLOOM_MIN
records, and is not even zeroed before that. Most transactions hold a
record or two, for which the scan is cheaper than hashing.
The filter takes 128 bytes, which the handle does not carry, as it lives
on the stack: the filter lives in the descriptor, past the records. A
descriptor of RCU_TXN_BLOOM_MIN records or more has room for it among
the slots of its records array, which the array's __counted_by() covers,
and a resize carries an armed filter over to the new descriptor.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
include/linux/rcu_txn.h | 89 ++++++++++++++++++--
include/linux/rcu_txn_bloom.h | 110 +++++++++++++++++++++++++
lib/rcu_txn.c | 36 ++++++++-
lib/tests/rcu_txn_kunit.c | 147 ++++++++++++++++++++++++++++++++++
4 files changed, 373 insertions(+), 9 deletions(-)
create mode 100644 include/linux/rcu_txn_bloom.h
diff --git a/include/linux/rcu_txn.h b/include/linux/rcu_txn.h
index 48e87ea2d90b..ee152100ab95 100644
--- a/include/linux/rcu_txn.h
+++ b/include/linux/rcu_txn.h
@@ -102,6 +102,7 @@
#include <linux/errno.h>
#include <linux/gfp_types.h>
#include <linux/rcupdate.h>
+#include <linux/rcu_txn_bloom.h> /* read-your-own-writes filter */
#include <linux/stddef.h>
#include <linux/string.h>
#include <linux/types.h>
@@ -197,7 +198,13 @@ struct rcu_txn_desc {
struct rcu_txn_group group; /* read by installed proxies */
struct rcu_head rcu_head; /* deferred-free handle */
unsigned int cap; /* records it can hold */
- struct rcu_txn_record records[] __counted_by(cap);
+ /*
+ * Record-sized slots of the allocation: the @cap records and, in a
+ * descriptor of RCU_TXN_BLOOM_MIN records or more, past them, the
+ * read-your-own-writes filter (__rcu_txn_bloom()).
+ */
+ unsigned int slots;
+ struct rcu_txn_record records[] __counted_by(slots);
};
/*
@@ -218,6 +225,7 @@ struct rcu_txn {
gfp_t gfp; /* for the descriptor */
enum rcu_txn_state state;
bool disjoint; /* declared slot-disjoint */
+ bool bloom_live; /* the descriptor's filter is armed */
};
/*
@@ -463,9 +471,10 @@ static inline void __rcu_txn_done(struct rcu_txn *t)
/*
* Scan this transaction's records for @slot, or NULL. Linear, so O(nr) per
- * call and O(nr^2) to build a write set. The returned pointer is invalidated
- * by the next record() (a grow may move the records), so use it before
- * recording again.
+ * call and O(nr^2) to build a write set -- __rcu_txn_find_ryw() below is what
+ * keeps that off the hot path. The returned pointer is invalidated by the
+ * next record() (a grow may move the records), so use it before recording
+ * again.
*/
static inline struct rcu_txn_record *__rcu_txn_scan(const struct rcu_txn *t,
void __rcu **slot)
@@ -478,6 +487,69 @@ static inline struct rcu_txn_record *__rcu_txn_scan(const struct rcu_txn *t,
return NULL;
}
+/*
+ * RCU_TXN_BLOOM_MIN: the record count from which the read-your-own-writes
+ * filter (<linux/rcu_txn_bloom.h>) is used. Below it the filter is never
+ * armed, nor even zeroed.
+ *
+ * rcu_txn_load() is only called on a slot about to be recorded, so loads
+ * track records one for one, and most transactions are tiny: one edge for an
+ * hlist operation, two for a list operation. Scanning that many records is
+ * cheaper than hashing.
+ *
+ * The filter is built and armed by the first __rcu_txn_find_ryw() that faces
+ * at least this many records, then kept current by every record(). Arming on
+ * the record that crosses the threshold instead would make a transaction of
+ * exactly RCU_TXN_BLOOM_MIN records build a filter it never tests.
+ *
+ * The filter lives in the descriptor, past the records, so that the handle on
+ * the stack does not carry it. A descriptor of this many records or more has
+ * room for it (rcu_txn_desc_alloc()).
+ */
+#define RCU_TXN_BLOOM_MIN 8
+
+static_assert(RCU_TXN_BLOOM_MIN > 1);
+
+/* The filter of the handle's descriptor: see RCU_TXN_BLOOM_MIN. */
+static inline u64 *__rcu_txn_bloom(const struct rcu_txn *t)
+{
+ return (u64 *)&t->desc->records[t->desc->cap];
+}
+
+/*
+ * Build the filter from the records so far and arm it. See
+ * RCU_TXN_BLOOM_MIN.
+ */
+static inline void __rcu_txn_bloom_arm(struct rcu_txn *t)
+{
+ u64 *bloom = __rcu_txn_bloom(t);
+ unsigned int i;
+
+ memset(bloom, 0, RCU_TXN_BLOOM_SIZE);
+ for (i = 0; i < t->nr; i++)
+ rcu_txn_bloom_set(bloom, t->records[i].slot);
+ t->bloom_live = true;
+}
+
+/*
+ * __rcu_txn_scan() behind the filter: a clear bit means the slot is not
+ * recorded, and the scan is skipped. All k bits set leaves the answer to the
+ * scan, which rules out a false positive. The filter never changes the answer.
+ */
+static inline struct rcu_txn_record *__rcu_txn_find_ryw(struct rcu_txn *t,
+ void __rcu **slot)
+{
+ if (unlikely(!t->bloom_live)) {
+ /* Armed on first use: see RCU_TXN_BLOOM_MIN. */
+ if (t->nr < RCU_TXN_BLOOM_MIN)
+ return __rcu_txn_scan(t, slot);
+ __rcu_txn_bloom_arm(t);
+ }
+ if (!rcu_txn_bloom_test(__rcu_txn_bloom(t), slot))
+ return NULL; /* definitely absent */
+ return __rcu_txn_scan(t, slot);
+}
+
/*
* Grow a full handle: double its capacity, or give an unreserved handle its
* first RCU_TXN_CAP records. Returns false if it cannot allocate.
@@ -531,6 +603,8 @@ static inline bool rcu_txn_record(struct rcu_txn *t, void __rcu **slot,
return false; /* OOM, now sticky */
r = &t->records[t->nr++];
__rcu_txn_record_set(r, slot, old_ptr, new_ptr, tag);
+ if (unlikely(t->bloom_live)) /* armed: keep it current */
+ rcu_txn_bloom_set(__rcu_txn_bloom(t), slot);
return true;
}
@@ -579,7 +653,8 @@ static inline bool rcu_txn_record(struct rcu_txn *t, void __rcu **slot,
* a list, may write the same slot.
*
* This skips the lookup among the records so far on every load and record, a
- * scan: a small saving from the second record on.
+ * scan or, past RCU_TXN_BLOOM_MIN records, a Bloom filter test: a small saving
+ * from the second record on.
*
* Recording a slot twice on such a handle breaks the rule of rcu_txn_record(),
* with the consequence stated there.
@@ -619,7 +694,7 @@ static inline void *rcu_txn_load(struct rcu_txn *t, void __rcu **slot,
/* A consumed handle has no record left: it reads committed values. */
WARN_ON_ONCE(t->state == RCU_TXN_DONE);
if (!t->disjoint) {
- r = __rcu_txn_scan(t, slot);
+ r = __rcu_txn_find_ryw(t, slot);
if (r)
return r->ptr[1]; /* our own pending write */
}
@@ -637,7 +712,7 @@ static inline bool __rcu_txn_chain(struct rcu_txn *t, void __rcu **slot,
{
struct rcu_txn_record *r;
- r = __rcu_txn_scan(t, slot);
+ r = __rcu_txn_find_ryw(t, slot);
if (!r)
return rcu_txn_record(t, slot, old_ptr, new_ptr, tag);
r->ptr[1] = new_ptr; /* committed old preserved */
diff --git a/include/linux/rcu_txn_bloom.h b/include/linux/rcu_txn_bloom.h
new file mode 100644
index 000000000000..c2b3b0e6ff01
--- /dev/null
+++ b/include/linux/rcu_txn_bloom.h
@@ -0,0 +1,110 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+/*
+ * Read-your-own-writes Bloom filter of the RCU pseudo-transactions
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+#ifndef _LINUX_RCU_TXN_BLOOM_H
+#define _LINUX_RCU_TXN_BLOOM_H
+
+/*
+ * An RCU pseudo-transaction with read-your-own-writes must decide, per
+ * access, whether a slot is already in this transaction's write set.
+ * The authoritative test is a linear scan of the records, so a write
+ * set of n slots costs O(n) per access and O(n^2) to build -- and the
+ * scan is pure overhead on the dominant case, the lookup miss.
+ *
+ * This Bloom filter is an optimisation to answer the miss in O(1)
+ * without false negatives: a clear bit means the slot is definitely
+ * absent, so the scan is skipped outright. All k bits set means
+ * present or a false positive, so the scan runs.
+ *
+ * Correctness never depends on the filter's presence, width or k. A false
+ * positive only costs time: the scan of the records that the filter fails to
+ * skip, which then finds the slot absent. Correctness does depend on the
+ * absence of false negatives, which is an obligation on the engine: the filter
+ * must be set for every slot recorded while it is live (arming rebuilds it
+ * from all records, so this holds from the moment it goes live).
+ *
+ * This is the mechanism only. The state belongs to the caller, which
+ * provides RCU_TXN_BLOOM_SIZE bytes and owns zeroing them; where they live and
+ * when the filter is armed is the engine's policy: see RCU_TXN_BLOOM_MIN in
+ * <linux/rcu_txn.h>. These helpers are pure functions over the array.
+ */
+
+#include <linux/compiler_types.h>
+#include <linux/types.h>
+
+/*
+ * RCU_TXN_BLOOM_WORDS sets the filter width (64 bits each; 16 = 1024 bits).
+ * With k bits over m = 64 * WORDS bits and n recorded slots the false-positive
+ * rate is ~(1 - e^{-kn/m})^k, which for a sparse filter is (kn/m)^k: doubling
+ * the width divides it by ~2^k.
+ *
+ * Both width and k only ever trade filter cost against the false-positive
+ * rate: correctness never depends on either.
+ */
+#define RCU_TXN_BLOOM_WORDS 16
+
+/*
+ * RCU_TXN_BLOOM_K sets the number of hash bits a slot maps to. Raising k by
+ * one multiplies the false-positive rate by the fill factor kn/m: a win only
+ * while the filter is sparse.
+ */
+#define RCU_TXN_BLOOM_K 3
+
+#define RCU_TXN_BLOOM_BITS (64ULL * RCU_TXN_BLOOM_WORDS)
+#define RCU_TXN_BLOOM_SIZE (sizeof(u64) * RCU_TXN_BLOOM_WORDS)
+
+/*
+ * Two independent hashes of the slot, for Kirsch-Mitzenmacher double hashing:
+ * one SplitMix64 avalanche (two multiplies), its 64 bits split into two 32-bit
+ * lanes. A single multiply would leave the k positions correlated, slot
+ * addresses being aligned and clustered. h2 is forced odd so that h1 + i*h2
+ * visits k distinct bits.
+ */
+static inline void rcu_txn_bloom_h1h2(void __rcu **slot, u64 *h1, u64 *h2)
+{
+ /*
+ * Shift by the alignment every slot has, 4 bytes on 32-bit: >> 3 would
+ * map adjacent slots there to the same k positions. On 64-bit the one
+ * zero bit left is mixed away by the avalanche.
+ */
+ u64 x = (u64)(unsigned long)slot >> 2;
+
+ x ^= x >> 30; x *= 0xbf58476d1ce4e5b9ULL;
+ x ^= x >> 27; x *= 0x94d049bb133111ebULL;
+ x ^= x >> 31;
+ *h1 = x & 0xffffffffULL; /* low lane */
+ *h2 = (x >> 32) | 1; /* high lane, odd stride */
+}
+
+static inline bool rcu_txn_bloom_test(const u64 *bloom, void __rcu **slot)
+{
+ u64 h1, h2;
+ unsigned int i;
+
+ rcu_txn_bloom_h1h2(slot, &h1, &h2);
+ for (i = 0; i < RCU_TXN_BLOOM_K; i++) {
+ u64 idx = (h1 + (u64)i * h2) % RCU_TXN_BLOOM_BITS;
+
+ if (!(bloom[idx >> 6] & ((u64)1 << (idx & 63))))
+ return false; /* a clear bit: definitely absent */
+ }
+ return true; /* all k bits set: present (or a false positive) */
+}
+
+static inline void rcu_txn_bloom_set(u64 *bloom, void __rcu **slot)
+{
+ u64 h1, h2;
+ unsigned int i;
+
+ rcu_txn_bloom_h1h2(slot, &h1, &h2);
+ for (i = 0; i < RCU_TXN_BLOOM_K; i++) {
+ u64 idx = (h1 + (u64)i * h2) % RCU_TXN_BLOOM_BITS;
+
+ bloom[idx >> 6] |= (u64)1 << (idx & 63);
+ }
+}
+
+#endif /* _LINUX_RCU_TXN_BLOOM_H */
diff --git a/lib/rcu_txn.c b/lib/rcu_txn.c
index 232ad715468a..b040af0a8cdb 100644
--- a/lib/rcu_txn.c
+++ b/lib/rcu_txn.c
@@ -9,6 +9,8 @@
*/
#include <linux/export.h>
+#include <linux/math.h>
+#include <linux/minmax.h>
#include <linux/overflow.h>
#include <linux/rcu_txn.h>
#include <linux/rcupdate.h>
@@ -17,6 +19,24 @@
#include "rcu_txn_internal.h"
+/* Record-sized slots that the filter takes, past the records. */
+#define RCU_TXN_BLOOM_SLOTS \
+ DIV_ROUND_UP(RCU_TXN_BLOOM_SIZE, sizeof(struct rcu_txn_record))
+
+/*
+ * The records that a descriptor of @slots record-sized slots holds. Once they
+ * are RCU_TXN_BLOOM_MIN or more, the filter takes the slots past them: a
+ * descriptor with slots for that many records but not for the filter as well
+ * holds RCU_TXN_BLOOM_MIN - 1 records.
+ */
+static unsigned int rcu_txn_desc_cap(unsigned int slots)
+{
+ if (slots < RCU_TXN_BLOOM_MIN)
+ return slots;
+ return max_t(unsigned int, slots - RCU_TXN_BLOOM_SLOTS,
+ RCU_TXN_BLOOM_MIN - 1);
+}
+
/*
* Allocate a transaction descriptor for at least @cap records, with its flip
* group initialized. Its capacity is what the allocation holds, which may be
@@ -32,12 +52,17 @@ static struct rcu_txn_desc *rcu_txn_desc_alloc(unsigned int cap, gfp_t gfp)
struct rcu_txn_desc *desc;
size_t size;
- size = kmalloc_size_roundup(struct_size(desc, records, cap));
+ size = struct_size(desc, records, cap);
+ if (cap >= RCU_TXN_BLOOM_MIN)
+ size = size_add(size, RCU_TXN_BLOOM_SLOTS *
+ sizeof(struct rcu_txn_record));
+ size = kmalloc_size_roundup(size);
desc = kmalloc(size, gfp);
if (!desc)
return NULL;
__rcu_txn_group_init(&desc->group);
- desc->cap = (size - sizeof(*desc)) / sizeof(desc->records[0]);
+ desc->slots = (size - sizeof(*desc)) / sizeof(desc->records[0]);
+ desc->cap = rcu_txn_desc_cap(desc->slots);
return desc;
}
@@ -66,6 +91,13 @@ bool __rcu_txn_resize(struct rcu_txn *t, unsigned int cap)
if (t->nr) /* @records is NULL until first allocated */
memcpy(desc->records, t->records,
(size_t)t->nr * sizeof(*desc->records));
+ /*
+ * An armed filter moves too: both descriptors hold RCU_TXN_BLOOM_MIN
+ * records or more, so both have room for it.
+ */
+ if (t->bloom_live)
+ memcpy(&desc->records[desc->cap], __rcu_txn_bloom(t),
+ RCU_TXN_BLOOM_SIZE);
kfree(t->desc); /* NULL if it had none */
t->desc = desc;
t->records = desc->records;
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index e5323e21157b..3111fc537ff3 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -436,6 +436,147 @@ static void rcu_txn_test_consumed_handle(struct kunit *test)
KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(slot), T(1));
}
+static void __rcu *bloom_slots[4096];
+
+/* An empty filter answers absent for every slot. */
+static void rcu_txn_test_bloom_empty(struct kunit *test)
+{
+ u64 bloom[RCU_TXN_BLOOM_WORDS] = {};
+ unsigned int i, present = 0;
+
+ for (i = 0; i < ARRAY_SIZE(bloom_slots); i++)
+ present += rcu_txn_bloom_test(bloom, &bloom_slots[i]);
+ KUNIT_EXPECT_EQ(test, present, 0);
+}
+
+/*
+ * No false negatives: every slot ever set still reads present, whatever is set
+ * after it. This is the property the engine's correctness rests on.
+ */
+static void rcu_txn_test_bloom_no_false_negative(struct kunit *test)
+{
+ u64 bloom[RCU_TXN_BLOOM_WORDS] = {};
+ unsigned int i, j, missing = 0;
+
+ for (i = 0; i < ARRAY_SIZE(bloom_slots); i++) {
+ rcu_txn_bloom_set(bloom, &bloom_slots[i]);
+ /* Testing all i + 1 each round is O(n^2): sample. */
+ for (j = 0; j <= i; j += i / 16 + 1)
+ missing += !rcu_txn_bloom_test(bloom, &bloom_slots[j]);
+ }
+ for (i = 0; i < ARRAY_SIZE(bloom_slots); i++)
+ missing += !rcu_txn_bloom_test(bloom, &bloom_slots[i]);
+ KUNIT_EXPECT_EQ(test, missing, 0);
+}
+
+/*
+ * The filter still discriminates past its design fill: with an eighth of the
+ * slots set, not every absent slot reads present. The bound only has to catch
+ * a degenerate hash.
+ */
+static void rcu_txn_test_bloom_discriminates(struct kunit *test)
+{
+ const unsigned int nr = ARRAY_SIZE(bloom_slots);
+ u64 bloom[RCU_TXN_BLOOM_WORDS] = {};
+ unsigned int i, false_positives = 0;
+
+ for (i = 0; i < nr / 8; i++)
+ rcu_txn_bloom_set(bloom, &bloom_slots[i]);
+ for (i = nr / 8; i < nr; i++)
+ false_positives += rcu_txn_bloom_test(bloom, &bloom_slots[i]);
+ KUNIT_EXPECT_LT(test, false_positives, nr - nr / 8);
+}
+
+/*
+ * The filter does not depend on insertion order. Arming rebuilds it from the
+ * records so far, so a rebuilt filter must match one grown incrementally.
+ */
+static void rcu_txn_test_bloom_rebuild(struct kunit *test)
+{
+ u64 bloom[RCU_TXN_BLOOM_WORDS] = {}, rebuilt[RCU_TXN_BLOOM_WORDS] = {};
+ unsigned int i;
+
+ for (i = 0; i < 64; i++)
+ rcu_txn_bloom_set(bloom, &bloom_slots[i]);
+ for (i = 64; i-- > 0; )
+ rcu_txn_bloom_set(rebuilt, &bloom_slots[i]);
+ KUNIT_EXPECT_MEMEQ(test, bloom, rebuilt, sizeof(bloom));
+}
+
+/*
+ * The engine arms the filter at the first lookup that faces RCU_TXN_BLOOM_MIN
+ * records, and keeps it current from then on: slots recorded before and after
+ * the arming are both found.
+ */
+static void rcu_txn_test_bloom_arming(struct kunit *test)
+{
+ enum { NR = 2 * RCU_TXN_BLOOM_MIN };
+ void __rcu *slots[NR];
+ struct rcu_txn txn;
+ unsigned int i;
+
+ for (i = 0; i < NR; i++)
+ RCU_INIT_POINTER(slots[i], T(i));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ /* The lookup that faces RCU_TXN_BLOOM_MIN records arms it. */
+ KUNIT_EXPECT_EQ(test, txn.bloom_live, i >= RCU_TXN_BLOOM_MIN);
+ rcu_txn_record_chain(&txn, &slots[i], v, T(20 + i), TAG);
+ }
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ KUNIT_EXPECT_PTR_EQ(test, v, T(20 + i));
+ rcu_txn_record_chain(&txn, &slots[i], v, T(100 + i), TAG);
+ }
+ KUNIT_EXPECT_EQ(test, txn.nr, NR); /* fused */
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
+
+ /* A handle declared disjoint never looks up, and never arms. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_declare_disjoint(&txn);
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ rcu_txn_record_chain(&txn, &slots[i], v, T(i), TAG);
+ }
+ KUNIT_EXPECT_FALSE(test, txn.bloom_live);
+ rcu_txn_cancel(&txn);
+}
+
+/*
+ * An armed filter moves with the records when the transaction outgrows its
+ * descriptor: every slot recorded before and after the move is still found.
+ */
+static void rcu_txn_test_bloom_grow(struct kunit *test)
+{
+ enum { NR = 8 * RCU_TXN_BLOOM_MIN };
+ void __rcu *slots[NR];
+ unsigned int i, cap = 0;
+ struct rcu_txn txn;
+
+ for (i = 0; i < NR; i++)
+ RCU_INIT_POINTER(slots[i], T(i));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ for (i = 0; i < NR; i++) {
+ void *v = rcu_txn_load(&txn, &slots[i], TAG);
+
+ if (txn.bloom_live && !cap)
+ cap = txn.cap; /* armed in this descriptor */
+ rcu_txn_record_chain(&txn, &slots[i], v, T(100 + i), TAG);
+ }
+ KUNIT_ASSERT_NE(test, cap, 0);
+ KUNIT_EXPECT_GT(test, txn.cap, cap); /* grown since */
+ for (i = 0; i < NR; i++)
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_load(&txn, &slots[i], TAG),
+ T(100 + i));
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -449,6 +590,12 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_ryw_chain),
KUNIT_CASE(rcu_txn_test_disjoint),
KUNIT_CASE(rcu_txn_test_consumed_handle),
+ KUNIT_CASE(rcu_txn_test_bloom_empty),
+ KUNIT_CASE(rcu_txn_test_bloom_no_false_negative),
+ KUNIT_CASE(rcu_txn_test_bloom_discriminates),
+ KUNIT_CASE(rcu_txn_test_bloom_rebuild),
+ KUNIT_CASE(rcu_txn_test_bloom_arming),
+ KUNIT_CASE(rcu_txn_test_bloom_grow),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 04/12] rcu_txn: Add CONFIG_DEBUG_RCU_TXN
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (2 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 03/12] rcu_txn: Skip the record lookup with a Bloom filter Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 05/12] rcu_txn: Add an hlist edited through transactions Mathieu Desnoyers
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
A pseudo-transaction relies on its caller to exclude other writers, and
has no way to notice when it does not: two writers racing on a slot
corrupt it silently. Add an option that checks the contract.
It validates the proxy tags, warns about a slot recorded twice in one
transaction, and validates the single-writer contract:
- a handle must be driven by the task that initialized it;
- a slot being recorded or installed to must not hold a proxy, which
would be another writer's;
- at install, and at the commit of a single record, a slot must still
hold the old value that was recorded;
- at settle, a slot must still hold the proxy that was installed.
An embedder may get a transaction ready in one task, initializing it and
reserving its records where an allocation failure can still be handled,
then record and commit it in another task, such as the one that
completes an asynchronous request. Nothing is recorded in between, so
nothing is mid-flight when the handle changes hands. rcu_txn_adopt() is
what the task that takes over calls first: it makes that task the one
the checks expect, and warns if the handle already holds records.
Without the option it does nothing.
These checks report every occurrence, with WARN(). A once-only report
would be consumed by the first user to trip it, and hide the others.
The tests stage each violation deterministically, by installing the
other writer's proxy in the slot by hand.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
include/linux/rcu_txn.h | 195 ++++++++++++++++++++-
kernel/rcu/Kconfig.debug | 16 ++
lib/rcu_txn.c | 9 +-
lib/rcu_txn_internal.h | 79 +++++++++
lib/tests/rcu_txn_kunit.c | 350 ++++++++++++++++++++++++++++++++++++++
5 files changed, 643 insertions(+), 6 deletions(-)
diff --git a/include/linux/rcu_txn.h b/include/linux/rcu_txn.h
index ee152100ab95..2946c63fb044 100644
--- a/include/linux/rcu_txn.h
+++ b/include/linux/rcu_txn.h
@@ -107,6 +107,21 @@
#include <linux/string.h>
#include <linux/types.h>
+#ifdef CONFIG_DEBUG_RCU_TXN
+#include <asm/current.h>
+#endif
+
+/*
+ * Contract checks that cost more than a test of the handle's state are built
+ * with CONFIG_DEBUG_RCU_TXN only. They report every occurrence: a once-only
+ * report would hide the violations of every embedder but the first.
+ */
+#ifdef CONFIG_DEBUG_RCU_TXN
+# define RCU_TXN_WARN_ON(cond) WARN_ON(cond)
+#else
+# define RCU_TXN_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond)
+#endif
+
/*
* First capacity of an unreserved handle: with its header, such a descriptor
* fits a 256-byte allocation. A transaction that records more should
@@ -226,6 +241,10 @@ struct rcu_txn {
enum rcu_txn_state state;
bool disjoint; /* declared slot-disjoint */
bool bloom_live; /* the descriptor's filter is armed */
+#ifdef CONFIG_DEBUG_RCU_TXN
+ /* Task that initialized or adopted the handle, and drives it. */
+ struct task_struct *owner;
+#endif
};
/*
@@ -251,6 +270,17 @@ static inline bool rcu_txn_is_proxy(const void *v, unsigned long tag)
return ((unsigned long)v & tag) == tag;
}
+/*
+ * Does @v, loaded from a slot transacted under @tag, hold an installed proxy?
+ * Unlike rcu_txn_is_proxy(), this answers false under a zero tag, which every
+ * value matches and which marks no proxy. A zero tag is legal for a
+ * transaction that installs none.
+ */
+static inline bool __rcu_txn_holds_proxy(const void *v, unsigned long tag)
+{
+ return tag && rcu_txn_is_proxy(v, tag);
+}
+
/*
* Recover the record's address from a slot value that holds an installed proxy.
*
@@ -261,6 +291,7 @@ static inline bool rcu_txn_is_proxy(const void *v, unsigned long tag)
*/
static inline struct rcu_txn_record *__rcu_txn_untag(void *v, unsigned long tag)
{
+ RCU_TXN_WARN_ON(!rcu_txn_is_proxy(v, tag));
return (struct rcu_txn_record *)((unsigned long)v - tag);
}
@@ -383,6 +414,96 @@ static inline void __rcu_txn_record_settle(struct rcu_txn_record *r)
rcu_assign_pointer(*r->slot, r->ptr[1]);
}
+/*
+ * CONFIG_DEBUG_RCU_TXN: runtime validation of the single-writer contract.
+ *
+ * This engine requires writer mutual exclusion -- it has no install-time CAS,
+ * no conflict detection and no abort, so two writers racing on one slot simply
+ * corrupt it, and a default build does not detect it. With
+ * CONFIG_DEBUG_RCU_TXN a violation warns, with a report identifying the
+ * violated handle or slot. Without it there is no check, and the handle does
+ * not carry the owner field.
+ *
+ * There is no per-structure object to claim an owner on, so the
+ * validator claims the two things that do exist:
+ *
+ * - the handle. Its owner is the task that initialized it, or the one
+ * that adopted it before anything was recorded, and reserve / record /
+ * load / commit / cancel must all run in that task. Catches a
+ * transaction handed between tasks mid-flight.
+ *
+ * - the slot, which is what two racing writers actually share, and so is
+ * where the real violation is visible. In a correct single-writer
+ * program a slot being recorded or installed to cannot already hold an
+ * installed proxy. A proxy sitting there therefore means another
+ * writer is mid-transaction on that very slot right now.
+ * Symmetrically, at settle each slot must still hold this writer's
+ * proxy; anything else means a concurrent writer overwrote it.
+ */
+
+#ifdef CONFIG_DEBUG_RCU_TXN
+
+static inline void __rcu_txn_excl_claim(struct rcu_txn *t)
+{
+ t->owner = current;
+}
+
+static inline void __rcu_txn_excl_owner(const struct rcu_txn *t,
+ const char *what)
+{
+ WARN(t->owner != current,
+ "rcu_txn %p: %s in a task other than the one that initialized or adopted the handle -- one transaction is driven end to end by one task\n",
+ t, what);
+}
+
+/* @slot must hold no proxy: it would be another writer's. */
+static inline void __rcu_txn_excl_slot_free(void __rcu **slot,
+ unsigned long tag, const char *what)
+{
+ void *v = rcu_access_pointer(*slot);
+
+ WARN(__rcu_txn_holds_proxy(v, tag),
+ "rcu_txn: slot %p already holds an installed proxy (%p) at %s: another writer is mid-transaction on it\n",
+ slot, v, what);
+}
+
+/*
+ * @r's slot must still hold the recorded old. This is the only witness of a
+ * single-record commit, which installs no proxy.
+ */
+static inline void __rcu_txn_excl_slot_unchanged(struct rcu_txn_record *r,
+ const char *what)
+{
+ void *v = rcu_access_pointer(*r->slot);
+
+ WARN(v != r->ptr[0],
+ "rcu_txn: slot %p holds %p at %s, but %p was recorded as its old: a concurrent writer changed it\n",
+ r->slot, v, what, r->ptr[0]);
+}
+
+#else /* !CONFIG_DEBUG_RCU_TXN */
+
+static inline void __rcu_txn_excl_claim(struct rcu_txn *t)
+{
+}
+
+static inline void __rcu_txn_excl_owner(const struct rcu_txn *t,
+ const char *what)
+{
+}
+
+static inline void __rcu_txn_excl_slot_free(void __rcu **slot,
+ unsigned long tag, const char *what)
+{
+}
+
+static inline void __rcu_txn_excl_slot_unchanged(struct rcu_txn_record *r,
+ const char *what)
+{
+}
+
+#endif /* CONFIG_DEBUG_RCU_TXN */
+
/*
* Writer side
* ===========
@@ -403,6 +524,7 @@ static inline void rcu_txn_init(struct rcu_txn *t, gfp_t gfp)
memset(t, 0, sizeof(*t));
t->state = RCU_TXN_PREPARE;
t->gfp = gfp;
+ __rcu_txn_excl_claim(t);
}
/*
@@ -439,11 +561,29 @@ static inline bool rcu_txn_reserve(struct rcu_txn *t, unsigned int cap)
WARN_ON_ONCE(t->state != RCU_TXN_OOM);
return false;
}
+ __rcu_txn_excl_owner(t, "reserve()");
if (t->records && cap <= t->cap)
return true; /* already large enough */
return __rcu_txn_resize(t, cap);
}
+/**
+ * rcu_txn_adopt() - take over a handle that another task got ready
+ * @t: the handle, initialized and holding no record yet
+ *
+ * One task drives a transaction end to end. An embedder may still get a
+ * handle ready in one task, where it can sleep and fail, with rcu_txn_init()
+ * and rcu_txn_reserve(), and have another record and commit, or cancel. The
+ * task that takes over calls this first. The embedder makes sure that the
+ * task which got the handle ready is done with it.
+ */
+static inline void rcu_txn_adopt(struct rcu_txn *t)
+{
+ /* A handle that changes hands with records in it is mid-flight. */
+ RCU_TXN_WARN_ON(t->nr);
+ __rcu_txn_excl_claim(t);
+}
+
/* Free a descriptor that installed no proxy: see lib/rcu_txn.c. */
void __rcu_txn_free_desc(struct rcu_txn_desc *desc);
@@ -581,8 +721,8 @@ bool __rcu_txn_grow(struct rcu_txn *t);
* Records must target pairwise-distinct slots. record() appends without
* looking for an earlier record of the same slot; recording a slot twice
* installs two proxies on it and settles both in record order, so the earlier
- * edit is lost. Use rcu_txn_record_chain() when the same slot may be written
- * twice.
+ * edit is lost. CONFIG_DEBUG_RCU_TXN warns about it at commit. Use
+ * rcu_txn_record_chain() when the same slot may be written twice.
*
* Context: The caller excludes other writers of @slot. May sleep if the
* handle is full and its gfp_t allows it.
@@ -599,6 +739,15 @@ static inline bool rcu_txn_record(struct rcu_txn *t, void __rcu **slot,
WARN_ON_ONCE(t->state != RCU_TXN_OOM);
return false;
}
+ /*
+ * Neither value may look like a proxy under @tag. After settle the
+ * slot holds @new_ptr directly: were it tagged, every later reader
+ * would resolve it as a proxy and dereference garbage.
+ */
+ RCU_TXN_WARN_ON(__rcu_txn_holds_proxy(old_ptr, tag));
+ RCU_TXN_WARN_ON(__rcu_txn_holds_proxy(new_ptr, tag));
+ __rcu_txn_excl_owner(t, "record()");
+ __rcu_txn_excl_slot_free(slot, tag, "record()");
if (unlikely(t->nr == t->cap) && !__rcu_txn_grow(t))
return false; /* OOM, now sticky */
r = &t->records[t->nr++];
@@ -657,7 +806,8 @@ static inline bool rcu_txn_record(struct rcu_txn *t, void __rcu **slot,
* from the second record on.
*
* Recording a slot twice on such a handle breaks the rule of rcu_txn_record(),
- * with the consequence stated there.
+ * with the consequence stated there. CONFIG_DEBUG_RCU_TXN warns about it at
+ * the offending rcu_txn_load() or rcu_txn_record_chain().
*
* Call after init and before the first record(). On a handle whose reserve()
* failed it does nothing: the failure is sticky, and commit() reports it.
@@ -672,6 +822,27 @@ static inline void rcu_txn_declare_disjoint(struct rcu_txn *t)
t->disjoint = true;
}
+/*
+ * On a handle declared disjoint, warn when a load or a record targets a slot
+ * that the transaction has already recorded. The load would return the
+ * committed value where its caller expects the pending one, and the record
+ * would install a second proxy on the slot.
+ */
+#ifdef CONFIG_DEBUG_RCU_TXN
+static inline void __rcu_txn_disjoint_check(const struct rcu_txn *t,
+ void __rcu **slot, const char *what)
+{
+ WARN(t->disjoint && __rcu_txn_scan(t, slot),
+ "rcu_txn %p: slot %p is already recorded at %s, on a handle declared disjoint\n",
+ t, slot, what);
+}
+#else
+static inline void __rcu_txn_disjoint_check(const struct rcu_txn *t,
+ void __rcu **slot, const char *what)
+{
+}
+#endif
+
/**
* rcu_txn_load() - load a slot as the transaction will leave it
* @t: the handle
@@ -681,6 +852,10 @@ static inline void rcu_txn_declare_disjoint(struct rcu_txn *t)
* On a handle declared disjoint, this returns the committed value without
* looking for a pending one.
*
+ * CONFIG_DEBUG_RCU_TXN checks that an unrecorded slot holds no proxy under
+ * @tag: this writer owns every store to the slot, and its earlier transactions
+ * settled before commit() returned.
+ *
* Context: The caller excludes other writers of @slot.
* Return: the pending new value of the transaction's record for @slot if it
* has one, else the slot's current value.
@@ -691,6 +866,7 @@ static inline void *rcu_txn_load(struct rcu_txn *t, void __rcu **slot,
struct rcu_txn_record *r;
void *v;
+ __rcu_txn_excl_owner(t, "load()");
/* A consumed handle has no record left: it reads committed values. */
WARN_ON_ONCE(t->state == RCU_TXN_DONE);
if (!t->disjoint) {
@@ -698,7 +874,9 @@ static inline void *rcu_txn_load(struct rcu_txn *t, void __rcu **slot,
if (r)
return r->ptr[1]; /* our own pending write */
}
+ __rcu_txn_disjoint_check(t, slot, "load()");
v = rcu_dereference_protected(*slot, 1);
+ RCU_TXN_WARN_ON(__rcu_txn_holds_proxy(v, tag));
return v;
}
@@ -715,6 +893,9 @@ static inline bool __rcu_txn_chain(struct rcu_txn *t, void __rcu **slot,
r = __rcu_txn_find_ryw(t, slot);
if (!r)
return rcu_txn_record(t, slot, old_ptr, new_ptr, tag);
+ RCU_TXN_WARN_ON(r->tag != tag);
+ /* The caller must have read its own write. */
+ RCU_TXN_WARN_ON(r->ptr[1] != old_ptr);
r->ptr[1] = new_ptr; /* committed old preserved */
return true;
}
@@ -733,7 +914,8 @@ static inline bool __rcu_txn_chain(struct rcu_txn *t, void __rcu **slot,
* rcu_txn_record() does. See the pair's contract above.
*
* On the chain path @old_ptr is the record's pending value, and a different
- * @old_ptr means the caller computed @new_ptr from a stale read.
+ * @old_ptr means the caller computed @new_ptr from a stale read, which
+ * CONFIG_DEBUG_RCU_TXN warns about.
*
* Context: As rcu_txn_record().
* Return: As rcu_txn_record().
@@ -747,7 +929,9 @@ static inline bool rcu_txn_record_chain(struct rcu_txn *t, void __rcu **slot,
WARN_ON_ONCE(t->state != RCU_TXN_OOM);
return false;
}
+ __rcu_txn_excl_owner(t, "record_chain()");
if (t->disjoint) { /* no slot repeats: skip the lookup */
+ __rcu_txn_disjoint_check(t, slot, "record_chain()");
return rcu_txn_record(t, slot, old_ptr, new_ptr, tag);
}
return __rcu_txn_chain(t, slot, old_ptr, new_ptr, tag);
@@ -770,6 +954,7 @@ static inline bool rcu_txn_record_chain(struct rcu_txn *t, void __rcu **slot,
*/
static inline void rcu_txn_cancel(struct rcu_txn *t)
{
+ __rcu_txn_excl_owner(t, "cancel()");
/* A consumed handle must be initialized again before reuse. */
if (WARN_ON_ONCE(t->state == RCU_TXN_DONE))
return;
@@ -811,6 +996,7 @@ static inline int rcu_txn_commit(struct rcu_txn *t)
{
const unsigned int nr = t->nr;
+ __rcu_txn_excl_owner(t, "commit()");
/* A consumed handle must be initialized again before reuse. */
if (WARN_ON_ONCE(t->state == RCU_TXN_DONE))
return -EINVAL;
@@ -823,6 +1009,7 @@ static inline int rcu_txn_commit(struct rcu_txn *t)
struct rcu_txn_record *r = &t->records[0];
/* One record: its settle store is the whole commit. */
+ __rcu_txn_excl_slot_unchanged(r, "commit");
__rcu_txn_record_settle(r);
}
__rcu_txn_release_desc(t); /* no proxy: free now */
diff --git a/kernel/rcu/Kconfig.debug b/kernel/rcu/Kconfig.debug
index e078e988773d..9faa8a5e801c 100644
--- a/kernel/rcu/Kconfig.debug
+++ b/kernel/rcu/Kconfig.debug
@@ -239,4 +239,20 @@ config TRIVIAL_PREEMPT_RCU
This has no value for production and is only for testing.
+config DEBUG_RCU_TXN
+ bool "Debug checks for single-writer RCU pseudo-transactions"
+ depends on DEBUG_KERNEL && RCU_TXN
+ help
+ Enable the costlier contract checks of <linux/rcu_txn.h>: validity
+ of the proxy tags, pairwise-distinct recorded slots, and a validator
+ of the single-writer contract. The validator warns when a
+ transaction is driven by two tasks, and when a slot it records,
+ installs to or settles was touched by another writer in the
+ meantime.
+
+ This adds an owner field to every transaction handle and a pairwise
+ scan of the records to every commit of two records or more.
+
+ Say N unless you are debugging a user of RCU pseudo-transactions.
+
endmenu # "RCU Debugging"
diff --git a/lib/rcu_txn.c b/lib/rcu_txn.c
index b040af0a8cdb..a4a71dfb2282 100644
--- a/lib/rcu_txn.c
+++ b/lib/rcu_txn.c
@@ -123,14 +123,19 @@ void __rcu_txn_commit_desc(struct rcu_txn_desc *desc, unsigned int nr)
struct rcu_txn_record *records = desc->records;
unsigned int i;
+ __rcu_txn_assert_distinct_slots(records, nr);
/* Install: readers still resolve to old, the selector being 0. */
- for (i = 0; i < nr; i++)
+ for (i = 0; i < nr; i++) {
+ __rcu_txn_excl_slot_installable(&records[i]);
__rcu_txn_record_install(&desc->group, &records[i]);
+ }
/* Flip: every proxy resolves to new. */
__rcu_txn_group_commit(&desc->group);
/* Settle. */
- for (i = 0; i < nr; i++)
+ for (i = 0; i < nr; i++) {
+ __rcu_txn_excl_slot_ours(&records[i]);
__rcu_txn_record_settle(&records[i]);
+ }
/* A reader may still hold a proxy, which lives in the descriptor. */
kfree_rcu(desc, rcu_head);
}
diff --git a/lib/rcu_txn_internal.h b/lib/rcu_txn_internal.h
index 7e79a48697a8..a654b2ffaaa6 100644
--- a/lib/rcu_txn_internal.h
+++ b/lib/rcu_txn_internal.h
@@ -42,9 +42,88 @@ static inline void __rcu_txn_group_commit(struct rcu_txn_group *group)
static inline void __rcu_txn_record_install(struct rcu_txn_group *group,
struct rcu_txn_record *r)
{
+ /*
+ * Installing requires a non-zero tag within RCU_TXN_TAG_MASK, none of
+ * whose bits is set in the record's address. Under a zero tag every
+ * plain value passes rcu_txn_is_proxy(), and readers would resolve live
+ * values as proxies. With a tag bit already set in the address, the OR
+ * changes nothing and __rcu_txn_untag() subtracts to a wrong address.
+ */
+ RCU_TXN_WARN_ON(!r->tag || (r->tag & ~RCU_TXN_TAG_MASK));
+ RCU_TXN_WARN_ON((unsigned long)r & r->tag);
r->group = group;
/* Store-release of slot pairs with rcu_dereference(). */
rcu_assign_pointer(*r->slot, (void *)((unsigned long)r | r->tag));
}
+/*
+ * The install and settle checks of CONFIG_DEBUG_RCU_TXN: see
+ * <linux/rcu_txn.h>.
+ */
+#ifdef CONFIG_DEBUG_RCU_TXN
+
+/* At settle, @r's slot must still hold the proxy this writer installed. */
+static inline void __rcu_txn_excl_slot_ours(struct rcu_txn_record *r)
+{
+ void *want = (void *)((unsigned long)r | r->tag);
+ void *v = rcu_access_pointer(*r->slot);
+
+ WARN(v != want,
+ "rcu_txn: slot %p holds %p at settle, expected our installed proxy %p: a concurrent writer overwrote it\n",
+ r->slot, v, want);
+}
+
+/*
+ * At the multi-record install, @r's slot must hold no proxy and must still
+ * hold the recorded old.
+ *
+ * The value check catches a racing writer that ran a complete transaction on
+ * the slot -- install, flip, settle -- between this transaction's record() and
+ * its install. That writer leaves a plain value behind, so the slot looks
+ * free; installing over it would publish a stale old to readers, then settle
+ * over the other writer's committed value.
+ *
+ * A correct single-writer program never trips it: this transaction has not
+ * written the slot yet, the writer's earlier transactions settled before
+ * returning, and chaining keeps the committed old in ptr[0].
+ */
+static inline void __rcu_txn_excl_slot_installable(struct rcu_txn_record *r)
+{
+ __rcu_txn_excl_slot_free(r->slot, r->tag, "install");
+ __rcu_txn_excl_slot_unchanged(r, "install");
+}
+
+/*
+ * Records must target pairwise-distinct slots: see rcu_txn_record(). They are
+ * in record order, hence the pairwise scan.
+ */
+static inline void
+__rcu_txn_assert_distinct_slots(const struct rcu_txn_record *records,
+ unsigned int nr)
+{
+ unsigned int i, j;
+
+ for (i = 1; i < nr; i++)
+ for (j = 0; j < i; j++)
+ WARN_ON(records[i].slot == records[j].slot);
+}
+
+#else /* !CONFIG_DEBUG_RCU_TXN */
+
+static inline void __rcu_txn_excl_slot_ours(struct rcu_txn_record *r)
+{
+}
+
+static inline void __rcu_txn_excl_slot_installable(struct rcu_txn_record *r)
+{
+}
+
+static inline void
+__rcu_txn_assert_distinct_slots(const struct rcu_txn_record *records,
+ unsigned int nr)
+{
+}
+
+#endif /* CONFIG_DEBUG_RCU_TXN */
+
#endif /* _LIB_RCU_TXN_INTERNAL_H */
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index 3111fc537ff3..3f005cdee4d0 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -9,6 +9,7 @@
#include <linux/bug.h>
#include <linux/rcu_txn.h>
#include <linux/rcupdate.h>
+#include <linux/sched/task.h>
#include <linux/string.h>
#include "../rcu_txn_internal.h" /* the steps of a commit */
@@ -577,6 +578,341 @@ static void rcu_txn_test_bloom_grow(struct kunit *test)
KUNIT_EXPECT_TRUE(test, all_settled(slots, NR));
}
+/*
+ * CONFIG_DEBUG_RCU_TXN
+ *
+ * Every case below is an embedder bug that the debug checks report. The
+ * writer/writer races are staged deterministically: the other writer's proxy is
+ * installed in the slot by hand, as a real race would leave it.
+ */
+
+static void need_debug(struct kunit *test)
+{
+ if (!IS_ENABLED(CONFIG_DEBUG_RCU_TXN))
+ kunit_skip(test, "CONFIG_DEBUG_RCU_TXN is off");
+}
+
+#ifdef CONFIG_DEBUG_RCU_TXN
+static void set_owner(struct rcu_txn *t, struct task_struct *task)
+{
+ t->owner = task;
+}
+#else
+static void set_owner(struct rcu_txn *t, struct task_struct *task)
+{
+}
+#endif
+
+/*
+ * Install a proxy into @slot by hand, as a writer stopped between its install
+ * and its flip leaves it. commit() runs install, flip and settle back to back,
+ * so one task cannot stop a transaction there.
+ */
+static void install_by_hand(struct rcu_txn_group *group,
+ struct rcu_txn_record *rec, void __rcu **slot,
+ void *old_ptr, void *new_ptr)
+{
+ __rcu_txn_group_init(group);
+ __rcu_txn_record_set(rec, slot, old_ptr, new_ptr, TAG);
+ __rcu_txn_record_install(group, rec);
+}
+
+/* A well-formed single-writer transaction trips nothing. */
+static void rcu_txn_test_debug_control(struct kunit *test)
+{
+ void __rcu *a = RCU_INITIALIZER(T(0));
+ void __rcu *b = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+ unsigned int i;
+
+ need_debug(test);
+ kunit_warning_suppress(test) {
+ for (i = 0; i < 100; i++) {
+ void *to = T(1 + i), *from = T(i);
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &a, from, to, TAG);
+ rcu_txn_record_chain(&txn, &b,
+ rcu_txn_load(&txn, &b, TAG), to,
+ TAG);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ }
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 0);
+ }
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(a), T(100));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(b), T(100));
+}
+
+/* A handle driven by a task other than the one that initialized it. */
+static void rcu_txn_test_debug_owner(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ set_owner(&txn, &init_task); /* as if another task initialized it */
+ kunit_warning_suppress(test) {
+ rcu_txn_record(&txn, &slot, T(0), T(1), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ set_owner(&txn, current);
+ rcu_txn_cancel(&txn);
+}
+
+/*
+ * A handle got ready by one task and adopted by the one that drives it. Not
+ * with records in it: that is a transaction changing hands mid-flight.
+ */
+static void rcu_txn_test_debug_adopt(struct kunit *test)
+{
+ void __rcu *a = RCU_INITIALIZER(T(0));
+ void __rcu *b = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ set_owner(&txn, &init_task); /* as if another task got it ready */
+ kunit_warning_suppress(test) {
+ rcu_txn_adopt(&txn);
+ rcu_txn_record(&txn, &a, T(0), T(1), TAG);
+ rcu_txn_record(&txn, &b, T(0), T(1), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 0);
+ rcu_txn_adopt(&txn);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ }
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(a), T(1));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(b), T(1));
+}
+
+/*
+ * A second writer records a slot where the first has already installed a
+ * proxy. This is the interleaving where the violation is visible: the first
+ * writer is between its install and its flip, so its proxy is sitting in the
+ * slot.
+ */
+static void rcu_txn_test_debug_record_over_proxy(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+ struct rcu_txn txn;
+
+ need_debug(test);
+ install_by_hand(&group, &rec, &slot, T(0), T(1));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ kunit_warning_suppress(test) {
+ rcu_txn_record(&txn, &slot, T(1), T(2), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+}
+
+/*
+ * A second writer installs onto a slot where the first has already installed a
+ * proxy. It recorded the slot while it was still plain, so record() saw
+ * nothing wrong; the overlap only becomes visible at its own install, which
+ * finds a proxy there and not the recorded old.
+ */
+static void rcu_txn_test_debug_install_over_proxy(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_record rec, other;
+ struct rcu_txn_group group;
+
+ need_debug(test);
+ /* Recorded while the slot is still plain. */
+ __rcu_txn_record_set(&rec, &slot, T(0), T(2), TAG);
+ install_by_hand(&group, &other, &slot, T(0), T(1));
+ kunit_warning_suppress(test) {
+ __rcu_txn_excl_slot_installable(&rec);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 2);
+ }
+}
+
+/*
+ * A peer ran a complete transaction on the slot between our record() and our
+ * commit(). It leaves a plain value behind, so the slot looks free; the
+ * witness is that it no longer holds the recorded old. The peer's committed
+ * write is staged by a plain store, which is what it leaves behind.
+ */
+static void rcu_txn_test_debug_install_over_settled(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_record rec;
+
+ need_debug(test);
+ __rcu_txn_record_set(&rec, &slot, T(0), T(1), TAG);
+ rcu_assign_pointer(slot, T(2)); /* a peer committed 0 -> 2 */
+ kunit_warning_suppress(test) {
+ __rcu_txn_excl_slot_installable(&rec);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+}
+
+/* Our proxy is gone by the time we settle: someone overwrote it. */
+static void rcu_txn_test_debug_settle_clobbered(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+
+ need_debug(test);
+ /* Stands for our install. */
+ install_by_hand(&group, &rec, &slot, T(0), T(1));
+ rcu_assign_pointer(slot, T(2)); /* a peer overwrites it */
+ kunit_warning_suppress(test) {
+ __rcu_txn_excl_slot_ours(&rec);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+}
+
+/*
+ * The single-record commit stores blind, with no proxy and no install, so its
+ * only witness is the slot's value: it must still hold the recorded old.
+ */
+static void rcu_txn_test_debug_single_record_changed(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record(&txn, &slot, T(0), T(1), TAG);
+ rcu_assign_pointer(slot, T(2)); /* a peer changed it under us */
+ kunit_warning_suppress(test) {
+ rcu_txn_commit(&txn);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+}
+
+/*
+ * One slot recorded twice, not adjacently: the commit would install both
+ * records and settle them in record order, dropping the earlier edit.
+ */
+static void rcu_txn_test_debug_duplicate_slot(struct kunit *test)
+{
+ void __rcu *a = RCU_INITIALIZER(T(0));
+ void __rcu *b = RCU_INITIALIZER(T(1));
+ struct rcu_txn_record recs[3];
+
+ need_debug(test);
+ __rcu_txn_record_set(&recs[0], &a, T(0), T(10), TAG);
+ __rcu_txn_record_set(&recs[1], &b, T(1), T(11), TAG);
+ __rcu_txn_record_set(&recs[2], &a, T(10), T(12), TAG);
+ kunit_warning_suppress(test) {
+ __rcu_txn_assert_distinct_slots(recs, ARRAY_SIZE(recs));
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+}
+
+/*
+ * No value stored in a transacted slot may carry the slot's tag: a reader
+ * would take it for a proxy. Caught at the record that introduces it, on the
+ * new value and on the old.
+ */
+static void rcu_txn_test_debug_tagged_value(struct kunit *test)
+{
+ void *tagged = (void *)((unsigned long)T(1) | TAG);
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ kunit_warning_suppress(test) {
+ rcu_txn_record(&txn, &slot, T(0), tagged, TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ kunit_warning_suppress(test) {
+ rcu_txn_record(&txn, &slot, tagged, T(2), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+}
+
+/* A proxy cannot be installed under a zero tag: nothing would mark it. */
+static void rcu_txn_test_debug_install_zero_tag(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+
+ need_debug(test);
+ __rcu_txn_group_init(&group);
+ __rcu_txn_record_set(&rec, &slot, T(0), T(1), 0);
+ kunit_warning_suppress(test) {
+ __rcu_txn_record_install(&group, &rec);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+}
+
+/*
+ * On a handle declared disjoint, a load or a record of a slot already recorded:
+ * the load would miss the pending value, and the record would put a second
+ * proxy on the slot.
+ */
+static void rcu_txn_test_debug_disjoint(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_declare_disjoint(&txn);
+ rcu_txn_record_chain(&txn, &slot, T(0), T(1), TAG);
+ kunit_warning_suppress(test) {
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_load(&txn, &slot, TAG), T(0));
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ kunit_warning_suppress(test) {
+ rcu_txn_record_chain(&txn, &slot, T(0), T(2), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+}
+
+/*
+ * record_chain() given an old value that is not the pending one: its caller
+ * computed the new value from a stale read.
+ */
+static void rcu_txn_test_debug_stale_chain(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn txn;
+
+ need_debug(test);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_record_chain(&txn, &slot, T(0), T(1), TAG);
+ kunit_warning_suppress(test) {
+ rcu_txn_record_chain(&txn, &slot, T(0), T(2), TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+}
+
+/* load() finds another writer's proxy in a slot it has not recorded. */
+static void rcu_txn_test_debug_load_proxy(struct kunit *test)
+{
+ void __rcu *slot = RCU_INITIALIZER(T(0));
+ struct rcu_txn_group group;
+ struct rcu_txn_record rec;
+ struct rcu_txn txn;
+
+ need_debug(test);
+ install_by_hand(&group, &rec, &slot, T(0), T(1));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ kunit_warning_suppress(test) {
+ rcu_txn_load(&txn, &slot, TAG);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -596,6 +932,20 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_bloom_rebuild),
KUNIT_CASE(rcu_txn_test_bloom_arming),
KUNIT_CASE(rcu_txn_test_bloom_grow),
+ KUNIT_CASE(rcu_txn_test_debug_control),
+ KUNIT_CASE(rcu_txn_test_debug_owner),
+ KUNIT_CASE(rcu_txn_test_debug_adopt),
+ KUNIT_CASE(rcu_txn_test_debug_record_over_proxy),
+ KUNIT_CASE(rcu_txn_test_debug_install_over_proxy),
+ KUNIT_CASE(rcu_txn_test_debug_install_over_settled),
+ KUNIT_CASE(rcu_txn_test_debug_settle_clobbered),
+ KUNIT_CASE(rcu_txn_test_debug_single_record_changed),
+ KUNIT_CASE(rcu_txn_test_debug_duplicate_slot),
+ KUNIT_CASE(rcu_txn_test_debug_tagged_value),
+ KUNIT_CASE(rcu_txn_test_debug_install_zero_tag),
+ KUNIT_CASE(rcu_txn_test_debug_disjoint),
+ KUNIT_CASE(rcu_txn_test_debug_stale_chain),
+ KUNIT_CASE(rcu_txn_test_debug_load_proxy),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 05/12] rcu_txn: Add an hlist edited through transactions
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (3 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 04/12] rcu_txn: Add CONFIG_DEBUG_RCU_TXN Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 06/12] rcu_txn: Add a doubly-linked list coherent in both directions Mathieu Desnoyers
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Add rcu_txn_hlist, a list with a single-pointer head shaped like hlist,
whose edits are recorded into an RCU pseudo-transaction. Its commit
makes all of them, on one list or on many, visible to readers at once:
an entry can move between two hash buckets, through a copy, and be in
exactly one of them at any instant.
Readers walk forward only and never load pprev, so a single update has
one reader-visible edge. The _rcu forms make such an update outside of
any transaction, as hlist_add_head_rcu() does: they publish that edge
with rcu_assign_pointer(), then write pprev with a plain store, which no
other writer reads in between. Every transaction settles its slots
before its commit returns, and the writers are excluded, so the slots
they read hold no proxy. They allocate nothing and cannot fail.
The _prepare forms record into a transaction of the caller's, and
transact pprev too, as a slot of its own. A later operation of the same
transaction then loads its neighbour's pprev as the transaction leaves
it, which lets edits of one bucket compose, and a transaction that fails
or is cancelled leaves every pprev as it was. An update with a
successor thus records two edges there, and takes the descriptor path at
commit even alone.
A writer that excludes the others walks a chain with the same iterators
as readers, outside of an RCU read-side critical section, by passing
them a lockdep condition for that exclusion, such as lockdep_is_held()
of its lock, as with hlist_for_each_entry_rcu(). rcu_txn_hlist_empty()
takes the same condition: unlike hlist_empty(), it may have to resolve a
proxy, which reads the descriptor of a commit in flight. Add
__rcu_txn_cond() to the engine, which picks that optional condition.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
include/linux/rcu_txn.h | 7 +
include/linux/rcu_txn_hlist.h | 483 ++++++++++++++++++++++++++++++++++
lib/tests/rcu_txn_kunit.c | 396 ++++++++++++++++++++++++++++
3 files changed, 886 insertions(+)
create mode 100644 include/linux/rcu_txn_hlist.h
diff --git a/include/linux/rcu_txn.h b/include/linux/rcu_txn.h
index 2946c63fb044..b9ee242a01ce 100644
--- a/include/linux/rcu_txn.h
+++ b/include/linux/rcu_txn.h
@@ -363,6 +363,13 @@ static inline void *rcu_txn_resolve(void *v, unsigned long tag)
*/
#define rcu_txn_dereference(p, tag) rcu_txn_dereference_check(p, tag, 0)
+/*
+ * The optional lockdep condition of an iterator, as for
+ * list_for_each_entry_rcu(): the condition if one was given, else 0.
+ */
+#define __rcu_txn_cond(cond...) ___rcu_txn_cond(0, ## cond, 0)
+#define ___rcu_txn_cond(zero, cond, extra...) ((zero) || (cond))
+
/**
* rcu_txn_slot() - the engine's view of an RCU-protected pointer's address
* @pp: address of an __rcu pointer, of any type
diff --git a/include/linux/rcu_txn_hlist.h b/include/linux/rcu_txn_hlist.h
new file mode 100644
index 000000000000..aecb1e4b94dc
--- /dev/null
+++ b/include/linux/rcu_txn_hlist.h
@@ -0,0 +1,483 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+/*
+ * RCU hlist edited through single-writer RCU pseudo-transactions
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+#ifndef _LINUX_RCU_TXN_HLIST_H
+#define _LINUX_RCU_TXN_HLIST_H
+
+/*
+ * rcu_txn_hlist: an RCU list with a single-pointer head, shaped like hlist, for
+ * a single writer (writers mutually excluded, as with hlist_add_head_rcu()).
+ * Its edits are recorded into a transaction of <linux/rcu_txn.h>, whose commit
+ * makes all of them, on one bucket or on many, visible to readers at once.
+ *
+ * Forward-only readers, and pprev
+ * -------------------------------
+ * A node's backward link is a pointer to the slot that points to it
+ * (&prev->next or &head->first), not to a node, so no operation has a head
+ * special case.
+ *
+ * Node: { struct rcu_txn_hlist_node *next; (node ptr; reader-visible)
+ * struct rcu_txn_hlist_node **pprev; (slot ptr; writer-only) }
+ *
+ * Readers walk forward only and never load pprev, so each operation has exactly
+ * one reader-visible edge: the next slot it re-points.
+ *
+ * Single updates and transactions
+ * -------------------------------
+ * The _rcu forms make a single update, outside of any transaction: they publish
+ * its one edge with rcu_assign_pointer(), as hlist_add_head_rcu() does, then
+ * write pprev with a plain store. The writers are mutually excluded, and every
+ * transaction settles its slots before its commit returns, so the slots they
+ * read hold no proxy, and no other writer reads pprev in between. A single
+ * update allocates nothing and cannot fail.
+ *
+ * The _prepare forms record into a transaction of the caller's, which may
+ * compose several updates, and which may fail or be cancelled. They transact
+ * pprev too, as a slot of its own. A later _prepare of the same transaction
+ * then loads its neighbour's pprev as the transaction leaves it, which is what
+ * makes edits of one bucket compose (see rcu_txn_hlist_del_prepare()), and a
+ * transaction that fails or is cancelled leaves every pprev as it was. An
+ * update with a successor records two edges, and so takes the descriptor path
+ * at commit even alone: a single update is cheapest in its _rcu form.
+ *
+ * Composing edits of one hlist
+ * ----------------------------
+ * Operations whose neighbourhoods touch may share a transaction: each _prepare
+ * loads the slots it reads, next/first and pprev, as the transaction leaves
+ * them, and chains a second write to a slot onto the first. Two rules remain
+ * for the caller:
+ *
+ * - Look up the nodes before editing, rather than by traversing the list in
+ * the middle of the transaction.
+ *
+ * - Every node an operation is anchored on must still be in the list as the
+ * transaction leaves it. A single-writer delete leaves no deletion mark:
+ * nothing in a deleted anchor distinguishes it from a live one.
+ *
+ * Example: 1 -> 2 -> 3, del_prepare(2) then add_behind_prepare(9, 2). The
+ * delete records {&1->next: 2 -> 3} and {&3->pprev: &2->next -> &1->next}.
+ * The add then records {&2->next: 3 -> 9}, and chains 3's pprev from
+ * &2->next, where the delete left it at &1->next. CONFIG_DEBUG_RCU_TXN
+ * warns there, in rcu_txn_record_chain(). The commit still returns 0, and
+ * the committed chain is 1 -> 3: node 9 is reachable only through the
+ * removed node 2, and 3->pprev points to a slot that no live node holds. A
+ * later del_rcu(3) follows that pprev, commits into the unreachable node 9
+ * and succeeds, while 1->next still points to 3, which the caller then
+ * frees.
+ *
+ * add_before and replace anchored on a deleted node chain onto the delete's
+ * record of 1->next instead, where CONFIG_DEBUG_RCU_TXN warns as well, and
+ * replace warns again on 3's pprev. The commit links the deleted node, or
+ * its replacement, back into the chain.
+ *
+ * Proxy tag
+ * ---------
+ * The slots, head->first, node->next and node->pprev, are transacted under
+ * RCU_TXN_HLIST_TAG, bit 0. No live value a slot holds carries it: a slot
+ * holds NULL, the address of a node, or, for pprev, the address of a pointer.
+ *
+ * Read side
+ * ---------
+ * Iterate within an RCU read-side critical section, through
+ * rcu_txn_hlist_first_rcu() / _next_rcu() or the macros below. Never use what
+ * rcu_dereference() returns for node->next directly: a slot may transiently
+ * hold a tagged proxy.
+ *
+ * Write side
+ * ----------
+ * Writers must be mutually excluded. A removed node keeps its own next, so a
+ * reader standing on it still reaches the live chain; the caller reclaims it
+ * after a grace period, as with hlist_del_rcu().
+ *
+ * The add forms take a node that no reader can reach, and build its next with
+ * a plain store, so they cannot move a node. To move an entry to another
+ * bucket, delete its node and add a copy's, in one transaction.
+ *
+ * A writer that excludes the others walks the chain with the same iterators,
+ * outside of an RCU read-side critical section, if it passes them a lockdep
+ * condition for that exclusion, such as lockdep_is_held() of its lock, as with
+ * hlist_for_each_entry_rcu(); rcu_txn_hlist_first_check(), _next_check() and
+ * rcu_txn_hlist_empty() take one too.
+ */
+
+#include <linux/build_bug.h>
+#include <linux/container_of.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcupdate.h>
+
+/* Proxy tag of the slots, head->first, node->next and node->pprev. */
+#define RCU_TXN_HLIST_TAG 1UL
+
+/*
+ * Under a zero tag every plain value would pass for a proxy. A tag beyond
+ * RCU_TXN_TAG_MASK would overlap the address of a record.
+ */
+static_assert(RCU_TXN_HLIST_TAG != 0 &&
+ (RCU_TXN_HLIST_TAG & ~RCU_TXN_TAG_MASK) == 0);
+
+struct rcu_txn_hlist_node {
+ struct rcu_txn_hlist_node __rcu *next; /* reader-visible */
+ struct rcu_txn_hlist_node __rcu * __rcu *pprev; /* writer-only */
+};
+
+/* A bucket head is one pointer, like struct hlist_head. */
+struct rcu_txn_hlist_head {
+ struct rcu_txn_hlist_node __rcu *first;
+};
+
+#define RCU_TXN_HLIST_HEAD_INIT { .first = NULL }
+
+static inline void rcu_txn_hlist_init(struct rcu_txn_hlist_head *head)
+{
+ RCU_INIT_POINTER(head->first, NULL);
+}
+
+/*
+ * First node and forward step, within an RCU read-side critical section or
+ * where @c, a lockdep condition, holds.
+ */
+#define rcu_txn_hlist_first_check(head, c) \
+ rcu_txn_dereference_check((head)->first, RCU_TXN_HLIST_TAG, c)
+
+#define rcu_txn_hlist_next_check(node, c) \
+ rcu_txn_dereference_check((node)->next, RCU_TXN_HLIST_TAG, c)
+
+/* First node and forward step, for use within an RCU read-side section. */
+static inline struct rcu_txn_hlist_node *
+rcu_txn_hlist_first_rcu(struct rcu_txn_hlist_head *head)
+{
+ return rcu_txn_dereference(head->first, RCU_TXN_HLIST_TAG);
+}
+
+static inline struct rcu_txn_hlist_node *
+rcu_txn_hlist_next_rcu(struct rcu_txn_hlist_node *node)
+{
+ return rcu_txn_dereference(node->next, RCU_TXN_HLIST_TAG);
+}
+
+/*
+ * True iff the chain is empty, within an RCU read-side critical section or
+ * where @cond, an optional lockdep condition, holds.
+ *
+ * Unlike hlist_empty(), which is a plain load, this needs one or the other:
+ * while a commit of two records or more is in flight on head->first, the slot
+ * holds a proxy, and telling an empty chain from another then reads the
+ * transaction's descriptor, which is freed after a grace period. A writer that
+ * excludes the others finds no proxy there, and passes a lockdep condition for
+ * that exclusion as @cond.
+ */
+#define rcu_txn_hlist_empty(head, cond...) \
+ (!rcu_txn_hlist_first_check(head, __rcu_txn_cond(cond)))
+
+/*
+ * Write-side read of a head->first or node->next slot: this transaction's
+ * pending write to it if it recorded one, else the slot's committed value. A
+ * typed wrapper over rcu_txn_load(). The _prepare forms read every forward
+ * link through this, which is what lets edits compose on one bucket (see
+ * rcu_txn_hlist_del_prepare()). Readers use rcu_txn_hlist_first_rcu() /
+ * _next_rcu() instead.
+ */
+static inline struct rcu_txn_hlist_node *
+rcu_txn_hlist_pending_next(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node __rcu **slot)
+{
+ return rcu_txn_load(txn, rcu_txn_slot(slot), RCU_TXN_HLIST_TAG);
+}
+
+/* The pprev counterpart of rcu_txn_hlist_pending_next(). */
+static inline struct rcu_txn_hlist_node __rcu **
+__rcu_txn_hlist_pending_pprev(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *node)
+{
+ return rcu_txn_load(txn, rcu_txn_slot(&node->pprev), RCU_TXN_HLIST_TAG);
+}
+
+/*
+ * Record that @node's pprev, @old as the transaction leaves it so far, becomes
+ * @new.
+ */
+static inline void
+__rcu_txn_hlist_record_pprev(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *node,
+ struct rcu_txn_hlist_node __rcu **old,
+ struct rcu_txn_hlist_node __rcu **new)
+{
+ rcu_txn_record_chain(txn, rcu_txn_slot(&node->pprev), old, new,
+ RCU_TXN_HLIST_TAG);
+}
+
+/**
+ * rcu_txn_hlist_insert_at_slot_prepare() - record an insertion at a slot
+ * @txn: the transaction
+ * @new: the node to insert
+ * @slot: the address of head->first, to insert at the head, or of pos->next,
+ * to insert behind @pos
+ * @succ: the node @slot points to as this transaction leaves it, or NULL
+ *
+ * Record the edge that makes @slot point to @new, without committing. When
+ * @succ is non-NULL, its pprev becomes the address of new->next.
+ *
+ * @succ must be read with rcu_txn_hlist_pending_next(): @new is built pointing
+ * at it, and a value read directly would link @new behind a node that an
+ * earlier edit of the same transaction displaced. The add_head and add_behind
+ * forms below do so.
+ */
+static inline void
+rcu_txn_hlist_insert_at_slot_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node __rcu **slot,
+ struct rcu_txn_hlist_node *succ)
+{
+ /* Build the fresh node's links before it becomes reachable. */
+ RCU_INIT_POINTER(new->next, succ);
+ RCU_INIT_POINTER(new->pprev, slot);
+
+ rcu_txn_record_chain(txn, rcu_txn_slot(slot), succ, new,
+ RCU_TXN_HLIST_TAG);
+ if (succ)
+ __rcu_txn_hlist_record_pprev(txn, succ, slot, &new->next);
+}
+
+/**
+ * rcu_txn_hlist_add_head_prepare() - record an insertion at the head
+ * @txn: the transaction
+ * @new: the node to insert
+ * @head: the list
+ *
+ * See rcu_txn_hlist_insert_at_slot_prepare().
+ */
+static inline void
+rcu_txn_hlist_add_head_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_head *head)
+{
+ struct rcu_txn_hlist_node *first =
+ rcu_txn_hlist_pending_next(txn, &head->first);
+
+ rcu_txn_hlist_insert_at_slot_prepare(txn, new, &head->first, first);
+}
+
+/**
+ * rcu_txn_hlist_add_behind_prepare() - record an insertion behind a node
+ * @txn: the transaction
+ * @new: the node to insert
+ * @pos: the node to insert it behind
+ *
+ * See rcu_txn_hlist_insert_at_slot_prepare().
+ */
+static inline void
+rcu_txn_hlist_add_behind_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node *pos)
+{
+ struct rcu_txn_hlist_node *next =
+ rcu_txn_hlist_pending_next(txn, &pos->next);
+
+ rcu_txn_hlist_insert_at_slot_prepare(txn, new, &pos->next, next);
+}
+
+/**
+ * rcu_txn_hlist_add_before_prepare() - record an insertion before a node
+ * @txn: the transaction
+ * @new: the node to insert
+ * @pos: the node to insert it before
+ *
+ * @pos's pprev points to the slot to re-point.
+ */
+static inline void
+rcu_txn_hlist_add_before_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node *pos)
+{
+ struct rcu_txn_hlist_node __rcu **slot =
+ __rcu_txn_hlist_pending_pprev(txn, pos);
+
+ RCU_INIT_POINTER(new->next, pos);
+ RCU_INIT_POINTER(new->pprev, slot);
+
+ rcu_txn_record_chain(txn, rcu_txn_slot(slot), pos, new,
+ RCU_TXN_HLIST_TAG);
+ __rcu_txn_hlist_record_pprev(txn, pos, slot, &new->next);
+}
+
+/**
+ * rcu_txn_hlist_del_prepare() - record the unlink of a node
+ * @txn: the transaction
+ * @elem: the node to unlink
+ *
+ * @elem's own next and pprev are left as they are, so a reader standing on it
+ * still reaches the live chain; the caller frees @elem a grace period after the
+ * commit.
+ *
+ * Composing on one bucket, by example: delete the adjacent A and B from head ->
+ * A -> B -> C in one transaction, a record being written {slot: old -> new}.
+ * del(A) records {head->first: A -> B}, and that B's pprev now points to
+ * head->first. del(B) loads B's pprev as the transaction leaves it and finds
+ * head->first, not the stale A->next, so it records against head->first, finds
+ * it recorded, and chains: {head->first: A -> C}. It then records that C's
+ * pprev points to head->first.
+ *
+ * Had del(B) read B->pprev directly, it would have found A->next, a slot inside
+ * the node that del(A) unlinks, and head->first would be left pointing to the
+ * deleted B.
+ */
+static inline void rcu_txn_hlist_del_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *elem)
+{
+ struct rcu_txn_hlist_node *next =
+ rcu_txn_hlist_pending_next(txn, &elem->next);
+ struct rcu_txn_hlist_node __rcu **ppv =
+ __rcu_txn_hlist_pending_pprev(txn, elem);
+
+ rcu_txn_record_chain(txn, rcu_txn_slot(ppv), elem, next,
+ RCU_TXN_HLIST_TAG);
+ if (next)
+ __rcu_txn_hlist_record_pprev(txn, next, &elem->next, ppv);
+}
+
+/**
+ * rcu_txn_hlist_replace_prepare() - record the replacement of a node
+ * @txn: the transaction
+ * @old: the node to replace
+ * @new: the node that replaces it
+ *
+ * @new takes @old's slot and successor, and @old keeps its own next for readers
+ * standing on it. Arguments are (old, new), as hlist_replace_rcu().
+ */
+static inline void rcu_txn_hlist_replace_prepare(struct rcu_txn *txn,
+ struct rcu_txn_hlist_node *old,
+ struct rcu_txn_hlist_node *new)
+{
+ struct rcu_txn_hlist_node *next =
+ rcu_txn_hlist_pending_next(txn, &old->next);
+ struct rcu_txn_hlist_node __rcu **ppv =
+ __rcu_txn_hlist_pending_pprev(txn, old);
+
+ RCU_INIT_POINTER(new->next, next);
+ RCU_INIT_POINTER(new->pprev, ppv);
+
+ rcu_txn_record_chain(txn, rcu_txn_slot(ppv), old, new,
+ RCU_TXN_HLIST_TAG);
+ if (next)
+ __rcu_txn_hlist_record_pprev(txn, next, &old->next,
+ &new->next);
+}
+
+/*
+ * The _rcu forms: each makes a single update, outside of any transaction. See
+ * "Single updates and transactions" above.
+ */
+
+/*
+ * Read a head->first or node->next slot outside of any transaction.
+ * CONFIG_DEBUG_RCU_TXN checks that it holds no proxy: one would be another
+ * writer's, in the middle of its commit.
+ */
+static inline struct rcu_txn_hlist_node *
+__rcu_txn_hlist_load(struct rcu_txn_hlist_node __rcu **slot)
+{
+ struct rcu_txn_hlist_node *v = rcu_dereference_protected(*slot, 1);
+
+ RCU_TXN_WARN_ON(__rcu_txn_holds_proxy(v, RCU_TXN_HLIST_TAG));
+ return v;
+}
+
+/* A node's pprev, which only the writers access, outside of any transaction. */
+static inline struct rcu_txn_hlist_node __rcu **
+__rcu_txn_hlist_pprev(struct rcu_txn_hlist_node *node)
+{
+ return rcu_dereference_protected(node->pprev, 1);
+}
+
+static inline void
+__rcu_txn_hlist_insert_at_slot_rcu(struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node __rcu **slot)
+{
+ struct rcu_txn_hlist_node *succ = __rcu_txn_hlist_load(slot);
+
+ RCU_INIT_POINTER(new->next, succ);
+ RCU_INIT_POINTER(new->pprev, slot);
+ rcu_assign_pointer(*slot, new);
+ if (succ)
+ RCU_INIT_POINTER(succ->pprev, &new->next);
+}
+
+static inline void rcu_txn_hlist_add_head_rcu(struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_head *head)
+{
+ __rcu_txn_hlist_insert_at_slot_rcu(new, &head->first);
+}
+
+static inline void rcu_txn_hlist_add_behind_rcu(struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node *pos)
+{
+ __rcu_txn_hlist_insert_at_slot_rcu(new, &pos->next);
+}
+
+static inline void rcu_txn_hlist_add_before_rcu(struct rcu_txn_hlist_node *new,
+ struct rcu_txn_hlist_node *pos)
+{
+ struct rcu_txn_hlist_node __rcu **slot = __rcu_txn_hlist_pprev(pos);
+
+ RCU_INIT_POINTER(new->next, pos);
+ RCU_INIT_POINTER(new->pprev, slot);
+ rcu_assign_pointer(*slot, new);
+ RCU_INIT_POINTER(pos->pprev, &new->next);
+}
+
+static inline void rcu_txn_hlist_del_rcu(struct rcu_txn_hlist_node *elem)
+{
+ struct rcu_txn_hlist_node __rcu **ppv = __rcu_txn_hlist_pprev(elem);
+ struct rcu_txn_hlist_node *next = __rcu_txn_hlist_load(&elem->next);
+
+ rcu_assign_pointer(*ppv, next);
+ if (next)
+ RCU_INIT_POINTER(next->pprev, ppv);
+}
+
+static inline void rcu_txn_hlist_replace_rcu(struct rcu_txn_hlist_node *old,
+ struct rcu_txn_hlist_node *new)
+{
+ struct rcu_txn_hlist_node __rcu **ppv = __rcu_txn_hlist_pprev(old);
+ struct rcu_txn_hlist_node *next = __rcu_txn_hlist_load(&old->next);
+
+ RCU_INIT_POINTER(new->next, next);
+ RCU_INIT_POINTER(new->pprev, ppv);
+ rcu_assign_pointer(*ppv, new);
+ if (next)
+ RCU_INIT_POINTER(next->pprev, &new->next);
+}
+
+#define rcu_txn_hlist_entry(ptr, type, member) \
+ container_of(ptr, type, member)
+
+#define rcu_txn_hlist_entry_safe(ptr, type, member) \
+({ \
+ typeof(ptr) ____ptr = (ptr); \
+ \
+ ____ptr ? rcu_txn_hlist_entry(____ptr, type, member) : NULL; \
+})
+
+/*
+ * Iterate forward over a bucket, within an RCU read-side critical section, or
+ * where @cond, an optional lockdep condition, holds.
+ */
+#define rcu_txn_hlist_for_each_rcu(pos, head, cond...) \
+ for (pos = rcu_txn_hlist_first_check(head, __rcu_txn_cond(cond)); \
+ pos; \
+ pos = rcu_txn_hlist_next_check(pos, __rcu_txn_cond(cond)))
+
+#define rcu_txn_hlist_for_each_entry_rcu(pos, head, member, cond...) \
+ for (pos = rcu_txn_hlist_entry_safe( \
+ rcu_txn_hlist_first_check(head, __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member); \
+ pos; \
+ pos = rcu_txn_hlist_entry_safe( \
+ rcu_txn_hlist_next_check(&(pos)->member, \
+ __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member))
+
+#endif /* _LINUX_RCU_TXN_HLIST_H */
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index 3f005cdee4d0..13e4bbae1053 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -8,6 +8,7 @@
#include <kunit/test.h>
#include <linux/bug.h>
#include <linux/rcu_txn.h>
+#include <linux/rcu_txn_hlist.h>
#include <linux/rcupdate.h>
#include <linux/sched/task.h>
#include <linux/string.h>
@@ -913,6 +914,387 @@ static void rcu_txn_test_debug_load_proxy(struct kunit *test)
rcu_txn_cancel(&txn);
}
+/*
+ * <linux/rcu_txn_hlist.h>
+ */
+
+struct hn {
+ struct rcu_txn_hlist_node node;
+ int key;
+};
+
+/* Freed with the test: a removed node stays valid for any reader. */
+static struct rcu_txn_hlist_node *hn_new(struct kunit *test, int key)
+{
+ struct hn *n = kunit_kzalloc(test, sizeof(*n), GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_NULL(test, n);
+ n->key = key;
+ return &n->node;
+}
+
+/* Build the list @keys by prepending in reverse. */
+static void hlist_build(struct kunit *test, struct rcu_txn_hlist_head *head,
+ const int *keys, int nr)
+{
+ int i;
+
+ for (i = nr - 1; i >= 0; i--)
+ rcu_txn_hlist_add_head_rcu(hn_new(test, keys[i]), head);
+}
+
+/* The test is the chain's only writer, and owns every node until it ends. */
+static struct rcu_txn_hlist_node *hlist_find(struct rcu_txn_hlist_head *head,
+ int key)
+{
+ struct hn *n;
+
+ rcu_txn_hlist_for_each_entry_rcu(n, head, node, true)
+ if (n->key == key)
+ return &n->node;
+ return NULL;
+}
+
+/*
+ * The forward walk gives @want, and each node's pprev points to the slot that
+ * points to it: &head->first for the first node, &prev->next thereafter. A
+ * torn pprev is invisible to the forward walk but breaks the next edit, hence
+ * the separate check.
+ */
+static void hlist_expect(struct kunit *test, struct rcu_txn_hlist_head *head,
+ const int *want, int nr)
+{
+ struct rcu_txn_hlist_node __rcu **slot = &head->first;
+ struct rcu_txn_hlist_node *p;
+ int i = 0;
+
+ rcu_txn_hlist_for_each_rcu(p, head, true) {
+ if (i < nr)
+ KUNIT_EXPECT_EQ(test,
+ container_of(p, struct hn, node)->key,
+ want[i]);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(p->pprev), slot);
+ slot = &p->next;
+ i++;
+ }
+ KUNIT_EXPECT_EQ(test, i, nr);
+}
+
+static void rcu_txn_test_hlist_add(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int behind[] = { 1, 2, 9, 3 };
+ static const int before[] = { 1, 8, 2, 9, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+
+ KUNIT_EXPECT_TRUE(test, rcu_txn_hlist_empty(&head, true));
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ hlist_expect(test, &head, keys, ARRAY_SIZE(keys));
+
+ rcu_txn_hlist_add_behind_rcu(hn_new(test, 9), hlist_find(&head, 2));
+ hlist_expect(test, &head, behind, ARRAY_SIZE(behind));
+
+ rcu_txn_hlist_add_before_rcu(hn_new(test, 8), hlist_find(&head, 2));
+ hlist_expect(test, &head, before, ARRAY_SIZE(before));
+}
+
+static void rcu_txn_test_hlist_del(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int middle[] = { 1, 2, 4, 5 };
+ static const int ends[] = { 2, 4 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *removed;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ removed = hlist_find(&head, 3);
+ rcu_txn_hlist_del_rcu(removed);
+ hlist_expect(test, &head, middle, ARRAY_SIZE(middle));
+ /* A reader standing on the removed node still reaches the chain. */
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_hlist_next_check(removed, true),
+ hlist_find(&head, 4));
+
+ rcu_txn_hlist_del_rcu(hlist_find(&head, 1)); /* first */
+ rcu_txn_hlist_del_rcu(hlist_find(&head, 5)); /* last */
+ hlist_expect(test, &head, ends, ARRAY_SIZE(ends));
+}
+
+static void rcu_txn_test_hlist_replace(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 1, 22, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ rcu_txn_hlist_replace_rcu(hlist_find(&head, 2), hn_new(test, 22));
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Composition on one bucket, with neighbourhoods that touch: the case the
+ * _prepare forms read their own writes for.
+ */
+
+/* Delete the adjacent 2 and 3 from 1..5 in one transaction. */
+static void rcu_txn_test_hlist_compose_adjacent_del(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 4, 5 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e2, *e3;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e2 = hlist_find(&head, 2);
+ e3 = hlist_find(&head, 3);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, e2);
+ rcu_txn_hlist_del_prepare(&txn, e3); /* the neighbour of e2 */
+ /* &1->next, chained, and the pprev of 3 and of 4. */
+ KUNIT_EXPECT_EQ(test, txn.nr, 3);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* Delete 2 and insert 9 into the same gap, behind 1, in one transaction. */
+static void rcu_txn_test_hlist_compose_del_and_add(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 1, 9, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e1, *e2;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = hlist_find(&head, 1);
+ e2 = hlist_find(&head, 2);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, e2);
+ rcu_txn_hlist_add_behind_prepare(&txn, hn_new(test, 9), e1);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* Three consecutive deletes in one transaction: &1->next is chained twice. */
+static void rcu_txn_test_hlist_compose_triple_del(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 5 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e[3];
+ struct rcu_txn txn;
+ int i;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ for (i = 0; i < ARRAY_SIZE(e); i++)
+ e[i] = hlist_find(&head, i + 2); /* 2, 3, 4 */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ for (i = 0; i < ARRAY_SIZE(e); i++)
+ rcu_txn_hlist_del_prepare(&txn, e[i]);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Insert 9 behind 1, then delete 1, in that order: the fresh node inherits
+ * &1->next as the slot that points to it, and the next operation removes the
+ * node that slot lives in. add_behind must have recorded &9->next as 2's
+ * pprev, and del(1) must then re-point head->first to the pending 9 rather than
+ * to the committed 2, and record &head->first as 9's pprev.
+ *
+ * A wrong pprev for 2 leaves the forward order intact here: only the pprev
+ * check catches it.
+ */
+static void rcu_txn_test_hlist_compose_add_then_del_host(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 9, 2, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e1;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = hlist_find(&head, 1);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_add_behind_prepare(&txn, hn_new(test, 9), e1);
+ rcu_txn_hlist_del_prepare(&txn, e1);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Delete the first node, then insert before its successor. add_before loads
+ * 2's pprev as the transaction leaves it, &head->first, which del(1) recorded.
+ * The committed &1->next would publish head->first == 1.
+ */
+static void rcu_txn_test_hlist_compose_del_then_add_before(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 9, 2, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e1, *e2;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = hlist_find(&head, 1);
+ e2 = hlist_find(&head, 2);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, e1);
+ rcu_txn_hlist_add_before_prepare(&txn, hn_new(test, 9), e2);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Two buckets in one transaction: a node leaves one and its replacement
+ * enters the other. The edges are installed as proxies, so that at any instant
+ * the key is in exactly one bucket.
+ */
+static void rcu_txn_test_hlist_move(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int from[] = { 1, 3 };
+ static const int to[] = { 2 };
+ struct rcu_txn_hlist_head a = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_head b = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn txn;
+
+ hlist_build(test, &a, keys, ARRAY_SIZE(keys));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, hlist_find(&a, 2));
+ rcu_txn_hlist_add_head_prepare(&txn, hn_new(test, 2), &b);
+ /* The two forward edges, and the pprev of 3. */
+ KUNIT_EXPECT_EQ(test, txn.nr, 3);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &a, from, ARRAY_SIZE(from));
+ hlist_expect(test, &b, to, ARRAY_SIZE(to));
+}
+
+/*
+ * One _prepare alone: an update with a successor records its forward edge and
+ * the successor's pprev, and commits through the descriptor.
+ */
+static void rcu_txn_test_hlist_prepare_alone(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 1, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, hlist_find(&head, 2));
+ KUNIT_EXPECT_EQ(test, txn.nr, 2);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Every kind of edit in one transaction over 1..5: del(2), add_behind(9, 1),
+ * replace(4, 44), add_before(8, 5). The last re-points the slot that the
+ * replace made 5's, inside the fresh node 44, and so loads 5's pending pprev
+ * to find it.
+ */
+static void hlist_prepare_mixed(struct kunit *test, struct rcu_txn *txn,
+ struct rcu_txn_hlist_head *head)
+{
+ struct rcu_txn_hlist_node *e1 = hlist_find(head, 1);
+ struct rcu_txn_hlist_node *e2 = hlist_find(head, 2);
+ struct rcu_txn_hlist_node *e4 = hlist_find(head, 4);
+ struct rcu_txn_hlist_node *e5 = hlist_find(head, 5);
+
+ rcu_txn_hlist_del_prepare(txn, e2);
+ rcu_txn_hlist_add_behind_prepare(txn, hn_new(test, 9), e1);
+ rcu_txn_hlist_replace_prepare(txn, e4, hn_new(test, 44));
+ rcu_txn_hlist_add_before_prepare(txn, hn_new(test, 8), e5);
+}
+
+static void rcu_txn_test_hlist_compose_mixed(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 9, 3, 44, 8, 5 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ hlist_prepare_mixed(test, &txn, &head);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * The same transaction, cancelled: nothing was written outside it, so the chain
+ * and every pprev are as they were. A later single update then finds the
+ * pprev it relies on.
+ */
+static void rcu_txn_test_hlist_cancel(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 2, 4, 5 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ hlist_prepare_mixed(test, &txn, &head);
+ rcu_txn_cancel(&txn);
+ hlist_expect(test, &head, keys, ARRAY_SIZE(keys));
+
+ rcu_txn_hlist_del_rcu(hlist_find(&head, 3));
+ hlist_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Out of memory after two edits, the sticky state set by hand as a failed
+ * record() leaves it: commit() reports it and publishes nothing, and the edits
+ * recorded before the failure leave no pprev behind either.
+ */
+static void rcu_txn_test_hlist_oom(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e1, *e2, *e4;
+ struct rcu_txn txn;
+
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = hlist_find(&head, 1);
+ e2 = hlist_find(&head, 2);
+ e4 = hlist_find(&head, 4);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, e2);
+ rcu_txn_hlist_add_behind_prepare(&txn, hn_new(test, 9), e1);
+ txn.state = RCU_TXN_OOM;
+ rcu_txn_hlist_replace_prepare(&txn, e4, hn_new(test, 44));
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), -ENOMEM);
+ hlist_expect(test, &head, keys, ARRAY_SIZE(keys));
+}
+
+/*
+ * An add_behind anchored on a node that the same transaction deleted: it
+ * records the successor's pprev from a slot that the delete already moved it
+ * from.
+ */
+static void rcu_txn_test_hlist_debug_ghost_anchor(struct kunit *test)
+{
+ static const int keys[] = { 1, 2, 3 };
+ struct rcu_txn_hlist_head head = RCU_TXN_HLIST_HEAD_INIT;
+ struct rcu_txn_hlist_node *e2;
+ struct rcu_txn txn;
+
+ need_debug(test);
+ hlist_build(test, &head, keys, ARRAY_SIZE(keys));
+ e2 = hlist_find(&head, 2);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_hlist_del_prepare(&txn, e2);
+ kunit_warning_suppress(test) {
+ rcu_txn_hlist_add_behind_prepare(&txn, hn_new(test, 9), e2);
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+ hlist_expect(test, &head, keys, ARRAY_SIZE(keys));
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -946,6 +1328,20 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_debug_disjoint),
KUNIT_CASE(rcu_txn_test_debug_stale_chain),
KUNIT_CASE(rcu_txn_test_debug_load_proxy),
+ KUNIT_CASE(rcu_txn_test_hlist_add),
+ KUNIT_CASE(rcu_txn_test_hlist_del),
+ KUNIT_CASE(rcu_txn_test_hlist_replace),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_adjacent_del),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_del_and_add),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_triple_del),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_add_then_del_host),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_del_then_add_before),
+ KUNIT_CASE(rcu_txn_test_hlist_move),
+ KUNIT_CASE(rcu_txn_test_hlist_prepare_alone),
+ KUNIT_CASE(rcu_txn_test_hlist_compose_mixed),
+ KUNIT_CASE(rcu_txn_test_hlist_cancel),
+ KUNIT_CASE(rcu_txn_test_hlist_oom),
+ KUNIT_CASE(rcu_txn_test_hlist_debug_ghost_anchor),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 06/12] rcu_txn: Add a doubly-linked list coherent in both directions
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (4 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 05/12] rcu_txn: Add an hlist edited through transactions Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 07/12] Documentation/RCU: Describe RCU pseudo-transactions Mathieu Desnoyers
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
An update of a doubly-linked list re-points a forward and a backward
edge. The RCU list does it with two stores, so a reader that walks in
both directions (list_bidir_prev_rcu()) can find the new forward edge
and then the old backward edge: having stepped forward over a node just
inserted, a step back can skip it.
Add rcu_txn_list, a circular doubly-linked list that commits both edges
of each operation as one RCU pseudo-transaction. At every instant,
a->next == b implies b->prev == a for the nodes of the list, and a
reader walking from node to node never finds the new value of an edge
and then the old value of the other.
Every operation switches two slots, so it installs proxies and allocates
a descriptor: the _rcu forms take a gfp_t and may return -ENOMEM,
leaving the list unchanged. The _prepare forms record into a
transaction of the caller's, which can be reserved beforehand and can
compose several operations, on this list or with other structures.
A node moves with rcu_txn_list_move_after_prepare() and
_before_prepare(), and the _rcu forms move_after, move_before, move (to
the front) and move_tail. A move records six edges: the two that unlink
the node, the two that link it at its new place, and the node's own next
and prev, which a reader standing on the node follows. The add forms
cannot move a node, as they build its links with plain stores. A move
to where the node already is records nothing.
Its iterators and rcu_txn_list_empty() take the same optional lockdep
condition as those of rcu_txn_hlist, and rcu_txn_list_next_check() and
_prev_check() are the accessors that take one.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
include/linux/rcu_txn_hlist.h | 7 +-
include/linux/rcu_txn_list.h | 544 ++++++++++++++++++++++++++++++++++
lib/tests/rcu_txn_kunit.c | 402 +++++++++++++++++++++++++
3 files changed, 951 insertions(+), 2 deletions(-)
create mode 100644 include/linux/rcu_txn_list.h
diff --git a/include/linux/rcu_txn_hlist.h b/include/linux/rcu_txn_hlist.h
index aecb1e4b94dc..070ccc131b76 100644
--- a/include/linux/rcu_txn_hlist.h
+++ b/include/linux/rcu_txn_hlist.h
@@ -11,7 +11,9 @@
* rcu_txn_hlist: an RCU list with a single-pointer head, shaped like hlist, for
* a single writer (writers mutually excluded, as with hlist_add_head_rcu()).
* Its edits are recorded into a transaction of <linux/rcu_txn.h>, whose commit
- * makes all of them, on one bucket or on many, visible to readers at once.
+ * makes all of them, on one bucket or on many, visible to readers at once. It
+ * is the hash-bucket sibling of the circular <linux/rcu_txn_list.h>: its head
+ * is one pointer, half the footprint of a sentinel-node head.
*
* Forward-only readers, and pprev
* -------------------------------
@@ -23,7 +25,8 @@
* struct rcu_txn_hlist_node **pprev; (slot ptr; writer-only) }
*
* Readers walk forward only and never load pprev, so each operation has exactly
- * one reader-visible edge: the next slot it re-points.
+ * one reader-visible edge: the next slot it re-points. This is where the hlist
+ * differs from <linux/rcu_txn_list.h>, whose prev is reader-visible.
*
* Single updates and transactions
* -------------------------------
diff --git a/include/linux/rcu_txn_list.h b/include/linux/rcu_txn_list.h
new file mode 100644
index 000000000000..6d99186e1ecf
--- /dev/null
+++ b/include/linux/rcu_txn_list.h
@@ -0,0 +1,544 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+/*
+ * RCU doubly-linked list whose two directions stay coherent
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ */
+#ifndef _LINUX_RCU_TXN_LIST_H
+#define _LINUX_RCU_TXN_LIST_H
+
+/*
+ * rcu_txn_list: a circular doubly-linked list whose forward and backward
+ * chains stay mutually coherent under concurrent RCU readers. A reader may
+ * iterate in either direction, or reverse direction mid-traversal, and never
+ * observe the two directions disagree.
+ *
+ * Why this is not just <linux/rculist.h>
+ * --------------------------------------
+ * The classic RCU list publishes the forward chain. list_del_rcu() poisons
+ * the removed entry's prev, and where a backward walk is allowed
+ * (list_bidir_prev_rcu(), with list_bidir_del_rcu()), the forward and the
+ * backward edge of an update are still two stores. Between them a reader sees
+ * one direction changed and not the other, so a step forward followed by a
+ * step backward need not bring it back to where it was.
+ *
+ * Every insertion into or deletion from a doubly-linked list re-points exactly
+ * two reader-visible edges, one forward and one backward:
+ *
+ * insert E between P and N: P->next: N -> E and N->prev: P -> E
+ * delete E between P and N: P->next: E -> N and N->prev: E -> P
+ *
+ * A move of E from between P and N to between Q and R re-points six: those of
+ * the delete and of the insert, and E's own next and prev, which a reader
+ * standing on E follows.
+ *
+ * rcu_txn_list commits both edges as one RCU pseudo-transaction
+ * (<linux/rcu_txn.h>): the two slots transiently hold tagged proxies that
+ * share one selector, and one store-release switches both edges from old to
+ * new. At every instant the forward and backward chains describe the same
+ * list.
+ *
+ * Coherence guarantee
+ * -------------------
+ * For any two nodes that are both members of the list at the instant they are
+ * resolved, the links are mutual inverses: a->next == b implies b->prev == a.
+ * No operation exposes a window where one direction sees a change and the other
+ * does not.
+ *
+ * As with any RCU structure, this does not freeze the list for a long-lived
+ * reader: it may observe an operation that commits during its walk. What it
+ * observes is coherent in both directions at each step.
+ *
+ * Coherence is a property of one resolution. Carrying it across two
+ * resolutions needs an ordering between them, which a dependency-chained walk
+ * has (next-then-next, or a prev hop taken from the node the previous load
+ * returned). A reader comparing two independently-reached slots must supply
+ * its own acquire on weakly-ordered hardware; see the monotonicity note in
+ * <linux/rcu_txn.h>.
+ *
+ * A removed node keeps its own next/prev pointing at its former neighbours, so
+ * a reader standing on it can still reach the live list in either direction,
+ * but the live list no longer points back at it. The caller reclaims the node
+ * after a grace period, as with list_del_rcu().
+ *
+ * Read side
+ * ---------
+ * Iterate within an RCU read-side critical section. A slot may transiently
+ * hold a tagged proxy, so next/prev must be read through
+ * rcu_txn_list_next_rcu() / rcu_txn_list_prev_rcu() or the iterators below,
+ * never directly. This is also why the node type is distinct from struct
+ * list_head: the list_*_rcu accessors do not resolve proxies.
+ *
+ * Write side
+ * ----------
+ * Writers must be mutually excluded, as with list_add_rcu(). Each mutator
+ * records its edges in a transaction, a forward and a backward one or six for
+ * a move, and commits them together. Such a transaction installs proxies, so
+ * it allocates a descriptor: an _rcu mutator takes a gfp_t and returns 0, or
+ * -ENOMEM if the transaction could not be allocated, in which case the list is
+ * unchanged. A writer that cannot fail there reserves its transaction first
+ * and uses the _prepare forms.
+ *
+ * A writer that excludes the others walks the list with the same iterators,
+ * outside of an RCU read-side critical section, if it passes them a lockdep
+ * condition for that exclusion, such as lockdep_is_held() of its lock, as with
+ * list_for_each_entry_rcu(); rcu_txn_list_next_check(), _prev_check() and
+ * rcu_txn_list_empty() take one too.
+ */
+
+#include <linux/container_of.h>
+#include <linux/gfp_types.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcupdate.h>
+
+/*
+ * The list node, embedded in the embedder's structure. next/prev hold a node
+ * pointer or, transiently during a mutation, a tagged proxy: always read them
+ * through the accessors.
+ */
+struct rcu_txn_list_node {
+ struct rcu_txn_list_node __rcu *next, *prev;
+};
+
+/* The list head embeds the circular sentinel node. */
+struct rcu_txn_list_head {
+ struct rcu_txn_list_node node;
+};
+
+#define RCU_TXN_LIST_HEAD_INIT(name) \
+ { .node = { .next = RCU_INITIALIZER(&(name).node), \
+ .prev = RCU_INITIALIZER(&(name).node) } }
+
+#define RCU_TXN_LIST_HEAD(name) \
+ struct rcu_txn_list_head name = RCU_TXN_LIST_HEAD_INIT(name)
+
+static inline void rcu_txn_list_init(struct rcu_txn_list_head *head)
+{
+ RCU_INIT_POINTER(head->node.next, &head->node);
+ RCU_INIT_POINTER(head->node.prev, &head->node);
+}
+
+/*
+ * A slot value with bit 0 set is a tagged struct rcu_txn_record * rather than
+ * a node pointer. Nodes and records are at least pointer-aligned, so bit 0 is
+ * free.
+ */
+#define RCU_TXN_LIST_TAG 1UL
+
+/*
+ * Write-side read of a link slot: this transaction's pending write to @slot if
+ * it recorded one, else the slot's committed value. A typed wrapper over
+ * rcu_txn_load().
+ *
+ * Every _prepare below reads its neighbour links through this rather than
+ * ->next / ->prev, which is what lets edits compose on one list (see
+ * rcu_txn_list_add_after_prepare()). Readers use rcu_txn_list_next_rcu() /
+ * _prev_rcu() instead.
+ */
+static inline struct rcu_txn_list_node *
+rcu_txn_list_pending(struct rcu_txn *txn, struct rcu_txn_list_node __rcu **slot)
+{
+ return rcu_txn_load(txn, rcu_txn_slot(slot), RCU_TXN_LIST_TAG);
+}
+
+/*
+ * Forward and backward steps, within an RCU read-side critical section or
+ * where @c, a lockdep condition, holds.
+ */
+#define rcu_txn_list_next_check(node, c) \
+ rcu_txn_dereference_check((node)->next, RCU_TXN_LIST_TAG, c)
+
+#define rcu_txn_list_prev_check(node, c) \
+ rcu_txn_dereference_check((node)->prev, RCU_TXN_LIST_TAG, c)
+
+/* Forward and backward steps, for use within an RCU read-side section. */
+static inline struct rcu_txn_list_node *
+rcu_txn_list_next_rcu(struct rcu_txn_list_node *node)
+{
+ return rcu_txn_dereference(node->next, RCU_TXN_LIST_TAG);
+}
+
+static inline struct rcu_txn_list_node *
+rcu_txn_list_prev_rcu(struct rcu_txn_list_node *node)
+{
+ return rcu_txn_dereference(node->prev, RCU_TXN_LIST_TAG);
+}
+
+/*
+ * True iff the list is empty, within an RCU read-side critical section or
+ * where @cond, an optional lockdep condition, holds.
+ *
+ * Unlike list_empty(), which is a plain load, this needs one or the other:
+ * while a commit is in flight on head->node.next, the slot holds a proxy, and
+ * telling an empty list from another then reads the transaction's descriptor,
+ * which is freed after a grace period. A writer that excludes the others finds
+ * no proxy there, and passes a lockdep condition for that exclusion as @cond.
+ */
+#define rcu_txn_list_empty(head, cond...) \
+ (&(head)->node == \
+ rcu_txn_list_next_check(&(head)->node, __rcu_txn_cond(cond)))
+
+/**
+ * rcu_txn_list_add_after_prepare() - record the insertion of a node after
+ * another
+ * @txn: the transaction
+ * @new: the node to insert
+ * @pos: the node to insert it after
+ *
+ * Record the two edges of the insertion into the caller's transaction, without
+ * committing. The caller owns the transaction, from init to commit, and may
+ * combine these records with others, including records of other structures:
+ * publish a node into a tree and splice it into this list atomically, for
+ * instance.
+ *
+ * Several _prepare calls may share a transaction, even when their
+ * neighbourhoods touch. Each reads its neighbour links through
+ * rcu_txn_list_pending() and records through rcu_txn_record_chain(), so it
+ * sees the transaction's own pending edits, and a second write to a slot
+ * chains onto the first.
+ *
+ * Example: P -> E1 -> E2 -> N, deleting E1 and E2 in one transaction, a record
+ * being written {slot: old -> new}. del(E1) records {P->next: E1 -> E2} and
+ * {E2->prev: E1 -> P}. del(E2) then reads E2->prev and gets the pending P, not
+ * the committed E1, so it records against P->next, finds it recorded, and
+ * chains: {P->next: E1 -> N}. With {N->prev: E2 -> P}, P and N end up linked
+ * and both victims unlinked.
+ *
+ * Two rules remain for the caller:
+ *
+ * - Look up the nodes before editing. A traversal made with the _rcu
+ * accessors sees the committed list, not the transaction's pending one, so
+ * a caller that walks and edits in the same transaction may find a node
+ * that its own earlier edit displaced.
+ *
+ * - Do not call rcu_txn_declare_disjoint() on a transaction that composes
+ * edits. It turns off both mechanisms above: in the example, del(E2) would
+ * read the committed E1 and record {E1->next: E2 -> N} and
+ * {N->prev: E2 -> E1}, and the commit would link P and N to deleted
+ * nodes. The four slots are distinct, so only CONFIG_DEBUG_RCU_TXN
+ * reports it, at the load. The _rcu forms below declare disjoint because
+ * each carries a single operation.
+ */
+static inline void rcu_txn_list_add_after_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *new,
+ struct rcu_txn_list_node *pos)
+{
+ struct rcu_txn_list_node *next = rcu_txn_list_pending(txn, &pos->next);
+
+ /* Build the fresh node's links before it becomes reachable. */
+ RCU_INIT_POINTER(new->prev, pos);
+ RCU_INIT_POINTER(new->next, next);
+
+ /* pos->next: next -> new ; next->prev: pos -> new */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&pos->next), next, new,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&next->prev), pos, new,
+ RCU_TXN_LIST_TAG);
+}
+
+/**
+ * rcu_txn_list_add_before_prepare() - record the insertion of a node before
+ * another
+ * @txn: the transaction
+ * @new: the node to insert
+ * @pos: the node to insert it before
+ *
+ * See rcu_txn_list_add_after_prepare().
+ */
+static inline void
+rcu_txn_list_add_before_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *new,
+ struct rcu_txn_list_node *pos)
+{
+ struct rcu_txn_list_node *prev = rcu_txn_list_pending(txn, &pos->prev);
+
+ RCU_INIT_POINTER(new->next, pos);
+ RCU_INIT_POINTER(new->prev, prev);
+
+ /* prev->next: pos -> new ; pos->prev: prev -> new */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&prev->next), pos, new,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&pos->prev), prev, new,
+ RCU_TXN_LIST_TAG);
+}
+
+/**
+ * rcu_txn_list_del_prepare() - record the unlink of a node
+ * @txn: the transaction
+ * @elem: the node to unlink
+ *
+ * See rcu_txn_list_add_after_prepare(). @elem's own next/prev are left as
+ * they are, so a reader standing on it can still leave in either direction;
+ * the caller frees @elem a grace period after the commit.
+ */
+static inline void rcu_txn_list_del_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *elem)
+{
+ struct rcu_txn_list_node *prev = rcu_txn_list_pending(txn, &elem->prev);
+ struct rcu_txn_list_node *next = rcu_txn_list_pending(txn, &elem->next);
+
+ /* prev->next: elem -> next ; next->prev: elem -> prev */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&prev->next), elem, next,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&next->prev), elem, prev,
+ RCU_TXN_LIST_TAG);
+}
+
+/**
+ * rcu_txn_list_replace_prepare() - record the replacement of a node
+ * @txn: the transaction
+ * @old: the node to replace
+ * @new: the node that replaces it
+ *
+ * See rcu_txn_list_add_after_prepare(). @new takes @old's neighbours, and
+ * @old keeps its own next/prev for readers standing on it.
+ */
+static inline void rcu_txn_list_replace_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *old,
+ struct rcu_txn_list_node *new)
+{
+ struct rcu_txn_list_node *prev = rcu_txn_list_pending(txn, &old->prev);
+ struct rcu_txn_list_node *next = rcu_txn_list_pending(txn, &old->next);
+
+ RCU_INIT_POINTER(new->prev, prev);
+ RCU_INIT_POINTER(new->next, next);
+
+ /* prev->next: old -> new ; next->prev: old -> new */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&prev->next), old, new,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&next->prev), old, new,
+ RCU_TXN_LIST_TAG);
+}
+
+/**
+ * rcu_txn_list_move_after_prepare() - record the move of a node after another
+ * @txn: the transaction
+ * @elem: the node to move, a node of the list
+ * @pos: the node to move it after
+ *
+ * See rcu_txn_list_add_after_prepare(). Six edges switch together: the two
+ * that unlink @elem, the two that link it after @pos, and @elem's own next and
+ * prev, so that a reader standing on @elem leaves it by its old neighbours
+ * until the commit and by its new ones from then on. The add forms cannot
+ * move a node: they build its links with plain stores, for a node that no
+ * reader can reach yet.
+ *
+ * A move of @elem after the node it already follows records nothing. @pos
+ * must not be @elem.
+ */
+static inline void
+rcu_txn_list_move_after_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_node *pos)
+{
+ struct rcu_txn_list_node *prev = rcu_txn_list_pending(txn, &elem->prev);
+ struct rcu_txn_list_node *next = rcu_txn_list_pending(txn, &elem->next);
+ struct rcu_txn_list_node *succ;
+
+ if (WARN_ON_ONCE(pos == elem) || pos == prev)
+ return;
+ succ = rcu_txn_list_pending(txn, &pos->next);
+
+ /* prev->next: elem -> next ; next->prev: elem -> prev */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&prev->next), elem, next,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&next->prev), elem, prev,
+ RCU_TXN_LIST_TAG);
+ /* pos->next: succ -> elem ; succ->prev: pos -> elem */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&pos->next), succ, elem,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&succ->prev), pos, elem,
+ RCU_TXN_LIST_TAG);
+ /* elem->next: next -> succ ; elem->prev: prev -> pos */
+ rcu_txn_record_chain(txn, rcu_txn_slot(&elem->next), next, succ,
+ RCU_TXN_LIST_TAG);
+ rcu_txn_record_chain(txn, rcu_txn_slot(&elem->prev), prev, pos,
+ RCU_TXN_LIST_TAG);
+}
+
+/**
+ * rcu_txn_list_move_before_prepare() - record the move of a node before
+ * another
+ * @txn: the transaction
+ * @elem: the node to move, a node of the list
+ * @pos: the node to move it before
+ *
+ * See rcu_txn_list_move_after_prepare().
+ */
+static inline void
+rcu_txn_list_move_before_prepare(struct rcu_txn *txn,
+ struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_node *pos)
+{
+ struct rcu_txn_list_node *prev = rcu_txn_list_pending(txn, &pos->prev);
+
+ if (WARN_ON_ONCE(pos == elem) || prev == elem)
+ return;
+ rcu_txn_list_move_after_prepare(txn, elem, prev);
+}
+
+/*
+ * The _rcu forms: each runs one operation in a transaction of its own. They
+ * return 0, or -ENOMEM if the transaction could not be allocated with @gfp, in
+ * which case the list is unchanged. The caller frees a removed or replaced
+ * node after a grace period.
+ */
+
+/*
+ * Start the transaction of a single operation, of @nr records: its slots are
+ * distinct.
+ */
+static inline void __rcu_txn_list_init_one(struct rcu_txn *txn, gfp_t gfp,
+ unsigned int nr)
+{
+ rcu_txn_init(txn, gfp);
+ rcu_txn_declare_disjoint(txn);
+ /* An OOM is sticky: commit reports it. */
+ rcu_txn_reserve(txn, nr);
+}
+
+static inline int rcu_txn_list_add_after_rcu(struct rcu_txn_list_node *new,
+ struct rcu_txn_list_node *pos,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 2);
+ rcu_txn_list_add_after_prepare(&txn, new, pos);
+ return rcu_txn_commit(&txn);
+}
+
+static inline int rcu_txn_list_add_before_rcu(struct rcu_txn_list_node *new,
+ struct rcu_txn_list_node *pos,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 2);
+ rcu_txn_list_add_before_prepare(&txn, new, pos);
+ return rcu_txn_commit(&txn);
+}
+
+/* Add @new at the head of the list (just after @head). */
+static inline int rcu_txn_list_add_rcu(struct rcu_txn_list_node *new,
+ struct rcu_txn_list_head *head,
+ gfp_t gfp)
+{
+ return rcu_txn_list_add_after_rcu(new, &head->node, gfp);
+}
+
+/* Add @new at the tail of the list (just before @head). */
+static inline int rcu_txn_list_add_tail_rcu(struct rcu_txn_list_node *new,
+ struct rcu_txn_list_head *head,
+ gfp_t gfp)
+{
+ return rcu_txn_list_add_before_rcu(new, &head->node, gfp);
+}
+
+static inline int rcu_txn_list_del_rcu(struct rcu_txn_list_node *elem,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 2);
+ rcu_txn_list_del_prepare(&txn, elem);
+ return rcu_txn_commit(&txn);
+}
+
+static inline int rcu_txn_list_replace_rcu(struct rcu_txn_list_node *old,
+ struct rcu_txn_list_node *new,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 2);
+ rcu_txn_list_replace_prepare(&txn, old, new);
+ return rcu_txn_commit(&txn);
+}
+
+static inline int rcu_txn_list_move_after_rcu(struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_node *pos,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 6);
+ rcu_txn_list_move_after_prepare(&txn, elem, pos);
+ return rcu_txn_commit(&txn);
+}
+
+static inline int rcu_txn_list_move_before_rcu(struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_node *pos,
+ gfp_t gfp)
+{
+ struct rcu_txn txn;
+
+ __rcu_txn_list_init_one(&txn, gfp, 6);
+ rcu_txn_list_move_before_prepare(&txn, elem, pos);
+ return rcu_txn_commit(&txn);
+}
+
+/* Move @elem to the head of the list (just after @head). */
+static inline int rcu_txn_list_move_rcu(struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_head *head,
+ gfp_t gfp)
+{
+ return rcu_txn_list_move_after_rcu(elem, &head->node, gfp);
+}
+
+/* Move @elem to the tail of the list (just before @head). */
+static inline int rcu_txn_list_move_tail_rcu(struct rcu_txn_list_node *elem,
+ struct rcu_txn_list_head *head,
+ gfp_t gfp)
+{
+ return rcu_txn_list_move_before_rcu(elem, &head->node, gfp);
+}
+
+#define rcu_txn_list_entry(ptr, type, member) \
+ container_of(ptr, type, member)
+
+#define rcu_txn_list_first_entry_rcu(head, type, member) \
+ rcu_txn_list_entry(rcu_txn_list_next_rcu(&(head)->node), type, member)
+
+#define rcu_txn_list_last_entry_rcu(head, type, member) \
+ rcu_txn_list_entry(rcu_txn_list_prev_rcu(&(head)->node), type, member)
+
+/*
+ * Iterate over the list, within an RCU read-side critical section, or where
+ * @cond, an optional lockdep condition, holds.
+ */
+#define rcu_txn_list_for_each_rcu(pos, head, cond...) \
+ for (pos = rcu_txn_list_next_check(&(head)->node, \
+ __rcu_txn_cond(cond)); \
+ pos != &(head)->node; \
+ pos = rcu_txn_list_next_check(pos, __rcu_txn_cond(cond)))
+
+#define rcu_txn_list_for_each_reverse_rcu(pos, head, cond...) \
+ for (pos = rcu_txn_list_prev_check(&(head)->node, \
+ __rcu_txn_cond(cond)); \
+ pos != &(head)->node; \
+ pos = rcu_txn_list_prev_check(pos, __rcu_txn_cond(cond)))
+
+#define rcu_txn_list_for_each_entry_rcu(pos, head, member, cond...) \
+ for (pos = rcu_txn_list_entry( \
+ rcu_txn_list_next_check(&(head)->node, \
+ __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member); \
+ &(pos)->member != &(head)->node; \
+ pos = rcu_txn_list_entry( \
+ rcu_txn_list_next_check(&(pos)->member, \
+ __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member))
+
+#define rcu_txn_list_for_each_entry_reverse_rcu(pos, head, member, cond...) \
+ for (pos = rcu_txn_list_entry( \
+ rcu_txn_list_prev_check(&(head)->node, \
+ __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member); \
+ &(pos)->member != &(head)->node; \
+ pos = rcu_txn_list_entry( \
+ rcu_txn_list_prev_check(&(pos)->member, \
+ __rcu_txn_cond(cond)), \
+ typeof(*(pos)), member))
+
+#endif /* _LINUX_RCU_TXN_LIST_H */
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index 13e4bbae1053..9b67caaa5db7 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -9,6 +9,7 @@
#include <linux/bug.h>
#include <linux/rcu_txn.h>
#include <linux/rcu_txn_hlist.h>
+#include <linux/rcu_txn_list.h>
#include <linux/rcupdate.h>
#include <linux/sched/task.h>
#include <linux/string.h>
@@ -1295,6 +1296,396 @@ static void rcu_txn_test_hlist_debug_ghost_anchor(struct kunit *test)
hlist_expect(test, &head, keys, ARRAY_SIZE(keys));
}
+/*
+ * <linux/rcu_txn_list.h>
+ */
+
+struct ln {
+ struct rcu_txn_list_node node;
+ int key;
+};
+
+/* Freed with the test: a removed node stays valid for any reader. */
+static struct rcu_txn_list_node *ln_new(struct kunit *test, int key)
+{
+ struct ln *n = kunit_kzalloc(test, sizeof(*n), GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_NULL(test, n);
+ n->key = key;
+ return &n->node;
+}
+
+/* Build the list @keys by appending. */
+static void list_build(struct kunit *test, struct rcu_txn_list_head *head,
+ const int *keys, int nr)
+{
+ int i;
+
+ for (i = 0; i < nr; i++)
+ KUNIT_ASSERT_EQ(test,
+ rcu_txn_list_add_tail_rcu(ln_new(test, keys[i]),
+ head, GFP_KERNEL),
+ 0);
+}
+
+/* The test is the list's only writer, and owns every node until it ends. */
+static struct rcu_txn_list_node *list_find(struct rcu_txn_list_head *head,
+ int key)
+{
+ struct ln *n;
+
+ rcu_txn_list_for_each_entry_rcu(n, head, node, true)
+ if (n->key == key)
+ return &n->node;
+ return NULL;
+}
+
+/*
+ * The forward walk gives @want, the backward walk gives its mirror, and next
+ * and prev are mutual inverses at every edge of the ring, sentinel included.
+ */
+static void list_expect(struct kunit *test, struct rcu_txn_list_head *head,
+ const int *want, int nr)
+{
+ struct rcu_txn_list_node *p, *q;
+ struct ln *n;
+ int i = 0;
+
+ rcu_txn_list_for_each_entry_rcu(n, head, node, true) {
+ if (i < nr)
+ KUNIT_EXPECT_EQ(test, n->key, want[i]);
+ i++;
+ }
+ KUNIT_EXPECT_EQ(test, i, nr);
+ rcu_txn_list_for_each_entry_reverse_rcu(n, head, node, true) {
+ i--;
+ if (i >= 0)
+ KUNIT_EXPECT_EQ(test, n->key, want[i]);
+ }
+ KUNIT_EXPECT_EQ(test, i, 0);
+ p = &head->node;
+ do {
+ q = rcu_txn_list_next_check(p, true);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(q, true), p);
+ p = q;
+ } while (p != &head->node);
+}
+
+static void rcu_txn_test_list_add(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ RCU_TXN_LIST_HEAD(tail);
+ static const int at_head[] = { 3, 2, 1 };
+ static const int keys[] = { 1, 2, 3 };
+ static const int middle[] = { 1, 8, 2, 9, 3 };
+ int i;
+
+ KUNIT_EXPECT_TRUE(test, rcu_txn_list_empty(&head, true));
+ list_expect(test, &head, NULL, 0);
+
+ for (i = 1; i <= 3; i++)
+ KUNIT_ASSERT_EQ(test,
+ rcu_txn_list_add_rcu(ln_new(test, i), &head,
+ GFP_KERNEL),
+ 0);
+ list_expect(test, &head, at_head, ARRAY_SIZE(at_head));
+
+ list_build(test, &tail, keys, ARRAY_SIZE(keys));
+ list_expect(test, &tail, keys, ARRAY_SIZE(keys));
+
+ KUNIT_EXPECT_EQ(test,
+ rcu_txn_list_add_after_rcu(ln_new(test, 9),
+ list_find(&tail, 2),
+ GFP_KERNEL),
+ 0);
+ KUNIT_EXPECT_EQ(test,
+ rcu_txn_list_add_before_rcu(ln_new(test, 8),
+ list_find(&tail, 2),
+ GFP_KERNEL),
+ 0);
+ list_expect(test, &tail, middle, ARRAY_SIZE(middle));
+}
+
+static void rcu_txn_test_list_del(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int middle[] = { 1, 2, 4, 5 };
+ static const int ends[] = { 2, 4 };
+ struct rcu_txn_list_node *removed;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ removed = list_find(&head, 3);
+ KUNIT_EXPECT_EQ(test, rcu_txn_list_del_rcu(removed, GFP_KERNEL), 0);
+ list_expect(test, &head, middle, ARRAY_SIZE(middle));
+ /* A reader standing on the removed node leaves in either direction. */
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(removed, true),
+ list_find(&head, 4));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(removed, true),
+ list_find(&head, 2));
+
+ rcu_txn_list_del_rcu(list_find(&head, 1), GFP_KERNEL); /* first */
+ rcu_txn_list_del_rcu(list_find(&head, 5), GFP_KERNEL); /* last */
+ list_expect(test, &head, ends, ARRAY_SIZE(ends));
+
+ rcu_txn_list_del_rcu(list_find(&head, 2), GFP_KERNEL);
+ rcu_txn_list_del_rcu(list_find(&head, 4), GFP_KERNEL);
+ list_expect(test, &head, NULL, 0);
+}
+
+static void rcu_txn_test_list_replace(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 1, 22, 3 };
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ KUNIT_EXPECT_EQ(test,
+ rcu_txn_list_replace_rcu(list_find(&head, 2),
+ ln_new(test, 22), GFP_KERNEL),
+ 0);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * The two edges of a delete, by hand, as a reader sees them across install,
+ * flip and settle: old in both directions until the single flip, new in both
+ * directions after.
+ *
+ * Ring: head <-> A <-> B <-> head. Delete A:
+ * forward edge: head->next: A -> B
+ * backward edge: B->prev: A -> head
+ */
+static void rcu_txn_test_list_proxy_phases(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2 };
+ static const int want[] = { 2 };
+ struct rcu_txn_list_node *a, *b;
+ struct rcu_txn_record rec[2];
+ struct rcu_txn_group group;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ a = list_find(&head, 1);
+ b = list_find(&head, 2);
+
+ __rcu_txn_group_init(&group);
+ __rcu_txn_record_set(&rec[0], rcu_txn_slot(&head.node.next), a, b,
+ RCU_TXN_LIST_TAG);
+ __rcu_txn_record_set(&rec[1], rcu_txn_slot(&b->prev), a, &head.node,
+ RCU_TXN_LIST_TAG);
+ __rcu_txn_record_install(&group, &rec[0]);
+ __rcu_txn_record_install(&group, &rec[1]);
+
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(&head.node, true), a);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(b, true), a);
+
+ __rcu_txn_group_commit(&group); /* one flip switches both edges */
+
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(&head.node, true), b);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(b, true), &head.node);
+
+ /* No reader holds a proxy here: the records need no grace period. */
+ __rcu_txn_record_settle(&rec[0]);
+ __rcu_txn_record_settle(&rec[1]);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* Delete the adjacent 2 and 3 from 1..5 in one transaction. */
+static void rcu_txn_test_list_compose_adjacent_del(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 4, 5 };
+ struct rcu_txn_list_node *e2, *e3;
+ struct rcu_txn txn;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ e2 = list_find(&head, 2);
+ e3 = list_find(&head, 3);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_list_del_prepare(&txn, e2);
+ rcu_txn_list_del_prepare(&txn, e3); /* the neighbour of e2 */
+ /* &1->next is chained; &3->prev and &4->prev are one record each. */
+ KUNIT_EXPECT_EQ(test, txn.nr, 3);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Delete 2 and insert 9 into the same gap in one transaction: both edges of
+ * the gap are written twice.
+ */
+static void rcu_txn_test_list_compose_del_and_add(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3 };
+ static const int want[] = { 1, 9, 3 };
+ struct rcu_txn_list_node *e1, *e2;
+ struct rcu_txn txn;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = list_find(&head, 1);
+ e2 = list_find(&head, 2);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_list_del_prepare(&txn, e2);
+ rcu_txn_list_add_after_prepare(&txn, ln_new(test, 9), e1);
+ KUNIT_EXPECT_EQ(test, txn.nr, 2);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* Three consecutive deletes in one transaction: &1->next is chained twice. */
+static void rcu_txn_test_list_compose_triple_del(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int want[] = { 1, 5 };
+ struct rcu_txn_list_node *e[3];
+ struct rcu_txn txn;
+ int i;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ for (i = 0; i < ARRAY_SIZE(e); i++)
+ e[i] = list_find(&head, i + 2); /* 2, 3, 4 */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ for (i = 0; i < ARRAY_SIZE(e); i++)
+ rcu_txn_list_del_prepare(&txn, e[i]);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* The moves of a node, in a transaction of their own. */
+static void rcu_txn_test_list_move(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4, 5 };
+ static const int front[] = { 4, 1, 2, 3, 5 };
+ static const int tail[] = { 1, 2, 3, 5, 4 };
+ static const int past_next[] = { 1, 3, 2, 5, 4 };
+ struct rcu_txn_list_node *e1, *e2, *e3, *e4;
+ struct rcu_txn txn;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ e1 = list_find(&head, 1);
+ e2 = list_find(&head, 2);
+ e3 = list_find(&head, 3);
+ e4 = list_find(&head, 4);
+
+ KUNIT_EXPECT_EQ(test, rcu_txn_list_move_rcu(e4, &head, GFP_KERNEL), 0);
+ list_expect(test, &head, front, ARRAY_SIZE(front));
+ KUNIT_EXPECT_EQ(test,
+ rcu_txn_list_move_tail_rcu(e4, &head, GFP_KERNEL), 0);
+ list_expect(test, &head, tail, ARRAY_SIZE(tail));
+ /* Past its next, then back before it: the neighbourhoods overlap. */
+ KUNIT_EXPECT_EQ(test, rcu_txn_list_move_after_rcu(e2, e3, GFP_KERNEL), 0);
+ list_expect(test, &head, past_next, ARRAY_SIZE(past_next));
+ KUNIT_EXPECT_EQ(test, rcu_txn_list_move_before_rcu(e2, e3, GFP_KERNEL), 0);
+ list_expect(test, &head, tail, ARRAY_SIZE(tail));
+
+ /* Where it already is: nothing is recorded. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_list_move_after_prepare(&txn, e2, e1);
+ rcu_txn_list_move_before_prepare(&txn, e1, e2);
+ KUNIT_EXPECT_EQ(test, txn.nr, 0);
+ rcu_txn_cancel(&txn);
+}
+
+/*
+ * The six edges of a move, as a reader sees them across install, flip and
+ * settle: the old list until the single flip, the new one after, the moved
+ * node's own links included.
+ *
+ * Ring: head <-> A <-> B <-> C <-> D <-> head. Move D after A.
+ */
+static void rcu_txn_test_list_move_phases(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4 };
+ static const int want[] = { 1, 4, 2, 3 };
+ struct rcu_txn_list_node *a, *b, *c, *d, *h = &head.node;
+ struct rcu_txn txn;
+ unsigned int i;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ a = list_find(&head, 1);
+ b = list_find(&head, 2);
+ c = list_find(&head, 3);
+ d = list_find(&head, 4);
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 6));
+ rcu_txn_list_move_after_prepare(&txn, d, a);
+ KUNIT_ASSERT_EQ(test, txn.nr, 6);
+ for (i = 0; i < txn.nr; i++)
+ __rcu_txn_record_install(&txn.desc->group, &txn.records[i]);
+
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(a, true), b);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(b, true), a);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(c, true), d);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(h, true), d);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(d, true), h);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(d, true), c);
+
+ __rcu_txn_group_commit(&txn.desc->group); /* one flip, six edges */
+
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(a, true), d);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(b, true), d);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(c, true), h);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(h, true), c);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_next_check(d, true), b);
+ KUNIT_EXPECT_PTR_EQ(test, rcu_txn_list_prev_check(d, true), a);
+
+ /* No reader holds a proxy here: the records need no grace period. */
+ for (i = 0; i < txn.nr; i++)
+ __rcu_txn_record_settle(&txn.records[i]);
+ __rcu_txn_free_desc(txn.desc);
+ __rcu_txn_done(&txn);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/* Two moves to the front in one transaction: their edges chain. */
+static void rcu_txn_test_list_compose_moves(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4 };
+ static const int want[] = { 4, 3, 1, 2 };
+ struct rcu_txn txn;
+
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_list_move_after_prepare(&txn, list_find(&head, 3), &head.node);
+ rcu_txn_list_move_after_prepare(&txn, list_find(&head, 4), &head.node);
+ KUNIT_EXPECT_EQ(test, rcu_txn_commit(&txn), 0);
+ list_expect(test, &head, want, ARRAY_SIZE(want));
+}
+
+/*
+ * Declaring disjoint a transaction that composes edits turns off what makes
+ * them compose: the second delete reads a link the first has already recorded.
+ */
+static void rcu_txn_test_list_debug_disjoint_compose(struct kunit *test)
+{
+ RCU_TXN_LIST_HEAD(head);
+ static const int keys[] = { 1, 2, 3, 4 };
+ struct rcu_txn_list_node *e2, *e3;
+ struct rcu_txn txn;
+
+ need_debug(test);
+ list_build(test, &head, keys, ARRAY_SIZE(keys));
+ e2 = list_find(&head, 2);
+ e3 = list_find(&head, 3);
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_declare_disjoint(&txn);
+ rcu_txn_list_del_prepare(&txn, e2);
+ kunit_warning_suppress(test) {
+ rcu_txn_list_del_prepare(&txn, e3); /* loads &3->prev */
+ KUNIT_EXPECT_SUPPRESSED_WARNING_COUNT(test, 1);
+ }
+ rcu_txn_cancel(&txn);
+ list_expect(test, &head, keys, ARRAY_SIZE(keys));
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -1342,6 +1733,17 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_hlist_cancel),
KUNIT_CASE(rcu_txn_test_hlist_oom),
KUNIT_CASE(rcu_txn_test_hlist_debug_ghost_anchor),
+ KUNIT_CASE(rcu_txn_test_list_add),
+ KUNIT_CASE(rcu_txn_test_list_del),
+ KUNIT_CASE(rcu_txn_test_list_replace),
+ KUNIT_CASE(rcu_txn_test_list_proxy_phases),
+ KUNIT_CASE(rcu_txn_test_list_compose_adjacent_del),
+ KUNIT_CASE(rcu_txn_test_list_compose_del_and_add),
+ KUNIT_CASE(rcu_txn_test_list_compose_triple_del),
+ KUNIT_CASE(rcu_txn_test_list_move),
+ KUNIT_CASE(rcu_txn_test_list_move_phases),
+ KUNIT_CASE(rcu_txn_test_list_compose_moves),
+ KUNIT_CASE(rcu_txn_test_list_debug_disjoint_compose),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 07/12] Documentation/RCU: Describe RCU pseudo-transactions
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (5 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 06/12] rcu_txn: Add a doubly-linked list coherent in both directions Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 08/12] Documentation/litmus-tests: Add RCU pseudo-transaction tests Mathieu Desnoyers
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Add a page on the RCU pseudo-transactions to the RCU handbook: when they
apply and what they cost, how a commit proceeds, what readers and
writers call, how a transaction allocates, and the two data structures
built on them. It ends with the API reference extracted from the
headers. The overview at the top of <linux/rcu_txn.h> points to it.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
Documentation/RCU/index.rst | 1 +
Documentation/RCU/rcu_txn.rst | 230 ++++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
include/linux/rcu_txn.h | 2 +
4 files changed, 234 insertions(+)
create mode 100644 Documentation/RCU/rcu_txn.rst
diff --git a/Documentation/RCU/index.rst b/Documentation/RCU/index.rst
index 035871687ee2..4bca16c8573b 100644
--- a/Documentation/RCU/index.rst
+++ b/Documentation/RCU/index.rst
@@ -18,6 +18,7 @@ RCU Handbook
rcu
rculist_nulls
rcuref
+ rcu_txn
torture
stallwarn
listRCU
diff --git a/Documentation/RCU/rcu_txn.rst b/Documentation/RCU/rcu_txn.rst
new file mode 100644
index 000000000000..6e48306ebc19
--- /dev/null
+++ b/Documentation/RCU/rcu_txn.rst
@@ -0,0 +1,230 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================
+Single-writer RCU pseudo-transactions
+=====================================
+
+An RCU pseudo-transaction switches a *set* of RCU-protected pointers from
+their old values to their new values with a single store, as observed by
+concurrent RCU readers: at any instant, every pointer of the set reads old, or
+every pointer reads new. A reader that follows several of them, each load
+depending on the previous one, sees old values, then new ones, and never an
+old value after a new one.
+
+The implementation is in ``include/linux/rcu_txn.h`` and ``lib/rcu_txn.c``.
+Code that uses it selects ``CONFIG_RCU_TXN``.
+
+When to use it
+==============
+
+``rcu_assign_pointer()`` publishes one pointer atomically. An update that
+re-points several pointers one at a time exposes its intermediate states: a
+reader that loads two of them may get the new value of one and then the old
+value of the other. Moving an object between two hash buckets is such an
+update, and so is every insertion into or removal from a doubly-linked list.
+
+A pseudo-transaction removes the intermediate states. It does not replace the
+update-side lock: the writers of a slot are excluded by the caller, and the
+transaction neither detects nor resolves conflicts between them. It is
+"pseudo" in that sense: there is no read set and no abort, and readers get no
+snapshot.
+
+A pseudo-transaction is not free, on the update side above all. The writer
+allocates a descriptor for each transaction, with the ``gfp_t`` given to
+``rcu_txn_init()``, and a transaction that switches two pointers or more frees
+it with ``kfree_rcu()``, a grace period after its commit, as readers may still
+be reading it: an allocation, which can fail, and a deferred free for every
+update, where ``rcu_assign_pointer()`` needs neither. The reader pays a test of
+the low bits of every pointer it loads, which is nearly free: only while a
+commit is in flight on that pointer does the reader also read the descriptor,
+with three more loads, one of them an acquire.
+
+How it works
+============
+
+The pointers a transaction switches are called *slots*: they are the *edges* of
+the data structure that the update re-points. The writer builds a *record* per
+slot, holding the slot's old and new targets, and commits in three steps:
+
+1. *Install.* Each slot is made to point at its record, with tag bits set that
+ mark the value as a *proxy* rather than a target. A reader that finds a
+ proxy resolves it through the record, which still answers with the old
+ target.
+
+2. *Commit.* The records of a transaction share one word, the *selector* of
+ their *flip group*. The writer stores 1 to it. From then on every proxy of
+ the transaction resolves to its new target.
+
+3. *Settle.* Each slot is overwritten with its new target, and the records
+ are freed after a grace period.
+
+The selector is written once, from 0 to 1, so a reader that resolves several
+slots of a transaction over time sees old targets, then new targets, and never
+an old target after a new one, provided that each of its loads depends on the
+previous one. The header describes the memory ordering in detail.
+
+A transaction of a single record installs no proxy: its commit is one
+``rcu_assign_pointer()``.
+
+Tags
+====
+
+The embedder chooses the tag that marks a proxy in its slots: typically bit 0,
+or a reserved value of a type code that its pointers already carry. The tag
+must be within ``RCU_TXN_TAG_MASK``, the low bits that the alignment of a
+record leaves clear, and no value that the embedder stores in a slot may carry
+all of the tag's bits.
+
+Readers
+=======
+
+A reader loads a slot with ``rcu_txn_dereference()`` instead of
+``rcu_dereference()``::
+
+ rcu_read_lock();
+ child = rcu_txn_dereference(node->left, FOO_TAG);
+ if (child)
+ do_something_with(child);
+ rcu_read_unlock();
+
+``rcu_txn_dereference()`` is ``rcu_dereference()`` followed by
+``rcu_txn_resolve()``. A reader that already has the loaded value, for
+instance because it tests other bits of it, calls ``rcu_txn_resolve()``
+itself.
+
+The load and the resolution must happen in one RCU read-side critical section:
+the record a proxy points at is freed with ``kfree_rcu()``.
+
+Writers
+=======
+
+The writer declares a handle on its stack, records the edges of the update and
+commits them::
+
+ struct rcu_txn txn;
+ struct foo *child;
+ int err;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 2);
+
+ spin_lock(&foo_lock);
+ child = rcu_dereference_protected(a->left, lockdep_is_held(&foo_lock));
+ rcu_txn_record(&txn, rcu_txn_slot(&a->left), child, NULL, FOO_TAG);
+ rcu_txn_record(&txn, rcu_txn_slot(&b->right), NULL, child, FOO_TAG);
+ err = rcu_txn_commit(&txn);
+ spin_unlock(&foo_lock);
+
+Here ``child`` moves from ``a->left`` to an empty ``b->right``: at any instant
+it is in exactly one of the two places.
+
+Slots are ``__rcu`` pointers. ``rcu_txn_slot()`` converts the address of one,
+of any type, to the ``void __rcu **`` the engine takes, and lets sparse check
+the annotation.
+
+A handle ends in ``rcu_txn_commit()`` or in ``rcu_txn_cancel()``. Either
+consumes it, and it must be initialized again before it is reused.
+
+Allocation
+----------
+
+A transaction allocates a descriptor, with the ``gfp_t`` given to
+``rcu_txn_init()``. ``rcu_txn_reserve()`` allocates it for a given number of
+records; otherwise the first ``rcu_txn_record()`` does, even for a single
+record, and a later one may grow it.
+
+A failed allocation is sticky: later records do nothing, and
+``rcu_txn_commit()`` publishes nothing and returns ``-ENOMEM``. A writer may
+therefore ignore the return values of ``rcu_txn_reserve()`` and
+``rcu_txn_record()`` and check that of ``rcu_txn_commit()`` only, as in the
+example above.
+
+A writer that records under a spinlock reserves its record bound before it
+takes the lock, as above, or initializes the handle with a non-sleeping
+``gfp_t``. Under a raw or a bit spinlock only the reserve will do, since
+``PREEMPT_RT`` does not allow the allocator there. ``rcu_txn_commit()`` and
+``rcu_txn_cancel()`` allocate nothing, do not sleep, and may be called under
+such a lock.
+
+An update that re-points a single slot needs no transaction. Every
+transaction settles its slots before its commit returns, and the writers of the
+slot are excluded, so the slot holds no proxy: ``rcu_assign_pointer()``
+publishes the update, as the ``_rcu`` operations of ``rcu_txn_hlist`` do.
+
+Writing a slot twice
+--------------------
+
+``rcu_txn_record()`` requires the records of a transaction to target distinct
+slots. A writer that may write the same slot twice in one transaction reads
+the slot with ``rcu_txn_load()``, which returns the transaction's pending value
+for it, and records with ``rcu_txn_record_chain()``, which updates the record
+the slot already has::
+
+ old = rcu_txn_load(&txn, slot, FOO_TAG);
+ rcu_txn_record_chain(&txn, slot, old, new, FOO_TAG);
+
+This is what allows several operations on one data structure to be composed in
+one transaction. The lookup among the transaction's records is a linear scan,
+behind a Bloom filter once the transaction holds enough of them.
+``rcu_txn_declare_disjoint()`` turns the lookup off for a transaction whose
+slots are known to be distinct.
+
+Data structures
+===============
+
+Two data structures are built on the engine. Each provides ``_prepare``
+operations, which record into a transaction owned by the caller, and ``_rcu``
+operations, which carry out a single operation by themselves.
+Readers walk them with ``_rcu`` iterators. A writer that excludes the others
+walks them with the same iterators outside of an RCU read-side critical
+section, if it passes them a lockdep condition for that exclusion, such as
+``lockdep_is_held()`` of its lock, as with ``list_for_each_entry_rcu()``.
+
+``include/linux/rcu_txn_hlist.h``
+ A list with a single-pointer head, like ``hlist``. Its ``_prepare``
+ operations compose: several nodes can be added to or removed from one
+ or several lists atomically. An ``_rcu`` operation publishes its
+ update with one ``rcu_assign_pointer()`` and allocates nothing.
+
+``include/linux/rcu_txn_list.h``
+ A circular doubly-linked list whose forward and backward chains are
+ mutually coherent at every instant, which allows readers to walk it in
+ both directions. An insertion or a removal switches a forward and a
+ backward edge, and a move six, the moved node's own included: each
+ allocates a descriptor.
+
+Debugging
+=========
+
+``CONFIG_DEBUG_RCU_TXN`` enables the checks that cost more than a test of the
+handle's state. It validates the tags, reports a slot recorded twice, and
+validates the single-writer contract: it warns when a handle is driven by a
+task other than the one that initialized it, and when a slot that a transaction
+records, installs to or settles was changed by another writer in the meantime.
+A task that takes over a handle which another got ready, with nothing recorded
+yet, calls ``rcu_txn_adopt()`` first, which makes it the task the check
+expects.
+
+Misuse of a handle, such as recording into one that was already committed,
+warns in every build.
+
+Testing
+=======
+
+``CONFIG_RCU_TXN_KUNIT_TEST`` builds the KUnit suite ``rcu_txn``, which covers
+the engine, both data structures and the debug checks::
+
+ ./tools/testing/kunit/kunit.py run --arch=x86_64 \
+ --kconfig_add CONFIG_RCU_TXN_KUNIT_TEST=y \
+ --kconfig_add CONFIG_DEBUG_KERNEL=y \
+ --kconfig_add CONFIG_DEBUG_RCU_TXN=y \
+ 'rcu_txn*'
+
+API
+===
+
+.. kernel-doc:: include/linux/rcu_txn.h
+
+.. kernel-doc:: include/linux/rcu_txn_hlist.h
+
+.. kernel-doc:: include/linux/rcu_txn_list.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 5126f4ece4e7..f34040369eb4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -22857,6 +22857,7 @@ RCU PSEUDO-TRANSACTIONS
M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
L: rcu@vger.kernel.org
S: Maintained
+F: Documentation/RCU/rcu_txn.rst
F: include/linux/rcu_txn*.h
F: lib/rcu_txn*
F: lib/tests/rcu_txn_kunit.c
diff --git a/include/linux/rcu_txn.h b/include/linux/rcu_txn.h
index b9ee242a01ce..19f1365ae8ae 100644
--- a/include/linux/rcu_txn.h
+++ b/include/linux/rcu_txn.h
@@ -12,6 +12,8 @@
* the slots, from their old to their new targets with a single store, as
* observed by concurrent RCU readers: at any instant, every slot of the set
* resolves to its old target, or every slot to its new one.
+ * Documentation/RCU/rcu_txn.rst says when to use them, and describes the data
+ * structures built on them.
*
* The slots are the edges of the data structure that an update re-points; the
* writer records each edge as its slot with the slot's old and new targets.
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 08/12] Documentation/litmus-tests: Add RCU pseudo-transaction tests
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (6 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 07/12] Documentation/RCU: Describe RCU pseudo-transactions Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 09/12] rcu_txn: Test a list writer against concurrent readers Mathieu Desnoyers
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Model the memory ordering of RCU pseudo-transactions with the Linux
kernel memory model: the store-release / load-acquire pairs that
<linux/rcu_txn.h> documents, and the guarantee they give readers.
- RCU-txn+install: a reader that finds a proxy reads its record
initialized.
- RCU-txn+monotonic+dep and +acquire: the slots of one transaction
resolve old, then new, never new then old, along a chain of dependent
loads or with a load-acquire between the loads.
- RCU-txn+list+step-back: a reader of an rcu_txn_list that steps
forward onto a node being inserted, then back, never skips it.
- RCU-txn+sync+free: a reader never reads a record freed after a grace
period.
- RCU-txn+publish and RCU-txn+remove: a transaction end to end, with
the writer doing every step. A reader of a transaction that
publishes an object sees the record initialized and the object built.
A reader of one that removes an object, which the writer frees with
the descriptor after a grace period, sees the record initialized, and
reads neither of them freed.
Each test checks an outcome that the code forbids, and herd7 finds that
it never happens. Error-injection variants validate that the model
catches a missing ordering: each drops one ordering that its test relies
on, and herd7 then finds the outcome possible. A store-release or a
load-acquire becomes a plain access, the grace period or the read-side
critical section goes, or, in RCU-txn+monotonic+unordered, the reader's
two loads are left with neither a dependency nor a load-acquire between
them.
herd7 can express neither a structure nor a tagged pointer: a record is
a few variables, and the reader's tag test compares the loaded value
with the record's address. Nor does it have arrays: where the code
indexes the record's targets with the selector, an address dependency,
the reader branches on the selector, and a branch orders no later load.
The model thus orders less than the code, and an outcome that it
forbids, the code forbids as well. For the same reason, the selector's
load-acquire is dropped only in RCU-txn+monotonic+acquire+once-resolve,
whose reader loads slot b independently of slot a: for a reader whose
next load depends on the target it resolved, the code's address
dependency still orders that load.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
Documentation/RCU/rcu_txn.rst | 12 +++
Documentation/litmus-tests/README | 37 ++++++++
.../rcu/RCU-txn+install+once-install.litmus | 55 ++++++++++++
.../litmus-tests/rcu/RCU-txn+install.litmus | 66 ++++++++++++++
.../RCU-txn+list+step-back+once-flip.litmus | 78 +++++++++++++++++
.../RCU-txn+list+step-back+once-settle.litmus | 78 +++++++++++++++++
.../rcu/RCU-txn+list+step-back.litmus | 85 +++++++++++++++++++
...-txn+monotonic+acquire+once-resolve.litmus | 79 +++++++++++++++++
.../rcu/RCU-txn+monotonic+acquire.litmus | 81 ++++++++++++++++++
.../RCU-txn+monotonic+dep+once-flip.litmus | 77 +++++++++++++++++
.../RCU-txn+monotonic+dep+once-settle.litmus | 77 +++++++++++++++++
.../rcu/RCU-txn+monotonic+dep.litmus | 83 ++++++++++++++++++
.../rcu/RCU-txn+monotonic+unordered.litmus | 84 ++++++++++++++++++
.../rcu/RCU-txn+publish+once-install.litmus | 60 +++++++++++++
.../rcu/RCU-txn+publish+once-settle.litmus | 60 +++++++++++++
.../litmus-tests/rcu/RCU-txn+publish.litmus | 71 ++++++++++++++++
.../rcu/RCU-txn+remove+nolock.litmus | 64 ++++++++++++++
.../rcu/RCU-txn+remove+nosync.litmus | 64 ++++++++++++++
.../rcu/RCU-txn+remove+once-install.litmus | 65 ++++++++++++++
.../litmus-tests/rcu/RCU-txn+remove.litmus | 72 ++++++++++++++++
.../rcu/RCU-txn+sync+free+nolock.litmus | 55 ++++++++++++
.../rcu/RCU-txn+sync+free+nosync.litmus | 55 ++++++++++++
.../litmus-tests/rcu/RCU-txn+sync+free.litmus | 60 +++++++++++++
MAINTAINERS | 1 +
24 files changed, 1519 insertions(+)
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+install+once-install.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+install.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-flip.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-settle.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+list+step-back.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire+once-resolve.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-flip.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-settle.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+monotonic+unordered.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+publish+once-install.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+publish+once-settle.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+publish.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+remove+nolock.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+remove+nosync.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+remove+once-install.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+remove.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+sync+free+nolock.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+sync+free+nosync.litmus
create mode 100644 Documentation/litmus-tests/rcu/RCU-txn+sync+free.litmus
diff --git a/Documentation/RCU/rcu_txn.rst b/Documentation/RCU/rcu_txn.rst
index 6e48306ebc19..7bac4bd02456 100644
--- a/Documentation/RCU/rcu_txn.rst
+++ b/Documentation/RCU/rcu_txn.rst
@@ -220,6 +220,18 @@ the engine, both data structures and the debug checks::
--kconfig_add CONFIG_DEBUG_RCU_TXN=y \
'rcu_txn*'
+The litmus tests ``Documentation/litmus-tests/rcu/RCU-txn+*.litmus`` model the
+memory ordering of the engine and of ``rcu_txn_list`` with the Linux kernel
+memory model. Each test checks an outcome that the code forbids, and herd7
+finds that it never happens. Error-injection variants validate that the model
+catches a missing ordering: each drops one ordering that its test relies on, and
+herd7 then finds the outcome possible. With herd7 installed as
+``tools/memory-model/README`` says::
+
+ cd tools/memory-model
+ scripts/checktheselitmus.sh --timeout 1h -- \
+ ../../Documentation/litmus-tests/rcu/RCU-txn+*.litmus
+
API
===
diff --git a/Documentation/litmus-tests/README b/Documentation/litmus-tests/README
index 6c666f3422ea..e18995b421a8 100644
--- a/Documentation/litmus-tests/README
+++ b/Documentation/litmus-tests/README
@@ -78,3 +78,40 @@ RCU+sync+read.litmus
RCU+sync+free.litmus
Both the above litmus tests demonstrate the RCU grace period guarantee
that an RCU read-side critical section can never span a grace period.
+
+RCU-txn+install.litmus
+ Demonstrates that the install of an RCU pseudo-transaction
+ (include/linux/rcu_txn.h) publishes its record: a reader that finds a
+ proxy reads the record initialized.
+
+RCU-txn+monotonic+dep.litmus
+RCU-txn+monotonic+acquire.litmus
+ Demonstrate that the slots of one RCU pseudo-transaction resolve old,
+ then new, never new then old, along a chain of dependent loads or with
+ a load-acquire between the loads.
+
+RCU-txn+list+step-back.litmus
+ Demonstrates that a reader of an rcu_txn_list that steps forward onto
+ a node being inserted, then back, never skips it.
+ NOTE: herd7 takes about half an hour on it, and on its variants.
+
+RCU-txn+sync+free.litmus
+ Demonstrates that a reader never reads the record of an RCU
+ pseudo-transaction once it is freed after a grace period.
+
+RCU-txn+publish.litmus
+RCU-txn+remove.litmus
+ Demonstrate RCU pseudo-transactions end to end, the writer doing every
+ step: a reader sees the record initialized and a published object
+ built, and never reads a removed object, nor the record, once they are
+ freed after a grace period.
+
+RCU-txn+*+once-*.litmus
+RCU-txn+monotonic+unordered.litmus
+RCU-txn+remove+nolock.litmus
+RCU-txn+remove+nosync.litmus
+RCU-txn+sync+free+nolock.litmus
+RCU-txn+sync+free+nosync.litmus
+ Error injection into the above RCU-txn tests. Each variant drops one
+ ordering that its test relies on, and herd7 then finds the outcome
+ possible, which validates that the model catches the missing ordering.
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+install+once-install.litmus b/Documentation/litmus-tests/rcu/RCU-txn+install+once-install.litmus
new file mode 100644
index 000000000000..7aca7fc07162
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+install+once-install.litmus
@@ -0,0 +1,55 @@
+C RCU-txn+install+once-install
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the install, a reader can find the proxy
+ * before the record it points at is written.
+ *
+ * Error injection into RCU-txn+install: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int n = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *n)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, n);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ WRITE_ONCE(*a, rg); // then the proxy, unordered
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) { // a proxy: resolve it
+ r1 = READ_ONCE(*r0); // proxy->group
+ r2 = smp_load_acquire(r1); // group->selector
+ if (r2) {
+ t = READ_ONCE(*rn); // proxy->ptr[1]
+ } else {
+ t = READ_ONCE(*ro); // proxy->ptr[0]
+ }
+ } else {
+ t = r0;
+ }
+ rcu_read_unlock();
+}
+
+exists (1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+install.litmus b/Documentation/litmus-tests/rcu/RCU-txn+install.litmus
new file mode 100644
index 000000000000..aafda3b7bc42
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+install.litmus
@@ -0,0 +1,66 @@
+C RCU-txn+install
+
+(*
+ * Result: Never
+ *
+ * The install of a proxy publishes its record: a reader that finds the proxy
+ * in the slot reads the record's group, the selector's initial 0 and the
+ * record's old target, never what was there before.
+ *
+ * The record is modeled by three variables: rg holds its group (the address
+ * of the selector), ro and rn its old and new targets. The slot holds the
+ * address of rg when it holds the proxy; the reader's tag test is the
+ * comparison with that address. Before the writer runs, every one of them
+ * holds garbage: junk, or 2 for the selector.
+ *
+ * The code resolves a proxy by indexing the record's targets with the
+ * selector, an address dependency. herd7 has no arrays, so the reader here
+ * branches on the selector instead, and a branch orders no later load: the
+ * model orders less than the code, and an outcome that it forbids, the code
+ * forbids as well.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int n = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *n)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, n);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) { // a proxy: resolve it
+ r1 = READ_ONCE(*r0); // proxy->group
+ r2 = smp_load_acquire(r1); // group->selector
+ if (r2) {
+ t = READ_ONCE(*rn); // proxy->ptr[1]
+ } else {
+ t = READ_ONCE(*ro); // proxy->ptr[0]
+ }
+ } else {
+ t = r0;
+ }
+ rcu_read_unlock();
+}
+
+exists (1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-flip.litmus b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-flip.litmus
new file mode 100644
index 000000000000..200adbb9b25b
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-flip.litmus
@@ -0,0 +1,78 @@
+C RCU-txn+list+step-back+once-flip
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the flip, a reader can resolve P->next to its
+ * new target, then find N->prev still holding its old one.
+ *
+ * Error injection into RCU-txn+list+step-back: everything else is as there.
+ *)
+
+{
+int sel = 0;
+int nodep = 0;
+int *nprev = &nodep;
+int *pnext = &nprev;
+int *xnext = &nprev;
+int *rpg = &sel;
+int *rpo = &nprev;
+int *rpn = &xnext;
+int *rng = &sel;
+int *rno = &nodep;
+int *rnn = &xnext;
+}
+
+P0(int **pnext, int **nprev, int **xnext, int **rpg, int **rng, int *sel)
+{
+ rcu_assign_pointer(*pnext, rpg); // install
+ rcu_assign_pointer(*nprev, rng);
+ WRITE_ONCE(*sel, 1); // flip, unordered
+ rcu_assign_pointer(*pnext, xnext); // settle
+ rcu_assign_pointer(*nprev, xnext);
+}
+
+P1(int **pnext, int **xnext, int **rpg, int **rpo, int **rpn, int **rng, int **rno, int **rnn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int *u;
+ int *s0;
+ int *s1;
+ int s2;
+ int *back;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*pnext); // forward from P
+ if (r0 == rpg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rpn);
+ } else {
+ t = READ_ONCE(*rpo);
+ }
+ } else {
+ t = r0;
+ }
+ if (t == xnext) {
+ u = rcu_dereference(*t); // forward from X, onto N
+ s0 = rcu_dereference(*u); // back from N
+ if (s0 == rng) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ back = READ_ONCE(*rnn);
+ } else {
+ back = READ_ONCE(*rno);
+ }
+ } else {
+ back = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:t=xnext /\ 1:back=nodep)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-settle.litmus b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-settle.litmus
new file mode 100644
index 000000000000..0ead3863fc9d
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back+once-settle.litmus
@@ -0,0 +1,78 @@
+C RCU-txn+list+step-back+once-settle
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the settle of P->next, a reader can find the
+ * new target settled there, then N->prev still holding its old one.
+ *
+ * Error injection into RCU-txn+list+step-back: everything else is as there.
+ *)
+
+{
+int sel = 0;
+int nodep = 0;
+int *nprev = &nodep;
+int *pnext = &nprev;
+int *xnext = &nprev;
+int *rpg = &sel;
+int *rpo = &nprev;
+int *rpn = &xnext;
+int *rng = &sel;
+int *rno = &nodep;
+int *rnn = &xnext;
+}
+
+P0(int **pnext, int **nprev, int **xnext, int **rpg, int **rng, int *sel)
+{
+ rcu_assign_pointer(*pnext, rpg); // install
+ rcu_assign_pointer(*nprev, rng);
+ smp_store_release(sel, 1); // flip
+ WRITE_ONCE(*pnext, xnext); // settle, unordered
+ rcu_assign_pointer(*nprev, xnext);
+}
+
+P1(int **pnext, int **xnext, int **rpg, int **rpo, int **rpn, int **rng, int **rno, int **rnn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int *u;
+ int *s0;
+ int *s1;
+ int s2;
+ int *back;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*pnext); // forward from P
+ if (r0 == rpg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rpn);
+ } else {
+ t = READ_ONCE(*rpo);
+ }
+ } else {
+ t = r0;
+ }
+ if (t == xnext) {
+ u = rcu_dereference(*t); // forward from X, onto N
+ s0 = rcu_dereference(*u); // back from N
+ if (s0 == rng) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ back = READ_ONCE(*rnn);
+ } else {
+ back = READ_ONCE(*rno);
+ }
+ } else {
+ back = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:t=xnext /\ 1:back=nodep)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+list+step-back.litmus b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back.litmus
new file mode 100644
index 000000000000..85d96e1ebec6
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+list+step-back.litmus
@@ -0,0 +1,85 @@
+C RCU-txn+list+step-back
+
+(*
+ * Result: Never
+ *
+ * An insertion of X between P and N into an rcu_txn_list switches P->next and
+ * N->prev in one transaction. A reader standing on P that steps forward onto
+ * X, forward again onto N, then back, comes back to X: it never finds the new
+ * forward edge and then the old backward one, which would take it back to P
+ * past X.
+ *
+ * A node is modeled by the address of the link a reader follows from it: X by
+ * its next link xnext, built pointing at N before the insertion, and N by its
+ * prev link nprev. P, which the reader only stands on, is nodep. P's next
+ * link is pnext. The records of the two slots are rp* and rn* (see
+ * RCU-txn+install).
+ *)
+
+{
+int sel = 0;
+int nodep = 0;
+int *nprev = &nodep;
+int *pnext = &nprev;
+int *xnext = &nprev;
+int *rpg = &sel;
+int *rpo = &nprev;
+int *rpn = &xnext;
+int *rng = &sel;
+int *rno = &nodep;
+int *rnn = &xnext;
+}
+
+P0(int **pnext, int **nprev, int **xnext, int **rpg, int **rng, int *sel)
+{
+ rcu_assign_pointer(*pnext, rpg); // install
+ rcu_assign_pointer(*nprev, rng);
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*pnext, xnext); // settle
+ rcu_assign_pointer(*nprev, xnext);
+}
+
+P1(int **pnext, int **xnext, int **rpg, int **rpo, int **rpn, int **rng, int **rno, int **rnn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int *u;
+ int *s0;
+ int *s1;
+ int s2;
+ int *back;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*pnext); // forward from P
+ if (r0 == rpg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rpn);
+ } else {
+ t = READ_ONCE(*rpo);
+ }
+ } else {
+ t = r0;
+ }
+ if (t == xnext) {
+ u = rcu_dereference(*t); // forward from X, onto N
+ s0 = rcu_dereference(*u); // back from N
+ if (s0 == rng) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ back = READ_ONCE(*rnn);
+ } else {
+ back = READ_ONCE(*rno);
+ }
+ } else {
+ back = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:t=xnext /\ 1:back=nodep)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire+once-resolve.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire+once-resolve.litmus
new file mode 100644
index 000000000000..8e8114bf66b6
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire+once-resolve.litmus
@@ -0,0 +1,79 @@
+C RCU-txn+monotonic+acquire+once-resolve
+
+(*
+ * Result: Sometimes
+ *
+ * Without the load-acquire of the selector when it resolves slot a, nothing
+ * orders the reader's load of slot b after its load of the selector: the
+ * reader can resolve slot a to its new target, then find slot b still holding
+ * its old one.
+ *
+ * Error injection into RCU-txn+monotonic+acquire: everything else is as there.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, b); // settle: a now points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = smp_load_acquire(a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = READ_ONCE(*r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*b);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire.litmus
new file mode 100644
index 000000000000..681e42b1d469
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+acquire.litmus
@@ -0,0 +1,81 @@
+C RCU-txn+monotonic+acquire
+
+(*
+ * Result: Never
+ *
+ * As RCU-txn+monotonic+dep, for a reader that loads slot b independently of
+ * slot a, and orders its two loads with a load-acquire of slot a.
+ *
+ * One transaction switches slot a from aold to anew and slot b from bold to
+ * bnew: the writer installs both proxies, flips the selector, then settles
+ * both slots. ra* and rb* are the two records (see RCU-txn+install). A node
+ * is modeled by the address of the link a reader follows from it: anew, the
+ * new target of slot a, is the node whose link is slot b.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, b); // settle: a now points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = smp_load_acquire(a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*b);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-flip.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-flip.litmus
new file mode 100644
index 000000000000..4eafbb122e20
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-flip.litmus
@@ -0,0 +1,77 @@
+C RCU-txn+monotonic+dep+once-flip
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the flip, a reader can resolve slot a to its
+ * new target, then find slot b still holding its old one.
+ *
+ * Error injection into RCU-txn+monotonic+dep: everything else is as there.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ WRITE_ONCE(*sel, 1); // flip, unordered
+ rcu_assign_pointer(*a, b); // settle: a now points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*ta);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-settle.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-settle.litmus
new file mode 100644
index 000000000000..7052aa325694
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep+once-settle.litmus
@@ -0,0 +1,77 @@
+C RCU-txn+monotonic+dep+once-settle
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the settle of slot a, a reader can find the
+ * new target settled there, then slot b still holding its old one.
+ *
+ * Error injection into RCU-txn+monotonic+dep: everything else is as there.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ smp_store_release(sel, 1); // flip
+ WRITE_ONCE(*a, b); // settle, unordered: a points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*ta);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep.litmus
new file mode 100644
index 000000000000..e60b43e9f76b
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+dep.litmus
@@ -0,0 +1,83 @@
+C RCU-txn+monotonic+dep
+
+(*
+ * Result: Never
+ *
+ * A reader that resolves slot a to its new target, then follows that target to
+ * slot b, resolves slot b to its new target as well: the slots of one
+ * transaction resolve old, then new, never new then old, along a chain of
+ * dependent loads.
+ *
+ * One transaction switches slot a from aold to anew and slot b from bold to
+ * bnew: the writer installs both proxies, flips the selector, then settles
+ * both slots. ra* and rb* are the two records (see RCU-txn+install). A node
+ * is modeled by the address of the link a reader follows from it: anew, the
+ * new target of slot a, is the node whose link is slot b.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, b); // settle: a now points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*ta);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+monotonic+unordered.litmus b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+unordered.litmus
new file mode 100644
index 000000000000..7591d5c048da
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+monotonic+unordered.litmus
@@ -0,0 +1,84 @@
+C RCU-txn+monotonic+unordered
+
+(*
+ * Result: Sometimes
+ *
+ * Without a dependency or a load-acquire between its two loads, a reader can
+ * resolve slot a to its new target, then find slot b still holding its old one.
+ *
+ * Error injection into RCU-txn+monotonic+dep and RCU-txn+monotonic+acquire:
+ * the reader loads slot b independently of slot a.
+ *
+ * One transaction switches slot a from aold to anew and slot b from bold to
+ * bnew: the writer installs both proxies, flips the selector, then settles
+ * both slots. ra* and rb* are the two records (see RCU-txn+install). A node
+ * is modeled by the address of the link a reader follows from it: anew, the
+ * new target of slot a, is the node whose link is slot b.
+ *)
+
+{
+int sel = 0;
+int aold = 0;
+int bold = 0;
+int bnew = 0;
+int *b = &bold;
+int *a = &aold;
+int *rag = &sel;
+int *rao = &aold;
+int *ran = &b;
+int *rbg = &sel;
+int *rbo = &bold;
+int *rbn = &bnew;
+}
+
+P0(int **a, int **b, int **rag, int **rbg, int *sel, int *bnew)
+{
+ rcu_assign_pointer(*a, rag); // install
+ rcu_assign_pointer(*b, rbg);
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, b); // settle: a now points to b's node
+ rcu_assign_pointer(*b, bnew);
+}
+
+P1(int **a, int **b, int **rag, int **rao, int **ran, int **rbg, int **rbo, int **rbn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *ta;
+ int *s0;
+ int *s1;
+ int s2;
+ int *tb;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rag) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ ta = READ_ONCE(*ran);
+ } else {
+ ta = READ_ONCE(*rao);
+ }
+ } else {
+ ta = r0;
+ }
+ if (ta == b) {
+ s0 = rcu_dereference(*b);
+ if (s0 == rbg) {
+ s1 = READ_ONCE(*s0);
+ s2 = smp_load_acquire(s1);
+ if (s2) {
+ tb = READ_ONCE(*rbn);
+ } else {
+ tb = READ_ONCE(*rbo);
+ }
+ } else {
+ tb = s0;
+ }
+ }
+ rcu_read_unlock();
+}
+
+exists (1:ta=b /\ 1:tb=bold)
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+publish+once-install.litmus b/Documentation/litmus-tests/rcu/RCU-txn+publish+once-install.litmus
new file mode 100644
index 000000000000..bcfc5a2841d6
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+publish+once-install.litmus
@@ -0,0 +1,60 @@
+C RCU-txn+publish+once-install
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the install, a reader can find the proxy
+ * before the record it points at is written.
+ *
+ * Error injection into RCU-txn+publish: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int n = 0;
+int *a = 0;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n)
+{
+ WRITE_ONCE(*n, 1); // build the new object
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, NULL); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, n);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ WRITE_ONCE(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, n); // __rcu_txn_record_settle()
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=n /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+publish+once-settle.litmus b/Documentation/litmus-tests/rcu/RCU-txn+publish+once-settle.litmus
new file mode 100644
index 000000000000..251832e9bbb3
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+publish+once-settle.litmus
@@ -0,0 +1,60 @@
+C RCU-txn+publish+once-settle
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the settle, a reader can find the new object
+ * in the slot before the object is built.
+ *
+ * Error injection into RCU-txn+publish: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int n = 0;
+int *a = 0;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n)
+{
+ WRITE_ONCE(*n, 1); // build the new object
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, NULL); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, n);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ WRITE_ONCE(*a, n); // __rcu_txn_record_settle()
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=n /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+publish.litmus b/Documentation/litmus-tests/rcu/RCU-txn+publish.litmus
new file mode 100644
index 000000000000..7a619330109f
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+publish.litmus
@@ -0,0 +1,71 @@
+C RCU-txn+publish
+
+(*
+ * Result: Never
+ *
+ * A transaction publishes a new object, end to end: the writer builds the
+ * object, initializes the record, installs the proxy, flips the selector and
+ * settles the slot. A reader that finds the proxy reads the record
+ * initialized, and a reader that reaches the object, through the proxy or in
+ * the slot once settled, reads it built.
+ *
+ * Slot a switches from NULL to the new object n, built by storing 1 to it.
+ * The record is rg, ro and rn (see RCU-txn+install), garbage before the writer
+ * runs: junk, or 2 for the selector.
+ *
+ * The object is built before the install, whose store-release orders it for a
+ * reader that finds the proxy: without the flip's store-release, this outcome
+ * stays forbidden. The flip's store-release orders the installs of a
+ * transaction before its selector, for a reader that resolves several of its
+ * slots: see RCU-txn+monotonic+dep+once-flip.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int n = 0;
+int *a = 0;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n)
+{
+ WRITE_ONCE(*n, 1); // build the new object
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, NULL); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, n);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, n); // __rcu_txn_record_settle()
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=n /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+remove+nolock.litmus b/Documentation/litmus-tests/rcu/RCU-txn+remove+nolock.litmus
new file mode 100644
index 000000000000..d32fe9a34d42
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+remove+nolock.litmus
@@ -0,0 +1,64 @@
+C RCU-txn+remove+nolock
+
+(*
+ * Result: Sometimes
+ *
+ * Without an RCU read-side critical section around the load and the
+ * resolution, the grace period does not wait for the reader, which can then
+ * read the record or the object after it is freed.
+ *
+ * Error injection into RCU-txn+remove: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *junk)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, NULL);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, NULL); // __rcu_txn_record_settle()
+ synchronize_rcu(); // both kfree_rcu()
+ WRITE_ONCE(*o, 0); // the allocator reuses the object
+ WRITE_ONCE(*sel, 2); // and the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=o /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+remove+nosync.litmus b/Documentation/litmus-tests/rcu/RCU-txn+remove+nosync.litmus
new file mode 100644
index 000000000000..76b2ed36b99e
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+remove+nosync.litmus
@@ -0,0 +1,64 @@
+C RCU-txn+remove+nosync
+
+(*
+ * Result: Sometimes
+ *
+ * Without a grace period between the settle and the frees, a reader can read
+ * the record or the object after it is freed.
+ *
+ * Error injection into RCU-txn+remove: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *junk)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, NULL);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, NULL); // __rcu_txn_record_settle()
+ WRITE_ONCE(*o, 0); // the allocator reuses the object
+ WRITE_ONCE(*sel, 2); // and the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=o /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+remove+once-install.litmus b/Documentation/litmus-tests/rcu/RCU-txn+remove+once-install.litmus
new file mode 100644
index 000000000000..514ff3c12e63
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+remove+once-install.litmus
@@ -0,0 +1,65 @@
+C RCU-txn+remove+once-install
+
+(*
+ * Result: Sometimes
+ *
+ * Without the store-release of the install, a reader can find the proxy
+ * before the record it points at is written.
+ *
+ * Error injection into RCU-txn+remove: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *junk)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, NULL);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ WRITE_ONCE(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, NULL); // __rcu_txn_record_settle()
+ synchronize_rcu(); // both kfree_rcu()
+ WRITE_ONCE(*o, 0); // the allocator reuses the object
+ WRITE_ONCE(*sel, 2); // and the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=o /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+remove.litmus b/Documentation/litmus-tests/rcu/RCU-txn+remove.litmus
new file mode 100644
index 000000000000..a1bedc74c238
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+remove.litmus
@@ -0,0 +1,72 @@
+C RCU-txn+remove
+
+(*
+ * Result: Never
+ *
+ * A transaction removes an object, end to end: the writer initializes the
+ * record, installs the proxy, flips the selector, settles the slot to NULL,
+ * then frees the descriptor and the object after a grace period, as their
+ * kfree_rcu() does. A reader that finds the proxy reads the record initialized
+ * and not yet freed, and a reader that reaches the object, in the slot before
+ * the install or through the proxy, never reads it freed.
+ *
+ * Slot a switches from the object o to NULL. The record is rg, ro and rn (see
+ * RCU-txn+install), garbage before the writer runs: junk, or 2 for the
+ * selector. Freeing stores 0 into the object and garbage into the descriptor,
+ * as the allocator may.
+ *)
+
+{
+int junk = 9;
+int sel = 2;
+int *rg = &junk;
+int *ro = &junk;
+int *rn = &junk;
+int o = 1;
+int *a = &o;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *o, int *junk)
+{
+ WRITE_ONCE(*sel, 0); // __rcu_txn_group_init()
+ WRITE_ONCE(*ro, o); // __rcu_txn_record_set()
+ WRITE_ONCE(*rn, NULL);
+ WRITE_ONCE(*rg, sel); // __rcu_txn_record_install(): r->group,
+ rcu_assign_pointer(*a, rg); // then the proxy
+ smp_store_release(sel, 1); // __rcu_txn_group_commit()
+ rcu_assign_pointer(*a, NULL); // __rcu_txn_record_settle()
+ synchronize_rcu(); // both kfree_rcu()
+ WRITE_ONCE(*o, 0); // the allocator reuses the object
+ WRITE_ONCE(*sel, 2); // and the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+ int d;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ if (t)
+ d = READ_ONCE(*t);
+ rcu_read_unlock();
+}
+
+exists ((1:r0=rg /\ (1:r1=junk \/ 1:r2=2 \/ 1:t=junk)) \/ (1:t=o /\ 1:d=0))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nolock.litmus b/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nolock.litmus
new file mode 100644
index 000000000000..3c4afa728506
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nolock.litmus
@@ -0,0 +1,55 @@
+C RCU-txn+sync+free+nolock
+
+(*
+ * Result: Sometimes
+ *
+ * Without an RCU read-side critical section around the load and the
+ * resolution, the grace period does not wait for the reader, which can then
+ * read the record after it is freed.
+ *
+ * Error injection into RCU-txn+sync+free: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 0;
+int o = 1;
+int n = 1;
+int *rg = &sel;
+int *ro = &o;
+int *rn = &n;
+int *a = &rg;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n, int *junk)
+{
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, n); // settle
+ synchronize_rcu(); // kfree_rcu() of the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+}
+
+exists (1:r0=rg /\ (1:r1=junk \/ 1:t=junk))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nosync.litmus b/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nosync.litmus
new file mode 100644
index 000000000000..85a69ea6edb4
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+sync+free+nosync.litmus
@@ -0,0 +1,55 @@
+C RCU-txn+sync+free+nosync
+
+(*
+ * Result: Sometimes
+ *
+ * Without a grace period between the settle and the free, a reader that
+ * loaded the proxy can read the record after it is freed.
+ *
+ * Error injection into RCU-txn+sync+free: everything else is as there.
+ *)
+
+{
+int junk = 9;
+int sel = 0;
+int o = 1;
+int n = 1;
+int *rg = &sel;
+int *ro = &o;
+int *rn = &n;
+int *a = &rg;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n, int *junk)
+{
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, n); // settle
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ rcu_read_unlock();
+}
+
+exists (1:r0=rg /\ (1:r1=junk \/ 1:t=junk))
diff --git a/Documentation/litmus-tests/rcu/RCU-txn+sync+free.litmus b/Documentation/litmus-tests/rcu/RCU-txn+sync+free.litmus
new file mode 100644
index 000000000000..d9dc0a833993
--- /dev/null
+++ b/Documentation/litmus-tests/rcu/RCU-txn+sync+free.litmus
@@ -0,0 +1,60 @@
+C RCU-txn+sync+free
+
+(*
+ * Result: Never
+ *
+ * A record outlives every reader that may hold its proxy: commit() frees the
+ * descriptor with kfree_rcu() after the settle, and a reader loads a slot and
+ * resolves what it loaded within one RCU read-side critical section. Such a
+ * reader never reads the freed record.
+ *
+ * The record is rg, ro and rn (see RCU-txn+install), installed when the writer
+ * starts. After the flip, the settle and a grace period, the writer stores
+ * junk into the record, as the allocator may once kfree_rcu() has run.
+ *)
+
+{
+int junk = 9;
+int sel = 0;
+int o = 1;
+int n = 1;
+int *rg = &sel;
+int *ro = &o;
+int *rn = &n;
+int *a = &rg;
+}
+
+P0(int **a, int **rg, int **ro, int **rn, int *sel, int *n, int *junk)
+{
+ smp_store_release(sel, 1); // flip
+ rcu_assign_pointer(*a, n); // settle
+ synchronize_rcu(); // kfree_rcu() of the descriptor
+ WRITE_ONCE(*rg, junk);
+ WRITE_ONCE(*ro, junk);
+ WRITE_ONCE(*rn, junk);
+}
+
+P1(int **a, int **rg, int **ro, int **rn)
+{
+ int *r0;
+ int *r1;
+ int r2;
+ int *t;
+
+ rcu_read_lock();
+ r0 = rcu_dereference(*a);
+ if (r0 == rg) {
+ r1 = READ_ONCE(*r0);
+ r2 = smp_load_acquire(r1);
+ if (r2) {
+ t = READ_ONCE(*rn);
+ } else {
+ t = READ_ONCE(*ro);
+ }
+ } else {
+ t = r0;
+ }
+ rcu_read_unlock();
+}
+
+exists (1:r0=rg /\ (1:r1=junk \/ 1:t=junk))
diff --git a/MAINTAINERS b/MAINTAINERS
index f34040369eb4..e0dd49be20f9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -22858,6 +22858,7 @@ M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
L: rcu@vger.kernel.org
S: Maintained
F: Documentation/RCU/rcu_txn.rst
+F: Documentation/litmus-tests/rcu/RCU-txn+*
F: include/linux/rcu_txn*.h
F: lib/rcu_txn*
F: lib/tests/rcu_txn_kunit.c
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 09/12] rcu_txn: Test a list writer against concurrent readers
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (7 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 08/12] Documentation/litmus-tests: Add RCU pseudo-transaction tests Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 10/12] rcu_txn: Test a commit from atomic context Mathieu Desnoyers
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Run one writer on a key-sorted rcu_txn_list while kthreads walk it
forward and backward.
Each walk must see sorted keys. That alone cannot catch an operation
whose two edges are published separately, since such a list is still
sorted in each direction taken alone. So, during phases where the
writer only inserts, each forward step p -> q also loads the prev of q
and requires its key to be at least that of p. That catches an insert
that publishes its forward edge before its backward one, as
list_add_rcu() does; the other order cannot be told from a completed
insert by key.
The test is marked slow. It needs more than one CPU to exercise real
concurrency.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
Documentation/RCU/rcu_txn.rst | 4 +-
lib/tests/rcu_txn_kunit.c | 307 ++++++++++++++++++++++++++++++++++
2 files changed, 310 insertions(+), 1 deletion(-)
diff --git a/Documentation/RCU/rcu_txn.rst b/Documentation/RCU/rcu_txn.rst
index 7bac4bd02456..133ef1ade43e 100644
--- a/Documentation/RCU/rcu_txn.rst
+++ b/Documentation/RCU/rcu_txn.rst
@@ -212,9 +212,11 @@ Testing
=======
``CONFIG_RCU_TXN_KUNIT_TEST`` builds the KUnit suite ``rcu_txn``, which covers
-the engine, both data structures and the debug checks::
+the engine, both data structures and the debug checks, and runs one writer
+against concurrent readers of a list::
./tools/testing/kunit/kunit.py run --arch=x86_64 \
+ --qemu_args="-smp 4" \
--kconfig_add CONFIG_RCU_TXN_KUNIT_TEST=y \
--kconfig_add CONFIG_DEBUG_KERNEL=y \
--kconfig_add CONFIG_DEBUG_RCU_TXN=y \
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index 9b67caaa5db7..fa01dcb7df2b 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -7,11 +7,15 @@
#include <kunit/test.h>
#include <linux/bug.h>
+#include <linux/kthread.h>
+#include <linux/minmax.h>
#include <linux/rcu_txn.h>
#include <linux/rcu_txn_hlist.h>
#include <linux/rcu_txn_list.h>
#include <linux/rcupdate.h>
+#include <linux/sched.h>
#include <linux/sched/task.h>
+#include <linux/slab.h>
#include <linux/string.h>
#include "../rcu_txn_internal.h" /* the steps of a commit */
@@ -1686,6 +1690,308 @@ static void rcu_txn_test_list_debug_disjoint_compose(struct kunit *test)
list_expect(test, &head, keys, ARRAY_SIZE(keys));
}
+/*
+ * One writer mutating a key-sorted list while several readers iterate it
+ * forward and backward.
+ *
+ * Two invariants, because the first one alone cannot see the property this
+ * test exists for.
+ *
+ * (1) Monotonicity, per direction. The writer keeps the list sorted by key at
+ * all times, so a forward walk must see strictly increasing keys and a reverse
+ * walk strictly decreasing ones. Out-of-range keys and runaway walks are
+ * caught here too.
+ *
+ * What invariant (1) cannot catch is a torn two-edge publish. Take an insert
+ * of X between P and Q publishing only its forward edge: the forward ring is
+ * P -> X -> Q, still strictly increasing, and the reverse ring is Q -> P, still
+ * strictly decreasing. Every torn state of a two-edge operation is sorted in
+ * each direction separately.
+ *
+ * (2) Cross-direction agreement, which is where a torn publish shows. After
+ * each forward step p -> q the reader also resolves the prev of q and requires
+ * its key to be at least p's. A forward-only torn insert yields the
+ * predecessor that is not back-linked yet, whose key is smaller than p's. A
+ * reverse-only torn insert is not caught: the prev of q points to X, whose key
+ * lies between p's and q's, as after a completed insert. list_add_rcu()
+ * publishes the forward edge first, which is the tear this check is for.
+ *
+ * Invariant (2) holds only in a phase without deletes. Under concurrent
+ * deletes it is simply false: delete p between the reader's two loads and the
+ * prev of q legitimately becomes the node before p, whose key is smaller. With
+ * inserts alone the legal outcomes are p itself or a node inserted between p
+ * and q, both with a key at least p's.
+ *
+ * The phase change is a handshake. The writer numbers its phases, the odd ones
+ * allowing deletes, and after publishing an odd phase waits for every reader
+ * to acknowledge that number. A reader samples the phase at the top of an
+ * iteration and acknowledges an odd one there, so that no cross-checking walk
+ * is in flight when the writer proceeds; an acknowledgment of an earlier
+ * phase does not count. Store-release and load-acquire order the walks and
+ * the deletes against the handshake.
+ */
+
+#define CONC_KEY_MAX 100 /* keys are 1..CONC_KEY_MAX */
+#define CONC_READERS 4
+#define CONC_STEP_LIMIT (CONC_KEY_MAX + 8) /* runaway-walk guard */
+#define CONC_INSERT_ROUNDS 20
+#define CONC_MIXED_OPS 20000
+
+struct cn {
+ struct rcu_txn_list_node node;
+ struct rcu_head rcu;
+ int key;
+};
+
+struct conc {
+ struct rcu_txn_list_head head;
+ unsigned long phase; /* odd: deletes as well as inserts */
+ struct cn *live[CONC_KEY_MAX + 1]; /* the writer's: node per key */
+};
+
+struct conc_reader {
+ struct conc *conc;
+ struct task_struct *task;
+ long walks;
+ long violations; /* invariant (1) */
+ long cross_checks;
+ long cross_violations; /* invariant (2) */
+ unsigned long ack; /* last odd phase seen: no cross-check in flight */
+};
+
+static int conc_reader_fn(void *arg)
+{
+ struct conc_reader *r = arg;
+ struct rcu_txn_list_head *head = &r->conc->head;
+
+ while (!kthread_should_stop()) {
+ /* Pairs with the writer's store-release of the phase. */
+ unsigned long phase = smp_load_acquire(&r->conc->phase);
+ bool insert_only = !(phase & 1);
+ struct rcu_txn_list_node *p;
+ int prev_key, steps;
+
+ if (!insert_only) {
+ /* Pairs with the writer's load-acquire of the ack. */
+ smp_store_release(&r->ack, phase);
+ }
+
+ /* Forward: strictly increasing keys, bounded. */
+ rcu_read_lock();
+ prev_key = 0;
+ steps = 0;
+ rcu_txn_list_for_each_rcu(p, head) {
+ int key = container_of(p, struct cn, node)->key;
+
+ if (key < 1 || key > CONC_KEY_MAX || key <= prev_key ||
+ ++steps > CONC_STEP_LIMIT) {
+ r->violations++;
+ break;
+ }
+ /*
+ * Invariant (2). Skipped for the first step, whose
+ * predecessor is the head.
+ */
+ if (insert_only && prev_key) {
+ struct rcu_txn_list_node *back =
+ rcu_txn_list_prev_rcu(p);
+
+ r->cross_checks++;
+ if (back != &head->node &&
+ container_of(back, struct cn,
+ node)->key < prev_key)
+ r->cross_violations++;
+ }
+ prev_key = key;
+ }
+ rcu_read_unlock();
+
+ /* Reverse: strictly decreasing keys, bounded. */
+ rcu_read_lock();
+ prev_key = CONC_KEY_MAX + 1;
+ steps = 0;
+ rcu_txn_list_for_each_reverse_rcu(p, head) {
+ int key = container_of(p, struct cn, node)->key;
+
+ if (key < 1 || key > CONC_KEY_MAX || key >= prev_key ||
+ ++steps > CONC_STEP_LIMIT) {
+ r->violations++;
+ break;
+ }
+ prev_key = key;
+ }
+ rcu_read_unlock();
+
+ r->walks += 2;
+ cond_resched();
+ }
+ return 0;
+}
+
+/* Insert @key, keeping the list sorted. Returns false if it cannot. */
+static bool conc_insert(struct conc *c, int key)
+{
+ struct rcu_txn_list_node *succ = &c->head.node; /* default: last */
+ struct cn *n = kmalloc(sizeof(*n), GFP_KERNEL);
+ int i;
+
+ if (!n)
+ return false;
+ n->key = key;
+ for (i = key + 1; i <= CONC_KEY_MAX; i++) {
+ if (c->live[i]) {
+ succ = &c->live[i]->node;
+ break;
+ }
+ }
+ if (rcu_txn_list_add_before_rcu(&n->node, succ, GFP_KERNEL)) {
+ kfree(n);
+ return false;
+ }
+ c->live[key] = n;
+ return true;
+}
+
+static bool conc_del(struct conc *c, int key)
+{
+ struct cn *n = c->live[key];
+
+ if (rcu_txn_list_del_rcu(&n->node, GFP_KERNEL))
+ return false;
+ c->live[key] = NULL;
+ kfree_rcu(n, rcu);
+ return true;
+}
+
+/* Replace the node of @key by a new one of the same key: order is kept. */
+static bool conc_replace(struct conc *c, int key)
+{
+ struct cn *old = c->live[key];
+ struct cn *n = kmalloc(sizeof(*n), GFP_KERNEL);
+
+ if (!n)
+ return false;
+ n->key = key;
+ if (rcu_txn_list_replace_rcu(&old->node, &n->node, GFP_KERNEL)) {
+ kfree(n);
+ return false;
+ }
+ c->live[key] = n;
+ kfree_rcu(old, rcu);
+ return true;
+}
+
+static void rcu_txn_test_list_concurrent(struct kunit *test)
+{
+ struct conc_reader *readers;
+ long walks = 0, violations = 0, cross_checks = 0, cross_violations = 0;
+ u32 rng = 2463534242u;
+ bool ok = true;
+ struct conc *c;
+ int i, key, round;
+
+ c = kunit_kzalloc(test, sizeof(*c), GFP_KERNEL);
+ readers = kunit_kcalloc(test, CONC_READERS, sizeof(*readers),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, c);
+ KUNIT_ASSERT_NOT_NULL(test, readers);
+ rcu_txn_list_init(&c->head);
+
+ for (i = 0; i < CONC_READERS; i++) {
+ readers[i].conc = c;
+ readers[i].task = kthread_run(conc_reader_fn, &readers[i],
+ "rcu_txn_reader/%d", i);
+ if (IS_ERR(readers[i].task)) {
+ readers[i].task = NULL;
+ ok = false;
+ }
+ }
+
+ /*
+ * An even phase, repeated: fill the list one key at a time, with
+ * inserts only, so that the readers cross-check against concurrent
+ * inserts. Then hand over to an odd phase and empty it with deletes.
+ */
+ for (round = 0; ok && round < CONC_INSERT_ROUNDS; round++) {
+ int order[CONC_KEY_MAX], other;
+
+ for (i = 0; i < CONC_KEY_MAX; i++)
+ order[i] = i + 1;
+ for (i = CONC_KEY_MAX - 1; i > 0; i--) { /* shuffle */
+ rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
+ other = rng % (i + 1);
+ swap(order[i], order[other]);
+ }
+
+ /* An even phase, after the deletes of the previous one. */
+ smp_store_release(&c->phase, (c->phase | 1) + 1);
+
+ for (i = 0; ok && i < CONC_KEY_MAX; i++) {
+ ok = conc_insert(c, order[i]);
+ /*
+ * Let the readers walk in between. On one cpu they
+ * only run in this phase if the writer sleeps in it.
+ */
+ if (i == CONC_KEY_MAX / 2)
+ schedule_timeout_uninterruptible(1);
+ else
+ cond_resched();
+ }
+
+ /* An odd phase: wait until no even phase walk is running. */
+ smp_store_release(&c->phase, c->phase + 1);
+ for (i = 0; i < CONC_READERS; i++) {
+ if (!readers[i].task)
+ continue;
+ /* Pairs with the reader's store-release of its ack. */
+ while (smp_load_acquire(&readers[i].ack) != c->phase)
+ schedule_timeout_uninterruptible(1);
+ }
+
+ for (key = 1; ok && key <= CONC_KEY_MAX; key++) {
+ if (c->live[key])
+ ok = conc_del(c, key);
+ cond_resched();
+ }
+ }
+
+ /* Then inserts, deletes and replaces, at random. */
+ for (i = 0; ok && i < CONC_MIXED_OPS; i++) {
+ rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
+ key = 1 + rng % CONC_KEY_MAX;
+
+ if (!c->live[key])
+ ok = conc_insert(c, key);
+ else if (!(rng & 0x300))
+ ok = conc_replace(c, key);
+ else
+ ok = conc_del(c, key);
+ cond_resched();
+ }
+
+ for (i = 0; i < CONC_READERS; i++) {
+ if (readers[i].task)
+ kthread_stop(readers[i].task);
+ walks += readers[i].walks;
+ violations += readers[i].violations;
+ cross_checks += readers[i].cross_checks;
+ cross_violations += readers[i].cross_violations;
+ }
+ for (key = 1; key <= CONC_KEY_MAX; key++)
+ if (c->live[key])
+ conc_del(c, key);
+
+ kunit_info(test, "%ld walks, %ld cross-direction checks\n", walks,
+ cross_checks);
+ KUNIT_EXPECT_TRUE(test, ok);
+ KUNIT_EXPECT_GT(test, walks, 0);
+ /* Forward and reverse walks stayed sorted. */
+ KUNIT_EXPECT_EQ(test, violations, 0);
+ /* No two-edge publish was ever seen torn. */
+ KUNIT_EXPECT_GT(test, cross_checks, 0);
+ KUNIT_EXPECT_EQ(test, cross_violations, 0);
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -1744,6 +2050,7 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_list_move_phases),
KUNIT_CASE(rcu_txn_test_list_compose_moves),
KUNIT_CASE(rcu_txn_test_list_debug_disjoint_compose),
+ KUNIT_CASE_SLOW(rcu_txn_test_list_concurrent),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 10/12] rcu_txn: Test a commit from atomic context
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (8 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 09/12] rcu_txn: Test a list writer against concurrent readers Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 11/12] samples: Add a cache on RCU pseudo-transactions under one mutex Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 12/12] samples: Add a cache on RCU pseudo-transactions with a lock per node Mathieu Desnoyers
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
A hash table with hlist_bl buckets excludes its writers with a bit
spinlock. On PREEMPT_RT a bit spinlock keeps the semantics of a raw
spinlock: nothing under it may take a sleeping lock, which rules out the
memory allocator.
Commit a transaction of two records under a bit spinlock, another under
a raw spinlock with interrupts off, and a single record and a cancel
under the bit spinlock, and check that neither lockdep nor the
might_sleep() checks report anything. The transactions reserve their
records before they take the lock, so recording and committing allocate
nothing, and the descriptor of a commit that installed proxies goes to
kfree_rcu(), which is allowed there. Each commit's return value is
checked once the lock is released, since a failed expectation allocates.
The test is only meaningful on a PREEMPT_RT kernel with lockdep,
PROVE_RAW_LOCK_NESTING and DEBUG_ATOMIC_SLEEP. It passes there, and
fails as expected when a spinlock_t is taken under the bit spinlock.
The last two cases release a descriptor that installed no proxy, which
has to be kfree_rcu() on PREEMPT_RT. They would pass with a kfree() as
well: a kfree() under a bit spinlock on PREEMPT_RT was reported by
neither lockdep nor the might_sleep() checks when tried.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
lib/tests/rcu_txn_kunit.c | 85 +++++++++++++++++++++++++++++++++++++++
1 file changed, 85 insertions(+)
diff --git a/lib/tests/rcu_txn_kunit.c b/lib/tests/rcu_txn_kunit.c
index fa01dcb7df2b..c534d8962960 100644
--- a/lib/tests/rcu_txn_kunit.c
+++ b/lib/tests/rcu_txn_kunit.c
@@ -7,8 +7,11 @@
#include <kunit/test.h>
#include <linux/bug.h>
+#include <linux/debug_locks.h>
#include <linux/kthread.h>
+#include <linux/list_bl.h>
#include <linux/minmax.h>
+#include <linux/panic.h>
#include <linux/rcu_txn.h>
#include <linux/rcu_txn_hlist.h>
#include <linux/rcu_txn_list.h>
@@ -1992,6 +1995,87 @@ static void rcu_txn_test_list_concurrent(struct kunit *test)
KUNIT_EXPECT_EQ(test, cross_violations, 0);
}
+/* Did the kernel report something since @tainted and @locks were sampled? */
+static void expect_no_report(struct kunit *test, bool tainted, int locks)
+{
+ KUNIT_EXPECT_EQ(test, test_taint(TAINT_WARN), tainted);
+ KUNIT_EXPECT_EQ(test, debug_locks, locks);
+}
+
+/*
+ * A commit from atomic context: under a bit spinlock, which is how a hash
+ * table with hlist_bl buckets excludes its writers, and under a raw spinlock.
+ * On PREEMPT_RT neither may take a sleeping lock, which lockdep and the
+ * might_sleep() checks would report.
+ *
+ * The transaction reserves its records before it takes the lock, so recording
+ * and committing allocate nothing, and the descriptor of a commit that
+ * installed proxies is handed to kfree_rcu().
+ *
+ * A descriptor that installed none is released under the lock as well, by a
+ * commit of one record or by a cancel. On PREEMPT_RT that has to be
+ * kfree_rcu() too, but these checks did not report a kfree() there when
+ * tried: those two cases only exercise the path.
+ */
+static void rcu_txn_test_commit_atomic(struct kunit *test)
+{
+ static DEFINE_RAW_SPINLOCK(raw_lock);
+ void __rcu *s1 = RCU_INITIALIZER(T(0));
+ void __rcu *s2 = RCU_INITIALIZER(T(1));
+ bool tainted = test_taint(TAINT_WARN);
+ struct hlist_bl_head bucket;
+ int locks = debug_locks;
+ unsigned long flags;
+ struct rcu_txn txn;
+ int err;
+
+ INIT_HLIST_BL_HEAD(&bucket);
+
+ /* Two records, under a bit spinlock. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ hlist_bl_lock(&bucket);
+ rcu_txn_record(&txn, &s1, T(0), T(10), TAG);
+ rcu_txn_record(&txn, &s2, T(1), T(11), TAG);
+ err = rcu_txn_commit(&txn);
+ hlist_bl_unlock(&bucket);
+ KUNIT_EXPECT_EQ(test, err, 0);
+ expect_no_report(test, tainted, locks);
+
+ /* Two records, under a raw spinlock with interrupts off. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ raw_spin_lock_irqsave(&raw_lock, flags);
+ rcu_txn_record(&txn, &s1, T(10), T(20), TAG);
+ rcu_txn_record(&txn, &s2, T(11), T(21), TAG);
+ err = rcu_txn_commit(&txn);
+ raw_spin_unlock_irqrestore(&raw_lock, flags);
+ KUNIT_EXPECT_EQ(test, err, 0);
+ expect_no_report(test, tainted, locks);
+
+ /* One record. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ hlist_bl_lock(&bucket);
+ rcu_txn_record(&txn, &s1, T(20), T(40), TAG);
+ err = rcu_txn_commit(&txn);
+ hlist_bl_unlock(&bucket);
+ KUNIT_EXPECT_EQ(test, err, 0);
+ expect_no_report(test, tainted, locks);
+
+ /* A cancel of such a handle. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ KUNIT_ASSERT_TRUE(test, rcu_txn_reserve(&txn, 2));
+ hlist_bl_lock(&bucket);
+ rcu_txn_record(&txn, &s1, T(40), T(50), TAG);
+ rcu_txn_cancel(&txn);
+ hlist_bl_unlock(&bucket);
+ expect_no_report(test, tainted, locks);
+
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s1), T(40));
+ KUNIT_EXPECT_PTR_EQ(test, rcu_access_pointer(s2), T(21));
+}
+
static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_single_record),
KUNIT_CASE(rcu_txn_test_multi_record),
@@ -2051,6 +2135,7 @@ static struct kunit_case rcu_txn_test_cases[] = {
KUNIT_CASE(rcu_txn_test_list_compose_moves),
KUNIT_CASE(rcu_txn_test_list_debug_disjoint_compose),
KUNIT_CASE_SLOW(rcu_txn_test_list_concurrent),
+ KUNIT_CASE(rcu_txn_test_commit_atomic),
{}
};
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 11/12] samples: Add a cache on RCU pseudo-transactions under one mutex
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (9 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 10/12] rcu_txn: Test a commit from atomic context Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
2026-10-16 13:33 ` [RFC PATCH 12/12] samples: Add a cache on RCU pseudo-transactions with a lock per node Mathieu Desnoyers
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Show the hlist and the list of RCU pseudo-transactions together: a
cache whose entries are found by key in a hash table of rcu_txn_hlist
buckets, and kept in an rcu_txn_list as their LRU.
Lookup threads find entries under RCU alone, and mark the ones they find
referenced. Writer threads insert, update and delete entries under one
mutex. An insertion into a full cache evicts the oldest entry, after
moving to the front of the list the referenced entries it finds at its
tail, as a second chance. Each change is one transaction over both
structures: an insertion that evicts links the new entry into its bucket
and at the front of the list, makes the moves, and unlinks the victim
from both, and at any instant the cache holds all of it or none of it.
Its records are reserved before the writer takes the mutex, which keeps
the allocation out of the critical section.
The module runs for duration_ms when it is loaded, then checks that the
hash table and the list hold the same entries, and reports what it did.
An allocation failure stops the writers there.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
Documentation/RCU/rcu_txn.rst | 9 +
MAINTAINERS | 1 +
samples/Kconfig | 13 +
samples/Makefile | 1 +
samples/rcu_txn/Makefile | 2 +
samples/rcu_txn/rcu_txn_cache_mutex.c | 354 ++++++++++++++++++++++++++
6 files changed, 380 insertions(+)
create mode 100644 samples/rcu_txn/Makefile
create mode 100644 samples/rcu_txn/rcu_txn_cache_mutex.c
diff --git a/Documentation/RCU/rcu_txn.rst b/Documentation/RCU/rcu_txn.rst
index 133ef1ade43e..8f63c1ce9c38 100644
--- a/Documentation/RCU/rcu_txn.rst
+++ b/Documentation/RCU/rcu_txn.rst
@@ -193,6 +193,15 @@ section, if it passes them a lockdep condition for that exclusion, such as
backward edge, and a move six, the moved node's own included: each
allocates a descriptor.
+Samples
+=======
+
+``samples/rcu_txn/`` holds a cache whose entries are found by key in a hash
+table of ``rcu_txn_hlist`` buckets, and kept in an ``rcu_txn_list`` as their
+LRU. Its lookups run under RCU alone, and each of its changes is one
+transaction over both structures. ``rcu_txn_cache_mutex`` excludes its
+writers with a single mutex. ``CONFIG_SAMPLE_RCU_TXN`` builds it.
+
Debugging
=========
diff --git a/MAINTAINERS b/MAINTAINERS
index e0dd49be20f9..99e0eb6d7331 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -22862,6 +22862,7 @@ F: Documentation/litmus-tests/rcu/RCU-txn+*
F: include/linux/rcu_txn*.h
F: lib/rcu_txn*
F: lib/tests/rcu_txn_kunit.c
+F: samples/rcu_txn/
RCUTORTURE TEST FRAMEWORK
M: "Paul E. McKenney" <paulmck@kernel.org>
diff --git a/samples/Kconfig b/samples/Kconfig
index a75e8e78330d..db4b5680e34d 100644
--- a/samples/Kconfig
+++ b/samples/Kconfig
@@ -103,6 +103,19 @@ config SAMPLE_KFIFO
If in doubt, say "N" here.
+config SAMPLE_RCU_TXN
+ tristate "Build RCU pseudo-transaction examples -- loadable modules only"
+ depends on m
+ select RCU_TXN
+ help
+ Build sample modules of a cache on RCU pseudo-transactions: a hash
+ table of rcu_txn_hlist buckets, with an rcu_txn_list as its LRU,
+ that threads look entries up in under RCU alone while other threads
+ insert, update, delete and evict them. Each module runs for a while
+ when it is loaded, then reports what it did.
+
+ If in doubt, say "N" here.
+
config SAMPLE_KDB
tristate "Build kdb command example -- loadable modules only"
depends on KGDB_KDB && m
diff --git a/samples/Makefile b/samples/Makefile
index 07641e177bd8..63bbf54e449d 100644
--- a/samples/Makefile
+++ b/samples/Makefile
@@ -18,6 +18,7 @@ subdir-$(CONFIG_SAMPLE_LANDLOCK) += landlock
obj-$(CONFIG_SAMPLE_LIVEPATCH) += livepatch/
subdir-$(CONFIG_SAMPLE_PIDFD) += pidfd
obj-$(CONFIG_SAMPLE_QMI_CLIENT) += qmi/
+obj-$(CONFIG_SAMPLE_RCU_TXN) += rcu_txn/
obj-$(CONFIG_SAMPLE_RPMSG_CLIENT) += rpmsg/
subdir-$(CONFIG_SAMPLE_SECCOMP) += seccomp
subdir-$(CONFIG_SAMPLE_TIMER) += timers
diff --git a/samples/rcu_txn/Makefile b/samples/rcu_txn/Makefile
new file mode 100644
index 000000000000..6d7845352afc
--- /dev/null
+++ b/samples/rcu_txn/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_SAMPLE_RCU_TXN) += rcu_txn_cache_mutex.o
diff --git a/samples/rcu_txn/rcu_txn_cache_mutex.c b/samples/rcu_txn/rcu_txn_cache_mutex.c
new file mode 100644
index 000000000000..58d17b579cba
--- /dev/null
+++ b/samples/rcu_txn/rcu_txn_cache_mutex.c
@@ -0,0 +1,354 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * A cache on RCU pseudo-transactions, its writers under one mutex
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * Entries are found by key in a hash table of rcu_txn_hlist buckets, and kept
+ * in an rcu_txn_list from the most recently inserted to the oldest. Lookup
+ * threads find entries under RCU alone, and mark the ones they find
+ * referenced. Writer threads insert, update and delete entries under one
+ * mutex. The cache holds CACHE_SIZE entries at most: an insertion into a full
+ * cache evicts the oldest entry, after giving a second chance at the front of
+ * the list to the referenced entries it finds at the tail, CACHE_CHANCES of
+ * them at most.
+ *
+ * Each change is one transaction over the hash table and the list together.
+ * An insertion that evicts links the new entry into its bucket and at the
+ * front of the list, moves the entries it gives a second chance, and unlinks
+ * the victim from its bucket and from the list: at any instant, the cache holds
+ * all of it, or none of it. An update replaces the entry in its bucket and
+ * moves the new one to the front of the list.
+ *
+ * Its records are reserved before the writer takes the mutex, which keeps the
+ * allocation out of the critical section.
+ *
+ * The sample runs for duration_ms when it is loaded, then checks that the hash
+ * table and the list hold the same entries, and reports what it did. An
+ * allocation failure stops the writers there.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/delay.h>
+#include <linux/hash.h>
+#include <linux/kthread.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/random.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcu_txn_hlist.h>
+#include <linux/rcu_txn_list.h>
+#include <linux/slab.h>
+
+#define CACHE_HASH_BITS 6
+#define CACHE_KEYS 512 /* keys are 0 to CACHE_KEYS - 1 */
+#define CACHE_SIZE 128 /* entries held at most */
+#define CACHE_CHANCES 4 /* second chances an eviction gives at most */
+#define CACHE_THREADS 16 /* of each kind, at most */
+
+/*
+ * What an insertion records at most: the entry into its bucket (two records,
+ * with its successor's pprev) and at the front of the list (two), the second
+ * chances (six each), and the eviction (four).
+ */
+#define CACHE_RECORDS (4 + 6 * CACHE_CHANCES + 4)
+
+static unsigned int nr_lookups = 2;
+module_param(nr_lookups, uint, 0444);
+MODULE_PARM_DESC(nr_lookups, "Lookup threads");
+
+static unsigned int nr_writers = 2;
+module_param(nr_writers, uint, 0444);
+MODULE_PARM_DESC(nr_writers, "Writer threads");
+
+static unsigned int duration_ms = 2000;
+module_param(duration_ms, uint, 0444);
+MODULE_PARM_DESC(duration_ms, "How long the threads run, from the load");
+
+struct cache_entry {
+ unsigned long key;
+ unsigned long value; /* key + CACHE_KEYS * version */
+ struct rcu_txn_hlist_node hnode; /* in its bucket */
+ struct rcu_txn_list_node lru; /* in the list, newest first */
+ struct rcu_head rcu;
+ bool referenced; /* found since it last moved */
+};
+
+static struct rcu_txn_hlist_head cache_hash[1 << CACHE_HASH_BITS];
+static RCU_TXN_LIST_HEAD(cache_lru);
+static DEFINE_MUTEX(cache_mutex); /* excludes the writers */
+static unsigned int cache_count; /* entries, under cache_mutex */
+
+static atomic_long_t nr_lookup, nr_hit, nr_bad, nr_insert, nr_update,
+ nr_delete, nr_evict, nr_chance, nr_nomem;
+
+/* Set at the first allocation failure, which stops the writers. */
+static bool cache_failed;
+
+static void cache_fail(void)
+{
+ atomic_long_inc(&nr_nomem);
+ WRITE_ONCE(cache_failed, true);
+}
+
+static struct rcu_txn_hlist_head *cache_bucket(unsigned long key)
+{
+ return &cache_hash[hash_long(key, CACHE_HASH_BITS)];
+}
+
+/* Within an RCU read-side critical section, or with the mutex held. */
+static struct cache_entry *cache_find(unsigned long key)
+{
+ struct cache_entry *e;
+
+ rcu_txn_hlist_for_each_entry_rcu(e, cache_bucket(key), hnode,
+ lockdep_is_held(&cache_mutex))
+ if (e->key == key)
+ return e;
+ return NULL;
+}
+
+static int cache_lookup_fn(void *arg)
+{
+ unsigned long lookups = 0, hits = 0, bad = 0;
+
+ while (!kthread_should_stop()) {
+ unsigned long key = get_random_u32_below(CACHE_KEYS);
+ struct cache_entry *e;
+
+ rcu_read_lock();
+ e = cache_find(key);
+ if (e) {
+ hits++;
+ if (e->value % CACHE_KEYS != key)
+ bad++;
+ if (!READ_ONCE(e->referenced))
+ WRITE_ONCE(e->referenced, true);
+ }
+ rcu_read_unlock();
+ lookups++;
+ cond_resched();
+ }
+ atomic_long_add(lookups, &nr_lookup);
+ atomic_long_add(hits, &nr_hit);
+ atomic_long_add(bad, &nr_bad);
+ return 0;
+}
+
+/*
+ * Record the eviction of the oldest entry that no lookup found since it last
+ * moved, after giving the referenced entries at the tail a second chance at
+ * the front. Returns the victim, to free a grace period after the commit.
+ */
+static struct cache_entry *cache_evict_prepare(struct rcu_txn *txn)
+{
+ struct rcu_txn_list_node *head = &cache_lru.node;
+ struct cache_entry *e;
+ unsigned int chances;
+
+ for (chances = 0; ; chances++) {
+ /* The tail as this transaction leaves it so far. */
+ e = container_of(rcu_txn_list_pending(txn, &head->prev),
+ struct cache_entry, lru);
+ if (chances == CACHE_CHANCES || !READ_ONCE(e->referenced))
+ break;
+ WRITE_ONCE(e->referenced, false);
+ rcu_txn_list_move_after_prepare(txn, &e->lru, head);
+ }
+ atomic_long_add(chances, &nr_chance);
+ rcu_txn_hlist_del_prepare(txn, &e->hnode);
+ rcu_txn_list_del_prepare(txn, &e->lru);
+ return e;
+}
+
+/* Insert @key, or update it, at version @version. */
+static void cache_insert(unsigned long key, unsigned long version)
+{
+ struct cache_entry *new, *old, *victim = NULL;
+ struct rcu_txn txn;
+
+ new = kzalloc_obj(*new, GFP_KERNEL);
+ if (!new) {
+ cache_fail();
+ return;
+ }
+ new->key = key;
+ new->value = key + CACHE_KEYS * version;
+
+ /* An allocation failure is sticky: the commit reports it. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, CACHE_RECORDS);
+
+ mutex_lock(&cache_mutex);
+ old = cache_find(key);
+ if (old) {
+ /* The new entry takes the old one's place in its bucket. */
+ rcu_txn_hlist_replace_prepare(&txn, &old->hnode, &new->hnode);
+ rcu_txn_list_del_prepare(&txn, &old->lru);
+ victim = old;
+ } else {
+ if (cache_count == CACHE_SIZE)
+ victim = cache_evict_prepare(&txn);
+ rcu_txn_hlist_add_head_prepare(&txn, &new->hnode,
+ cache_bucket(key));
+ }
+ rcu_txn_list_add_after_prepare(&txn, &new->lru, &cache_lru.node);
+ if (rcu_txn_commit(&txn)) {
+ /* Nothing was published. */
+ mutex_unlock(&cache_mutex);
+ cache_fail();
+ kfree(new);
+ return;
+ }
+ if (!victim)
+ cache_count++;
+ mutex_unlock(&cache_mutex);
+
+ if (old)
+ atomic_long_inc(&nr_update);
+ else
+ atomic_long_inc(victim ? &nr_evict : &nr_insert);
+ if (victim)
+ kfree_rcu(victim, rcu);
+}
+
+static void cache_delete(unsigned long key)
+{
+ struct cache_entry *e;
+ struct rcu_txn txn;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 4);
+
+ mutex_lock(&cache_mutex);
+ e = cache_find(key);
+ if (e) {
+ rcu_txn_hlist_del_prepare(&txn, &e->hnode);
+ rcu_txn_list_del_prepare(&txn, &e->lru);
+ }
+ if (rcu_txn_commit(&txn)) {
+ mutex_unlock(&cache_mutex);
+ cache_fail();
+ return;
+ }
+ if (e)
+ cache_count--;
+ mutex_unlock(&cache_mutex);
+
+ if (e) {
+ atomic_long_inc(&nr_delete);
+ kfree_rcu(e, rcu);
+ }
+}
+
+static int cache_writer_fn(void *arg)
+{
+ unsigned long version = 0;
+
+ while (!kthread_should_stop()) {
+ unsigned long key = get_random_u32_below(CACHE_KEYS);
+
+ /* After an allocation failure, wait to be stopped. */
+ if (READ_ONCE(cache_failed)) {
+ msleep(20);
+ continue;
+ }
+ if (get_random_u32_below(8))
+ cache_insert(key, ++version);
+ else
+ cache_delete(key);
+ cond_resched();
+ }
+ return 0;
+}
+
+/*
+ * With the threads stopped: every entry of the hash table is in its bucket and
+ * in the list, both ways, and there are cache_count of them.
+ */
+static bool cache_check(void)
+{
+ unsigned int in_hash = 0, in_list = 0, in_reverse = 0, i;
+ struct cache_entry *e;
+ bool ok = true;
+
+ rcu_read_lock();
+ for (i = 0; i < ARRAY_SIZE(cache_hash); i++) {
+ rcu_txn_hlist_for_each_entry_rcu(e, &cache_hash[i], hnode) {
+ ok &= cache_find(e->key) == e;
+ in_hash++;
+ }
+ }
+ rcu_txn_list_for_each_entry_rcu(e, &cache_lru, lru) {
+ ok &= cache_find(e->key) == e;
+ in_list++;
+ }
+ rcu_txn_list_for_each_entry_reverse_rcu(e, &cache_lru, lru)
+ in_reverse++;
+ rcu_read_unlock();
+ return ok && in_hash == cache_count && in_list == cache_count &&
+ in_reverse == cache_count;
+}
+
+/*
+ * With the threads stopped, no reader or writer is left: free every entry
+ * without unlinking it, which allocates nothing and so cannot fail.
+ */
+static void cache_free_all(void)
+{
+ struct rcu_txn_list_node *node, *next;
+ unsigned int i;
+
+ for (node = rcu_txn_list_next_check(&cache_lru.node, true);
+ node != &cache_lru.node; node = next) {
+ next = rcu_txn_list_next_check(node, true);
+ kfree(container_of(node, struct cache_entry, lru));
+ }
+ rcu_txn_list_init(&cache_lru);
+ for (i = 0; i < ARRAY_SIZE(cache_hash); i++)
+ rcu_txn_hlist_init(&cache_hash[i]);
+ cache_count = 0;
+}
+
+static int __init cache_init(void)
+{
+ struct task_struct *threads[2 * CACHE_THREADS];
+ unsigned int i, n = 0;
+ bool ok;
+
+ for (i = 0; i < min(nr_lookups, CACHE_THREADS); i++)
+ threads[n++] = kthread_run(cache_lookup_fn, NULL,
+ "rcu_txn_lookup/%u", i);
+ for (i = 0; i < min(nr_writers, CACHE_THREADS); i++)
+ threads[n++] = kthread_run(cache_writer_fn, NULL,
+ "rcu_txn_writer/%u", i);
+ msleep(duration_ms);
+ for (i = 0; i < n; i++)
+ if (!IS_ERR(threads[i]))
+ kthread_stop(threads[i]);
+
+ ok = cache_check();
+ pr_info("%ld lookups, %ld hits, %ld bad; %ld insertions, %ld updates, %ld deletions, %ld evictions, %ld second chances, %ld allocation failures; %u entries\n",
+ atomic_long_read(&nr_lookup), atomic_long_read(&nr_hit),
+ atomic_long_read(&nr_bad), atomic_long_read(&nr_insert),
+ atomic_long_read(&nr_update), atomic_long_read(&nr_delete),
+ atomic_long_read(&nr_evict), atomic_long_read(&nr_chance),
+ atomic_long_read(&nr_nomem), cache_count);
+ if (!ok || atomic_long_read(&nr_bad))
+ pr_err("the hash table and the list disagree\n");
+ if (READ_ONCE(cache_failed))
+ pr_warn("an allocation failed, which stopped the writers\n");
+ cache_free_all();
+ return 0;
+}
+
+static void __exit cache_exit(void)
+{
+}
+
+module_init(cache_init);
+module_exit(cache_exit);
+
+MODULE_DESCRIPTION("A cache on RCU pseudo-transactions, its writers under one mutex");
+MODULE_LICENSE("GPL");
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH 12/12] samples: Add a cache on RCU pseudo-transactions with a lock per node
2026-10-16 13:33 [RFC PATCH 00/12] RCU pseudo-transactions Mathieu Desnoyers
` (10 preceding siblings ...)
2026-10-16 13:33 ` [RFC PATCH 11/12] samples: Add a cache on RCU pseudo-transactions under one mutex Mathieu Desnoyers
@ 2026-10-16 13:33 ` Mathieu Desnoyers
11 siblings, 0 replies; 13+ messages in thread
From: Mathieu Desnoyers @ 2026-10-16 13:33 UTC (permalink / raw)
To: Paul E . McKenney
Cc: linux-kernel, Mathieu Desnoyers, Boqun Feng, Josh Triplett,
Frederic Weisbecker, Neeraj Upadhyay, Joel Fernandes,
Uladzislau Rezki, Steven Rostedt, Lai Jiangshan, Zqiang,
Linus Torvalds, Matthew Wilcox, Christian Brauner,
Greg Kroah-Hartman, Liam R . Howlett, Lorenzo Stoakes,
Alice Ryhl, Andrew Ballance, Peter Zijlstra, Will Deacon,
Mark Rutland, Ingo Molnar, Thomas Gleixner, rcu
Show RCU pseudo-transactions under fine-grained locking: the same cache
as the mutex sample, with writers that exclude each other per node
rather than under one mutex, so that writers updating distinct entries
run in parallel. Every entry, every bucket and the head of the list
has a spinlock, and a writer holds the lock of every node updated by its
transaction.
A writer plans under RCU, with no lock held: it finds the entry by
walking its bucket, and reads the other nodes around it from its links.
It then takes the locks of those nodes, in the order of the locks'
addresses, and validates the plan: no planned node is dead, and the
planned links are in place, in the entry and in its neighbours. If not,
it releases the locks and plans again. The dead flag is an entry's
tombstone, set by the writer that unlinks it, since an unlinked entry
keeps its links for the readers standing on it.
The operation is one RCU read-side critical section from the plan to the
commit, which keeps every node it found allocated; its records are
reserved before that section. The locks are of one lockdep class, with
a comparison function that gives lockdep their order.
As in the mutex sample, lookup threads find entries under RCU alone, and
each insertion, update, deletion or eviction is one transaction over the
hash table and the list; a second chance moves a referenced entry from
the tail of the list to its front. The module runs and reports as the
mutex sample does, with its writers' retries and lost races in the
report.
Assisted-by: LLM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
Documentation/RCU/rcu_txn.rst | 4 +-
samples/rcu_txn/Makefile | 2 +-
samples/rcu_txn/rcu_txn_cache_node_lock.c | 713 ++++++++++++++++++++++
3 files changed, 717 insertions(+), 2 deletions(-)
create mode 100644 samples/rcu_txn/rcu_txn_cache_node_lock.c
diff --git a/Documentation/RCU/rcu_txn.rst b/Documentation/RCU/rcu_txn.rst
index 8f63c1ce9c38..b8c4d58c8cca 100644
--- a/Documentation/RCU/rcu_txn.rst
+++ b/Documentation/RCU/rcu_txn.rst
@@ -200,7 +200,9 @@ Samples
table of ``rcu_txn_hlist`` buckets, and kept in an ``rcu_txn_list`` as their
LRU. Its lookups run under RCU alone, and each of its changes is one
transaction over both structures. ``rcu_txn_cache_mutex`` excludes its
-writers with a single mutex. ``CONFIG_SAMPLE_RCU_TXN`` builds it.
+writers with a single mutex; ``rcu_txn_cache_node_lock`` with a spinlock in
+every node, taken in the order of their addresses, so that its writers run in
+parallel. ``CONFIG_SAMPLE_RCU_TXN`` builds them.
Debugging
=========
diff --git a/samples/rcu_txn/Makefile b/samples/rcu_txn/Makefile
index 6d7845352afc..1ceee3c6e903 100644
--- a/samples/rcu_txn/Makefile
+++ b/samples/rcu_txn/Makefile
@@ -1,2 +1,2 @@
# SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_SAMPLE_RCU_TXN) += rcu_txn_cache_mutex.o
+obj-$(CONFIG_SAMPLE_RCU_TXN) += rcu_txn_cache_mutex.o rcu_txn_cache_node_lock.o
diff --git a/samples/rcu_txn/rcu_txn_cache_node_lock.c b/samples/rcu_txn/rcu_txn_cache_node_lock.c
new file mode 100644
index 000000000000..423567a07e48
--- /dev/null
+++ b/samples/rcu_txn/rcu_txn_cache_node_lock.c
@@ -0,0 +1,713 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * A cache on RCU pseudo-transactions, its writers under a lock per node
+ *
+ * Copyright (C) 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+ *
+ * The same cache as rcu_txn_cache_mutex.c, with writers that exclude each other
+ * per node rather than under one mutex, so that writers updating distinct
+ * entries run in parallel. Every entry, every bucket and the head of the list
+ * has a spinlock, and a writer holds the lock of every node whose links its
+ * transaction changes. An entry's links are its next and pprev in its bucket
+ * and its next and prev in the list; a bucket's is its first entry; the head's
+ * are the first and the last entry of the list.
+ *
+ * A writer plans under RCU, with no lock held: it finds the entry it works on
+ * by walking its bucket, which finds the associated node, and reads the other
+ * nodes around it from its links. It then takes the surrounding nodes' locks
+ * in the order of the locks' addresses, so that two writers cannot deadlock,
+ * and validates the plan: no planned node is dead, and the planned links are in
+ * place, in the entry and in its neighbours. If not, the writer releases the
+ * locks and plans again.
+ *
+ * The dead flag is an entry's tombstone, set with its lock held by the writer
+ * that unlinked it. An unlinked entry keeps its links, for the readers
+ * standing on it, so its neighbours' links alone are not enough to determine
+ * that it is gone.
+ *
+ * The operation is one RCU read-side critical section from the plan to the
+ * commit, which keeps every node it found allocated. lockdep is told the
+ * order of the locks: they are one class, with a comparison function.
+ *
+ * Each operation is one transaction over the hash table and the list, recorded
+ * and committed with the locks held, its records reserved before. An
+ * insertion links the new entry into its bucket and at the front of the list;
+ * an update replaces the entry in both; a deletion or an eviction unlinks one
+ * from both; a second chance moves a referenced entry from the tail of the
+ * list to its front. Lookup threads find entries under RCU alone, and mark
+ * the ones they find referenced.
+ *
+ * The sample runs for duration_ms when it is loaded, then checks that the hash
+ * table and the list hold the same entries, and reports what it did. An
+ * allocation failure stops the writers there.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/delay.h>
+#include <linux/hash.h>
+#include <linux/kthread.h>
+#include <linux/module.h>
+#include <linux/random.h>
+#include <linux/rcu_txn.h>
+#include <linux/rcu_txn_hlist.h>
+#include <linux/rcu_txn_list.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include <linux/spinlock.h>
+
+#define CACHE_HASH_BITS 6
+#define CACHE_KEYS 512 /* keys are 0 to CACHE_KEYS - 1 */
+#define CACHE_SIZE 128 /* entries held, about */
+#define CACHE_CHANCES 4 /* second chances an eviction gives at most */
+#define CACHE_THREADS 16 /* of each kind, at most */
+#define CACHE_LOCKS 5 /* locks an operation holds at most */
+
+static unsigned int nr_lookups = 2;
+module_param(nr_lookups, uint, 0444);
+MODULE_PARM_DESC(nr_lookups, "Lookup threads");
+
+static unsigned int nr_writers = 4;
+module_param(nr_writers, uint, 0444);
+MODULE_PARM_DESC(nr_writers, "Writer threads");
+
+static unsigned int duration_ms = 2000;
+module_param(duration_ms, uint, 0444);
+MODULE_PARM_DESC(duration_ms, "How long the threads run, from the load");
+
+struct cache_entry {
+ spinlock_t lock; /* its links, and @dead */
+ unsigned long key;
+ unsigned long value; /* key + CACHE_KEYS * version */
+ struct rcu_txn_hlist_node hnode; /* in its bucket */
+ struct rcu_txn_list_node lru; /* in the list, newest first */
+ struct rcu_head rcu;
+ bool dead; /* the tombstone: unlinked */
+ bool referenced; /* found since it last moved */
+};
+
+struct cache_bucket {
+ spinlock_t lock; /* its first entry */
+ struct rcu_txn_hlist_head head;
+};
+
+static struct cache_bucket cache_hash[1 << CACHE_HASH_BITS];
+static struct {
+ spinlock_t lock; /* the first and last entries */
+ struct rcu_txn_list_head head;
+} cache_lru = { .head = RCU_TXN_LIST_HEAD_INIT(cache_lru.head) };
+static atomic_t cache_count;
+
+static atomic_long_t nr_lookup, nr_hit, nr_bad, nr_insert, nr_update,
+ nr_delete, nr_evict, nr_chance, nr_retry, nr_lost,
+ nr_nomem;
+
+/* Set at the first allocation failure, which stops the writers. */
+static bool cache_failed;
+
+static void cache_fail(void)
+{
+ atomic_long_inc(&nr_nomem);
+ WRITE_ONCE(cache_failed, true);
+}
+
+#ifdef CONFIG_PROVE_LOCKING
+/* The locks of the cache are taken in the order of their addresses. */
+static int cache_lock_cmp(const struct lockdep_map *a,
+ const struct lockdep_map *b)
+{
+ return cmp_int((unsigned long)a, (unsigned long)b);
+}
+#endif
+
+/* Every cache lock is of one lockdep class, ordered by cache_lock_cmp(). */
+static void cache_lock_init(spinlock_t *lock)
+{
+ spin_lock_init(lock);
+ lock_set_cmp_fn(lock, cache_lock_cmp, NULL);
+}
+
+static struct cache_bucket *cache_bucket(unsigned long key)
+{
+ return &cache_hash[hash_long(key, CACHE_HASH_BITS)];
+}
+
+static struct cache_entry *hnode_entry(struct rcu_txn_hlist_node *node)
+{
+ return node ? container_of(node, struct cache_entry, hnode) : NULL;
+}
+
+/* The entry of a node of the list, or NULL for the head. */
+static struct cache_entry *lru_entry(struct rcu_txn_list_node *node)
+{
+ if (node == &cache_lru.head.node)
+ return NULL;
+ return container_of(node, struct cache_entry, lru);
+}
+
+static spinlock_t *lru_lock(struct rcu_txn_list_node *node)
+{
+ struct cache_entry *e = lru_entry(node);
+
+ return e ? &e->lock : &cache_lru.lock;
+}
+
+/* The first entry of a bucket, with the bucket's lock held. */
+static struct rcu_txn_hlist_node *bucket_first(struct cache_bucket *b)
+{
+ return rcu_txn_hlist_first_check(&b->head, lockdep_is_held(&b->lock));
+}
+
+/* The first node of the list, with the head's lock held. */
+static struct rcu_txn_list_node *lru_first(void)
+{
+ return rcu_txn_list_next_check(&cache_lru.head.node,
+ lockdep_is_held(&cache_lru.lock));
+}
+
+/* Within an RCU read-side critical section. */
+static struct cache_entry *cache_find(unsigned long key)
+{
+ struct cache_entry *e;
+
+ rcu_txn_hlist_for_each_entry_rcu(e, &cache_bucket(key)->head, hnode)
+ if (e->key == key)
+ return e;
+ return NULL;
+}
+
+/* What a writer found under RCU, and the locks it takes. */
+struct cache_plan {
+ struct cache_entry *e; /* the entry it works on */
+ struct rcu_txn_hlist_node __rcu **pprev; /* the slot pointing to @e, */
+ struct cache_entry *pprev_entry; /* an entry's, or NULL */
+ spinlock_t *pprev_lock; /* and its lock */
+ struct rcu_txn_hlist_node *hnext; /* next in the bucket */
+ struct rcu_txn_list_node *prev, *next; /* in the list */
+ struct rcu_txn_list_node *first; /* of the list */
+ spinlock_t *locks[CACHE_LOCKS]; /* taken in address order */
+ unsigned int nr_locks;
+};
+
+static void cache_want(struct cache_plan *p, spinlock_t *lock)
+{
+ unsigned int i;
+
+ for (i = 0; i < p->nr_locks; i++)
+ if (p->locks[i] == lock)
+ return;
+ if (!WARN_ON(p->nr_locks == CACHE_LOCKS))
+ p->locks[p->nr_locks++] = lock;
+}
+
+/* Take the locks in address order. */
+static void cache_lock(struct cache_plan *p)
+{
+ unsigned int i, j;
+
+ for (i = 1; i < p->nr_locks; i++)
+ for (j = i; j > 0 && p->locks[j - 1] > p->locks[j]; j--)
+ swap(p->locks[j - 1], p->locks[j]);
+ for (i = 0; i < p->nr_locks; i++)
+ spin_lock(p->locks[i]);
+}
+
+static void cache_unlock(struct cache_plan *p)
+{
+ while (p->nr_locks)
+ spin_unlock(p->locks[--p->nr_locks]);
+}
+
+/*
+ * Find the entry of @key by walking its bucket, which gives the slot that
+ * points to it there: the bucket's first, or the next of the entry before it.
+ */
+static struct cache_entry *cache_find_slot(unsigned long key,
+ struct cache_plan *p)
+{
+ struct cache_bucket *b = cache_bucket(key);
+ struct rcu_txn_hlist_node __rcu **slot = &b->head.first;
+ struct cache_entry *before = NULL;
+ struct rcu_txn_hlist_node *node;
+
+ for (node = rcu_txn_hlist_first_rcu(&b->head); node;
+ node = rcu_txn_hlist_next_rcu(node)) {
+ struct cache_entry *e = hnode_entry(node);
+
+ if (e->key == key) {
+ p->e = e;
+ p->pprev = slot;
+ p->pprev_entry = before;
+ p->pprev_lock = before ? &before->lock : &b->lock;
+ return e;
+ }
+ slot = &node->next;
+ before = e;
+ }
+ return NULL;
+}
+
+/* What an operation on an entry changes besides its list neighbours. */
+enum cache_mode {
+ CACHE_UNLINK, /* the bucket: a deletion, an update */
+ CACHE_FRONT, /* the front of the list: a second chance */
+};
+
+/* Is @e, a planned entry or NULL for a bucket or the head, unlinked? */
+static bool cache_dead(const struct cache_entry *e)
+{
+ return e && e->dead;
+}
+
+/* A slot of a bucket, with the lock of its owner held. */
+static struct rcu_txn_hlist_node *
+hlist_link(struct rcu_txn_hlist_node __rcu **slot, spinlock_t *lock)
+{
+ return rcu_txn_dereference_check(*slot, RCU_TXN_HLIST_TAG,
+ lockdep_is_held(lock));
+}
+
+/* A link of the list, with the lock of its node held. */
+static struct rcu_txn_list_node *
+lru_link(struct rcu_txn_list_node *node, struct rcu_txn_list_node __rcu **link)
+{
+ return rcu_txn_dereference_check(*link, RCU_TXN_LIST_TAG,
+ lockdep_is_held(lru_lock(node)));
+}
+
+/*
+ * With the locks held: no planned node is dead, and the planned links are in
+ * place, as @p->e and as its neighbours see them.
+ */
+static bool cache_plan_holds(const struct cache_plan *p, enum cache_mode mode)
+{
+ struct cache_entry *e = p->e;
+
+ if (e->dead || cache_dead(lru_entry(p->prev)) ||
+ cache_dead(lru_entry(p->next)))
+ return false;
+ if (lru_link(&e->lru, &e->lru.prev) != p->prev ||
+ lru_link(&e->lru, &e->lru.next) != p->next ||
+ lru_link(p->prev, &p->prev->next) != &e->lru ||
+ lru_link(p->next, &p->next->prev) != &e->lru)
+ return false;
+
+ if (mode == CACHE_UNLINK) {
+ struct cache_entry *n = hnode_entry(p->hnext);
+
+ if (cache_dead(p->pprev_entry) || cache_dead(n))
+ return false;
+ if (rcu_access_pointer(e->hnode.pprev) != p->pprev ||
+ hlist_link(p->pprev, p->pprev_lock) != &e->hnode ||
+ hlist_link(&e->hnode.next, &e->lock) != p->hnext ||
+ (n && rcu_access_pointer(n->hnode.pprev) != &e->hnode.next))
+ return false;
+ }
+ if (mode == CACHE_FRONT)
+ return !cache_dead(lru_entry(p->first)) &&
+ lru_first() == p->first;
+ return true;
+}
+
+/*
+ * Plan an operation of @mode on the entry of @key, take the locks of the nodes
+ * whose links it changes, and look again. Returns false, holding no lock, if
+ * there is no such entry, or if it is not @want when that is given. Within an
+ * RCU read-side critical section, which keeps what it finds allocated.
+ */
+static bool cache_lock_entry(struct cache_plan *p, unsigned long key,
+ struct cache_entry *want, enum cache_mode mode)
+{
+ struct rcu_txn_list_node *head = &cache_lru.head.node;
+ bool dead;
+
+ for (;;) {
+ p->nr_locks = 0;
+ if (!cache_find_slot(key, p) || (want && p->e != want))
+ return false;
+ p->hnext = rcu_txn_hlist_next_rcu(&p->e->hnode);
+ p->prev = rcu_txn_list_prev_rcu(&p->e->lru);
+ p->next = rcu_txn_list_next_rcu(&p->e->lru);
+ p->first = rcu_txn_list_next_rcu(head);
+
+ cache_want(p, &p->e->lock);
+ if (mode == CACHE_UNLINK) {
+ cache_want(p, p->pprev_lock);
+ if (p->hnext)
+ cache_want(p, &hnode_entry(p->hnext)->lock);
+ }
+ cache_want(p, lru_lock(p->prev));
+ cache_want(p, lru_lock(p->next));
+ if (mode == CACHE_FRONT) {
+ cache_want(p, &cache_lru.lock);
+ cache_want(p, lru_lock(p->first));
+ }
+ cache_lock(p);
+ if (cache_plan_holds(p, mode))
+ return true;
+ dead = p->e->dead;
+ cache_unlock(p);
+ if (dead && want)
+ return false;
+ atomic_long_inc(&nr_retry);
+ }
+}
+
+/*
+ * Unlink the entry of @key from its bucket and from the list, or replace it
+ * with @new in both, with @txn, which has four records reserved and which this
+ * consumes. When @want is given, only if it is that entry. Returns false if
+ * there is no such entry, or if the transaction could not be allocated.
+ * Within an RCU read-side critical section, which keeps @want allocated.
+ */
+static bool __cache_unlink(struct rcu_txn *txn, unsigned long key,
+ struct cache_entry *want, struct cache_entry *new)
+{
+ struct cache_plan p;
+ bool done = false;
+
+ if (!cache_lock_entry(&p, key, want, CACHE_UNLINK)) {
+ rcu_txn_cancel(txn);
+ return false;
+ }
+ if (new) {
+ rcu_txn_hlist_replace_prepare(txn, &p.e->hnode, &new->hnode);
+ rcu_txn_list_replace_prepare(txn, &p.e->lru, &new->lru);
+ } else {
+ rcu_txn_hlist_del_prepare(txn, &p.e->hnode);
+ rcu_txn_list_del_prepare(txn, &p.e->lru);
+ }
+ if (!rcu_txn_commit(txn)) {
+ p.e->dead = true;
+ done = true;
+ }
+ cache_unlock(&p);
+ if (done) {
+ kfree_rcu(p.e, rcu);
+ if (!new)
+ atomic_dec(&cache_count);
+ } else {
+ cache_fail();
+ }
+ return done;
+}
+
+/* Unlink the entry of @key, or replace it with @new: see __cache_unlink(). */
+static bool cache_unlink(unsigned long key, struct cache_entry *new)
+{
+ struct rcu_txn txn;
+ bool done;
+
+ /* Before RCU and the locks, as it may sleep. */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 4);
+
+ rcu_read_lock();
+ done = __cache_unlink(&txn, key, NULL, new);
+ rcu_read_unlock();
+ return done;
+}
+
+/*
+ * Move @e to the front of the list, with @txn, which has six records reserved
+ * and which this consumes. Within an RCU read-side critical section, which
+ * keeps @e allocated.
+ */
+static void cache_move_front(struct rcu_txn *txn, struct cache_entry *e)
+{
+ struct cache_plan p;
+
+ if (!cache_lock_entry(&p, e->key, e, CACHE_FRONT)) {
+ rcu_txn_cancel(txn);
+ return;
+ }
+ /* Nothing is recorded if it is there already. */
+ rcu_txn_list_move_after_prepare(txn, &e->lru, &cache_lru.head.node);
+ if (rcu_txn_commit(txn))
+ cache_fail();
+ cache_unlock(&p);
+}
+
+/*
+ * Evict the oldest entry that no lookup found since it last moved, after giving
+ * the referenced entries at the tail a second chance at the front.
+ */
+static void cache_evict(void)
+{
+ unsigned int chances = 0;
+
+ for (;;) {
+ bool evicted = false;
+ struct cache_entry *e;
+ struct rcu_txn txn;
+
+ /* Before RCU, for a move (six records) or an unlink (four). */
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 6);
+
+ /* One critical section, which keeps @e allocated throughout. */
+ rcu_read_lock();
+ e = lru_entry(rcu_txn_list_prev_rcu(&cache_lru.head.node));
+ if (!e) {
+ rcu_read_unlock();
+ rcu_txn_cancel(&txn);
+ break;
+ }
+ if (chances < CACHE_CHANCES && READ_ONCE(e->referenced)) {
+ WRITE_ONCE(e->referenced, false);
+ cache_move_front(&txn, e);
+ chances++;
+ } else {
+ evicted = __cache_unlink(&txn, e->key, e, NULL);
+ }
+ rcu_read_unlock();
+ if (evicted) {
+ atomic_long_inc(&nr_evict);
+ break;
+ }
+ if (READ_ONCE(cache_failed))
+ break;
+ /* Another writer unlinked it first: try the new tail. */
+ }
+ atomic_long_add(chances, &nr_chance);
+}
+
+/*
+ * Link @new into its bucket and at the front of the list. Returns false if
+ * its key is in the cache already, or if the transaction could not be
+ * allocated.
+ */
+static bool cache_link(struct cache_entry *new)
+{
+ struct cache_bucket *b = cache_bucket(new->key);
+ struct rcu_txn_list_node *head = &cache_lru.head.node;
+ struct cache_plan p;
+ struct rcu_txn txn;
+ bool there;
+
+ rcu_txn_init(&txn, GFP_KERNEL);
+ rcu_txn_reserve(&txn, 4);
+
+ rcu_read_lock();
+ for (;;) {
+ /* The first entries of the bucket and of the list change. */
+ p.nr_locks = 0;
+ p.hnext = rcu_txn_hlist_first_rcu(&b->head);
+ p.first = rcu_txn_list_next_rcu(head);
+ cache_want(&p, &b->lock);
+ if (p.hnext)
+ cache_want(&p, &hnode_entry(p.hnext)->lock);
+ cache_want(&p, &cache_lru.lock);
+ cache_want(&p, lru_lock(p.first));
+ cache_lock(&p);
+ if (bucket_first(b) == p.hnext && lru_first() == p.first)
+ break;
+ cache_unlock(&p);
+ atomic_long_inc(&nr_retry);
+ }
+
+ /* No other writer inserts into the bucket while its lock is held. */
+ there = !!cache_find(new->key);
+ if (!there) {
+ rcu_txn_hlist_add_head_prepare(&txn, &new->hnode, &b->head);
+ rcu_txn_list_add_after_prepare(&txn, &new->lru, head);
+ }
+ if (rcu_txn_commit(&txn)) {
+ cache_fail();
+ there = true;
+ } else if (!there) {
+ atomic_inc(&cache_count);
+ }
+ cache_unlock(&p);
+ rcu_read_unlock();
+ return !there;
+}
+
+/* Insert @key, or update it, at version @version. */
+static void cache_insert(unsigned long key, unsigned long version)
+{
+ struct cache_entry *new;
+ bool old;
+
+ new = kzalloc_obj(*new, GFP_KERNEL);
+ if (!new) {
+ cache_fail();
+ return;
+ }
+ cache_lock_init(&new->lock);
+ new->key = key;
+ new->value = key + CACHE_KEYS * version;
+
+ rcu_read_lock();
+ old = !!cache_find(key);
+ rcu_read_unlock();
+ if (old) {
+ if (cache_unlink(key, new)) {
+ atomic_long_inc(&nr_update);
+ return;
+ }
+ } else {
+ if (atomic_read(&cache_count) >= CACHE_SIZE)
+ cache_evict();
+ if (cache_link(new)) {
+ atomic_long_inc(&nr_insert);
+ return;
+ }
+ }
+ atomic_long_inc(&nr_lost); /* another writer was first */
+ kfree(new);
+}
+
+static void cache_delete(unsigned long key)
+{
+ if (cache_unlink(key, NULL))
+ atomic_long_inc(&nr_delete);
+}
+
+static int cache_lookup_fn(void *arg)
+{
+ unsigned long lookups = 0, hits = 0, bad = 0;
+
+ while (!kthread_should_stop()) {
+ unsigned long key = get_random_u32_below(CACHE_KEYS);
+ struct cache_entry *e;
+
+ rcu_read_lock();
+ e = cache_find(key);
+ if (e) {
+ hits++;
+ if (e->value % CACHE_KEYS != key)
+ bad++;
+ if (!READ_ONCE(e->referenced))
+ WRITE_ONCE(e->referenced, true);
+ }
+ rcu_read_unlock();
+ lookups++;
+ cond_resched();
+ }
+ atomic_long_add(lookups, &nr_lookup);
+ atomic_long_add(hits, &nr_hit);
+ atomic_long_add(bad, &nr_bad);
+ return 0;
+}
+
+static int cache_writer_fn(void *arg)
+{
+ unsigned long version = 0;
+
+ while (!kthread_should_stop()) {
+ unsigned long key = get_random_u32_below(CACHE_KEYS);
+
+ /* After an allocation failure, wait to be stopped. */
+ if (READ_ONCE(cache_failed)) {
+ msleep(20);
+ continue;
+ }
+ if (get_random_u32_below(8))
+ cache_insert(key, ++version);
+ else
+ cache_delete(key);
+ cond_resched();
+ }
+ return 0;
+}
+
+/*
+ * With the threads stopped: every entry of the hash table is in its bucket and
+ * in the list, both ways, and there are cache_count of them.
+ */
+static bool cache_check(void)
+{
+ unsigned int in_hash = 0, in_list = 0, in_reverse = 0, i;
+ unsigned int count = atomic_read(&cache_count);
+ struct cache_entry *e;
+ bool ok = true;
+
+ rcu_read_lock();
+ for (i = 0; i < ARRAY_SIZE(cache_hash); i++) {
+ rcu_txn_hlist_for_each_entry_rcu(e, &cache_hash[i].head,
+ hnode) {
+ ok &= cache_find(e->key) == e;
+ in_hash++;
+ }
+ }
+ rcu_txn_list_for_each_entry_rcu(e, &cache_lru.head, lru) {
+ ok &= cache_find(e->key) == e;
+ in_list++;
+ }
+ rcu_txn_list_for_each_entry_reverse_rcu(e, &cache_lru.head, lru)
+ in_reverse++;
+ rcu_read_unlock();
+ return ok && in_hash == count && in_list == count &&
+ in_reverse == count;
+}
+
+/*
+ * With the threads stopped, no reader or writer is left: free every entry
+ * without unlinking it, which allocates nothing and so cannot fail.
+ */
+static void cache_free_all(void)
+{
+ struct rcu_txn_list_node *head = &cache_lru.head.node;
+ struct rcu_txn_list_node *node, *next;
+ unsigned int i;
+
+ for (node = rcu_txn_list_next_check(head, true); node != head;
+ node = next) {
+ next = rcu_txn_list_next_check(node, true);
+ kfree(lru_entry(node));
+ }
+ rcu_txn_list_init(&cache_lru.head);
+ for (i = 0; i < ARRAY_SIZE(cache_hash); i++)
+ rcu_txn_hlist_init(&cache_hash[i].head);
+ atomic_set(&cache_count, 0);
+}
+
+static int __init cache_init(void)
+{
+ struct task_struct *threads[2 * CACHE_THREADS];
+ unsigned int i, n = 0;
+ bool ok;
+
+ for (i = 0; i < ARRAY_SIZE(cache_hash); i++) {
+ cache_lock_init(&cache_hash[i].lock);
+ rcu_txn_hlist_init(&cache_hash[i].head);
+ }
+ cache_lock_init(&cache_lru.lock);
+
+ for (i = 0; i < min(nr_lookups, CACHE_THREADS); i++)
+ threads[n++] = kthread_run(cache_lookup_fn, NULL,
+ "rcu_txn_lookup/%u", i);
+ for (i = 0; i < min(nr_writers, CACHE_THREADS); i++)
+ threads[n++] = kthread_run(cache_writer_fn, NULL,
+ "rcu_txn_writer/%u", i);
+ msleep(duration_ms);
+ for (i = 0; i < n; i++)
+ if (!IS_ERR(threads[i]))
+ kthread_stop(threads[i]);
+
+ ok = cache_check();
+ pr_info("%ld lookups, %ld hits, %ld bad; %ld insertions, %ld updates, %ld deletions, %ld evictions, %ld second chances, %ld retries, %ld lost races, %ld allocation failures; %d entries\n",
+ atomic_long_read(&nr_lookup), atomic_long_read(&nr_hit),
+ atomic_long_read(&nr_bad), atomic_long_read(&nr_insert),
+ atomic_long_read(&nr_update), atomic_long_read(&nr_delete),
+ atomic_long_read(&nr_evict), atomic_long_read(&nr_chance),
+ atomic_long_read(&nr_retry), atomic_long_read(&nr_lost),
+ atomic_long_read(&nr_nomem), atomic_read(&cache_count));
+ if (!ok || atomic_long_read(&nr_bad))
+ pr_err("the hash table and the list disagree\n");
+ if (READ_ONCE(cache_failed))
+ pr_warn("an allocation failed, which stopped the writers\n");
+ cache_free_all();
+ return 0;
+}
+
+static void __exit cache_exit(void)
+{
+}
+
+module_init(cache_init);
+module_exit(cache_exit);
+
+MODULE_DESCRIPTION("A cache on RCU pseudo-transactions, its writers under a lock per node");
+MODULE_LICENSE("GPL");
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread