mm/slub: has_pcs_used() reads remote per-CPU sheaf state from flush_all_cpus_locked() without synchronization (data race; KCSAN blind spot: slub.o is uninstrumented)

[徐晗翔 <xuhx@hust.edu.cn>]

[-- Attachment #1: Type: text/plain, Size: 4809 bytes --]

flush_all_cpus_locked() calls has_pcs_used() for every CPU to decide
which flush work to skip, reading each remote CPU's pcs->spare /
pcs->rcu_free and dereferencing pcs->main->size -- all locklessly
(mm/slub.c:4041-4051, inlined into flush_all_cpus_locked) -- while
the owning CPU's alloc/free fast paths mutate exactly those fields
under only the per-CPU sheaf lock (pcs->main->size++/--, the
pcs->main pointer swap in __pcs_replace_empty_main, etc.).  The
reader runs from kmem_cache_shrink() (sysfs shrink attribute, memory
offline) and is, as far as I can enumerate, the only cross-CPU
consumer of this state outside the sysfs slabinfo walkers (excluded
-- they read other fields under a different protocol).  I verified
the race with KCSAN after discovering why nobody has ever seen it
reported: mm/Makefile sets KCSAN_SANITIZE_slub.o := n, so BOTH sides
of this race are structurally invisible to KCSAN on any stock kernel
(verified at the disassembly level: the four accesses in
flush_all_cpus_locked compile to bare cmp/mov with zero __tsan
calls).  On a diagnostic build that only flips that Makefile switch
(no source change), a shrink+churn workload on a dedicated cache
produces 436 reader-pair reports covering all five writer families,
including an 8-byte pcs->main pointer swap with an explicit "value
changed" line -- the snapshot prerequisite of the (extremely
unlikely, source-argued) use-after-free chain: a stale pcs->main can
outlive its sheaf's retirement (spare -> rcu_free -> call_rcu ->
kfree) if the reader is preempted between load and dereference.  The
practical impact of the race itself is a wrong skip decision in
sfw->skip; the fix is to move the emptiness check into the target
CPU's flush callback, where the state is local and locked.

The detailed description:

* The race (Linux 7.3, mm/slub.c)

  Reader (inline has_pcs_used, from flush_all_cpus_locked, called by
  __kmem_cache_shrink / kmem_cache_shrink -- sysfs "shrink" and
  memory offline, privileged):
      reads pcs->spare, pcs->rcu_free, pcs->main, pcs->main->size of
      REMOTE cpus, holding no lock on that cpu's sheaf.

  Writers (per-CPU fast paths under the sheaf lock of their own cpu):
      alloc/free paths' main->size++/-- (kmem_cache_free's free_to_pcs,
      bulk variants' size += batch, alloc's size--), and
      __pcs_replace_empty_main's pcs->main = full / pcs->spare = old
      pointer stores.

  Sheaves are enabled by default (no slub_debug boot param needed).

* Why KCSAN has never reported it (part of the finding)

  mm/Makefile:15 KCSAN_SANITIZE_slub.o := n excludes the entire file
  from instrumentation on both sides.  nm/disassembly on a stock 7.3
  KCSAN kernel: has_pcs_used is inlined into flush_all_cpus_locked
  and the four accesses at +0x166/+0x173 etc. have no __tsan_* calls
  around them.  So "no KCSAN report" is not evidence of absence for
  anything inside slub.o; this race was invisible by construction.

* Evidence (KCSAN, stock-vs-diagnostic A/B; reproducer attached)

  - Stock kernel: three runs, zero reports (structurally, as above).
  - Diagnostic kernel (ONLY the Makefile switch flipped, source
    untouched): dedicated non-mergeable cache, 3 churn threads
    (single+bulk8 alloc/free) + 2 shrink threads -> 436 reader-pair
    reports, all five writer families:
      * size field (4 bytes, same address across CPUs):
        read by flush_all_cpus_locked+0x173 (shrink, cpu 3) vs
        read-write kmem_cache_free_bulk+0x90a (churn, cpu 1)
      * pcs->main pointer (8 bytes): write by
        __pcs_replace_empty_main+0x5c1 (pcs->main = full) vs read by
        flush_all_cpus_locked+0x166, KCSAN printing
        "value changed: ...0300 -> ...78800" -- the reader observed
        the sheaf pointer being concurrently replaced, i.e. the
        stale-snapshot precondition of the UAF chain, live.
  - Control (same diagnostic kernel, churn-only, shrink=0): zero
    reader-pair reports -- the reports are caused by the shrink-side
    reader, not by the churn load.
  - Honest limits: the (c)-layer UAF read (stale pcs->main surviving
    the retirement chain during a preempted reader) is source-argued,
  not dynamically triggered -- it needs a millisecond-scale preemption
  of the reader landing inside a retirement window; KCSAN demonstrates
  the racing accesses and the pointer-swap visibility, which is the
  right tool's right scope.

  Reproducer (attached fxslub.c): registers a private cache with a
  ctor (non-mergeable, sheaf-enabled) and hammers
  kmem_cache_shrink() from N threads while other threads churn
  single+bulk allocations.  On a KCSAN kernel with slub.o
  instrumented it produces the reports above within minutes; on a
  stock kernel it demonstrably cannot (that is the point).


Regards,
Hanxiang Xu
xuhx@hust.edu.cn

[-- Attachment #2: fxslub.c --]
[-- Type: text/plain, Size: 6525 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * fxslub.c - KCSAN hammer for Case 1 of the kernel/futex manual audit:
 *
 *   has_pcs_used() (mm/slub.c:4041-4051) reads remote per-CPU sheaf state
 *   (pcs->spare / pcs->rcu_free / pcs->main->size) with no lock and no
 *   annotation from flush_all_cpus_locked() during kmem_cache_shrink(),
 *   racing the owning CPU's local_trylock-protected writers
 *   (free_to_pcs / alloc_from_pcs / __free_to_pcs_batch /
 *    __pcs_replace_empty_main / __pcs_replace_full_main /
 *    __kfree_rcu_sheaf ...).
 *
 * Method: dedicated unmergeable cache (ctor) + churn kthreads pinned on
 * CPUs 0..N-1 driving all sheaf write paths (single fastpath, bulk, and
 * kfree_rcu batching) + shrink kthreads looping kmem_cache_shrink() on
 * other CPUs. No kernel source modification; EXPORTED APIs only.
 *
 * Params:
 *   churn_threads  - pinned per-CPU alloc/free/rcu threads (default 6)
 *   shrink_threads - kmem_cache_shrink() loopers (default 2, cpus 6,7)
 *   shrink         - 1 enable shrink loop (fx1), 0 negative control (fx1ctl)
 *   secs           - self-reported runtime budget hint for /init (default 90)
 */
#define pr_fmt(fmt) "FXSLUB: " fmt

#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/slab.h>
#include <linux/delay.h>
#include <linux/rcupdate.h>
#include <linux/cpu.h>
#include <linux/nodemask.h>
#include <linux/mm.h>

static int churn_threads = 6;
static int shrink_threads = 2;
static bool shrink = true;
static int secs = 90;

module_param(churn_threads, int, 0);
module_param(shrink_threads, int, 0);
module_param(shrink, bool, 0);
module_param(secs, int, 0);

struct fxobj {
	struct rcu_head rcu;
	char pad[512 - sizeof(struct rcu_head)];
};

static struct kmem_cache *fxcache;
static struct task_struct *churn_task[8];
static struct task_struct *shrink_task[4];
static atomic_t stop_flag;

static atomic64_t n_alloc, n_bulk, n_rcu, n_shrink, n_free_single;

/* ctor => unmergeable => dedicated sheaves/barns for this cache */
static void fx_ctor(void *p) { }

static int fx_churn(void *arg)
{
	void *bulk[8];
	struct fxobj *o;
	int n;

	while (!kthread_should_stop() && !atomic_read(&stop_flag)) {
		/* single fastpath: pcs->main->size ++/-- on this cpu */
		o = kmem_cache_alloc(fxcache, GFP_KERNEL);
		if (o) {
			atomic64_inc(&n_alloc);
			kmem_cache_free(fxcache, o);
			atomic64_inc(&n_free_single);
		}

		/* bulk: drives __pcs_replace_empty_main / free_to_pcs_bulk /
		 *       barn traffic (pcs->main/spare pointer stores).
		 * Bulk size 8 (not 32): smaller batches keep the free-side
		 * barn_replace_full_sheaf() rate (whose MAX_FULL_SHEAVES check
		 * is outside barn->lock by design) from stacking unboundedly
		 * under KCSAN-widened check->lock gaps (attempt4: 5.8GB of
		 * freed objects parked in barn sheaves -> OOM). */
		n = kmem_cache_alloc_bulk(fxcache, GFP_KERNEL, 8, bulk);
		if (n > 0) {
			atomic64_add(n, &n_bulk);
			kmem_cache_free_bulk(fxcache, n, bulk);
		}

		/* memory watermark throttle: barn parking of freed objects is
		 * only soft-bounded (see above); do not let the hammer OOM the VM */
		if (si_mem_available() < (512UL << 20 >> PAGE_SHIFT))
			msleep(100);

		/*
		 * No kfree_rcu phase: for a non-kmalloc cache, kfree_rcu() takes
		 * the per-object call_rcu(kvfree_rcu_cb) path (NOT the sheaf
		 * batching), so it contributes nothing to the sheaf-state writers
		 * under test while its RCU backlog grows without bound under
		 * KCSAN taxation (attempt3: 5.8GB unreclaimable, OOM panic).
		 * All target writers (free_to_pcs / alloc_from_pcs /
		 * __pcs_replace_*) are driven by the single+bulk churn above.
		 */

		if (!(atomic64_read(&n_alloc) & 0xfff))
			cond_resched();
	}
	return 0;
}

static int fx_shrink(void *arg)
{
	while (!kthread_should_stop() && !atomic_read(&stop_flag)) {
		kmem_cache_shrink(fxcache);
		atomic64_inc(&n_shrink);
		cond_resched();
	}
	return 0;
}

static int __init fxslub_init(void)
{
	long i;
	int ret;

	fxcache = kmem_cache_create("fxsheaf", sizeof(struct fxobj), 0,
				    SLAB_PANIC, fx_ctor);
	if (!fxcache)
		return -ENOMEM;

	if (churn_threads > 8)
		churn_threads = 8;
	if (shrink_threads > 4)
		shrink_threads = 4;

	pr_info("cache fxsheaf created, churn=%d shrink=%d(%s) secs=%d\n",
		churn_threads, shrink_threads,
		shrink ? "on" : "OFF (negative control)", secs);

	for (i = 0; i < churn_threads; i++) {
		struct task_struct *t = kthread_create(fx_churn, (void *)i,
						       "fxchurn/%ld", i);
		if (IS_ERR(t)) { ret = PTR_ERR(t); goto err; }
		kthread_bind(t, i % nr_cpu_ids);
		wake_up_process(t);
		churn_task[i] = t;
	}

	if (shrink) {
		for (i = 0; i < shrink_threads; i++) {
			struct task_struct *t = kthread_create(fx_shrink, (void *)i,
							       "fxshrink/%ld", i);
			if (IS_ERR(t)) { ret = PTR_ERR(t); goto err; }
			kthread_bind(t, (churn_threads + i) % nr_cpu_ids);
			wake_up_process(t);
			shrink_task[i] = t;
		}
	}

	pr_info("threads running\n");
	return 0;
err:
	atomic_set(&stop_flag, 1);
	for (i = 0; i < 8; i++)
		if (churn_task[i])
			kthread_stop(churn_task[i]);
	kmem_cache_destroy(fxcache);
	return ret;
}

static void __exit fxslub_exit(void)
{
	int i;

	atomic_set(&stop_flag, 1);
	for (i = 0; i < 8; i++)
		if (churn_task[i])
			kthread_stop(churn_task[i]);
	for (i = 0; i < 4; i++)
		if (shrink_task[i])
			kthread_stop(shrink_task[i]);

	pr_info("totals: shrink_calls=%lld allocs=%lld bulk=%lld rcu=%lld single_free=%lld\n",
		atomic64_read(&n_shrink), atomic64_read(&n_alloc),
		atomic64_read(&n_bulk), atomic64_read(&n_rcu),
		atomic64_read(&n_free_single));
	/* see comment below before any future re-addition of destroy */

	/*
	 * The cache is deliberately NOT destroyed: kfree_rcu() from a non-
	 * kmalloc cache queues one call_rcu(kvfree_rcu_cb) per object
	 * (kvfree_call_rcu(), mm/slab_common.c), and those callbacks free back
	 * into per-CPU sheaves; draining ~200k of them under KCSAN taxation
	 * takes minutes and then floods "Objects remaining on
	 * __kmem_cache_shutdown()" warnings (observed: 1441/1604 warnings in
	 * the first two control attempts). That noise is a module-contract
	 * artifact, irrelevant to the race under test; this VM is throwaway,
	 * so we leak the idle cache and keep the log clean.
	 */
	pr_info("leaking idle cache fxsheaf (teardown noise avoided)\n");
	pr_info("cache destroyed, module out\n");
}

module_init(fxslub_init);
module_exit(fxslub_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("KCSAN hammer: has_pcs_used() vs per-CPU sheaf writers (audit Case 1)");