genirq: NMI dispatch reads struct irqaction with no quiescence -- use-after-free across free_nmi()'s kfree (x86 verification on the arch-independent core path)

[徐晗翔 <xuhx@hust.edu.cn>]

[-- Attachment #1: Type: text/plain, Size: 6922 bytes --]

handle_fasteoi_nmi() (kernel/irq/chip.c:792-810), the SPI-NMI flow
handler, snapshots desc->action at entry with no lock (:795) and then
uses that pointer in NMI context: the tracepoint(s) (:801/:806), the
indirect call action->handler(irq, action->dev_id) (:805) and the
trailing irq_eoi.  The unregister path free_nmi() -> __cleanup_nmi()
(kernel/irq/manage.c:2032-2065) removes the publication under
desc->lock, then kfree()s the action OUTSIDE the lock (:2057) --
there is no mechanism anywhere that waits for an in-flight NMI
handler: the flow handler never sets IRQD_IRQ_INPROGRESS (no
IRQD_IRQ_INPROGRESS in the whole function, so __synchronize_hardirq's
spin can never observe an in-flight NMI), there is no reference
count, no RCU, and no in-flight counter on struct irqaction.  When
the remove side and an in-flight NMI run concurrently, the in-flight
handler keeps dereferencing the kfree()d action -- including reading
the function pointer and indirect-calling it -- which happens in NMI
context (panic or hard lockup; no safe recovery).  I verified this
one-shot on a pristine Linux 7.3 KASAN kernel by driving the
arch-independent genirq core with a fake NMI-capable irqchip module
(IRQCHIP_SUPPORTS_NMI, mirroring the GIC driver's nmi_setup/teardown
semantics): with the dispatch thread held by kprobe right after the
snapshot (handle_fasteoi_nmi+0x56) and free_nmi completing its kfree
on another CPU, the release produced exactly the two predicted
slab-use-after-free reports (action->handler load at +0x5e, dev_id
load at +0x6b -- matching the objdump-decoded load offsets byte for
byte), followed by an indirect call through the garbage function
pointer (Oops RIP 0x0) from NMI context.  The sibling drivers
pciehp-style comparison does not exist here, but the plain-IRQ path
has the complete quiescence machinery (IRQD_IRQ_INPROGRESS +
__synchronize_hardirq + request_mutex + dev_id matching) that the NMI
paths lack entirely -- the defect is that NMI dispatch bypasses all
of it.

The detailed description:

* The two paths (Linux 7.3 line numbers)

  Dispatch (SPI-NMI): generic_handle_domain_nmi() -> handle_irq_desc()
  -> desc->handle_irq = handle_fasteoi_nmi (kernel/irq/chip.c:792-810):
  :795 snapshot; :799 kstat; :801/:806 tracepoints; :805
  action->handler(irq, action->dev_id); :808-809 irq_eoi.

  Unregister: free_nmi() (manage.c:2067-2082) -> __cleanup_nmi()
  (manage.c:2032-2065): lock :2037, irq_nmi_teardown :2038 (flow
  handler switched back by the irqchip driver, gic-v3.c:646),
  desc->action = NULL :2047, irq_shutdown_and_deactivate :2050,
  unlock, kfree(action) :2057, irq_release_resources :2059,
  irq_chip_pm_put :2061, module_put :2062.  No wait of any kind
  before the kfree.

* Why the depth/mask handling is not protection (closed)

  free_nmi() on an enabled NMI (depth==0) only WARNs and disables
  (manage.c:2077-2079); disable_nmi_nosync() documents that it does
  not wait for running instances (manage.c:755-768) and a synchronous
  disable_nmi() does not exist in the tree; a GICv3-level mask stops
  NEW deliveries but SPI pending state survives disable (GICv3 spec:
  only PPI pending is cleared on disable), so "disabled but still
  deliverable" widens the window.  struct irqaction has no refcount
  and no RCU field (include/linux/interrupt.h:123-139); the kfree is
  direct.

  The plain free_irq() path, by contrast, has the full machinery:
  IRQD_IRQ_INPROGRESS (handle.c:255-267) + __synchronize_hardirq
  (manage.c:50-82) + request_mutex (manage.c:1868) + dev_id matching
  with an already-free early return (manage.c:1881-1889).  NMI
  dispatch bypasses handle_irq_event(), so it never participates.

* Reachability (upstream form)

  Requires arm64 + GICv3 pseudo-NMI (irqchip.gicv3_pseudo_nmi=1) and
  an SPI-type NMI registration; in-tree registrants: ACPI AGDI
  (frees without disabling first, agdi.c:150-151), Fujitsu A64FX diag,
  and arm_pmu's SPI overflow (its probe-failure path removes the
  cpuhp instance with _nocalls -- teardown callbacks do not run, NMI
  stays enabled -- then frees, arm_pmu.c:858-859/243/51).  Trigger
  pairs an admin remove with any in-flight firmware/PMU NMI; the
  window is microseconds, and NMI frequency scales it linearly.
  Alternative delivery paths (resend, software re-trigger, injection,
  suspend/resume) were exhaustively ruled out for NMI descs -- the
  only execution of handle_fasteoi_nmi is the NMI entry itself, so
  the exposure is exactly this window.

* Evidence (pristine 7.3 KASAN, x86_64 QEMU; instruments attached)

  nmi_race.ko registers a fake NMI irqchip (the GIC mirror) and a
  dispatch kthread on the exported-equivalent paths; nmi_stall.ko
  parks the dispatcher at handle_fasteoi_nmi+0x56 (after the
  snapshot, before the handler/dev_id loads).  free_nmi from another
  CPU completes inside the stall:

    BUG: KASAN: slab-use-after-free in handle_fasteoi_nmi+0x5e
         (action->handler load)   <- objdump-decoded offsets match
    BUG: KASAN: slab-use-after-free in handle_fasteoi_nmi+0x6b
         (action->dev_id load)
    Oops: 0010 RIP: 0010:0x0     <- indirect call through the garbage
                                    function pointer read from freed
                                    memory, in NMI context
    three stacks closed: use handle_fasteoi_nmi+0x5e <- handle_irq_desc
      (CPU 2); alloc __kmalloc_cache_noprof <- request_nmi (task 112);
      free kfree <- __cleanup_nmi <- free_thread (CPU 3);
      object kmalloc-128 (struct irqaction)

  Fidelity, stated plainly: the audited defect code (chip.c:792-810
  snapshot/use and manage.c:2032-2065 unlock-and-free) is the
  architecture-independent genirq core and executes unmodified; the
  delivery ENTRY (GIC pseudo-NMI exception path) is platform code and
  is replaced in the x86 VM by a module irqchip whose setup/teardown
  mirror the GIC semantics -- the racing pair (dispatch snapshot vs
  unregister free) is unaffected by which entry raises the interrupt.

* Suggested fix direction

  Option A (minimal): in __cleanup_nmi(), after removing the
  publication and before the out-of-lock kfree, wait for in-flight
  NMIs: __synchronize_hardirq(desc, true) covers the active window
  (EOI clears it), plus a per-desc in-flight counter incremented/
  decremented by handle_fasteoi_nmi (atomic; spin-waited to zero)
  to close the small ack-then-sample blind spot; a pre-wait
  PENDING clear accounts for the GICv3 SPI pending-survival rule.

  Option B (structural): make struct irqaction RCU-freed for NMI
  flows (rcu_dereference at the snapshot, kfree_rcu at cleanup) --
  consistent with the sparse-irq RCU direction and it also fixes the
  percpu variant (my separate observation).  Either way, free_nmi()
  deserves a kernel-doc stating the quiescence contract (none exists
  today).


  Please keep me in Cc; I am not subscribed to the lists.


Regards,
Hanxiang Xu
xuhx@hust.edu.cn

[-- Attachment #2: nmi_stall.c --]
[-- Type: text/plain, Size: 2814 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * nmi_stall.c - generic kprobe controlled-window staller for the
 * kernel/irq NMI verification (2026-11-09), following the r4_stall
 * methodology (controlled window; does not measure natural hit rates).
 *
 * Params:
 *   symbol=   function to probe (required)
 *   off=      kprobe offset inside symbol (instruction offset, hex ok)
 *   stall_us= max busy-wait duration once armed+hit (default 40s)
 *   arm=      sysfs gate (0600): stall only while armed
 *   release=  sysfs gate (0600): end the stall
 *
 * The pre_handler stalls the FIRST armed hit until release=1 (or timeout).
 * Later hits are counted only.
 */
#define pr_fmt(fmt) "NMI1STALL: " fmt

#include <linux/module.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>
#include <linux/delay.h>
#include <linux/atomic.h>

static char *symbol = "";
module_param(symbol, charp, 0400);
MODULE_PARM_DESC(symbol, "kprobe target symbol");

static unsigned long off;
module_param(off, ulong, 0400);
MODULE_PARM_DESC(off, "kprobe offset inside symbol");

static int stall_us = 40000000;
module_param(stall_us, int, 0600);
MODULE_PARM_DESC(stall_us, "max stall duration in us");

static int arm;
module_param(arm, int, 0600);
MODULE_PARM_DESC(arm, "1 = stall next hit");

static int release;
module_param(release, int, 0600);
MODULE_PARM_DESC(release, "1 = end the stall");

static atomic_t hits = ATOMIC_INIT(0);
static bool consumed;

static int stall_pre(struct kprobe *p, struct pt_regs *regs)
{
	int h = atomic_inc_return(&hits);

	if (!READ_ONCE(arm) || consumed) {
		if (h <= 5)
			pr_warn("hit #%d (not stalled, arm=%d)\n", h, READ_ONCE(arm));
		return 0;
	}
	WRITE_ONCE(consumed, true);
	pr_warn("STALL HIT at %s+0x%lx stall_us=%d\n", p->symbol_name, off,
		stall_us);
	/* hard deadline so a dead orchestrator cannot hang the VM forever */
	for (int waited = 0; waited < stall_us && !READ_ONCE(release);
	     waited += 1000)
		udelay(1000);
	pr_warn("RELEASED (release=%d waited_hits=%d)\n", READ_ONCE(release),
		atomic_read(&hits));
	return 0;
}

static struct kprobe kp;

static int __init nmi_stall_init(void)
{
	if (!symbol[0]) {
		pr_err("no symbol given\n");
		return -EINVAL;
	}
	kp.symbol_name = symbol;
	kp.offset = off;
	kp.pre_handler = stall_pre;
	if (register_kprobe(&kp)) {
		pr_err("cannot register kprobe on %s+0x%lx\n", symbol, off);
		return -EINVAL;
	}
	pr_warn("armed-able: %s+0x%lx stall_us=%d (waiting for arm=1)\n",
		symbol, off, stall_us);
	return 0;
}

static void __exit nmi_stall_exit(void)
{
	unregister_kprobe(&kp);
	pr_warn("removed hits=%d stalled=%d\n", atomic_read(&hits), consumed);
}

module_init(nmi_stall_init);
module_exit(nmi_stall_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("generic kprobe stall for kernel/irq NMI verification");

[-- Attachment #3: nmi_race.c --]
[-- Type: text/plain, Size: 10724 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * nmi_race.c - driver module for the kernel/irq NMI race verification
 * (2026-11-09). It exercises the REAL genirq NMI core paths
 * (request_nmi/free_nmi/enable_irq + handle_fasteoi_nmi) on a fake
 * NMI-capable irq_chip, mirroring what GICv3 pseudo-NMI does on arm64.
 *
 * request_nmi()/free_nmi() are not exported, so they are resolved with the
 * kprobe-addr trick (same technique as r4_stall's symbol lookup).
 *
 * modes:
 *   1  Case-1 controlled window: dispatch thread enters handle_fasteoi_nmi
 *      and is stalled by nmi_stall.ko (kprobe at +0x56 in build-kasan-7.3,
 *      after the desc->action snapshot, before the action->handler /
 *      action->dev_id loads); free thread frees the action meanwhile; the
 *      released dispatcher then reads freed memory -> KASAN.
 *   3  natural hammer: writer request/sleep/free/sleep cycles + dense
 *      dispatcher (READ_ONCE(installed) pre-check to dodge the NULL window;
 *      residual NULL / freed reads are captured by KASAN/KCSAN).
 *      churn=1 variant: writer only cycles enable/disable (negative control,
 *      desc->action untouched).
 *   5  W5 double-free: two threads call free_nmi() concurrently on an
 *      installed line; the loser passes the irq_is_nmi() gate before the
 *      winner clears IRQS_NMI, then serializes behind desc->lock and runs
 *      a SECOND cleanup (WARN_ON(desc->action==NULL) + unconditional
 *      release tail) -> teardown/release_resources counted twice, module
 *      refcount goes negative in /proc/modules.
 */
#define pr_fmt(fmt) "NMI1RACE: " fmt

#include <linux/module.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>
#include <linux/irq.h>
#include <linux/irqdesc.h>
#include <linux/interrupt.h>
#include <linux/kthread.h>
#include <linux/delay.h>
#include <linux/atomic.h>
#include <linux/sched/task.h>

typedef int (*request_nmi_t)(unsigned int, irq_handler_t, unsigned long,
			     const char *, void *);
typedef const void *(*free_nmi_t)(unsigned int, void *);

static int mode = 1;
module_param(mode, int, 0400);
MODULE_PARM_DESC(mode, "1=case1 controlled, 3=hammer, 5=W5 double-free");

static int secs = 60;
module_param(secs, int, 0400);
MODULE_PARM_DESC(secs, "hammer duration (mode 3)");

static int hold_ms = 1500;
module_param(hold_ms, int, 0400);
MODULE_PARM_DESC(hold_ms, "installed hold time per cycle (mode 3)");

static int gap_ms = 100;
module_param(gap_ms, int, 0400);
MODULE_PARM_DESC(gap_ms, "freed gap time per cycle (mode 3)");

static int churn;
module_param(churn, int, 0400);
MODULE_PARM_DESC(churn, "mode3 control: writer cycles enable/disable only");

static unsigned int cpu_disp = 2, cpu_free = 3;
module_param(cpu_disp, uint, 0400);
module_param(cpu_free, uint, 0400);

static request_nmi_t request_nmi_fn;
static free_nmi_t free_nmi_fn;

static unsigned int virq;
static void *cookie;
static atomic_t installed = ATOMIC_INIT(0);
static atomic_t go = ATOMIC_INIT(0);
static atomic_t teardown_cnt = ATOMIC_INIT(0);
static atomic_t relres_cnt = ATOMIC_INIT(0);
static atomic_t dispatches = ATOMIC_INIT(0);
static atomic_t disp_null = ATOMIC_INIT(0);
static struct task_struct *t_disp, *t_free;

/* ---- fake NMI-capable chip (core sees the same callbacks as GICv3) ---- */

static int fake_nmi_setup(struct irq_data *d)
{
	pr_warn("chip: irq_nmi_setup irq=%u\n", d->irq);
	return 0;
}

static void fake_nmi_teardown(struct irq_data *d)
{
	pr_warn("chip: irq_nmi_teardown #%d irq=%u\n",
		atomic_inc_return(&teardown_cnt), d->irq);
}

static void fake_relres(struct irq_data *d)
{
	pr_warn("chip: irq_release_resources #%d irq=%u\n",
		atomic_inc_return(&relres_cnt), d->irq);
}

static void fake_mask(struct irq_data *d) {}
static void fake_unmask(struct irq_data *d) {}
static void fake_eoi(struct irq_data *d) {}

static struct irq_chip fake_chip = {
	.name			= "nmi-fake",
	.flags			= IRQCHIP_SUPPORTS_NMI,
	.irq_mask		= fake_mask,
	.irq_unmask		= fake_unmask,
	.irq_eoi		= fake_eoi,
	.irq_nmi_setup		= fake_nmi_setup,
	.irq_nmi_teardown	= fake_nmi_teardown,
	.irq_release_resources	= fake_relres,
};

static atomic_t handler_calls = ATOMIC_INIT(0);

static irqreturn_t fake_nmi_handler(int irq, void *dev_id)
{
	/* print sparsely: printk-path accesses would dilute KCSAN sampling */
	if (atomic_inc_return(&handler_calls) % 512 == 1)
		pr_warn("HANDLER CALLED #%d irq=%d dev_id=%px\n",
			atomic_read(&handler_calls), irq, dev_id);
	return IRQ_HANDLED;
}

static void park(void)
{
	while (!kthread_should_stop())
		msleep(100);
}

static unsigned long sym_addr(const char *name)
{
	struct kprobe p = { .symbol_name = name };
	unsigned long addr = 0;

	if (!register_kprobe(&p)) {
		addr = (unsigned long)p.addr;
		unregister_kprobe(&p);
	}
	return addr;
}

static int do_request(void)
{
	int ret;

	ret = request_nmi_fn(virq, fake_nmi_handler,
			     IRQF_PERCPU | IRQF_NOBALANCING | IRQF_NO_AUTOEN |
			     IRQF_NO_THREAD,
			     "nmi-race", &cookie);
	if (!ret) {
		atomic_inc(&installed);
		enable_irq(virq); /* == enable_nmi */
		/*
		 * The GIC driver installs handle_fasteoi_nmi from its
		 * irq_nmi_setup callback (desc->lock held); the core does not
		 * do it. struct irq_desc is opaque to modules, so we install
		 * the audited flow handler via the exported API instead.
		 */
		__irq_set_handler(virq, handle_fasteoi_nmi, 0, NULL);
	}
	pr_warn("request_nmi -> %d (installed=%d)\n", ret,
		atomic_read(&installed));
	return ret;
}

static void do_free(void)
{
	const void *devname;

	atomic_dec(&installed);
	devname = free_nmi_fn(virq, &cookie);
	pr_warn("free_nmi -> %p (installed=%d)\n", devname,
		atomic_read(&installed));
}

/* mode 1: dispatcher stalls inside handle_fasteoi_nmi via nmi_stall.ko */
static int disp_thread_m1(void *arg)
{
	while (!atomic_read(&go))
		cpu_relax();
	pr_warn("DISPATCH_BEGIN (cpu=%u)\n", smp_processor_id());
	generic_handle_irq(virq);
	pr_warn("DISPATCH_DONE\n");
	park();
	return 0;
}

static int free_thread_m1(void *arg)
{
	msleep(2000);
	pr_warn("release go -> dispatch enters handle_fasteoi_nmi\n");
	atomic_inc(&go);
	msleep(3000); /* dispatcher is stalled inside handle_fasteoi_nmi now */
	do_free();
	pr_warn("FREE side finished (dispatcher still stalled on freed action)\n");
	park();
	return 0;
}

/* mode 3: natural hammer */
static int writer_thread(void *arg)
{
	unsigned long deadline = jiffies + secs * HZ;
	int cycles = 0;

	while (time_before(jiffies, deadline)) {
		if (churn) {
			/* control: no management writes at all */
			msleep_interruptible(hold_ms);
		} else {
			if (do_request())
				break;
			msleep_interruptible(hold_ms);
			do_free();
			msleep_interruptible(gap_ms);
		}
		cycles++;
	}
	if (churn)
		cycles = 0;
	pr_warn("writer done: cycles=%d churn=%d teardown=%d relres=%d\n",
		cycles, churn, atomic_read(&teardown_cnt),
		atomic_read(&relres_cnt));
	park();
	return 0;
}

static int disp_thread_m3(void *arg)
{
	unsigned long deadline = jiffies + secs * HZ;

	while (time_before(jiffies, deadline)) {
		if (atomic_read(&installed)) {
			generic_handle_irq(virq);
			atomic_inc(&dispatches);
		} else {
			atomic_inc(&disp_null);
			cpu_relax();
		}
	}
	pr_warn("dispatcher done: dispatches=%d null-skips=%d\n",
		atomic_read(&dispatches), atomic_read(&disp_null));
	park();
	return 0;
}

/* mode 5: concurrent double free_nmi (W5) */
static int w5_thread(void *arg)
{
	int id = (int)(long)arg;
	const void *ret;
	int i;

	/* rendezvous so BOTH threads pass free_nmi's irq_is_nmi() gate before
	 * either one clears IRQS_NMI inside the cleanup lock region */
	atomic_inc(&go);
	while (atomic_read(&go) < 2)
		cpu_relax();

	ret = free_nmi_fn(virq, &cookie);
	pr_warn("W5 thread %d free -> %p teardown=%d relres=%d\n", id, ret,
		atomic_read(&teardown_cnt), atomic_read(&relres_cnt));
	/* fallback retries in case this thread lost the istate race */
	for (i = 0; i < 30 && atomic_read(&teardown_cnt) < 2; i++)
		udelay(500);
	pr_warn("W5 thread %d done: teardown=%d relres=%d (expect 2/2)\n", id,
		atomic_read(&teardown_cnt), atomic_read(&relres_cnt));
	park();
	return 0;
}

static int __init nmi_race_init(void)
{
	unsigned long a_req = sym_addr("request_nmi");
	unsigned long a_free = sym_addr("free_nmi");
	int ret = -EINVAL;

	request_nmi_fn = (request_nmi_t)a_req;
	free_nmi_fn = (free_nmi_t)a_free;
	if (!request_nmi_fn || !free_nmi_fn) {
		pr_err("cannot resolve request_nmi=%lx free_nmi=%lx\n", a_req,
		       a_free);
		return -EINVAL;
	}
	pr_warn("resolved request_nmi=%px free_nmi=%px mode=%d\n",
		request_nmi_fn, free_nmi_fn, mode);

	virq = __irq_alloc_descs(-1, 0, 1, NUMA_NO_NODE, THIS_MODULE, NULL);
	if (virq < 0) {
		pr_err("irq_alloc_descs failed %d\n", virq);
		return virq;
	}
	pr_warn("allocated virq=%u (desc owner=this module)\n", virq);
	irq_set_chip(virq, &fake_chip);

	switch (mode) {
	case 1:
		ret = do_request();
		if (ret)
			return ret;
		t_free = kthread_create_on_cpu(free_thread_m1, NULL, cpu_free,
					       "nmi1_free");
		t_disp = kthread_create_on_cpu(disp_thread_m1, NULL, cpu_disp,
					       "nmi1_disp");
		if (IS_ERR(t_free) || IS_ERR(t_disp))
			return -EINVAL;
		wake_up_process(t_free);
		wake_up_process(t_disp);
		break;
	case 3:
		/* experiment: the writer installs in its first cycle.
		 * churn control: install once here so the dispatch density
		 * matches the experiment, and the writer only sleeps. */
		if (churn && do_request())
			return ret;
		t_free = kthread_create_on_cpu(writer_thread, NULL, cpu_free,
					       "nmi1_writer");
		t_disp = kthread_create_on_cpu(disp_thread_m3, NULL, cpu_disp,
					       "nmi1_disp");
		if (IS_ERR(t_free) || IS_ERR(t_disp))
			return -EINVAL;
		wake_up_process(t_free);
		wake_up_process(t_disp);
		break;
	case 5:
		ret = do_request();
		if (ret)
			return ret;
		t_free = kthread_create_on_cpu(w5_thread, (void *)1L, cpu_free,
					       "nmi_w5a");
		t_disp = kthread_create_on_cpu(w5_thread, (void *)2L, cpu_disp,
					       "nmi_w5b");
		if (IS_ERR(t_free) || IS_ERR(t_disp))
			return -EINVAL;
		wake_up_process(t_free);
		wake_up_process(t_disp);
		break;
	default:
		pr_err("bad mode %d\n", mode);
		return -EINVAL;
	}
	return 0;
}

static void __exit nmi_race_exit(void)
{
	if (t_free && !IS_ERR(t_free))
		kthread_stop(t_free);
	if (t_disp && !IS_ERR(t_disp))
		kthread_stop(t_disp);
	/* best-effort cleanup of a still-installed action (mode 3 churn off) */
	if (atomic_read(&installed) && mode == 3 && !churn)
		do_free();
	pr_warn("removed mode=%d teardown=%d relres=%d\n", mode,
		atomic_read(&teardown_cnt), atomic_read(&relres_cnt));
}

module_init(nmi_race_init);
module_exit(nmi_race_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("kernel/irq NMI race verification driver");