* [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep
2026-11-20 10:59 [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
2 siblings, 0 replies; 4+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_perf_event_output() takes one of three per-CPU perf_sample_data
slots per nesting level, and fails with -EBUSY and a WARN_ON_ONCE() when
it is entered a fourth time on a CPU. Programs get there without any
bug: a kprobe program calls it, and fentry programs on
perf_event_output() call it again from inside. Each program's own
recursion check lets it in once, so three of them reach the fourth
level. With panic_on_warn, that takes the machine down.
Keep the -EBUSY and drop the warning, as commit 3880cdbed1c4 ("bpf: Fix
WARN() in get_bpf_raw_tp_regs") did for the raw tracepoint regs slots.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index b4ff9d4e8..8910d5d05 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -702,7 +702,7 @@ BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map,
sds = this_cpu_ptr(&bpf_trace_sds);
nest_level = this_cpu_inc_return(bpf_trace_nest_level);
- if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
+ if (nest_level > ARRAY_SIZE(sds->sds)) {
err = -EBUSY;
goto out;
}
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread* [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers
2026-11-20 10:59 [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
2 siblings, 0 replies; 4+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_get_stack() and bpf_get_stackid() with BPF_F_USER_BUILD_ID, and
bpf_find_vma(), take mmap_read_trylock() on the current mm. That fires
the mmap_lock tracepoints, so a program on one of them that calls these
helpers runs the tracepoint again from inside itself:
prog 1 on mmap_lock_acquire_returned
bpf_get_stack(BPF_F_USER_STACK | BPF_F_USER_BUILD_ID)
mmap_read_trylock()
prog 2 on mmap_lock_acquire_returned
bpf_get_stack(...)
mmap_read_trylock()
...
The recursion check of each program only stops it from running inside
itself, so this nests as deep as there are programs attached, and each
level runs all programs that aren't on the stack yet: the work per
tracepoint hit grows with the factorial of the number of programs. With
irqs disabled, the per-CPU irq_work already allows only one level; with
irqs enabled nothing limits it.
Count the levels per CPU with irqs enabled too and fail with -EBUSY past
three, the same depth as the other per-CPU nesting limits of BPF
tracing. The helpers then fall back as they do when the trylock fails:
bpf_get_stack() and bpf_get_stackid() return IPs instead of build IDs,
and bpf_find_vma() returns -EBUSY.
The counter is per CPU, and programs with irqs enabled can be
preempted, so a preempted holder counts against the other tasks on its
CPU. They can hit the limit early and get the fallback; they never get
wrong data.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/bpf/mmap_unlock_work.h | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h
index 1834db20b..02af564e6 100644
--- a/kernel/bpf/mmap_unlock_work.h
+++ b/kernel/bpf/mmap_unlock_work.h
@@ -13,6 +13,7 @@ struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
atomic_t active;
+ int nest_level;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -28,8 +29,16 @@ static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
struct mmap_unlock_irq_work *work;
- if (!irqs_disabled())
- return NULL;
+ if (!irqs_disabled()) {
+ /*
+ * mmap_read_trylock() fires the mmap_lock tracepoints, and
+ * programs on them can get here again.
+ */
+ if (this_cpu_inc_return(mmap_unlock_work.nest_level) <= 3)
+ return NULL;
+ this_cpu_dec(mmap_unlock_work.nest_level);
+ return ERR_PTR(-EBUSY);
+ }
/*
* PREEMPT_RT does not allow to trylock mmap sem in interrupt
@@ -51,12 +60,15 @@ bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
{
if (work)
atomic_set_release(&work->active, 0);
+ else
+ this_cpu_dec(mmap_unlock_work.nest_level);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)
{
if (!work) {
mmap_read_unlock(mm);
+ bpf_mmap_unlock_guard_put(NULL);
} else {
work->mm = mm;
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread* [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
2026-11-20 10:59 [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
2 siblings, 0 replies; 4+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs have no pt_regs, so they fill in one of
three per-CPU slots, picked by a per-CPU nest counter:
get_bpf_raw_tp_regs() level++, use regs[level - 1]
put_bpf_raw_tp_regs() level--
That only works if the users of a CPU leave in the reverse order they
came, as a task, an interrupt and an NMI do. These programs run with
migration disabled but preemption enabled, so tasks can leave in any
order:
task A task B task C
get: regs[0]
<preempted>
get: regs[1]
<preempted>
put: level 1
get: regs[1]
fills in its regs
uses regs[1]
B sends a sample or a stack trace with the registers of C. Profilers and
tracers then attribute it to the wrong task, and nothing reports an
error. The per-CPU recursion counter of a program doesn't prevent it:
the slots are shared by all programs, so it takes two of them.
Fill in a pt_regs on the stack of each helper instead, as
bpf_get_stack_raw_tp_sleepable() already does, and remove the slots.
Each caller then has its own registers, however it nests or gets
preempted. This also removes the three-level nesting limit the slots put
on the three helpers. bpf_perf_event_output() keeps its own limit, and
the mmap_read_trylock() for build IDs has a separate one.
The cost is sizeof(struct pt_regs) of stack per call, 168 bytes on
x86-64, in helpers that can also run in IRQ and NMI context, and
zeroing it on every call. It has to be zeroed: perf_fetch_caller_regs()
sets only a few registers, and a sample with PERF_SAMPLE_REGS_INTR
copies all of them to user space.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 78 +++++++-----------------------------------------
1 file changed, 11 insertions(+), 67 deletions(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 8910d5d05..5c49c139d 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1624,51 +1624,13 @@ pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
}
}
-/*
- * bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
- * to avoid potential recursive reuse issue when/if tracepoints are added
- * inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
- *
- * Since raw tracepoints run despite bpf_prog_active, support concurrent usage
- * in normal, irq, and nmi context.
- */
-struct bpf_raw_tp_regs {
- struct pt_regs regs[3];
-};
-static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
-static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
-static struct pt_regs *get_bpf_raw_tp_regs(void)
-{
- struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
- int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
-
- if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
- this_cpu_dec(bpf_raw_tp_nest_level);
- return ERR_PTR(-EBUSY);
- }
-
- return &tp_regs->regs[nest_level - 1];
-}
-
-static void put_bpf_raw_tp_regs(void)
-{
- this_cpu_dec(bpf_raw_tp_nest_level);
-}
-
BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
-
- perf_fetch_caller_regs(regs);
- ret = ____bpf_perf_event_output(regs, map, flags, data, size);
+ struct pt_regs regs = {};
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return ____bpf_perf_event_output(®s, map, flags, data, size);
}
static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
@@ -1689,18 +1651,12 @@ extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
+ perf_fetch_caller_regs(®s);
/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
- ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
- flags, 0, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ return bpf_get_stackid((unsigned long) ®s, (unsigned long) map,
+ flags, 0, 0);
}
static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
@@ -1715,17 +1671,11 @@ static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
- ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
- (unsigned long) size, flags, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return bpf_get_stack((unsigned long) ®s, (unsigned long) buf,
+ (unsigned long) size, flags, 0);
}
static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
@@ -1741,12 +1691,6 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp_sleepable, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- /*
- * Don't take a per-CPU bpf_raw_tp_regs slot here: those slots are a
- * LIFO nest counter and are vulnerable to slot overwrite and
- * corruption across sleep. Sleepable programs never run in
- * IRQ/NMI context, so a local pt_regs is sufficient.
- */
struct pt_regs regs = {};
perf_fetch_caller_regs(®s);
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread