mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH bpf-next v2] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
@ 2026-11-19 10:37 Mike Bazov via B4 Relay
  2026-11-19 11:14 ` bot+bpf-ci
  0 siblings, 1 reply; 3+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-19 10:37 UTC (permalink / raw)
  To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
	Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
	Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
	Masami Hiramatsu, Mathieu Desnoyers
  Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov

From: Mike Bazov <mike.bazov@gmail.com>

bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs have no pt_regs, so they fill in one of
three per-CPU slots, picked by a per-CPU nest counter:

  get_bpf_raw_tp_regs()   level++, use regs[level - 1]
  put_bpf_raw_tp_regs()   level--

That only works if the users of a CPU leave in the reverse order they
came, as a task, an interrupt and an NMI do. These programs run with
migration disabled but preemption enabled, so tasks can leave in any
order:

  task A              task B              task C
  get: regs[0]
  <preempted>
                      get: regs[1]
                      <preempted>
  put: level 1
                                          get: regs[1]
                                          fills in its regs
                      uses regs[1]

B sends a sample or a stack trace with the registers of C. Profilers and
tracers then attribute it to the wrong task, and nothing reports an
error. The per-CPU recursion counter of a program doesn't prevent it:
the slots are shared by all programs, so it takes two of them.

Fill in a pt_regs on the stack of each helper instead, as
bpf_get_stack_raw_tp_sleepable() already does, and remove the slots.
Each caller then has its own registers, however it nests or gets
preempted, and the helpers no longer fail with -EBUSY when nested more
than three deep.

The cost is sizeof(struct pt_regs) of stack per call, 168 bytes on
x86-64, in helpers that can also run in IRQ and NMI context, and
zeroing it on every call. It has to be zeroed: perf_fetch_caller_regs()
sets only a few registers, and a sample with PERF_SAMPLE_REGS_INTR
copies all of them to user space.

Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
A reproducer, x86-64: two raw tracepoint programs on sys_enter call
bpf_perf_event_output() once each, with the program's number as data.
Two threads make syscalls on CPU 0, and a SCHED_FIFO thread preempts
them every 20us. The IP of a sample is regs->ip, the return address of
the helper call. The reproducer reads that address for both programs
from their JIT images and checks every sample.

Without this patch, 7 to 18 samples per program and 10s run have the IP
of the other program, out of 1.4M. With it, none. bpf_get_stack() and
bpf_get_stackid() in the same setup return an empty kernel stack
instead: the unwinder looks for the other task's sp on the current
stack and doesn't find it.

5 runs of 10s, debug config, CONFIG_PREEMPT=y, preempt=full, on
bpf-next 0fb18ce8a2e9:

                                     without     with
  bpf_perf_event_output, wrong IP    7-18        0      per program
  bpf_get_stack, empty stacks        362-535     0
  bpf_get_stackid, empty stacks      168-364     0

The bpf_get_stackid() case with bpftrace, whose kstack calls it; @a[]
and @b[] count the empty stacks:

bpftrace -e '
  rawtracepoint:sys_enter /comm == "dd"/ { @a[kstack(1)] = count(); }
  rawtracepoint:sys_enter /comm == "dd"/ { @b[kstack(1)] = count(); }' &
  taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
  taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
  taskset -c 0 chrt -f 1 perl -e 'select(undef, undef, undef, 2e-5) while 1' &
---
Changes in v2:
- Use a pt_regs on the stack of each helper and remove the per-CPU
  slots, instead of disabling preemption while a slot is in use
  (per Andrii's suggestion).
- Link to v1: https://lore.kernel.org/r/20261116-bpf-raw-tp-regs-preempt-v1-1-d01b6dd8573c@gmail.com
---
 kernel/trace/bpf_trace.c | 78 +++++++-----------------------------------------
 1 file changed, 11 insertions(+), 67 deletions(-)

diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index b4ff9d4e8..3f1901ae6 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1624,51 +1624,13 @@ pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 	}
 }
 
-/*
- * bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
- * to avoid potential recursive reuse issue when/if tracepoints are added
- * inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
- *
- * Since raw tracepoints run despite bpf_prog_active, support concurrent usage
- * in normal, irq, and nmi context.
- */
-struct bpf_raw_tp_regs {
-	struct pt_regs regs[3];
-};
-static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
-static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
-static struct pt_regs *get_bpf_raw_tp_regs(void)
-{
-	struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
-	int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
-
-	if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
-		this_cpu_dec(bpf_raw_tp_nest_level);
-		return ERR_PTR(-EBUSY);
-	}
-
-	return &tp_regs->regs[nest_level - 1];
-}
-
-static void put_bpf_raw_tp_regs(void)
-{
-	this_cpu_dec(bpf_raw_tp_nest_level);
-}
-
 BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
 	   struct bpf_map *, map, u64, flags, void *, data, u64, size)
 {
-	struct pt_regs *regs = get_bpf_raw_tp_regs();
-	int ret;
-
-	if (IS_ERR(regs))
-		return PTR_ERR(regs);
-
-	perf_fetch_caller_regs(regs);
-	ret = ____bpf_perf_event_output(regs, map, flags, data, size);
+	struct pt_regs regs = {};
 
-	put_bpf_raw_tp_regs();
-	return ret;
+	perf_fetch_caller_regs(&regs);
+	return ____bpf_perf_event_output(&regs, map, flags, data, size);
 }
 
 static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
@@ -1689,18 +1651,12 @@ extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
 BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
 	   struct bpf_map *, map, u64, flags)
 {
-	struct pt_regs *regs = get_bpf_raw_tp_regs();
-	int ret;
-
-	if (IS_ERR(regs))
-		return PTR_ERR(regs);
+	struct pt_regs regs = {};
 
-	perf_fetch_caller_regs(regs);
+	perf_fetch_caller_regs(&regs);
 	/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
-	ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
-			      flags, 0, 0);
-	put_bpf_raw_tp_regs();
-	return ret;
+	return bpf_get_stackid((unsigned long) &regs, (unsigned long) map,
+			       flags, 0, 0);
 }
 
 static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
@@ -1715,17 +1671,11 @@ static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
 BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
 	   void *, buf, u32, size, u64, flags)
 {
-	struct pt_regs *regs = get_bpf_raw_tp_regs();
-	int ret;
-
-	if (IS_ERR(regs))
-		return PTR_ERR(regs);
+	struct pt_regs regs = {};
 
-	perf_fetch_caller_regs(regs);
-	ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
-			    (unsigned long) size, flags, 0);
-	put_bpf_raw_tp_regs();
-	return ret;
+	perf_fetch_caller_regs(&regs);
+	return bpf_get_stack((unsigned long) &regs, (unsigned long) buf,
+			     (unsigned long) size, flags, 0);
 }
 
 static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
@@ -1741,12 +1691,6 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
 BPF_CALL_4(bpf_get_stack_raw_tp_sleepable, struct bpf_raw_tracepoint_args *, args,
 	   void *, buf, u32, size, u64, flags)
 {
-	/*
-	 * Don't take a per-CPU bpf_raw_tp_regs slot here: those slots are a
-	 * LIFO nest counter and are vulnerable to slot overwrite and
-	 * corruption across sleep.  Sleepable programs never run in
-	 * IRQ/NMI context, so a local pt_regs is sufficient.
-	 */
 	struct pt_regs regs = {};
 
 	perf_fetch_caller_regs(&regs);

---
base-commit: 8835113107b2cf75df33a71e265cdeec5d5d13e0
change-id: 20261116-bpf-raw-tp-regs-preempt-1f7019989ab9

Best regards,
-- 
Mike Bazov <mike.bazov@gmail.com>



^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH bpf-next v2] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
  2026-11-19 10:37 [PATCH bpf-next v2] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
@ 2026-11-19 11:14 ` bot+bpf-ci
  2026-11-19 19:28   ` Jiri Olsa
  0 siblings, 1 reply; 3+ messages in thread
From: bot+bpf-ci @ 2026-11-19 11:14 UTC (permalink / raw)
  To: mike.bazov, song, jolsa, ihor.solodrai, ast, daniel, andrii,
	eddyz87, memxor, martin.lau, yonghong.song, emil, kpsingh, matt,
	rostedt, mhiramat, mathieu.desnoyers
  Cc: bpf, linux-kernel, linux-trace-kernel, mike.bazov, ast, andrii,
	daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

> bpf: Use an on-stack pt_regs in the raw tracepoint helpers

TL;DR: Without the per-CPU slots, nested raw_tp/fentry programs may hit the
WARN_ON_ONCE in bpf_perf_event_output() (panic with panic_on_warn); the commit
message's claim of no more -EBUSY also looks wrong.

> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c

[ ... ]

>  BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
>  	   struct bpf_map *, map, u64, flags, void *, data, u64, size)
>  {
> -	struct pt_regs *regs = get_bpf_raw_tp_regs();
> -	int ret;
> -
> -	if (IS_ERR(regs))
> -		return PTR_ERR(regs);
> -
> -	perf_fetch_caller_regs(regs);
> -	ret = ____bpf_perf_event_output(regs, map, flags, data, size);
> +	struct pt_regs regs = {};
>  
> -	put_bpf_raw_tp_regs();
> -	return ret;
> +	perf_fetch_caller_regs(&regs);
> +	return ____bpf_perf_event_output(&regs, map, flags, data, size);
>  }

Can this reach the WARN_ON_ONCE() in bpf_perf_event_output() using only
raw_tp and tracing programs?

bpf_perf_event_output() still has a three-level limit:

    nest_level = this_cpu_inc_return(bpf_trace_nest_level);

    if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
        err = -EBUSY;
        goto out;
    }

Before this patch, every raw_tp/tracing caller first took a
bpf_raw_tp_nest_level slot, so a fourth nested caller got a silent -EBUSY
from get_bpf_raw_tp_regs().  That function's WARN_ON_ONCE was removed by
commit 3880cdbed1c4 ("bpf: Fix WARN() in get_bpf_raw_tp_regs") after a
syzbot report of this kind of nesting.  The WARN above could only fire if a
kprobe, tracepoint or perf_event program was also in the chain.

With the slots gone, a chain of only raw_tp/tracing programs can get there.
For example, a raw_tp program A calls bpf_perf_event_output() (nest 1),
and its __bpf_perf_event_output() calls perf_event_output().  Fentry
programs B, C and D attached to perf_event_output() also call
bpf_perf_event_output(), and each is let through once by its own
prog->active check, so they run at nest 2, 3 and 4.  At D the WARN_ON_ONCE
fires, and with panic_on_warn the kernel panics.  The same depth is
reachable across task, softirq, hardirq and NMI context.

This also seems to contradict the commit message:

> the helpers no longer fail with -EBUSY when nested more than three deep

For bpf_perf_event_output() the three-level limit remains, coming from
bpf_trace_sds and bpf_trace_nest_level.

Should the WARN_ON_ONCE in bpf_perf_event_output() be dropped, as
3880cdbed1c4 did for get_bpf_raw_tp_regs(), and should the commit message
say that bpf_perf_event_output() keeps its three-level limit?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44508926690

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH bpf-next v2] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
  2026-11-19 11:14 ` bot+bpf-ci
@ 2026-11-19 19:28   ` Jiri Olsa
  0 siblings, 0 replies; 3+ messages in thread
From: Jiri Olsa @ 2026-11-19 19:28 UTC (permalink / raw)
  To: bot+bpf-ci
  Cc: mike.bazov, song, ihor.solodrai, ast, daniel, andrii, eddyz87,
	memxor, martin.lau, yonghong.song, emil, kpsingh, matt, rostedt,
	mhiramat, mathieu.desnoyers, bpf, linux-kernel,
	linux-trace-kernel, martin.lau, mason

On Thu, Nov 19, 2026 at 11:14:51AM +0000, bot+bpf-ci@kernel.org wrote:
> > bpf: Use an on-stack pt_regs in the raw tracepoint helpers
> 
> TL;DR: Without the per-CPU slots, nested raw_tp/fentry programs may hit the
> WARN_ON_ONCE in bpf_perf_event_output() (panic with panic_on_warn); the commit
> message's claim of no more -EBUSY also looks wrong.
> 
> > diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> > --- a/kernel/trace/bpf_trace.c
> > +++ b/kernel/trace/bpf_trace.c
> 
> [ ... ]
> 
> >  BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
> >  	   struct bpf_map *, map, u64, flags, void *, data, u64, size)
> >  {
> > -	struct pt_regs *regs = get_bpf_raw_tp_regs();
> > -	int ret;
> > -
> > -	if (IS_ERR(regs))
> > -		return PTR_ERR(regs);
> > -
> > -	perf_fetch_caller_regs(regs);
> > -	ret = ____bpf_perf_event_output(regs, map, flags, data, size);
> > +	struct pt_regs regs = {};
> >  
> > -	put_bpf_raw_tp_regs();
> > -	return ret;
> > +	perf_fetch_caller_regs(&regs);
> > +	return ____bpf_perf_event_output(&regs, map, flags, data, size);
> >  }
> 
> Can this reach the WARN_ON_ONCE() in bpf_perf_event_output() using only
> raw_tp and tracing programs?
> 
> bpf_perf_event_output() still has a three-level limit:
> 
>     nest_level = this_cpu_inc_return(bpf_trace_nest_level);
> 
>     if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
>         err = -EBUSY;
>         goto out;
>     }
> 
> Before this patch, every raw_tp/tracing caller first took a
> bpf_raw_tp_nest_level slot, so a fourth nested caller got a silent -EBUSY
> from get_bpf_raw_tp_regs().  That function's WARN_ON_ONCE was removed by
> commit 3880cdbed1c4 ("bpf: Fix WARN() in get_bpf_raw_tp_regs") after a
> syzbot report of this kind of nesting.  The WARN above could only fire if a
> kprobe, tracepoint or perf_event program was also in the chain.
> 
> With the slots gone, a chain of only raw_tp/tracing programs can get there.
> For example, a raw_tp program A calls bpf_perf_event_output() (nest 1),
> and its __bpf_perf_event_output() calls perf_event_output().  Fentry
> programs B, C and D attached to perf_event_output() also call
> bpf_perf_event_output(), and each is let through once by its own
> prog->active check, so they run at nest 2, 3 and 4.  At D the WARN_ON_ONCE
> fires, and with panic_on_warn the kernel panics.  The same depth is
> reachable across task, softirq, hardirq and NMI context.
> 
> This also seems to contradict the commit message:
> 
> > the helpers no longer fail with -EBUSY when nested more than three deep
> 
> For bpf_perf_event_output() the three-level limit remains, coming from
> bpf_trace_sds and bpf_trace_nest_level.
> 
> Should the WARN_ON_ONCE in bpf_perf_event_output() be dropped, as
> 3880cdbed1c4 did for get_bpf_raw_tp_regs(), and should the commit message
> say that bpf_perf_event_output() keeps its three-level limit?

+1 we can now nest more than 3 times, hitting that limit shouldn't warn

jirka

> 
> 
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
> 
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44508926690


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-19 19:28 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 10:37 [PATCH bpf-next v2] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov via B4 Relay
2026-11-19 11:14 ` bot+bpf-ci
2026-11-19 19:28   ` Jiri Olsa

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®