* [PATCH bpf v3 0/3] bpf: Fix use-after-free of progs detached from busy trampolines
@ 2026-09-24 17:05 Florent Revest (Anthropic)
2026-09-24 17:05 ` [PATCH bpf v3 1/3] bpf: Wait for an RCU tasks grace period before freeing trampoline progs Florent Revest (Anthropic)
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Florent Revest (Anthropic) @ 2026-09-24 17:05 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: Florent Revest (Anthropic),
Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi,
Song Liu, Yonghong Song, Jiri Olsa, KP Singh, John Fastabend,
Leon Hwang, Junseo Lim, Sechang Lim, Puranjay Mohan, Xu Kuohai,
Ilya Leoshkevich, Hari Bathini, Christophe Leroy, Naveen N Rao,
Björn Töpel, Pu Lehui, Tiezhu Yang, Hengqi Chen,
linux-kernel
A task running in a trampoline image can call a prog that was detached
and freed in the meantime, when it slept in a sleepable prog before
reaching the detached one or was preempted right before calling it.
Patch 1 handles the preempted case with an RCU tasks grace period,
patch 2 handles the sleeping case by extending the ip_after_call nop
patching to every prog in the image and patch 3 adds a selftest for the
sleeping case.
v2 only patched the nop of the detached prog, at detach time, in the
images still in use, which needed the trampoline to track them. v3 goes
with Alexei's simpler proposal: bpf_tramp_image_put() patches all the
nops of the image it retires, the same way it patched ip_after_call,
which goes away. A task that is in an old image when the trampoline is
updated now skips the rest of that image's progs, attached or not, like
it already skipped its fexit progs.
v3 doesn't jump over the call to the original function. With a fexit
prog on a regular kernel function that call is the function itself, so a
task that is before it in an old image would return to its caller
without the function having run. The fentry_fexit subtest of patch 3
checks that it still runs.
sashiko noted that on riscv and loongarch a task can be preempted in
the middle of a multi-instruction patch site. ip_after_call has this too
and it isn't addressed here, pending input from the JIT maintainers on
whether a single instruction site is fine for these in-image jumps.
Tested on x86_64 under KVM and on arm64, s390x, powerpc64le, riscv64
and loongarch64 under qemu TCG, all with KASAN: the new selftest crashes
the unpatched kernel and passes with the series on every one of them,
along with trampoline_count, fentry/fexit, fentry_fexit, modify_return
and lsm_cgroup (and the full test_progs on x86_64). Same on bpf-next,
where patch 2 has a trivial conflict in x86's
arch_bpf_trampoline_size().
Changes since v2
(https://lore.kernel.org/bpf/20260912095924.866254-1-florent.revest@linux.dev/):
- Patch all the nops in bpf_tramp_image_put() instead of the detached
prog's nop at detach time, drop the image list, the trampoline
backpointer and ip_after_call (Alexei)
- Wait for an RCU tasks grace period before freeing progs that were
linked to a trampoline, for tasks preempted right before the enter
helper (Junseo, sashiko)
- Initialize the jit ctx in loongarch's arch_bpf_trampoline_size() and
the dummy image in every arch_bpf_trampoline_size() (bpf-ci)
- selftest: move the userfaultfd helper to testing_helpers.c and share
it with bpf_mod_race, comment fixes (bpf-ci), add a subtest where the
original function runs between the sleeping prog and the detached one
Changes since v1
(https://lore.kernel.org/bpf/20260819122252.1782790-1-florent.revest@linux.dev/):
- Patch nops in front of detached progs instead of taking prog
references from the image (Alexei)
- Lower BPF_MAX_TRAMP_LINKS on arm64, loongarch and powerpc so the
image still fits in a page
- Explain that the sleepable case doesn't depend on CONFIG_PREEMPTION
(Kumar, Alexei)
- Add a selftest (Jiri, Alexei)
- Add Sechang's Reported-by (Junseo, Kumar)
- Drop Leon's and Kumar's acks since the code changed entirely
Florent Revest (Anthropic) (3):
bpf: Wait for an RCU tasks grace period before freeing trampoline
progs
bpf: Skip the progs of trampoline images that are being freed
selftests/bpf: Detach a trampoline prog while a task sleeps before it
arch/arm64/net/bpf_jit_comp.c | 33 ++--
arch/loongarch/net/bpf_jit.c | 45 ++---
arch/powerpc/net/bpf_jit_comp.c | 45 ++---
arch/riscv/net/bpf_jit_comp64.c | 39 +++--
arch/s390/net/bpf_jit_comp.c | 46 +++--
arch/x86/net/bpf_jit_comp.c | 26 +--
include/linux/bpf.h | 36 +++-
kernel/bpf/syscall.c | 19 ++-
kernel/bpf/trampoline.c | 37 ++--
.../selftests/bpf/prog_tests/bpf_mod_race.c | 34 +---
.../bpf/prog_tests/tramp_prog_detach.c | 158 ++++++++++++++++++
.../selftests/bpf/progs/tramp_prog_detach.c | 56 +++++++
tools/testing/selftests/bpf/testing_helpers.c | 28 ++++
tools/testing/selftests/bpf/testing_helpers.h | 2 +
14 files changed, 453 insertions(+), 151 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c
create mode 100644 tools/testing/selftests/bpf/progs/tramp_prog_detach.c
base-commit: 5fc5768c7ca92895ccd1de94dc521e5a55ae7896
--
2.55.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH bpf v3 1/3] bpf: Wait for an RCU tasks grace period before freeing trampoline progs 2026-09-24 17:05 [PATCH bpf v3 0/3] bpf: Fix use-after-free of progs detached from busy trampolines Florent Revest (Anthropic) @ 2026-09-24 17:05 ` Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it Florent Revest (Anthropic) 2 siblings, 0 replies; 6+ messages in thread From: Florent Revest (Anthropic) @ 2026-09-24 17:05 UTC (permalink / raw) To: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko Cc: Florent Revest (Anthropic), Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa, KP Singh, John Fastabend, Leon Hwang, Junseo Lim, Sechang Lim, Puranjay Mohan, Xu Kuohai, Ilya Leoshkevich, Hari Bathini, Christophe Leroy, Naveen N Rao, Björn Töpel, Pu Lehui, Tiezhu Yang, Hengqi Chen, linux-kernel When a prog is detached from a trampoline, it is freed after an RCU grace period, or an RCU tasks trace one if it is sleepable. This covers the tasks that are running the prog, since the prog's enter helper takes the matching read lock before the prog is called. On a preemptible kernel, it doesn't cover a task that was preempted in the trampoline just before the enter helper. That task holds no lock yet, and it calls the prog after it was freed: BUG: KASAN: vmalloc-out-of-bounds in __bpf_prog_enter_recur+0x3a5/0x3f0 Read of size 8 at addr ffffc90000055040 by task candidate/110 CPU: 1 UID: 0 PID: 110 Comm: candidate Not tainted 7.3.0-rc2-00014-g15071f2a1263-dirty #2 PREEMPT(full) Call Trace: <TASK> __bpf_prog_enter_recur+0x3a5/0x3f0 bpf_trampoline_6442509193+0x37/0xf1 __x64_sys_futex+0x9/0x410 do_syscall_64+0xb0/0x530 ... Wait for an RCU tasks grace period before the existing one when freeing a prog that was linked to a trampoline. An RCU tasks grace period only ends once the tasks that were preempted have run again, and bpf_tramp_image_put() already relies on it to free the image. It doesn't wait for tasks that sleep in the trampoline, the next commit takes care of those. Only progs that were linked to a trampoline can be called this way, so bpf_trampoline_add_prog() marks them and other progs are still freed as before. Fixes: e21aa341785c ("bpf: Fix fexit trampoline.") Reported-by: Junseo Lim <zirajs7@gmail.com> Closes: https://lore.kernel.org/bpf/aqdrwVpanH3WGurX@omen-arch/ Assisted-by: Claude:unspecified Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev> --- include/linux/bpf.h | 1 + kernel/bpf/syscall.c | 19 ++++++++++++++++++- kernel/bpf/trampoline.c | 1 + 3 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1d2676782d70..d4f732996b47 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1770,6 +1770,7 @@ struct bpf_prog_aux { bool offload_requested; /* Program is bound and offloaded to the netdev. */ bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */ bool attach_tracing_prog; /* true if tracing another tracing program */ + bool tramp_linked; /* true if it was ever called from a trampoline */ bool func_proto_unreliable; bool tail_call_reachable; bool xdp_has_frags; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 244a939b9d2d..96217b99399d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2448,6 +2448,21 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu) bpf_prog_free(aux->prog); } +/* + * Progs called from a trampoline can also be reached by a task that was + * preempted in the trampoline before the prog's enter helper took its RCU + * read lock, wait for those first. + */ +static void __bpf_prog_put_rcu_tasks(struct rcu_head *rcu) +{ + struct bpf_prog *prog = container_of(rcu, struct bpf_prog_aux, rcu)->prog; + + if (prog->sleepable) + call_rcu_tasks_trace(rcu, __bpf_prog_put_rcu); + else + call_rcu(rcu, __bpf_prog_put_rcu); +} + static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) { bpf_prog_kallsyms_del_all(prog); @@ -2461,7 +2476,9 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) btf_put(prog->aux->attach_btf); if (deferred) { - if (prog->sleepable) + if (IS_ENABLED(CONFIG_TASKS_RCU) && prog->aux->tramp_linked) + call_rcu_tasks(&prog->aux->rcu, __bpf_prog_put_rcu_tasks); + else if (prog->sleepable) call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu); else call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 90b70ea0d370..9d69c066a817 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -907,6 +907,7 @@ static int bpf_trampoline_add_prog(struct bpf_trampoline *tr, } hlist_add_head(&node->tramp_hlist, prog_list); + node->link->prog->aux->tramp_linked = true; if (kind == BPF_TRAMP_FSESSION) { tr->progs_cnt[BPF_TRAMP_FENTRY]++; fexit = fsession_exit(node); -- 2.55.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed 2026-09-24 17:05 [PATCH bpf v3 0/3] bpf: Fix use-after-free of progs detached from busy trampolines Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 1/3] bpf: Wait for an RCU tasks grace period before freeing trampoline progs Florent Revest (Anthropic) @ 2026-09-24 17:05 ` Florent Revest (Anthropic) 2026-09-24 18:07 ` bot+bpf-ci 2026-09-24 17:05 ` [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it Florent Revest (Anthropic) 2 siblings, 1 reply; 6+ messages in thread From: Florent Revest (Anthropic) @ 2026-09-24 17:05 UTC (permalink / raw) To: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko Cc: Florent Revest (Anthropic), Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa, KP Singh, John Fastabend, Leon Hwang, Junseo Lim, Sechang Lim, Puranjay Mohan, Xu Kuohai, Ilya Leoshkevich, Hari Bathini, Christophe Leroy, Naveen N Rao, Björn Töpel, Pu Lehui, Tiezhu Yang, Hengqi Chen, linux-kernel bpf_tramp_image_put() makes sure a trampoline image is not freed while a task may still be running in it, but nothing similar is done for the progs called by that image. Detach drops the last prog reference right away and the prog is freed after grace periods, on the basis that a task still in the traced function skips the fexit progs once the nop at ip_after_call is patched to a jump. That leaves out a task sleeping in a sleepable prog that runs before the detached one, which no grace period waits for: CPU 0 CPU 1 in image I, sleeping in prog S detach P from I's trampoline -> new image, bpf_tramp_image_put(I) bpf_prog_put(P), last ref grace periods, P freed back from S __bpf_prog_enter(P) call P->bpf_func If S and P are fexit progs the task is already past the patched jump, and fentry only images don't have one. On x86 this is an int3 in poisoned bpf_prog_pack memory: Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ... We hit this in production when progs attached through trampolines got detached while their hooks were busy, and it was independently found with a fuzzer and KASAN. Extend what ip_after_call does to every prog: have the JITs emit a patchable nop in front of each prog call sequence and record it in the image, and have bpf_tramp_image_put() patch them all to jumps over the call sequences, which replaces ip_after_call. Tasks that are still in an image that was put skip all of its progs from then on, including the ones that are still attached, like they already skipped its fexit progs. The call to the original function is left alone, a task that is before it in the image still has to run it. With the extra nops, BPF_MAX_TRAMP_LINKS progs no longer fit in a page on arm64 and loongarch (and already didn't on powerpc), so lower the limit there like s390 does. Fixes: e21aa341785c ("bpf: Fix fexit trampoline.") Reported-by: Sechang Lim <rhkrqnwk98@gmail.com> Closes: https://lore.kernel.org/bpf/20260815071927.147049-1-zirajs7@gmail.com/ Suggested-by: Alexei Starovoitov <ast@kernel.org> Assisted-by: Claude:unspecified Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev> --- arch/arm64/net/bpf_jit_comp.c | 33 ++++++++++++----------- arch/loongarch/net/bpf_jit.c | 45 +++++++++++++++++--------------- arch/powerpc/net/bpf_jit_comp.c | 45 ++++++++++++++++---------------- arch/riscv/net/bpf_jit_comp64.c | 39 ++++++++++++++++------------ arch/s390/net/bpf_jit_comp.c | 46 +++++++++++++++++++++------------ arch/x86/net/bpf_jit_comp.c | 26 ++++++++++++------- include/linux/bpf.h | 35 ++++++++++++++++++++++--- kernel/bpf/trampoline.c | 36 +++++++++++++++++--------- 8 files changed, 188 insertions(+), 117 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c5f55d6161fe..e3a8e7e12127 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2418,10 +2418,11 @@ bool bpf_jit_supports_subprog_tailcalls(void) return true; } -static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node, - int bargs_off, int retval_off, int run_ctx_off, - bool save_ret) +static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im, + struct bpf_tramp_node *node, int bargs_off, + int retval_off, int run_ctx_off, bool save_ret) { + void *skip; __le32 *branch; u64 enter_prog; u64 exit_prog; @@ -2431,6 +2432,10 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node, enter_prog = (u64)bpf_trampoline_enter(p); exit_prog = (u64)bpf_trampoline_exit(p); + /* nop, patched to skip this prog when the image is put */ + skip = ctx->ro_image + ctx->idx; + emit(A64_NOP, ctx); + if (node->cookie == 0) { /* if cookie is zero, one instruction is enough to store it */ emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx); @@ -2483,11 +2488,13 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node, emit(A64_ADD_I(1, A64_R(2), A64_SP, run_ctx_off), ctx); emit_call(exit_prog, ctx); + + bpf_tramp_image_add_skip(im, skip, ctx->ro_image + ctx->idx); } -static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, - int bargs_off, int retval_off, int run_ctx_off, - __le32 **branches) +static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_image *im, + struct bpf_tramp_nodes *tn, int bargs_off, + int retval_off, int run_ctx_off, __le32 **branches) { int i; @@ -2496,7 +2503,7 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, */ emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx); for (i = 0; i < tn->nr_nodes; i++) { - invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off, + invoke_bpf_prog(ctx, im, tn->nodes[i], bargs_off, retval_off, run_ctx_off, true); /* if (*(u64 *)(sp + retval_off) != 0) * goto do_fexit; @@ -2882,7 +2889,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, store_func_meta(ctx, meta, func_meta_off); cookie_bargs_off--; } - invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off, + invoke_bpf_prog(ctx, im, fentry->nodes[i], bargs_off, retval_off, run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET); } @@ -2893,7 +2900,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, if (!branches) return -ENOMEM; - invoke_bpf_mod_ret(ctx, fmod_ret, bargs_off, retval_off, + invoke_bpf_mod_ret(ctx, im, fmod_ret, bargs_off, retval_off, run_ctx_off, branches); } @@ -2906,9 +2913,6 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, emit(A64_RET(A64_R(10)), ctx); /* store return value */ emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); - /* reserve a nop for bpf_tramp_image_put */ - im->ip_after_call = ctx->ro_image + ctx->idx; - emit(A64_NOP, ctx); } /* update the branches saved in invoke_bpf_mod_ret with cbnz */ @@ -2930,12 +2934,11 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, store_func_meta(ctx, meta, func_meta_off); cookie_bargs_off--; } - invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off, + invoke_bpf_prog(ctx, im, fexit->nodes[i], bargs_off, retval_off, run_ctx_off, false); } if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = ctx->ro_image + ctx->idx; /* for the first pass, assume the worst case */ if (!ctx->image) ctx->idx += 4; @@ -2994,7 +2997,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, .image = NULL, .idx = 0, }; - struct bpf_tramp_image im; + struct bpf_tramp_image im = {}; struct arg_aux aaux; int ret; diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c index 4da278900938..d8a5ab288570 100644 --- a/arch/loongarch/net/bpf_jit.c +++ b/arch/loongarch/net/bpf_jit.c @@ -1696,13 +1696,19 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off, } } -static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n, - int args_off, int retval_off, int run_ctx_off, bool save_ret) +static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im, + struct bpf_tramp_node *n, int args_off, int retval_off, + int run_ctx_off, bool save_ret) { - int ret; + int i, ret; u32 *branch; struct bpf_prog *p = n->link->prog; int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); + void *skip = ctx->ro_image + ctx->idx; + + /* nops for move_imm+jirl, patched to skip this prog when the image is put */ + for (i = 0; i < LOONGARCH_LONG_JUMP_NINSNS; i++) + emit_insn(ctx, nop); if (n->cookie) emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, @@ -1755,13 +1761,17 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n, /* arg3: &run_ctx */ emit_insn(ctx, addid, LOONGARCH_GPR_A2, LOONGARCH_GPR_FP, -run_ctx_off); ret = emit_call(ctx, (const u64)bpf_trampoline_exit(p)); + if (ret) + return ret; - return ret; + bpf_tramp_image_add_skip(im, skip, ctx->ro_image + ctx->idx); + return 0; } -static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, - int args_off, int retval_off, int run_ctx_off, - int func_meta_off, bool save_ret, u64 func_meta, int cookie_off) +static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_image *im, + struct bpf_tramp_nodes *tn, int args_off, int retval_off, + int run_ctx_off, int func_meta_off, bool save_ret, + u64 func_meta, int cookie_off) { int i, cur_cookie = (cookie_off - args_off) / 8; @@ -1774,7 +1784,8 @@ static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta); cur_cookie--; } - err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret); + err = invoke_bpf_prog(ctx, im, tn->nodes[i], args_off, retval_off, + run_ctx_off, save_ret); if (err) return err; } @@ -2017,7 +2028,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i } if (fentry->nr_nodes) { - ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off, + ret = invoke_bpf(ctx, im, fentry, args_off, retval_off, run_ctx_off, func_meta_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off); if (ret) return ret; @@ -2029,7 +2040,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off); for (i = 0; i < fmod_ret->nr_nodes; i++) { - ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i], + ret = invoke_bpf_prog(ctx, im, fmod_ret->nodes[i], args_off, retval_off, run_ctx_off, true); if (ret) goto out; @@ -2051,10 +2062,6 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i goto out; emit_insn(ctx, std, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -retval_off); emit_insn(ctx, std, regmap[BPF_REG_0], LOONGARCH_GPR_FP, -(retval_off - 8)); - im->ip_after_call = ctx->ro_image + ctx->idx; - /* Reserve space for the move_imm + jirl instruction */ - for (i = 0; i < LOONGARCH_LONG_JUMP_NINSNS; i++) - emit_insn(ctx, nop); } for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) { @@ -2068,14 +2075,13 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta); if (fexit->nr_nodes) { - ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off, + ret = invoke_bpf(ctx, im, fexit, args_off, retval_off, run_ctx_off, func_meta_off, false, func_meta, cookie_off); if (ret) goto out; } if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = ctx->ro_image + ctx->idx; move_addr(ctx, LOONGARCH_GPR_A0, (const u64)im); ret = emit_call(ctx, (const u64)__bpf_tramp_exit); if (ret) @@ -2177,11 +2183,8 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { int ret; - struct jit_ctx ctx; - struct bpf_tramp_image im; - - ctx.image = NULL; - ctx.idx = 0; + struct jit_ctx ctx = {}; + struct bpf_tramp_image im = {}; ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags); diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index 7b07b43575f1..5b71ae6f1014 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -602,14 +602,18 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) } static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx, - struct bpf_tramp_node *n, int regs_off, int retval_off, - int run_ctx_off, bool save_ret) + struct bpf_tramp_image *im, struct bpf_tramp_node *n, + int regs_off, int retval_off, int run_ctx_off, bool save_ret) { struct bpf_prog *p = n->link->prog; ppc_inst_t branch_insn; - u32 jmp_idx; + u32 jmp_idx, skip_idx; int ret = 0; + /* nop, patched to skip this prog when the image is put */ + skip_idx = ctx->idx; + EMIT(PPC_RAW_NOP()); + /* Save cookie */ if (IS_ENABLED(CONFIG_PPC64)) { PPC_LI64(_R3, n->cookie); @@ -679,13 +683,17 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct EMIT(PPC_RAW_ADDI(_R5, _R1, run_ctx_off)); ret = bpf_jit_emit_func_call_rel(image, ro_image, ctx, (unsigned long)bpf_trampoline_exit(p)); + if (ret) + return ret; - return ret; + if (ro_image) /* image is NULL for dummy pass */ + bpf_tramp_image_add_skip(im, &ro_image[skip_idx], &ro_image[ctx->idx]); + return 0; } static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx, - struct bpf_tramp_nodes *tn, int regs_off, int retval_off, - int run_ctx_off, u32 *branches) + struct bpf_tramp_image *im, struct bpf_tramp_nodes *tn, + int regs_off, int retval_off, int run_ctx_off, u32 *branches) { int i; @@ -696,8 +704,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context EMIT(PPC_RAW_LI(_R3, 0)); EMIT(PPC_RAW_STL(_R3, _R1, retval_off)); for (i = 0; i < tn->nr_nodes; i++) { - if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off, - run_ctx_off, true)) + if (invoke_bpf_prog(image, ro_image, ctx, im, tn->nodes[i], regs_off, + retval_off, run_ctx_off, true)) return -EINVAL; /* @@ -1043,8 +1051,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im cookie_ctx_off--; } - if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off, - run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) + if (invoke_bpf_prog(image, ro_image, ctx, im, fentry->nodes[i], regs_off, + retval_off, run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) return -EINVAL; } @@ -1053,7 +1061,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im if (!branches) return -ENOMEM; - if (invoke_bpf_mod_ret(image, ro_image, ctx, fmod_ret, regs_off, retval_off, + if (invoke_bpf_mod_ret(image, ro_image, ctx, im, fmod_ret, regs_off, retval_off, run_ctx_off, branches)) { ret = -EINVAL; goto cleanup; @@ -1090,11 +1098,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im /* Restore updated tail_call_cnt */ if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) bpf_trampoline_restore_tail_call_cnt(image, ctx, bpf_frame_size, r4_off); - - /* Reserve space to patch branch instruction to skip fexit progs */ - if (ro_image) /* image is NULL for dummy pass */ - im->ip_after_call = &((u32 *)ro_image)[ctx->idx]; - EMIT(PPC_RAW_NOP()); } /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */ @@ -1123,16 +1126,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im cookie_ctx_off--; } - if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off, - run_ctx_off, false)) { + if (invoke_bpf_prog(image, ro_image, ctx, im, fexit->nodes[i], regs_off, + retval_off, run_ctx_off, false)) { ret = -EINVAL; goto cleanup; } } if (flags & BPF_TRAMP_F_CALL_ORIG) { - if (ro_image) /* image is NULL for dummy pass */ - im->ip_epilogue = &((u32 *)ro_image)[ctx->idx]; PPC_LI_ADDR(_R3, im); ret = bpf_jit_emit_func_call_rel(image, ro_image, ctx, (unsigned long)__bpf_tramp_exit); @@ -1192,7 +1193,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { - struct bpf_tramp_image im; + struct bpf_tramp_image im = {}; int ret; ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr); @@ -1320,7 +1321,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, /* * If we are not poking at bpf prog entry, then we are simply patching in/out - * an unconditional branch instruction at im->ip_after_call + * an unconditional branch instruction in a trampoline image */ if (offset) { if (old_t == BPF_MOD_CALL || new_t == BPF_MOD_CALL) { diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 151031e97a24..a4c6c4db23e0 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -904,12 +904,18 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64, emit_sd(RV_REG_FP, stack_off, reg, ctx); } -static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off, - int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) +static int invoke_bpf_prog(struct bpf_tramp_image *im, struct bpf_tramp_node *node, + int args_off, int retval_off, int run_ctx_off, bool save_ret, + struct rv_jit_context *ctx) { int ret, branch_off; struct bpf_prog *p = node->link->prog; int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); + void *skip = ctx->ro_insns + ctx->ninsns; + + /* 2 nops for auipc+jalr, patched to skip this prog when the image is put */ + emit(rv_nop(), ctx); + emit(rv_nop(), ctx); if (node->cookie) emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx); @@ -962,13 +968,17 @@ static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval /* arg3: &run_ctx */ emit_addi(RV_REG_A2, RV_REG_FP, -run_ctx_off, ctx); ret = emit_call((const u64)bpf_trampoline_exit(p), true, ctx); + if (ret) + return ret; - return ret; + bpf_tramp_image_add_skip(im, skip, ctx->ro_insns + ctx->ninsns); + return 0; } -static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off, - int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta, - int cookie_off, struct rv_jit_context *ctx) +static int invoke_bpf(struct bpf_tramp_image *im, struct bpf_tramp_nodes *tn, + int args_off, int retval_off, int run_ctx_off, int func_meta_off, + bool save_ret, u64 func_meta, int cookie_off, + struct rv_jit_context *ctx) { int i, cur_cookie = (cookie_off - args_off) / 8; @@ -981,8 +991,8 @@ static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off, emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx); cur_cookie--; } - err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off, - save_ret, ctx); + err = invoke_bpf_prog(im, tn->nodes[i], args_off, retval_off, + run_ctx_off, save_ret, ctx); if (err) return err; } @@ -1170,7 +1180,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, } if (fentry->nr_nodes) { - ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off, + ret = invoke_bpf(im, fentry, args_off, retval_off, run_ctx_off, func_meta_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx); if (ret) return ret; @@ -1184,7 +1194,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, /* cleanup to avoid garbage return value confusion */ emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx); for (i = 0; i < fmod_ret->nr_nodes; i++) { - ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off, + ret = invoke_bpf_prog(im, fmod_ret->nodes[i], args_off, retval_off, run_ctx_off, true, ctx); if (ret) goto out; @@ -1211,10 +1221,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx); emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx); emit_sd(RV_REG_FP, -(retval_off - 8), regmap[BPF_REG_0], ctx); - im->ip_after_call = ctx->ro_insns + ctx->ninsns; - /* 2 nops reserved for auipc+jalr pair */ - emit(rv_nop(), ctx); - emit(rv_nop(), ctx); } /* update branches saved in invoke_bpf_mod_ret with bnez */ @@ -1230,14 +1236,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx); if (fexit->nr_nodes) { - ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off, + ret = invoke_bpf(im, fexit, args_off, retval_off, run_ctx_off, func_meta_off, false, func_meta, cookie_off, ctx); if (ret) goto out; } if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = ctx->ro_insns + ctx->ninsns; emit_imm(RV_REG_A0, ctx->insns ? (const s64)im : RV_MAX_COUNT_IMM, ctx); ret = emit_call((const u64)__bpf_tramp_exit, true, ctx); if (ret) @@ -1299,7 +1304,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { - struct bpf_tramp_image im; + struct bpf_tramp_image im = {}; struct rv_jit_context ctx; int ret; diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index c4b47070bb59..2c18f2a142a9 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2566,6 +2566,8 @@ struct bpf_tramp_jit { int r14_off; /* Offset of saved %r14, has to be at the * bottom */ int do_fexit; /* do_fexit: label */ + int skip[BPF_MAX_TRAMP_LINKS]; /* skip: labels after each prog */ + int nr_progs; }; static void load_imm64(struct bpf_jit *jit, int dst_reg, u64 val) @@ -2584,6 +2586,7 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o } static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, + struct bpf_tramp_image *im, const struct btf_func_model *m, struct bpf_tramp_node *node, bool save_ret) { @@ -2591,8 +2594,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, int cookie_off = tjit->run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog *p = node->link->prog; + void *skip = jit->prg_buf + jit->prg; + int idx = tjit->nr_progs++; int patch; + if (idx >= ARRAY_SIZE(tjit->skip)) + return -E2BIG; + + /* + * nop, patched to skip this prog when the image is put + */ + + /* brcl 0,skip */ + EMIT6_PCREL_RILC(0xc0040000, 0, tjit->skip[idx]); + /* * run_ctx.cookie = node->cookie; */ @@ -2652,10 +2667,15 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, /* brasl %r14,__bpf_prog_exit */ EMIT6_PCREL_RILB_PTR(0xc0050000, REG_14, bpf_trampoline_exit(p)); + /* skip: */ + tjit->skip[idx] = jit->prg; + bpf_tramp_image_add_skip(im, skip, jit->prg_buf + jit->prg); + return 0; } static int invoke_bpf(struct bpf_tramp_jit *tjit, + struct bpf_tramp_image *im, const struct btf_func_model *m, struct bpf_tramp_nodes *tn, bool save_ret, u64 func_meta, int cookie_off) @@ -2670,7 +2690,7 @@ static int invoke_bpf(struct bpf_tramp_jit *tjit, emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta); cur_cookie--; } - if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret)) + if (invoke_bpf_prog(tjit, im, m, tn->nodes[i], save_ret)) return -EINVAL; } @@ -2712,6 +2732,11 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, u64 func_meta; int i, j; + /* The skip labels are taken from the previous pass. */ + tjit->nr_progs = 0; + if (im) + im->nr_skips = 0; + /* Support as many stack arguments as "mvc" instruction can handle. */ nr_reg_args = min_t(int, m->nr_args, MAX_NR_REG_ARGS); nr_stack_args = m->nr_args - nr_reg_args; @@ -2875,7 +2900,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, emit_store_stack_imm64(jit, REG_0, tjit->retval_off, 0); } - if (invoke_bpf(tjit, m, fentry, flags & BPF_TRAMP_F_RET_FENTRY_RET, + if (invoke_bpf(tjit, im, m, fentry, flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off)) return -EINVAL; @@ -2889,7 +2914,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, 0xf000 | tjit->retval_off); for (i = 0; i < fmod_ret->nr_nodes; i++) { - if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true)) + if (invoke_bpf_prog(tjit, im, m, fmod_ret->nodes[i], true)) return -EINVAL; /* @@ -2943,15 +2968,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, /* mvc tccnt_off(%r15),tail_call_cnt(4,%r15) */ _EMIT6(0xd203f000 | tjit->tccnt_off, 0xf000 | offsetof(struct prog_frame, tail_call_cnt)); - - im->ip_after_call = jit->prg_buf + jit->prg; - - /* - * The following nop will be patched by bpf_tramp_image_put(). - */ - - /* brcl 0,im->ip_epilogue */ - EMIT6_PCREL_RILC(0xc0040000, 0, (u64)im->ip_epilogue); } /* Set the "is_return" flag for fsession. */ @@ -2962,12 +2978,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, /* do_fexit: */ tjit->do_fexit = jit->prg; - if (invoke_bpf(tjit, m, fexit, false, func_meta, cookie_off)) + if (invoke_bpf(tjit, im, m, fexit, false, func_meta, cookie_off)) return -EINVAL; if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = jit->prg_buf + jit->prg; - /* * __bpf_tramp_exit(im); */ @@ -3016,7 +3030,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *orig_call) { - struct bpf_tramp_image im; + struct bpf_tramp_image im = {}; struct bpf_tramp_jit tjit; int ret; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2853e87797a7..7ef5c8efc882 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3217,16 +3217,21 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, + struct bpf_tramp_image *im, struct bpf_tramp_node *node, int stack_size, int run_ctx_off, bool save_ret, void *image, void *rw_image) { u8 *prog = *pprog; - u8 *jmp_insn; + u8 *jmp_insn, *skip; int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog *p = node->link->prog; u64 cookie = node->cookie; + /* nop, patched to skip this prog when the image is put */ + skip = image + (prog - (u8 *)rw_image); + emit_nops(&prog, X86_PATCH_SIZE); + /* mov rdi, cookie */ emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); @@ -3301,6 +3306,8 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 *)rw_image))) return -EINVAL; + bpf_tramp_image_add_skip(im, skip, image + (prog - (u8 *)rw_image)); + *pprog = prog; return 0; } @@ -3332,6 +3339,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) } static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, + struct bpf_tramp_image *im, struct bpf_tramp_nodes *tl, int stack_size, int run_ctx_off, int func_meta_off, bool save_ret, void *image, void *rw_image, u64 func_meta, @@ -3346,7 +3354,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); cur_cookie--; } - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, + if (invoke_bpf_prog(m, &prog, im, tl->nodes[i], stack_size, run_ctx_off, save_ret, image, rw_image)) return -EINVAL; } @@ -3355,6 +3363,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, } static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, + struct bpf_tramp_image *im, struct bpf_tramp_nodes *tl, int stack_size, int run_ctx_off, u8 **branches, void *image, void *rw_image) @@ -3368,7 +3377,7 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, emit_mov_imm32(&prog, false, BPF_REG_0, 0); emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); for (i = 0; i < tl->nr_nodes; i++) { - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, + if (invoke_bpf_prog(m, &prog, im, tl->nodes[i], stack_size, run_ctx_off, true, image, rw_image)) return -EINVAL; @@ -3640,7 +3649,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im } if (fentry->nr_nodes) { - if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, &prog, im, fentry, regs_off, run_ctx_off, func_meta_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, func_meta, cookie_off)) return -EINVAL; @@ -3652,7 +3661,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im if (!branches) return -ENOMEM; - if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off, + if (invoke_bpf_mod_ret(m, &prog, im, fmod_ret, regs_off, run_ctx_off, branches, image, rw_image)) { ret = -EINVAL; goto cleanup; @@ -3682,8 +3691,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im } /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); - im->ip_after_call = image + (prog - (u8 *)rw_image); - emit_nops(&prog, X86_PATCH_SIZE); } if (fmod_ret->nr_nodes) { @@ -3708,7 +3715,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); if (fexit->nr_nodes) { - if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, &prog, im, fexit, regs_off, run_ctx_off, func_meta_off, false, image, rw_image, func_meta, cookie_off)) { ret = -EINVAL; goto cleanup; @@ -3723,7 +3730,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im * restored to R0. */ if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = image + (prog - (u8 *)rw_image); /* arg1: mov rdi, im */ emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 *)rw_image))) { @@ -3811,7 +3817,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { - struct bpf_tramp_image im; + struct bpf_tramp_image im = {}; void *image; int ret; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index d4f732996b47..9c61803380c2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1258,11 +1258,15 @@ struct btf_func_model { #define BPF_TRAMP_F_INDIRECT BIT(8) /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 - * bytes on x86. + * bytes on x86. The trampoline image has to fit in PAGE_SIZE. */ enum { -#if defined(__s390x__) +#if defined(__s390x__) || defined(__powerpc64__) BPF_MAX_TRAMP_LINKS = 27, +#elif defined(__loongarch__) + BPF_MAX_TRAMP_LINKS = 33, +#elif defined(__aarch64__) + BPF_MAX_TRAMP_LINKS = 37, #else BPF_MAX_TRAMP_LINKS = 38, #endif @@ -1363,19 +1367,42 @@ enum bpf_tramp_prog_type { BPF_TRAMP_FSESSION, }; +/* + * Each prog call in a trampoline image is preceded by a nop. When the image is + * put, the nops are patched to jumps to target, right after each call, so that + * tasks still running in the image skip the progs, which can be freed by then. + */ +struct bpf_tramp_skip { + void *nop; + void *target; +}; + struct bpf_tramp_image { void *image; int size; struct bpf_ksym ksym; struct percpu_ref pcref; - void *ip_after_call; - void *ip_epilogue; + bool call_orig; + int nr_skips; + struct bpf_tramp_skip *skips; union { struct rcu_head rcu; struct work_struct work; }; }; +static inline void bpf_tramp_image_add_skip(struct bpf_tramp_image *im, void *nop, void *target) +{ + struct bpf_tramp_skip *skip; + + /* struct_ops trampolines and arch_bpf_trampoline_size() have no image */ + if (!im || !im->skips) + return; + skip = &im->skips[im->nr_skips++]; + skip->nop = nop; + skip->target = target; +} + struct bpf_trampoline { /* hlist for trampoline_key_table */ struct hlist_node hlist_key; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 9d69c066a817..0d7aa1b0fe30 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -565,6 +565,7 @@ static void bpf_tramp_image_free(struct bpf_tramp_image *im) arch_free_bpf_trampoline(im->image, im->size); bpf_jit_uncharge_modmem(im->size); percpu_ref_exit(&im->pcref); + kfree(im->skips); kfree_rcu(im, rcu); } @@ -601,7 +602,7 @@ static void __bpf_tramp_image_put_rcu_tasks(struct rcu_head *rcu) struct bpf_tramp_image *im; im = container_of(rcu, struct bpf_tramp_image, rcu); - if (im->ip_after_call) + if (im->call_orig) /* the case of fmod_ret/fexit trampoline and CONFIG_PREEMPTION=y */ percpu_ref_kill(&im->pcref); else @@ -611,6 +612,8 @@ static void __bpf_tramp_image_put_rcu_tasks(struct rcu_head *rcu) static void bpf_tramp_image_put(struct bpf_tramp_image *im) { + int i, err; + /* The trampoline image that calls original function is using: * rcu_read_lock_trace to protect sleepable bpf progs * rcu_read_lock to protect normal bpf progs @@ -621,10 +624,17 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im) * * The trampoline is unreachable before bpf_tramp_image_put(). * - * First, patch the trampoline to avoid calling into fexit progs. - * The progs will be freed even if the original function is still - * executing or sleeping. - * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on + * First, patch the trampoline to avoid calling into progs. The progs + * will be freed even if a task is still in the trampoline, e.g. + * sleeping in the original function or in a sleepable prog. + */ + for (i = 0; i < im->nr_skips; i++) { + err = bpf_arch_text_poke(im->skips[i].nop, BPF_MOD_NOP, + BPF_MOD_JUMP, NULL, im->skips[i].target); + WARN_ON_ONCE(err); + } + + /* In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on * first few asm instructions to execute and call into * __bpf_tramp_enter->percpu_ref_get. * Then use percpu_ref_kill to wait for the trampoline and the original @@ -637,11 +647,7 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im) * percpu_ref_kill will be waiting for. Hence the first * call_rcu_tasks() is not necessary. */ - if (im->ip_after_call) { - int err = bpf_arch_text_poke(im->ip_after_call, BPF_MOD_NOP, - BPF_MOD_JUMP, NULL, - im->ip_epilogue); - WARN_ON(err); + if (im->call_orig) { if (IS_ENABLED(CONFIG_TASKS_RCU)) call_rcu_tasks(&im->rcu, __bpf_tramp_image_put_rcu_tasks); else @@ -658,7 +664,7 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im) call_rcu_tasks_trace(&im->rcu, __bpf_tramp_image_put_rcu_tasks); } -static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size) +static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size, int nr_progs) { struct bpf_tramp_image *im; struct bpf_ksym *ksym; @@ -669,6 +675,10 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size) if (!im) goto out; + im->skips = kzalloc_objs(*im->skips, nr_progs); + if (!im->skips) + goto out_free_im; + err = bpf_jit_charge_modmem(size); if (err) goto out_free_im; @@ -695,6 +705,7 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size) out_uncharge: bpf_jit_uncharge_modmem(size); out_free_im: + kfree(im->skips); kfree(im); out: return ERR_PTR(err); @@ -771,11 +782,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } - im = bpf_tramp_image_alloc(tr->key, size); + im = bpf_tramp_image_alloc(tr->key, size, total); if (IS_ERR(im)) { err = PTR_ERR(im); goto out; } + im->call_orig = tr->flags & BPF_TRAMP_F_CALL_ORIG; err = arch_prepare_bpf_trampoline(im, im->image, im->image + size, &tr->func.model, tr->flags, tnodes, -- 2.55.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed 2026-09-24 17:05 ` [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed Florent Revest (Anthropic) @ 2026-09-24 18:07 ` bot+bpf-ci 0 siblings, 0 replies; 6+ messages in thread From: bot+bpf-ci @ 2026-09-24 18:07 UTC (permalink / raw) To: florent.revest, bpf, ast, daniel, andrii Cc: florent.revest, martin.lau, eddyz87, memxor, song, yonghong.song, jolsa, kpsingh, john.fastabend, leon.hwang, zirajs7, rhkrqnwk98, puranjay, xukuohai, iii, hbathini, chleroy, naveen, bjorn, pulehui, yangtiezhu, hengqi.chen, linux-kernel, ast, andrii, daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai [-- Attachment #1: Type: text/plain, Size: 10213 bytes --] > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c > index 2853e87797a7..7ef5c8efc882 100644 > --- a/arch/x86/net/bpf_jit_comp.c > +++ b/arch/x86/net/bpf_jit_comp.c > @@ -3217,16 +3217,21 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, > } > > static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, > + struct bpf_tramp_image *im, > struct bpf_tramp_node *node, int stack_size, > int run_ctx_off, bool save_ret, > void *image, void *rw_image) > { > u8 *prog = *pprog; > - u8 *jmp_insn; > + u8 *jmp_insn, *skip; > int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); > struct bpf_prog *p = node->link->prog; > u64 cookie = node->cookie; > > + /* nop, patched to skip this prog when the image is put */ > + skip = image + (prog - (u8 *)rw_image); > + emit_nops(&prog, X86_PATCH_SIZE); > + The patch adds a 5-byte nop in front of every prog call sequence on x86 but keeps BPF_MAX_TRAMP_LINKS at 38. This adds 38 * 5 - 5 = 185 bytes to the worst case image size, which was already close to PAGE_SIZE. Example worst case: 19 fsession progs (38 link slots because each counts twice) attached to a function with 7-9 argument registers, each calling bpf_session_cookie() and bpf_get_func_ip() with a 64-bit link cookie. With 7 args the image reaches about 4040-4043 bytes, above PAGE_SIZE - BPF_INSN_SAFETY = 4032. With 10 args it reaches about 4106-4109 bytes. An attach that used to succeed now fails with -EFAULT and a WARN_ON_ONCE() splat. The size pass writes to a single PAGE_SIZE execmem buffer with only an end-of-pass WARN_ON_ONCE check: arch/x86/net/bpf_jit_comp.c:arch_bpf_trampoline_size() { image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE); ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, ...); arch/x86/net/bpf_jit_comp.c:__arch_prepare_bpf_trampoline() { if (WARN_ON_ONCE(prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) { ret = -EFAULT; goto cleanup; } With 10 args the size pass writes past the end of the execmem buffer before the check is reached. The commit message says the extra nops break the page budget only on arm64 and loongarch. Should x86 get a lower BPF_MAX_TRAMP_LINKS too, or should the x86 size pass bound-check as it emits? [ ... ] > @@ -3368,7 +3377,7 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, > emit_mov_imm32(&prog, false, BPF_REG_0, 0); > emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); > for (i = 0; i < tl->nr_nodes; i++) { > - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, > + if (invoke_bpf_prog(m, &prog, im, tl->nodes[i], stack_size, run_ctx_off, true, > image, rw_image)) > return -EINVAL; > > /* mod_ret prog stored return value into [rbp - 8]. Emit: > * if (*(u64 *)(rbp - 8) != 0) > * goto do_fexit; > */ invoke_bpf_mod_ret() now passes im to invoke_bpf_prog(), so every fmod_ret prog gets a skip nop that bpf_tramp_image_put() patches to a jump. The skip target is before the cmp/jne check. A skipped fmod_ret prog therefore leaves the previous return value in place (0 from the initial mov after xor eax,eax), and the original function is called. Before this change only fexit progs were skipped in a put image, and those only observe. fmod_ret progs always ran. Any trampoline update puts the old image, including attaching or detaching an unrelated prog. BPF LSM progs on int hooks become BPF_TRAMP_MODIFY_RETURN: kernel/bpf/verifier.c:bpf_attach_type_to_tramp() { case BPF_LSM_MAC: if (!prog->aux->attach_func_proto->type) return BPF_TRAMP_FEXIT; else return BPF_TRAMP_MODIFY_RETURN; Hooks such as bpf_lsm_file_open and bpf_lsm_bprm_check_security are in sleepable_lsm_hooks. A possible sequence: task T in image I of bpf_lsm_file_open, sleeping in sleepable LSM prog S (returns 0) attach another prog to the hook -> new image, bpf_tramp_image_put(I) -> nop before enforcing prog P patched to jmp T returns from S jmp over P (P is still attached) cmp [rbp-8], 0 -> equal, call bpf_lsm_file_open() -> 0 open allowed although P would have denied it On a preemptible kernel the window exists even without sleepable progs: anywhere between trampoline entry and P's nop. The commit message says still-attached progs are skipped "like they already skipped its fexit progs". For fmod_ret, though, skipping changes the traced function's return value instead of just losing an event. Is it intended that a concurrent attach or detach on the same hook can bypass an attached LSM or fmod_ret prog? > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index d4f732996b47..9c61803380c2 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1258,11 +1258,15 @@ struct btf_func_model { > #define BPF_TRAMP_F_INDIRECT BIT(8) > > /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 > - * bytes on x86. > + * bytes on x86. The trampoline image has to fit in PAGE_SIZE. > */ > enum { > -#if defined(__s390x__) > +#if defined(__s390x__) || defined(__powerpc64__) > BPF_MAX_TRAMP_LINKS = 27, > +#elif defined(__loongarch__) > + BPF_MAX_TRAMP_LINKS = 33, > +#elif defined(__aarch64__) > + BPF_MAX_TRAMP_LINKS = 37, > #else > BPF_MAX_TRAMP_LINKS = 38, > #endif BPF_MAX_TRAMP_LINKS drops from 38 to 27 for every __powerpc64__ build (and to 37 for every __aarch64__ build), whatever the page size. The updated comment says the constraint is that the image has to fit in PAGE_SIZE, and bpf_trampoline_update() does check the size against PAGE_SIZE. PPC_64K_PAGES is the default for PPC_BOOK3S_64, and 38 progs fit easily in a 64K image. On common ppc64le configurations (and arm64 with 16K/64K pages), attaching the 28th to 38th prog to one function now fails with -E2BIG where it used to work. The commit message's "already didn't on powerpc" only holds for 4K pages. Could the limit depend on PAGE_SIZE instead, keeping 38 when the page is larger than 4K? > diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c > index 9d69c066a817..0d7aa1b0fe30 100644 > --- a/kernel/bpf/trampoline.c > +++ b/kernel/bpf/trampoline.c [ ... ] > @@ -611,6 +612,8 @@ static void __bpf_tramp_image_put_rcu_tasks(struct rcu_head *rcu) > > static void bpf_tramp_image_put(struct bpf_tramp_image *im) > { > + int i, err; > + > /* The trampoline image that calls original function is using: > * rcu_read_lock_trace to protect sleepable bpf progs > * rcu_read_lock to protect normal bpf progs > @@ -621,10 +624,17 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im) > * > * The trampoline is unreachable before bpf_tramp_image_put(). > * > - * First, patch the trampoline to avoid calling into fexit progs. > - * The progs will be freed even if the original function is still > - * executing or sleeping. > - * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on > + * First, patch the trampoline to avoid calling into progs. The progs > + * will be freed even if a task is still in the trampoline, e.g. > + * sleeping in the original function or in a sleepable prog. > + */ > + for (i = 0; i < im->nr_skips; i++) { > + err = bpf_arch_text_poke(im->skips[i].nop, BPF_MOD_NOP, > + BPF_MOD_JUMP, NULL, im->skips[i].target); > + WARN_ON_ONCE(err); > + } Now that every prog nop in the old image is patched, a task still in that image also skips the fmod_ret progs that are still attached, not only fentry/fexit progs. Before this change only ip_after_call was patched. It sits after the original function call, so fentry and fmod_ret progs always ran and only fexit progs could be skipped. The commit message justifies skipping attached progs "like they already skipped its fexit progs". That comparison holds for observe-only fentry/fexit progs, but fmod_ret progs decide the return value and whether the original function runs. BPF LSM progs on int-returning hooks are attached as BPF_TRAMP_MODIFY_RETURN. Scenario: 1. Task T enters image I_old for bpf_lsm_file_open. It is either preempted or sleeping in an earlier sleepable lsm.s prog, e.g. blocked in bpf_copy_from_user(). Either way it has not yet reached the nop of enforcing LSM prog L. 2. Someone attaches or detaches any other prog on the same trampoline. bpf_trampoline_update() -> modify_fentry() -> bpf_tramp_image_put(I_old) turns all nops of I_old into jumps, including L's. 3. T resumes and jumps over L's call sequence. In x86 invoke_bpf_mod_ret() the result slot [rbp-8] was zeroed and nothing stores to it, so the following cmp/jne does not branch to do_fexit. The original bpf_lsm_file_open() stub is called and returns 0. The hook fails open for T even though L is still attached and would have denied access. The sequential pokes can also leave T running some fmod_ret progs but not others. The sleepable case makes the window as long as the earlier prog blocks. Should fmod_ret progs that are still attached keep running, with only the progs being removed skipped? Otherwise, is this fail-open behaviour for BPF LSM and fmod_ret acceptable, and should the commit message and comment say so? > + > + /* In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on > * first few asm instructions to execute and call into > * __bpf_tramp_enter->percpu_ref_get. > * Then use percpu_ref_kill to wait for the trampoline and the original This isn't a bug, but the multi-line comment has the opening slash-star on the same line as the comment text. BPF subsystem style in kernel/bpf/ has the opening slash-star on its own line for new multi-line comments: /* * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on * first few asm instructions to execute and call into --- AI reviewed your patch. Please fix the bug or email reply why it's not a bug. See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36034251810 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it 2026-09-24 17:05 [PATCH bpf v3 0/3] bpf: Fix use-after-free of progs detached from busy trampolines Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 1/3] bpf: Wait for an RCU tasks grace period before freeing trampoline progs Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed Florent Revest (Anthropic) @ 2026-09-24 17:05 ` Florent Revest (Anthropic) 2026-09-24 17:53 ` bot+bpf-ci 2 siblings, 1 reply; 6+ messages in thread From: Florent Revest (Anthropic) @ 2026-09-24 17:05 UTC (permalink / raw) To: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko Cc: Florent Revest (Anthropic), Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa, KP Singh, John Fastabend, Leon Hwang, Junseo Lim, Sechang Lim, Puranjay Mohan, Xu Kuohai, Ilya Leoshkevich, Hari Bathini, Christophe Leroy, Naveen N Rao, Björn Töpel, Pu Lehui, Tiezhu Yang, Hengqi Chen, linux-kernel Add a test for the use-after-free fixed by the previous commit. A sleepable prog on bpf_fentry_test1() blocks a task on a userfaultfd page, like bpf_mod_race does, while the prog that runs after it in the same trampoline is detached and freed. The task is then released and must not call into the freed prog. This is done with fentry progs, with fexit progs, where the task is already past the original function, and with a task sleeping in a fentry prog while a fexit prog is detached, where the original function must still be called. bpf_mod_race's userfaultfd helper moves to testing_helpers.c so that both tests can use it. Without the fix, on a kernel with KASAN: BUG: KASAN: vmalloc-out-of-bounds in __bpf_prog_enter_recur+0xed/0x1e0 Read of size 8 at addr ffa0000000144040 by task test_progs/171 CPU: 6 UID: 0 PID: 171 Comm: test_progs Tainted: G OE 7.2.0+ #1 PREEMPT(full) Call Trace: <TASK> __bpf_prog_enter_recur+0xed/0x1e0 bpf_trampoline_6442545468+0x72/0xe3 bpf_fentry_test1+0x9/0x20 bpf_prog_test_run_tracing+0x183/0x3e0 __sys_bpf+0xd3f/0x38f0 ... Assisted-by: Claude:unspecified Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev> --- .../selftests/bpf/prog_tests/bpf_mod_race.c | 34 +--- .../bpf/prog_tests/tramp_prog_detach.c | 158 ++++++++++++++++++ .../selftests/bpf/progs/tramp_prog_detach.c | 56 +++++++ tools/testing/selftests/bpf/testing_helpers.c | 28 ++++ tools/testing/selftests/bpf/testing_helpers.h | 2 + 5 files changed, 245 insertions(+), 33 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c create mode 100644 tools/testing/selftests/bpf/progs/tramp_prog_detach.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c b/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c index ecc3d47919ad..f8497e764beb 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c @@ -55,38 +55,6 @@ static void *load_module_thread(void *p) return p; } -static int sys_userfaultfd(int flags) -{ - return syscall(__NR_userfaultfd, flags); -} - -static int test_setup_uffd(void *fault_addr) -{ - struct uffdio_register uffd_register = {}; - struct uffdio_api uffd_api = {}; - int uffd; - - uffd = sys_userfaultfd(O_CLOEXEC); - if (uffd < 0) - return -errno; - - uffd_api.api = UFFD_API; - uffd_api.features = 0; - if (ioctl(uffd, UFFDIO_API, &uffd_api)) { - close(uffd); - return -1; - } - - uffd_register.range.start = (unsigned long)fault_addr; - uffd_register.range.len = getpagesize(); - uffd_register.mode = UFFDIO_REGISTER_MODE_MISSING; - if (ioctl(uffd, UFFDIO_REGISTER, &uffd_register)) { - close(uffd); - return -1; - } - return uffd; -} - static void test_bpf_mod_race_config(const struct test_config *config) { void *fault_addr, *skel_fail; @@ -117,7 +85,7 @@ static void test_bpf_mod_race_config(const struct test_config *config) if (!ASSERT_OK(bpf_mod_race__attach(skel), "bpf_mod_kfunc_race__attach")) goto end_destroy; - uffd = test_setup_uffd(fault_addr); + uffd = uffd_block_page(fault_addr); if (!ASSERT_GE(uffd, 0, "userfaultfd open + register address")) goto end_destroy; diff --git a/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c b/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c new file mode 100644 index 000000000000..002eb4920c91 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include <pthread.h> +#include <poll.h> +#include <sys/mman.h> +#include <linux/userfaultfd.h> +#include "tramp_prog_detach.skel.h" +#include "testing_helpers.h" + +/* + * Detach and free a prog while a task sleeps in the prog that runs before it + * in the same trampoline image, then let that task continue through the + * image. It must not call into the freed prog. + * + * The task is held in a sleepable prog with userfaultfd, like bpf_mod_race + * does. + */ + +static struct bpf_program *pick_prog(struct tramp_prog_detach *skel, + bool fexit, bool sleepable) +{ + if (fexit) + return sleepable ? skel->progs.fexit_sleepable : + skel->progs.fexit_victim; + return sleepable ? skel->progs.fentry_sleepable : + skel->progs.fentry_victim; +} + +static struct bpf_program *sleepable_prog; + +static void *run_sleepable(void *arg) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + + /* calls bpf_fentry_test1() */ + return (void *)(long)bpf_prog_test_run_opts(bpf_program__fd(sleepable_prog), + &topts); +} + +static struct tramp_prog_detach *load_one(bool fexit, bool sleepable) +{ + struct tramp_prog_detach *skel; + int err; + + skel = tramp_prog_detach__open(); + if (!ASSERT_OK_PTR(skel, "open")) + return NULL; + + bpf_program__set_autoload(pick_prog(skel, fexit, sleepable), true); + err = tramp_prog_detach__load(skel); + if (!ASSERT_OK(err, "load")) + goto err; + skel->bss->pid = getpid(); + err = tramp_prog_detach__attach(skel); + if (!ASSERT_OK(err, "attach")) + goto err; + return skel; +err: + tramp_prog_detach__destroy(skel); + return NULL; +} + +static void test_detach(bool sleepable_fexit, bool victim_fexit) +{ + struct tramp_prog_detach *sleepable = NULL, *victim = NULL; + struct pollfd pfd = { .events = POLLIN }; + struct uffdio_copy uffd_copy = {}; + struct uffd_msg uffd_msg; + void *fault_page, *src_page = MAP_FAILED; + long page_size = getpagesize(); + bool started = false; + void *thread_ret; + pthread_t thread; + int uffd = -1; + + fault_page = mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (!ASSERT_NEQ(fault_page, MAP_FAILED, "mmap fault_page")) + return; + src_page = mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (!ASSERT_NEQ(src_page, MAP_FAILED, "mmap src_page")) + goto out; + + /* The most recently attached prog runs first */ + victim = load_one(victim_fexit, false); + if (!victim) + goto out; + sleepable = load_one(sleepable_fexit, true); + if (!sleepable) + goto out; + sleepable_prog = pick_prog(sleepable, sleepable_fexit, true); + + /* Not armed yet so this doesn't block, make sure sleepable runs first */ + if (!ASSERT_OK((long)run_sleepable(NULL), "dry run")) + goto out; + if (!ASSERT_LT(sleepable->bss->ts, victim->bss->ts, "prog order")) + goto out; + + uffd = uffd_block_page(fault_page); + if (!ASSERT_GE(uffd, 0, "userfaultfd open + register address")) + goto out; + sleepable->bss->fault_addr = fault_page; + + if (!ASSERT_OK(pthread_create(&thread, NULL, run_sleepable, NULL), + "pthread_create")) + goto out; + started = true; + + /* Wait for the thread to sleep in bpf_copy_from_user() */ + pfd.fd = uffd; + if (!ASSERT_EQ(poll(&pfd, 1, 10000), 1, "poll uffd")) + goto out; + if (!ASSERT_EQ(read(uffd, &uffd_msg, sizeof(uffd_msg)), sizeof(uffd_msg), + "read uffd")) + goto out; + if (!ASSERT_EQ(uffd_msg.event, UFFD_EVENT_PAGEFAULT, "uffd pagefault")) + goto out; + + /* Detach and unload the victim prog, and make sure it is gone */ + tramp_prog_detach__destroy(victim); + victim = NULL; + kern_sync_rcu(); + usleep(100 * 1000); + kern_sync_rcu(); + +out: + /* Let the thread proceed with the rest of the trampoline */ + if (uffd >= 0) { + uffd_copy.dst = (unsigned long)fault_page; + uffd_copy.src = (unsigned long)src_page; + uffd_copy.len = page_size; + ASSERT_OK(ioctl(uffd, UFFDIO_COPY, &uffd_copy), "uffd copy"); + close(uffd); + } + if (started && + ASSERT_OK(pthread_join(thread, &thread_ret), "pthread_join")) + ASSERT_NULL(thread_ret, "blocking run"); + + tramp_prog_detach__destroy(victim); + tramp_prog_detach__destroy(sleepable); + if (src_page != MAP_FAILED) + munmap(src_page, page_size); + munmap(fault_page, page_size); +} + +void serial_test_tramp_prog_detach(void) +{ + /* a task sleeping before the original function is called */ + if (test__start_subtest("fentry")) + test_detach(false, false); + /* a task sleeping after the original function returned */ + if (test__start_subtest("fexit")) + test_detach(true, true); + /* the original function runs in between and must still be called */ + if (test__start_subtest("fentry_fexit")) + test_detach(false, true); +} diff --git a/tools/testing/selftests/bpf/progs/tramp_prog_detach.c b/tools/testing/selftests/bpf/progs/tramp_prog_detach.c new file mode 100644 index 000000000000..507d372167ec --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tramp_prog_detach.c @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +int pid; +void *fault_addr; +__u64 ts; + +static int do_sleepable(void) +{ + char dst; + + if (bpf_get_current_pid_tgid() >> 32 != pid) + return 0; + + ts = bpf_ktime_get_ns(); + /* blocks for as long as user space wants when fault_addr is armed */ + bpf_copy_from_user(&dst, sizeof(dst), fault_addr); + return 0; +} + +static int do_victim(void) +{ + if (bpf_get_current_pid_tgid() >> 32 != pid) + return 0; + + ts = bpf_ktime_get_ns(); + return 0; +} + +SEC("?fentry.s/bpf_fentry_test1") +int BPF_PROG(fentry_sleepable, int a) +{ + return do_sleepable(); +} + +SEC("?fentry/bpf_fentry_test1") +int BPF_PROG(fentry_victim, int a) +{ + return do_victim(); +} + +SEC("?fexit.s/bpf_fentry_test1") +int BPF_PROG(fexit_sleepable, int a, int ret) +{ + return do_sleepable(); +} + +SEC("?fexit/bpf_fentry_test1") +int BPF_PROG(fexit_victim, int a, int ret) +{ + return do_victim(); +} diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c index c970e7793dfc..9f672d0c9dda 100644 --- a/tools/testing/selftests/bpf/testing_helpers.c +++ b/tools/testing/selftests/bpf/testing_helpers.c @@ -13,6 +13,7 @@ #include "test_progs.h" #include "testing_helpers.h" #include <linux/membarrier.h> +#include <linux/userfaultfd.h> int parse_num_list(const char *s, bool **num_set, int *num_set_len) { @@ -459,6 +460,33 @@ int kern_sync_rcu(void) return syscall(__NR_membarrier, MEMBARRIER_CMD_SHARED, 0, 0); } +int uffd_block_page(void *fault_addr) +{ + struct uffdio_register uffd_register = {}; + struct uffdio_api uffd_api = {}; + int uffd; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC); + if (uffd < 0) + return -errno; + + uffd_api.api = UFFD_API; + uffd_api.features = 0; + if (ioctl(uffd, UFFDIO_API, &uffd_api)) { + close(uffd); + return -1; + } + + uffd_register.range.start = (unsigned long)fault_addr; + uffd_register.range.len = getpagesize(); + uffd_register.mode = UFFDIO_REGISTER_MODE_MISSING; + if (ioctl(uffd, UFFDIO_REGISTER, &uffd_register)) { + close(uffd); + return -1; + } + return uffd; +} + int get_xlated_program(int fd_prog, struct bpf_insn **buf, __u32 *cnt) { __u32 buf_element_size = sizeof(struct bpf_insn); diff --git a/tools/testing/selftests/bpf/testing_helpers.h b/tools/testing/selftests/bpf/testing_helpers.h index 2edc6fb7fc52..1f83b6080716 100644 --- a/tools/testing/selftests/bpf/testing_helpers.h +++ b/tools/testing/selftests/bpf/testing_helpers.h @@ -36,6 +36,8 @@ __u64 read_perf_max_sample_freq(void); int load_bpf_testmod(bool verbose); int unload_bpf_testmod(bool verbose); int kern_sync_rcu(void); +/* returns a userfaultfd that makes accesses to fault_addr's page block */ +int uffd_block_page(void *fault_addr); int finit_module(int fd, const char *param_values, int flags); int delete_module(const char *name, int flags); int load_module(const char *path, bool verbose); -- 2.55.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it 2026-09-24 17:05 ` [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it Florent Revest (Anthropic) @ 2026-09-24 17:53 ` bot+bpf-ci 0 siblings, 0 replies; 6+ messages in thread From: bot+bpf-ci @ 2026-09-24 17:53 UTC (permalink / raw) To: florent.revest, bpf, ast, daniel, andrii Cc: florent.revest, martin.lau, eddyz87, memxor, song, yonghong.song, jolsa, kpsingh, john.fastabend, leon.hwang, zirajs7, rhkrqnwk98, puranjay, xukuohai, iii, hbathini, chleroy, naveen, bjorn, pulehui, yangtiezhu, hengqi.chen, linux-kernel, ast, andrii, daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai [-- Attachment #1: Type: text/plain, Size: 5064 bytes --] > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c b/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c > index ecc3d47919ad..f8497e764beb 100644 > --- a/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_mod_race.c [ ... ] > diff --git a/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c b/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c > new file mode 100644 > index 000000000000..002eb4920c91 > --- /dev/null > +++ b/tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c [ ... ] > +static void test_detach(bool sleepable_fexit, bool victim_fexit) > +{ > + struct tramp_prog_detach *sleepable = NULL, *victim = NULL; > + struct pollfd pfd = { .events = POLLIN }; > + struct uffdio_copy uffd_copy = {}; > + struct uffd_msg uffd_msg; > + void *fault_page, *src_page = MAP_FAILED; > + long page_size = getpagesize(); > + bool started = false; > + void *thread_ret; > + pthread_t thread; > + int uffd = -1; [ ... ] > + sleepable_prog = pick_prog(sleepable, sleepable_fexit, true); > + > + /* Not armed yet so this doesn't block, make sure sleepable runs first */ > + if (!ASSERT_OK((long)run_sleepable(NULL), "dry run")) > + goto out; > + if (!ASSERT_LT(sleepable->bss->ts, victim->bss->ts, "prog order")) > + goto out; > + > + uffd = uffd_block_page(fault_page); > + if (!ASSERT_GE(uffd, 0, "userfaultfd open + register address")) > + goto out; A subsystem pattern flags this as potentially concerning: uffd_block_page() returns -errno when syscall(__NR_userfaultfd, O_CLOEXEC) fails. On a kernel without CONFIG_USERFAULTFD that is -ENOSYS, and the test then fails through ASSERT_GE() instead of calling test__skip() with a message saying userfaultfd is not available. While tools/testing/selftests/bpf/config sets CONFIG_USERFAULTFD=y so the BPF CI kernel always has it, and the same ASSERT_GE pattern already exists in bpf_mod_race.c (serial_test_bpf_mod_race) from before this commit, is this the intended behavior for environments without userfaultfd support? > + sleepable->bss->fault_addr = fault_page; > + > + if (!ASSERT_OK(pthread_create(&thread, NULL, run_sleepable, NULL), > + "pthread_create")) > + goto out; > + started = true; > + > + /* Wait for the thread to sleep in bpf_copy_from_user() */ > + pfd.fd = uffd; > + if (!ASSERT_EQ(poll(&pfd, 1, 10000), 1, "poll uffd")) > + goto out; > + if (!ASSERT_EQ(read(uffd, &uffd_msg, sizeof(uffd_msg)), sizeof(uffd_msg), > + "read uffd")) > + goto out; > + if (!ASSERT_EQ(uffd_msg.event, UFFD_EVENT_PAGEFAULT, "uffd pagefault")) > + goto out; > + > + /* Detach and unload the victim prog, and make sure it is gone */ > + tramp_prog_detach__destroy(victim); > + victim = NULL; > + kern_sync_rcu(); > + usleep(100 * 1000); > + kern_sync_rcu(); Does this wait actually ensure the victim prog is gone before the task resumes? The test only catches the use-after-free if the victim prog has actually been freed before UFFDIO_COPY releases the sleeping task. The wait here does not guarantee that, and on the kernel this series produces it most likely is not freed yet. Commit 1c2316cf73ba ("bpf: Wait for an RCU tasks grace period before freeing trampoline progs") changed __bpf_prog_put_noref() so that a tramp_linked prog is freed through call_rcu_tasks(&prog->aux->rcu, __bpf_prog_put_rcu_tasks), then call_rcu(), then bpf_prog_free(), which calls schedule_work() for the final vfree. RCU Tasks callbacks are lazy. call_rcu_tasks_generic() only sets needwake for wakeme_after_rcu (a synchronous waiter) or when the queue reaches rcu_task_lazy_lim. Otherwise it arms lazy_timer with rcu_tasks.lazy_jiffies = DIV_ROUND_UP(HZ, 4) (kernel/rcu/tasks.h:145), which is about 250ms by default. Nothing on the detach path forces a synchronous RCU Tasks wait. Detaching the victim while the sleepable prog stays attached goes through modify_ftrace_direct(), and its tmp_ops is static, so it is not FTRACE_OPS_FL_DYNAMIC and ftrace_shutdown() does not call synchronize_rcu_tasks(). The test waits only kern_sync_rcu() + 100ms + kern_sync_rcu(). kern_sync_rcu() is membarrier(MEMBARRIER_CMD_SHARED), which is synchronize_rcu() on SMP and a no-op with one CPU online. That is well short of 250ms lazy delay + RCU Tasks GP + RCU GP + workqueue. So when the thread resumes in the old image, the victim's bpf_prog and JIT image are usually still allocated. On a kernel that has 1c2316cf73ba but lacks 2ece3357872c (or breaks it later), the thread would run the victim prog without a KASAN report and the test would pass. Consider waiting well beyond the RCU Tasks lazy period (for example about 1s, or polling until the victim prog ID is no longer returned by bpf_prog_get_next_id()/bpf_prog_get_fd_by_id()) before releasing the thread. [ ... ] --- AI reviewed your patch. Please fix the bug or email reply why it's not a bug. See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36034251810 ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-24 18:07 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-09-24 17:05 [PATCH bpf v3 0/3] bpf: Fix use-after-free of progs detached from busy trampolines Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 1/3] bpf: Wait for an RCU tasks grace period before freeing trampoline progs Florent Revest (Anthropic) 2026-09-24 17:05 ` [PATCH bpf v3 2/3] bpf: Skip the progs of trampoline images that are being freed Florent Revest (Anthropic) 2026-09-24 18:07 ` bot+bpf-ci 2026-09-24 17:05 ` [PATCH bpf v3 3/3] selftests/bpf: Detach a trampoline prog while a task sleeps before it Florent Revest (Anthropic) 2026-09-24 17:53 ` bot+bpf-ci
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®