[PATCH bpf-next v2 0/5] bpf: Check recursion per task for preemptible programs

[PATCH bpf-next v2 0/5] bpf: Check recursion per task for preemptible programs

[Mike Bazov via B4 Relay]

Tracing programs run preemptibly: trampoline programs only disable
migration, tracepoint probes run under SRCU-fast, and syscall tracepoint
and sleepable programs can fault. Their recursion check is a per-CPU
counter:

  CPU N
  -----
  task A enters prog -> prog->active on CPU N becomes 1
  task A is preempted inside prog
  task B enters prog on CPU N -> prog->active is already raised:
    the run is dropped and counted as a recursion miss
  task A resumes and finishes -> prog->active drops to 0

Nothing recursed, but task B's event is lost. Before 7.0 this was mostly
hidden, because distribution kernels used PREEMPT_VOLUNTARY. Commit
7dadeaa6e851 ("sched: Further restrict the preemption modes") left only
lazy and full preemption on x86 and arm64, so 7.0+ distribution kernels
drop such events in normal operation. Programs can't work around it: the
counter is checked before their first instruction.

In a production security agent with paired fentry/fexit programs on
inet_sendmsg() and inet_recvmsg(), a dropped half breaks the entry/exit
pairing and desynchronizes stateful protocol parsing. On a stock 7.0
kernel, 4,063 of 38.0M runs were dropped under 10 minutes of HTTP load.
v1 has the detailed analysis and a standalone reproducer.

This series checks recursion per task instead. Each run already links a
run context into the task's chain (current->bpf_ctx); the run context
now records the program, and the check walks the task's chain. Real
recursion, including through interrupts and NMIs, which run on top of
the interrupted task's chain, is still rejected; a second task is not.
Programs that keep a private stack keep the per-CPU check.

Patch 1 comes first and stands on its own. A per-CPU private stack is
only safe if a program never runs in two tasks at once on one CPU, and
that already happens on bpf-next today: two raw_tp dispatchers
tail-calling one handler, or one tracepoint program attached to a
syscall and a regular tracepoint, let a second task overwrite the
preempted task's stack. The selftests reproduce both. Patch 1 stops
giving tracing, raw_tp and tracepoint programs a private stack. Without
it, the per-task check would let a dispatcher in while its tail-call
target still uses a per-CPU stack.

  1. bpf: Avoid per-CPU private stack in preemptible tracing progs
  2. bpf: Check program recursion per task via bpf_ctx
  3. bpf: Check recursion per task for programs on syscall tracepoints
  4. selftests/bpf: Add trampoline and raw_tp preemption recursion test
  5. selftests/bpf: Test syscall tracepoint programs under preemption

Behavior changes worth a look:

- On the kernel stack, the frames of a call chain share one stack
  budget, while a private stack checks each frame on its own. Programs
  whose frames only fit separately no longer load, e.g. a raw_tp program
  with two 2000-byte frames: "combined stack size of 2 calls is 4000.
  Too large". Is that acceptable, or would an opt-in private stack, which
  would keep the per-CPU check, be preferred as a follow-up?
- Raw tracepoint programs were the only ones with a private stack that
  BPF_PROG_TEST_RUN runs, so unwinding on a private stack is now checked
  in the generated code only. A run-time test could use bpf_testmod's
  struct_ops that requests a private stack: wanted in this series or as
  a follow-up?
- The same program can now run in two tasks at once on one CPU, so a
  program using a per-CPU map as scratch space can see another run's
  data.
- Timed may_goto keeps its budget per program and CPU, as before: a run
  in another task can renew the budget of a preempted run.

Testing:

x86-64 VM with 4 vCPUs, full preemption, bpf-next 0fb18ce8a2e9 with and
without the series.

- The same test_progs binary on both kernels:

                                            bpf-next    series
  trampoline_preempt, 14 preemption         all fail    all pass
    subtests (20 rounds each, the
    overlap is checked in every round)
  trampoline_preempt, 4 recursion           all pass    all pass
    subtests
  verifier_unwind, may_goto_priv_stack,     pass        pass
    timed_may_goto, test_global_funcs,
    recursion, task_local_storage

  On bpf-next the second task is dropped in every round of every
  preemption subtest, and two_dispatchers and tp_dual also corrupt the
  preempted task's stack.

- HTTP load: nginx -> gunicorn -> Redis on the VM, the production
  fentry/fexit probe (8 programs on inet[6]_{send,recv}msg()), 4
  CPU-bound tasks, and 200,000 requests from wrk on each kernel:

                probe runs    recursion misses
  bpf-next       7,966,760               9,254
  series         7,976,662                   0

Changes in v2:
- v1 kept recursion-checked trampoline programs non-preemptible. As
  discussed on v1, v2 checks recursion per task instead, using the
  bpf_ctx chain, so programs stay preemptible.
- Cover raw_tp/tp_btf, sleepable trampoline programs, BPF_PROG_TEST_RUN
  and syscall tracepoints, not only trampolines.
- New patch 1: no per-CPU private stack for these program types, which
  also fixes two existing stack corruptions.
- New deterministic selftests: every round forces a second task into the
  program while the first is preempted inside it, proves the overlap,
  and checks the preempted task's stack; plus true-recursion tests.

v1: https://lore.kernel.org/bpf/20261025-b4-bpf-trampoline-preempt-v1-0-198dcdc0c6ac@sentinelone.com/

---
Mike Bazov (5):
      bpf: Avoid per-CPU private stack in preemptible tracing progs
      bpf: Check program recursion per task via bpf_ctx
      bpf: Check recursion per task for programs on syscall tracepoints
      selftests/bpf: Add trampoline and raw_tp preemption recursion test
      selftests/bpf: Test syscall tracepoint programs under preemption

 include/linux/bpf.h                                |  54 ++-
 kernel/bpf/trampoline.c                            |  28 +-
 kernel/bpf/verifier.c                              |  43 +-
 kernel/trace/bpf_trace.c                           |  12 +-
 net/bpf/test_run.c                                 |   4 +-
 net/ipv4/bpf_tcp_ops.c                             |   6 +-
 .../selftests/bpf/prog_tests/may_goto_priv_stack.c |   6 +-
 .../selftests/bpf/prog_tests/timed_may_goto.c      |   6 +-
 .../selftests/bpf/prog_tests/trampoline_preempt.c  | 464 +++++++++++++++++++++
 .../selftests/bpf/progs/may_goto_priv_stack.c      |  14 +-
 .../selftests/bpf/progs/test_global_func3.c        |   9 +
 .../selftests/bpf/progs/timed_may_goto_user.c      |  16 +-
 .../selftests/bpf/progs/trampoline_preempt.c       | 302 ++++++++++++++
 .../selftests/bpf/progs/verifier_private_stack.c   |  65 +--
 .../testing/selftests/bpf/progs/verifier_unwind.c  |  33 +-
 15 files changed, 975 insertions(+), 87 deletions(-)
---
base-commit: 0fb18ce8a2e9de830433923cf65838933d615359
change-id: 20261111-bpf-preempt-v2-r2-f9f9a0dbd5d4

Best regards,
-- 
Mike Bazov <mikeba@sentinelone.com>

[PATCH bpf-next v2 1/5] bpf: Avoid per-CPU private stack in preemptible tracing progs

[Mike Bazov via B4 Relay]

From: Mike Bazov <mikeba@sentinelone.com>

A per-CPU private stack is only safe if a program never runs in two
tasks at once on one CPU. Tracing, raw_tp and tracepoint programs run
preemptibly, and that already happens today:

- Two raw_tp dispatchers that tail-call the same handler each pass their
  own per-CPU recursion check, so two tasks can run the handler at once
  on one CPU and share its private stack.

  event 1 -> dispatcher A --+
                            +--> jump_table[0] --> handler H
  event 2 -> dispatcher B --+

- A tracepoint program attached to a syscall tracepoint, which runs
  preemptibly since commit 57918341dd19 ("bpf: Add sleepable support for
  classic tracepoint programs"), and to a regular tracepoint can be
  entered by a second task through the regular tracepoint, which only
  checks bpf_prog_active.

In both cases the second task overwrites the preempted task's stack.
Never give these program types a private stack. Checking their recursion
per task, as the next patches do, makes such overlap the normal case.

check_recursion() told tracing programs apart by whether they can get
a private stack, so it would start accepting recursive subprograms in
these program types. Make it check the program type instead, and test
it for raw_tp.

On the kernel stack, the frames of a call chain share one stack budget,
while a private stack checks each frame on its own. A program whose
main function and callee each use close to the per-frame limit no
longer loads: two 2000-byte frames are rejected with "combined stack
size of 2 calls is 4000. Too large".

Update the selftests that relied on these program types having a
private stack to use kprobe.multi and perf_event programs. Raw
tracepoint programs were the only ones with a private stack that
BPF_PROG_TEST_RUN actually runs, so unwinding on a private stack is now
checked in the generated code only, not at run time.

Fixes: a76ab5731e32 ("bpf: Find eligible subprogs for private stack support")
Fixes: 57918341dd19 ("bpf: Add sleepable support for classic tracepoint programs")
Signed-off-by: Mike Bazov <mikeba@sentinelone.com>
---
 kernel/bpf/verifier.c                              | 43 +++++++++-----
 .../selftests/bpf/prog_tests/may_goto_priv_stack.c |  6 +-
 .../selftests/bpf/prog_tests/timed_may_goto.c      |  6 +-
 .../selftests/bpf/progs/may_goto_priv_stack.c      | 14 ++++-
 .../selftests/bpf/progs/test_global_func3.c        |  9 +++
 .../selftests/bpf/progs/timed_may_goto_user.c      | 16 +++++-
 .../selftests/bpf/progs/verifier_private_stack.c   | 65 ++++++++++++----------
 .../testing/selftests/bpf/progs/verifier_unwind.c  | 33 +++++++++--
 8 files changed, 139 insertions(+), 53 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 20b1b9ff574c..fe59a0e27380 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5470,9 +5470,16 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)
 	if (!bpf_jit_supports_private_stack())
 		return NO_PRIV_STACK;
 
-	/* bpf_prog_check_recur() checks all prog types that use bpf trampoline
-	 * while kprobe/tp/perf_event/raw_tp don't use trampoline hence checked
-	 * explicitly.
+	/*
+	 * A private stack is per CPU, so a program using it must never run in two
+	 * tasks at once on one CPU. Tracing, raw_tp and tracepoint programs run
+	 * preemptibly, and the same program can be entered by another task on the
+	 * same CPU, e.g. through a second tail-call dispatcher, or as a tracepoint
+	 * program attached to both a syscall and a regular tracepoint. They always
+	 * use the task's kernel stack. Kprobe multi and session programs are
+	 * serialized per CPU by bpf_prog_active, perf_event programs run in
+	 * interrupt context, and struct_ops programs that request a private stack
+	 * keep per-CPU recursion protection.
 	 */
 	switch (prog->type) {
 	case BPF_PROG_TYPE_KPROBE:
@@ -5485,14 +5492,10 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)
 		    prog->expected_attach_type != BPF_TRACE_KPROBE_SESSION)
 			return NO_PRIV_STACK;
 		fallthrough;
-	case BPF_PROG_TYPE_TRACEPOINT:
 	case BPF_PROG_TYPE_PERF_EVENT:
-	case BPF_PROG_TYPE_RAW_TRACEPOINT:
 		return PRIV_STACK_ADAPTIVE;
-	case BPF_PROG_TYPE_TRACING:
-	case BPF_PROG_TYPE_LSM:
 	case BPF_PROG_TYPE_STRUCT_OPS:
-		if (prog->aux->priv_stack_requested || bpf_prog_check_recur(prog))
+		if (prog->aux->priv_stack_requested)
 			return PRIV_STACK_ADAPTIVE;
 		fallthrough;
 	default:
@@ -5502,6 +5505,22 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)
 	return NO_PRIV_STACK;
 }
 
+/* Progs that run on top of other progs, whose frames are still on the stack */
+static bool bpf_prog_runs_nested(struct bpf_prog *prog)
+{
+	switch (prog->type) {
+	case BPF_PROG_TYPE_KPROBE:
+	case BPF_PROG_TYPE_PERF_EVENT:
+	case BPF_PROG_TYPE_TRACEPOINT:
+	case BPF_PROG_TYPE_RAW_TRACEPOINT:
+		return true;
+	case BPF_PROG_TYPE_TRACING:
+		return bpf_prog_check_recur(prog);
+	default:
+		return false;
+	}
+}
+
 static int round_up_stack_depth(struct bpf_verifier_env *env, int stack_depth)
 {
 	if (env->prog->jit_requested)
@@ -5684,12 +5703,10 @@ static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *
 		}
 		if (env->subprog_info[s].recursive) {
 			/*
-			 * These progs run on top of other progs. Their frames belong in the
-			 * private stack, which has room for one frame of a subprog. A kprobe prog
-			 * that can be attached to a uprobe has no private stack, and is no better.
+			 * These progs run on top of other progs. A private stack has room for one
+			 * frame of a subprog, and the kernel stack under them is already in use.
 			 */
-			if ((bpf_enable_priv_stack(env->prog) != NO_PRIV_STACK ||
-			     env->prog->type == BPF_PROG_TYPE_KPROBE) &&
+			if (bpf_prog_runs_nested(env->prog) &&
 			    !env->prog->aux->priv_stack_requested) {
 				verbose(env, "recursive %s() in a tracing program\n",
 					bpf_subprog_name(env, s));
diff --git a/tools/testing/selftests/bpf/prog_tests/may_goto_priv_stack.c b/tools/testing/selftests/bpf/prog_tests/may_goto_priv_stack.c
index 4f8476d36250..b4aab50b8a77 100644
--- a/tools/testing/selftests/bpf/prog_tests/may_goto_priv_stack.c
+++ b/tools/testing/selftests/bpf/prog_tests/may_goto_priv_stack.c
@@ -24,13 +24,17 @@ void test_may_goto_priv_stack(void)
 		return;
 
 	err = may_goto_priv_stack__attach(skel);
+	if (err == -EOPNOTSUPP) {
+		test__skip();
+		goto out;
+	}
 	if (!ASSERT_OK(err, "attach"))
 		goto out;
 
 	memset(skel->bss->regs_64, 0xff, sizeof(skel->bss->regs_64));
 	memset(skel->bss->regs_128, 0xff, sizeof(skel->bss->regs_128));
 
-	err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.priv_stack_64), &topts);
+	err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger), &topts);
 	if (!ASSERT_OK(err, "test_run"))
 		goto out;
 
diff --git a/tools/testing/selftests/bpf/prog_tests/timed_may_goto.c b/tools/testing/selftests/bpf/prog_tests/timed_may_goto.c
index 3da3f33a2898..ee7b79ef77e0 100644
--- a/tools/testing/selftests/bpf/prog_tests/timed_may_goto.c
+++ b/tools/testing/selftests/bpf/prog_tests/timed_may_goto.c
@@ -131,10 +131,14 @@ static void test_user_ringbuf(struct timed_may_goto_user *skel)
 static void test_priv_stack(struct timed_may_goto_user *skel)
 {
 	LIBBPF_OPTS(bpf_test_run_opts, opts);
-	int fd = bpf_program__fd(skel->progs.priv_stack);
+	int fd = bpf_program__fd(skel->progs.trigger);
 	struct bpf_link *link;
 
 	link = bpf_program__attach(skel->progs.priv_stack);
+	if (!link && errno == EOPNOTSUPP) {
+		test__skip();
+		return;
+	}
 	if (!ASSERT_OK_PTR(link, "attach"))
 		return;
 	ASSERT_OK(bpf_prog_test_run_opts(fd, &opts), "test_run");
diff --git a/tools/testing/selftests/bpf/progs/may_goto_priv_stack.c b/tools/testing/selftests/bpf/progs/may_goto_priv_stack.c
index 5c2aa0aa5055..e11a7f7493a6 100644
--- a/tools/testing/selftests/bpf/progs/may_goto_priv_stack.c
+++ b/tools/testing/selftests/bpf/progs/may_goto_priv_stack.c
@@ -10,11 +10,11 @@ __u64 regs_64[7];
 __u64 regs_128[7];
 
 /*
- * fentry prog with 64 or more bytes of stack uses private stack.
+ * kprobe.multi prog with 64 or more bytes of stack uses private stack.
  * Loop until may_goto expires. r0-r5 should stay zero.
  */
 #define TIMED_MAY_GOTO_PRIV_STACK(size, func)				\
-SEC("fentry/" #func)							\
+SEC("kprobe.multi/" #func)						\
 __naked void priv_stack_##size(void)					\
 {									\
 	asm volatile (							\
@@ -49,4 +49,14 @@ __naked void priv_stack_##size(void)					\
 TIMED_MAY_GOTO_PRIV_STACK(64, bpf_fentry_test1)
 TIMED_MAY_GOTO_PRIV_STACK(128, bpf_fentry_test2)
 
+/*
+ * No tests in here, just to trigger 'bpf_fentry_test*'
+ * through tracing test_run
+ */
+SEC("fentry/bpf_modify_return_test")
+int trigger(void *ctx)
+{
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c
index b916355c5e59..a473bed346d2 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func3.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func3.c
@@ -267,6 +267,15 @@ int global_func3_recursion_in_tracing_prog(void *ctx)
 	return depth(again + 7);
 }
 
+SEC("raw_tp")
+__arch_x86_64
+__arch_arm64
+__failure __msg("recursive depth() in a tracing program")
+int global_func3_recursion_in_raw_tp(void *ctx)
+{
+	return depth(again + 7);
+}
+
 __attribute__ ((noinline))
 int redirects(struct __sk_buff *skb, int n)
 {
diff --git a/tools/testing/selftests/bpf/progs/timed_may_goto_user.c b/tools/testing/selftests/bpf/progs/timed_may_goto_user.c
index 8b85815d45d2..ed56b6afb607 100644
--- a/tools/testing/selftests/bpf/progs/timed_may_goto_user.c
+++ b/tools/testing/selftests/bpf/progs/timed_may_goto_user.c
@@ -277,7 +277,7 @@ int drain(void *ctx)
 }
 
 /*
- * A tracing prog with large frames: they are in a private stack.
+ * A kprobe.multi prog with large frames: they are in a private stack.
  */
 int priv_stack_spun, priv_stack_cb_spun;
 
@@ -296,8 +296,8 @@ static int priv_stack_cb(__u64 i, int *spun)
 	return 0;
 }
 
-SEC("fentry/bpf_fentry_test1")
-int BPF_PROG(priv_stack)
+SEC("kprobe.multi/bpf_fentry_test1")
+int BPF_KPROBE(priv_stack)
 {
 	volatile char buf[128] = {};
 	int i, n = buf[zero & 127], spun = 0;
@@ -309,3 +309,13 @@ int BPF_PROG(priv_stack)
 	priv_stack_cb_spun = spun;
 	return 0;
 }
+
+/*
+ * No tests in here, just to trigger 'bpf_fentry_test*'
+ * through tracing test_run
+ */
+SEC("fentry/bpf_modify_return_test")
+int BPF_PROG(trigger)
+{
+	return 0;
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_private_stack.c b/tools/testing/selftests/bpf/progs/verifier_private_stack.c
index e7f410a74a21..bebdb0aff23f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_private_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_private_stack.c
@@ -127,6 +127,23 @@ __naked void no_private_stack_kprobe(void)
 "	::: __clobber_all);
 }
 
+SEC("fentry/bpf_fentry_test9")
+__description("No private stack, fentry")
+__success
+__arch_x86_64
+__jited("	subq	$0x100, %rsp")
+__arch_arm64
+__jited("	sub	sp, sp, #0x100")
+__naked void no_private_stack_fentry(void)
+{
+	asm volatile ("			\
+	r1 = 42;			\
+	*(u64 *)(r10 - 256) = r1;	\
+	r0 = 0;				\
+	exit;				\
+"	::: __clobber_all);
+}
+
 SEC("uprobe")
 __description("No private stack, uprobe")
 __success
@@ -215,7 +232,7 @@ static unsigned long loop_callback(void)
 	: __clobber_common);
 }
 
-SEC("raw_tp")
+SEC("kprobe.multi")
 __description("Private stack, callback")
 __success
 __arch_x86_64
@@ -273,9 +290,9 @@ __naked void private_stack_callback(void)
 	: __clobber_common);
 }
 
-SEC("fentry/bpf_fentry_test9")
+SEC("perf_event")
 __description("Private stack, exception in main prog")
-__success __retval(0)
+__success
 __arch_x86_64
 __jited("	pushq	%r9")
 __jited("...")
@@ -321,9 +338,9 @@ __used static int subprog_exception(void)
 	return 0;
 }
 
-SEC("fentry/bpf_fentry_test9")
+SEC("perf_event")
 __description("Private stack, exception in subprog")
-__success __retval(0)
+__success
 __arch_x86_64
 __jited("	movq	%rdi, -0x200(%r9)")
 __jited("	pushq	%r9")
@@ -355,19 +372,15 @@ int private_stack_exception_sub_prog(void)
 }
 
 /*
- * Nothing attaches the two fentry progs above, so they are loaded, but never
- * run. BPF_PROG_TEST_RUN does run a raw_tp prog. The default exception callback
- * returns the cookie of bpf_throw().
+ * Nothing attaches the two perf_event progs above, so they are loaded, but
+ * never run. BPF_PROG_TEST_RUN does run a raw_tp prog. The default exception
+ * callback returns the cookie of bpf_throw().
  */
 SEC("raw_tp")
-__description("Private stack, run, exception in main prog")
+__description("No private stack, run, exception in main prog")
 __success __retval(1)
 __arch_x86_64
-__jited("	movq	%rdi, -0x200(%r9)")
-__jited("	movl	$0x1, %edi")
-__jited("	pushq	%r9")
-__jited("...")
-__jited("	callq")
+__jited("	subq	$0x200, %rsp")
 __naked void private_stack_run_exception_main_prog(void)
 {
 	asm volatile ("			\
@@ -396,14 +409,10 @@ __naked static void subprog_throw(void)
 }
 
 SEC("raw_tp")
-__description("Private stack, run, exception in subprog")
+__description("No private stack, run, exception in subprog")
 __success __retval(2)
 __arch_x86_64
-__jited("	movq	%rdi, -0x200(%r9)")
-__jited("	pushq	%r9")
-__jited("...")
-__jited("	callq")
-__jited("	popq	%r9")
+__jited("	subq	$0x200, %rsp")
 __naked void private_stack_run_exception_sub_prog(void)
 {
 	asm volatile ("			\
@@ -432,16 +441,14 @@ __naked static void private_stack_subprog_throw(void)
 
 /* Only the frame of the main prog counts, and it has nothing pushed at the call */
 SEC("raw_tp")
-__description("Private stack, run, exception in subprog, no private stack in main prog")
+__description("No private stack, run, exception in subprog with a large stack")
 __success __retval(3)
 __arch_x86_64
 __jited("	subq	$0x8, %rsp")
 __jited("...")
 __jited("func #1")
 __jited("...")
-__jited("	movq	%rdi, -0x200(%r9)")
-__jited("	movl	$0x3, %edi")
-__jited("	pushq	%r9")
+__jited("	subq	$0x200, %rsp")
 __naked void no_private_stack_run_exception_sub_prog(void)
 {
 	asm volatile ("				\
@@ -478,9 +485,9 @@ __noinline static int timer_cb2(void *map, int *key, struct bpf_timer *timer)
 	return 0;
 }
 
-SEC("fentry/bpf_fentry_test9")
+SEC("kprobe.multi")
 __description("Private stack, async callback, not nested")
-__success __retval(0)
+__success
 __arch_x86_64
 __jited("	movabsq	$0x{{.*}}, %r9")
 __arch_arm64
@@ -503,9 +510,9 @@ int private_stack_async_callback_1(void)
 	return 0;
 }
 
-SEC("fentry/bpf_fentry_test9")
+SEC("kprobe.multi")
 __description("Private stack, async callback, potential nesting")
-__success __retval(0)
+__success
 __load_if_JITed()
 __log_level(4) __msg("stack depth max 272")
 __msg("subprog 0 (private_stack_async_callback_2) main {{.*}} stack 8")
@@ -532,7 +539,7 @@ int private_stack_async_callback_2(void)
 	return 0;
 }
 
-SEC("fentry/bpf_fentry_test9")
+SEC("kprobe.multi")
 __description("private stack, max stack depth is private stack")
 __success
 __log_level(4) __msg("stack depth max 256")
diff --git a/tools/testing/selftests/bpf/progs/verifier_unwind.c b/tools/testing/selftests/bpf/progs/verifier_unwind.c
index 5a21345c34d4..5cdfda16d22f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unwind.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unwind.c
@@ -1165,9 +1165,9 @@ static unsigned long may_unwind_big_stack(void)
  * x86-64 keeps the pointer to the private stack in r9 and saves it around a call.
  * The value that is read back in the pad is there only when r9 is restored.
  */
-SEC("raw_tp")
+SEC("kprobe.multi")
 __description("unwind: private stack, the callee unwinds")
-__success __retval(47)
+__success
 __load_if_JITed()
 __arch_x86_64
 __jited("	movl	$0x1234, %esi") /* past the load, see above */
@@ -1200,12 +1200,37 @@ __naked int private_stack_unwinds(void)
 }
 
 SEC("raw_tp")
-__description("unwind: private stack, the callee returns")
+__description("unwind: big stack, the callee unwinds")
+__success __retval(47)
+__load_if_JITed()
+__arch_x86_64
+__arch_arm64
+__naked int big_stack_unwinds(void)
+{
+	asm volatile (
+		"*(u64 *)(r10 - 200) = 42;"
+		"r1 = %[one] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"call may_unwind_big_stack;"
+		unwind_goto(1)
+		"exit;"
+		"r1 = *(u64 *)(r10 - 200);"
+		"r0 += r1;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+SEC("raw_tp")
+__description("unwind: big stack, the callee returns")
 __success __retval(49)
 __load_if_JITed()
 __arch_x86_64
 __arch_arm64
-__naked int private_stack_returns(void)
+__naked int big_stack_returns(void)
 {
 	asm volatile (
 		"*(u64 *)(r10 - 200) = 42;"

-- 
2.43.0

[PATCH bpf-next v2 2/5] bpf: Check program recursion per task via bpf_ctx

[Mike Bazov via B4 Relay]

From: Mike Bazov <mikeba@sentinelone.com>

Trampoline programs (fentry, fexit, fsession, fmod_ret and their
sleepable variants) and raw tracepoint programs (raw_tp, raw_tp.s,
tp_btf) run preemptibly: their runners only disable migration, and with
PREEMPT_RCU the RCU read section doesn't disable preemption either.
Raw tracepoint programs do so since tracepoints switched to SRCU-fast in
commit a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of
__DO_TRACE_CALL() with SRCU-fast").
Their recursion check is a per-CPU, per-program counter. When a task is
preempted inside a program on CPU N, the counter stays raised, and every
other task that enters the same program on CPU N is dropped and counted
as a recursion miss, although nothing recursed.

Check recursion per task instead. Each of these runners already links a
stack-allocated bpf_run_ctx into current->bpf_ctx. Move saved_run_ctx
into struct bpf_run_ctx, so every frame links to the outer one, and
record in each frame the program it runs. A program is recursing if a
frame in the current task's chain already runs it. Interrupts and NMIs
run on top of the interrupted task's chain, so recursion through them is
still caught; only other tasks are no longer blocked.

bpf_set_run_ctx() initializes the frame before publishing it, and
bpf_reset_run_ctx() unlinks it before its stack storage can be reused.
bpf_set_run_ctx() also accepts a NULL ctx, which io_uring passes to hide
the BPF context while submitting user requests.

Programs that use a per-CPU private stack keep the per-CPU check, since
two tasks must never use that stack on one CPU. After the previous
patch, these are only struct_ops programs that request one.

As a result, the same program can now run in two tasks at once on one
CPU. Programs that use a per-CPU map as scratch space across helper
calls may see another run's data, where the run was dropped before.
And as commit ec274bb4665c ("bpf: One may_goto time slice per prog
invocation") notes, a second invocation of a program on the same CPU
gives its may_goto loops more time; that invocation can now come from
another task, while the first one is preempted.

Link: https://lore.kernel.org/bpf/20261025-b4-bpf-trampoline-preempt-v1-0-198dcdc0c6ac@sentinelone.com/
Fixes: ca06f55b9002 ("bpf: Add per-program recursion prevention mechanism")
Fixes: a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")
Fixes: 439ebd5b5708 ("bpf: Add sleepable support for raw tracepoint programs")
Signed-off-by: Mike Bazov <mikeba@sentinelone.com>
---
 include/linux/bpf.h      | 48 +++++++++++++++++++++++++++++++++++++++++++-----
 kernel/bpf/trampoline.c  | 28 ++++++++++++++--------------
 kernel/trace/bpf_trace.c | 12 +++++-------
 net/bpf/test_run.c       |  4 ++--
 net/ipv4/bpf_tcp_ops.c   |  6 +++---
 5 files changed, 67 insertions(+), 31 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 520fd72e21fd..52e29b374b4a 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2785,7 +2785,10 @@ int bpf_prog_array_copy(struct bpf_prog_array *old_array,
 			struct bpf_prog_array **new_array);
 struct bpf_prog *bpf_prog_dummy(void);
 
-struct bpf_run_ctx {};
+struct bpf_run_ctx {
+	struct bpf_run_ctx *saved_run_ctx;
+	const struct bpf_prog *prog;
+};
 
 struct bpf_cg_run_ctx {
 	struct bpf_run_ctx run_ctx;
@@ -2803,7 +2806,6 @@ struct bpf_tramp_run_ctx {
 	struct bpf_run_ctx run_ctx;
 	u64 bpf_cookie;
 	int retval;
-	struct bpf_run_ctx *saved_run_ctx;
 };
 
 static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
@@ -2811,8 +2813,17 @@ static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
 	struct bpf_run_ctx *old_ctx = NULL;
 
 #ifdef CONFIG_BPF_SYSCALL
-	old_ctx = current->bpf_ctx;
-	current->bpf_ctx = new_ctx;
+	old_ctx = READ_ONCE(current->bpf_ctx);
+	if (new_ctx) {
+		new_ctx->saved_run_ctx = old_ctx;
+		new_ctx->prog = NULL;
+	}
+	/* Initialize the frame before interrupts and NMIs on this CPU see it. */
+	barrier();
+	WRITE_ONCE(current->bpf_ctx, new_ctx);
+#else
+	if (new_ctx)
+		new_ctx->saved_run_ctx = NULL;
 #endif
 	return old_ctx;
 }
@@ -2820,10 +2831,37 @@ static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
 static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 {
 #ifdef CONFIG_BPF_SYSCALL
-	current->bpf_ctx = old_ctx;
+	WRITE_ONCE(current->bpf_ctx, old_ctx);
+	/* Unlink the frame before its stack storage can be reused. */
+	barrier();
 #endif
 }
 
+static __always_inline bool
+bpf_prog_enter_recur_ctx(struct bpf_prog *prog, struct bpf_run_ctx *run_ctx)
+{
+	if (unlikely(prog->aux->jits_use_priv_stack))
+		return bpf_prog_get_recursion_context(prog);
+#ifdef CONFIG_BPF_SYSCALL
+	for (struct bpf_run_ctx *ctx = run_ctx->saved_run_ctx;
+	     ctx; ctx = ctx->saved_run_ctx) {
+		if (READ_ONCE(ctx->prog) == prog)
+			return false;
+	}
+	WRITE_ONCE(run_ctx->prog, prog);
+#endif
+	return true;
+}
+
+static __always_inline void
+bpf_prog_exit_recur_ctx(struct bpf_prog *prog, struct bpf_run_ctx *run_ctx)
+{
+	if (unlikely(prog->aux->jits_use_priv_stack))
+		bpf_prog_put_recursion_context(prog);
+	else
+		WRITE_ONCE(run_ctx->prog, NULL);
+}
+
 /* BPF program asks to bypass CAP_NET_BIND_SERVICE in bind. */
 #define BPF_RET_BIND_NO_CAP_NET_BIND_SERVICE			(1 << 0)
 /* BPF program asks to set CN on the packet. */
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 21d460866dfe..d66aa688688f 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -1342,9 +1342,9 @@ static u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tram
 {
 	rcu_read_lock_dont_migrate();
 
-	run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
+	bpf_set_run_ctx(&run_ctx->run_ctx);
 
-	if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+	if (unlikely(!bpf_prog_enter_recur_ctx(prog, &run_ctx->run_ctx))) {
 		bpf_prog_inc_misses_counter(prog);
 		if (prog->aux->recursion_detected)
 			prog->aux->recursion_detected(prog);
@@ -1385,10 +1385,10 @@ static void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start,
 					  struct bpf_tramp_run_ctx *run_ctx)
 	__releases(RCU)
 {
-	bpf_reset_run_ctx(run_ctx->saved_run_ctx);
+	bpf_reset_run_ctx(run_ctx->run_ctx.saved_run_ctx);
 
 	update_prog_stats(prog, start);
-	bpf_prog_put_recursion_context(prog);
+	bpf_prog_exit_recur_ctx(prog, &run_ctx->run_ctx);
 	rcu_read_unlock_migrate();
 }
 
@@ -1401,7 +1401,7 @@ static u64 notrace __bpf_prog_enter_lsm_cgroup(struct bpf_prog *prog,
 	 */
 	rcu_read_lock_dont_migrate();
 
-	run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
+	bpf_set_run_ctx(&run_ctx->run_ctx);
 
 	return NO_START_TIME;
 }
@@ -1410,7 +1410,7 @@ static void notrace __bpf_prog_exit_lsm_cgroup(struct bpf_prog *prog, u64 start,
 					       struct bpf_tramp_run_ctx *run_ctx)
 	__releases(RCU)
 {
-	bpf_reset_run_ctx(run_ctx->saved_run_ctx);
+	bpf_reset_run_ctx(run_ctx->run_ctx.saved_run_ctx);
 
 	rcu_read_unlock_migrate();
 }
@@ -1422,9 +1422,9 @@ u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
 	migrate_disable();
 	might_fault();
 
-	run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
+	bpf_set_run_ctx(&run_ctx->run_ctx);
 
-	if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+	if (unlikely(!bpf_prog_enter_recur_ctx(prog, &run_ctx->run_ctx))) {
 		bpf_prog_inc_misses_counter(prog);
 		if (prog->aux->recursion_detected)
 			prog->aux->recursion_detected(prog);
@@ -1436,10 +1436,10 @@ u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
 void notrace __bpf_prog_exit_sleepable_recur(struct bpf_prog *prog, u64 start,
 					     struct bpf_tramp_run_ctx *run_ctx)
 {
-	bpf_reset_run_ctx(run_ctx->saved_run_ctx);
+	bpf_reset_run_ctx(run_ctx->run_ctx.saved_run_ctx);
 
 	update_prog_stats(prog, start);
-	bpf_prog_put_recursion_context(prog);
+	bpf_prog_exit_recur_ctx(prog, &run_ctx->run_ctx);
 	migrate_enable();
 	rcu_read_unlock_trace();
 }
@@ -1451,7 +1451,7 @@ static u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog,
 	migrate_disable();
 	might_fault();
 
-	run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
+	bpf_set_run_ctx(&run_ctx->run_ctx);
 
 	return bpf_prog_start_time();
 }
@@ -1459,7 +1459,7 @@ static u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog,
 static void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start,
 					      struct bpf_tramp_run_ctx *run_ctx)
 {
-	bpf_reset_run_ctx(run_ctx->saved_run_ctx);
+	bpf_reset_run_ctx(run_ctx->run_ctx.saved_run_ctx);
 
 	update_prog_stats(prog, start);
 	migrate_enable();
@@ -1472,7 +1472,7 @@ static u64 notrace __bpf_prog_enter(struct bpf_prog *prog,
 {
 	rcu_read_lock_dont_migrate();
 
-	run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);
+	bpf_set_run_ctx(&run_ctx->run_ctx);
 
 	return bpf_prog_start_time();
 }
@@ -1481,7 +1481,7 @@ static void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start,
 				    struct bpf_tramp_run_ctx *run_ctx)
 	__releases(RCU)
 {
-	bpf_reset_run_ctx(run_ctx->saved_run_ctx);
+	bpf_reset_run_ctx(run_ctx->run_ctx.saved_run_ctx);
 
 	update_prog_stats(prog, start);
 	rcu_read_unlock_migrate();
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 2aec56d7f3d3..d596c333b7eb 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -2127,19 +2127,17 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
 		rcu_read_lock_dont_migrate();
 	}
 
-	if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+	run_ctx.bpf_cookie = link->cookie;
+	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+	if (unlikely(!bpf_prog_enter_recur_ctx(prog, &run_ctx.run_ctx))) {
 		bpf_prog_inc_misses_counter(prog);
 		goto out;
 	}
 
-	run_ctx.bpf_cookie = link->cookie;
-	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
-
 	(void)bpf_prog_run(prog, args);
-
-	bpf_reset_run_ctx(old_run_ctx);
 out:
-	bpf_prog_put_recursion_context(prog);
+	bpf_reset_run_ctx(old_run_ctx);
+	bpf_prog_exit_recur_ctx(prog, &run_ctx.run_ctx);
 
 	if (sleepable) {
 		migrate_enable();
diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
index d6c9dc62182e..362d9ec0d8f8 100644
--- a/net/bpf/test_run.c
+++ b/net/bpf/test_run.c
@@ -759,7 +759,7 @@ __bpf_prog_test_run_raw_tp(void *data)
 		rcu_read_lock();
 	}
 
-	if (unlikely(!bpf_prog_get_recursion_context(info->prog))) {
+	if (unlikely(!bpf_prog_enter_recur_ctx(info->prog, &run_ctx.run_ctx))) {
 		bpf_prog_inc_misses_counter(info->prog);
 		goto out;
 	}
@@ -767,7 +767,7 @@ __bpf_prog_test_run_raw_tp(void *data)
 	info->retval = bpf_prog_run(info->prog, info->ctx);
 
 out:
-	bpf_prog_put_recursion_context(info->prog);
+	bpf_prog_exit_recur_ctx(info->prog, &run_ctx.run_ctx);
 
 	if (info->prog->sleepable) {
 		migrate_enable();
diff --git a/net/ipv4/bpf_tcp_ops.c b/net/ipv4/bpf_tcp_ops.c
index 2ba73dd6c52c..e1ca34e63de0 100644
--- a/net/ipv4/bpf_tcp_ops.c
+++ b/net/ipv4/bpf_tcp_ops.c
@@ -207,11 +207,11 @@ BPF_CALL_0(bpf_tcp_ops_get_retval)
 	 * return value chained across the bpf_tcp_ops attached to the cgroup
 	 * and is this program's saved_run_ctx.
 	 */
-	if (WARN_ON_ONCE(!ctx->saved_run_ctx))
+	if (WARN_ON_ONCE(!ctx->run_ctx.saved_run_ctx))
 		return 0;
 
-	return container_of(ctx->saved_run_ctx, struct bpf_tramp_run_ctx,
-			    run_ctx)->retval;
+	return container_of(ctx->run_ctx.saved_run_ctx,
+			    struct bpf_tramp_run_ctx, run_ctx)->retval;
 }
 
 const struct bpf_func_proto bpf_tcp_ops_get_retval_proto = {

-- 
2.43.0

[PATCH bpf-next v2 3/5] bpf: Check recursion per task for programs on syscall tracepoints

[Mike Bazov via B4 Relay]

From: Mike Bazov <mikeba@sentinelone.com>

Since commit 57918341dd19 ("bpf: Add sleepable support for classic
tracepoint programs"), classic tracepoint programs on syscall
tracepoints run before perf disables preemption, through
trace_call_bpf_faultable() and bpf_prog_run_array_sleepable(). That
runner still checks recursion with the per-CPU counter, so a task
preempted inside such a program blocks every other task on the same
CPU, and their runs are dropped as recursion misses. A sleepable
program that faults keeps the counter raised for the whole sleep.

Use the per-task check from the previous patch. The runner publishes one
frame for the whole program array, so the program is recorded in it per
item and cleared before the next one.

Fixes: 57918341dd19 ("bpf: Add sleepable support for classic tracepoint programs")
Signed-off-by: Mike Bazov <mikeba@sentinelone.com>
---
 include/linux/bpf.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 52e29b374b4a..649ff2950fd6 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3549,9 +3549,9 @@ bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
 			continue;
 		}
 
-		if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+		if (unlikely(!bpf_prog_enter_recur_ctx(prog, &run_ctx.run_ctx))) {
 			bpf_prog_inc_misses_counter(prog);
-			bpf_prog_put_recursion_context(prog);
+			bpf_prog_exit_recur_ctx(prog, &run_ctx.run_ctx);
 			item++;
 			continue;
 		}
@@ -3565,7 +3565,7 @@ bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
 			ret &= run_prog(prog, ctx);
 		}
 
-		bpf_prog_put_recursion_context(prog);
+		bpf_prog_exit_recur_ctx(prog, &run_ctx.run_ctx);
 		item++;
 	}
 	bpf_reset_run_ctx(old_run_ctx);

-- 
2.43.0

[PATCH bpf-next v2 4/5] selftests/bpf: Add trampoline and raw_tp preemption recursion test

[Mike Bazov via B4 Relay]

From: Mike Bazov <mikeba@sentinelone.com>

Pin two threads to one CPU. Thread A enters the program under test and,
from inside it, fills a 64-byte stack buffer and wakes thread B, which
is SCHED_FIFO, so B preempts A in the middle of the program and enters
the same program on the same CPU. Right before entering, B checks that A
is still inside the program, and A confirms that B's run ended before A
checks its buffer. When A resumes, it checks its buffer against its
thread ID, read again from the kernel. Each subtest runs 20 rounds and
asserts that B preempted A inside the program in every round, that both
threads ran every round, that there were no recursion misses, and that
A's stack is intact. If B never preempted A inside the program, the
subtest is skipped.

Subtests: fentry, fexit, fentry.s, fexit.s, fsession, raw_tp, raw_tp.s
and tp_btf. Without the per-task recursion check, B is dropped in every
round.

tailcall_raw_tp and two_dispatchers enter a large-stack handler through
tail calls, from one dispatcher and from two dispatchers. Without the
private-stack change, two_dispatchers corrupts A's stack. nested_prog
wakes B from a second program that runs nested inside A's.

recursion_raw_tp, recursion_tp_btf and recursion_test_run check that
true recursion is still rejected: the program calls bpf_printk(), which
fires the bpf_trace_printk tracepoint it is attached to, and the nested
run must be dropped and counted as a recursion miss. recursion_test_run
starts the outer run with BPF_PROG_TEST_RUN. recursion_two_hops has a
second program run in between, so the check has to look past the frame
right below.

Each subtest loads only its own programs, and is skipped if the kernel
doesn't support its program type.

Signed-off-by: Mike Bazov <mikeba@sentinelone.com>
---
 .../selftests/bpf/prog_tests/trampoline_preempt.c  | 429 +++++++++++++++++++++
 .../selftests/bpf/progs/trampoline_preempt.c       | 265 +++++++++++++
 2 files changed, 694 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c b/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c
new file mode 100644
index 000000000000..e01a2ba8916a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c
@@ -0,0 +1,429 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <errno.h>
+#include <pthread.h>
+#include <sched.h>
+#include <signal.h>
+#include <sys/prctl.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+#include <test_progs.h>
+#include "trampoline_preempt.skel.h"
+
+#define NR_ROUNDS 20
+
+#ifndef ENOTSUPP
+#define ENOTSUPP 524
+#endif
+
+enum subtest_kind {
+	SUBTEST_FENTRY,
+	SUBTEST_FEXIT,
+	SUBTEST_FENTRY_S,
+	SUBTEST_FEXIT_S,
+	SUBTEST_RAW_TP,
+	SUBTEST_RAW_TP_S,
+	SUBTEST_TP_BTF,
+	SUBTEST_FSESSION,
+	SUBTEST_TAILCALL_RAW_TP,
+	SUBTEST_TWO_DISPATCHERS,
+	SUBTEST_NESTED_PROG,
+};
+
+enum recursion_kind {
+	RECURSION_RAW_TP,
+	RECURSION_TP_BTF,
+	RECURSION_TEST_RUN,
+	RECURSION_TWO_HOPS,
+};
+
+struct ctx {
+	struct trampoline_preempt *skel;
+	int cpu;
+	int go_pipe[2];
+	int done_pipe[2];
+	sigset_t sigset;
+	int err_a;
+	int err_b;
+	int overlaps;
+};
+
+static int pin_to_cpu(int cpu)
+{
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	if (sched_setaffinity(0, sizeof(set), &set))
+		return -errno;
+	return 0;
+}
+
+static void *worker_a(void *arg)
+{
+	struct ctx *c = arg;
+	char comm[16] = {};
+	char ch = 'a';
+
+	c->err_a = pin_to_cpu(c->cpu);
+	if (!c->err_a && prctl(PR_GET_NAME, comm, 0, 0, 0))
+		c->err_a = -errno;
+	c->skel->bss->tid_a = (int)syscall(SYS_gettid);
+
+	if (write(c->done_pipe[1], &ch, 1) != 1 || c->err_a)
+		goto out;
+
+	while (read(c->go_pipe[0], &ch, 1) == 1) {
+		if (prctl(PR_SET_NAME, comm, 0, 0, 0)) {
+			c->err_a = -errno;
+			break;
+		}
+		if (write(c->done_pipe[1], &ch, 1) != 1) {
+			c->err_a = -EIO;
+			break;
+		}
+	}
+
+out:
+	close(c->done_pipe[1]);
+	c->done_pipe[1] = -1;
+	return NULL;
+}
+
+static void *worker_b(void *arg)
+{
+	struct sched_param sp = { .sched_priority = 1 };
+	struct timespec ts = { .tv_sec = 5 };
+	struct ctx *c = arg;
+	char comm[16] = {};
+	char ch = 'b';
+	int i;
+
+	c->err_b = pin_to_cpu(c->cpu);
+	if (c->err_b)
+		goto out;
+	if (sched_setscheduler(0, SCHED_FIFO, &sp)) {
+		c->err_b = -errno;
+		goto out;
+	}
+	if (prctl(PR_GET_NAME, comm, 0, 0, 0)) {
+		c->err_b = -errno;
+		goto out;
+	}
+	c->skel->bss->tid_b = (int)syscall(SYS_gettid);
+
+	for (i = 0; i < NR_ROUNDS; i++) {
+		if (write(c->go_pipe[1], &ch, 1) != 1) {
+			c->err_b = -EIO;
+			break;
+		}
+		while (sigtimedwait(&c->sigset, NULL, &ts) < 0) {
+			if (errno == EINTR)
+				continue;
+			c->err_b = -errno;
+			break;
+		}
+		if (c->err_b)
+			break;
+		/* B preempted A only if A is still inside the program */
+		if (*(volatile int *)&c->skel->bss->a_inside)
+			c->overlaps++;
+		if (prctl(PR_SET_NAME, comm, 0, 0, 0)) {
+			c->err_b = -errno;
+			break;
+		}
+		if (read(c->done_pipe[0], &ch, 1) != 1) {
+			c->err_b = -EIO;
+			break;
+		}
+	}
+
+out:
+	close(c->go_pipe[1]);
+	c->go_pipe[1] = -1;
+	return NULL;
+}
+
+static void run_subtest(enum subtest_kind kind)
+{
+	struct ctx c = {
+		.go_pipe = { -1, -1 },
+		.done_pipe = { -1, -1 },
+	};
+	struct bpf_prog_info info = {};
+	__u32 info_len = sizeof(info);
+	struct bpf_program *prog = NULL, *extra = NULL, *tail = NULL, *p;
+	struct bpf_link *link = NULL, *link2 = NULL, *wakeup_link = NULL;
+	struct bpf_map *jmp = NULL;
+	sigset_t oldset;
+	pthread_t ta, tb;
+	bool ta_started = false, tb_started = false, sigmask_set = false;
+	bool via_map;
+	__u32 zero = 0;
+	int tail_fd;
+	char ch;
+	int err;
+
+	c.skel = trampoline_preempt__open();
+	if (!ASSERT_OK_PTR(c.skel, "open"))
+		return;
+
+	switch (kind) {
+	case SUBTEST_FENTRY:
+		prog = c.skel->progs.on_fentry;
+		break;
+	case SUBTEST_FEXIT:
+		prog = c.skel->progs.on_fexit;
+		break;
+	case SUBTEST_FENTRY_S:
+		prog = c.skel->progs.on_fentry_s;
+		break;
+	case SUBTEST_FEXIT_S:
+		prog = c.skel->progs.on_fexit_s;
+		break;
+	case SUBTEST_RAW_TP:
+		prog = c.skel->progs.on_raw_tp;
+		break;
+	case SUBTEST_RAW_TP_S:
+		prog = c.skel->progs.on_raw_tp_s;
+		break;
+	case SUBTEST_TP_BTF:
+		prog = c.skel->progs.on_tp_btf;
+		break;
+	case SUBTEST_FSESSION:
+		prog = c.skel->progs.on_fsession;
+		break;
+	case SUBTEST_TAILCALL_RAW_TP:
+		prog = c.skel->progs.dispatcher_raw_tp;
+		tail = c.skel->progs.handler_raw_tp;
+		jmp = c.skel->maps.jmp_raw_tp;
+		break;
+	case SUBTEST_TWO_DISPATCHERS:
+		prog = c.skel->progs.dispatcher_a;
+		extra = c.skel->progs.dispatcher_b;
+		tail = c.skel->progs.handler_raw_tp;
+		jmp = c.skel->maps.jmp_raw_tp;
+		break;
+	case SUBTEST_NESTED_PROG:
+		prog = c.skel->progs.on_raw_tp_nested;
+		break;
+	}
+	via_map = kind == SUBTEST_NESTED_PROG;
+	bpf_object__for_each_program(p, c.skel->obj)
+		bpf_program__set_autoload(p, p == prog || p == extra || p == tail ||
+					  (via_map && p == c.skel->progs.on_wakeup_delete));
+
+	err = trampoline_preempt__load(c.skel);
+	if (err == -EOPNOTSUPP || err == -ENOTSUPP) {
+		test__skip();
+		goto out;
+	}
+	if (!ASSERT_OK(err, "load"))
+		goto out;
+
+	c.cpu = sched_getcpu();
+	if (!ASSERT_GE(c.cpu, 0, "sched_getcpu"))
+		goto out;
+
+	if (!ASSERT_OK(pipe(c.go_pipe), "go_pipe"))
+		goto out;
+	if (!ASSERT_OK(pipe(c.done_pipe), "done_pipe"))
+		goto out;
+
+	sigemptyset(&c.sigset);
+	sigaddset(&c.sigset, SIGUSR1);
+	if (!ASSERT_OK(pthread_sigmask(SIG_BLOCK, &c.sigset, &oldset), "sigmask"))
+		goto out;
+	sigmask_set = true;
+
+	c.skel->bss->sig = SIGUSR1;
+	if (via_map) {
+		wakeup_link = bpf_program__attach(c.skel->progs.on_wakeup_delete);
+		if (!ASSERT_OK_PTR(wakeup_link, "attach_wakeup"))
+			goto out;
+	}
+	link = bpf_program__attach(prog);
+	if (!ASSERT_OK_PTR(link, "attach"))
+		goto out;
+	if (extra) {
+		link2 = bpf_program__attach(extra);
+		if (!ASSERT_OK_PTR(link2, "attach_extra"))
+			goto out;
+	}
+	if (tail) {
+		tail_fd = bpf_program__fd(tail);
+		if (!ASSERT_OK(bpf_map_update_elem(bpf_map__fd(jmp), &zero, &tail_fd,
+						   BPF_ANY), "jmp_update"))
+			goto out;
+	}
+
+	if (!ASSERT_OK(pthread_create(&ta, NULL, worker_a, &c), "pthread_a"))
+		goto out;
+	ta_started = true;
+
+	if (!ASSERT_EQ(read(c.done_pipe[0], &ch, 1), 1, "ready_a"))
+		goto out;
+	if (!ASSERT_OK(c.err_a, "setup_a"))
+		goto out;
+
+	if (!ASSERT_OK(pthread_create(&tb, NULL, worker_b, &c), "pthread_b"))
+		goto out;
+	tb_started = true;
+
+	pthread_join(tb, NULL);
+	tb_started = false;
+	pthread_join(ta, NULL);
+	ta_started = false;
+
+	if (c.err_b == -EPERM || c.err_b == -EACCES) {
+		test__skip();
+		goto out;
+	}
+
+	ASSERT_OK(c.err_a, "worker_a_err");
+	ASSERT_OK(c.err_b, "worker_b_err");
+	ASSERT_EQ(c.skel->bss->sig_err, 0, "sig_err");
+	if (!c.overlaps) {
+		/* B never preempted A inside the program, so nothing was tested */
+		test__skip();
+		goto out;
+	}
+	ASSERT_EQ(c.overlaps, NR_ROUNDS, "overlaps");
+	ASSERT_EQ(c.skel->bss->stack_corrupt, 0, "stack_corrupt");
+	ASSERT_EQ(c.skel->bss->runs_a, NR_ROUNDS, "runs_a");
+	ASSERT_EQ(c.skel->bss->runs_b, NR_ROUNDS, "runs_b");
+	ASSERT_EQ(c.skel->bss->b_inside_a, c.skel->bss->runs_b, "b_inside_a");
+
+	err = bpf_prog_get_info_by_fd(bpf_program__fd(prog), &info, &info_len);
+	if (ASSERT_OK(err, "prog_info"))
+		ASSERT_EQ(info.recursion_misses, 0, "recursion_misses");
+
+out:
+	if (c.go_pipe[1] >= 0)
+		close(c.go_pipe[1]);
+	if (tb_started)
+		pthread_join(tb, NULL);
+	if (ta_started)
+		pthread_join(ta, NULL);
+	if (c.go_pipe[0] >= 0)
+		close(c.go_pipe[0]);
+	if (c.done_pipe[0] >= 0)
+		close(c.done_pipe[0]);
+	if (c.done_pipe[1] >= 0)
+		close(c.done_pipe[1]);
+	if (sigmask_set)
+		pthread_sigmask(SIG_SETMASK, &oldset, NULL);
+	bpf_link__destroy(link);
+	bpf_link__destroy(link2);
+	bpf_link__destroy(wakeup_link);
+	trampoline_preempt__destroy(c.skel);
+}
+
+/*
+ * The program calls bpf_printk(), which fires the bpf_trace_printk tracepoint
+ * the program is attached to. The nested run must be rejected as recursion.
+ * With two hops, a second program runs in between, so the check has to look
+ * past the frame right below.
+ */
+static void run_recursion_subtest(enum recursion_kind kind)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct bpf_prog_info info = {};
+	__u32 info_len = sizeof(info);
+	struct trampoline_preempt *skel;
+	struct bpf_program *prog, *hop = NULL, *trigger, *p;
+	struct bpf_link *link = NULL, *hop_link = NULL;
+	int err;
+
+	skel = trampoline_preempt__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	switch (kind) {
+	case RECURSION_TP_BTF:
+		prog = skel->progs.recur_tp_btf;
+		break;
+	case RECURSION_TWO_HOPS:
+		prog = skel->progs.recur_two_hops;
+		hop = skel->progs.recur_hop;
+		break;
+	default:
+		prog = skel->progs.recur_raw_tp;
+		break;
+	}
+	trigger = kind == RECURSION_TEST_RUN ? prog : skel->progs.recur_trigger;
+	bpf_object__for_each_program(p, skel->obj)
+		bpf_program__set_autoload(p, p == prog || p == hop || p == trigger);
+
+	err = trampoline_preempt__load(skel);
+	if (err == -EOPNOTSUPP || err == -ENOTSUPP) {
+		test__skip();
+		goto out;
+	}
+	if (!ASSERT_OK(err, "load"))
+		goto out;
+	link = bpf_program__attach(prog);
+	if (!ASSERT_OK_PTR(link, "attach"))
+		goto out;
+	if (hop) {
+		hop_link = bpf_program__attach(hop);
+		if (!ASSERT_OK_PTR(hop_link, "attach_hop"))
+			goto out;
+	}
+
+	skel->bss->recur_tid = (int)syscall(SYS_gettid);
+	err = bpf_prog_test_run_opts(bpf_program__fd(trigger), &opts);
+	skel->bss->recur_tid = 0;
+	if (!ASSERT_OK(err, "test_run"))
+		goto out;
+
+	ASSERT_EQ(skel->bss->recur_runs, 1, "recur_runs");
+	err = bpf_prog_get_info_by_fd(bpf_program__fd(prog), &info, &info_len);
+	if (ASSERT_OK(err, "prog_info"))
+		ASSERT_EQ(info.recursion_misses, 1, "recursion_misses");
+	if (hop) {
+		ASSERT_EQ(skel->bss->hop_runs, 1, "hop_runs");
+		memset(&info, 0, sizeof(info));
+		err = bpf_prog_get_info_by_fd(bpf_program__fd(hop), &info, &info_len);
+		if (ASSERT_OK(err, "hop_info"))
+			ASSERT_EQ(info.recursion_misses, 0, "hop_recursion_misses");
+	}
+out:
+	bpf_link__destroy(hop_link);
+	bpf_link__destroy(link);
+	trampoline_preempt__destroy(skel);
+}
+
+void test_trampoline_preempt(void)
+{
+	if (test__start_subtest("fentry"))
+		run_subtest(SUBTEST_FENTRY);
+	if (test__start_subtest("fexit"))
+		run_subtest(SUBTEST_FEXIT);
+	if (test__start_subtest("fentry_s"))
+		run_subtest(SUBTEST_FENTRY_S);
+	if (test__start_subtest("fexit_s"))
+		run_subtest(SUBTEST_FEXIT_S);
+	if (test__start_subtest("raw_tp"))
+		run_subtest(SUBTEST_RAW_TP);
+	if (test__start_subtest("raw_tp_s"))
+		run_subtest(SUBTEST_RAW_TP_S);
+	if (test__start_subtest("tp_btf"))
+		run_subtest(SUBTEST_TP_BTF);
+	if (test__start_subtest("fsession"))
+		run_subtest(SUBTEST_FSESSION);
+	if (test__start_subtest("tailcall_raw_tp"))
+		run_subtest(SUBTEST_TAILCALL_RAW_TP);
+	if (test__start_subtest("two_dispatchers"))
+		run_subtest(SUBTEST_TWO_DISPATCHERS);
+	if (test__start_subtest("nested_prog"))
+		run_subtest(SUBTEST_NESTED_PROG);
+	if (test__start_subtest("recursion_raw_tp"))
+		run_recursion_subtest(RECURSION_RAW_TP);
+	if (test__start_subtest("recursion_tp_btf"))
+		run_recursion_subtest(RECURSION_TP_BTF);
+	if (test__start_subtest("recursion_test_run"))
+		run_recursion_subtest(RECURSION_TEST_RUN);
+	if (test__start_subtest("recursion_two_hops"))
+		run_recursion_subtest(RECURSION_TWO_HOPS);
+}
diff --git a/tools/testing/selftests/bpf/progs/trampoline_preempt.c b/tools/testing/selftests/bpf/progs/trampoline_preempt.c
new file mode 100644
index 000000000000..78ed74627a79
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/trampoline_preempt.c
@@ -0,0 +1,265 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "vmlinux.h"
+#include <asm/unistd.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct task_struct *bpf_task_from_pid(s32 pid) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+int bpf_send_signal_task(struct task_struct *task, int sig, enum pid_type type, u64 value) __ksym;
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, int);
+} wakeup_map SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u32);
+} jmp_raw_tp SEC(".maps");
+
+int tid_a;
+int tid_b;
+int sig;
+int runs_a;
+int runs_b;
+int sig_err;
+int stack_corrupt;
+int a_inside;
+int b_inside_a;
+int recur_tid;
+int recur_runs;
+
+static __always_inline void fill_stack(__u64 *buf, __u32 tid)
+{
+	#pragma unroll
+	for (int i = 0; i < 8; i++)
+		buf[i] = ((__u64)tid << 32) | (0x5a5a0000U + i);
+}
+
+static __always_inline void check_stack(const __u64 *buf, __u32 tid)
+{
+	#pragma unroll
+	for (int i = 0; i < 8; i++) {
+		if (buf[i] != (((__u64)tid << 32) | (0x5a5a0000U + i)))
+			stack_corrupt = 1;
+	}
+}
+
+static void signal_worker_b(void)
+{
+	struct task_struct *task;
+
+	bpf_rcu_read_lock();
+	task = bpf_task_from_pid(tid_b);
+	if (!task) {
+		sig_err = 1;
+		bpf_rcu_read_unlock();
+		return;
+	}
+	if (bpf_send_signal_task(task, sig, PIDTYPE_PID, 0))
+		sig_err = 1;
+	bpf_task_release(task);
+	bpf_rcu_read_unlock();
+}
+
+static __always_inline int handle_comm(bool via_map)
+{
+	__u32 tid = (__u32)bpf_get_current_pid_tgid();
+	__u64 stack_buf[8];
+	int zero = 0;
+
+	if (tid != tid_a && tid != tid_b)
+		return 0;
+
+	fill_stack(stack_buf, tid);
+	asm volatile("" : "+m"(stack_buf));
+
+	if (tid == tid_a) {
+		int runs_b_before = *(volatile int *)&runs_b;
+
+		runs_a++;
+		*(volatile int *)&a_inside = 1;
+		if (via_map)
+			bpf_map_delete_elem(&wakeup_map, &zero);
+		else
+			signal_worker_b();
+		/* B's whole run happened after the wakeup and before the stack check */
+		if (*(volatile int *)&runs_b != runs_b_before)
+			b_inside_a++;
+	} else {
+		runs_b++;
+	}
+
+	asm volatile("" : "+m"(stack_buf));
+	/* Don't trust a copy of tid that may have been spilled to the stack under test */
+	tid = (__u32)bpf_get_current_pid_tgid();
+	check_stack(stack_buf, tid);
+	if (tid == tid_a)
+		*(volatile int *)&a_inside = 0;
+	return 0;
+}
+
+static __always_inline int recur_body(void)
+{
+	if ((__u32)bpf_get_current_pid_tgid() != recur_tid)
+		return 0;
+	recur_runs++;
+	bpf_printk("recur");
+	return 0;
+}
+
+SEC("fentry/__set_task_comm")
+int BPF_PROG(on_fentry)
+{
+	return handle_comm(false);
+}
+
+SEC("fexit/__set_task_comm")
+int BPF_PROG(on_fexit)
+{
+	return handle_comm(false);
+}
+
+SEC("fentry.s/" SYS_PREFIX "sys_prctl")
+int BPF_PROG(on_fentry_s)
+{
+	return handle_comm(false);
+}
+
+SEC("fexit.s/" SYS_PREFIX "sys_prctl")
+int BPF_PROG(on_fexit_s)
+{
+	return handle_comm(false);
+}
+
+SEC("fentry/htab_map_delete_elem")
+int BPF_PROG(on_wakeup_delete, struct bpf_map *map)
+{
+	if (map == (struct bpf_map *)&wakeup_map)
+		signal_worker_b();
+	return 0;
+}
+
+SEC("raw_tp/task_rename")
+int BPF_PROG(on_raw_tp)
+{
+	return handle_comm(false);
+}
+
+SEC("raw_tp/task_rename")
+int BPF_PROG(on_raw_tp_nested)
+{
+	return handle_comm(true);
+}
+
+SEC("raw_tp.s/sys_enter")
+int BPF_PROG(on_raw_tp_s, struct pt_regs *regs, long id)
+{
+	if (id != __NR_prctl)
+		return 0;
+	return handle_comm(false);
+}
+
+SEC("tp_btf/task_rename")
+int BPF_PROG(on_tp_btf)
+{
+	return handle_comm(false);
+}
+
+SEC("fsession/__set_task_comm")
+int BPF_PROG(on_fsession)
+{
+	if (bpf_session_is_return(ctx))
+		return 0;
+	return handle_comm(false);
+}
+
+SEC("raw_tp/task_rename")
+int dispatcher_raw_tp(void *ctx)
+{
+	bpf_tail_call(ctx, &jmp_raw_tp, 0);
+	return 0;
+}
+
+SEC("raw_tp/task_rename")
+int handler_raw_tp(void *ctx)
+{
+	return handle_comm(false);
+}
+
+SEC("raw_tp/task_rename")
+int dispatcher_a(void *ctx)
+{
+	if ((__u32)bpf_get_current_pid_tgid() == tid_a)
+		bpf_tail_call(ctx, &jmp_raw_tp, 0);
+	return 0;
+}
+
+SEC("raw_tp/sys_enter")
+int BPF_PROG(dispatcher_b, struct pt_regs *regs, long id)
+{
+	if (id == __NR_prctl && (__u32)bpf_get_current_pid_tgid() == tid_b)
+		bpf_tail_call(ctx, &jmp_raw_tp, 0);
+	return 0;
+}
+
+SEC("raw_tp/bpf_trace_printk")
+int recur_raw_tp(void *ctx)
+{
+	return recur_body();
+}
+
+SEC("tp_btf/bpf_trace_printk")
+int BPF_PROG(recur_tp_btf)
+{
+	return recur_body();
+}
+
+SEC("syscall")
+int recur_trigger(void *ctx)
+{
+	bpf_printk("trigger");
+	return 0;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, int);
+} recur_map SEC(".maps");
+
+int hop_runs;
+
+SEC("raw_tp/bpf_trace_printk")
+int recur_two_hops(void *ctx)
+{
+	int zero = 0;
+
+	if ((__u32)bpf_get_current_pid_tgid() != recur_tid)
+		return 0;
+	recur_runs++;
+	bpf_map_delete_elem(&recur_map, &zero);
+	return 0;
+}
+
+SEC("fentry/htab_map_delete_elem")
+int BPF_PROG(recur_hop, struct bpf_map *map)
+{
+	if (map != (struct bpf_map *)&recur_map ||
+	    (__u32)bpf_get_current_pid_tgid() != recur_tid)
+		return 0;
+	hop_runs++;
+	bpf_printk("hop");
+	return 0;
+}

-- 
2.43.0

[PATCH bpf-next v2 5/5] selftests/bpf: Test syscall tracepoint programs under preemption

[Mike Bazov via B4 Relay]

From: Mike Bazov <mikeba@sentinelone.com>

Add subtests for classic tracepoint programs on sys_enter_prctl, which
run preemptibly:

- tp_syscall: the same two-thread scenario as the other subtests.
- tailcall_tp: a tracepoint dispatcher tail-calls a large-stack handler.
- tp_dual: one program attached to sys_enter_prctl and to the regular
  task_rename tracepoint, told apart by their attach cookies. A enters
  it through the syscall tracepoint, B through task_rename.

Without the per-task check, every subtest counts recursion misses:
tp_syscall and tailcall_tp drop B's run, and tp_dual drops B's run
through sys_enter_prctl. Without the private-stack change, tailcall_tp
and tp_dual corrupt A's stack.

Signed-off-by: Mike Bazov <mikeba@sentinelone.com>
---
 .../selftests/bpf/prog_tests/trampoline_preempt.c  | 43 ++++++++++++++++++++--
 .../selftests/bpf/progs/trampoline_preempt.c       | 37 +++++++++++++++++++
 2 files changed, 76 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c b/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c
index e01a2ba8916a..a8109290e88d 100644
--- a/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c
+++ b/tools/testing/selftests/bpf/prog_tests/trampoline_preempt.c
@@ -25,8 +25,11 @@ enum subtest_kind {
 	SUBTEST_RAW_TP_S,
 	SUBTEST_TP_BTF,
 	SUBTEST_FSESSION,
+	SUBTEST_TP_SYSCALL,
 	SUBTEST_TAILCALL_RAW_TP,
+	SUBTEST_TAILCALL_TP,
 	SUBTEST_TWO_DISPATCHERS,
+	SUBTEST_TP_DUAL,
 	SUBTEST_NESTED_PROG,
 };
 
@@ -158,7 +161,7 @@ static void run_subtest(enum subtest_kind kind)
 	sigset_t oldset;
 	pthread_t ta, tb;
 	bool ta_started = false, tb_started = false, sigmask_set = false;
-	bool via_map;
+	bool via_map, dual = false;
 	__u32 zero = 0;
 	int tail_fd;
 	char ch;
@@ -193,17 +196,29 @@ static void run_subtest(enum subtest_kind kind)
 	case SUBTEST_FSESSION:
 		prog = c.skel->progs.on_fsession;
 		break;
+	case SUBTEST_TP_SYSCALL:
+		prog = c.skel->progs.on_tp_syscall;
+		break;
 	case SUBTEST_TAILCALL_RAW_TP:
 		prog = c.skel->progs.dispatcher_raw_tp;
 		tail = c.skel->progs.handler_raw_tp;
 		jmp = c.skel->maps.jmp_raw_tp;
 		break;
+	case SUBTEST_TAILCALL_TP:
+		prog = c.skel->progs.dispatcher_tp;
+		tail = c.skel->progs.handler_tp;
+		jmp = c.skel->maps.jmp_tp;
+		break;
 	case SUBTEST_TWO_DISPATCHERS:
 		prog = c.skel->progs.dispatcher_a;
 		extra = c.skel->progs.dispatcher_b;
 		tail = c.skel->progs.handler_raw_tp;
 		jmp = c.skel->maps.jmp_raw_tp;
 		break;
+	case SUBTEST_TP_DUAL:
+		prog = c.skel->progs.on_tp_dual;
+		dual = true;
+		break;
 	case SUBTEST_NESTED_PROG:
 		prog = c.skel->progs.on_raw_tp_nested;
 		break;
@@ -242,9 +257,23 @@ static void run_subtest(enum subtest_kind kind)
 		if (!ASSERT_OK_PTR(wakeup_link, "attach_wakeup"))
 			goto out;
 	}
-	link = bpf_program__attach(prog);
-	if (!ASSERT_OK_PTR(link, "attach"))
-		goto out;
+	if (dual) {
+		LIBBPF_OPTS(bpf_tracepoint_opts, topts, .bpf_cookie = 1);
+
+		link = bpf_program__attach_tracepoint_opts(prog, "syscalls",
+							   "sys_enter_prctl", &topts);
+		if (!ASSERT_OK_PTR(link, "attach_sys_enter"))
+			goto out;
+		topts.bpf_cookie = 2;
+		link2 = bpf_program__attach_tracepoint_opts(prog, "task",
+							    "task_rename", &topts);
+		if (!ASSERT_OK_PTR(link2, "attach_task_rename"))
+			goto out;
+	} else {
+		link = bpf_program__attach(prog);
+		if (!ASSERT_OK_PTR(link, "attach"))
+			goto out;
+	}
 	if (extra) {
 		link2 = bpf_program__attach(extra);
 		if (!ASSERT_OK_PTR(link2, "attach_extra"))
@@ -412,10 +441,16 @@ void test_trampoline_preempt(void)
 		run_subtest(SUBTEST_TP_BTF);
 	if (test__start_subtest("fsession"))
 		run_subtest(SUBTEST_FSESSION);
+	if (test__start_subtest("tp_syscall"))
+		run_subtest(SUBTEST_TP_SYSCALL);
 	if (test__start_subtest("tailcall_raw_tp"))
 		run_subtest(SUBTEST_TAILCALL_RAW_TP);
+	if (test__start_subtest("tailcall_tp"))
+		run_subtest(SUBTEST_TAILCALL_TP);
 	if (test__start_subtest("two_dispatchers"))
 		run_subtest(SUBTEST_TWO_DISPATCHERS);
+	if (test__start_subtest("tp_dual"))
+		run_subtest(SUBTEST_TP_DUAL);
 	if (test__start_subtest("nested_prog"))
 		run_subtest(SUBTEST_NESTED_PROG);
 	if (test__start_subtest("recursion_raw_tp"))
diff --git a/tools/testing/selftests/bpf/progs/trampoline_preempt.c b/tools/testing/selftests/bpf/progs/trampoline_preempt.c
index 78ed74627a79..e8e610fc885e 100644
--- a/tools/testing/selftests/bpf/progs/trampoline_preempt.c
+++ b/tools/testing/selftests/bpf/progs/trampoline_preempt.c
@@ -27,6 +27,13 @@ struct {
 	__type(value, __u32);
 } jmp_raw_tp SEC(".maps");
 
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u32);
+} jmp_tp SEC(".maps");
+
 int tid_a;
 int tid_b;
 int sig;
@@ -184,6 +191,12 @@ int BPF_PROG(on_fsession)
 	return handle_comm(false);
 }
 
+SEC("tp/syscalls/sys_enter_prctl")
+int on_tp_syscall(void *ctx)
+{
+	return handle_comm(false);
+}
+
 SEC("raw_tp/task_rename")
 int dispatcher_raw_tp(void *ctx)
 {
@@ -197,6 +210,19 @@ int handler_raw_tp(void *ctx)
 	return handle_comm(false);
 }
 
+SEC("tp/syscalls/sys_enter_prctl")
+int dispatcher_tp(void *ctx)
+{
+	bpf_tail_call(ctx, &jmp_tp, 0);
+	return 0;
+}
+
+SEC("tp/syscalls/sys_enter_prctl")
+int handler_tp(void *ctx)
+{
+	return handle_comm(false);
+}
+
 SEC("raw_tp/task_rename")
 int dispatcher_a(void *ctx)
 {
@@ -213,6 +239,17 @@ int BPF_PROG(dispatcher_b, struct pt_regs *regs, long id)
 	return 0;
 }
 
+SEC("tp/syscalls/sys_enter_prctl")
+int on_tp_dual(void *ctx)
+{
+	__u32 tid = (__u32)bpf_get_current_pid_tgid();
+	__u64 cookie = bpf_get_attach_cookie(ctx);
+
+	if ((tid == tid_a && cookie != 1) || (tid == tid_b && cookie != 2))
+		return 0;
+	return handle_comm(false);
+}
+
 SEC("raw_tp/bpf_trace_printk")
 int recur_raw_tp(void *ctx)
 {

-- 
2.43.0

Re: [PATCH bpf-next v2 1/5] bpf: Avoid per-CPU private stack in preemptible tracing progs

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]

> bpf: Avoid per-CPU private stack in preemptible tracing progs

TL;DR: Selftest nits only: renamed "No private stack" tests in
verifier_private_stack.c keep old "private_stack" function names, and one
comment still describes the old mixed case.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_private_stack.c b/tools/testing/selftests/bpf/progs/verifier_private_stack.c
> --- a/tools/testing/selftests/bpf/progs/verifier_private_stack.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_private_stack.c

[ ... ]

>  SEC("raw_tp")
> -__description("Private stack, run, exception in main prog")
> +__description("No private stack, run, exception in main prog")
>  __success __retval(1)
>  __arch_x86_64
> -__jited("	movq	%rdi, -0x200(%r9)")
> -__jited("	movl	$0x1, %edi")
> -__jited("	pushq	%r9")
> -__jited("...")
> -__jited("	callq")
> +__jited("	subq	$0x200, %rsp")
>  __naked void private_stack_run_exception_main_prog(void)
>  {

this isn't a bug, but would it be clearer to rename
private_stack_run_exception_main_prog() to match its new "No private stack"
description?

The same applies to private_stack_run_exception_sub_prog() and
private_stack_subprog_throw().  Elsewhere in this file the function name
follows the description, for example no_private_stack_kprobe(), and
private_stack_returns() was renamed to big_stack_returns() in
verifier_unwind.c when its description changed.

[ ... ]

>  /* Only the frame of the main prog counts, and it has nothing pushed at the call */
>  SEC("raw_tp")
> -__description("Private stack, run, exception in subprog, no private stack in main prog")
> +__description("No private stack, run, exception in subprog with a large stack")
>  __success __retval(3)

Also, the comment above no_private_stack_run_exception_sub_prog() still
describes the old mixed case, a main prog with nothing pushed at the call.
Now that neither frame has a private stack, should it be dropped or reworded?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43092697553