[PATCH bpf 0/2] LoongArch: BPF: Fix tail call count handling

[PATCH bpf 0/2] LoongArch: BPF: Fix tail call count handling

[George Guo]

Two independent fixes for the LoongArch BPF JIT's tail call count (TCC)
handling, both found while enabling the arena and tailcall selftests on
LoongArch.

Patch 1 fixes memory corruption / a kernel panic. For an arena program the
extra REG_ARENA slot in the prologue shifts the callee-saved area down by
one word, but BPF_TAIL_CALL_CNT_PTR_STACK_OFF() did not account for it, so
on every tail call or bpf2bpf call the JIT loaded the counter *value* and
dereferenced it as the TCC pointer.

Patch 2 fixes an off-by-one: emit_bpf_tail_call() increments the tail call
count before the NULL-slot check, so jumping through an empty prog_array
slot is charged against MAX_TAIL_CALL_CNT and programs reach the limit one
call early.

Both carry Fixes: tags and Cc: stable. They touch only emit_bpf_tail_call()
and the new offset helper; there is no functional dependency between them.

George Guo (2):
  LoongArch: BPF: Fix tail call count pointer offset for arena programs
  LoongArch: BPF: Don't charge an empty prog_array slot to the tail call
    count

 arch/loongarch/net/bpf_jit.c | 34 +++++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)

-- 
2.25.1

[PATCH bpf 1/2] LoongArch: BPF: Fix tail call count pointer offset for arena programs

[George Guo]

From: George Guo <guodongtai@kylinos.cn>

The tail call count (TCC) and its pointer occupy the two deepest slots of
the callee-saved area set up by build_prologue(). An arena program reserves
one extra word for REG_ARENA (arena_vm_start) right above them:

    ra fp s0 s1 s2 s3 s4 s5      <- 8 words
    [ REG_ARENA ]                <- only if ctx->arena_vm_start
    tail_call_cnt
    tail_call_cnt_ptr            <- loaded on tail call / bpf2bpf call

BPF_TAIL_CALL_CNT_PTR_STACK_OFF() hardcodes the pointer at
round_up(stack, 16) - 80, which is only correct when REG_ARENA is absent.
For an arena program the extra word shifts every slot below it down by 8
bytes, so the macro resolves to the tail_call_cnt slot (the counter value)
instead of tail_call_cnt_ptr. The JIT then loads that small integer and
dereferences it as the TCC pointer, corrupting memory or panicking the
kernel whenever an arena program performs a tail call or a bpf2bpf call.

Replace the macro with a helper that accounts for the REG_ARENA slot,
mirroring the reservation logic in build_prologue().

Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")
Cc: stable@vger.kernel.org
Signed-off-by: George Guo <guodongtai@kylinos.cn>
---
 arch/loongarch/net/bpf_jit.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 24913dc7f4e8..f705de099f23 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -18,7 +18,23 @@
 
 #define REG_TCC		LOONGARCH_GPR_A6
 #define REG_ARENA	LOONGARCH_GPR_S6 /* For storing arena_vm_start */
-#define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack) (round_up(stack, 16) - 80)
+
+static int tail_call_cnt_ptr_stack_off(struct jit_ctx *ctx)
+{
+	/* Ten words are pushed below the BPF stack: ra, fp, s0-s5, and the
+	 * tail call count plus its pointer, which occupy the two deepest
+	 * slots of the callee-saved area.
+	 */
+	int offset = sizeof(long) * 10;
+
+	/* An arena program reserves one extra word above them (REG_ARENA),
+	 * which pushes the tail call count pointer down by one slot.
+	 */
+	if (ctx->arena_vm_start)
+		offset += sizeof(long);
+
+	return round_up(ctx->stack_size, 16) - offset;
+}
 
 static const int regmap[] = {
 	/* return value from in-kernel function, and exit value for eBPF program */
@@ -278,7 +294,7 @@ bool bpf_jit_supports_far_kfunc_call(void)
 static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 {
 	int off, tc_ninsn = 0;
-	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
+	int tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
 	u8 a1 = LOONGARCH_GPR_A1;
 	u8 a2 = LOONGARCH_GPR_A2;
 	u8 t1 = LOONGARCH_GPR_T1;
@@ -1153,7 +1169,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
 			return ret;
 
 		if (insn->src_reg == BPF_PSEUDO_CALL) {
-			tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
+			tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
 			emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, tcc_ptr_off);
 		}
 
-- 
2.25.1

[PATCH bpf 2/2] LoongArch: BPF: Don't charge an empty prog_array slot to the tail call count

[George Guo]

From: George Guo <guodongtai@kylinos.cn>

emit_bpf_tail_call() bumped the tail call count and stored it back to
*tcc_ptr before loading array->ptrs[index] and testing it for NULL.  A
tail call that targets an empty slot therefore consumed one unit of the
tail call budget even though control never transferred.

The interpreter increments tail_call_cnt only after the prog pointer is
found to be non-NULL (kernel/bpf/core.c, BPF_TAIL_CALL), so a fall-through
to an empty slot leaves the count untouched.  The JIT must do the same.

This is visible with selftests/bpf tailcalls/tailcall_3, whose entry prog
tail-calls an empty slot before the real target: the observed count is 32
instead of the expected 33.

Defer the store of the bumped count until after the NULL check.  The limit
comparison is unchanged: t3 = *tcc_ptr + 1, and "t3 > MAX_TAIL_CALL_CNT" is
equivalent to "*tcc_ptr >= MAX_TAIL_CALL_CNT".

The check-before-NULL ordering dates back to the original JIT; commit
c0fcc955ff82 ("LoongArch: BPF: Fix the tailcall hierarchy") reworked the
counter into the *tcc_ptr form but preserved the same ordering.

Fixes: 5dc615520c4d ("LoongArch: Add BPF JIT support")
Cc: stable@vger.kernel.org
Signed-off-by: George Guo <guodongtai@kylinos.cn>
---
 arch/loongarch/net/bpf_jit.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index f705de099f23..f2aa0b7f65ad 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -323,13 +323,18 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 		goto toofar;
 
 	/*
-	 * if ((*tcc_ptr)++ >= MAX_TAIL_CALL_CNT)
+	 * if (*tcc_ptr + 1 > MAX_TAIL_CALL_CNT)
 	 *      goto out;
+	 *
+	 * Compute the bumped count but do not write it back yet: the
+	 * interpreter increments tail_call_cnt only after the prog pointer is
+	 * found to be non-NULL, so a tail call to an empty slot must not
+	 * consume the tail call budget.  The store is deferred until the call
+	 * is known to be taken (below).
 	 */
 	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, tcc_ptr_off);
 	emit_insn(ctx, ldd, t3, REG_TCC, 0);
 	emit_insn(ctx, addid, t3, t3, 1);
-	emit_insn(ctx, std, t3, REG_TCC, 0);
 	emit_insn(ctx, addid, t2, LOONGARCH_GPR_ZERO, MAX_TAIL_CALL_CNT);
 	if (emit_tailcall_jmp(ctx, BPF_JSGT, t3, t2, jmp_offset) < 0)
 		goto toofar;
@@ -346,6 +351,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 	if (emit_tailcall_jmp(ctx, BPF_JEQ, t2, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
 		goto toofar;
 
+	/* (*tcc_ptr)++; the tail call is taken, so commit the bumped count */
+	emit_insn(ctx, std, t3, REG_TCC, 0);
+
 	/* goto *(prog->bpf_func + 4); */
 	off = offsetof(struct bpf_prog, bpf_func);
 	emit_insn(ctx, ldd, t3, t2, off);
-- 
2.25.1

[PATCH bpf v2] LoongArch: BPF: Fix tail call count pointer offset for arena programs

[George Guo]

From: George Guo <guodongtai@kylinos.cn>

The tail call count (TCC) and its pointer occupy the two deepest slots of
the callee-saved area set up by build_prologue(). An arena program reserves
one extra word for REG_ARENA (arena_vm_start) right above them:

    ra fp s0 s1 s2 s3 s4 s5      <- 8 words
    [ REG_ARENA ]                <- only if ctx->arena_vm_start
    tail_call_cnt
    tail_call_cnt_ptr            <- loaded on tail call / bpf2bpf call

BPF_TAIL_CALL_CNT_PTR_STACK_OFF() hardcodes the pointer at
round_up(stack, 16) - 80, which is only correct when REG_ARENA is absent.
For an arena program the extra word shifts every slot below it down by 8
bytes, so the macro resolves to the tail_call_cnt slot (the counter value)
instead of tail_call_cnt_ptr. The JIT then loads the counter value and
dereferences it as the TCC pointer, corrupting memory or panicking the
kernel whenever an arena program performs a tail call or a bpf2bpf call.

Replace the macro with a helper that accounts for the REG_ARENA slot,
mirroring the reservation logic in build_prologue().

Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")
Cc: stable@vger.kernel.org
Signed-off-by: George Guo <guodongtai@kylinos.cn>
---
v2:
 - Dropped the second patch ("Don't charge an empty prog_array slot to
   the tail call count"); that off-by-one was fixed independently by
   commit 0379d10f09bc ("LoongArch: BPF: Fix off-by-one error in tail
   call"), now in 7.2-rc1. The arena tail call count pointer offset bug
   addressed here is independent and still unfixed.
 - No code change; reworded the commit message and rebased on 7.2-rc1.
 - v1: https://lore.kernel.org/all/20260625083212.277417-1-dongtai.guo@linux.dev

 arch/loongarch/net/bpf_jit.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index ad7e28375aa9..5e34e9e3f508 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -25,7 +25,23 @@
 
 #define REG_TCC		LOONGARCH_GPR_A6
 #define REG_ARENA	LOONGARCH_GPR_S6 /* For storing arena_vm_start */
-#define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack) (round_up(stack, 16) - 80)
+
+static int tail_call_cnt_ptr_stack_off(struct jit_ctx *ctx)
+{
+	/* Ten words are pushed below the BPF stack: ra, fp, s0-s5, and the
+	 * tail call count plus its pointer, which occupy the two deepest
+	 * slots of the callee-saved area.
+	 */
+	int offset = sizeof(long) * 10;
+
+	/* An arena program reserves one extra word above them (REG_ARENA),
+	 * which pushes the tail call count pointer down by one slot.
+	 */
+	if (ctx->arena_vm_start)
+		offset += sizeof(long);
+
+	return round_up(ctx->stack_size, 16) - offset;
+}
 
 static const int regmap[] = {
 	/* return value from in-kernel function, and exit value for eBPF program */
@@ -291,7 +307,7 @@ bool bpf_jit_supports_far_kfunc_call(void)
 static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 {
 	int off, tc_ninsn = 0;
-	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
+	int tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
 	u8 a1 = LOONGARCH_GPR_A1;
 	u8 a2 = LOONGARCH_GPR_A2;
 	u8 t1 = LOONGARCH_GPR_T1;
@@ -1181,7 +1197,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
 			return ret;
 
 		if (insn->src_reg == BPF_PSEUDO_CALL) {
-			tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
+			tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
 			emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, tcc_ptr_off);
 		}
 
-- 
2.25.1

Re: [PATCH bpf v2] LoongArch: BPF: Fix tail call count pointer offset for arena programs

[Tiezhu Yang]

On 2026/6/29 下午4:55, George Guo wrote:
> From: George Guo <guodongtai@kylinos.cn>
> 
> The tail call count (TCC) and its pointer occupy the two deepest slots of
> the callee-saved area set up by build_prologue(). An arena program reserves
> one extra word for REG_ARENA (arena_vm_start) right above them:
> 
>      ra fp s0 s1 s2 s3 s4 s5      <- 8 words
>      [ REG_ARENA ]                <- only if ctx->arena_vm_start
>      tail_call_cnt
>      tail_call_cnt_ptr            <- loaded on tail call / bpf2bpf call
> 
> BPF_TAIL_CALL_CNT_PTR_STACK_OFF() hardcodes the pointer at
> round_up(stack, 16) - 80, which is only correct when REG_ARENA is absent.
> For an arena program the extra word shifts every slot below it down by 8
> bytes, so the macro resolves to the tail_call_cnt slot (the counter value)
> instead of tail_call_cnt_ptr. The JIT then loads the counter value and
> dereferences it as the TCC pointer, corrupting memory or panicking the
> kernel whenever an arena program performs a tail call or a bpf2bpf call.
> 
> Replace the macro with a helper that accounts for the REG_ARENA slot,
> mirroring the reservation logic in build_prologue().
> 
> Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")
> Cc: stable@vger.kernel.org
> Signed-off-by: George Guo <guodongtai@kylinos.cn>
> ---
> v2:
>   - Dropped the second patch ("Don't charge an empty prog_array slot to
>     the tail call count"); that off-by-one was fixed independently by
>     commit 0379d10f09bc ("LoongArch: BPF: Fix off-by-one error in tail
>     call"), now in 7.2-rc1. The arena tail call count pointer offset bug
>     addressed here is independent and still unfixed.
>   - No code change; reworded the commit message and rebased on 7.2-rc1.
>   - v1: https://lore.kernel.org/all/20260625083212.277417-1-dongtai.guo@linux.dev
> 
>   arch/loongarch/net/bpf_jit.c | 22 +++++++++++++++++++---
>   1 file changed, 19 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index ad7e28375aa9..5e34e9e3f508 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -25,7 +25,23 @@
>   
>   #define REG_TCC		LOONGARCH_GPR_A6
>   #define REG_ARENA	LOONGARCH_GPR_S6 /* For storing arena_vm_start */
> -#define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack) (round_up(stack, 16) - 80)
> +
> +static int tail_call_cnt_ptr_stack_off(struct jit_ctx *ctx)
> +{
> +	/* Ten words are pushed below the BPF stack: ra, fp, s0-s5, and the
> +	 * tail call count plus its pointer, which occupy the two deepest
> +	 * slots of the callee-saved area.
> +	 */
> +	int offset = sizeof(long) * 10;
> +
> +	/* An arena program reserves one extra word above them (REG_ARENA),
> +	 * which pushes the tail call count pointer down by one slot.
> +	 */
> +	if (ctx->arena_vm_start)
> +		offset += sizeof(long);
> +
> +	return round_up(ctx->stack_size, 16) - offset;
> +}
>   
>   static const int regmap[] = {
>   	/* return value from in-kernel function, and exit value for eBPF program */
> @@ -291,7 +307,7 @@ bool bpf_jit_supports_far_kfunc_call(void)
>   static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
>   {
>   	int off, tc_ninsn = 0;
> -	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
> +	int tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
>   	u8 a1 = LOONGARCH_GPR_A1;
>   	u8 a2 = LOONGARCH_GPR_A2;
>   	u8 t1 = LOONGARCH_GPR_T1;
> @@ -1181,7 +1197,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
>   			return ret;
>   
>   		if (insn->src_reg == BPF_PSEUDO_CALL) {
> -			tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
> +			tcc_ptr_off = tail_call_cnt_ptr_stack_off(ctx);
>   			emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, tcc_ptr_off);
>   		}
>   
> 

Please see the discussion in the other thread:

https://lore.kernel.org/loongarch/d7f2c8d3-4a46-e978-5cce-f59b84e632f0@loongson.cn/

Thanks,
Tiezhu