[PATCH v2 0/4] Optimize BPF tailcall for LoongArch

[PATCH v2 0/4] Optimize BPF tailcall for LoongArch

[Tiezhu Yang]

Based on 7.2-rc1, tested on a Loongson-3A6000 physical machine.

This version addresses the review comments of AI, changes are isolated
to patch 3/4, the rest of the series remains unchanged.

v1 -> v2:
- [Patch 3/4] Use ctx->image to inject a safe dummy offset during
  Pass 1, safely bypassing the 16-bit signed immediate range check.
- [Patch 3/4] Update the commit message to reflect the new code.

Here are the test results:

  $ sudo modprobe test_bpf test_suite=test_tail_calls
  $ dmesg -t | tail -1
  test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]

  $ sudo ./test_progs -t tailcalls | tail -1
  Summary: 1/37 PASSED, 0 SKIPPED, 0 FAILED

Tiezhu Yang (4):
  LoongArch: BPF: Optimize redundant TCC loads in epilogue
  LoongArch: BPF: Move arena register slot below TCC context
  LoongArch: BPF: Refactor jump offset calculation in tail call
  LoongArch: BPF: Implement branchless conditional move for TCC

 arch/loongarch/include/asm/inst.h |   6 ++
 arch/loongarch/net/bpf_jit.c      | 112 +++++++++++++++---------------
 2 files changed, 63 insertions(+), 55 deletions(-)

-- 
2.42.0

[PATCH v2 1/4] LoongArch: BPF: Optimize redundant TCC loads in epilogue

[Tiezhu Yang]

The legacy epilogue implementation pops the tail call counter (TCC)
context via a redundant double-load pattern. It first decrements the
load_offset by 2 slots to fetch 'tcc_ptr', and then immediately bumps
it back up by 1 slot to load the original 'tcc' value into REG_TCC,
meaninglessly overwriting the register.

Optimize this sequence by adjusting the load_offset by only 1 slot.
This aligns the offset directly with the higher stack slot containing
the entry TCC counter (or caller state), allowing us to restore the
REG_TCC register safely with a single load.

This removes one redundant instruction from the epilogue hot path,
improves code readability, and ensures the correct TCC register
context is handed back cleanly upon normal return.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/net/bpf_jit.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index ad7e28375aa9..719dc4ab7a98 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -246,14 +246,8 @@ static void __build_epilogue(struct jit_ctx *ctx, bool is_tail_call)
 		emit_insn(ctx, ldd, REG_ARENA, LOONGARCH_GPR_SP, load_offset);
 	}
 
-	/*
-	 * When push into the stack, follow the order of tcc then tcc_ptr.
-	 * When pop from the stack, first pop tcc_ptr then followed by tcc.
-	 */
-	load_offset -= 2 * sizeof(long);
-	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, load_offset);
-
-	load_offset += sizeof(long);
+	/* Only restore the TCC state into REG_TCC from the higher slot */
+	load_offset -= sizeof(long);
 	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, load_offset);
 
 	emit_insn(ctx, addid, LOONGARCH_GPR_SP, LOONGARCH_GPR_SP, stack_adjust);
-- 
2.42.0

[PATCH v2 2/4] LoongArch: BPF: Move arena register slot below TCC context

[Tiezhu Yang]

Currently, the stack layout places the optional arena register slot above
the tail call counter context. When arena_vm_start is dynamically enabled,
it shifts the relative offset of the tcc_ptr slot within the stack frame,
causing hardcoded tracking macros to mismatch and potentially leading to
memory misalignment or corruption.

To fix this, move the arena register save and restore sequences below the
tail call counter context slots in both build_prologue() and epilogue. It
needs to update __build_epilogue() to insert a proper offset decrement to
safely skip the unneeded tcc_ptr reading block while accurately aligning
with the relocated arena slot at the very bottom.

With this patch, the tcc_ptr slot is always positioned at a fixed distance
directly underneath the base callee-saved registers that is independent of
whether the arena features are on.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/net/bpf_jit.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 719dc4ab7a98..95e3b372a44f 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -124,6 +124,9 @@ static void prepare_bpf_tail_call_cnt(struct jit_ctx *ctx, int *store_offset)
  *                            |           tcc           |
  *                            +-------------------------+
  *                            |           tcc_ptr       |
+ *                            +-------------------------+
+ *                            |           arena         |
+ *                            |         (optional)      |
  *                            +-------------------------+ <--BPF_REG_FP
  *                            |  prog->aux->stack_depth |
  *                            |        (optional)       |
@@ -145,7 +148,7 @@ static void build_prologue(struct jit_ctx *ctx)
 	stack_adjust += sizeof(long) * 2;
 
 	if (ctx->arena_vm_start)
-		stack_adjust += 8;
+		stack_adjust += sizeof(long);
 
 	stack_adjust = round_up(stack_adjust, 16);
 	stack_adjust += bpf_stack_adjust;
@@ -194,13 +197,13 @@ static void build_prologue(struct jit_ctx *ctx)
 	store_offset -= sizeof(long);
 	emit_insn(ctx, std, LOONGARCH_GPR_S5, LOONGARCH_GPR_SP, store_offset);
 
+	prepare_bpf_tail_call_cnt(ctx, &store_offset);
+
 	if (ctx->arena_vm_start) {
 		store_offset -= sizeof(long);
 		emit_insn(ctx, std, REG_ARENA, LOONGARCH_GPR_SP, store_offset);
 	}
 
-	prepare_bpf_tail_call_cnt(ctx, &store_offset);
-
 	emit_insn(ctx, addid, LOONGARCH_GPR_FP, LOONGARCH_GPR_SP, stack_adjust);
 
 	if (bpf_stack_adjust)
@@ -241,15 +244,18 @@ static void __build_epilogue(struct jit_ctx *ctx, bool is_tail_call)
 	load_offset -= sizeof(long);
 	emit_insn(ctx, ldd, LOONGARCH_GPR_S5, LOONGARCH_GPR_SP, load_offset);
 
+	/* Only restore the TCC state into REG_TCC from the higher slot */
+	load_offset -= sizeof(long);
+	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, load_offset);
+
+	/* Skip the unused local 'tcc_ptr' slot to align with arena */
+	load_offset -= sizeof(long);
+
 	if (ctx->arena_vm_start) {
 		load_offset -= sizeof(long);
 		emit_insn(ctx, ldd, REG_ARENA, LOONGARCH_GPR_SP, load_offset);
 	}
 
-	/* Only restore the TCC state into REG_TCC from the higher slot */
-	load_offset -= sizeof(long);
-	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, load_offset);
-
 	emit_insn(ctx, addid, LOONGARCH_GPR_SP, LOONGARCH_GPR_SP, stack_adjust);
 
 	if (!is_tail_call) {
-- 
2.42.0

[PATCH v2 3/4] LoongArch: BPF: Refactor jump offset calculation in tail call

[Tiezhu Yang]

In emit_bpf_tail_call(), the macro-based jmp_offset calculation relies
on a fragile prediction equation: "tc_ninsn - (ctx->idx - idx0)". This
calculation model utilizes tc_ninsn from the previous pass to estimate
the remaining stride of the active tail call code block.

Given that the macro relies on stale layout data from a prior pass to
compute the active relative stride, the conditional branches end up
encoding wrong immediate offsets under extra JIT passes. Consequently,
the execution flow leaps into misaligned instruction boundaries,
triggering soft lockups or early loop collapse.

To fix this issue, drop the pass-dependent macro and the intermediate
variable entirely. Instead, explicitly calculate "jmp_offset" using an
absolute target reference anchor: "ctx->offset[insn + 1] - ctx->idx".

However, during the initial size-estimation pass, the target address
"ctx->offset[insn + 1]" is not yet populated and will evaluate to 0,
resulting in "jmp_offset = -ctx->idx". For large eBPF programs, this
inflated negative offset would overflow the 16-bit signed immediate
limit of LoongArch conditional branch instructions, causing the range
validation inside emit_tailcall_jmp() to abort the JIT compilation
prematurely.

To bypass this initialization bottleneck cleanly without changing any
function signatures, leverage the "ctx->image" pointer to safely detect
the initial pass. Since conditional branch instructions on LoongArch
maintain a fixed 4-byte width regardless of the target displacement,
injecting a safe dummy offset (0) when "ctx->image == NULL" preserves
accurate size estimation while preventing 16-bit range check failures.

Within this new deterministic arithmetic model:
1) "ctx->offset[insn + 1]" maps directly to the absolute instruction
   index of the immediate next eBPF business instruction, serving as
   a rock-solid target label.
2) "ctx->idx" tracks the real-time instruction index of the conditional
   branch currently being emitted.

This guarantees the generated immediate offsets always land precisely
on the next instruction boundary under all multi-pass compilation paths
and all code size dimensions.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/net/bpf_jit.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 95e3b372a44f..45364a15ae5e 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -290,17 +290,13 @@ bool bpf_jit_supports_far_kfunc_call(void)
 
 static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 {
-	int off, tc_ninsn = 0;
+	int off, jmp_offset;
 	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(ctx->stack_size);
 	u8 a1 = LOONGARCH_GPR_A1;
 	u8 a2 = LOONGARCH_GPR_A2;
 	u8 t1 = LOONGARCH_GPR_T1;
 	u8 t2 = LOONGARCH_GPR_T2;
 	u8 t3 = LOONGARCH_GPR_T3;
-	const int idx0 = ctx->idx;
-
-#define cur_offset (ctx->idx - idx0)
-#define jmp_offset (tc_ninsn - (cur_offset))
 
 	/*
 	 * a0: &ctx
@@ -310,12 +306,12 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 	 * if (index >= array->map.max_entries)
 	 *	 goto out;
 	 */
-	tc_ninsn = insn ? ctx->offset[insn+1] - ctx->offset[insn] : ctx->offset[0];
 	emit_zext_32(ctx, a2, true);
 
 	off = offsetof(struct bpf_array, map.max_entries);
 	emit_insn(ctx, ldwu, t1, a1, off);
 	/* bgeu $a2, $t1, jmp_offset */
+	jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
 	if (emit_tailcall_jmp(ctx, BPF_JGE, a2, t1, jmp_offset) < 0)
 		goto toofar;
 
@@ -326,6 +322,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 	emit_insn(ctx, ldd, REG_TCC, LOONGARCH_GPR_SP, tcc_ptr_off);
 	emit_insn(ctx, ldd, t3, REG_TCC, 0);
 	emit_insn(ctx, addid, t2, LOONGARCH_GPR_ZERO, MAX_TAIL_CALL_CNT);
+	jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
 	if (emit_tailcall_jmp(ctx, BPF_JSGE, t3, t2, jmp_offset) < 0)
 		goto toofar;
 
@@ -340,6 +337,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 	off = offsetof(struct bpf_array, ptrs);
 	emit_insn(ctx, ldd, t2, t2, off);
 	/* beq $t2, $zero, jmp_offset */
+	jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
 	if (emit_tailcall_jmp(ctx, BPF_JEQ, t2, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
 		goto toofar;
 
@@ -355,8 +353,6 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
 toofar:
 	pr_info_once("tail_call: jump too far\n");
 	return -1;
-#undef cur_offset
-#undef jmp_offset
 }
 
 static void emit_store_stack_imm64(struct jit_ctx *ctx, int reg, int stack_off, u64 imm64)
-- 
2.42.0

[PATCH v2 4/4] LoongArch: BPF: Implement branchless conditional move for TCC

[Tiezhu Yang]

The current implementation handles combined bpf2bpf and tail calls by
checking at runtime whether REG_TCC holds a count or an address via a
conditional jump (BPF_JGT against 33). This adds non-negligible branch
prediction overhead in the hot path of tail call execution.

To eliminate this runtime conditional branch in prologue, refactor the
logic of prepare_bpf_tail_call_cnt() with a branchless conditional move
mechanism using unsigned comparison and mask instructions.

When the program is a main program (is_main_prog), the branchless logic
classifies and blends the incoming REG_TCC based on its physical content
according to three possible entrance statuses:

1) Initial Entry (Scalar): Enter via a standard event, where REG_TCC is
   initialized to a pure scalar 0.
2) Inherited Entry (Scalar): Enter via a flat main-to-main tail call,
   where REG_TCC carries an accumulated scalar count (1 to 33) reloaded
   from the prior stack frame.
3) Inherited Entry (Pointer): Enter via a tail call from a subprogram,
   where REG_TCC inherits and carries a massive kernel pointer address
   reloaded from the prior stack frame.

To handle these cases seamlessly without branching:
- sltui identifies the type of REG_TCC. For cases 1 and 2 (scalar count),
  it sets the temporary register T8 to 1. For case 3 (kernel pointer), it
  sets T8 to 0.
- maskeqz handles the local stack pointer (T7). If T8 == 1, it keeps T7
  intact. If T8 == 0, it clears T7 to 0.
- masknez handles the incoming REG_TCC to prevent register pollution.
  If T8 == 1, it clears REG_TCC to 0 (erasing 1~33 scalars). If T8 == 0,
  it keeps the massive kernel pointer intact.
- or combines the results. The scalar cases yield a clean local stack
  pointer by evaluating REG_TCC (0) | T7, while the pointer case leaves
  the inherited global pointer intact by evaluating REG_TCC | T7 (0).

For subprograms, the logic directly backs up the verified TCC pointer
inherited via REG_TCC into its own local "tcc" slot, ensuring the global
counter reference remains safely propagated across function frames.

Finally, the finalized REG_TCC pointer is unconditionally backed up into
the adjacent "tcc_ptr" slot, completing the expected dual-slot stack
layout identically across all entry paths.

This optimization refactors the inner logic of the helper function,
unifies the offset decrement at the function entry, and uses safe
internal temporary registers (T7 and T8) to prevent any JIT register
conflicts, completely removing all runtime branching from the prologue
hot path.

When exposed to complex bpf2bpf + tailcall interleaved control flows,
where inputs shuffle dynamically between a scalar count (0 to 33) and
a massive kernel pointer address, clear improvements are demonstrated.
Furthermore, the optimization significantly eliminated uncertainty in
the execution path, making the performance more stable and predictable.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/include/asm/inst.h |  6 +++
 arch/loongarch/net/bpf_jit.c      | 74 +++++++++++++++++--------------
 2 files changed, 46 insertions(+), 34 deletions(-)

diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
index 76b723590023..19feae166e1c 100644
--- a/arch/loongarch/include/asm/inst.h
+++ b/arch/loongarch/include/asm/inst.h
@@ -97,6 +97,7 @@ enum reg2i6_op {
 };
 
 enum reg2i12_op {
+	sltui_op	= 0x09,
 	addiw_op	= 0x0a,
 	addid_op	= 0x0b,
 	lu52id_op	= 0x0c,
@@ -153,6 +154,8 @@ enum reg3_op {
 	addd_op		= 0x21,
 	subw_op		= 0x22,
 	subd_op		= 0x23,
+	maskeqz_op	= 0x26,
+	masknez_op	= 0x27,
 	nor_op		= 0x28,
 	and_op		= 0x29,
 	or_op		= 0x2a,
@@ -644,6 +647,7 @@ static inline void emit_##NAME(union loongarch_instruction *insn,	\
 	insn->reg2i12_format.rj = rj;					\
 }
 
+DEF_EMIT_REG2I12_FORMAT(sltui, sltui_op)
 DEF_EMIT_REG2I12_FORMAT(addiw, addiw_op)
 DEF_EMIT_REG2I12_FORMAT(addid, addid_op)
 DEF_EMIT_REG2I12_FORMAT(lu52id, lu52id_op)
@@ -749,6 +753,8 @@ DEF_EMIT_REG3_FORMAT(divd, divd_op)
 DEF_EMIT_REG3_FORMAT(modd, modd_op)
 DEF_EMIT_REG3_FORMAT(divdu, divdu_op)
 DEF_EMIT_REG3_FORMAT(moddu, moddu_op)
+DEF_EMIT_REG3_FORMAT(maskeqz, maskeqz_op)
+DEF_EMIT_REG3_FORMAT(masknez, masknez_op)
 DEF_EMIT_REG3_FORMAT(and, and_op)
 DEF_EMIT_REG3_FORMAT(or, or_op)
 DEF_EMIT_REG3_FORMAT(xor, xor_op)
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 45364a15ae5e..b56ae7ed3e06 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -52,50 +52,56 @@ static void prepare_bpf_tail_call_cnt(struct jit_ctx *ctx, int *store_offset)
 	const struct bpf_prog *prog = ctx->prog;
 	const bool is_main_prog = !bpf_is_subprog(prog);
 
+	*store_offset -= sizeof(long);
 	if (is_main_prog) {
-		/*
-		 * LOONGARCH_GPR_T3 = MAX_TAIL_CALL_CNT
-		 * if (REG_TCC > T3 )
-		 *	std REG_TCC -> LOONGARCH_GPR_SP + store_offset
-		 * else
-		 *	std REG_TCC -> LOONGARCH_GPR_SP + store_offset
-		 *	REG_TCC = LOONGARCH_GPR_SP + store_offset
-		 *
-		 * std REG_TCC -> LOONGARCH_GPR_SP + store_offset
-		 *
-		 * The purpose of this code is to first push the TCC into stack,
-		 * and then push the address of TCC into stack.
-		 * In cases where bpf2bpf and tailcall are used in combination,
-		 * the value in REG_TCC may be a count or an address,
-		 * these two cases need to be judged and handled separately.
-		 */
-		emit_insn(ctx, addid, LOONGARCH_GPR_T3, LOONGARCH_GPR_ZERO, MAX_TAIL_CALL_CNT);
-		*store_offset -= sizeof(long);
-
-		emit_cond_jmp(ctx, BPF_JGT, REG_TCC, LOONGARCH_GPR_T3, 4);
-
-		/*
-		 * If REG_TCC < MAX_TAIL_CALL_CNT, the value in REG_TCC is a count,
-		 * push tcc into stack
-		 */
+		/* Save entrance TCC state (scalar count or kernel pointer) to local 'tcc' slot */
 		emit_insn(ctx, std, REG_TCC, LOONGARCH_GPR_SP, *store_offset);
 
-		/* Push the address of TCC into the REG_TCC */
-		emit_insn(ctx, addid, REG_TCC, LOONGARCH_GPR_SP, *store_offset);
-
-		emit_uncond_jmp(ctx, 2);
+		/* Compute the absolute pointer to the local 'tcc' slot */
+		emit_insn(ctx, addid, LOONGARCH_GPR_T7, LOONGARCH_GPR_SP, *store_offset);
 
 		/*
-		 * If REG_TCC > MAX_TAIL_CALL_CNT, the value in REG_TCC is an address,
-		 * push tcc_ptr into stack
+		 * Branchless classification and blending:
+		 *
+		 * In combined bpf2bpf and tailcall scenarios, REG_TCC can carry
+		 * either a scalar count (0 to 33) or an inherited kernel pointer address.
+		 *
+		 * Entrance status for a main program based on REG_TCC physical content:
+		 * 1) Initial Entry (Scalar): Enter via a standard event, where REG_TCC
+		 *    is initialized to a pure scalar 0.
+		 * 2) Inherited Entry (Scalar): Enter via a flat main-to-main tail call,
+		 *    where REG_TCC carries an accumulated scalar count (1 to 33) reloaded
+		 *    from the prior stack frame.
+		 * 3) Inherited Entry (Pointer): Enter via a tail call from a subprogram,
+		 *    where REG_TCC inherits and carries a massive kernel pointer address
+		 *    reloaded from the prior stack frame.
+		 *
+		 * sltui:   Identify the type of REG_TCC.
+		 *          If REG_TCC < MAX_TAIL_CALL_CNT + 1 (scalar 0~33), sets T8 = 1.
+		 *          If REG_TCC >= MAX_TAIL_CALL_CNT + 1 (kernel pointer), sets T8 = 0.
+		 *
+		 * maskeqz: Handle the local stack pointer (T7).
+		 *          If T8 == 1, keeps T7 intact.
+		 *          If T8 == 0, clears T7 to 0.
+		 *
+		 * masknez: Handle the incoming REG_TCC to prevent register pollution.
+		 *          If T8 == 1, clears REG_TCC to 0 (erasing 1~33 scalars).
+		 *          If T8 == 0, keeps the massive kernel pointer intact.
+		 *
+		 * or:      Combine the results.
+		 *          Scalar Case:  REG_TCC = 0 | T7       -> Clean local stack pointer.
+		 *          Pointer Case: REG_TCC = REG_TCC | 0  -> Clean inherited global pointer.
 		 */
-		emit_insn(ctx, std, REG_TCC, LOONGARCH_GPR_SP, *store_offset);
+		emit_insn(ctx, sltui, LOONGARCH_GPR_T8, REG_TCC, MAX_TAIL_CALL_CNT + 1);
+		emit_insn(ctx, maskeqz, LOONGARCH_GPR_T7, LOONGARCH_GPR_T7, LOONGARCH_GPR_T8);
+		emit_insn(ctx, masknez, REG_TCC, REG_TCC, LOONGARCH_GPR_T8);
+		emit_insn(ctx, or, REG_TCC, REG_TCC, LOONGARCH_GPR_T7);
 	} else {
-		*store_offset -= sizeof(long);
+		/* Subprograms: backup the verified TCC pointer inherited via REG_TCC */
 		emit_insn(ctx, std, REG_TCC, LOONGARCH_GPR_SP, *store_offset);
 	}
 
-	/* Push tcc_ptr into stack */
+	/* Store the finalized TCC pointer value securely into the local 'tcc_ptr' slot */
 	*store_offset -= sizeof(long);
 	emit_insn(ctx, std, REG_TCC, LOONGARCH_GPR_SP, *store_offset);
 }
-- 
2.42.0

Re: [PATCH v2 2/4] LoongArch: BPF: Move arena register slot below TCC context

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

> LoongArch: BPF: Move arena register slot below TCC context
>
> Currently, the stack layout places the optional arena register slot above
> the tail call counter context. When arena_vm_start is dynamically enabled,
> it shifts the relative offset of the tcc_ptr slot within the stack frame,
> causing hardcoded tracking macros to mismatch and potentially leading to
> memory misalignment or corruption.
> To fix this, move the arena register save and restore sequences below the
> tail call counter context slots in both build_prologue() and epilogue. It
> needs to update __build_epilogue() to insert a proper offset decrement to
> safely skip the unneeded tcc_ptr reading block while accurately aligning
> with the relocated arena slot at the very bottom.
> With this patch, the tcc_ptr slot is always positioned at a fixed distance
> directly underneath the base callee-saved registers that is independent of
> whether the arena features are on.
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>

This looks like a fix for a memory misalignment or corruption bug affecting
arena BPF programs.  Should it carry a Fixes: tag?

The buggy stack layout, where the arena register save/restore and the extra
stack reservation sit above the tail call counter context, appears to have
been introduced by:

  Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")

An independent fix for the same root cause ("LoongArch: BPF: Fix tail call
count pointer offset for arena programs") carries this same Fixes tag along
with Cc: stable, so a stable backport may be warranted here as well.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28766636714

Re: [PATCH v2 4/4] LoongArch: BPF: Implement branchless conditional move for TCC

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 6483 bytes --]

> The current implementation handles combined bpf2bpf and tail calls by
> checking at runtime whether REG_TCC holds a count or an address via a
> conditional jump (BPF_JGT against 33). This adds non-negligible branch
> prediction overhead in the hot path of tail call execution.
> To eliminate this runtime conditional branch in prologue, refactor the
> logic of prepare_bpf_tail_call_cnt() with a branchless conditional move
> mechanism using unsigned comparison and mask instructions.
> When the program is a main program (is_main_prog), the branchless logic
> classifies and blends the incoming REG_TCC based on its physical content
> according to three possible entrance statuses:
> 1) Initial Entry (Scalar): Enter via a standard event, where REG_TCC is
>    initialized to a pure scalar 0.
> 2) Inherited Entry (Scalar): Enter via a flat main-to-main tail call,
>    where REG_TCC carries an accumulated scalar count (1 to 33) reloaded
>    from the prior stack frame.
> 3) Inherited Entry (Pointer): Enter via a tail call from a subprogram,
>    where REG_TCC inherits and carries a massive kernel pointer address
>    reloaded from the prior stack frame.
> To handle these cases seamlessly without branching:
> - sltui identifies the type of REG_TCC. For cases 1 and 2 (scalar count),
>   it sets the temporary register T8 to 1. For case 3 (kernel pointer), it
>   sets T8 to 0.
> - maskeqz handles the local stack pointer (T7). If T8 == 1, it keeps T7
>   intact. If T8 == 0, it clears T7 to 0.
> - masknez handles the incoming REG_TCC to prevent register pollution.
>   If T8 == 1, it clears REG_TCC to 0 (erasing 1~33 scalars). If T8 == 0,
>   it keeps the massive kernel pointer intact.
> - or combines the results. The scalar cases yield a clean local stack
>   pointer by evaluating REG_TCC (0) | T7, while the pointer case leaves
>   the inherited global pointer intact by evaluating REG_TCC | T7 (0).
> For subprograms, the logic directly backs up the verified TCC pointer
> inherited via REG_TCC into its own local "tcc" slot, ensuring the global
> counter reference remains safely propagated across function frames.
> Finally, the finalized REG_TCC pointer is unconditionally backed up into
> the adjacent "tcc_ptr" slot, completing the expected dual-slot stack
> layout identically across all entry paths.
> This optimization refactors the inner logic of the helper function,
> unifies the offset decrement at the function entry, and uses safe
> internal temporary registers (T7 and T8) to prevent any JIT register
> conflicts, completely removing all runtime branching from the prologue
> hot path.
> When exposed to complex bpf2bpf + tailcall interleaved control flows,
> where inputs shuffle dynamically between a scalar count (0 to 33) and
> a massive kernel pointer address, clear improvements are demonstrated.
> Furthermore, the optimization significantly eliminated uncertainty in
> the execution path, making the performance more stable and predictable.
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>

This isn't a bug, but the closing lines:

> When exposed to complex bpf2bpf + tailcall interleaved control flows,
> where inputs shuffle dynamically between a scalar count (0 to 33) and
> a massive kernel pointer address, clear improvements are demonstrated.
> Furthermore, the optimization significantly eliminated uncertainty in
> the execution path, making the performance more stable and predictable.

state performance improvements without any number or benchmark behind them,
and the rationale (removing the runtime branch from the prologue hot path)
is already covered earlier in the message.  Could these closing claims be
backed by a concrete measurement, or trimmed?

> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index 8d6738103f59..b5419cafc984 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -52,50 +52,56 @@ static void prepare_bpf_tail_call_cnt(struct jit_ctx *ctx, int *store_offset)

[ ... ]

> +		/*
> +		 * Branchless classification and blending:
> +		 *
> +		 * In combined bpf2bpf and tailcall scenarios, REG_TCC can carry
> +		 * either a scalar count (0 to 33) or an inherited kernel pointer address.
> +		 *
> +		 * Entrance status for a main program based on REG_TCC physical content:
> +		 * 1) Initial Entry (Scalar): Enter via a standard event, where REG_TCC
> +		 *    is initialized to a pure scalar 0.
> +		 * 2) Inherited Entry (Scalar): Enter via a flat main-to-main tail call,
> +		 *    where REG_TCC carries an accumulated scalar count (1 to 33) reloaded
> +		 *    from the prior stack frame.
> +		 * 3) Inherited Entry (Pointer): Enter via a tail call from a subprogram,
> +		 *    where REG_TCC inherits and carries a massive kernel pointer address
> +		 *    reloaded from the prior stack frame.
> +		 *
> +		 * sltui:   Identify the type of REG_TCC.
> +		 *          If REG_TCC < MAX_TAIL_CALL_CNT + 1 (scalar 0~33), sets T8 = 1.
> +		 *          If REG_TCC >= MAX_TAIL_CALL_CNT + 1 (kernel pointer), sets T8 = 0.
> +		 *
> +		 * maskeqz: Handle the local stack pointer (T7).
> +		 *          If T8 == 1, keeps T7 intact.
> +		 *          If T8 == 0, clears T7 to 0.
> +		 *
> +		 * masknez: Handle the incoming REG_TCC to prevent register pollution.
> +		 *          If T8 == 1, clears REG_TCC to 0 (erasing 1~33 scalars).
> +		 *          If T8 == 0, keeps the massive kernel pointer intact.
> +		 *
> +		 * or:      Combine the results.
> +		 *          Scalar Case:  REG_TCC = 0 | T7       -> Clean local stack pointer.
> +		 *          Pointer Case: REG_TCC = REG_TCC | 0  -> Clean inherited global pointer.
>  		 */
> +		emit_insn(ctx, sltui, LOONGARCH_GPR_T8, REG_TCC, MAX_TAIL_CALL_CNT + 1);
> +		emit_insn(ctx, maskeqz, LOONGARCH_GPR_T7, LOONGARCH_GPR_T7, LOONGARCH_GPR_T8);
> +		emit_insn(ctx, masknez, REG_TCC, REG_TCC, LOONGARCH_GPR_T8);
> +		emit_insn(ctx, or, REG_TCC, REG_TCC, LOONGARCH_GPR_T7);

This isn't a bug, but the comment narrates each of the four emitted
instructions line by line, and the sltui/maskeqz/masknez/or walk-through
largely restates the entrance-status text already present in the commit
message as well as the code right below it.  Could this be condensed?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28766636714

Re: [PATCH v2 3/4] LoongArch: BPF: Refactor jump offset calculation in tail call

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3378 bytes --]

> In emit_bpf_tail_call(), the macro-based jmp_offset calculation relies
> on a fragile prediction equation: "tc_ninsn - (ctx->idx - idx0)". This
> calculation model utilizes tc_ninsn from the previous pass to estimate
> the remaining stride of the active tail call code block.
>
> Given that the macro relies on stale layout data from a prior pass to
> compute the active relative stride, the conditional branches end up
> encoding wrong immediate offsets under extra JIT passes. Consequently,
> the execution flow leaps into misaligned instruction boundaries,
> triggering soft lockups or early loop collapse.
>
> To fix this issue, drop the pass-dependent macro and the intermediate
> variable entirely. Instead, explicitly calculate "jmp_offset" using an
> absolute target reference anchor: "ctx->offset[insn + 1] - ctx->idx".
>
> However, during the initial size-estimation pass, the target address
> "ctx->offset[insn + 1]" is not yet populated and will evaluate to 0,
> resulting in "jmp_offset = -ctx->idx". For large eBPF programs, this
> inflated negative offset would overflow the 16-bit signed immediate
> limit of LoongArch conditional branch instructions, causing the range
> validation inside emit_tailcall_jmp() to abort the JIT compilation
> prematurely.
>
> To bypass this initialization bottleneck cleanly without changing any
> function signatures, leverage the "ctx->image" pointer to safely detect
> the initial pass. Since conditional branch instructions on LoongArch
> maintain a fixed 4-byte width regardless of the target displacement,
> injecting a safe dummy offset (0) when "ctx->image == NULL" preserves
> accurate size estimation while preventing 16-bit range check failures.
>
> Within this new deterministic arithmetic model:
> 1) "ctx->offset[insn + 1]" maps directly to the absolute instruction
>    index of the immediate next eBPF business instruction, serving as
>    a rock-solid target label.
> 2) "ctx->idx" tracks the real-time instruction index of the conditional
>    branch currently being emitted.
>
> This guarantees the generated immediate offsets always land precisely
> on the next instruction boundary under all multi-pass compilation paths
> and all code size dimensions.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>

This looks like a fix for wrong branch offsets in the tail call path.
Should it carry a Fixes: tag?  The tc_ninsn macro and the

  tc_ninsn = insn ? ctx->offset[insn+1] - ctx->offset[insn] : ctx->offset[0];

calculation being removed here were introduced by:

  Fixes: cd39d9e6b7e4 ("LoongArch: BPF: Fix jump offset calculation in tailcall")

This isn't a bug, but could the changelog be tightened?  Phrases such as
"rock-solid target label", "deterministic arithmetic model",
"initialization bottleneck", "eBPF business instruction" and "all code
size dimensions" read as fairly ornate for an offset-formula refactor.

Could it just state that the old macro derived jmp_offset from a stale
prior-pass stride while the new form anchors on ctx->offset[insn + 1],
with the ctx->image == NULL guard avoiding a false 16-bit range abort
during size estimation?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28766636714

Re: [PATCH v2 2/4] LoongArch: BPF: Move arena register slot below TCC context

[Tiezhu Yang]

On 2026/7/6 下午12:29, bot+bpf-ci@kernel.org wrote:
>> LoongArch: BPF: Move arena register slot below TCC context
>>
>> Currently, the stack layout places the optional arena register slot above
>> the tail call counter context. When arena_vm_start is dynamically enabled,
>> it shifts the relative offset of the tcc_ptr slot within the stack frame,
>> causing hardcoded tracking macros to mismatch and potentially leading to
>> memory misalignment or corruption.
>> To fix this, move the arena register save and restore sequences below the
>> tail call counter context slots in both build_prologue() and epilogue. It
>> needs to update __build_epilogue() to insert a proper offset decrement to
>> safely skip the unneeded tcc_ptr reading block while accurately aligning
>> with the relocated arena slot at the very bottom.
>> With this patch, the tcc_ptr slot is always positioned at a fixed distance
>> directly underneath the base callee-saved registers that is independent of
>> whether the arena features are on.
>> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> 
> This looks like a fix for a memory misalignment or corruption bug affecting
> arena BPF programs.  Should it carry a Fixes: tag?
> 
> The buggy stack layout, where the arena register save/restore and the extra
> stack reservation sit above the tail call counter context, appears to have
> been introduced by:
> 
>    Fixes: ef54c517a937 ("LoongArch: BPF: Implement PROBE_MEM32 pseudo instructions")

OK, will add it in v3.

Thanks,
Tiezhu

Re: [PATCH v2 4/4] LoongArch: BPF: Implement branchless conditional move for TCC

[Tiezhu Yang]

On 2026/7/6 下午12:30, bot+bpf-ci@kernel.org wrote:
>> The current implementation handles combined bpf2bpf and tail calls by
>> checking at runtime whether REG_TCC holds a count or an address via a
>> conditional jump (BPF_JGT against 33). This adds non-negligible branch
>> prediction overhead in the hot path of tail call execution.
>> To eliminate this runtime conditional branch in prologue, refactor the
>> logic of prepare_bpf_tail_call_cnt() with a branchless conditional move
>> mechanism using unsigned comparison and mask instructions.

...

> 
> This isn't a bug, but the closing lines:
> 
>> When exposed to complex bpf2bpf + tailcall interleaved control flows,
>> where inputs shuffle dynamically between a scalar count (0 to 33) and
>> a massive kernel pointer address, clear improvements are demonstrated.
>> Furthermore, the optimization significantly eliminated uncertainty in
>> the execution path, making the performance more stable and predictable.
> 
> state performance improvements without any number or benchmark behind them,
> and the rationale (removing the runtime branch from the prologue hot path)
> is already covered earlier in the message.  Could these closing claims be
> backed by a concrete measurement, or trimmed?

I will trim the commit message in v3.

>> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
>> index 8d6738103f59..b5419cafc984 100644
>> --- a/arch/loongarch/net/bpf_jit.c
>> +++ b/arch/loongarch/net/bpf_jit.c
>> @@ -52,50 +52,56 @@ static void prepare_bpf_tail_call_cnt(struct jit_ctx *ctx, int *store_offset)
> 
> [ ... ]
> 
>> +		/*
>> +		 * Branchless classification and blending:
>> +		 *
>> +		 * In combined bpf2bpf and tailcall scenarios, REG_TCC can carry
>> +		 * either a scalar count (0 to 33) or an inherited kernel pointer address.
>> +		 *
>> +		 * Entrance status for a main program based on REG_TCC physical content:
>> +		 * 1) Initial Entry (Scalar): Enter via a standard event, where REG_TCC
>> +		 *    is initialized to a pure scalar 0.
>> +		 * 2) Inherited Entry (Scalar): Enter via a flat main-to-main tail call,
>> +		 *    where REG_TCC carries an accumulated scalar count (1 to 33) reloaded
>> +		 *    from the prior stack frame.
>> +		 * 3) Inherited Entry (Pointer): Enter via a tail call from a subprogram,
>> +		 *    where REG_TCC inherits and carries a massive kernel pointer address
>> +		 *    reloaded from the prior stack frame.
>> +		 *
>> +		 * sltui:   Identify the type of REG_TCC.
>> +		 *          If REG_TCC < MAX_TAIL_CALL_CNT + 1 (scalar 0~33), sets T8 = 1.
>> +		 *          If REG_TCC >= MAX_TAIL_CALL_CNT + 1 (kernel pointer), sets T8 = 0.
>> +		 *
>> +		 * maskeqz: Handle the local stack pointer (T7).
>> +		 *          If T8 == 1, keeps T7 intact.
>> +		 *          If T8 == 0, clears T7 to 0.
>> +		 *
>> +		 * masknez: Handle the incoming REG_TCC to prevent register pollution.
>> +		 *          If T8 == 1, clears REG_TCC to 0 (erasing 1~33 scalars).
>> +		 *          If T8 == 0, keeps the massive kernel pointer intact.
>> +		 *
>> +		 * or:      Combine the results.
>> +		 *          Scalar Case:  REG_TCC = 0 | T7       -> Clean local stack pointer.
>> +		 *          Pointer Case: REG_TCC = REG_TCC | 0  -> Clean inherited global pointer.
>>   		 */
>> +		emit_insn(ctx, sltui, LOONGARCH_GPR_T8, REG_TCC, MAX_TAIL_CALL_CNT + 1);
>> +		emit_insn(ctx, maskeqz, LOONGARCH_GPR_T7, LOONGARCH_GPR_T7, LOONGARCH_GPR_T8);
>> +		emit_insn(ctx, masknez, REG_TCC, REG_TCC, LOONGARCH_GPR_T8);
>> +		emit_insn(ctx, or, REG_TCC, REG_TCC, LOONGARCH_GPR_T7);
> 
> This isn't a bug, but the comment narrates each of the four emitted
> instructions line by line, and the sltui/maskeqz/masknez/or walk-through
> largely restates the entrance-status text already present in the commit
> message as well as the code right below it.  Could this be condensed?

I will condense the code comments, only remain necessary in v3.

Thanks,
Tiezhu

Re: [PATCH v2 3/4] LoongArch: BPF: Refactor jump offset calculation in tail call

[Tiezhu Yang]

On 2026/7/6 下午12:43, bot+bpf-ci@kernel.org wrote:
>> In emit_bpf_tail_call(), the macro-based jmp_offset calculation relies
>> on a fragile prediction equation: "tc_ninsn - (ctx->idx - idx0)". This
>> calculation model utilizes tc_ninsn from the previous pass to estimate
>> the remaining stride of the active tail call code block.

...

> 
> This looks like a fix for wrong branch offsets in the tail call path.
> Should it carry a Fixes: tag?  The tc_ninsn macro and the
> 
>    tc_ninsn = insn ? ctx->offset[insn+1] - ctx->offset[insn] : ctx->offset[0];
> 
> calculation being removed here were introduced by:
> 
>    Fixes: cd39d9e6b7e4 ("LoongArch: BPF: Fix jump offset calculation in tailcall")
> 
> This isn't a bug, but could the changelog be tightened?  Phrases such as
> "rock-solid target label", "deterministic arithmetic model",
> "initialization bottleneck", "eBPF business instruction" and "all code
> size dimensions" read as fairly ornate for an offset-formula refactor.
> 
> Could it just state that the old macro derived jmp_offset from a stale
> prior-pass stride while the new form anchors on ctx->offset[insn + 1],
> with the ctx->image == NULL guard avoiding a false 16-bit range abort
> during size estimation?

OK, I will add the Fixes tag and trim the commit message in v3.

Thanks,
Tiezhu