* [PATCH 0/1] riscv: Optimize user copy with efficient unaligned access support
@ 2025-12-02 7:41 maohan4761
2025-12-02 7:41 ` [PATCH 1/1] " maohan4761
0 siblings, 1 reply; 3+ messages in thread
From: maohan4761 @ 2025-12-02 7:41 UTC (permalink / raw)
To: pjw, palmer; +Cc: guoren, linux-riscv, linux-kernel, Mao Han
From: Mao Han <han_mao@linux.alibaba.com>
Many modern high-performance processors now handle unaligned memory
accesses with performance nearly on par with aligned accesses. However,
the current kernel implementation of
fallback_scalar_usercopy_sum_enabled still defaults to aligning accesses
to register size boundaries. This path incurs additional
shift-and-combine operations that fail to fully leverage hardware
capabilities and cannot utilize wide-load/store instructions for
trailing data chunks smaller than 9 × SZREG.
This patch introduces an optimized code path enabled by
RISCV_EFFICIENT_UNALIGNED_ACCESS, which relies on hardware support for
efficient unaligned memory accesses. By doing so, the software can fully
exploit the maximum available load/store instruction width and avoids
complex bit-manipulation logic for reassembly.
The optimization significantly improves performance of
__asm_copy_to/from_user for transfers smaller than
riscv_v_usercopy_threshold. In particular, for small data sizes (8–72
bytes) under unaligned scenarios, performance gains range from 40% to
600%.
lmbench’s sig hndl/catch test shows approximately a 20% improvement,
and iperf small-packet throughput sees noticeable gains as well.
Mao Han (1):
riscv: Optimize user copy with efficient unaligned access support
arch/riscv/lib/uaccess.S | 113 +++++++++++++++++++++++++++++++++++++++
1 file changed, 113 insertions(+)
--
2.25.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/1] riscv: Optimize user copy with efficient unaligned access support
2025-12-02 7:41 [PATCH 0/1] riscv: Optimize user copy with efficient unaligned access support maohan4761
@ 2025-12-02 7:41 ` maohan4761
2026-02-09 17:53 ` Andrew Jones
0 siblings, 1 reply; 3+ messages in thread
From: maohan4761 @ 2025-12-02 7:41 UTC (permalink / raw)
To: pjw, palmer; +Cc: guoren, linux-riscv, linux-kernel, Mao Han
From: Mao Han <han_mao@linux.alibaba.com>
Introduce an optimized path in fallback_scalar_usercopy_sum_enabled for
systems that support efficient unaligned memory accesses (i.e., when
CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS is enabled).
This eliminates the overhead of bit-shifting and OR-ing partial words to
reconstruct misaligned values, which was previously required for handling
protential alignments. Medium-sized buffers between 8 and 9*SZREG also see
noticeable improvement, as the original path would fall back to
byte-by-byte copying.
Signed-off-by: Mao Han <han_mao@linux.alibaba.com>
---
arch/riscv/lib/uaccess.S | 113 +++++++++++++++++++++++++++++++++++++++
1 file changed, 113 insertions(+)
diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S
index 4efea1b..bf124a1 100644
--- a/arch/riscv/lib/uaccess.S
+++ b/arch/riscv/lib/uaccess.S
@@ -54,6 +54,118 @@ EXPORT_SYMBOL(__asm_copy_from_user_sum_enabled)
EXPORT_SYMBOL(__asm_copy_to_user_sum_enabled)
SYM_FUNC_START(fallback_scalar_usercopy_sum_enabled)
+ /*
+ * Save the terminal address which will be used to compute the number
+ * of bytes copied in case of a fixup exception.
+ */
+ add t5, a0, a2
+
+ /*
+ * Register allocation for code below:
+ * a0 - start of uncopied dst
+ * a1 - start of uncopied src
+ * a2 - size
+ * t0 - end of uncopied dst
+ */
+ add t0, a0, a2
+#ifdef CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS
+ /* If length < 8, go to byte copy */
+ li a3, 8
+ bltu a2, a3, .Lbyte_copy_tail
+
+ /* check length >= 128 */
+ li t1, 128
+ bltu a2, t1, .L_len_less_16x_szreg
+
+.L_loop_16x_reg:
+ fixup REG_L a4, 0(a1), 10f
+ fixup REG_L a5, SZREG(a1), 10f
+ fixup REG_L a6, 2*SZREG(a1), 10f
+ fixup REG_L a7, 3*SZREG(a1), 10f
+ fixup REG_S a4, 0(a0), 10f
+ fixup REG_S a5, SZREG(a0), 10f
+ fixup REG_S a6, 2*SZREG(a0), 10f
+ fixup REG_S a7, 3*SZREG(a0), 10f
+
+ fixup REG_L t1, 4*SZREG(a1), 10f
+ fixup REG_L t2, 5*SZREG(a1), 10f
+ fixup REG_L t3, 6*SZREG(a1), 10f
+ fixup REG_L t4, 7*SZREG(a1), 10f
+ fixup REG_S t1, 4*SZREG(a0), 10f
+ fixup REG_S t2, 5*SZREG(a0), 10f
+ fixup REG_S t3, 6*SZREG(a0), 10f
+ fixup REG_S t4, 7*SZREG(a0), 10f
+
+ fixup REG_L a4, 8*SZREG(a1), 10f
+ fixup REG_L a5, 9*SZREG(a1), 10f
+ fixup REG_L a6, 10*SZREG(a1), 10f
+ fixup REG_L a7, 11*SZREG(a1), 10f
+ fixup REG_S a4, 8*SZREG(a0), 10f
+ fixup REG_S a5, 9*SZREG(a0), 10f
+ fixup REG_S a6, 10*SZREG(a0), 10f
+ fixup REG_S a7, 11*SZREG(a0), 10f
+
+ fixup REG_L t1, 12*SZREG(a1), 10f
+ fixup REG_L t2, 13*SZREG(a1), 10f
+ fixup REG_L t3, 14*SZREG(a1), 10f
+ fixup REG_L t4, 15*SZREG(a1), 10f
+ fixup REG_S t1, 12*SZREG(a0), 10f
+ fixup REG_S t2, 13*SZREG(a0), 10f
+ fixup REG_S t3, 14*SZREG(a0), 10f
+ fixup REG_S t4, 15*SZREG(a0), 10f
+
+ addi a1, a1, 16*SZREG
+ addi a0, a0, 16*SZREG
+
+ addi t1, a0, 16*SZREG
+ bleu t1, t0, .L_loop_16x_reg
+
+.L_len_less_16x_szreg:
+ # Pre-check: ensure at least one register copy is possible
+ addi t1, a0, 4*SZREG
+ bgtu t1, t0, .L_len_less_4x_szreg
+
+.L_loop_4x_reg:
+ fixup REG_L a4, 0(a1), 10f
+ fixup REG_L a5, SZREG(a1), 10f
+ fixup REG_L a6, 2*SZREG(a1), 10f
+ fixup REG_L a7, 3*SZREG(a1), 10f
+ fixup REG_S a4, 0(a0), 10f
+ fixup REG_S a5, SZREG(a0), 10f
+ fixup REG_S a6, 2*SZREG(a0), 10f
+ fixup REG_S a7, 3*SZREG(a0), 10f
+ addi a1, a1, 4*SZREG
+ addi a0, a0, 4*SZREG
+
+ # Check if another register copy is safe
+ addi t1, a0, 4*SZREG
+ bleu t1, t0, .L_loop_4x_reg
+
+.L_len_less_4x_szreg:
+ # Pre-check: ensure at least one register copy is possible
+ add t1, a0, SZREG
+ bgtu t1, t0, .Lbyte_copy_word
+
+.L_loop_reg:
+ fixup REG_L a4, 0(a1), 10f
+ addi a1, a1, SZREG
+ fixup REG_S a4, 0(a0), 10f
+ addi a0, a0, SZREG
+
+ # Check if another register copy is safe
+ addi t1, a0, SZREG
+ bleu t1, t0, .L_loop_reg
+.Lbyte_copy_word:
+#if __riscv_xlen == 64
+ add t1, a0, 4
+ bgtu t1, t0, .Lbyte_copy_tail
+
+ fixup lw a4, 0(a1), 10f
+ addi a1, a1, 4
+ fixup sw a4, 0(a0), 10f
+ addi a0, a0, 4
+#endif
+#else
/*
* Save the terminal address which will be used to compute the number
* of bytes copied in case of a fixup exception.
@@ -190,6 +302,7 @@ SYM_FUNC_START(fallback_scalar_usercopy_sum_enabled)
/* Revert src to original unaligned value */
add a1, a1, a3
+#endif /* CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS */
.Lbyte_copy_tail:
/*
* Byte copy anything left.
--
2.25.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH 1/1] riscv: Optimize user copy with efficient unaligned access support
2025-12-02 7:41 ` [PATCH 1/1] " maohan4761
@ 2026-02-09 17:53 ` Andrew Jones
0 siblings, 0 replies; 3+ messages in thread
From: Andrew Jones @ 2026-02-09 17:53 UTC (permalink / raw)
To: maohan4761; +Cc: pjw, palmer, guoren, linux-riscv, linux-kernel, Mao Han
On Tue, Dec 02, 2025 at 03:41:21PM +0800, maohan4761@gmail.com wrote:
...
> +.L_len_less_16x_szreg:
> + # Pre-check: ensure at least one register copy is possible
ensure at least four register copies are possible
> + addi t1, a0, 4*SZREG
> + bgtu t1, t0, .L_len_less_4x_szreg
> +
> +.L_loop_4x_reg:
> + fixup REG_L a4, 0(a1), 10f
> + fixup REG_L a5, SZREG(a1), 10f
> + fixup REG_L a6, 2*SZREG(a1), 10f
> + fixup REG_L a7, 3*SZREG(a1), 10f
> + fixup REG_S a4, 0(a0), 10f
> + fixup REG_S a5, SZREG(a0), 10f
> + fixup REG_S a6, 2*SZREG(a0), 10f
> + fixup REG_S a7, 3*SZREG(a0), 10f
> + addi a1, a1, 4*SZREG
> + addi a0, a0, 4*SZREG
> +
> + # Check if another register copy is safe
Check if another four register copies are safe
> + addi t1, a0, 4*SZREG
> + bleu t1, t0, .L_loop_4x_reg
> +
Thanks,
drew
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-02-09 17:53 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-12-02 7:41 [PATCH 0/1] riscv: Optimize user copy with efficient unaligned access support maohan4761
2025-12-02 7:41 ` [PATCH 1/1] " maohan4761
2026-02-09 17:53 ` Andrew Jones
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®