mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v3 0/2] arm64: csum: Add fused copy and Internet checksum
@ 2026-11-20 22:03 Demian Shulhan
  2026-11-20 22:03 ` [PATCH v3 1/2] " Demian Shulhan
  2026-11-20 22:03 ` [PATCH v3 2/2] lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck() Demian Shulhan
  0 siblings, 2 replies; 3+ messages in thread
From: Demian Shulhan @ 2026-11-20 22:03 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon
  Cc: Robin Murphy, David Laight, Nathan Chancellor, Jakub Kicinski,
	Eric Dumazet, Andrew Morton, David Gow, Brendan Higgins,
	linux-arm-kernel, netdev, kunit-dev, linux-kselftest,
	linux-kernel

Add a single-pass csum_partial_copy_nocheck() for arm64 and KUnit
coverage (plus an optional throughput benchmark) in checksum_kunit.

On ARM Neoverse-V2 (3.0 GHz), this reaches 39.2-45.1 GB/s aligned
(1.8-1.9x) and 36.5-43.8 GB/s unaligned (2.0-2.2x) for >= 1368B buffers
vs generic memcpy() + csum_partial().

Changes in v3:
- Use a 64B carry-tree primitive unrolled 4x to 256B (~21 insns/64B,
  pure leaf function, zero spills on GCC 15 and Clang 21).
- Align @src to 16B for len >= 272, stashing the odd-offset bit in
  bit 62 of the carry counter.
- Add zero-length, 72/256/272B threshold, and dst[-1]/dst[len]
  sentinel checks to checksum_kunit.

Changes in v2:
- Drop the NEON path and separate csum-copy* files, folding the scalar
  implementation directly into arch/arm64/lib/csum.c (resolves the
  <asm/neon-intrinsics.h> include issue noted by Nathan Chancellor).
- Switch the scalar loop to four __uint128_t accumulators (David Laight).
- Canonicalize 0xffff (-0) to 0x0000 (+0) after csum_fold() in KUnit to
  fix x86_64 test failures on zero/all-zero inputs (Jakub Kicinski).

Link: https://lore.kernel.org/r/20261114045500.v2-1-demyansh@gmail.com

Demian Shulhan (2):
  arm64: csum: Add fused copy and Internet checksum
  lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck()

 arch/arm64/include/asm/checksum.h |   3 +
 arch/arm64/lib/csum.c             | 189 +++++++++++++
 lib/Kconfig.debug                 |  10 +
 lib/tests/checksum_kunit.c        | 453 ++++++++++++++++++++++++++++++
 4 files changed, 655 insertions(+)


base-commit: 97d73c8251226cf27cc6656b4c542869bb8c69fa
-- 
2.53.0


^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH v3 1/2] arm64: csum: Add fused copy and Internet checksum
  2026-11-20 22:03 [PATCH v3 0/2] arm64: csum: Add fused copy and Internet checksum Demian Shulhan
@ 2026-11-20 22:03 ` Demian Shulhan
  2026-11-20 22:03 ` [PATCH v3 2/2] lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck() Demian Shulhan
  1 sibling, 0 replies; 3+ messages in thread
From: Demian Shulhan @ 2026-11-20 22:03 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon
  Cc: Robin Murphy, David Laight, Nathan Chancellor, Jakub Kicinski,
	Eric Dumazet, Andrew Morton, David Gow, Brendan Higgins,
	linux-arm-kernel, netdev, kunit-dev, linux-kselftest,
	linux-kernel

arm64 falls back to the generic csum_partial_copy_nocheck(), which does
two passes (memcpy() + csum_partial()) and hits software GSO paths
(e.g. TCP over WireGuard) and UDP/ICMP replies.

Implement a single-pass csum_partial_copy_nocheck() in
arch/arm64/lib/csum.c using a 64-byte carry-tree primitive unrolled 4x
to 256 bytes (~21 instructions per 64B, pure leaf function with zero
spills on GCC 15 and Clang 21) and 16-byte @src alignment for
len >= 272. Each summed word is read once into GPRs before storing and
summing, so the checksum always matches @dst under concurrent @src
modification.

In-kernel KUnit benchmark on ARM Neoverse-V2 (3.0 GHz,
CONFIG_CHECKSUM_KUNIT_BENCHMARK=y, GCC 15.2 / Clang 21.1, median of 3):

  case (src/dst)    memcpy+csum_partial  fused (GCC / Clang)    speedup
  ----------------  -------------------  -------------------  -----------
  16B    (+0 / +0)     4.0 /  3.7 GB/s      6.9 /  5.9 GB/s   1.73x/1.61x
  64B    (+0 / +0)     9.4 / 10.5 GB/s     19.1 / 22.0 GB/s   2.04x/2.08x
  256B   (+0 / +0)    16.3 / 18.4 GB/s     33.2 / 35.2 GB/s   2.04x/1.92x
  512B   (+0 / +0)    19.4 / 21.0 GB/s     36.5 / 39.3 GB/s   1.88x/1.87x
  1368B  (+0 / +0)    21.7 / 22.2 GB/s     39.5 / 41.2 GB/s   1.82x/1.85x
  1448B  (+0 / +0)    21.8 / 22.2 GB/s     39.2 / 41.1 GB/s   1.79x/1.86x
  4096B  (+0 / +0)    23.4 / 23.5 GB/s     42.5 / 43.9 GB/s   1.82x/1.87x
  16384B (+0 / +0)    23.8 / 24.0 GB/s     43.3 / 45.1 GB/s   1.82x/1.87x
  16B    (+1 / +3)     1.5 /  2.6 GB/s      6.9 /  5.9 GB/s   4.44x/2.25x
  128B   (+1 / +3)     6.4 / 10.5 GB/s     24.3 / 27.2 GB/s   3.78x/2.60x
  1448B  (+1 / +3)    17.8 / 18.0 GB/s     38.0 / 39.0 GB/s   2.14x/2.16x
  1500B  (+1 / +3)    17.9 / 18.2 GB/s     36.5 / 39.6 GB/s   2.04x/2.18x
  4096B  (+1 / +3)    19.1 / 19.5 GB/s     40.6 / 42.0 GB/s   2.12x/2.15x
  16384B (+1 / +3)    19.4 / 19.6 GB/s     42.3 / 43.8 GB/s   2.18x/2.24x

Suggested-by: David Laight <david.laight.linux@gmail.com>
Signed-off-by: Demian Shulhan <demyansh@gmail.com>
---
 arch/arm64/include/asm/checksum.h |   3 +
 arch/arm64/lib/csum.c             | 189 ++++++++++++++++++++++++++++++
 2 files changed, 192 insertions(+)

diff --git a/arch/arm64/include/asm/checksum.h b/arch/arm64/include/asm/checksum.h
index dc52b733675d..9f766db01152 100644
--- a/arch/arm64/include/asm/checksum.h
+++ b/arch/arm64/include/asm/checksum.h
@@ -44,6 +44,9 @@ static inline __sum16 ip_fast_csum(const void *iph, unsigned int ihl)
 extern unsigned int do_csum(const unsigned char *buff, int len);
 #define do_csum do_csum
 
+#define _HAVE_ARCH_CSUM_AND_COPY
+__wsum csum_partial_copy_nocheck(const void *src, void *dst, int len);
+
 #include <asm-generic/checksum.h>
 
 #endif	/* __ASM_CHECKSUM_H */
diff --git a/arch/arm64/lib/csum.c b/arch/arm64/lib/csum.c
index 75db718e5b6a..2bf2d63cfd42 100644
--- a/arch/arm64/lib/csum.c
+++ b/arch/arm64/lib/csum.c
@@ -1,9 +1,12 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright (C) 2019-2020 Arm Ltd.
 
+#include <linux/bitops.h>
 #include <linux/compiler.h>
+#include <linux/export.h>
 #include <linux/kasan-checks.h>
 #include <linux/kernel.h>
+#include <linux/unaligned.h>
 
 #include <net/checksum.h>
 
@@ -116,6 +119,192 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len)
 	return sum >> 16;
 }
 
+static __always_inline __uint128_t get_unaligned128(const u8 *p)
+{
+	u64 lo = get_unaligned((const u64 *)p);
+	u64 hi = get_unaligned((const u64 *)(p + 8));
+
+	return ((__uint128_t)hi << 64) | lo;
+}
+
+static __always_inline void put_unaligned128(__uint128_t val, u8 *p)
+{
+	put_unaligned((u64)val, (u64 *)p);
+	put_unaligned((u64)(val >> 64), (u64 *)(p + 8));
+}
+
+static __always_inline __wsum csum_fold_acc_cy(__uint128_t acc, u64 cy)
+{
+	__uint128_t t;
+	u64 s;
+
+	acc += (acc >> 64) | (acc << 64);
+	t = (__uint128_t)(u64)(acc >> 64) + cy;
+	s = (u64)t + (u64)(t >> 64);
+	s += (s >> 32) | (s << 32);
+	return (__force __wsum)(u32)(s >> 32);
+}
+
+static __always_inline void csum_tree64_ovf(const u8 *s, u8 *d,
+					    __uint128_t *acc, s64 *cy)
+{
+	__uint128_t v0 = get_unaligned128(s);
+	__uint128_t v1 = get_unaligned128(s + 16);
+	__uint128_t v2, v3;
+	s64 o;
+
+	put_unaligned128(v0, d);
+	put_unaligned128(v1, d + 16);
+	o = __builtin_add_overflow(v0, v1, &v0);
+
+	v2 = get_unaligned128(s + 32);
+	v3 = get_unaligned128(s + 48);
+	put_unaligned128(v2, d + 32);
+	put_unaligned128(v3, d + 48);
+	o += __builtin_add_overflow(v2, v3, &v2);
+
+	o += __builtin_add_overflow(v0, v2, &v0);
+	*cy += o + __builtin_add_overflow(*acc, v0, acc);
+}
+
+static __always_inline void csum_align_src16(const u8 **ps, u8 **pd,
+					     unsigned int *pulen,
+					     __uint128_t *pacc, s64 *pc0)
+{
+	uintptr_t off = (uintptr_t)(*ps) & 15;
+
+	if (unlikely(off) && *pulen >= 272U) {
+		unsigned int adv = 16U - (unsigned int)off;
+		u64 lo = get_unaligned((const u64 *)(*ps));
+		u64 hi = get_unaligned((const u64 *)(*ps + 8));
+		u64 odd = (u64)off & 1ULL;
+		u64 m = ~0ULL >> ((off & 7) * 8);
+		u64 h;
+
+		put_unaligned(lo, (u64 *)(*pd));
+		put_unaligned(hi, (u64 *)(*pd + 8));
+		lo = (off & 8) ? (lo & m) : lo;
+		hi = (off & 8) ? 0ULL : (hi & m);
+		h = lo + hi;
+		h += h < lo;
+		*pacc = ror64(h, (unsigned int)(odd * 56ULL));
+		/*
+		 * At len <= INT_MAX, the 256B loop adds <= 16 carries per
+		 * iteration (< 2^27 total before bit 62 is cleared, and c0
+		 * stays < 2^33 overall), so bit 62 of @c0 cannot overflow.
+		 */
+		*pc0 = (s64)(odd << 62);
+		*ps += adv;
+		*pd += adv;
+		*pulen -= adv;
+	}
+}
+
+static __always_inline __wsum csum_fold_final(__uint128_t acc, u64 cy, u64 odd)
+{
+	__wsum res = csum_fold_acc_cy(acc, cy);
+
+	if (unlikely(odd))
+		return (__force __wsum)ror32((__force u32)res, 8);
+	return res;
+}
+
+/*
+ * Single-pass copy and ones' complement sum. Each summed word is read once
+ * into a GPR before storing and summing, so the returned sum always matches
+ * @dst even if @src is concurrently modified.
+ */
+__wsum __no_sanitize_address csum_partial_copy_nocheck(const void *src, void *dst, int len)
+{
+	const u8 *s = src;
+	u8 *d = dst;
+	__uint128_t acc = 0;
+	s64 c0 = 0;
+	u64 odd = 0;
+	unsigned int ulen = (unsigned int)len;
+
+	if (unlikely(len <= 0))
+		return 0;
+
+	kasan_check_read(src, len);
+	kasan_check_write(dst, len);
+
+	if (ulen >= 64U) {
+		if (ulen >= 256U) {
+			csum_align_src16(&s, &d, &ulen, &acc, &c0);
+			ulen -= 256U;
+			do {
+				csum_tree64_ovf(s, d, &acc, &c0);
+				csum_tree64_ovf(s + 64, d + 64, &acc, &c0);
+				csum_tree64_ovf(s + 128, d + 128, &acc, &c0);
+				csum_tree64_ovf(s + 192, d + 192, &acc, &c0);
+				s += 256;
+				d += 256;
+			} while (!__builtin_sub_overflow(ulen, 256U, &ulen));
+			odd = (u64)c0 >> 62;
+			c0 &= (1LL << 62) - 1;
+			ulen &= 255U;
+			if (!ulen)
+				goto fold;
+		}
+		if (ulen >= 64U) {
+			ulen -= 64U;
+			do {
+				csum_tree64_ovf(s, d, &acc, &c0);
+				s += 64;
+				d += 64;
+			} while (!__builtin_sub_overflow(ulen, 64U, &ulen));
+			ulen &= 63U;
+			if (!ulen)
+				goto fold;
+		}
+	}
+	if (likely(ulen >= 16U)) {
+		ulen -= 16U;
+		do {
+			__uint128_t v = get_unaligned128(s);
+
+			put_unaligned128(v, d);
+			s += 16;
+			d += 16;
+			c0 += __builtin_add_overflow(acc, v, &acc);
+		} while (!__builtin_sub_overflow(ulen, 16U, &ulen));
+	}
+	if (likely(ulen & 8U)) {
+		u64 v = get_unaligned((const u64 *)s);
+
+		put_unaligned(v, (u64 *)d);
+		s += 8;
+		d += 8;
+		c0 += __builtin_add_overflow(acc, v, &acc);
+	}
+	if (likely(ulen & 4U)) {
+		u32 v = get_unaligned((const u32 *)s);
+
+		put_unaligned(v, (u32 *)d);
+		s += 4;
+		d += 4;
+		c0 += v;
+	}
+	if (likely(ulen & 2U)) {
+		u16 v = get_unaligned((const u16 *)s);
+
+		put_unaligned(v, (u16 *)d);
+		s += 2;
+		d += 2;
+		c0 += v;
+	}
+	if (ulen & 1U) {
+		u8 v = *s;
+
+		*d = v;
+		c0 += v;
+	}
+fold:
+	return csum_fold_final(acc, (u64)c0, odd);
+}
+EXPORT_SYMBOL(csum_partial_copy_nocheck);
+
 __sum16 csum_ipv6_magic(const struct in6_addr *saddr,
 			const struct in6_addr *daddr,
 			__u32 len, __u8 proto, __wsum csum)
-- 
2.53.0


^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH v3 2/2] lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck()
  2026-11-20 22:03 [PATCH v3 0/2] arm64: csum: Add fused copy and Internet checksum Demian Shulhan
  2026-11-20 22:03 ` [PATCH v3 1/2] " Demian Shulhan
@ 2026-11-20 22:03 ` Demian Shulhan
  1 sibling, 0 replies; 3+ messages in thread
From: Demian Shulhan @ 2026-11-20 22:03 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon
  Cc: Robin Murphy, David Laight, Nathan Chancellor, Jakub Kicinski,
	Eric Dumazet, Andrew Morton, David Gow, Brendan Higgins,
	linux-arm-kernel, netdev, kunit-dev, linux-kselftest,
	linux-kernel

Add KUnit tests for csum_partial_copy_nocheck() covering 0..64B
exhaustive alignments, threshold and MTU/page lengths up to 4352B,
carry-stressing patterns, vmalloc() guard pages and dst canaries,
zero-length calls, and task/softirq/hardirq execution.

Checksums are verified against a standalone C reference and
csum_partial(), canonicalizing 0xffff (-0) to 0x0000 (+0) per RFC 1071.
Also add an optional throughput benchmark gated behind
CONFIG_CHECKSUM_KUNIT_BENCHMARK (default n).

Signed-off-by: Demian Shulhan <demyansh@gmail.com>
---
 lib/Kconfig.debug          |  10 +
 lib/tests/checksum_kunit.c | 453 +++++++++++++++++++++++++++++++++++++
 2 files changed, 463 insertions(+)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2d22638ec13a..588343ced260 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2770,6 +2770,16 @@ config CHECKSUM_KUNIT
 
 	  If unsure, say N.
 
+config CHECKSUM_KUNIT_BENCHMARK
+	bool "Benchmark for the checksum functions"
+	depends on CHECKSUM_KUNIT
+	help
+	  Include a benchmark of csum_partial_copy_nocheck() against
+	  memcpy() followed by csum_partial() in the checksum KUnit test
+	  suite, for a range of buffer lengths and misalignments. The
+	  throughput of both is logged as test output. Only meaningful on
+	  real hardware and is not intended for production builds.
+
 config UTIL_MACROS_KUNIT
 	tristate "KUnit test util_macros.h functions at runtime" if !KUNIT_ALL_TESTS
 	depends on KUNIT
diff --git a/lib/tests/checksum_kunit.c b/lib/tests/checksum_kunit.c
index be04aa42125c..ac4be0a27c7d 100644
--- a/lib/tests/checksum_kunit.c
+++ b/lib/tests/checksum_kunit.c
@@ -3,8 +3,15 @@
  * Test cases csum_partial, csum_fold, ip_fast_csum, csum_ipv6_magic
  */
 
+#include <kunit/run-in-irq-context.h>
 #include <kunit/test.h>
+#include <linux/math64.h>
+#include <linux/prandom.h>
+#include <linux/preempt.h>
+#include <linux/string.h>
+#include <linux/vmalloc.h>
 #include <asm/checksum.h>
+#include <net/checksum.h>
 #include <net/ip6_checksum.h>
 
 #define MAX_LEN 512
@@ -619,18 +626,464 @@ static void test_csum_ipv6_magic(struct kunit *test)
 	}
 }
 
+/*
+ * csum_partial_copy_nocheck() tests: verify copied bytes, folded checksum
+ * against a standalone reference and csum_partial(), canary/guard-page
+ * bounds, and concurrent task/softirq/hardirq execution.
+ */
+#define COPY_MAX_OFF 16
+#define COPY_MAX_LEN 4352
+#define COPY_BENCH_MAX_LEN 16384
+#define COPY_GUARD 128 /* canary span on each side of dst */
+#define COPY_CANARY 0xa5
+#define COPY_SEED 0x5eed
+
+static struct rnd_state copy_rng;
+/* vmalloc()ed with a page-aligned length, so each is followed by a guard page */
+static u8 *copy_src_area;
+static u8 *copy_dst_area;
+static u8 *copy_shadow; /* expected contents of copy_src_area */
+static size_t copy_area_len;
+
+static int checksum_suite_init(struct kunit_suite *suite)
+{
+	copy_area_len = round_up(COPY_BENCH_MAX_LEN + COPY_MAX_OFF, PAGE_SIZE);
+	copy_src_area = vmalloc(copy_area_len);
+	copy_dst_area = vmalloc(copy_area_len);
+	copy_shadow = vmalloc(copy_area_len);
+	if (!copy_src_area || !copy_dst_area || !copy_shadow) {
+		vfree(copy_src_area);
+		vfree(copy_dst_area);
+		vfree(copy_shadow);
+		copy_src_area = NULL;
+		copy_dst_area = NULL;
+		copy_shadow = NULL;
+		return -ENOMEM;
+	}
+	prandom_seed_state(&copy_rng, COPY_SEED);
+	return 0;
+}
+
+static void checksum_suite_exit(struct kunit_suite *suite)
+{
+	vfree(copy_src_area);
+	vfree(copy_dst_area);
+	vfree(copy_shadow);
+	copy_src_area = NULL;
+	copy_dst_area = NULL;
+	copy_shadow = NULL;
+}
+
+static void copy_fill_random(void)
+{
+	prandom_bytes_state(&copy_rng, copy_src_area, copy_area_len);
+	memcpy(copy_shadow, copy_src_area, copy_area_len);
+}
+
+static void copy_fill_bytes(const u8 *pattern, int pattern_len)
+{
+	size_t i;
+
+	for (i = 0; i < copy_area_len; i++)
+		copy_src_area[i] = pattern[i % pattern_len];
+	memcpy(copy_shadow, copy_src_area, copy_area_len);
+}
+
+/* Place @len bytes at misalignment @off as close as possible to the guard page. */
+static u8 *copy_place_at_end(u8 *area, int off, int len)
+{
+	u8 *end = area + copy_area_len;
+	int pad;
+
+	/*
+	 * UML lacks EX_TYPE_ZEROPAD fixup on vmalloc guard pages in x86
+	 * csum_partial(), and RISC-V do_csum_no_alignment() over-reads up to
+	 * sizeof(unsigned long) - 1 bytes past the buffer end.
+	 */
+	if (IS_ENABLED(CONFIG_UML) || IS_ENABLED(CONFIG_RISCV))
+		end -= sizeof(unsigned long);
+
+	pad = ((unsigned long)(end - len) - off) & (COPY_MAX_OFF - 1);
+	return end - len - pad;
+}
+
+/* Naive reference: folded ones' complement sum, same semantics as csum_fold(). */
+static u16 ref_csum_fold(const u8 *buf, int len)
+{
+	u64 sum = 0;
+	u16 word;
+	int i;
+
+	for (i = 0; i + 1 < len; i += 2) {
+		memcpy(&word, buf + i, sizeof(word)); /* native endian */
+		sum += word;
+	}
+	if (len & 1) {
+		u8 pad[2] = { buf[len - 1], 0 };
+
+		memcpy(&word, pad, sizeof(word)); /* LE: low byte, BE: high byte */
+		sum += word;
+	}
+	while (sum >> 16)
+		sum = (sum & 0xffff) + (sum >> 16);
+
+	sum = ~sum & 0xffff;
+	return sum == 0xffff ? 0 : sum;
+}
+
+/*
+ * RFC 1071: 0xffff (-0) and 0x0000 (+0) are equivalent. Canonicalize to 0 so
+ * ~0U-seeded archs (e.g. x86_64) match 0-seeded refs on zero/all-zero inputs.
+ */
+static u16 fold_csum(__wsum csum)
+{
+	u16 sum = (__force u16)csum_fold(csum);
+
+	return sum == 0xffff ? 0 : sum;
+}
+
+static int first_non_canary(const u8 *p, int n)
+{
+	int i;
+
+	for (i = 0; i < n; i++)
+		if (p[i] != COPY_CANARY)
+			return i;
+	return -1;
+}
+
+/* Verify one csum_partial_copy_nocheck() call with @before/@after canaries. */
+static void check_csum_copy_at(struct kunit *test, const u8 *src, u8 *dst,
+			       int len, int before, int after)
+{
+	unsigned long src_off = (unsigned long)src & (COPY_MAX_OFF - 1);
+	unsigned long dst_off = (unsigned long)dst & (COPY_MAX_OFF - 1);
+	const u8 *lo, *hi;
+	u16 expect, got;
+	int bad;
+
+	memset(dst - before, COPY_CANARY, before + len + after);
+	expect = ref_csum_fold(src, len);
+
+	got = fold_csum(csum_partial_copy_nocheck(src, dst, len));
+
+	KUNIT_ASSERT_EQ_MSG(test, got, expect,
+			    "checksum mismatch src_off=%lu dst_off=%lu len=%d",
+			    src_off, dst_off, len);
+	KUNIT_ASSERT_EQ_MSG(test, memcmp(src, dst, len), 0,
+			    "copied data mismatch src_off=%lu dst_off=%lu len=%d",
+			    src_off, dst_off, len);
+
+	if (before > 0)
+		KUNIT_ASSERT_EQ_MSG(test, dst[-1], COPY_CANARY,
+				    "clobbered dst[-1] (src_off=%lu dst_off=%lu len=%d)",
+				    src_off, dst_off, len);
+	if (after > 0)
+		KUNIT_ASSERT_EQ_MSG(test, dst[len], COPY_CANARY,
+				    "clobbered dst[len] (src_off=%lu dst_off=%lu len=%d)",
+				    src_off, dst_off, len);
+
+	bad = first_non_canary(dst - before, before);
+	KUNIT_ASSERT_EQ_MSG(test, bad, -1,
+			    "wrote %d bytes before dst (src_off=%lu dst_off=%lu len=%d)",
+			    before - bad, src_off, dst_off, len);
+
+	bad = first_non_canary(dst + len, after);
+	KUNIT_ASSERT_EQ_MSG(test, bad, -1,
+			    "wrote at dst+len+%d (src_off=%lu dst_off=%lu len=%d)",
+			    bad, src_off, dst_off, len);
+
+	/* Source untouched, including a margin on either side. */
+	lo = max(src - COPY_GUARD, (const u8 *)copy_src_area);
+	hi = min(src + len + COPY_GUARD, (const u8 *)copy_src_area + copy_area_len);
+	KUNIT_ASSERT_EQ_MSG(test,
+			    memcmp(lo, copy_shadow + (lo - copy_src_area), hi - lo), 0,
+			    "source buffer modified (src_off=%lu dst_off=%lu len=%d)",
+			    src_off, dst_off, len);
+
+	/* Must be interchangeable with the arch csum_partial() semantics. */
+	KUNIT_ASSERT_EQ_MSG(test, got,
+			    fold_csum(csum_partial(src, len, 0)),
+			    "disagrees with csum_partial() src_off=%lu dst_off=%lu len=%d",
+			    src_off, dst_off, len);
+}
+
+/* Source against its guard page, destination between canaries. */
+static void check_csum_copy(struct kunit *test, int src_off, int dst_off,
+			    int len)
+{
+	KUNIT_ASSERT_TRUE(test, src_off >= 0 && src_off < COPY_MAX_OFF);
+	KUNIT_ASSERT_TRUE(test, dst_off >= 0 && dst_off < COPY_MAX_OFF);
+	KUNIT_ASSERT_TRUE(test, len >= 0 && len <= COPY_MAX_LEN);
+
+	check_csum_copy_at(test, copy_place_at_end(copy_src_area, src_off, len),
+			   copy_dst_area + COPY_GUARD + dst_off, len,
+			   COPY_GUARD, COPY_GUARD);
+}
+
+/* Representative source offsets used with all 16 destination offsets. */
+static const int copy_src_offs[] = { 0, 1, 3, 8, 15 };
+
+/* Every len 0..64 with every src/dst misalignment: the scalar tail ladder. */
+static void test_csum_copy_small_all_alignments(struct kunit *test)
+{
+	int len, src_off, dst_off;
+
+	copy_fill_random();
+	for (len = 0; len <= 64; len++)
+		for (src_off = 0; src_off < COPY_MAX_OFF; src_off++)
+			for (dst_off = 0; dst_off < COPY_MAX_OFF; dst_off++)
+				check_csum_copy(test, src_off, dst_off, len);
+}
+
+/* Threshold boundaries & large unaligned buffers across [0..15]x[0..7] and [0..15]. */
+static void test_csum_copy_large_unaligned(struct kunit *test)
+{
+	static const int threshold_lens[] = {
+		48, 63, 64, 65, 71, 72, 73, 79, 80, 96, 127, 128,
+		255, 256, 257, 271, 272, 273,
+		1368, 1448, 1500, 4096,
+	};
+	static const int extra_lens[] = {
+		1368, 1448, 1500, 2033, 2034, 2035, 2039, 2040, 2041, 2047,
+		2048, 2049, 2055, 2056, 2057, 2062, 2063, 3072, 4095, 4096,
+		4097, 4352,
+	};
+	int len, i, j, src_off, dst_off;
+
+	copy_fill_random();
+	for (j = 0; j < ARRAY_SIZE(threshold_lens); j++)
+		for (src_off = 0; src_off < COPY_MAX_OFF; src_off++)
+			for (dst_off = 0; dst_off < 8; dst_off++)
+				check_csum_copy(test, src_off, dst_off,
+						threshold_lens[j]);
+
+	for (len = 1024; len < 1024 + 80; len++)
+		for (i = 0; i < ARRAY_SIZE(copy_src_offs); i++)
+			for (dst_off = 0; dst_off < COPY_MAX_OFF; dst_off++)
+				check_csum_copy(test, copy_src_offs[i], dst_off,
+						len);
+
+	for (j = 0; j < ARRAY_SIZE(extra_lens); j++)
+		for (i = 0; i < ARRAY_SIZE(copy_src_offs); i++)
+			for (dst_off = 0; dst_off < COPY_MAX_OFF; dst_off++)
+				check_csum_copy(test, copy_src_offs[i], dst_off,
+						extra_lens[j]);
+}
+
+/* Patterns stressing carries, zero/MSB bytes, and odd-offset byte order. */
+static void test_csum_copy_patterns(struct kunit *test)
+{
+	static const u8 pat_ff[] = { 0xff };
+	static const u8 pat_00[] = { 0x00 };
+	static const u8 pat_80[] = { 0x80 };
+	static const u8 pat_ff00[] = { 0xff, 0x00 };
+	static const u8 pat_ramp[] = { 0x01, 0x02, 0x04, 0x08,
+				       0x10, 0x20, 0x40, 0x80 };
+	static const struct {
+		const u8 *pat;
+		int len;
+	} patterns[] = {
+		{ pat_ff, ARRAY_SIZE(pat_ff) },
+		{ pat_00, ARRAY_SIZE(pat_00) },
+		{ pat_80, ARRAY_SIZE(pat_80) },
+		{ pat_ff00, ARRAY_SIZE(pat_ff00) },
+		{ pat_ramp, ARRAY_SIZE(pat_ramp) },
+		{ NULL, 0 }, /* random */
+	};
+	static const int lens[] = {
+		1, 2, 3, 7, 8, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128,
+		255, 256, 511, 512, 1023, 1024, 1025, 1368, 1448, 1500, 2048,
+		4095, 4096, 4352,
+	};
+	static const int offs[][2] = {
+		{ 0, 0 }, { 1, 0 }, { 0, 1 }, { 1, 1 }, { 3, 7 }, { 7, 3 },
+		{ 9, 5 }, { 15, 15 },
+	};
+	int p, l, o;
+
+	for (p = 0; p < ARRAY_SIZE(patterns); p++) {
+		if (patterns[p].pat)
+			copy_fill_bytes(patterns[p].pat, patterns[p].len);
+		else
+			copy_fill_random();
+
+		for (l = 0; l < ARRAY_SIZE(lens); l++)
+			for (o = 0; o < ARRAY_SIZE(offs); o++)
+				check_csum_copy(test, offs[o][0], offs[o][1],
+						lens[l]);
+	}
+}
+
+/* Place dst against its trailing guard page so any over-write faults. */
+static void test_csum_copy_dst_guard_page(struct kunit *test)
+{
+	static const int lens[] = {
+		1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
+		31, 32, 33, 63, 64, 65, 71, 72, 73, 79, 80, 81, 96, 127, 128,
+		129, 255, 256, 257, 271, 272, 273, 1023, 1024, 1025, 1368,
+		1448, 1500, 2048, 4095, 4096, 4097,
+	};
+	int l, src_off, dst_off;
+
+	copy_fill_random();
+	for (l = 0; l < ARRAY_SIZE(lens); l++)
+		for (dst_off = 0; dst_off < COPY_MAX_OFF; dst_off++) {
+			u8 *dst = copy_place_at_end(copy_dst_area, dst_off, lens[l]);
+			int after = copy_dst_area + copy_area_len - (dst + lens[l]);
+
+			src_off = (dst_off * 7 + lens[l]) & (COPY_MAX_OFF - 1);
+			check_csum_copy_at(test,
+					   copy_place_at_end(copy_src_area,
+							     src_off, lens[l]),
+					   dst, lens[l], COPY_GUARD, after);
+		}
+}
+
+/* len == 0: fold to 0 and never touch src or dst, even against guard pages. */
+static void test_csum_copy_zero_len(struct kunit *test)
+{
+	u8 *unmapped_end = copy_src_area + copy_area_len;
+	u8 *dst = copy_dst_area + COPY_GUARD;
+	__wsum sum;
+
+	copy_fill_random();
+	memset(copy_dst_area, COPY_CANARY, 2 * COPY_GUARD);
+	sum = csum_partial_copy_nocheck(unmapped_end, dst, 0);
+	KUNIT_EXPECT_EQ(test, fold_csum(sum), 0);
+	KUNIT_EXPECT_EQ(test, first_non_canary(copy_dst_area, 2 * COPY_GUARD), -1);
+}
+
+/* Concurrent execution across task, softirq, and hardirq contexts. */
+#define COPY_IRQ_NUM_MSGS 3
+#define COPY_IRQ_NUM_CTX 3 /* task, softirq, hardirq */
+
+static const int copy_irq_lens[COPY_IRQ_NUM_MSGS] = { 100, 1500, 4097 };
+
+struct copy_irq_test_state {
+	const u8 *src[COPY_IRQ_NUM_MSGS];
+	u16 expected[COPY_IRQ_NUM_MSGS];
+	u8 *dst[COPY_IRQ_NUM_CTX];
+	atomic_t seqno;
+};
+
+static bool copy_irq_test_func(void *state_)
+{
+	struct copy_irq_test_state *state = state_;
+	u32 i = (u32)atomic_inc_return(&state->seqno) % COPY_IRQ_NUM_MSGS;
+	int ctx = in_hardirq() ? 2 : in_serving_softirq() ? 1 : 0;
+	const u8 *src = state->src[i];
+	u8 *dst = state->dst[ctx];
+	int len = copy_irq_lens[i];
+	u16 got;
+
+	got = fold_csum(csum_partial_copy_nocheck(src, dst, len));
+	return got == state->expected[i] && memcmp(src, dst, len) == 0;
+}
+
+static void test_csum_copy_interrupt_context(struct kunit *test)
+{
+	struct copy_irq_test_state state = { };
+	int i;
+
+	copy_fill_random();
+	for (i = 0; i < COPY_IRQ_NUM_MSGS; i++) {
+		/* Distinct, misaligned messages. */
+		state.src[i] = copy_src_area + i * (COPY_MAX_LEN + 64) + 2 * i + 1;
+		state.expected[i] = ref_csum_fold(state.src[i], copy_irq_lens[i]);
+	}
+	for (i = 0; i < COPY_IRQ_NUM_CTX; i++)
+		state.dst[i] = copy_dst_area + i * (COPY_MAX_LEN + 64) + 2 * i + 1;
+
+	kunit_run_irq_test(test, copy_irq_test_func, 100000, &state);
+}
+
+/* Optional throughput benchmark vs memcpy() + csum_partial(). */
+static void test_csum_copy_benchmark(struct kunit *test)
+{
+	static const int lens[] = {
+		16, 40, 64, 128, 256, 512, 1000, 1024, 1368, 1448, 1500,
+		2048, 4096, 16384,
+	};
+	static const int offs[][2] = { { 0, 0 }, { 1, 3 } };
+	int l, o, i, len, num_iters;
+	u64 t, t_fused, t_split;
+	u32 acc = 0; /* keeps every call alive via OPTIMIZER_HIDE_VAR() */
+
+	if (!IS_ENABLED(CONFIG_CHECKSUM_KUNIT_BENCHMARK))
+		kunit_skip(test, "not enabled");
+
+	copy_fill_random();
+
+	/* warm-up */
+	for (i = 0; i < 10000000; i += COPY_BENCH_MAX_LEN)
+		acc ^= (__force u32)csum_partial_copy_nocheck(copy_src_area,
+							      copy_dst_area,
+							      COPY_BENCH_MAX_LEN);
+	OPTIMIZER_HIDE_VAR(acc);
+
+	for (o = 0; o < ARRAY_SIZE(offs); o++) {
+		const u8 *src = copy_src_area + offs[o][0];
+		u8 *dst = copy_dst_area + offs[o][1];
+
+		for (l = 0; l < ARRAY_SIZE(lens); l++) {
+			len = lens[l];
+			num_iters = 10000000 / (len + 128);
+
+			preempt_disable();
+			t = ktime_get_ns();
+			for (i = 0; i < num_iters; i++)
+				acc ^= (__force u32)csum_partial_copy_nocheck(src, dst, len);
+			t_fused = ktime_get_ns() - t;
+			OPTIMIZER_HIDE_VAR(acc);
+
+			t = ktime_get_ns();
+			for (i = 0; i < num_iters; i++) {
+				memcpy(dst, src, len);
+				acc ^= (__force u32)csum_partial(dst, len, 0);
+			}
+			t_split = ktime_get_ns() - t;
+			OPTIMIZER_HIDE_VAR(acc);
+			preempt_enable();
+
+			kunit_info(test,
+				   "src+%d dst+%d len=%5d: csum_partial_copy_nocheck %6llu MB/s, memcpy+csum_partial %6llu MB/s\n",
+				   offs[o][0], offs[o][1], len,
+				   div64_u64((u64)len * num_iters * 1000,
+					     max_t(u64, t_fused, 1)),
+				   div64_u64((u64)len * num_iters * 1000,
+					     max_t(u64, t_split, 1)));
+		}
+	}
+	{
+		u32 ref_acc = acc;
+
+		OPTIMIZER_HIDE_VAR(acc);
+		KUNIT_ASSERT_EQ(test, acc, ref_acc);
+	}
+}
+
 static struct kunit_case __refdata checksum_test_cases[] = {
 	KUNIT_CASE(test_csum_fixed_random_inputs),
 	KUNIT_CASE(test_csum_all_carry_inputs),
 	KUNIT_CASE(test_csum_no_carry_inputs),
 	KUNIT_CASE(test_ip_fast_csum),
 	KUNIT_CASE(test_csum_ipv6_magic),
+	KUNIT_CASE(test_csum_copy_small_all_alignments),
+	KUNIT_CASE(test_csum_copy_large_unaligned),
+	KUNIT_CASE(test_csum_copy_patterns),
+	KUNIT_CASE(test_csum_copy_dst_guard_page),
+	KUNIT_CASE(test_csum_copy_zero_len),
+	KUNIT_CASE(test_csum_copy_interrupt_context),
+	KUNIT_CASE(test_csum_copy_benchmark),
 	{}
 };
 
 static struct kunit_suite checksum_test_suite = {
 	.name = "checksum",
 	.test_cases = checksum_test_cases,
+	.suite_init = checksum_suite_init,
+	.suite_exit = checksum_suite_exit,
 };
 
 kunit_test_suites(&checksum_test_suite);
-- 
2.53.0


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-20 22:03 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 22:03 [PATCH v3 0/2] arm64: csum: Add fused copy and Internet checksum Demian Shulhan
2026-11-20 22:03 ` [PATCH v3 1/2] " Demian Shulhan
2026-11-20 22:03 ` [PATCH v3 2/2] lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck() Demian Shulhan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®