[PATCH 2/7] x86/word-at-a-time: Instrument load_unaligned_zeropad()

[Bill Wendling <morbo@google.com>]

load_unaligned_zeropad() is implemented as inline asm plus an
exception table entry on every architecture, so the compiler never
sees the actual memory access and cannot emit KASAN/KCSAN
instrumentation for it.

Resurrect the instrumentation that was lost when migrating from
read_word_at_a_time() by renaming load_unaligned_zeropad() to
arch_load_unaligned_zeropad() and adding a common instrumented
load_unaligned_zeropad() in the new
asm-generic/word-at-a-time-instrumented.h, pulled in from the bottom
of asm/word-at-a-time.h. This mirrors the existing
arch_atomic_*()/atomic_*() split in linux/atomic/atomic-instrumented.h,
where the arch_* version is the raw, uninstrumented primitive and the
plain name is the instrumented one everyone should call.

The wrapper is __always_inline, so it compiles down to exactly the
same inline asm as before plus instrument_read() -- confirmed via
objdump that no calls or relocations to load_unaligned_zeropad(),
instrument_read(), or the KASAN/KCSAN check functions remain when
KASAN and KCSAN are both disabled.

Convert x86 to the new split here. Follow-up patches convert the
remaining architectures (arm64, arm, riscv, s390, powerpc) that
implement load_unaligned_zeropad().

Fixes: d94c12bd97d5 ("string: Add load_unaligned_zeropad() code path to sized_strscpy()")
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Bill Wendling <morbo@google.com>
---
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Kees Cook <kees@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Peter Collingbourne <peter@pcc.me.uk>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Cc: linux-riscv@lists.infradead.org
Cc: x86@kernel.org
---
 arch/x86/include/asm/word-at-a-time.h         |  4 ++-
 .../asm-generic/word-at-a-time-instrumented.h | 25 +++++++++++++++++++
 2 files changed, 28 insertions(+), 1 deletion(-)
 create mode 100644 include/asm-generic/word-at-a-time-instrumented.h

diff --git a/arch/x86/include/asm/word-at-a-time.h b/arch/x86/include/asm/word-at-a-time.h
index 422a47746657..483d27dc96b2 100644
--- a/arch/x86/include/asm/word-at-a-time.h
+++ b/arch/x86/include/asm/word-at-a-time.h
@@ -67,7 +67,7 @@ static inline unsigned long find_zero(unsigned long mask)
  * and the next page not being mapped, take the exception and
  * return zeroes in the non-existing part.
  */
-static inline unsigned long load_unaligned_zeropad(const void *addr)
+static inline unsigned long arch_load_unaligned_zeropad(const void *addr)
 {
 	unsigned long ret;
 
@@ -81,4 +81,6 @@ static inline unsigned long load_unaligned_zeropad(const void *addr)
 	return ret;
 }
 
+#include <asm-generic/word-at-a-time-instrumented.h>
+
 #endif /* _ASM_WORD_AT_A_TIME_H */
diff --git a/include/asm-generic/word-at-a-time-instrumented.h b/include/asm-generic/word-at-a-time-instrumented.h
new file mode 100644
index 000000000000..8a66d53bf538
--- /dev/null
+++ b/include/asm-generic/word-at-a-time-instrumented.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_GENERIC_WORD_AT_A_TIME_INSTRUMENTED_H
+#define _ASM_GENERIC_WORD_AT_A_TIME_INSTRUMENTED_H
+
+#include <linux/instrumented.h>
+
+/*
+ * Each arch's asm/word-at-a-time.h defines arch_load_unaligned_zeropad()
+ * as inline asm plus an exception table entry, so the compiler never sees
+ * the actual memory access and cannot emit KASAN/KCSAN instrumentation
+ * for it. Provide the common, instrumented load_unaligned_zeropad() that
+ * every caller uses by wrapping the arch version with instrument_read().
+ *
+ * This is marked __always_inline, and arch_load_unaligned_zeropad() is
+ * itself a tiny inline-asm-only function, so this compiles down to the
+ * same inline asm as before plus instrument_read() (which itself compiles
+ * to nothing when KASAN and KCSAN are both disabled) -- no actual call.
+ */
+static __always_inline unsigned long load_unaligned_zeropad(const void *addr)
+{
+	instrument_read(addr, 1);
+	return arch_load_unaligned_zeropad(addr);
+}
+
+#endif /* _ASM_GENERIC_WORD_AT_A_TIME_INSTRUMENTED_H */
-- 
2.47.3