[PATCH v2] x86: add hintable NOPs emulation

[Marcos Del Sol Vives <marcos@orca.pet>]

Hintable NOPs are a series of instructions introduced by Intel with the
Pentium Pro (i686), and described in US patent US5701442A.

These instructions were reserved to allow backwards-compatible changes
in the instruction set possible, by having old processors treat them as
variable-length NOPs, while having other semantics in modern processors.

Some modern uses are:
 - Multi-byte/long NOPs
 - Indirect Branch Tracking (ENDBR32)
 - Shadow Stack (part of CET)

Some processors advertising i686 compatibility lack full support for
them, which may cause #UD to be incorrectly triggered, crashing software
that uses then with an unexpected SIGILL.

One such software is sudo in Debian bookworm, which is compiled with
GCC -fcf-protection=branch and contains ENDBR32 instructions. It crashes
on my Vortex86DX3 processor and VIA C3 Nehalem processors [1].

This patch is a much simplified version of my previous patch for x86
instruction emulation [2], that only emulates hintable NOPs.

When #UD is raised, it checks if the opcode corresponds to a hintable NOP
in user space. If true, it warns the user via the dmesg and advances the
instruction pointer, thus emulating its expected NOP behaviour.

[1]: https://lists.debian.org/debian-devel/2023/10/msg00118.html
[2]: https://lore.kernel.org/all/20210626130313.1283485-1-marcos@orca.pet/

Signed-off-by: Marcos Del Sol Vives <marcos@orca.pet>
---
 arch/x86/Kconfig        | 29 +++++++++++++++++++++++++++++
 arch/x86/kernel/traps.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 58d890fe2100..a6daebdc2573 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1286,6 +1286,35 @@ config X86_IOPL_IOPERM
 	  ability to disable interrupts from user space which would be
 	  granted if the hardware IOPL mechanism would be used.
 
+config X86_HNOP_EMU
+	bool "Hintable NOPs emulation"
+	depends on X86_32
+	default y
+	help
+	  Hintable NOPs are a series of instructions introduced by Intel with
+	  the Pentium Pro (i686), and described in US patent US5701442A.
+
+	  These instructions were reserved to allow backwards-compatible
+	  changes in the instruction set possible, by having old processors
+	  treat them as variable-length NOPs, while having other semantics in
+	  modern processors.
+
+	  Some modern uses are:
+	   - Multi-byte/long NOPs
+	   - Indirect Branch Tracking (ENDBR32)
+	   - Shadow Stack (part of CET)
+
+	  Some processors advertising i686 compatibility (such as Cyrix MII,
+	  VIA C3 Nehalem or DM&P Vortex86DX3) lack full support for them,
+	  which may cause SIGILL to be incorrectly raised in user space when
+	  a hintable NOP is encountered.
+
+	  Say Y here if you want the kernel to emulate them, allowing programs
+	  that make use of them to run transparently on such processors.
+
+	  This emulation has no performance penalty for processors that
+	  properly support them, so if unsure, enable it.
+
 config TOSHIBA
 	tristate "Toshiba Laptop support"
 	depends on X86_32
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 36354b470590..22b51c4186e7 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -295,12 +295,45 @@ DEFINE_IDTENTRY(exc_overflow)
 	do_error_trap(regs, 0, "overflow", X86_TRAP_OF, SIGSEGV, 0, NULL);
 }
 
+static bool handle_hnop(struct pt_regs *regs)
+{
+	unsigned char buf[MAX_INSN_SIZE];
+	unsigned long nr_copied;
+	struct insn insn;
+
+	if (!IS_ENABLED(CONFIG_X86_HNOP_EMU))
+		return false;
+
+	nr_copied = insn_fetch_from_user(regs, buf);
+	if (nr_copied <= 0)
+		return false;
+
+	if (!insn_decode_from_regs(&insn, regs, buf, nr_copied))
+		return false;
+
+	/* Hintable NOPs cover 0F 18 to 0F 1F */
+	if (insn.opcode.bytes[0] != 0x0F ||
+	    insn.opcode.bytes[1] < 0x18 || insn.opcode.bytes[1] > 0x1F)
+		return false;
+
+	pr_warn_once("%s[%d] uses hintable NOPs that your processor does not support.\n"
+		     "The kernel is emulating them; the performance of this "
+		     "and other executables using them will be impacted.\n",
+		     current->comm, task_pid_nr(current));
+
+	regs->ip += insn.length;
+	return true;
+}
+
 #ifdef CONFIG_X86_F00F_BUG
 void handle_invalid_op(struct pt_regs *regs)
 #else
 static inline void handle_invalid_op(struct pt_regs *regs)
 #endif
 {
+	if (user_mode(regs) && handle_hnop(regs))
+		return;
+
 	do_error_trap(regs, 0, "invalid opcode", X86_TRAP_UD, SIGILL,
 		      ILL_ILLOPN, error_get_trap_addr(regs));
 }
-- 
2.34.1