[PATCH] x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

[PATCH] x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

[Danish Khateeb]

Since commit 1b1f3b3e1b39 ("x86/entry: Get rid of the sys_ni_syscall()
indirection"), an invalid syscall number leaves regs->ax alone, relying
on the entry code having set it to -ENOSYS. But a tracer stopped at
syscall entry can change regs->ax. The value it wrote is now returned
for any out-of-range number, while a number that hits a hole in the
syscall table still ends up in sys_ni_syscall() and returns -ENOSYS.

For example, with the tracer setting regs->ax to -9999 at entry, 64-bit
syscall 400 (a hole) returns -ENOSYS, but 473 (past the end) and -2
return -9999. The int $0x80 path behaves the same way. v7.2 returns
-ENOSYS for all of them.

This undoes commit b337b4965e3a ("x86/entry: Treat out of range and gap
system calls the same"), which made -1 the only invalid number that
keeps the return value. The syscall_numbering_64 selftest checks this
under ptrace and has failed since v7.3-rc1.

Restore the -1 check in the 64-bit and 32-bit paths, setting -ENOSYS
directly rather than calling sys_ni_syscall(). -1 still keeps the
return value, so a tracer or a tracepoint program that skips the
syscall by setting -1 is not affected.

Fixes: 1b1f3b3e1b39 ("x86/entry: Get rid of the sys_ni_syscall() indirection")
Assisted-by: LLM
Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
---

Notes:
    Tested on v7.3-rc5 x86_64 under virtme-ng (KASAN, lockdep), with
    CONFIG_X86_X32_ABI both off and on:
    
    - selftests/x86/syscall_numbering_64: 37800 failures before, all in the
      ptrace passes that change regs->ax. All checks pass after, including
      the x32 checks with CONFIG_X86_X32_ABI=y.
    - A small tracer that sets regs->ax to -9999 at the syscall-entry stop:
      before, 473, 1000 and -2 returned -9999 while the hole 400 returned
      -ENOSYS. After, all of them return -ENOSYS and only -1 returns -9999,
      as on v7.2. The same holds for int $0x80 (hole 388).
    - ptrace_syscall, syscall_nt, single_step_syscall, syscall_arg_fault and
      check_initial_reg_state (32- and 64-bit), sysret_rip_64 and seccomp_bpf
      (111/111) pass before and after. No new kernel warnings.
    
    The native 32-bit kernel was only build-tested (i386_defconfig). gcc W=1,
    clang W=1 and sparse are clean.

 arch/x86/entry/syscall_32.c |  6 ++++++
 arch/x86/entry/syscall_64.c | 12 +++++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/arch/x86/entry/syscall_32.c b/arch/x86/entry/syscall_32.c
index 91123a90710c..ea38537b22c2 100644
--- a/arch/x86/entry/syscall_32.c
+++ b/arch/x86/entry/syscall_32.c
@@ -77,6 +77,12 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs, unsigned
 	if (likely(nr < IA32_NR_syscalls)) {
 		nr = array_index_nospec(nr, IA32_NR_syscalls);
 		regs->ax = ia32_sys_call(regs, (unsigned int)nr);
+	} else if ((long)nr != -1) {
+		/*
+		 * ptrace or seccomp may have changed the return value. Only
+		 * -1 keeps it; any other invalid number fails with -ENOSYS.
+		 */
+		regs->ax = -ENOSYS;
 	}
 }
 
diff --git a/arch/x86/entry/syscall_64.c b/arch/x86/entry/syscall_64.c
index 966d3d1b7586..ac5ffa0a502a 100644
--- a/arch/x86/entry/syscall_64.c
+++ b/arch/x86/entry/syscall_64.c
@@ -64,7 +64,7 @@ static __always_inline bool do_syscall_x64(struct pt_regs *regs, unsigned long n
 	return false;
 }
 
-static __always_inline void do_syscall_x32(struct pt_regs *regs, unsigned long nr)
+static __always_inline bool do_syscall_x32(struct pt_regs *regs, unsigned long nr)
 {
 	/* Adjust the starting offset of the table */
 	nr -= __X32_SYSCALL_BIT;
@@ -72,7 +72,9 @@ static __always_inline void do_syscall_x32(struct pt_regs *regs, unsigned long n
 	if (IS_ENABLED(CONFIG_X86_X32_ABI) && likely(nr < X32_NR_syscalls)) {
 		nr = array_index_nospec(nr, X32_NR_syscalls);
 		regs->ax = x32_sys_call(regs, (unsigned int)nr);
+		return true;
 	}
+	return false;
 }
 
 /* Returns true to return using SYSRET, or false to use IRET */
@@ -81,8 +83,12 @@ __visible noinstr bool do_syscall_64(struct pt_regs *regs, long nr)
 	if (likely(syscall_enter_from_user_mode_randomize_stack(regs, &nr))) {
 		instrumentation_begin();
 
-		if (!do_syscall_x64(regs, nr))
-			do_syscall_x32(regs, nr);
+		/*
+		 * ptrace or seccomp may have changed the return value. Only
+		 * -1 keeps it; any other invalid number fails with -ENOSYS.
+		 */
+		if (!do_syscall_x64(regs, nr) && !do_syscall_x32(regs, nr) && nr != -1)
+			regs->ax = -ENOSYS;
 
 		instrumentation_end();
 	}

base-commit: 72d3fcf802c45d00b300f25b848a93c3a2bd7c7e
-- 
2.55.0

Re: [PATCH] x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

[Ting-Han Hou]

On Tue, Sep 29, 2026 at 12:33:45PM -0500, Danish Khateeb wrote:
> x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

I tested this on top of current mainline a5ebb76233b7
(v7.3-rc6-386-ga5ebb76233b7): x86_64 defconfig + kvm_guest.config,
booted in QEMU/KVM (-cpu host), with the x86 selftests built from the
same tree.

selftests/x86/syscall_numbering_64:

  without the patch:
    [FAIL]  A total of 37800 system calls had incorrect behavior
  (12600 each in the "modifying the default return", "clobbering the
  top 32 bits" and "sign-extending the syscall number" ptrace passes,
  e.g. "x32 syscall 0:-64 returned -9999, but it should have returned
  -ENOSYS")

  with the patch:
    [OK]    All system calls succeeded or failed as expected

It also passes with CONFIG_X86_X32_ABI=y ("x32 is supported").

The selftest does not cover the syscall_32.c hunk, so I also used a
small tracer that sets the return register to -9999 at the
syscall-entry stop of int $0x80, from both a 32-bit and a 64-bit
process. Without the patch, numbers 1000, 0x7fffffff, -2 and -1000 all
return -9999. With the patch they return -ENOSYS. -1 returns -9999 in
both cases, as expected.

ptrace_syscall_{32,64}, syscall_nt_{32,64}, syscall_arg_fault_{32,64},
single_step_syscall_{32,64}, check_initial_reg_state_{32,64},
sysret_rip_64, sigreturn_64 and test_syscall_vdso_32 pass with and
without the patch. No warnings in dmesg on the patched kernels. I did
not test a native 32-bit kernel.

The kernel test robot report for this is
https://lore.kernel.org/all/202609210251.4f7bf339-lkp@intel.com/ ,
which asked for:

Reported-by: kernel test robot <yi1.lai@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202609210251.4f7bf339-lkp@intel.com

I used an AI assistant (Claude Code) to help build and run these tests
and to read through the change; I checked the results myself.

Tested-by: Ting-Han Hou <ue081723@gmail.com>
Reviewed-by: Ting-Han Hou <ue081723@gmail.com>