From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7A74408624; Thu, 30 Jul 2026 11:04:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785409442; cv=none; b=lRxmaI0yquiXJxJdbtbE6XbmhXsSjdk3VpDmXYuVY96XA1ELX1VX/lLQxz93Fif9q7Yv2/oz4mR2XgAiP0+kn79gsfyYaQvZlCo2REiAon4kXxtWUtkgrdiYbvIaJhXsvgKuwI/5YC5yu0bZdx+NXetpUHSuT7rc7OGMrV4ttPg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785409442; c=relaxed/simple; bh=t7mGqlhCGvnxD+LIxMaDte2x0c0sdcIybZ7iDkYPGjM=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=T4sZZx8WAV+0r/SuQ+f7dwgnxSPjZOjSLUYzTNvED8E9CZp+8uybGrA+TZAUbayAaHCuHLMkUykqGx4ookDz1cO7phiW3LelK3fXaUOBE6qQRsi5Gdw/E/8vS8sbe92h8EuhgwEGABYVZoWIs9okVMhOv0PuOrPqp63FyigWUNY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=0bszRDQQ; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=n/N8iYbG; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="0bszRDQQ"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="n/N8iYbG" Date: Thu, 30 Jul 2026 11:03:57 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785409439; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FQVRWB2m3o6acA15+WMZPEaPO9so7XW4MEFrWduJzRk=; b=0bszRDQQrbLPAtwDJfIcvu4YZEb8x0WTQZHRslQDKgGJOIAbUwmidWOR2rWpiwc5SdMEO/ 2kXfdJbTU/OzW6wR+EGk9EXdwiiJ6Y4R06UZlSDvJNh8XHJPHzqW4u9H1EOb0xnLX5Eag1 eUm3buxGm7XJG8nsD/osEqyfGLfCMFGpPkGB3/HN9lSzeznPuPiQpRwcCSBnZ6kfAiMmVP wokkUrOCuszkjLkr8GqQIoToMj3zg5M3psnnOpRygw/B6yWytjM1E/BmIFAszvIWAQiBQ0 +N4ZaEjt5EcSbCOEhVlxbbCF1LZXFpb6ZlwqAT63oCRnE9FqnfcpswR60GYAMw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785409439; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FQVRWB2m3o6acA15+WMZPEaPO9so7XW4MEFrWduJzRk=; b=n/N8iYbGBMfpUssPeNLwXq7/AP3H4fkIL8gRaNg2mjexBsMfCxuQcAMQEXTA2/fvjQJ1lR ithh3X00XwAv8jDg== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/core] x86/kcfi: Optimize call sequence Cc: Scott D Constable , "Peter Zijlstra (Intel)" , "Borislav Petkov (AMD)" , Nathan Chancellor , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260612071506.GQ187714@noisy.programming.kicks-ass.net> References: <20260612071506.GQ187714@noisy.programming.kicks-ass.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <178540943737.112641.6283931680368927637.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the x86/core branch of tip: Commit-ID: b2250d63e642228911db80238f7e437dc405cc63 Gitweb: https://git.kernel.org/tip/b2250d63e642228911db80238f7e437dc40= 5cc63 Author: Peter Zijlstra AuthorDate: Thu, 11 Jun 2026 21:51:13 +02:00 Committer: Peter Zijlstra CommitterDate: Thu, 30 Jul 2026 12:59:22 +02:00 x86/kcfi: Optimize call sequence As noted in commit 85a2d4a890dc ("x86,ibt: Use UDB instead of 0xEA") Jcc shou= ld be assumed not-taken, however the normal kCFI (ABI) emits the following seque= nce: movl $(-hash), %r10d addl -15(%r11), %r10d je 1f ud2 1: cs call __x86_indirect_thunk_r11 (when used in conjunction with -mretpoline-external-thunk). Notably, the Jcc here is always taken, resulting in lower throughput than wou= ld be ideal. Replace it with the following sequence on boot: movl $(-hash), %r10d addl -15(%r11), %r10d jne . + 3 test $0xd6, %al cs call __x86_indirect_thunk_r11 This jumps to the UDB instruction used as an immediate byte in the test instruction. The test instruction will clobber eflags, but that is immaterial, eflags is already changed by the preceding addl. Intel recommends the FineIBT sequence on platforms that support IBT; older platforms are still widely used and would benefit from this. An earlier PoC was benchmarked by Scott: Indirect branch miss rate (br_misp_retired.indirect:k / br_inst_retired.indir= ect:k) BHI_DIS_S=3D1 Benchmark Baseline IBT kCFI kCFI-o= pt ---------------------------------------------------------------------------= -- iperf3 UDP 0.103764 0.103180 0.104311 0.1029= 45 hackbench 0.000885 0.000876 0.001996 0.0008= 26 lmbench syscall 0.005089 0.004486 0.016990 0.0058= 52 lmbench fork+exit 0.018454 0.019176 0.031085 0.0151= 53 lmbench fork+exec 0.017147 0.021613 0.029129 0.0163= 37 redis 0.032220 0.032655 0.045540 0.0279= 46 nginx+wrk 0.109033 0.112765 0.132557 0.1024= 17 fio randread 0.009704 0.009620 0.008548 0.0009= 62 fio seqwrite 0.006927 0.006707 0.019372 0.0045= 90 kbuild 0.056748 0.057324 0.064640 0.0481= 36 BHI_DIS_S=3D0 Benchmark Baseline IBT kCFI kCFI-o= pt ---------------------------------------------------------------------------= -- iperf3 UDP 0.000077 0.000106 0.000186 0.0000= 73 hackbench 0.000123 0.000132 0.000367 0.0000= 97 lmbench syscall 0.023259 0.018319 0.040903 0.0127= 72 lmbench fork+exit 0.011494 0.011887 0.029079 0.0164= 15 lmbench fork+exec 0.037782 0.038994 0.055378 0.0263= 81 redis 0.002481 0.003152 0.017073 0.0001= 84 nginx+wrk 0.015478 0.016266 0.033637 0.0002= 68 fio randread 0.009836 0.007949 0.007096 0.0001= 43 fio seqwrite 0.014587 0.014165 0.041792 0.0021= 57 kbuild 0.055774 0.055249 0.062590 0.0465= 46 Suggested-by: Scott D Constable Signed-off-by: Peter Zijlstra (Intel) Acked-by: Borislav Petkov (AMD) Tested-by: Nathan Chancellor Link: https://patch.msgid.link/20260612071506.GQ187714%40noisy.programming.ki= cks-ass.net --- arch/x86/kernel/alternative.c | 21 ++++++++++++++++++++- arch/x86/kernel/cfi.c | 11 +++++++++-- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 62936a3..ba7d205 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1356,6 +1356,20 @@ early_param("cfi", cfi_parse_cmdline); * "Make conditional jumps most often not taken: The efficiency and through= put * for not-taken branches is better than for taken branches on most * processors. Therefore, it is good to place the most frequent branch fir= st" + * + * NOTE: Update the kCFI caller sequence to make use of this observation: + * + * kCFI kCFI-OPT + * + * caller: caller: + * movl $(-0x12345678),%r10d // 6 movl $(-0x12345678),%r10d // 6 + * addl $-15(%r11),%r10d // 4 addl $-15(%r11),%r10d // 4 + * je 1f // 2 jne . + 3 // 2 + * ud2 // 2 test $0xd6, %al // 2 + * 1: cs call __x86_indirect_thunk_r11 // 6 1: cs call __x86_indirect_thun= k_r11 // 6 + * + * This new test clobbers eflags, but those are clobbered by the hash test + * anyway. */ =20 /* @@ -1518,8 +1532,9 @@ static int cfi_disable_callers(s32 *start, s32 *end) static int cfi_enable_callers(s32 *start, s32 *end) { /* - * Re-enable kCFI, undo what cfi_disable_callers() did. + * Re-enable (and update) kCFI, undo what cfi_disable_callers() did. */ + const u8 udne[] =3D { 0x75, 0x01, 0xa8, 0xd6 }; const u8 mov[] =3D { 0x41, 0xba }; s32 *s; =20 @@ -1532,6 +1547,10 @@ static int cfi_enable_callers(s32 *start, s32 *end) if (!hash) /* nocfi callers */ continue; =20 + /* + * See the kCFI/FineIBT comment above -- update note. + */ + text_poke_early(addr + 10, udne, 4); text_poke_early(addr, mov, 2); } =20 diff --git a/arch/x86/kernel/cfi.c b/arch/x86/kernel/cfi.c index 638eb5c..1df26df 100644 --- a/arch/x86/kernel/cfi.c +++ b/arch/x86/kernel/cfi.c @@ -72,8 +72,15 @@ enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) =20 switch (cfi_mode) { case CFI_KCFI: - if (!is_cfi_trap(addr)) - return BUG_TRAP_TYPE_NONE; + if (!is_cfi_trap(addr)) { + /* + * The updated kCFI sequence has "test $0xd6, %al" instead of + * "ud2", adjust the offset. + */ + addr -=3D 1; + if (!is_cfi_trap(addr)) + return BUG_TRAP_TYPE_NONE; + } =20 if (!decode_cfi_insn(regs, &target, &type)) return report_cfi_failure_noaddr(regs, addr);