From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 348C1A937 for ; Thu, 11 Dec 2025 13:46:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765460781; cv=none; b=Urryjpfzuqt3cqDT3OHj6FGFlrpKjFuXG3xi7t3z9Ikel8fRIEYAB+2sYYtc6Zk6y7daimCUly5/hyRhRUgr1ZEIoFZBVJGx9V7kcpvggG7DkhqLlkU5GrJiQx8fBbwwXk84th6dTsfG8zyut9mUiDDPu3bg7Apcsf6rBIdaINc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765460781; c=relaxed/simple; bh=cteSiBxyLCm25ScinzMyERljhxyzhRU2cUF+/nWsaCk=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=BFsUX7l01Iprzz3/nqUsHUnY3XDJe7eKUWWbEYJFuPqy3q0bcEm55dBJ5CbCb5VQU7DLT5V/EiehjpavkmeSyOG1pm2wtcfKsH1oQKMkBtkhW2e3qGyaRmotKiFeOXo0uozlsH1k50bLzoqjZZQaiXpPUSjUmCqnfw4z0aCPSPk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BDM5MHAZ; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=I5flS44K; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BDM5MHAZ"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="I5flS44K" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1765460778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KW7IIg1ZAhy1yX6x6fK2/pEDvpxlIRaCqfVt49AdYhg=; b=BDM5MHAZMpb4JSEF7DPWBLfPY4ZFz/18vdwOOyICA8rGaWH72C8ZQS6y8hI+uePbM/oNaf EYpw6FeROApYcaQL07fhYGCIFRHPFkUlHwom5k7KNuONNgUggWkb8z0NZmFGivlSKOwmh/ CB2OWZvfBsByaO1XObgBGD2attfhvPQ= Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-371-ORKbF0hPOlaTzWZNf53ecw-1; Thu, 11 Dec 2025 08:46:16 -0500 X-MC-Unique: ORKbF0hPOlaTzWZNf53ecw-1 X-Mimecast-MFC-AGG-ID: ORKbF0hPOlaTzWZNf53ecw_1765460776 Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-340bc4ef67fso66831a91.3 for ; Thu, 11 Dec 2025 05:46:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1765460776; x=1766065576; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:from:to :cc:subject:date:message-id:reply-to; bh=KW7IIg1ZAhy1yX6x6fK2/pEDvpxlIRaCqfVt49AdYhg=; b=I5flS44K5357mhV2UuV7cbsPX5px6MnnHAqKtWtbsOso5Z4iO9fPk4oa9m5/gdF3hs B+fE7Hp1B5Ctx6iEoBNTVe33wMNa6Jt+upbfJLjLEdnk8hYfJmErOHecGIBJ8BgKTh3q rOJAMaZHg/m34ELIxChXKecDuAdGXLmkqVMRp0qraMaot/NZI0FLKsEiTuIxQ1wVaceB HHjA3iQGiWDcabptL5NkbneJf/eu4+hMv/LjDhr1WPu3h1pIzItUlVStXeEh0g7/zNdj LIEq5EhFdWxvNjxg71VuTVay3EHy1+Z3mrjx20rKcezRGDG9tNPY0STGAaKfIl5ya8xJ i6og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765460776; x=1766065576; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KW7IIg1ZAhy1yX6x6fK2/pEDvpxlIRaCqfVt49AdYhg=; b=VPCpzfpYH3GPn16d4TGzNgaZa9peYgIjaMiR9S/xojaCXyExfSuk3TAkSAvNkQPr/J Xrhhcusa86HqnpMNzr+txBP5wOCVmaciTfS4pOoQ3j90nIVZuBV/4h92gq+080QFZ+hD 8k6q+dC7ktjpGlH1Yae/+4J3aH+bsB/vyLsbkzc1pZuLZG6wky4+Xe6cJbu1IbelFS9q DoF1IJsL6HNUUgiGeS91E/Bvp/Af2ap335sMavDkgDyAm/7FWp9SwzSUQe3FbUa7cL9n 8p5fejlzeUexTmIYkrpj4Yw949YnCSr9y0YH6e724M3NvdhSPHwQrRUlb49obScecszF UA5A== X-Gm-Message-State: AOJu0YyBA31KiEkt9kf0j1NqJmbvtF4nj9fMplarXk8yD9Qe3j53yAYb 5BQ5IAImx5aQTxjxxrLB1ryotaxt4RLWNdLBcv/kmxvOjj/kSbGnqCGmKEX6S1/tzVOKKJKqCaF EdC/9kqwSUmgw0SElanxvjC0MvvtWfjdIjbFlhJOCgikXG+q/BkTmm0UsIiNpGKU/ZA== X-Gm-Gg: AY/fxX7O89TAK9VkLtC1Ci4WZ72qq2ta3UjrRQTVwPciWp5q0SpyRnUeStyBU23O5iA FYRo14ktjVQydE8C5mfs8Z3oARSznzQX2+xNkUZlkQNbdhlvzvPoophfX93b50Uq7R4RHfZeE9X 2a5Sl5BfCAu3zku+H5HZw8DsukDPrQtTfHy4NuCtiw691o1BYB03j7nopheZRDgWZF6f2ZUnzbO O0+YlJUWTTpdEYPN0v6ljTNl5aWy6mnHMHeUOuROOjdGMDObCmdDkBptIMWF2Lt0nzs19z2JD3z LWw0GKoz5wh86FgMaQodriPKjAZCphM+fRJNiJR9VW+zcgNZK6OCb+kox/ELuI0yXROE2eLp4gt lCn6g2JywSyse3l1ykeUXQ8I5 X-Received: by 2002:a05:7022:108:b0:11b:c86b:386a with SMTP id a92af1059eb24-11f29669e49mr5727919c88.5.1765460775640; Thu, 11 Dec 2025 05:46:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IEmdUExgraRk+1Ui2l8EGns5r6vwb6F14882sJ83+arvIQXimvQKTEx+Zgm0MwA7ElP2UIGaA== X-Received: by 2002:a05:7022:108:b0:11b:c86b:386a with SMTP id a92af1059eb24-11f29669e49mr5727882c88.5.1765460775003; Thu, 11 Dec 2025 05:46:15 -0800 (PST) Received: from [192.168.3.252] ([74.75.144.57]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-11f2e1bb3b4sm8073350c88.4.2025.12.11.05.46.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 11 Dec 2025 05:46:14 -0800 (PST) Message-ID: <358c63d6-5a25-462d-af04-1703bb7840e8@redhat.com> Date: Thu, 11 Dec 2025 08:46:08 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [tboot-devel] [PATCH 1/1] Disable CET when calling tboot shutdown procedure. From: Tony Camuso To: Bagas Sanjaya , ning.sun@intel.com, tboot-devel@lists.sourceforge.net Cc: linux-kernel@vger.kernel.org, rppt@kernel.org, tglx@linutronix.de, mingo@kernel.org, bp@alien8.de, michal.camacho.romero@linux.intel.com References: <20251017073619.547993-1-michal.camacho.romero@linux.intel.com> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 12/10/2025 1:10 PM, Tony Camuso wrote: > On 11/24/2025 7:34 PM, Bagas Sanjaya wrote: >> On Thu, Nov 13, 2025 at 09:37:14AM -0500, Tony Camuso wrote: >>> The tboot->shutdown_entry is effectively bios code and CET needs to be >>> disabled before calling it. >>> >>> It resolves TBOOT shutdown failure bug, reported on the SLES (SUSE Linux >>> Enterprise Server) 16.0 OS. OS power off, called by the "init 0" command, >>> was failing, due to activated Intel Control-Flow Enforcement Technology >>> (CET). >>> Disabling CET has allowed to execute OS and TBOOT shutdown properly. >> >> Are ``systemctl poweroff`` and ``shutdown -P`` are also affected? >> >> Confused... >> > > Yes, all shutdown methods on kernels launched with tboot, on systems that > expose the CPU ibt flag to kernels v6.12+ will cause the stack trace appended > below. > > The stack trace demonstrates that CET enforcement collides with legacy > BIOS shutdown code that lacks ENDBR markers. The kernel BUG at > cet.c:102 is a direct result of CET being active when jumping into > tboot->shutdown_entry. > > Legacy BIOS/tboot code without ENDBR now traps, requiring CET to be disabled > around that call. > > The patch: >     Prevents CET from falsely trapping on non-CET BIOS code. Need to clarify further: The patch prevents CET from trapping when tboot invokes the BIOS-provided shutdown_entry routine, which lacks ENDBR instructions. tboot side: In tboot_shutdown(), the kernel switches to the tboot page tables and then calls: shutdown = (void(*)(void))(unsigned long)tboot->shutdown_entry; shutdown(); That shutdown_entry pointer comes from the tboot structure, populated at boot. In the tboot project directory, see include/tboot.h In the kernel, see include/linux/tboot.h BIOS side: The actual routine behind shutdown_entry is implemented in BIOS/firmware. It’s not compiled with CET/IBT support, so it lacks the required ENDBR64 instruction at its entry point When CET is still enabled, the CPU enforces IBT. Jumping into that BIOS routine without ENDBR triggers a #CP (control protection exception), which is what the stack trace shows. So it is the BIOS shutdown_entry function itself that causes the trap, but only because tboot is handing control to it while CET is active. What happens: From the stack trace Missing ENDBR: 0x8041d0 kernel BUG at arch/x86/kernel/cet.c:102! RIP: 0010:0x8041d0 This shows the CPU trapping on entry into the shutdown routine at address 0x8041d0, which is the pointer stored in tboot->shutdown_entry The shutdown_entry field is explicitly documented as the physical address of the BIOS shutdown routine. This structure is populated by tboot at boot. > >     Maintains system stability during shutdown. > >     Preserves CET protection elsewhere, only disabling it for the >     narrow window where legacy firmware must run. > > > [  169.420078] reboot: Power down > [  169.427516] Missing ENDBR: 0x8041d0 > [  169.431128] ------------[ cut here ]------------ > [  169.435805] kernel BUG at arch/x86/kernel/cet.c:102! > [  169.440840] Oops: invalid opcode: 0000 [#1] SMP NOPTI > [  169.445966] CPU: 0 UID: 0 PID: 3354 Comm: poweroff Kdump: loaded Not tainted 6.12.0-124.8.1.el10_1.x86_64 #1 PREEMPT(voluntary) > [  169.457580] Hardware name: Dell Inc. PowerEdge R570/03TJR3, BIOS 1.2.1 01/23/2025 > [  169.465113] RIP: 0010:exc_control_protection+0x18c/0x190 > [  169.470490] Code: 1c ff 45 31 c9 49 89 d8 b9 09 00 00 00 48 8b 93 80 00 00 00 be 63 00 00 00 48 c7 c7 a4 85 e5 a4 e8 79 92 30 ff e9 02 ff ff ff <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f > [  169.489292] RSP: 0018:ff55b3cba167fa88 EFLAGS: 00010002 > [  169.494581] RAX: 0000000000000017 RBX: ff55b3cba167faa8 RCX: 00000000ffff7fff > [  169.501765] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 > [  169.508949] RBP: 0000000000000003 R08: 0000000000000000 R09: ffffffffa59e2b08 > [  169.516132] R10: ffffffffa5922ac8 R11: 0000000000000003 R12: 0000000000000000 > [  169.523316] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 > [  169.530514] FS:  00007f5f9a122140(0000) GS:ff39175c2de00000(0000) knlGS:0000000000000000 > [  169.538659] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [  169.544454] CR2: 0000559386dc5320 CR3: 000000010fbe2000 CR4: 0000000000f71ef0 > [  169.551651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [  169.558835] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 > [  169.566019] PKRU: 55555554 > [  169.568784] Call Trace: > [  169.571295]  > [  169.573458]  ? show_trace_log_lvl+0x1b0/0x2f0 > [  169.577880]  ? show_trace_log_lvl+0x1b0/0x2f0 > [  169.582301]  ? asm_exc_control_protection+0x26/0x30 > [  169.587244]  ? exc_control_protection+0x18c/0x190 > [  169.592011]  ? __die_body.cold+0x8/0x12 > [  169.595910]  ? die+0x2e/0x50 > [  169.598863]  ? do_trap+0xca/0x110 > [  169.602243]  ? do_error_trap+0x65/0x80 > [  169.606049]  ? exc_control_protection+0x18c/0x190 > [  169.610816]  ? exc_invalid_op+0x50/0x70 > [  169.614715]  ? exc_control_protection+0x18c/0x190 > [  169.619482]  ? asm_exc_invalid_op+0x1a/0x20 > [  169.623728]  ? exc_control_protection+0x18c/0x190 > [  169.628496]  ? exc_control_protection+0x14f/0x190 > [  169.633263]  asm_exc_control_protection+0x26/0x30 > [  169.638030] RIP: 0010:0x8041d0 > [  169.641142] Code: Unable to access opcode bytes at 0x8041a6. > [  169.646857] RSP: 0018:ff55b3cba167fb50 EFLAGS: 00010007 > [  169.652144] RAX: 00000000008041d0 RBX: 0000000000000000 RCX: 0000000000000005 > [  169.659341] RDX: 00c6e8a7c0000000 RSI: 0000000000000001 RDI: ffffffffff1ff000 > [  169.666525] RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000000ffff > [  169.673709] R10: 0000000000000000 R11: ffffffffffff0000 R12: 0000000000002001 > [  169.680906] R13: ffffffffa5ae02c8 R14: 00000000ffffffff R15: 0000000000000000 > [  169.688091]  ? tboot_shutdown+0x5b/0x140 > [  169.692084]  ? tboot_sleep+0x12c/0x140 > [  169.695890]  ? acpi_os_enter_sleep+0x2b/0x60 > [  169.700221]  ? acpi_hw_legacy_sleep+0x140/0x1c0 > [  169.704816]  ? acpi_power_off+0x16/0x40 > [  169.708715]  ? sys_off_notify+0x48/0x70 > [  169.712615]  ? notifier_call_chain+0x5a/0xd0 > [  169.716943]  ? atomic_notifier_call_chain+0x32/0x50 > [  169.721885]  ? do_kernel_power_off+0x3e/0x50 > [  169.726213]  ? native_machine_power_off+0x21/0x40 > [  169.730983]  ? __do_sys_reboot+0x1d2/0x240 > [  169.735151]  ? do_syscall_64+0x7d/0x160 > [  169.739053]  ? syscall_exit_work+0xf3/0x120 > [  169.743302]  ? syscall_exit_to_user_mode+0x32/0x190 > [  169.748243]  ? do_syscall_64+0x89/0x160 > [  169.752143]  ? __count_memcg_events+0xdf/0x170 > [  169.756645]  ? handle_mm_fault+0x256/0x370 > [  169.760813]  ? do_user_addr_fault+0x347/0x640 > [  169.765235]  ? exc_page_fault+0x73/0x160 > [  169.769228]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e >