Re: [tboot-devel] [PATCH 1/1] Disable CET when calling tboot shutdown procedure.

[Tony Camuso <tcamuso@redhat.com>]

On 12/11/25 8:46 AM, Tony Camuso wrote:
> 
> On 12/10/2025 1:10 PM, Tony Camuso wrote:
>> On 11/24/2025 7:34 PM, Bagas Sanjaya wrote:
>>> On Thu, Nov 13, 2025 at 09:37:14AM -0500, Tony Camuso wrote:
>>>> The tboot->shutdown_entry is effectively bios code and CET needs to be
>>>> disabled before calling it.
>>>>
>>>> It resolves TBOOT shutdown failure bug, reported on the SLES (SUSE 
>>>> Linux
>>>> Enterprise Server) 16.0 OS. OS power off, called by the "init 0" 
>>>> command,
>>>> was failing, due to activated Intel Control-Flow Enforcement Technology
>>>> (CET).
>>>> Disabling CET has allowed to execute OS and TBOOT shutdown properly.
>>>
>>> Are ``systemctl poweroff`` and ``shutdown -P`` are also affected?
>>>
>>> Confused...
>>>
>>
>> Yes, all shutdown methods on kernels launched with tboot, on systems that
>> expose the CPU ibt flag to kernels v6.12+ will cause the stack trace 
>> appended
>> below.
>>
>> The stack trace demonstrates that CET enforcement collides with legacy
>> BIOS shutdown code that lacks ENDBR markers. The kernel BUG at
>> cet.c:102 is a direct result of CET being active when jumping into
>> tboot->shutdown_entry.
>>
>> Legacy BIOS/tboot code without ENDBR now traps, requiring CET to be 
>> disabled
>> around that call.
>>
>> The patch:
>>      Prevents CET from falsely trapping on non-CET BIOS code.
> 
> Need to clarify further:
> 
> The patch prevents CET from trapping when tboot invokes the BIOS-provided
> shutdown_entry routine, which lacks ENDBR instructions.
> 
> tboot side:
>    In tboot_shutdown(), the kernel switches to the tboot page tables and
>    then calls:
>      shutdown = (void(*)(void))(unsigned long)tboot->shutdown_entry;
>      shutdown();
> 
>    That shutdown_entry pointer comes from the tboot structure, populated
>    at boot.
>      In the tboot project directory, see include/tboot.h
>      In the kernel, see include/linux/tboot.h
> 
> BIOS side:
>    The actual routine behind shutdown_entry is implemented in BIOS/ 
> firmware.
>    It’s not compiled with CET/IBT support, so it lacks the required ENDBR64
>    instruction at its entry point
> 
>    When CET is still enabled, the CPU enforces IBT. Jumping into that BIOS
>    routine without ENDBR triggers a #CP (control protection exception),
>    which is what the stack trace shows.
> 
>    So it is the BIOS shutdown_entry function itself that causes the trap,
>    but only because tboot is handing control to it while CET is active.
> 
> What happens:
>    From the stack trace
>      Missing ENDBR: 0x8041d0
>      kernel BUG at arch/x86/kernel/cet.c:102!
>      RIP: 0010:0x8041d0
>    This shows the CPU trapping on entry into the shutdown routine at 
> address
>    0x8041d0, which is the pointer stored in tboot->shutdown_entry
> 
>    The shutdown_entry field is explicitly documented as the physical 
> address
>    of the BIOS shutdown routine. This structure is populated by tboot at 
> boot.
> 
>>
>>      Maintains system stability during shutdown.
>>
>>      Preserves CET protection elsewhere, only disabling it for the
>>      narrow window where legacy firmware must run.
>>
>>
>> [  169.420078] reboot: Power down
>> [  169.427516] Missing ENDBR: 0x8041d0
>> [  169.431128] ------------[ cut here ]------------
>> [  169.435805] kernel BUG at arch/x86/kernel/cet.c:102!
>> [  169.440840] Oops: invalid opcode: 0000 [#1] SMP NOPTI
>> [  169.445966] CPU: 0 UID: 0 PID: 3354 Comm: poweroff Kdump: loaded 
>> Not tainted 6.12.0-124.8.1.el10_1.x86_64 #1 PREEMPT(voluntary)
>> [  169.457580] Hardware name: Dell Inc. PowerEdge R570/03TJR3, BIOS 
>> 1.2.1 01/23/2025
>> [  169.465113] RIP: 0010:exc_control_protection+0x18c/0x190
>> [  169.470490] Code: 1c ff 45 31 c9 49 89 d8 b9 09 00 00 00 48 8b 93 
>> 80 00 00 00 be 63 00 00 00 48 c7 c7 a4 85 e5 a4 e8 79 92 30 ff e9 02 
>> ff ff ff <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 
>> 66 0f
>> [  169.489292] RSP: 0018:ff55b3cba167fa88 EFLAGS: 00010002
>> [  169.494581] RAX: 0000000000000017 RBX: ff55b3cba167faa8 RCX: 
>> 00000000ffff7fff
>> [  169.501765] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 
>> 0000000000000001
>> [  169.508949] RBP: 0000000000000003 R08: 0000000000000000 R09: 
>> ffffffffa59e2b08
>> [  169.516132] R10: ffffffffa5922ac8 R11: 0000000000000003 R12: 
>> 0000000000000000
>> [  169.523316] R13: 0000000000000000 R14: 0000000000000000 R15: 
>> 0000000000000000
>> [  169.530514] FS:  00007f5f9a122140(0000) GS:ff39175c2de00000(0000) 
>> knlGS:0000000000000000
>> [  169.538659] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  169.544454] CR2: 0000559386dc5320 CR3: 000000010fbe2000 CR4: 
>> 0000000000f71ef0
>> [  169.551651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
>> 0000000000000000
>> [  169.558835] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 
>> 0000000000000400
>> [  169.566019] PKRU: 55555554
>> [  169.568784] Call Trace:
>> [  169.571295]  <TASK>
>> [  169.573458]  ? show_trace_log_lvl+0x1b0/0x2f0
>> [  169.577880]  ? show_trace_log_lvl+0x1b0/0x2f0
>> [  169.582301]  ? asm_exc_control_protection+0x26/0x30
>> [  169.587244]  ? exc_control_protection+0x18c/0x190
>> [  169.592011]  ? __die_body.cold+0x8/0x12
>> [  169.595910]  ? die+0x2e/0x50
>> [  169.598863]  ? do_trap+0xca/0x110
>> [  169.602243]  ? do_error_trap+0x65/0x80
>> [  169.606049]  ? exc_control_protection+0x18c/0x190
>> [  169.610816]  ? exc_invalid_op+0x50/0x70
>> [  169.614715]  ? exc_control_protection+0x18c/0x190
>> [  169.619482]  ? asm_exc_invalid_op+0x1a/0x20
>> [  169.623728]  ? exc_control_protection+0x18c/0x190
>> [  169.628496]  ? exc_control_protection+0x14f/0x190
>> [  169.633263]  asm_exc_control_protection+0x26/0x30
>> [  169.638030] RIP: 0010:0x8041d0
>> [  169.641142] Code: Unable to access opcode bytes at 0x8041a6.
>> [  169.646857] RSP: 0018:ff55b3cba167fb50 EFLAGS: 00010007
>> [  169.652144] RAX: 00000000008041d0 RBX: 0000000000000000 RCX: 
>> 0000000000000005
>> [  169.659341] RDX: 00c6e8a7c0000000 RSI: 0000000000000001 RDI: 
>> ffffffffff1ff000
>> [  169.666525] RBP: 0000000000000005 R08: 0000000000000000 R09: 
>> 000000000000ffff
>> [  169.673709] R10: 0000000000000000 R11: ffffffffffff0000 R12: 
>> 0000000000002001
>> [  169.680906] R13: ffffffffa5ae02c8 R14: 00000000ffffffff R15: 
>> 0000000000000000
>> [  169.688091]  ? tboot_shutdown+0x5b/0x140
>> [  169.692084]  ? tboot_sleep+0x12c/0x140
>> [  169.695890]  ? acpi_os_enter_sleep+0x2b/0x60
>> [  169.700221]  ? acpi_hw_legacy_sleep+0x140/0x1c0
>> [  169.704816]  ? acpi_power_off+0x16/0x40
>> [  169.708715]  ? sys_off_notify+0x48/0x70
>> [  169.712615]  ? notifier_call_chain+0x5a/0xd0
>> [  169.716943]  ? atomic_notifier_call_chain+0x32/0x50
>> [  169.721885]  ? do_kernel_power_off+0x3e/0x50
>> [  169.726213]  ? native_machine_power_off+0x21/0x40
>> [  169.730983]  ? __do_sys_reboot+0x1d2/0x240
>> [  169.735151]  ? do_syscall_64+0x7d/0x160
>> [  169.739053]  ? syscall_exit_work+0xf3/0x120
>> [  169.743302]  ? syscall_exit_to_user_mode+0x32/0x190
>> [  169.748243]  ? do_syscall_64+0x89/0x160
>> [  169.752143]  ? __count_memcg_events+0xdf/0x170
>> [  169.756645]  ? handle_mm_fault+0x256/0x370
>> [  169.760813]  ? do_user_addr_fault+0x347/0x640
>> [  169.765235]  ? exc_page_fault+0x73/0x160
>> [  169.769228]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>
> 

Hi, Bagas.

I wanted to follow up on the tboot/CET patch I sent on 2025-12-11. You 
had asked whether the IBT failure occurred only in certain shutdown 
paths. I’ve since tested all the shutdown and reboot flows available on 
this system, and they all hit the same tboot_shutdown() → shutdown_entry 
path and trigger a #CP when CET is enabled.

To recap the findings:

     shutdown_entry ultimately points to a BIOS/firmware routine.

     That routine isn’t built with CET/IBT, so it lacks an ENDBR64 at
     its entry.

     With CET still active, IBT enforcement causes an immediate #CP
     when control is transferred.

     Disabling CET before calling shutdown_entry resolves the issue
     across all tested shutdown/reboot paths.

If there’s anything you’d like clarified or adjusted in the patch or 
changelog, I’m happy to update it. Just wanted to check in now that 
we’re past the holiday lull.

Thanks,
Tony Camuso <tcamuso@redhat.com>