From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56D68240604 for ; Mon, 12 Jan 2026 17:31:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768239101; cv=none; b=YXLY+RvOM5RawMpfV/TaivED/ITookEjh3hx3clRv71wmg/xT4z9GOxAMNOnvgUTZPtgTbf/co3xfanBtrn1pDn0gsW4SphHjbh2+5Ui5G6+DAnbeh3Ps5t1VCLnAI8aiGGQX7Xgnmuikgkt39RUKjKBj4nyuOlzCAGyxi1XSys= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768239101; c=relaxed/simple; bh=I6vinIbm+C2pgLa4A7ygkXgFKgOfP+K3wTH2X2JYciw=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=DmxnWXzds9dxMERoNuZ1CB79S+mjHfhdXCNhGhMt7O16gcRBP7V7WuYSWYJ/DBUsKK0Fhjc7+ilQku0Yh9gQnsXWOca9Ky7SLFhEnHG1FK7mpYn+xnd5it7wOM9gv0rDrWy5it45q/ETG6iVCcn1Dy6NaFwWX2cZQnV67RBNVX4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fovfocKv; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=TQNZIgHM; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fovfocKv"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="TQNZIgHM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1768239099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tOwk8VMztEIkNafL45QZZwD7L3p3oXT7KAteupa3pZo=; b=fovfocKvXpoM5fWXTAL5MsUz4lVeihX33KverBpV9sz7SmkEG3J6H1fDQo2uZiuvgkz+Fj JhUMRnG4sCNl1zWCEJfUoqo0SSmqDo9Z42kBO/0vsvkD4CUuOQPZ4LYKWKiXe/L2umHXO7 cBd3UNrxt16imZeH1J8k/dKxB3G6Rg4= Received: from mail-vk1-f199.google.com (mail-vk1-f199.google.com [209.85.221.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-333-1mx_VFPZPJSKayMa5M1GzQ-1; Mon, 12 Jan 2026 12:31:38 -0500 X-MC-Unique: 1mx_VFPZPJSKayMa5M1GzQ-1 X-Mimecast-MFC-AGG-ID: 1mx_VFPZPJSKayMa5M1GzQ_1768239097 Received: by mail-vk1-f199.google.com with SMTP id 71dfb90a1353d-5637112ac92so6448766e0c.0 for ; Mon, 12 Jan 2026 09:31:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1768239097; x=1768843897; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:from:to :cc:subject:date:message-id:reply-to; bh=tOwk8VMztEIkNafL45QZZwD7L3p3oXT7KAteupa3pZo=; b=TQNZIgHMaweH9XIPmFPN3o55finUWx7C4Udhvd3jIFWwl5+cNcucMkF59idmWsX6Oy S8OwovrusCr2bwozhgDgx+zHJacCfbneNN9mh+YU/ic5PYMCCHeQ5kbN2Pv3yfZ0OHU6 cTwpub2evXhPZ260MHP+D3w099ctjGwC0WTjovS61kL1kAevkGV22l2JsDIBApR7zYLh 4YF8hl6FJu3mxzrepPwt9k1JMOh9GVVvJewM6VIyHqZZcsavyz66Vv/T+PVN9J0lJKEo dERy4EQwqnMZ7mXAIOdXRwp+XrZBwGO5AwZcoG+uzXercmk2j1W1qcf0lfaSJYdOeC0K 3M0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768239097; x=1768843897; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tOwk8VMztEIkNafL45QZZwD7L3p3oXT7KAteupa3pZo=; b=K6U4Mo7t/nIPIV85tRKPFnu7EErmthhhv6HcaXQfxADSi8Qm60UTY3i7sEuljQLvnt SyL34tAGBAqW0MEbUfew7/ZNdXOj9YDiDJiv/p3qHi2HGY8IUNPf17MqRRgerQZx/2Hc YClHtoXi4bPrSG7s2JaIjv2qI/tbnXxHSD+OTtspDVbBdz6dQ+NLhYFx1Kl6QuaxfS0Y CNgXF1VrjAt6CXgG2Y3fQE8CIhCs8wtWpHHb8rqh1gyWQ9Fux3NMgqlFo/Wogqd4kKuY +1jyo1+tkRHXw0XbC7hBV9Uu1cC/6PQwtp/pWf08CN/vllJ2xwhk4hcIMLkuPnP3FlfJ /9YQ== X-Gm-Message-State: AOJu0Yw+hRZQGTm9tEtzOChLIio3kvmjNTT8GpFGon2M1zt8rtCkAFxs DdYbgRHPvhvpDdiDhGCgTsiyEMNzVpsSNLawBP3QYHa5PNhQMzI/Qm7C48CWWNRI92xvx6F3vOK e52pnGWEXkgVn0hPktOeu6lFBRgqkQqbp7QznFtr6lLeYS7sCmpcB2qBOMiQXs+AhBQ== X-Gm-Gg: AY/fxX6Bgf3wE2nXzkbGv8woItvllpDJ4gBPoQKC7E3AHhFH+4d7dmhPYc3lUWg9HZN V7C31BIvDyvjKVTNjybbbD89D9inKoaX7Uh6y+026UJFe/C7C2qp2OrfgRBH2jdvy44KHU1WHU/ k55IiUTok9jdfhbsyLAishMM4vICn4XJexLbbyAKNAIKCa7zFhSseJaTlHIof3o7xmblcDhjJFp 6AwbNYBYsnlFbLXbLjFgiNe+txGKfgeXzD6vkTn2QksaZxwAHGd+oBfX4Aq7yhy5sK177rMiJ9D YnvU0smV5EfoxO8LiuqhRag+E6hL1xx90mpS/ndhUNPCvK2gD/GmUFOgI1/tX2iRphMD1w4fK2d C4g2adCadBWk= X-Received: by 2002:a05:6122:6b07:b0:563:60ce:9d53 with SMTP id 71dfb90a1353d-56360cea08bmr5093918e0c.9.1768239097299; Mon, 12 Jan 2026 09:31:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IEWfozEG2RCdr776/dd7/OqD3BQPe90AOS190CjujXwz1xvy2qiNzp1kouWtBVenK9Hwzrjtg== X-Received: by 2002:a05:6122:6b07:b0:563:60ce:9d53 with SMTP id 71dfb90a1353d-56360cea08bmr5093896e0c.9.1768239096750; Mon, 12 Jan 2026 09:31:36 -0800 (PST) Received: from [10.6.10.143] ([66.187.232.140]) by smtp.gmail.com with ESMTPSA id 71dfb90a1353d-5636134be9fsm11677998e0c.16.2026.01.12.09.31.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Jan 2026 09:31:35 -0800 (PST) Message-ID: <6323166f-e7f8-4a42-b8b1-a67ddb4c3ec7@redhat.com> Date: Mon, 12 Jan 2026 12:31:30 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [tboot-devel] [PATCH 1/1] Disable CET when calling tboot shutdown procedure. From: Tony Camuso To: Bagas Sanjaya , ning.sun@intel.com, tboot-devel@lists.sourceforge.net Cc: linux-kernel@vger.kernel.org, rppt@kernel.org, tglx@linutronix.de, mingo@kernel.org, bp@alien8.de, michal.camacho.romero@linux.intel.com References: <20251017073619.547993-1-michal.camacho.romero@linux.intel.com> <358c63d6-5a25-462d-af04-1703bb7840e8@redhat.com> Content-Language: en-US In-Reply-To: <358c63d6-5a25-462d-af04-1703bb7840e8@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 12/11/25 8:46 AM, Tony Camuso wrote: > > On 12/10/2025 1:10 PM, Tony Camuso wrote: >> On 11/24/2025 7:34 PM, Bagas Sanjaya wrote: >>> On Thu, Nov 13, 2025 at 09:37:14AM -0500, Tony Camuso wrote: >>>> The tboot->shutdown_entry is effectively bios code and CET needs to be >>>> disabled before calling it. >>>> >>>> It resolves TBOOT shutdown failure bug, reported on the SLES (SUSE >>>> Linux >>>> Enterprise Server) 16.0 OS. OS power off, called by the "init 0" >>>> command, >>>> was failing, due to activated Intel Control-Flow Enforcement Technology >>>> (CET). >>>> Disabling CET has allowed to execute OS and TBOOT shutdown properly. >>> >>> Are ``systemctl poweroff`` and ``shutdown -P`` are also affected? >>> >>> Confused... >>> >> >> Yes, all shutdown methods on kernels launched with tboot, on systems that >> expose the CPU ibt flag to kernels v6.12+ will cause the stack trace >> appended >> below. >> >> The stack trace demonstrates that CET enforcement collides with legacy >> BIOS shutdown code that lacks ENDBR markers. The kernel BUG at >> cet.c:102 is a direct result of CET being active when jumping into >> tboot->shutdown_entry. >> >> Legacy BIOS/tboot code without ENDBR now traps, requiring CET to be >> disabled >> around that call. >> >> The patch: >>      Prevents CET from falsely trapping on non-CET BIOS code. > > Need to clarify further: > > The patch prevents CET from trapping when tboot invokes the BIOS-provided > shutdown_entry routine, which lacks ENDBR instructions. > > tboot side: >   In tboot_shutdown(), the kernel switches to the tboot page tables and >   then calls: >     shutdown = (void(*)(void))(unsigned long)tboot->shutdown_entry; >     shutdown(); > >   That shutdown_entry pointer comes from the tboot structure, populated >   at boot. >     In the tboot project directory, see include/tboot.h >     In the kernel, see include/linux/tboot.h > > BIOS side: >   The actual routine behind shutdown_entry is implemented in BIOS/ > firmware. >   It’s not compiled with CET/IBT support, so it lacks the required ENDBR64 >   instruction at its entry point > >   When CET is still enabled, the CPU enforces IBT. Jumping into that BIOS >   routine without ENDBR triggers a #CP (control protection exception), >   which is what the stack trace shows. > >   So it is the BIOS shutdown_entry function itself that causes the trap, >   but only because tboot is handing control to it while CET is active. > > What happens: >   From the stack trace >     Missing ENDBR: 0x8041d0 >     kernel BUG at arch/x86/kernel/cet.c:102! >     RIP: 0010:0x8041d0 >   This shows the CPU trapping on entry into the shutdown routine at > address >   0x8041d0, which is the pointer stored in tboot->shutdown_entry > >   The shutdown_entry field is explicitly documented as the physical > address >   of the BIOS shutdown routine. This structure is populated by tboot at > boot. > >> >>      Maintains system stability during shutdown. >> >>      Preserves CET protection elsewhere, only disabling it for the >>      narrow window where legacy firmware must run. >> >> >> [  169.420078] reboot: Power down >> [  169.427516] Missing ENDBR: 0x8041d0 >> [  169.431128] ------------[ cut here ]------------ >> [  169.435805] kernel BUG at arch/x86/kernel/cet.c:102! >> [  169.440840] Oops: invalid opcode: 0000 [#1] SMP NOPTI >> [  169.445966] CPU: 0 UID: 0 PID: 3354 Comm: poweroff Kdump: loaded >> Not tainted 6.12.0-124.8.1.el10_1.x86_64 #1 PREEMPT(voluntary) >> [  169.457580] Hardware name: Dell Inc. PowerEdge R570/03TJR3, BIOS >> 1.2.1 01/23/2025 >> [  169.465113] RIP: 0010:exc_control_protection+0x18c/0x190 >> [  169.470490] Code: 1c ff 45 31 c9 49 89 d8 b9 09 00 00 00 48 8b 93 >> 80 00 00 00 be 63 00 00 00 48 c7 c7 a4 85 e5 a4 e8 79 92 30 ff e9 02 >> ff ff ff <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 >> 66 0f >> [  169.489292] RSP: 0018:ff55b3cba167fa88 EFLAGS: 00010002 >> [  169.494581] RAX: 0000000000000017 RBX: ff55b3cba167faa8 RCX: >> 00000000ffff7fff >> [  169.501765] RDX: 0000000000000000 RSI: 0000000000000003 RDI: >> 0000000000000001 >> [  169.508949] RBP: 0000000000000003 R08: 0000000000000000 R09: >> ffffffffa59e2b08 >> [  169.516132] R10: ffffffffa5922ac8 R11: 0000000000000003 R12: >> 0000000000000000 >> [  169.523316] R13: 0000000000000000 R14: 0000000000000000 R15: >> 0000000000000000 >> [  169.530514] FS:  00007f5f9a122140(0000) GS:ff39175c2de00000(0000) >> knlGS:0000000000000000 >> [  169.538659] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [  169.544454] CR2: 0000559386dc5320 CR3: 000000010fbe2000 CR4: >> 0000000000f71ef0 >> [  169.551651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: >> 0000000000000000 >> [  169.558835] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: >> 0000000000000400 >> [  169.566019] PKRU: 55555554 >> [  169.568784] Call Trace: >> [  169.571295]  >> [  169.573458]  ? show_trace_log_lvl+0x1b0/0x2f0 >> [  169.577880]  ? show_trace_log_lvl+0x1b0/0x2f0 >> [  169.582301]  ? asm_exc_control_protection+0x26/0x30 >> [  169.587244]  ? exc_control_protection+0x18c/0x190 >> [  169.592011]  ? __die_body.cold+0x8/0x12 >> [  169.595910]  ? die+0x2e/0x50 >> [  169.598863]  ? do_trap+0xca/0x110 >> [  169.602243]  ? do_error_trap+0x65/0x80 >> [  169.606049]  ? exc_control_protection+0x18c/0x190 >> [  169.610816]  ? exc_invalid_op+0x50/0x70 >> [  169.614715]  ? exc_control_protection+0x18c/0x190 >> [  169.619482]  ? asm_exc_invalid_op+0x1a/0x20 >> [  169.623728]  ? exc_control_protection+0x18c/0x190 >> [  169.628496]  ? exc_control_protection+0x14f/0x190 >> [  169.633263]  asm_exc_control_protection+0x26/0x30 >> [  169.638030] RIP: 0010:0x8041d0 >> [  169.641142] Code: Unable to access opcode bytes at 0x8041a6. >> [  169.646857] RSP: 0018:ff55b3cba167fb50 EFLAGS: 00010007 >> [  169.652144] RAX: 00000000008041d0 RBX: 0000000000000000 RCX: >> 0000000000000005 >> [  169.659341] RDX: 00c6e8a7c0000000 RSI: 0000000000000001 RDI: >> ffffffffff1ff000 >> [  169.666525] RBP: 0000000000000005 R08: 0000000000000000 R09: >> 000000000000ffff >> [  169.673709] R10: 0000000000000000 R11: ffffffffffff0000 R12: >> 0000000000002001 >> [  169.680906] R13: ffffffffa5ae02c8 R14: 00000000ffffffff R15: >> 0000000000000000 >> [  169.688091]  ? tboot_shutdown+0x5b/0x140 >> [  169.692084]  ? tboot_sleep+0x12c/0x140 >> [  169.695890]  ? acpi_os_enter_sleep+0x2b/0x60 >> [  169.700221]  ? acpi_hw_legacy_sleep+0x140/0x1c0 >> [  169.704816]  ? acpi_power_off+0x16/0x40 >> [  169.708715]  ? sys_off_notify+0x48/0x70 >> [  169.712615]  ? notifier_call_chain+0x5a/0xd0 >> [  169.716943]  ? atomic_notifier_call_chain+0x32/0x50 >> [  169.721885]  ? do_kernel_power_off+0x3e/0x50 >> [  169.726213]  ? native_machine_power_off+0x21/0x40 >> [  169.730983]  ? __do_sys_reboot+0x1d2/0x240 >> [  169.735151]  ? do_syscall_64+0x7d/0x160 >> [  169.739053]  ? syscall_exit_work+0xf3/0x120 >> [  169.743302]  ? syscall_exit_to_user_mode+0x32/0x190 >> [  169.748243]  ? do_syscall_64+0x89/0x160 >> [  169.752143]  ? __count_memcg_events+0xdf/0x170 >> [  169.756645]  ? handle_mm_fault+0x256/0x370 >> [  169.760813]  ? do_user_addr_fault+0x347/0x640 >> [  169.765235]  ? exc_page_fault+0x73/0x160 >> [  169.769228]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e >> > Hi, Bagas. I wanted to follow up on the tboot/CET patch I sent on 2025-12-11. You had asked whether the IBT failure occurred only in certain shutdown paths. I’ve since tested all the shutdown and reboot flows available on this system, and they all hit the same tboot_shutdown() → shutdown_entry path and trigger a #CP when CET is enabled. To recap the findings: shutdown_entry ultimately points to a BIOS/firmware routine. That routine isn’t built with CET/IBT, so it lacks an ENDBR64 at its entry. With CET still active, IBT enforcement causes an immediate #CP when control is transferred. Disabling CET before calling shutdown_entry resolves the issue across all tested shutdown/reboot paths. If there’s anything you’d like clarified or adjusted in the patch or changelog, I’m happy to update it. Just wanted to check in now that we’re past the holiday lull. Thanks, Tony Camuso