From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93488C4167D for ; Wed, 13 Dec 2023 16:41:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379091AbjLMQlC (ORCPT ); Wed, 13 Dec 2023 11:41:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232663AbjLMQkl (ORCPT ); Wed, 13 Dec 2023 11:40:41 -0500 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6761FDD for ; Wed, 13 Dec 2023 08:40:46 -0800 (PST) Received: from in01.mta.xmission.com ([166.70.13.51]:34198) by out03.mta.xmission.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1rDSHq-000F5P-IN; Wed, 13 Dec 2023 09:40:42 -0700 Received: from ip68-227-168-167.om.om.cox.net ([68.227.168.167]:57650 helo=email.froward.int.ebiederm.org.xmission.com) by in01.mta.xmission.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1rDSHp-007ayE-7B; Wed, 13 Dec 2023 09:40:42 -0700 From: "Eric W. Biederman" To: James Gowans Cc: "Sean Christopherson" , , , Paolo Bonzini , Marc Zyngier , Arnd Bergmann , Tony Luck , Borislav Petkov , Thomas Gleixner , Ingo Molnar , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , "Pavel Machek" , Sebastian Reichel , Orson Zhai , Alexander Graf , "Jan H . Schoenherr" References: <20231213064004.2419447-1-jgowans@amazon.com> Date: Wed, 13 Dec 2023 10:39:52 -0600 In-Reply-To: <20231213064004.2419447-1-jgowans@amazon.com> (James Gowans's message of "Wed, 13 Dec 2023 08:40:04 +0200") Message-ID: <874jgm9huv.fsf@email.froward.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1rDSHp-007ayE-7B;;;mid=<874jgm9huv.fsf@email.froward.int.ebiederm.org>;;;hst=in01.mta.xmission.com;;;ip=68.227.168.167;;;frm=ebiederm@xmission.com;;;spf=pass X-XM-AID: U2FsdGVkX1+gZJHyqWjEBJwoHvIZu/SXC+PpnQcK/vQ= X-SA-Exim-Connect-IP: 68.227.168.167 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: [PATCH] kexec: do syscore_shutdown() in kernel_kexec X-SA-Exim-Version: 4.2.1 (built Sat, 08 Feb 2020 21:53:50 +0000) X-SA-Exim-Scanned: Yes (on in01.mta.xmission.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org James Gowans writes: > syscore_shutdown() runs driver and module callbacks to get the system > into a state where it can be correctly shut down. In commit > 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") > syscore_shutdown() was removed from kernel_restart_prepare() and hence > got (incorrectly?) removed from the kexec flow. This was innocuous until > commit 6735150b6997 ("KVM: Use syscore_ops instead of reboot_notifier to hook restart/shutdown") > changed the way that KVM registered its shutdown callbacks, switching from > reboot notifiers to syscore_ops.shutdown. As syscore_shutdown() is > missing from kexec, KVM's shutdown hook is not run and virtualisation is > left enabled on the boot CPU which results in triple faults when > switching to the new kernel on Intel x86 VT-x with VMXE enabled. > > Fix this by adding syscore_shutdown() to the kexec sequence. In terms of > where to add it, it is being added after migrating the kexec task to the > boot CPU, but before APs are shut down. It is not totally clear if this > is the best place: in commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") > it is stated that "syscore_ops operations should be carried with one > CPU on-line and interrupts disabled." APs are only offlined later in > machine_shutdown(), so this syscore_shutdown() is being run while APs > are still online. This seems to be the correct place as it matches where > syscore_shutdown() is run in the reboot and halt flows - they also run > it before APs are shut down. The assumption is that the commit message > in commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") > is no longer valid. > > KVM has been discussed here as it is what broke loudly by not having > syscore_shutdown() in kexec, but this change impacts more than just KVM; > all drivers/modules which register a syscore_ops.shutdown callback will > now be invoked in the kexec flow. Looking at some of them like x86 MCE > it is probably more correct to also shut these down during kexec. > Maintainers of all drivers which use syscore_ops.shutdown are added on > CC for visibility. They are: > > arch/powerpc/platforms/cell/spu_base.c .shutdown = spu_shutdown, > arch/x86/kernel/cpu/mce/core.c .shutdown = mce_syscore_shutdown, > arch/x86/kernel/i8259.c .shutdown = i8259A_shutdown, > drivers/irqchip/irq-i8259.c .shutdown = i8259A_shutdown, > drivers/irqchip/irq-sun6i-r.c .shutdown = sun6i_r_intc_shutdown, > drivers/leds/trigger/ledtrig-cpu.c .shutdown = ledtrig_cpu_syscore_shutdown, > drivers/power/reset/sc27xx-poweroff.c .shutdown = sc27xx_poweroff_shutdown, > kernel/irq/generic-chip.c .shutdown = irq_gc_shutdown, > virt/kvm/kvm_main.c .shutdown = kvm_shutdown, > > This has been tested by doing a kexec on x86_64 and aarch64. >From the 10,000 foot perspective: Acked-by: "Eric W. Biederman" Eric > Fixes: 6735150b6997 ("KVM: Use syscore_ops instead of reboot_notifier to hook restart/shutdown") > > Signed-off-by: James Gowans > Cc: Eric Biederman > Cc: Paolo Bonzini > Cc: Sean Christopherson > Cc: Marc Zyngier > Cc: Arnd Bergmann > Cc: Tony Luck > Cc: Borislav Petkov > Cc: Thomas Gleixner > Cc: Ingo Molnar > Cc: Chen-Yu Tsai > Cc: Jernej Skrabec > Cc: Samuel Holland > Cc: Pavel Machek > Cc: Sebastian Reichel > Cc: Orson Zhai > Cc: Alexander Graf > Cc: Jan H. Schoenherr > --- > kernel/kexec_core.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c > index be5642a4ec49..b926c4db8a91 100644 > --- a/kernel/kexec_core.c > +++ b/kernel/kexec_core.c > @@ -1254,6 +1254,7 @@ int kernel_kexec(void) > kexec_in_progress = true; > kernel_restart_prepare("kexec reboot"); > migrate_to_reboot_cpu(); > + syscore_shutdown(); > > /* > * migrate_to_reboot_cpu() disables CPU hotplug assuming that