From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AC56C4332F for ; Mon, 11 Dec 2023 23:51:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345401AbjLKXu4 (ORCPT ); Mon, 11 Dec 2023 18:50:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345163AbjLKXuy (ORCPT ); Mon, 11 Dec 2023 18:50:54 -0500 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E1B99B; Mon, 11 Dec 2023 15:51:00 -0800 (PST) Received: from in02.mta.xmission.com ([166.70.13.52]:37898) by out03.mta.xmission.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1rCq36-00EVyC-FL; Mon, 11 Dec 2023 16:50:56 -0700 Received: from ip68-227-168-167.om.om.cox.net ([68.227.168.167]:49742 helo=email.froward.int.ebiederm.org.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1rCq35-00F4Dc-Cg; Mon, 11 Dec 2023 16:50:56 -0700 From: "Eric W. Biederman" To: "Gowans, James" Cc: "Graf (AWS), Alexander" , "seanjc@google.com" , =?utf-8?Q?Sch=C3=B6nherr=2C_Jan_H=2E?= , "yuzenghui@huawei.com" , "kvm-riscv@lists.infradead.org" , "kexec@lists.infradead.org" , "james.morse@arm.com" , "oliver.upton@linux.dev" , "suzuki.poulose@arm.com" , "chenhuacai@kernel.org" , "atishp@atishpatra.org" , "linux-kernel@vger.kernel.org" , "maz@kernel.org" , "pbonzini@redhat.com" , "kvm@vger.kernel.org" , "kvmarm@lists.linux.dev" , "anup@brainfault.org" , "aleksandar.qemu.devel@gmail.com" References: <20230512233127.804012-1-seanjc@google.com> <20230512233127.804012-2-seanjc@google.com> <871qbud5f9.fsf@email.froward.int.ebiederm.org> <7e30cfc2359dfef39d038e3734f7e5e3d9e82d68.camel@amazon.com> Date: Mon, 11 Dec 2023 17:50:33 -0600 In-Reply-To: <7e30cfc2359dfef39d038e3734f7e5e3d9e82d68.camel@amazon.com> (James Gowans's message of "Mon, 11 Dec 2023 10:27:15 +0000") Message-ID: <87wmtk9u46.fsf@email.froward.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1rCq35-00F4Dc-Cg;;;mid=<87wmtk9u46.fsf@email.froward.int.ebiederm.org>;;;hst=in02.mta.xmission.com;;;ip=68.227.168.167;;;frm=ebiederm@xmission.com;;;spf=pass X-XM-AID: U2FsdGVkX183erp/ctJyjnDuDEXnwg8YrcYEg8oGrjc= X-SA-Exim-Connect-IP: 68.227.168.167 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: [PATCH v2 1/2] KVM: Use syscore_ops instead of reboot_notifier to hook restart/shutdown X-SA-Exim-Version: 4.2.1 (built Sat, 08 Feb 2020 21:53:50 +0000) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org "Gowans, James" writes: > On Mon, 2023-12-11 at 09:54 +0200, James Gowans wrote: >> > >> > What problem are you running into with your rebase that worked with >> > reboot notifiers that is not working with syscore_shutdown? >> >> Prior to this commit [1] which changed KVM from reboot notifiers to >> syscore_ops, KVM's reboot notifier shutdown callback was invoked on >> kexec via kernel_restart_prepare. >> >> After this commit, KVM is not being shut down because currently the >> kexec flow does not call syscore_shutdown. > > I think I missed what you're asking here; you're asking for a reproducer > for the specific failure? > > 1. Launch a QEMU VM with -enable-kvm flag > > 2. Do an immediate (-f flag) kexec: > kexec -f --reuse-cmdline ./bzImage > > Somewhere after doing the RET to new kernel in the relocate_kernel asm > function the new kernel starts triple faulting; I can't exactly figure > out where but I think it has to do with the new kernel trying to modify > CR3 while the VMXE bit is still set in CR4 causing the triple fault. > > If KVM has been shut down via the shutdown callback, or alternatively if > the QEMU process has actually been killed first (by not doing a -f exec) > then the VMXE bit is clear and the kexec goes smoothly. > > So, TL;DR: kexec -f use to work with a KVM VM active, now it goes into a > triple fault crash. You mentioned I rebase so I thought your were backporting kernel patches. By rebase do you mean you porting your userspace to a newer kernel? In any event I believe the bug with respect to kexec was introduced in commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()"). That is where syscore_shutdown was removed from kernel_restart_prepare(). At this point it looks like someone just needs to add the missing syscore_shutdown call into kernel_kexec() right after migrate_to_reboot_cpu() is called. That said I am not seeing the reboot notifiers being called on the kexec path either so your issue with kvm might be deeper. Eric