From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE3E48F40 for ; Sat, 4 Jan 2025 01:51:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735955463; cv=none; b=gtGmcruBu2ez8lyJYawOGzEzsiNKVRfCzri+FHsuBYdM6O36QfzGYFJQ7PQ/3JICveOrBZ8zNqk0B53nmoM6a0ngxVh654zc8ZGsWXcsfVu2vA16Z5onVkD46738lSU+8pvSQ3c49PoTKukKD/gt3er6iCT63FVhOVVjL0WK1Kg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735955463; c=relaxed/simple; bh=BagrBYqDgnY7ta1n1z30+Cfu4+0Q26WIAtazgM4uVVs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=QuJPYZ5u51DUR00KDYk8t2mI8+TXAyBGykXe+FC5VuABUuXrOIpAWuJ+wyaGykp6eIsephIiVmEelbPb8Lct3UuC+V335a/R3azXS2C4FuEoLw7uat0ef3waIC3O1Jn0RYM38BuE5U2Kn6/rdBDu/9i/B0czN9e4W1FNsU2YFuk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F3i2MY9R; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F3i2MY9R" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33FABC4CED6; Sat, 4 Jan 2025 01:51:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735955463; bh=BagrBYqDgnY7ta1n1z30+Cfu4+0Q26WIAtazgM4uVVs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=F3i2MY9RWpB1inFcdcr8XawjZFkxH+i+uCAVs8j6cbjoqj1nt/vtV3WwYSECm1AHG X2s8vHlvQs8jtacGjBAGDvW5JP0sqjxdSBUMcCSXTS9eCd9ugPSpZ+X9yeMoj6WQkR /eIWscJ7DyniofK+uIQzdwzwS5xuInhTfhAYGQkEBHtjDeLOs4u+gJdIjZ/Ax2QlZd pGjiV0iyLnUYXLAJlHWBjEk7BXDrbMA+oHB34S6NIO9WMnpE8ufDxR2996MYud2pHt LxqrrPCmyj8sQP+YcKBqaeaCe1RXErlbOalTj6PUCiV0teIWVCiQtOt8jAZKrNVZMq fQzHSDZP8yBJA== Date: Fri, 3 Jan 2025 17:51:01 -0800 From: Luis Chamberlain To: Adam Williamson Cc: linux-kernel@vger.kernel.org, jforbes@redhat.com, rppt@kernel.org Subject: Re: Kernel crash while doing chroot'ed grub2-mkconfig on qemu-emulated Nehalem CPU since late November 6.13 snapshot Message-ID: References: <9a3c747052fb82274ab3c4a84eaf64c1273117ce.camel@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Thu, Jan 02, 2025 at 12:16:03PM -0800, Adam Williamson wrote: > On Wed, 2024-12-11 at 08:51 -0800, Adam Williamson wrote: > > Hi, folks. Please CC me on replies, I'm not subscribed to the list. The > > downstream bug report for this is > > https://bugzilla.redhat.com/show_bug.cgi?id=2329581 . I also filed > > https://bugzilla.kernel.org/show_bug.cgi?id=219554 but it looks like > > nobody is monitoring that ATM, hence this email. Sorry, I don't know > > where to send it that would be more targeted. > > > > I maintain Fedora's openQA instance - https://openqa.fedoraproject.org/ > > (openQA is an automated testing system which runs jobs on qemu VMs, > > inputting keyboard and mouse events via VNC, and monitoring results via > > screenshots and the serial console). > > > > In openQA testing we've noticed a lot of failures of install tests > > since kernel-6.13.0-0.rc0.20241125git9f16d5e6f220.8.fc42 landed in > > Rawhide - that is, a snapshot of upstream git 9f16d5e6f220 . The > > previous build, kernel-6.13.0-0.rc0.20241119git158f238aa69d.2.fc42 - a > > snapshot of upstream 158f238aa69d - did not show this problem. The > > problems persist with the latest kernel build, kernel-6.13.0- > > 0.rc2.22.fc42 (a build of 6.13 rc2 exactly). > > > > Both BIOS and UEFI x86_64 installs are frequently hitting kernel > > crashes when the Fedora installer runs grub2-mkconfig as part of the > > install process. In the BIOS case, this causes the system to hang > > permanently. In the UEFI case, the system hangs for a while then > > reboots, and fails to boot properly as the installation did not > > complete. > > > > I've reproduced both BIOS and UEFI failures locally with a qemu VM > > configured like the one we use in the affected tests: 2 vCPUs, 4G RAM, > > and CPU model Nehalem - that's `-cpu Nehalem` argument to qemu. If I > > use host CPU config instead, the bug doesn't happen. We intentionally > > use the Nehalem model in this testing to ensure Fedora doesn't > > inadvertently stop supporting the CPU baseline it intends to support. > > > > This happens on more than 50% of install attempts, but not all of them > > (sometimes they work; I've set our test system to retry failures five > > times for now to mitigate the effects of this bug). > > > > The details of the traces we get in the kernel logs differ between > > occurrences and also between BIOS and UEFI, which someone suggested > > indicate this may be some kind of memory corruption issue. But the > > broad shape is consistent: the installer reaches grub2-mkconfig and we > > get a kernel crash. > > > > I did also try reproducing this by running `grub2-mkconfig -o > > /boot/grub/grub2.cfg` multiple times on an *installed* VM with the same > > kernel and VM config, but could not trigger a crash in this case. There > > must be something specific about how this happens in the installer > > environment (for one thing, the installer runs the command chroot'ed > > into the installed system environment). > > > > I'll attach sample logs from a UEFI failure and a BIOS failure. > > > > I haven't attempted to bisect this yet as I find bisecting kernel > > issues pretty painful (the Fedora kernel package spec is a bit weird if > > you're not used to it, building a full kernel takes a long time, I > > don't know how to do intermittent builds with the Fedora kernel spec, > > and since I can't yet reproduce this outside the installer I then have > > to build an installer image with the kernel build in to test it...). > > But if needs must I'll bite the bullet and do it. If anyone could e.g. > > guess at a commit or commit series that might be causing this so I > > could try a targeted reversion, though, that'd be great. > > Update on this: over the holidays, I bisected it to > 5185e7f9f3bd754ab60680814afd714e2673ef88 . A kernel with that commit > reverted does not hit the bug. > > I also did some testing with various CPU model configurations. I think > this actually isn't to do with Nehalem per se, but "virtual machines > where the CPU configuration does not exactly match the host", or > something like that. > > I tried a bunch of qemu CPU model settings - nehalem, sandybridge, > haswell, Skylake-Client and Cascadelake-Server - and got failures with > all of them, but when I set the model to "host", all tests passed. > > The tests get farmed out to a cluster of systems which have different > CPUs - one is Broadwell, one is Skylake, one is Cascade Lake - so I > think when I set the model to anything specific, it will match the host > CPU on some or none of those systems, but never *all* of them, so the > bug will always show up. > > I have emailed the author and reviewer of > 5185e7f9f3bd754ab60680814afd714e2673ef88 (also CCed on this mail) but > have not heard back from them yet. I've sunk over a week into this bug > at this point so it'd be great if someone could look at it. It's not > the biggest regression in the world, but it is a bit awkward for our > automated testing (I'll have to fiddle around to try and set CPU model > 'host' for the most badly-affected tests but ensure we still have > enough tests with 'nehalem' to confirm our baseline isn't moved). > > Thanks, and happy new year! Can you test this: https://lore.kernel.org/lkml/20250103065631.26459-1-jgross@suse.com/T/#u Luis