From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02D2625EFAE for ; Mon, 10 Feb 2025 23:56:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739231797; cv=none; b=oBINzviR+IrZHWgXTwstkDNu9M7K/Mw2ECHoKwIr5clzFZeNsxKaWIQUB2JIINYuB0hxiHvEnqN9q3Ty0dCMdH67jZUPr6gu6CmW2unFlu117W15iG0tLYQos/HhDYuFknq6cMEapZPJ56UtkMb/yPMRYk51Y8cnx2oslexP1sY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739231797; c=relaxed/simple; bh=D4hzEZ90I1yTKCrbrFTPLTeJGB3TmoI5HZKosC2O5A8=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=oCpaVJWzaTJhrwERpegWOZ6BGJLrEa9OSTcnie5nLq1p/SpwhAXrXl/U8bGY2Ea0yqeJC1/EoCC7xEOdcV1A4Hw+eS4Lqw4fzzpCLlHlVnANZbTJm/RnwBJzX8a4bHXwt7BRJ9bGKF6ljBbfwXL9EGCvTXF1GN+r42dxwE8jrNQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GSes+v/Y; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GSes+v/Y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739231794; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=00zVrDfVjXaItDva0W2RZwy2B+2RKPwCMMc/ltLdmxE=; b=GSes+v/YT356e+IAlfyP4DwM67JSTNYsyk7mys6zkQQUVqlB6nSPPjnmuLsAM9sAkCq7Tm mKYzhqqprfKJ1NBJhphG3v0bBACUiPTUDq9N4KU1VPhuPirpuftAVEWHv1i8jhej6FNCqS gr0JAjP/YiZd/c0CfAPb00Lj0ZEpa3I= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-149-hKMrYxVNNYyTUJ9OJWZ4HQ-1; Mon, 10 Feb 2025 18:56:32 -0500 X-MC-Unique: hKMrYxVNNYyTUJ9OJWZ4HQ-1 X-Mimecast-MFC-AGG-ID: hKMrYxVNNYyTUJ9OJWZ4HQ Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6e4434d78e5so117999746d6.0 for ; Mon, 10 Feb 2025 15:56:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739231792; x=1739836592; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=00zVrDfVjXaItDva0W2RZwy2B+2RKPwCMMc/ltLdmxE=; b=DRBEWpGb9kEcO5jcX/GadXWAnk1fKIdHMOaDQQ3gaoWgSzUfab3MvdTtPHVnrchnca ETTSCC4nIT8E9hFAcb+FEcsGZ3zJJ62/UvQmJWQaJE63Myo8Y+6adyM3bJH4evtivy06 0jtnlPU7XqyRQG17yC74Yii7XjnPylkJjDJt1mK7Q9yvsAhH5wFwrHqveKsLrnu9zHtI OIvyPNS2kQ/wkNJHyvPimNk7cb+J1oKwFXd7OVqDdgXtvPMENUpH9VQrk9JFrv9eL3yU J9aETmqHGilTRD9Z5Uap3h7AZIFA9cLcMRdYL4QWnZNYxyW7OI/Qn018KaeHlyyibior C9YQ== X-Forwarded-Encrypted: i=1; AJvYcCXtBFK7Iq312e9sE2aM70jhYq9TLq0oEnneBtsw+haHRTaamYpJqguGXb5WzQnqUwH6RsCC3wO95gtRN5M=@vger.kernel.org X-Gm-Message-State: AOJu0YylXeQFz3292ONlM9T8KU1x7Z0tevMtAIqtUJQXCBY7C9SYbNYi xS5eJHDX3Di/DjNb1jmLh0o/d4+xArUQws//5WK6qXCJj6i+n1LgLE3zbVZUkzLOMFdJTVq6EXU Xfh0S3rJnWbKEJASzM/8NfhXfcAvG8yBrJDBgxanAgGhEHZuPdqdTktSdHU00yQ== X-Gm-Gg: ASbGncv0i8rzTAfy6+xbckeuEMhNyTIIQGOweKZTrpf7fJTxFw+d3pWh0Covr+q4JRN NqiDUoDJWLkPxi3YAHETNv6yyPFV6yEWg+6N+IoantPUHqVxBJDndQ8JcoH1bdtTnwFcRN8cjNL 7asZpd1erT4wc/MCPBPL/EvmDx7R4Ilr3B9cepc/gC64OuZc696RSHUBN4cFebY7OPJHdFlXmzF bvaPsDdJG0tmycXxyDX0N3EwReOCX0kNgBSFUT7UoJ7Z6H1eP6kbIeJtfZ1mpcZnJFOqL/r+K/k VMvA X-Received: by 2002:a05:6214:4108:b0:6e4:2f4c:f2d2 with SMTP id 6a1803df08f44-6e4456d9da8mr232522056d6.31.1739231792046; Mon, 10 Feb 2025 15:56:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmUoRcB8F2i3rLS9u2hkgfZbI19/TiD5QZMRyg7oxpwPHi67YZY6q7qMfmv5QJMuG4tXI+/g== X-Received: by 2002:a05:6214:4108:b0:6e4:2f4c:f2d2 with SMTP id 6a1803df08f44-6e4456d9da8mr232521766d6.31.1739231791656; Mon, 10 Feb 2025 15:56:31 -0800 (PST) Received: from starship ([2607:fea8:fc01:8d8d:6adb:55ff:feaa:b156]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e43ba36d5csm53280906d6.26.2025.02.10.15.56.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Feb 2025 15:56:31 -0800 (PST) Message-ID: <3c06cc1bb206f1ff9925dc4c7cf5a23f3f6c3505.camel@redhat.com> Subject: Re: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec From: Maxim Levitsky To: Michael Kelley , "thomas.tai@oracle.com" , "mhkelley58@gmail.com" , "haiyangz@microsoft.com" , "wei.liu@kernel.org" , "decui@microsoft.com" , "drawat.floss@gmail.com" , "javierm@redhat.com" , Helge Deller , "daniel@ffwll.ch" , "airlied@gmail.com" , "tzimmermann@suse.de" Cc: "dri-devel@lists.freedesktop.org" , "linux-fbdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-hyperv@vger.kernel.org" Date: Mon, 10 Feb 2025 18:56:30 -0500 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit On Mon, 2025-02-10 at 21:35 +0000, Michael Kelley wrote: > From: thomas.tai@oracle.com Sent: Monday, February 10, 2025 7:08 AM > > > > > > > > Then the question is why the efifb driver doesn't work in the kdump > > > > kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel > > > > on the Oracle Linux 9.4 system, and transferred the kernel image binary > > > > and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the > > > > efifb driver is loaded as part of the kdump kernel, and it doesn't cause > > > > any problems. But there's an interesting difference. In the Oracle Linux > > > > 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on > > > > the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This > > > > difference is due to differences in how the screen_info variable gets > > > > setup in the two VMs. > > > > > > > > When the normal kernel starts in a freshly booted VM, Hyper-V provides > > > > the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V > > > > FB driver or Hyper-V DRM driver has initialized, Linux has picked a > > > > different MMIO address range and told Hyper-V to use the new > > > > address range (which often starts at 0x40900000). A kexec does *not* > > > > reset Hyper-V's transition to the new range, so when the efifb driver > > > > tries to use the framebuffer at 0x40000000, the accesses trap to > > > > Hyper-V and probably fail or timeout (I'm not sure of the details). After > > > > the guest does some number of these bad references, Hyper-V considers > > > > itself to be under attack from an ill-behaving guest, and throttles the > > > > guest so that it doesn't run for a few seconds. The throttling repeats, > > > > and results in extremely slow running in the kdump kernel. > > > > > > > > Somehow in the Ubuntu 20.04 VM, the location of the frame buffer > > > > as stored in screen_info.lfb_base gets updated to be 0x40900000. I > > > > haven't fully debugged how that happens. But with that update, the > > > > efifb driver is using the updated framebuffer address and it works. On > > > > the Oracle Linux 9.4 system, that update doesn't appear to happen, > > > > and the problem occurs. > > > > > > > > This in an interim update on the problem. I'm still investigating how > > > > screen_info.lfb_base is set in the kdump kernel, and why it is different > > > > in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is > > > > well understood, we can contemplate how to fix the problem. Undoing > > > > the revert that is commit 2bebc3cd4870 doesn't seem like the solution > > > > since the original code there was reported to cause many other issues. > > > > The solution focus will likely be on how to ensure the kdump kernel gets > > > > the correct framebuffer address so the efifb driver works, since the > > > > framebuffer address changing is a quirk of Hyper-V behavior. > > > > > > > > If anyone else has insight into what's going on here, please chime in. > > > > What I've learned so far is still somewhat tentative. > > > > > > > Here's what is happening. On Ubuntu 20.04, the kdump image is > > > loaded into crash memory using the kexec command. Ubuntu 20.04 > > > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1, > > > and per the kexec man page, it defaults to using the older kexec_load() > > > system call. When using kexec_load(), the contents to be loaded into > > > crash memory is constructed in user space by the kexec command. > > > The kexec command gets the "screen_info" settings, including the > > > physical address of the frame buffer, via the FBIOGET_FSCREENINFO > > > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself > > > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns > > > the updated framebuffer address. So the efifb driver loads and runs > > > correctly. > > > > > > On Oracle Linux 9.4, the kdump image is also loaded with the > > > kexec command, but from kexec-tools package version > > > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than > > > the version on Ubuntu 20.04. This newer kexec defaults to using the > > > newer kexec_file_load() system call. This system call gets the > > > framebuffer address from the screen_info variable in the kernel, which > > > has not been updated to reflect the new framebuffer address. Hence > > > in the kdump kernel, the efifb driver uses the old framebuffer address, > > > and hence the problem. > > > > > > To further complicate matters, the kexec on Oracle Linux 9.4 seems to > > > have a bug when the -c option forces the use of kexec_load() instead > > > of kexec_file_load(). As an experiment, I modified the kdumpctl shell > > > script to add the "-c" option to kexec, but in that case the value "0x0" > > > is passed as the framebuffer address, which is wrong. Furthermore, > > > the " screen_info.orig_video_isVGA" value (which I mentioned earlier > > > in connection with commit 2bebc3cd4870) is also set to 0, so the > > > kdump kernel no longer thinks it has an EFI framebuffer. Hence the > > > efifb driver isn't loaded, and the kdump works, though for the wrong > > > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4 > > > VM, then kdump works as expected, with the efifb driver being loaded > > > and using the correct framebuffer address. So something is going wrong > > > with kexec 2.0.28 in how it sets up the screen_info when the -c option > > > is used. I'll leave the debugging of the kexec bug to someone else. > > > > Hi Michael, > > > > Do you think we need to handle Azure Gen2 VM differently in the kexec? > > > > Or should we change the kexec_file_load() system call to retrieve the correct > > framebuffer address? > > I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers. > Commit c25a19afb81c may also be a cause of the problem -- see precursor > commit 3cb73bc3fa2a, which describes exactly the problem. I still need to > do some testing, but without that commit, kdump won't detect that it has > an EFI framebuffer, won't load the efifb driver, and so won't encounter the > problem. But we probably need to get Thomas Zimmerman to weigh in on > the implications of reverting c25a19afb81c. > > There's one additional variation of the problem. Assume the Hyper-V FB > driver is loaded (for example) during boot and moves the framebuffer. Then > system runs kexec as part of arming kdump during the boot sequence. > The most recent location of the framebuffer (and whether it is an EFI framebuffer) > gets picked at the time kexec runs, and is stored in the crash kernel memory area. > But what if the framebuffer later moves, perhaps because the Hyper-V FB driver > is unbound? The crash kernel memory area doesn’t get updated and kdump > could still have the wrong framebuffer address. This anomaly argues for the > commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't > load. Of course that approach means that the kdump kernel *must* contain > either the Hyper-V FB or Hyper-V DRM driver in order to work on a system > with only a framebuffer for text output. The efifb driver won't work. But > perhaps that's OK. > > Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and > always use kexec_load() instead of kexec_file_load() sounds like a big hack > to me. And with that approach, you give up the ability to enforce loading only > properly signed kdump images. This is something kexec_file_load() provides > that kexec_load() doesn't, and is one of the main reasons that kexec_file_load() > was added. > > Whether the kexec_file_load() system call could be enhanced to get the > frame buffer information from the /dev/fb0 device, I'm not sure. That might > be a reasonable approach, though it still has the problem that the framebuffer > address could change *after* kexec_file_load() runs. > > Anyway, that's a dump of my current thoughts. I haven't reached a final > conclusion or recommendation yet. Comments from others on the > thread are welcome. Hi! Asking because I also had to do some digging in this area: Do you think that the kernel can *ask* the hypervisor where the framebuffer is instead of relying on bios, the bootloader and/or kexec to somehow provide this information? If hyperv doesn't provide this API, how hard it would be in your opinion to provide it? I am asking because, I also had to debug a RHEL downstream issue where a slightly botched backport ensured that the first stage of the compressed uefi boot image, stopped passing the 'screen_info' to the second stage (the kernel itself), and as a result of this, the second stage stopped loading simplefb, and as a result of *this*, the PCI driver started to try to use the framebuffer range for its own use which failed and resulted in a cryptic error. If the kernel was to just issue some form of a hypercall to ask the hypervisor where the framebuffer currently is, we could avoid a whole class of bugs similar to this. What do you think? Best regards, Maxim Levitsky > > Michael > > > Thank you, > > Thomas > > > > > I'm still thinking about alternatives to fix this mess. Please chime > > > in if you have suggestions. > > > > > > Michael