mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Christian König" <christian.koenig@amd.com>
To: Nickolai Zeldovich <nickolai@csail.mit.edu>,
	Alex Deucher <alexander.deucher@amd.com>
Cc: David Airlie <airlied@gmail.com>, Simona Vetter <simona@ffwll.ch>,
	amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1] drm/radeon: stop the GPU before kexec
Date: Wed, 18 Nov 2026 13:11:34 +0100	[thread overview]
Message-ID: <d7179bdf-daa3-4ed2-a45e-ef2b99d116f1@amd.com> (raw)
In-Reply-To: <20261117170328.56738-1-nickolai@csail.mit.edu>

On 11/17/26 18:03, Nickolai Zeldovich wrote:
> On a kexec reboot, radeon leaves the GPU running, and on a system
> without an IOMMU the GPU then writes into the next kernel's memory as
> soon as radeon probes there.
> 
> The PCI core clears bus mastering before the kexec, but the interrupt
> handler (IH) stays enabled, with its ring and write-pointer writeback
> mapped through the old kernel's GART. The new kernel's
> evergreen_mc_program() raises page-flip interrupts before the new GART
> is set up; the GPU holds the IH writes for them while bus mastering is
> off and issues them, to the old kernel's pages, once r600_irq_init()
> sets it again.

Good catch.
 
> Suspend the GPU in radeon_pci_shutdown() on kexec, as PPC64 and Loongson
> already do on every shutdown. Normal shutdown and reboot are unchanged:
> commit 8d13c187c42e ("Revert "drm/radeon: Fix EEH during kexec"")
> removed the same call there because it hung some HD 5770 systems.

Yeah that is clearly not the right approach.

The kexec is often done to switch to a new kernel because the old one has encountered some severe problem, doing a HW suspend in such a situation is not guaranteed to work.

What should be done instead is to fix the initialization order and/or disable GART before the bus mastering is enabled again.

Even without kexec this situation can happen because the PCI reset handling on those HW generations is often not working correctly.

So in situations like VM path through you run into the same problem after a spontaneous reset and there you don't have the change to suspend the HW.

Regards,
Christian.

> 
> Tested on riscv64 with a Radeon HD 6450 by pointing the stale GART at
> canary pages after a kexec: without this patch the page-flip interrupts
> write IH entries into them, with it nothing is written.
> 
> Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
> ---
> 
> Notes:
>     Tested on a Milk-V Pioneer (SG2042, no IOMMU) with a Radeon HD 6450:
>     9 of 9 kexec reboots of patched 7.4-rc3 into itself came up clean, with
>     radeon and the desktop running. Unpatched, kexec boots there
>     intermittently oopsed, logged "BUG: Bad rss-counter state" soon after
>     radeon probed, or hung early in boot.
> 
>     Commit 8d13c187c42e removed this call from every shutdown after two
>     HD 5770 x86 systems hung on shutdown and restart
>     (https://bugzilla.kernel.org/show_bug.cgi?id=205147), but the cause was
>     unclear. Kexec runs the same radeon_suspend_kms() as shutdown and only
>     skips the firmware that follows it. Shutdown and restart end in
>     different firmware paths and both hung, so the hang was more likely in
>     the suspend itself and may recur on kexec on those systems. Also, with
>     this patch the GPU reaches the next kernel in D3hot, so an early console
>     on it (vgacon, efifb) is not decoded until radeon wakes it; PPC64 has
>     done this since 2019, but it is untested on x86.
> 
>     Separately from any hang, I would expect that kexec without this patch
>     would let the GPU corrupt the new kernel's memory on those systems, as
>     it does here. If you prefer to avoid the full suspend, a narrower fix
>     would disable only the IH: clear rdev->ih.enabled and call
>     radeon_irq_set(), which on R600 through CIK turns off the IH ring and
>     all interrupt sources without waiting on the GPU. That skips the code
>     that may have hung, but leaves the CP, DMA, UVD and the old GART set up;
>     they stayed idle in my tests, but nothing stops them.
> 
>     Reproducer (Evergreen register offsets): run it as root after a kexec,
>     with radeon not loaded in the new kernel (modprobe.blacklist=radeon). It
>     points every entry of the old GART, still live in VRAM, at canary pages,
>     so stray GPU writes land there instead of in kernel memory; rewrites the
>     D1-D4 surface addresses to raise page-flip interrupts, as
>     evergreen_mc_program() does; sets bus master, as r600_irq_init() does;
>     and counts the canary bytes the GPU changed. The IH usually holds its
>     writes until bus master is set; in one run of eight it did not wait.
> 
>       gcc -std=c99 -D_GNU_SOURCE -O2 -o ihrepro ihrepro.c
>       echo 0 > /proc/sys/vm/compact_unevictable_allowed   # canaries stay put
>       ./ihrepro
> 
>       old kernel unpatched:  bus master off, IH on, GART on
>                              68 canary bytes written by the GPU
>                              (four 16-byte IH entries, 4-byte write pointer)
>       old kernel patched:    bus master off, IH off, GART off
>                              0 canary bytes written by the GPU
> 
>     #include <fcntl.h>
>     #include <stdint.h>
>     #include <stdio.h>
>     #include <string.h>
>     #include <sys/mman.h>
>     #include <unistd.h>
> 
>     #define DEV "/sys/bus/pci/devices/0000:01:00.0/"
>     #define NPG 1024
> 
>     static void *bar(const char *f)
>     {
>         int fd = open(f, O_RDWR | O_SYNC);
>         off_t sz = lseek(fd, 0, SEEK_END);
>         void *p = mmap(NULL, sz, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
>         return p == MAP_FAILED ? NULL : p;
>     }
> 
>     int main(void)
>     {
>         static const uint32_t crtc[] = { 0x0, 0xc00, 0x9800, 0xa400 };
>         volatile uint32_t *mmio = bar(DEV "resource2");
>         volatile uint64_t *vram = bar(DEV "resource0");
>         int cfg = open(DEV "config", O_RDWR), pm = open("/proc/self/pagemap", O_RDONLY);
>         uint16_t cmd;
> 
>         if (!mmio || !vram || cfg < 0 || pm < 0)
>                 return perror("open"), 1;
>         pread(cfg, &cmd, 2, 4);
>         printf("bus master %s, IH %s, GART %s\n", cmd & 4 ? "on" : "off",
>                mmio[0x3e00 / 4] & 1 ? "on" : "off", mmio[0x1410 / 4] & 1 ? "on" : "off");
> 
>         /* point every GTT page of the old GART table at a canary page */
>         uint8_t *c = mmap(NULL, NPG * 4096, PROT_READ | PROT_WRITE,
>                           MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE | MAP_LOCKED, -1, 0);
>         memset(c, 0xa5, NPG * 4096);
>         volatile uint64_t *pt = vram + ((uint64_t)mmio[0x153c / 4] << 12) / 8;
>         uint64_t flags = pt[0] & 0xfff, n = mmio[0x157c / 4] - mmio[0x155c / 4] + 1;
>         for (uint64_t i = 0; i < n; i++) {
>                 uint64_t e;
>                 pread(pm, &e, 8, ((uintptr_t)(c + (i % NPG) * 4096) / 4096) * 8);
>                 pt[i] = ((e & ((1ULL << 55) - 1)) << 12) | flags;
>         }
>         mmio[0x5480 / 4] = 1;   /* HDP flush */
>         mmio[0x1470 / 4] = 1;   /* GART TLB invalidate */
> 
>         /* raise a page-flip interrupt on each CRTC, then enable bus master */
>         for (int i = 0; i < 4; i++)
>                 mmio[(0x6810 + crtc[i]) / 4] = mmio[(0x6810 + crtc[i]) / 4];
>         sleep(1);
>         cmd |= 4;
>         pwrite(cfg, &cmd, 2, 4);
>         sleep(1);
> 
>         int hits = 0;
>         for (int i = 0; i < NPG * 4096; i++)
>                 hits += c[i] != 0xa5;
>         printf("%d canary bytes written by the GPU\n", hits);
>         return hits != 0;
>     }
> 
>     This patch was written with the help of an AI tool.
> 
>  drivers/gpu/drm/radeon/radeon_drv.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/gpu/drm/radeon/radeon_drv.c b/drivers/gpu/drm/radeon/radeon_drv.c
> index 53d06053dec8..cd786c3264ff 100644
> --- a/drivers/gpu/drm/radeon/radeon_drv.c
> +++ b/drivers/gpu/drm/radeon/radeon_drv.c
> @@ -31,6 +31,7 @@
> 
>  #include <linux/aperture.h>
>  #include <linux/compat.h>
> +#include <linux/kexec.h>
>  #include <linux/module.h>
>  #include <linux/pm_runtime.h>
>  #include <linux/vga_switcheroo.h>
> @@ -379,6 +380,10 @@ radeon_pci_shutdown(struct pci_dev *pdev)
>          * it breaks some other boards.
>          */
>         radeon_suspend_kms(pci_get_drvdata(pdev), true, true, false);
> +#else
> +       /* Clearing bus master does not stop the IH writing to stale addresses */
> +       if (kexec_is_in_progress())
> +               radeon_suspend_kms(pci_get_drvdata(pdev), true, true, false);
>  #endif
>  }
> 
> 
> base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
> --
> 2.47.3
> 


      reply	other threads:[~2026-11-18 12:11 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 17:03 Nickolai Zeldovich
2026-11-18 12:11 ` Christian König [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d7179bdf-daa3-4ed2-a45e-ef2b99d116f1@amd.com \
    --to=christian.koenig@amd.com \
    --cc=airlied@gmail.com \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=nickolai@csail.mit.edu \
    --cc=simona@ffwll.ch \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®