From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=128.30.2.78 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794935017; b=4OdAMhShGrS22pZ8AU7hw7grSgmmME28D01JYjyW/IqWoEdD0M8ChEIcgqV4lwhIlApl cttcyHtUomkTmk1oL+N5Z6xCP7dymeZvABjD8vcmh5iHCsiVL5Fkp9pBL7ayPMNy+TQqn KwfKJUnJB/fFrS4pvFT94aI/h0O++d6f4+Vj0IapCYVKznCxTextKfgKRD1SM0kTUcAsR vuu3e4489aVnjIwUgBMDbICpsnFGWQhHhu6S+ilqK9uZAG24vxulW/56fniB3rg5eR0gx T/lzKiK+U9sdbFSY/pE2Iu5Gl7JPit9T1/MyZynRQ3/52xR8jgQXVAvX0sfE+5w1laQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794935017; h=DMARC-Filter:Received:From:To:Cc:Subject:Date:Message-ID:X-Mailer: MIME-Version:Content-Transfer-Encoding; bh=uI17tEexINtQgsX65dt29zg6Kh079wmiKMEsTv7UKs4=; b=CSQuld8Cnbt/EiFlMdNe1+hYy7XAbHebvECllrmXFS0MB0Y9m24XkBzkQ0gHmis/S+MY 6Y4LjJ339uaH7F0ueRIvT0+yjLUylFo8CIetffeViWip87TnYv8pnrcexpbRoEpf/sUHH qq5pYXk1AHGhJ3fui2fX/8YzhbKexIAMhgDKtEtMJvFPdP8eJcwxsIvrH2dxNqKX3jrvN n84G4yJpdzIsVDblqK+h0GLSZcNuvkzCdys5am+3yEKnjdTcpUv6wjqD7vt3N9h18TUW1 XsB2dlPZR/AiXG1sT4DfeOMB7jXTpmyTvcmvhWdMaclC5cqIlQg/5nyABTaXhtGHWJQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=csail.mit.edu; spf=pass smtp.mailfrom=csail.mit.edu; arc=none smtp.remote-ip=128.30.2.78 Received: from outgoing2021.csail.mit.edu (outgoing2021.csail.mit.edu [128.30.2.78]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19003606A331 for ; Tue, 17 Nov 2026 17:03:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 19003606A331 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=csail.mit.edu Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=csail.mit.edu Received: from escalante.csail.mit.edu ([128.52.128.198]) by outgoing2021.csail.mit.edu with esmtp (Exim 4.95) (envelope-from ) id 1xTMae-00C8zH-IN; Tue, 17 Nov 2026 12:03:28 -0500 From: Nickolai Zeldovich To: Alex Deucher , =?UTF-8?q?Christian=20K=C3=B6nig?= Cc: David Airlie , Simona Vetter , amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, Nickolai Zeldovich Subject: [PATCH v1] drm/radeon: stop the GPU before kexec Date: Tue, 17 Nov 2026 12:03:28 -0500 Message-ID: <20261117170328.56738-1-nickolai@csail.mit.edu> X-Mailer: git-send-email 2.53.0 X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit On a kexec reboot, radeon leaves the GPU running, and on a system without an IOMMU the GPU then writes into the next kernel's memory as soon as radeon probes there. The PCI core clears bus mastering before the kexec, but the interrupt handler (IH) stays enabled, with its ring and write-pointer writeback mapped through the old kernel's GART. The new kernel's evergreen_mc_program() raises page-flip interrupts before the new GART is set up; the GPU holds the IH writes for them while bus mastering is off and issues them, to the old kernel's pages, once r600_irq_init() sets it again. Suspend the GPU in radeon_pci_shutdown() on kexec, as PPC64 and Loongson already do on every shutdown. Normal shutdown and reboot are unchanged: commit 8d13c187c42e ("Revert "drm/radeon: Fix EEH during kexec"") removed the same call there because it hung some HD 5770 systems. Tested on riscv64 with a Radeon HD 6450 by pointing the stale GART at canary pages after a kexec: without this patch the page-flip interrupts write IH entries into them, with it nothing is written. Signed-off-by: Nickolai Zeldovich --- Notes: Tested on a Milk-V Pioneer (SG2042, no IOMMU) with a Radeon HD 6450: 9 of 9 kexec reboots of patched 7.4-rc3 into itself came up clean, with radeon and the desktop running. Unpatched, kexec boots there intermittently oopsed, logged "BUG: Bad rss-counter state" soon after radeon probed, or hung early in boot. Commit 8d13c187c42e removed this call from every shutdown after two HD 5770 x86 systems hung on shutdown and restart (https://bugzilla.kernel.org/show_bug.cgi?id=205147), but the cause was unclear. Kexec runs the same radeon_suspend_kms() as shutdown and only skips the firmware that follows it. Shutdown and restart end in different firmware paths and both hung, so the hang was more likely in the suspend itself and may recur on kexec on those systems. Also, with this patch the GPU reaches the next kernel in D3hot, so an early console on it (vgacon, efifb) is not decoded until radeon wakes it; PPC64 has done this since 2019, but it is untested on x86. Separately from any hang, I would expect that kexec without this patch would let the GPU corrupt the new kernel's memory on those systems, as it does here. If you prefer to avoid the full suspend, a narrower fix would disable only the IH: clear rdev->ih.enabled and call radeon_irq_set(), which on R600 through CIK turns off the IH ring and all interrupt sources without waiting on the GPU. That skips the code that may have hung, but leaves the CP, DMA, UVD and the old GART set up; they stayed idle in my tests, but nothing stops them. Reproducer (Evergreen register offsets): run it as root after a kexec, with radeon not loaded in the new kernel (modprobe.blacklist=radeon). It points every entry of the old GART, still live in VRAM, at canary pages, so stray GPU writes land there instead of in kernel memory; rewrites the D1-D4 surface addresses to raise page-flip interrupts, as evergreen_mc_program() does; sets bus master, as r600_irq_init() does; and counts the canary bytes the GPU changed. The IH usually holds its writes until bus master is set; in one run of eight it did not wait. gcc -std=c99 -D_GNU_SOURCE -O2 -o ihrepro ihrepro.c echo 0 > /proc/sys/vm/compact_unevictable_allowed # canaries stay put ./ihrepro old kernel unpatched: bus master off, IH on, GART on 68 canary bytes written by the GPU (four 16-byte IH entries, 4-byte write pointer) old kernel patched: bus master off, IH off, GART off 0 canary bytes written by the GPU #include #include #include #include #include #include #define DEV "/sys/bus/pci/devices/0000:01:00.0/" #define NPG 1024 static void *bar(const char *f) { int fd = open(f, O_RDWR | O_SYNC); off_t sz = lseek(fd, 0, SEEK_END); void *p = mmap(NULL, sz, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); return p == MAP_FAILED ? NULL : p; } int main(void) { static const uint32_t crtc[] = { 0x0, 0xc00, 0x9800, 0xa400 }; volatile uint32_t *mmio = bar(DEV "resource2"); volatile uint64_t *vram = bar(DEV "resource0"); int cfg = open(DEV "config", O_RDWR), pm = open("/proc/self/pagemap", O_RDONLY); uint16_t cmd; if (!mmio || !vram || cfg < 0 || pm < 0) return perror("open"), 1; pread(cfg, &cmd, 2, 4); printf("bus master %s, IH %s, GART %s\n", cmd & 4 ? "on" : "off", mmio[0x3e00 / 4] & 1 ? "on" : "off", mmio[0x1410 / 4] & 1 ? "on" : "off"); /* point every GTT page of the old GART table at a canary page */ uint8_t *c = mmap(NULL, NPG * 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE | MAP_LOCKED, -1, 0); memset(c, 0xa5, NPG * 4096); volatile uint64_t *pt = vram + ((uint64_t)mmio[0x153c / 4] << 12) / 8; uint64_t flags = pt[0] & 0xfff, n = mmio[0x157c / 4] - mmio[0x155c / 4] + 1; for (uint64_t i = 0; i < n; i++) { uint64_t e; pread(pm, &e, 8, ((uintptr_t)(c + (i % NPG) * 4096) / 4096) * 8); pt[i] = ((e & ((1ULL << 55) - 1)) << 12) | flags; } mmio[0x5480 / 4] = 1; /* HDP flush */ mmio[0x1470 / 4] = 1; /* GART TLB invalidate */ /* raise a page-flip interrupt on each CRTC, then enable bus master */ for (int i = 0; i < 4; i++) mmio[(0x6810 + crtc[i]) / 4] = mmio[(0x6810 + crtc[i]) / 4]; sleep(1); cmd |= 4; pwrite(cfg, &cmd, 2, 4); sleep(1); int hits = 0; for (int i = 0; i < NPG * 4096; i++) hits += c[i] != 0xa5; printf("%d canary bytes written by the GPU\n", hits); return hits != 0; } This patch was written with the help of an AI tool. drivers/gpu/drm/radeon/radeon_drv.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/gpu/drm/radeon/radeon_drv.c b/drivers/gpu/drm/radeon/radeon_drv.c index 53d06053dec8..cd786c3264ff 100644 --- a/drivers/gpu/drm/radeon/radeon_drv.c +++ b/drivers/gpu/drm/radeon/radeon_drv.c @@ -31,6 +31,7 @@ #include #include +#include #include #include #include @@ -379,6 +380,10 @@ radeon_pci_shutdown(struct pci_dev *pdev) * it breaks some other boards. */ radeon_suspend_kms(pci_get_drvdata(pdev), true, true, false); +#else + /* Clearing bus master does not stop the IH writing to stale addresses */ + if (kexec_is_in_progress()) + radeon_suspend_kms(pci_get_drvdata(pdev), true, true, false); #endif } base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4 -- 2.47.3