From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66FB93368A5 for ; Wed, 19 Aug 2026 18:53:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787165637; cv=none; b=uAnOVVEevMOdBCimJzm8oFk8YuqZk483y54IXP9aYvBbO10j+++0aIUqUt0o14MGB4IG+k/2F5KdNRI4hD5Lhlipj9EeqjEwSmtCXWfcru09pJiHHWXD4hi3/Wgx59o0yVVFT1utpAjMeF9nRBjVE56cPTNw6Q7Cogg7/pXpJNY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787165637; c=relaxed/simple; bh=NBq3AZMxLGZ/CiCFg3nZVOzaV/Rdh8V8ae2QO+umOvY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=n4Fmm48U7jTntECfIJZ5W4oLe29PQYZhLIjEuIGrR7oNkaughV1OUQz1VVLBcVqyIWdTwatL+G23bnP8Cg0r8UDmGaA6EUtm9SCVZkMNMkNrNZQQJztyVgR2pYrA24Q27ll0LPdlU39dbyjxlZuGzZCSCpcDgl6taLWEWHl+OnU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DmWNv4es; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DmWNv4es" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4996f1ee4a4so12087395e9.2 for ; Wed, 19 Aug 2026 11:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787165634; x=1787770434; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=LeaiBk7jtPj1JV25dNEYhySyvqZDMBq0LxTmZgiU7l8=; b=DmWNv4esFn7DdiJItMgn6QRWY0tl2+184KOopMumeesGjLOJo5EYUnckS/byrIvopu eluOV1QZ1hd+hOv8NjRspETi6Ps6seiv5B4ICg81erxDq0xm02AqKZAZdY5YE/m02E2q xGQy/SuC0GajeRaBRRSiRBYfEQR6dItJcVzk0h/ce46dLp14qpP608alZwZMdDLqcDp/ K1+WF4DluhvGUrGL7bpeROByuqysjJBS3lntd9lRAcvlocs0yu/WFsRok3jutxfyxTef DUqEfIdie2LSkIkxX704x8W6CMKKA/L8MsQ1VLnbmkDcTniCbYpTit5x9PQV9kc1c2ay zpxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787165634; x=1787770434; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LeaiBk7jtPj1JV25dNEYhySyvqZDMBq0LxTmZgiU7l8=; b=gzzF8JivxUMB475PO8D7ClRN7aP7ww/i7qfDBlttwMGgi13GSqPaX/bQNyP55CV084 DNBp+EFcElKiRxMDS9CB02dy0YXtBzAI0uyG6lOg37W2CfK5VOcu5/ESkNq01mHZJtxt 68NOKbtK8UVgSqSXFtWjLYopV2OvOVZhVui/gl4aYLDrXROjgAP8mhR1koKRQq5c9CXG hEvn+/BIS1GA7Qeh4HYggyyqgy2kzH9SmwehfqRiIv62atS6gRGro0Ere9pzJ0kxTBDr 0mnPmPfIz52uSsvXdWueY9yIuSRkhjl5AH2G3Vlwi/6y5bG9c0A25nJQfsUgHzLFIE1o VWcA== X-Forwarded-Encrypted: i=1; AHgh+Rq0CNAIlW9OtEw0mEhtBUhwHffQtDFdGqet1OFjWeY4rSLVS6XljyvIUg8Z74fd9k20EskeXph/Z8f53Bw=@vger.kernel.org X-Gm-Message-State: AOJu0Yw8o3kfMrao39aCqw/O85pXM+17KMiSYlrkMNnHhgs38+tHFO5D Ym2oXJohK72FAmH3bpWZP4uEH/cBucslsD0JNDgvaIA0wodp6L8O0yU8 X-Gm-Gg: AR+sD11h7IQsCy6y1NfsHhrQuQLFHYtcKkD4x0SPKIHQrIffAzkza1Gl63PQH1vEPyK aG4+Lhjf3HfM4Op9RlSYiWGx9VxzCeKbGYStIpH8CgZKqNprsvQev4BQip6VfjS1wl41hz6dt3U 7oEsymZp14SfOV0s02Le4CzFfd6poj4K/pjxi51pmxaCQXquPS5igWch5LtKawDkwab8HwHhuR8 K6aU9o3zhGOJmKSniDlOB88CBuObN3jGi8IwJYQEUgGmxa8aLJSUIUWu8GVUk8GBBS4KP1a4SGR LytenCYBfW7d0wVLmwwnL5Ux5/4TZbi+myjz9tulI7/kozdFclnRSBuB0jyZhB5zxQrRnophs9K b0SP+3utmJx53eUYC/5fCqeNacqGxRJWIeUlxe+jiyEkiUHftyglx/04b0+KzICm5MiXF4iu6H7 pOJ2QdF+ADIIZuWi33QJmFreBVrUuYYGZ0a4Ynoa8q8iw6MuzV7e9uCfrW4JNzqn/GgMZwv+PZU 34jGsjTWqdVaHeuvI1dc2nUbpAe9jFkeNyuspCbyco6Nb1SYiHzsOKMS8UhHL+cgAZP/Ddc7koo BBpPv03y6Ccunbf+ X-Received: by 2002:a05:600c:4ec9:b0:495:6e68:5df2 with SMTP id 5b1f17b1804b1-499aa1e33femr110408125e9.12.1787165633486; Wed, 19 Aug 2026 11:53:53 -0700 (PDT) Received: from horsehead.lan (p548c9e2a.dip0.t-ipconnect.de. [84.140.158.42]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-499aa0d6721sm104205315e9.11.2026.08.19.11.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Aug 2026 11:53:53 -0700 (PDT) From: Denis Pisarev To: amd-gfx@lists.freedesktop.org Cc: alexander.deucher@amd.com, christian.koenig@amd.com, mario.limonciello@amd.com, ionut_n2001@yahoo.com, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, Denis Pisarev Subject: [RFC PATCH 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume Date: Wed, 19 Aug 2026 20:53:48 +0200 Message-ID: <20260819185349.29407-1-pisarevden@gmail.com> X-Mailer: git-send-email 2.55.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Hi, This RFC follows up on bugzilla 219492 ("failed to write reg 28b4 wait reg 28c6" after hibernation) and on the resume_gpu_stable proposal rejected in January: https://lore.kernel.org/amd-gfx/20260106125929.25214-6-sunlightlinux@gmail.com/ The rejection argued that (a) kiq sched.ready is managed correctly - set false in suspend, set true again only after ring tests pass on resume - and (b) the proposed flag would have disabled the KIQ flush path permanently. Both points are correct, and this RFC takes a different route: nothing is disabled proactively; the driver reacts to observed failures. MEASURED FAILURE MODE (Cezanne iGPU, Ryzen 7 PRO 5850U, 7.1.8, KDE Wayland, no ROCm): - S4 resume -> "failed to write reg 28b4/28c6" + "1a6f4/1a706" pairs begin in the same second as the thaw and continue at 80-140/hour for 9+ hours until reboot. kiq sched.ready is true throughout (the ring test at resume passed). - Two user-visible consequences: each failed flush burns the ~5 s KIQ retry window (desktop-wide sluggishness), and the invalidation is then silently dropped (stale TLBs - correctness). TRIGGER ISOLATION (single-variable tests, one hibernate cycle each): - stock (GFXOFF allowed across S4): ~70-140 errors / 30 min of use - GFXOFF held off across S4: 0 errors (30 min) - GFXOFF re-enabled 40 min after the clean resume: 0 errors (15 min; gating resumption not independently verified) So the wedge forms in the S4 resume window while GFXOFF is allowed, and is consistent with the existing semaphore workaround comment in gmc_v9_0.c about losing invalidate-acknowledge state across power-gating cycles. A boot-time amdgpu.pg_mask=0 does not even initialize on this part (probe -ETIMEDOUT), so runtime debugfs control was used. THE PATCH (1/1): - amdgpu_gmc_fw_reg_write_reg_wait() returns -ETIME on failure and counts consecutive failures - gmc_v9_0_flush_gpu_tlb() uses its existing MMIO path as fallback when the KIQ submit fails (invalidations no longer dropped), and after 3 consecutive failures skips the KIQ path entirely until the counter resets - so wedged systems stop paying 5 s per flush - counter resets on every success and in gmc_v9_0_hw_fini(); each suspend/resume cycle re-arms KIQ. No proactive disabling anywhere. - gmc_v10/v11/v12 call sites unchanged and compile-safe (int return used as statement); extending the fallback to them is trivial once the approach is agreed. This mirrors what the sibling PASID path already does (KIQ submit, -ETIME on timeout, MMIO variant as fallback), applied to the per-VMID path. OPEN QUESTIONS: 1. Root cause in the S4 window: RLC/ME not processing KIQ invalidates while GFXOFF is allowed? Happy to run tracing (amdgpu.vm_debug, VM_INVALIDATE ack registers) on the affected hardware on request. 2. Should the latch be per-xcc-inst? 3. Should amdgpu_gmc_flush_gpu_tlb_pasid()'s KIQ branch share the latch? Patch 1/1 follows. Comments very welcome - especially on whether the failure-driven fallback is the right shape vs. fixing the S4 resume ordering directly. Denis Pisarev Denis Pisarev (1): drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is unresponsive drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 15 +++++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 4 +++- drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 18 ++++++++++++++---- 4 files changed, 30 insertions(+), 9 deletions(-) -- 2.55.0