From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E59A64195A1 for ; Thu, 20 Aug 2026 10:57:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787223452; cv=none; b=gqdGzYzLb+z0KeAEgtxQiMDM6W4YDGsrwyrbmjexr88EVoc7dedip5lk2LNUVdsmrKHHWZ4sQgDcNEKKks55hYtykG3KTQHfMHuOL+FuXiqqJcwsqUerLBwVfjkRa3/RhqTDWZkYc/L2QzmGFDFPwpCiLhbuEVM1aAd/a5/Zarg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787223452; c=relaxed/simple; bh=5V0B3Fgjy8M6qn8siVM9CIvU8ncEq6RljTWEG5IBY5w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=haCJ8APwXg9/Fso7K3x3nhtfBWf+bK/zOAjeOUcRQCH/TuqP0peeYGfk3WGrWpa0muVBPH4qMV+43zKuXWgRc/9d6j+78v/dr7/9vM2G/YEaO6MX0tACdRl8U0xj++naCyMN8kNY3xshAoOHMJoJ0R7X3hZgFaTd2fd0agQSkb0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=oCDMLZbZ; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oCDMLZbZ" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-47de0093c42so2010824f8f.3 for ; Thu, 20 Aug 2026 03:57:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787223449; x=1787828249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6vW9ld8Q7uobfbuxJfABp8mlFR5KPp0lNPptjS3cwDc=; b=oCDMLZbZPNo32iwXcL1BTdtKj+b3+7RrA1eydjWDsCDnmp/o2f1kww3lYnBY4WXJI2 zgwRANTZPFVl51p9r4ObQs3SSGlolT9S81iv9eE45RLIWG2+l02LHfIOr6hw1JGOg6au 3HS2hxxXVjDn6+JWcPQMY7jeuJGr/kcCZFX2jft3mAUqF6P/lU8Lav3LmsPl5BA/ATIq lpaYk9I7Mi5CemxfXssMeJo92Hzdaas/MYHauBQsw1Sju08WInGk/cYckiBLZbqHkt0E 2n5wKNX4LGeKbut7DgxfjxGhdPQJ8ceDg2iAsspq9eFFVL5JZI+FPZtJAUiQj1nXPsLf hQGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787223449; x=1787828249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6vW9ld8Q7uobfbuxJfABp8mlFR5KPp0lNPptjS3cwDc=; b=pCMXKiOPJdVh0DCiC7Nr0PHVDifvL1s37RvygjnJ16ZQpueR+wbRQSvHyw2tvZKeyE aQeh3E5d/xWq0MsnYUuLNFPFlz6KH5OHDG6fwfzQT3RMlXVLHkklniRfrvZ7BKIa8/2E UvN5wdowCjPCAPpBlOqCm1ogC5WcExyWsfvM0jQ14e5dBTAQTEj15Ct8cvzaBz1ynrvO 7vyN6lNcljvi0OVti3tihiFbmBvGiiJfy7zxPVeTvInOudyhUSAEKpzzSCGjy0Q6pB7i HnDw7ELvCxBYh0XYY+4McRpEGCvLj86AZganjqF+6Et+p2NWERN4jkS7DgRF2XqPEvRK FoCg== X-Forwarded-Encrypted: i=1; AHgh+RpcZCu3YRj6HZv9DTxSzrEfoV9o4aewPcvabmpPaHo92KWHFWEhSEstMwNBYi5gONfeL5bvDbsNzvmWP3M=@vger.kernel.org X-Gm-Message-State: AOJu0YxP6Z+7v7lNVSeRiCvQxHvKs1p56wQTWhcODVXBUFmqqnAcD5uH xJcyFP3er5i3IqSxsjBOn4330SJ+qHtMSVcnNgELpbRcS52avWdk15Yt X-Gm-Gg: AR+sD10sb2sM95fHaVb9HjoLH7l3ulqNzIeiwd6JhQdbCBfcWkXkjNWhbnShlt0mMg3 CqS8qm6AW3+/y2w/YTC0fiK+hYXIkTOpYZDEphtH+TUznBV+Eg9w+eyK1nGXP5jIMc1Nzh95IQJ WV5syiIbefx1haGLvPoZAm3nq7d7wNOwkfIplxm0/Ixbj5cbb6uZXgJFy17UJRBL1XsUZCNE3nU XfrFRh4fioXtTGFyPCtqf51bc0A6ifUElMZ3Z4l/MT7pGBVCh+BSkyrTlrxZ1iMwmh9IOy7btDz fWHVPZ/+bJ9xEDxHNotC1l/cABPKRuvhnjVXNAmFcXmN1y9dvT1x/us7rzibCJ6cXpr+/IXNg86 dY8UGZjdlXQehkea21S2bN7ka0idLkZggPsmcsU/oeGS1V6SvV9WvIH3//B0uZY/0z/pHJfpNuc QitCPdoLOH7UPYxNSlRIvMx6CMTI/hGWbjhvgNdlhG+HvRIMnJ5aRI49hdeVEL/6yL0mi87umzK 1qRCCyh6QijwS976XhQ8Pllr4331xU0kaytrysl/q7G9VnXzczlru0gejnlCF5RES0bp0O1Nkle ER62HA== X-Received: by 2002:a05:600c:a30f:b0:499:b01e:911c with SMTP id 5b1f17b1804b1-499b01e9125mr89248005e9.2.1787223448742; Thu, 20 Aug 2026 03:57:28 -0700 (PDT) Received: from horsehead.lan (p548c9e2a.dip0.t-ipconnect.de. [84.140.158.42]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-499b4568a65sm13872775e9.2.2026.08.20.03.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 03:57:28 -0700 (PDT) From: Denis Pisarev To: amd-gfx@lists.freedesktop.org Cc: alexander.deucher@amd.com, christian.koenig@amd.com, mario.limonciello@amd.com, ionut_n2001@yahoo.com, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, Denis Pisarev Subject: [RFC PATCH v2 1/1] drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is unresponsive Date: Thu, 20 Aug 2026 12:57:16 +0200 Message-ID: <20260820105716.148732-2-pisarevden@gmail.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260820105716.148732-1-pisarevden@gmail.com> References: <20260819185349.29407-1-pisarevden@gmail.com> <20260820105716.148732-1-pisarevden@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit After resume from S4 (hibernation) on gmc_v9 parts with GFXOFF (observed on Cezanne / Ryzen 7 PRO 5850U, kernel 7.1.8), KIQ-based TLB flushes start failing at the moment of the thaw and keep failing for hours of normal desktop use: amdgpu 0000:07:00.0: failed to write reg 28b4 wait reg 28c6 amdgpu 0000:07:00.0: failed to write reg 1a6f4 wait reg 1a706 (80-140 errors/hour measured over 9+ hours; bugzilla 219492). The KIQ ring stays sched.ready throughout, so readiness does not reflect the state of the hardware in this failure mode. Two problems follow from the current code: every failed flush burns the full ~5 s KIQ retry window before erroring out (desktop-wide sluggishness), and the invalidation is then silently dropped (stale TLBs - correctness). Make the failure observable and self-healing: - amdgpu_gmc_fw_reg_write_reg_wait() returns 0/-ETIME, propagates MES errors, and counts consecutive failures per KIQ instance - gmc_v9_0_flush_gpu_tlb() falls back to a new gmc_v9_0_flush_gpu_tlb_mmio() helper (the former pre-KIQ MMIO code, now with irqsave locking since it is reachable at runtime) when the KIQ submit fails, so the invalidation is no longer dropped - after AMDGPU_KIQ_FLUSH_MAX_FAIL (3) consecutive failures the KIQ path is skipped entirely until the counter resets, so wedged systems stop paying the 5 s retry window per flush - the MMIO fallback is restricted to process context on bare metal: amdgpu_gfx_off_ctrl() is used to hold the GC block awake across the direct register access (it may sleep), and SR-IOV VFs keep proxying through KIQ as before - the counter resets on every success and in gmc_v9_0_hw_fini(); every suspend/resume cycle re-arms the KIQ path; nothing is disabled proactively Changes since v1 (addressing the sashiko-bot review): - hold GFXOFF off around runtime MMIO access (was: unguarded - could hit power-gated registers) - per-KIQ-instance failure counter instead of a global one (multi-XCC cross-talk) - invalidate_lock is now taken irqsave (the path is runtime-reachable) - VFs and interrupt context never take the runtime MMIO fallback - MES path errors are propagated instead of hardcoded success - reworded the threshold warning to not promise a fallback that gmc_v10/v11/v12 do not implement Known limitation / open question: a KIQ command that already timed out stays queued in the ring; if the ring recovers late it could emit a duplicate invalidation concurrently with a CPU MMIO flush. The command is semantically an idempotent invalidate, but reviewer input on whether the req/ack handshake needs protection here is welcome. gmc_v10/v11/v12 call sites are unchanged and compile-safe (int return used as statement). They can get the same fallback once the approach is agreed for gmc_v9. The sibling PASID path (amdgpu_gmc_flush_gpu_tlb_pasid) already has an -ETIME/MMIO split; this brings the per-VMID path in line with it. Root-cause note: with GFXOFF held off across the S4 cycle (debugfs amdgpu_gfxoff), zero errors occur across resume and 30 min of use vs ~70-140 in the control arm; the wedge forms in the S4 resume window while GFXOFF is allowed, consistent with the existing semaphore workaround comment in gmc_v9_0.c. Signed-off-by: Denis Pisarev --- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 18 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 111 ++++++++++++++++++------ 5 files changed, 102 insertions(+), 33 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h index 7b09410d6..cd5d9e56e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -360,6 +360,8 @@ enum amdgpu_kiq_irq { #define MAX_KIQ_REG_WAIT 5000 /* in usecs, 5ms */ #define MAX_KIQ_REG_BAILOUT_INTERVAL 5 /* in msecs, 5ms */ #define MAX_KIQ_REG_TRY 1000 +/* consecutive KIQ TLB flush failures before falling back to MMIO */ +#define AMDGPU_KIQ_FLUSH_MAX_FAIL 3 /* * BIOS. diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h index 54c1eb9c4..e2aceb99c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h @@ -167,6 +167,8 @@ struct amdgpu_kiq { struct amdgpu_irq_src irq; const struct kiq_pm4_funcs *pmf; void *mqd_backup; + /* consecutive TLB flush reg access failures on this instance */ + atomic_t flush_failures; }; /* diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c index 5d6149ba7..49d3d6651 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c @@ -874,7 +874,7 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid, return r; } -void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, +int amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, uint32_t reg0, uint32_t reg1, uint32_t ref, uint32_t mask, uint32_t xcc_inst) @@ -886,9 +886,8 @@ void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, uint32_t seq; if (adev->mes.ring[MES_PIPE_INST(xcc_inst, 0)].sched.ready) { - amdgpu_mes_reg_write_reg_wait(adev, reg0, reg1, - ref, mask, xcc_inst); - return; + return amdgpu_mes_reg_write_reg_wait(adev, reg0, reg1, + ref, mask, xcc_inst); } spin_lock_irqsave(&kiq->ring_lock, flags); @@ -919,13 +918,20 @@ void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, if (cnt > MAX_KIQ_REG_TRY) goto failed_kiq; - return; + atomic_set(&adev->gfx.kiq[xcc_inst].flush_failures, 0); + return 0; failed_undo: amdgpu_ring_undo(ring); spin_unlock_irqrestore(&kiq->ring_lock, flags); failed_kiq: - dev_err(adev->dev, "failed to write reg %x wait reg %x\n", reg0, reg1); + if (atomic_inc_return(&adev->gfx.kiq[xcc_inst].flush_failures) == + AMDGPU_KIQ_FLUSH_MAX_FAIL) + dev_warn(adev->dev, + "KIQ reg access keeps failing, MMIO fallback recommended\n"); + dev_err_ratelimited(adev->dev, + "failed to write reg %x wait reg %x\n", reg0, reg1); + return -ETIME; } /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h index ddb0d500e..303e0ee98 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h @@ -447,7 +447,7 @@ void amdgpu_gmc_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid, uint32_t flush_type, bool all_hub, uint32_t inst); -void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, +int amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, uint32_t reg0, uint32_t reg1, uint32_t ref, uint32_t mask, uint32_t xcc_inst); diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c index 8a5c44810..11fc9085e 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c @@ -764,27 +764,28 @@ static bool gmc_v9_0_get_atc_vmid_pasid_mapping_info(struct amdgpu_device *adev, */ /** - * gmc_v9_0_flush_gpu_tlb - tlb flush with certain type + * gmc_v9_0_flush_gpu_tlb_mmio - tlb flush via direct MMIO * * @adev: amdgpu_device pointer + * @hub: vmhub to flush * @vmid: vm instance to flush * @vmhub: which hub to flush - * @flush_type: the flush type + * @inv_req: invalidation request payload * - * Flush the TLB for the requested page table using certain type. + * Direct CPU access to the invalidation engine. Callers must ensure + * the target block cannot power gate across the access (GFXOFF needs + * to be held off at runtime) and must hold no other locks. */ -static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, - uint32_t vmhub, uint32_t flush_type) +static void gmc_v9_0_flush_gpu_tlb_mmio(struct amdgpu_device *adev, + struct amdgpu_vmhub *hub, + uint32_t vmid, uint32_t vmhub, + u32 inv_req) { bool use_semaphore = gmc_v9_0_use_invalidate_semaphore(adev, vmhub); - u32 j, inv_req, tmp, sem, req, ack, inst; const unsigned int eng = 17; - struct amdgpu_vmhub *hub; - - BUG_ON(vmhub >= AMDGPU_MAX_VMHUBS); + unsigned long flags; + u32 j, tmp, sem, req, ack, inst; - hub = &adev->vmhub[vmhub]; - inv_req = gmc_v9_0_get_invalidate_req(vmid, flush_type); sem = hub->vm_inv_eng0_sem + hub->eng_distance * eng; req = hub->vm_inv_eng0_req + hub->eng_distance * eng; ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng; @@ -794,21 +795,8 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, else inst = vmhub; - /* This is necessary for SRIOV as well as for GFXOFF to function - * properly under bare metal - */ - if (adev->gfx.kiq[inst].ring.sched.ready && - (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev))) { - uint32_t req = hub->vm_inv_eng0_req + hub->eng_distance * eng; - uint32_t ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng; - - amdgpu_gmc_fw_reg_write_reg_wait(adev, req, ack, inv_req, - 1 << vmid, inst); - return; - } - /* This path is needed before KIQ/MES/GFXOFF are set up */ - spin_lock(&adev->gmc.invalidate_lock); + spin_lock_irqsave(&adev->gmc.invalidate_lock, flags); /* * It may lose gpuvm invalidate acknowldege state across power-gating @@ -870,7 +858,7 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, WREG32_SOC15_IP_NO_KIQ(GC, sem, 0, GET_INST(GC, inst)); } - spin_unlock(&adev->gmc.invalidate_lock); + spin_unlock_irqrestore(&adev->gmc.invalidate_lock, flags); if (j < adev->usec_timeout) return; @@ -878,6 +866,70 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, DRM_ERROR("Timeout waiting for VM flush ACK!\n"); } +/** + * gmc_v9_0_flush_gpu_tlb - tlb flush with certain type + * + * @adev: amdgpu_device pointer + * @vmid: vm instance to flush + * @vmhub: which hub to flush + * @flush_type: the flush type + * + * Flush the TLB for the requested page table using certain type. + */ +static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, + uint32_t vmhub, uint32_t flush_type) +{ + u32 inv_req; + const unsigned int eng = 17; + struct amdgpu_vmhub *hub; + u32 inst; + + BUG_ON(vmhub >= AMDGPU_MAX_VMHUBS); + + hub = &adev->vmhub[vmhub]; + inv_req = gmc_v9_0_get_invalidate_req(vmid, flush_type); + + if (vmhub >= AMDGPU_MMHUB0(0)) + inst = 0; + else + inst = vmhub; + + /* This is necessary for SRIOV as well as for GFXOFF to function + * properly under bare metal + */ + if (adev->gfx.kiq[inst].ring.sched.ready && + (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev))) { + uint32_t req = hub->vm_inv_eng0_req + hub->eng_distance * eng; + uint32_t ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng; + + if (atomic_read(&adev->gfx.kiq[inst].flush_failures) < + AMDGPU_KIQ_FLUSH_MAX_FAIL) { + if (!amdgpu_gmc_fw_reg_write_reg_wait(adev, req, ack, + inv_req, + 1 << vmid, inst)) + return; + /* KIQ submit failed; error already logged */ + } + + /* + * MMIO fallback: invalidation must not be silently dropped + * when KIQ is unresponsive. Direct register access is only + * safe in process context (amdgpu_gfx_off_ctrl may sleep) + * and on bare metal; SR-IOV VFs must keep proxying through + * KIQ. + */ + if (in_interrupt() || amdgpu_sriov_vf(adev)) + return; + + amdgpu_gfx_off_ctrl(adev, false); + gmc_v9_0_flush_gpu_tlb_mmio(adev, hub, vmid, vmhub, inv_req); + amdgpu_gfx_off_ctrl(adev, true); + return; + } + + gmc_v9_0_flush_gpu_tlb_mmio(adev, hub, vmid, vmhub, inv_req); +} + /** * gmc_v9_0_flush_gpu_tlb_pasid - tlb flush via pasid * @@ -2237,6 +2289,13 @@ static void gmc_v9_0_gart_disable(struct amdgpu_device *adev) static int gmc_v9_0_hw_fini(struct amdgpu_ip_block *ip_block) { struct amdgpu_device *adev = ip_block->adev; + int i; + + /* KIQ instances are re-initialized on the next resume; re-arm + * the MMIO fallback logic + */ + for (i = 0; i < AMDGPU_MAX_GC_INSTANCES; i++) + atomic_set(&adev->gfx.kiq[i].flush_failures, 0); gmc_v9_0_gart_disable(adev); -- 2.55.0