mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust
@ 2026-11-20 15:28 Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers Joe Damato
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev
  Cc: andrew+netdev, davem, edumazet, kuba, pabeni, horms,
	michael.chan, pavan.chebbi, selvin.xavier,
	kalesh-anakkur.purayil, linux-kernel, Joe Damato

Greetings:

On two production systems, I saw the following dmesg pattern:

  NETDEV WATCHDOG: transmit queue 0 timed out 6073 ms
  Resp cmpl intr err msg: 0x51                  x20
  hwrm_ring_free type 1 failed                  x12
  hwrm_ring_free type 2 failed                  x8
  AMD-Vi: IO_PAGE_FAULT  x3

This suggests that, for some currently unknown reason, TX completions stall
and the netdev watchdog fires. The driver asks FW to free the rings, this
times out, but the driver ignores the possible failure and frees ring memory.
Since the FW didn't respond to the ring free command, it is possible that the
FW is still DMAing to the memory which was freed.

To address this, this series makes the driver stop DMA before freeing ring
memory when the firmware did not answer RING_FREE.

Since the RFCv4, I found a third system that hit this repeatedly:

  NETDEV WATCHDOG: CPU: 35: transmit queue 0 timed out 5459 ms
  hwrm_ring_free type 1 failed. rc:fffffff0 err:0    x64
  hwrm_ring_free type 2 failed. rc:fffffff0 err:0    x32
  hwrm_ring_free type 0 failed. rc:fffffffb err:1    x64
  hwrm_ring_alloc type 1 failed. rc:ffffffe4 err:4
  hwrm_ring_free type 1 failed. rc:fffffff0 err:0    x44
  Abandoning msg {0x51 0x3e35} len: 0 due to firmware status: 0x408100
  nic open fail (rc: fffffffb)
  DMAR: [DMA Write NO_PASID] Request device [47:00.0] fault addr
  0xffffa07ef000 [fault reason 0x05] PTE Write access is not set

This case differs from the first two in a few ways, but this series would have
still helped:

 - The ring frees fail on the close path and again on the error path of
   the reopen (bnxt_init_chip()), and the device then writes to memory the
   driver has already unmapped. The series stops DMA on both paths.

 - The interface came back up later, the watchdog fired again, and the
   cycle started over. The host logged 79 DMAR write faults in about 20
   hours. Patch 4 breaks this loop by refusing to reopen after DMA has
   been stopped.

 - The firmware answers each RING_FREE about 5 seconds late, after the
   driver has given up, and finally reports an error in its health
   register. Patch 2 does not treat these late responses as success because
   their sequence ids do not match.

In this most recent case, I suspect the hardware is faulty, but the driver
shouldn't hand memory back to the system while the device could still write to
it.

In all of the above cases, the IOMMU caught the error. If the system had no
IOMMU or the IOMMU was in passthrough mode, data corruption or crash would
have occurred.

In the future, this could possibly be extended to make recovery automatic.

I wonder if Sashiko reads cover letters and then avoids flagging stuff, let's
find out. (Unrelated: I wonder if anyone reads cover letters).

Sashiko: you pointed out a few issues which are real bugs but were not
observed in production and I am leaving out of this series which only
addresses an issue I've seen in production.

In particular, I am leaving out:
  - the per queue paths (real, but not observed in production)
  - completion and nq failures are not counted as stuck (all rings fail in
    prod, have not observed only nq failures)
  - The polling path in __hwrm_send() probably needs a dma_rmb (sashiko
    pointed out this pre-existing issue that is probably unlikely but also
    unobserved)
  - bnxt_resume, bnxt_fw_reset_task, and bnxt_io_slot_reset all call
    pci_set_master without checking the DMA bit

All of the above was outlined in the thread about the v5 [1].

Thanks,
Joe

[1]: https://lore.kernel.org/netdev/avIpEpoCYvLHYl3j@devvm20253.cco0.facebook.com/

v6:
 - Commit messages across all patches were updated and a new patch was added
   to the series, patch 5.
 - Patch 1:
    - Return the errno from hwrm_req_send instead of always -EIO.

 - Patch 2:
    - Respect BNXT_HWRM_CTX_SILENT for the new warning.
    - Read the valid byte with READ_ONCE(), then dma_rmb() before reading
      the rest of the response, including error_code.
    - Only check the response for HWRM_RING_FREE. HWRM_SELFTEST_IRQ and
      HWRM_RING_RESET keep the existing behavior.

 - Patch 3:
    - Count a ring as stuck only when no response arrived, not when the
      firmware replied with an error.
    - Don't stop DMA when BNXT_STATE_IN_FW_RESET or BNXT_STATE_FW_FATAL_COND is
      set.

 - Patch 4:
   - Add BNXT_STATE_DMA_STOPPED to BNXT_NO_FW_ACCESS() so HWRM commands sent
     after DMA is stopped fail right away instead of waiting for the timeout.

 - Patch 5 (new):
   - stop the ULP around bnxt_reset() and leave it stopped if DMA was
     stopped. The silent reset after a VF config change leaves the ULP
     running.

v5: https://lore.kernel.org/netdev/20261106191423.4194081-1-joe@dama.to/
 - Switched from RFC to actual submission
 - Patch 4 updated to use an un-used bit to make backporting easier.

v4: https://lore.kernel.org/netdev/20260925174404.2789072-1-joe@dama.to/
  - No changes to patch 1 or 2
  - Patch 3: reworded the message logged when DMA is stopped, a rebind is
    what recovers the device, not a firmware reset. No functional change.
  - Patch 4 added which adds a new bit (BNXT_STATE_DMA_STOPPED) that is set
    when DMA is disabled. When this bit is set, the device requires user
    intervention to bring back up.

v3: https://lore.kernel.org/netdev/20260923210744.3406861-1-joe@dama.to/
  - No changes to patch 1
  - Patch 2: Don't poll for the valid bit as Michael suggested.
  - Patch 3: bnxt_hwrm_ring_free now returns -EIO instead of stopping the
    device, so the remaining resources can be freed and the remaining commands
    can be sent before stopping the device, as Michael suggested. Note the
    switch to using pci_clear_master in this patch instead of
    pci_disable_device. This was done so that the normal shutdown paths can
    call pci_disable_device without generating a warning.

v2: https://lore.kernel.org/netdev/20260922182405.1290749-1-joe@dama.to/
  - No changes to patch 1
  - Patch 2 from v1 dropped
  - Patch 2 in the v2 now checks the response and logs state before giving up
  - Patch 3 in the v2 disables the device to stop DMA before freeing ring
    memory

RFCv1: https://lore.kernel.org/netdev/20260917233218.1160001-1-joe@dama.to/


Joe Damato (5):
  bnxt_en: return the RING_FREE status to callers
  bnxt_en: check HWRM response if completion never arrives
  bnxt_en: stop DMA before releasing rings the firmware did not free
  bnxt_en: refuse to open a device with stopped DMA
  bnxt_en: stop the ULP around the reset task

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 141 +++++++++++++-----
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |   2 +
 .../net/ethernet/broadcom/bnxt/bnxt_hwrm.c    |  47 +++++-
 3 files changed, 149 insertions(+), 41 deletions(-)


base-commit: 00f445721c7f0e48aaff9813cfe1c065c6eb3b71
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers
  2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
@ 2026-11-20 15:28 ` Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 2/5] bnxt_en: check HWRM response if completion never arrives Joe Damato
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev, Michael Chan, Pavan Chebbi, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Prashant Sreedharan
  Cc: edumazet, horms, selvin.xavier, kalesh-anakkur.purayil,
	linux-kernel, Joe Damato

hwrm_ring_free_send_msg() reports failure to its caller, but always as
-EIO, so a caller cannot tell whether the firmware rejected
HWRM_RING_FREE or never answered it. The TX, RX and AGG ring free
helpers discard the value.

Return the errno from hwrm_req_send() instead. -EBUSY means no response
arrived: the wait timed out, was abandoned, or the firmware channel is
down. An error reported by the firmware keeps its own errno, for example
a firmware BUSY reply becomes -EAGAIN. Return the value from the TX, RX
and AGG ring free helpers.

No caller acts on it yet, so there is no functional change.

Fixes: 74608fc98d28 ("bnxt_en: Ring free response from close path should use completion ring")
Signed-off-by: Joe Damato <joe@dama.to>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 50 +++++++++++++----------
 1 file changed, 29 insertions(+), 21 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index e78cf7ad3cbc..c3413354e758 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -7673,55 +7673,62 @@ static int hwrm_ring_free_send_msg(struct bnxt *bp,
 	if (rc || error_code) {
 		netdev_err(bp->dev, "hwrm_ring_free type %d failed. rc:%x err:%x\n",
 			   ring_type, rc, error_code);
+		if (rc)
+			return rc;
 		return -EIO;
 	}
 	return 0;
 }
 
-static void bnxt_hwrm_tx_ring_free(struct bnxt *bp,
-				   struct bnxt_tx_ring_info *txr,
-				   bool close_path)
+static int bnxt_hwrm_tx_ring_free(struct bnxt *bp,
+				  struct bnxt_tx_ring_info *txr,
+				  bool close_path)
 {
 	struct bnxt_ring_struct *ring = &txr->tx_ring_struct;
 	u32 cmpl_ring_id;
+	int rc;
 
 	if (ring->fw_ring_id == INVALID_HW_RING_ID)
-		return;
+		return 0;
 
 	cmpl_ring_id = close_path ? bnxt_cp_ring_for_tx(bp, txr) :
 		       INVALID_HW_RING_ID;
-	hwrm_ring_free_send_msg(bp, ring, RING_FREE_REQ_RING_TYPE_TX,
-				cmpl_ring_id);
+	rc = hwrm_ring_free_send_msg(bp, ring, RING_FREE_REQ_RING_TYPE_TX,
+				     cmpl_ring_id);
 	ring->fw_ring_id = INVALID_HW_RING_ID;
+	return rc;
 }
 
-static void bnxt_hwrm_rx_ring_free(struct bnxt *bp,
-				   struct bnxt_rx_ring_info *rxr,
-				   bool close_path)
+static int bnxt_hwrm_rx_ring_free(struct bnxt *bp,
+				  struct bnxt_rx_ring_info *rxr,
+				  bool close_path)
 {
 	struct bnxt_ring_struct *ring = &rxr->rx_ring_struct;
 	u32 grp_idx = rxr->bnapi->index;
 	u32 cmpl_ring_id;
+	int rc;
 
 	if (ring->fw_ring_id == INVALID_HW_RING_ID)
-		return;
+		return 0;
 
 	cmpl_ring_id = bnxt_cp_ring_for_rx(bp, rxr);
-	hwrm_ring_free_send_msg(bp, ring,
-				RING_FREE_REQ_RING_TYPE_RX,
-				close_path ? cmpl_ring_id :
-				INVALID_HW_RING_ID);
+	rc = hwrm_ring_free_send_msg(bp, ring,
+				     RING_FREE_REQ_RING_TYPE_RX,
+				     close_path ? cmpl_ring_id :
+				     INVALID_HW_RING_ID);
 	ring->fw_ring_id = INVALID_HW_RING_ID;
 	bp->grp_info[grp_idx].rx_fw_ring_id = INVALID_HW_RING_ID;
+	return rc;
 }
 
-static void bnxt_hwrm_rx_agg_ring_free(struct bnxt *bp,
-				       struct bnxt_rx_ring_info *rxr,
-				       bool close_path)
+static int bnxt_hwrm_rx_agg_ring_free(struct bnxt *bp,
+				      struct bnxt_rx_ring_info *rxr,
+				      bool close_path)
 {
 	struct bnxt_ring_struct *ring = &rxr->rx_agg_ring_struct;
 	u32 grp_idx = rxr->bnapi->index;
 	u32 type, cmpl_ring_id;
+	int rc;
 
 	if (bp->flags & BNXT_FLAG_CHIP_P5_PLUS)
 		type = RING_FREE_REQ_RING_TYPE_RX_AGG;
@@ -7729,14 +7736,15 @@ static void bnxt_hwrm_rx_agg_ring_free(struct bnxt *bp,
 		type = RING_FREE_REQ_RING_TYPE_RX;
 
 	if (ring->fw_ring_id == INVALID_HW_RING_ID)
-		return;
+		return 0;
 
 	cmpl_ring_id = bnxt_cp_ring_for_rx(bp, rxr);
-	hwrm_ring_free_send_msg(bp, ring, type,
-				close_path ? cmpl_ring_id :
-				INVALID_HW_RING_ID);
+	rc = hwrm_ring_free_send_msg(bp, ring, type,
+				     close_path ? cmpl_ring_id :
+				     INVALID_HW_RING_ID);
 	ring->fw_ring_id = INVALID_HW_RING_ID;
 	bp->grp_info[grp_idx].agg_fw_ring_id = INVALID_HW_RING_ID;
+	return rc;
 }
 
 static void bnxt_hwrm_cp_ring_free(struct bnxt *bp,
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH net v6 2/5] bnxt_en: check HWRM response if completion never arrives
  2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers Joe Damato
@ 2026-11-20 15:28 ` Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 3/5] bnxt_en: stop DMA before releasing rings the firmware did not free Joe Damato
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev, Michael Chan, Pavan Chebbi, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Prashant Sreedharan
  Cc: edumazet, horms, selvin.xavier, kalesh-anakkur.purayil,
	linux-kernel, Joe Damato

When HWRM_RING_FREE is sent over a completion ring, __hwrm_send() waits
for NAPI to consume the completion and gives up if it never arrives,
without looking at the response.

If a completion is not posted within the timeout, check the response
before giving up. If resp_len is set, the sequence id matches, and the
valid byte is set then the firmware completed the command and only the
notification was lost. Fall through to the normal error_code handling in
that case.

This check runs only after the full command timeout. A response that is
still being written reads a zero valid byte, so the command is still
treated as failed. The rest of the response is read only after the valid
byte has been seen.

Log the response state on both paths so there is more data when this rare
event occurs.

Fixes: 74608fc98d28 ("bnxt_en: Ring free response from close path should use completion ring")
Signed-off-by: Joe Damato <joe@dama.to>
---
 .../net/ethernet/broadcom/bnxt/bnxt_hwrm.c    | 47 +++++++++++++++++--
 1 file changed, 43 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_hwrm.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_hwrm.c
index de02d1baa751..53667c0c38aa 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_hwrm.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_hwrm.c
@@ -582,11 +582,50 @@ static int __hwrm_send(struct bnxt *bp, struct bnxt_hwrm_ctx *ctx)
 		}
 
 		if (READ_ONCE(token->state) != BNXT_HWRM_COMPLETE) {
-			hwrm_err(bp, ctx, "Resp cmpl intr err msg: 0x%x\n",
-				 req_type);
-			goto exit;
+			__le16 resp_seq_id;
+			u8 valid_byte = 0;
+
+			/* Only HWRM_RING_FREE checks the response here. The
+			 * other users of a completion ring depend on the
+			 * completion itself: the IRQ self test, and
+			 * HWRM_RING_RESET, which reuses the ring. A cancelled
+			 * wait means the device is being torn down.
+			 */
+			if (req_type != HWRM_RING_FREE ||
+			    READ_ONCE(token->state) == BNXT_HWRM_CANCELLED) {
+				hwrm_err(bp, ctx, "Resp cmpl intr err msg: 0x%x\n",
+					 req_type);
+				goto exit;
+			}
+
+			/* The completion was not delivered, but the firmware
+			 * may still have freed the ring. Check the response
+			 * before giving up and log the state.
+			 */
+			resp_seq_id = READ_ONCE(ctx->resp->seq_id);
+			len = le16_to_cpu(READ_ONCE(ctx->resp->resp_len));
+			if (len && resp_seq_id == ctx->req->seq_id)
+				valid_byte = READ_ONCE(*((u8 *)ctx->resp + len - 1));
+
+			if (!valid_byte) {
+				hwrm_err(bp, ctx,
+					 "Resp cmpl intr err msg: 0x%x len:%d seq:0x%x/0x%x\n",
+					 req_type, len,
+					 le16_to_cpu(resp_seq_id),
+					 le16_to_cpu(ctx->req->seq_id));
+				goto exit;
+			}
+
+			/* Read the rest of the response after the valid byte */
+			dma_rmb();
+			if (!(ctx->flags & BNXT_HWRM_CTX_SILENT))
+				netdev_warn(bp->dev,
+					    "Resp cmpl intr not delivered, msg: 0x%x completed anyway (len:%d valid:0x%x err:0x%x)\n",
+					    req_type, len, valid_byte,
+					    le16_to_cpu(ctx->resp->error_code));
+		} else {
+			len = le16_to_cpu(READ_ONCE(ctx->resp->resp_len));
 		}
-		len = le16_to_cpu(READ_ONCE(ctx->resp->resp_len));
 		valid = ((u8 *)ctx->resp) + len - 1;
 	} else {
 		__le16 seen_out_of_seq = ctx->req->seq_id; /* will never see */
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH net v6 3/5] bnxt_en: stop DMA before releasing rings the firmware did not free
  2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 2/5] bnxt_en: check HWRM response if completion never arrives Joe Damato
@ 2026-11-20 15:28 ` Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 4/5] bnxt_en: refuse to open a device with stopped DMA Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 5/5] bnxt_en: stop the ULP around the reset task Joe Damato
  4 siblings, 0 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev, Michael Chan, Pavan Chebbi, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Prashant Sreedharan
  Cc: edumazet, horms, selvin.xavier, kalesh-anakkur.purayil,
	linux-kernel, Joe Damato

When HWRM_RING_FREE is not answered, bnxt_hwrm_ring_free() clears
fw_ring_id and __bnxt_close_nic() goes on to call bnxt_free_mem(), which
unmaps the ring memory and the RX buffers that the FW may still be
using.

This is reachable in production. On a BCM57504 the first sign is the TX
watchdog; the close that follows times out a subset of its RING_FREEs and
the driver releases those rings anyway:

  05:30:12  NETDEV WATCHDOG: transmit queue 0 timed out 6073 ms
  05:30:12  Resp cmpl intr err msg: 0x51                  x20
  05:30:12  hwrm_ring_free type 1 failed                  x12
  05:30:12  hwrm_ring_free type 2 failed                  x8
  05:30:12  AMD-Vi: IO_PAGE_FAULT  x3

Count the TX, RX and AGG rings whose RING_FREE got no response and
report that to the caller, which stops DMA before the ring memory is
released. This is skipped during a firmware reset or after a fatal
firmware error, where the reset path already handles the device.

The device stays unusable until the driver is rebound.

Fixes: 74608fc98d28 ("bnxt_en: Ring free response from close path should use completion ring")
Signed-off-by: Joe Damato <joe@dama.to>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 61 ++++++++++++++++++-----
 1 file changed, 48 insertions(+), 13 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index c3413354e758..600a8c866f3c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -7774,21 +7774,31 @@ static void bnxt_clear_one_cp_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cp
 			memset(cpr->cp_desc_ring[i], 0, size);
 }
 
-static void bnxt_hwrm_ring_free(struct bnxt *bp, bool close_path)
+static int bnxt_hwrm_ring_free(struct bnxt *bp, bool close_path)
 {
+	int stuck = 0;
 	u32 type;
 	int i;
 
 	if (!bp->bnapi)
-		return;
+		return 0;
 
+	/* -EBUSY means the firmware never answered, so it may still own
+	 * the ring.
+	 */
 	for (i = 0; i < bp->tx_nr_rings; i++)
-		bnxt_hwrm_tx_ring_free(bp, &bp->tx_ring[i], close_path);
+		if (bnxt_hwrm_tx_ring_free(bp, &bp->tx_ring[i],
+					   close_path) == -EBUSY)
+			stuck++;
 
 	bnxt_cancel_dim(bp);
 	for (i = 0; i < bp->rx_nr_rings; i++) {
-		bnxt_hwrm_rx_ring_free(bp, &bp->rx_ring[i], close_path);
-		bnxt_hwrm_rx_agg_ring_free(bp, &bp->rx_ring[i], close_path);
+		struct bnxt_rx_ring_info *rxr = &bp->rx_ring[i];
+
+		if (bnxt_hwrm_rx_ring_free(bp, rxr, close_path) == -EBUSY)
+			stuck++;
+		if (bnxt_hwrm_rx_agg_ring_free(bp, rxr, close_path) == -EBUSY)
+			stuck++;
 	}
 
 	/* The completion rings are about to be freed.  After that the
@@ -7818,6 +7828,26 @@ static void bnxt_hwrm_ring_free(struct bnxt *bp, bool close_path)
 			bp->grp_info[i].cp_fw_ring_id = INVALID_HW_RING_ID;
 		}
 	}
+
+	if (!stuck)
+		return 0;
+
+	netdev_err(bp->dev, "Firmware did not free %d ring(s)\n", stuck);
+	return -EIO;
+}
+
+static void bnxt_stop_dma(struct bnxt *bp)
+{
+	/* A firmware reset resets the device, and the fatal error path
+	 * already disables it.
+	 */
+	if (test_bit(BNXT_STATE_IN_FW_RESET, &bp->state) ||
+	    test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state))
+		return;
+
+	netdev_err(bp->dev,
+		   "Disabling DMA before releasing ring memory, the driver must be rebound to recover\n");
+	pci_clear_master(bp->pdev);
 }
 
 static int __bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max,
@@ -10852,16 +10882,19 @@ static void bnxt_clear_vnic(struct bnxt *bp)
 		bnxt_hwrm_vnic_ctx_free(bp);
 }
 
-static void bnxt_hwrm_resource_free(struct bnxt *bp, bool close_path,
-				    bool irq_re_init)
+static int bnxt_hwrm_resource_free(struct bnxt *bp, bool close_path,
+				   bool irq_re_init)
 {
+	int rc;
+
 	bnxt_clear_vnic(bp);
-	bnxt_hwrm_ring_free(bp, close_path);
+	rc = bnxt_hwrm_ring_free(bp, close_path);
 	bnxt_hwrm_ring_grp_free(bp);
 	if (irq_re_init) {
 		bnxt_hwrm_stat_ctx_free(bp);
 		bnxt_hwrm_free_tunnel_ports(bp);
 	}
+	return rc;
 }
 
 static int bnxt_hwrm_set_br_mode(struct bnxt *bp, u16 br_mode)
@@ -11383,15 +11416,15 @@ static int bnxt_init_chip(struct bnxt *bp, bool irq_re_init)
 	return 0;
 
 err_out:
-	bnxt_hwrm_resource_free(bp, 0, true);
+	if (bnxt_hwrm_resource_free(bp, 0, true))
+		bnxt_stop_dma(bp);
 
 	return rc;
 }
 
 static int bnxt_shutdown_nic(struct bnxt *bp, bool irq_re_init)
 {
-	bnxt_hwrm_resource_free(bp, 1, irq_re_init);
-	return 0;
+	return bnxt_hwrm_resource_free(bp, 1, irq_re_init);
 }
 
 static int bnxt_init_nic(struct bnxt *bp, bool irq_re_init)
@@ -13462,7 +13495,8 @@ int bnxt_half_open_nic(struct bnxt *bp)
  */
 void bnxt_half_close_nic(struct bnxt *bp)
 {
-	bnxt_hwrm_resource_free(bp, false, true);
+	if (bnxt_hwrm_resource_free(bp, false, true))
+		bnxt_stop_dma(bp);
 	bnxt_del_napi(bp);
 	bnxt_free_skbs(bp);
 	bnxt_free_mem(bp, true);
@@ -13541,7 +13575,8 @@ static void __bnxt_close_nic(struct bnxt *bp, bool irq_re_init,
 	if (BNXT_SUPPORTS_MULTI_RSS_CTX(bp))
 		bnxt_clear_rss_ctxs(bp);
 	/* Flush rings and disable interrupts */
-	bnxt_shutdown_nic(bp, irq_re_init);
+	if (bnxt_shutdown_nic(bp, irq_re_init))
+		bnxt_stop_dma(bp);
 
 	/* TODO CHIMP_FW: Link/PHY related cleanup if (link_re_init) */
 
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH net v6 4/5] bnxt_en: refuse to open a device with stopped DMA
  2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
                   ` (2 preceding siblings ...)
  2026-11-20 15:28 ` [PATCH net v6 3/5] bnxt_en: stop DMA before releasing rings the firmware did not free Joe Damato
@ 2026-11-20 15:28 ` Joe Damato
  2026-11-20 15:28 ` [PATCH net v6 5/5] bnxt_en: stop the ULP around the reset task Joe Damato
  4 siblings, 0 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev, Michael Chan, Pavan Chebbi, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Prashant Sreedharan
  Cc: edumazet, horms, selvin.xavier, kalesh-anakkur.purayil,
	linux-kernel, Joe Damato

Add BNXT_STATE_DMA_STOPPED and set it in bnxt_stop_dma() to signal that
the device had DMA disabled. When opening the device later, check this
bit and exit with an error. Also add the bit to BNXT_NO_FW_ACCESS() so
HWRM commands sent after DMA is stopped fail right away instead of
waiting for the timeout.

This is a very rare case, but when it happens the driver must be rebound
to make the device usable again.

Fixes: 74608fc98d28 ("bnxt_en: Ring free response from close path should use completion ring")
Signed-off-by: Joe Damato <joe@dama.to>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 26 ++++++++++++++++++++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h |  2 ++
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 600a8c866f3c..9daf053293a4 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -7847,9 +7847,20 @@ static void bnxt_stop_dma(struct bnxt *bp)
 
 	netdev_err(bp->dev,
 		   "Disabling DMA before releasing ring memory, the driver must be rebound to recover\n");
+	set_bit(BNXT_STATE_DMA_STOPPED, &bp->state);
 	pci_clear_master(bp->pdev);
 }
 
+static int bnxt_check_dma_stopped(struct bnxt *bp)
+{
+	if (!test_bit(BNXT_STATE_DMA_STOPPED, &bp->state))
+		return 0;
+
+	netdev_err(bp->dev,
+		   "DMA was disabled after the firmware failed to free rings, rebind the driver to recover\n");
+	return -ENODEV;
+}
+
 static int __bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max,
 			     bool shared);
 static int bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max,
@@ -13440,9 +13451,10 @@ static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
 
 int bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
 {
-	int rc = 0;
+	int rc;
 
-	if (test_bit(BNXT_STATE_ABORT_ERR, &bp->state))
+	rc = bnxt_check_dma_stopped(bp);
+	if (!rc && test_bit(BNXT_STATE_ABORT_ERR, &bp->state))
 		rc = -EIO;
 	if (!rc)
 		rc = __bnxt_open_nic(bp, irq_re_init, link_re_init);
@@ -13459,7 +13471,11 @@ int bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
  */
 int bnxt_half_open_nic(struct bnxt *bp)
 {
-	int rc = 0;
+	int rc;
+
+	rc = bnxt_check_dma_stopped(bp);
+	if (rc)
+		goto half_open_err;
 
 	if (test_bit(BNXT_STATE_ABORT_ERR, &bp->state)) {
 		netdev_err(bp->dev, "A previous firmware reset has not completed, aborting half open\n");
@@ -13519,6 +13535,10 @@ static int bnxt_open(struct net_device *dev)
 	struct bnxt *bp = netdev_priv(dev);
 	int rc;
 
+	rc = bnxt_check_dma_stopped(bp);
+	if (rc)
+		return rc;
+
 	if (test_bit(BNXT_STATE_ABORT_ERR, &bp->state)) {
 		rc = bnxt_reinit_after_abort(bp);
 		if (rc) {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index 41d05efd6307..50c024dd3c6c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -2474,9 +2474,11 @@ struct bnxt {
 #define BNXT_STATE_FW_NON_FATAL_COND	13
 #define BNXT_STATE_FW_ACTIVATE_RESET	14
 #define BNXT_STATE_HALF_OPEN		15	/* For offline ethtool tests */
+#define BNXT_STATE_DMA_STOPPED		16
 
 #define BNXT_NO_FW_ACCESS(bp)					\
 	(test_bit(BNXT_STATE_FW_FATAL_COND, &(bp)->state) ||	\
+	 test_bit(BNXT_STATE_DMA_STOPPED, &(bp)->state) ||	\
 	 pci_channel_offline((bp)->pdev))
 
 	struct bnxt_irq	*irq_tbl;
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH net v6 5/5] bnxt_en: stop the ULP around the reset task
  2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
                   ` (3 preceding siblings ...)
  2026-11-20 15:28 ` [PATCH net v6 4/5] bnxt_en: refuse to open a device with stopped DMA Joe Damato
@ 2026-11-20 15:28 ` Joe Damato
  4 siblings, 0 replies; 6+ messages in thread
From: Joe Damato @ 2026-11-20 15:28 UTC (permalink / raw)
  To: netdev, Michael Chan, Pavan Chebbi, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Prashant Sreedharan
  Cc: edumazet, horms, selvin.xavier, kalesh-anakkur.purayil,
	linux-kernel, Joe Damato

The reset task can now stop DMA when the firmware does not free the
rings, but the RDMA driver is not told and keeps using the device.

Stop the ULP before bnxt_reset() takes its locks, as bnxt_fw_reset()
does, and start it again afterwards unless DMA was stopped. The silent
reset for a VF config change is left alone.

Fixes: 74608fc98d28 ("bnxt_en: Ring free response from close path should use completion ring")
Signed-off-by: Joe Damato <joe@dama.to>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 9daf053293a4..2abb721d87de 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -14710,10 +14710,14 @@ static void bnxt_tph_update(struct bnxt *bp)
 /* Only called from bnxt_sp_task() */
 static void bnxt_reset(struct bnxt *bp, bool silent)
 {
+	if (!silent)
+		bnxt_ulp_stop(bp);
 	bnxt_rtnl_lock_sp(bp);
 	if (test_bit(BNXT_STATE_OPEN, &bp->state))
 		bnxt_reset_task(bp, silent);
 	bnxt_rtnl_unlock_sp(bp);
+	if (!silent && !test_bit(BNXT_STATE_DMA_STOPPED, &bp->state))
+		bnxt_ulp_start(bp);
 }
 
 /* Only called from bnxt_sp_task() */
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-20 15:28 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 15:28 [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust Joe Damato
2026-11-20 15:28 ` [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers Joe Damato
2026-11-20 15:28 ` [PATCH net v6 2/5] bnxt_en: check HWRM response if completion never arrives Joe Damato
2026-11-20 15:28 ` [PATCH net v6 3/5] bnxt_en: stop DMA before releasing rings the firmware did not free Joe Damato
2026-11-20 15:28 ` [PATCH net v6 4/5] bnxt_en: refuse to open a device with stopped DMA Joe Damato
2026-11-20 15:28 ` [PATCH net v6 5/5] bnxt_en: stop the ULP around the reset task Joe Damato

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®