mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Joe Damato <joe@dama.to>
To: netdev@vger.kernel.org
Cc: andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com,
	kuba@kernel.org, pabeni@redhat.com, horms@kernel.org,
	michael.chan@broadcom.com, pavan.chebbi@broadcom.com,
	selvin.xavier@broadcom.com, kalesh-anakkur.purayil@broadcom.com,
	linux-kernel@vger.kernel.org, Joe Damato <joe@dama.to>
Subject: [PATCH net v6 0/5] bnxt_en: Make RING FREE more robust
Date: Fri, 20 Nov 2026 07:28:22 -0800	[thread overview]
Message-ID: <20261120152830.699712-1-joe@dama.to> (raw)

Greetings:

On two production systems, I saw the following dmesg pattern:

  NETDEV WATCHDOG: transmit queue 0 timed out 6073 ms
  Resp cmpl intr err msg: 0x51                  x20
  hwrm_ring_free type 1 failed                  x12
  hwrm_ring_free type 2 failed                  x8
  AMD-Vi: IO_PAGE_FAULT  x3

This suggests that, for some currently unknown reason, TX completions stall
and the netdev watchdog fires. The driver asks FW to free the rings, this
times out, but the driver ignores the possible failure and frees ring memory.
Since the FW didn't respond to the ring free command, it is possible that the
FW is still DMAing to the memory which was freed.

To address this, this series makes the driver stop DMA before freeing ring
memory when the firmware did not answer RING_FREE.

Since the RFCv4, I found a third system that hit this repeatedly:

  NETDEV WATCHDOG: CPU: 35: transmit queue 0 timed out 5459 ms
  hwrm_ring_free type 1 failed. rc:fffffff0 err:0    x64
  hwrm_ring_free type 2 failed. rc:fffffff0 err:0    x32
  hwrm_ring_free type 0 failed. rc:fffffffb err:1    x64
  hwrm_ring_alloc type 1 failed. rc:ffffffe4 err:4
  hwrm_ring_free type 1 failed. rc:fffffff0 err:0    x44
  Abandoning msg {0x51 0x3e35} len: 0 due to firmware status: 0x408100
  nic open fail (rc: fffffffb)
  DMAR: [DMA Write NO_PASID] Request device [47:00.0] fault addr
  0xffffa07ef000 [fault reason 0x05] PTE Write access is not set

This case differs from the first two in a few ways, but this series would have
still helped:

 - The ring frees fail on the close path and again on the error path of
   the reopen (bnxt_init_chip()), and the device then writes to memory the
   driver has already unmapped. The series stops DMA on both paths.

 - The interface came back up later, the watchdog fired again, and the
   cycle started over. The host logged 79 DMAR write faults in about 20
   hours. Patch 4 breaks this loop by refusing to reopen after DMA has
   been stopped.

 - The firmware answers each RING_FREE about 5 seconds late, after the
   driver has given up, and finally reports an error in its health
   register. Patch 2 does not treat these late responses as success because
   their sequence ids do not match.

In this most recent case, I suspect the hardware is faulty, but the driver
shouldn't hand memory back to the system while the device could still write to
it.

In all of the above cases, the IOMMU caught the error. If the system had no
IOMMU or the IOMMU was in passthrough mode, data corruption or crash would
have occurred.

In the future, this could possibly be extended to make recovery automatic.

I wonder if Sashiko reads cover letters and then avoids flagging stuff, let's
find out. (Unrelated: I wonder if anyone reads cover letters).

Sashiko: you pointed out a few issues which are real bugs but were not
observed in production and I am leaving out of this series which only
addresses an issue I've seen in production.

In particular, I am leaving out:
  - the per queue paths (real, but not observed in production)
  - completion and nq failures are not counted as stuck (all rings fail in
    prod, have not observed only nq failures)
  - The polling path in __hwrm_send() probably needs a dma_rmb (sashiko
    pointed out this pre-existing issue that is probably unlikely but also
    unobserved)
  - bnxt_resume, bnxt_fw_reset_task, and bnxt_io_slot_reset all call
    pci_set_master without checking the DMA bit

All of the above was outlined in the thread about the v5 [1].

Thanks,
Joe

[1]: https://lore.kernel.org/netdev/avIpEpoCYvLHYl3j@devvm20253.cco0.facebook.com/

v6:
 - Commit messages across all patches were updated and a new patch was added
   to the series, patch 5.
 - Patch 1:
    - Return the errno from hwrm_req_send instead of always -EIO.

 - Patch 2:
    - Respect BNXT_HWRM_CTX_SILENT for the new warning.
    - Read the valid byte with READ_ONCE(), then dma_rmb() before reading
      the rest of the response, including error_code.
    - Only check the response for HWRM_RING_FREE. HWRM_SELFTEST_IRQ and
      HWRM_RING_RESET keep the existing behavior.

 - Patch 3:
    - Count a ring as stuck only when no response arrived, not when the
      firmware replied with an error.
    - Don't stop DMA when BNXT_STATE_IN_FW_RESET or BNXT_STATE_FW_FATAL_COND is
      set.

 - Patch 4:
   - Add BNXT_STATE_DMA_STOPPED to BNXT_NO_FW_ACCESS() so HWRM commands sent
     after DMA is stopped fail right away instead of waiting for the timeout.

 - Patch 5 (new):
   - stop the ULP around bnxt_reset() and leave it stopped if DMA was
     stopped. The silent reset after a VF config change leaves the ULP
     running.

v5: https://lore.kernel.org/netdev/20261106191423.4194081-1-joe@dama.to/
 - Switched from RFC to actual submission
 - Patch 4 updated to use an un-used bit to make backporting easier.

v4: https://lore.kernel.org/netdev/20260925174404.2789072-1-joe@dama.to/
  - No changes to patch 1 or 2
  - Patch 3: reworded the message logged when DMA is stopped, a rebind is
    what recovers the device, not a firmware reset. No functional change.
  - Patch 4 added which adds a new bit (BNXT_STATE_DMA_STOPPED) that is set
    when DMA is disabled. When this bit is set, the device requires user
    intervention to bring back up.

v3: https://lore.kernel.org/netdev/20260923210744.3406861-1-joe@dama.to/
  - No changes to patch 1
  - Patch 2: Don't poll for the valid bit as Michael suggested.
  - Patch 3: bnxt_hwrm_ring_free now returns -EIO instead of stopping the
    device, so the remaining resources can be freed and the remaining commands
    can be sent before stopping the device, as Michael suggested. Note the
    switch to using pci_clear_master in this patch instead of
    pci_disable_device. This was done so that the normal shutdown paths can
    call pci_disable_device without generating a warning.

v2: https://lore.kernel.org/netdev/20260922182405.1290749-1-joe@dama.to/
  - No changes to patch 1
  - Patch 2 from v1 dropped
  - Patch 2 in the v2 now checks the response and logs state before giving up
  - Patch 3 in the v2 disables the device to stop DMA before freeing ring
    memory

RFCv1: https://lore.kernel.org/netdev/20260917233218.1160001-1-joe@dama.to/


Joe Damato (5):
  bnxt_en: return the RING_FREE status to callers
  bnxt_en: check HWRM response if completion never arrives
  bnxt_en: stop DMA before releasing rings the firmware did not free
  bnxt_en: refuse to open a device with stopped DMA
  bnxt_en: stop the ULP around the reset task

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 141 +++++++++++++-----
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |   2 +
 .../net/ethernet/broadcom/bnxt/bnxt_hwrm.c    |  47 +++++-
 3 files changed, 149 insertions(+), 41 deletions(-)


base-commit: 00f445721c7f0e48aaff9813cfe1c065c6eb3b71
-- 
2.53.1-Meta


             reply	other threads:[~2026-11-20 15:28 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 15:28 Joe Damato [this message]
2026-11-20 15:28 ` [PATCH net v6 1/5] bnxt_en: return the RING_FREE status to callers Joe Damato
2026-11-20 15:28 ` [PATCH net v6 2/5] bnxt_en: check HWRM response if completion never arrives Joe Damato
2026-11-20 15:28 ` [PATCH net v6 3/5] bnxt_en: stop DMA before releasing rings the firmware did not free Joe Damato
2026-11-20 15:28 ` [PATCH net v6 4/5] bnxt_en: refuse to open a device with stopped DMA Joe Damato
2026-11-20 15:28 ` [PATCH net v6 5/5] bnxt_en: stop the ULP around the reset task Joe Damato

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120152830.699712-1-joe@dama.to \
    --to=joe@dama.to \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=horms@kernel.org \
    --cc=kalesh-anakkur.purayil@broadcom.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=michael.chan@broadcom.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=pavan.chebbi@broadcom.com \
    --cc=selvin.xavier@broadcom.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®