mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic
@ 2026-11-20 20:24 Chris Babroski
  2026-11-20 20:24 ` [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX Chris Babroski
  2026-11-20 20:24 ` [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic Chris Babroski
  0 siblings, 2 replies; 3+ messages in thread
From: Chris Babroski @ 2026-11-20 20:24 UTC (permalink / raw)
  To: netdev, linux-kernel
  Cc: andrew+netdev, davem, edumazet, kuba, pabeni, davthompson, cbabroski

Background
----------
On BlueField 2 and BlueField 3 the MAC HW is designed to repeatedly send
pause frames to its link partner when the RX FIFO is full. The internal
Ethernet switch on BlueField that is connected to this port is also
designed to stop traffic on all ports when it receives a high number of
continuous pause frames from one of the ports.

What this means is that when the MAC RX FIFO cannot be drained (e.g. due
to panic), it is possible to reproduce a scenario under steady and high
network load where the network becomes inaccessible for other devices
connected to the 3-port switch, namely the OOB host and DPU-BMC.

What this patch series does
---------------------------
Register a panic notifier at highest priority that disables MAC RX and
clears the RX FIFO to ensure that the MAC can no longer receive packets
or send pause frames after a kernel panic.

Add protection in the mlxbf_gige driver for the case when firmware
disables the MAC after handling a fatal error or panic that the kernel
could not detect.

Testing
-------
The following testing was done:
* Verified no performance difference when testing with iperf.
* No regressions found when copying large files to and from DUT.
* Repeated module load/unload succeeds.
* Bringing interface down and up repeatedly succeeds.
* Flooded the interface with broadcast packets to keep MAC RX FIFO full
  while monitoring pause frames received on the internal 3 port switch.
  With this test setup verified pause frames stopped after:
  * ATF FW panic
  * kernel panic triggered with sysrq
  * power off
  * reboot
  * module unload
* No unexpected kernel Oops or warnings when implementing custom ATF FW
  to disable OOB MAC while mlxbf_gige is actively draining the RX queue.

Chris Babroski (2):
  mlxbf_gige: protect against MAC disable during TX
  mlxbf_gige: disable MAC after kernel panic

 .../ethernet/mellanox/mlxbf_gige/mlxbf_gige.h |  2 +
 .../mellanox/mlxbf_gige/mlxbf_gige_main.c     | 39 +++++++++++++++++++
 .../mellanox/mlxbf_gige/mlxbf_gige_tx.c       | 29 +++++++++-----
 3 files changed, 60 insertions(+), 10 deletions(-)


base-commit: 2208ab00f4f213ade91ab775e7f8c4dacdcc856e
-- 
2.34.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX
  2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
@ 2026-11-20 20:24 ` Chris Babroski
  2026-11-20 20:24 ` [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic Chris Babroski
  1 sibling, 0 replies; 3+ messages in thread
From: Chris Babroski @ 2026-11-20 20:24 UTC (permalink / raw)
  To: netdev, linux-kernel
  Cc: andrew+netdev, davem, edumazet, kuba, pabeni, davthompson, cbabroski

On BlueField, firmware can disable and reset the OOB MAC when it handles
a fatal error or panic. If Linux is still actively transmitting packets
while the MAC is disabled, then it's possible for the HW consumer index
to be reset while the SW completion index in the driver remains the
same. This can cause the two indices to suddenly become far apart, with
a difference that is potentially larger than the TX queue length itself.

Previously, mlxbf_gige_handle_tx_complete() would continue to loop
around the TX queue, counting packets and freeing their memory, until
the two indices were equal. If entries were already freed the first time
through the queue, then a second attempt could cause a NULL pointer
dereference.

Bound the loop over the length of the TX queue so entries are not
handled more than once, and check that the skb at tx_wqe_index is not
NULL before handling.

After the loop, set the SW completion index to the new HW consumer index
to prevent future polls from walking the queue again. In normal
operation the loop already exits with the two indices equal, so there is
no change in behavior.

TX is not recovered after a sudden MAC disable. If firmware has disabled
the MAC then the system is in a state where it will be rebooting
automatically or needs manual intervention to recover.

Signed-off-by: Chris Babroski <cbabroski@nvidia.com>
---
 .../mellanox/mlxbf_gige/mlxbf_gige_tx.c       | 29 ++++++++++++-------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
index 04982e888c63..73c3033591d5 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
@@ -130,6 +130,7 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
 	u64 *tx_wqe_addr;
 	u64 tx_status;
 	u16 tx_ci;
+	u16 entry;
 
 	tx_status = readq(priv->base + MLXBF_GIGE_TX_STATUS);
 	if (tx_status & MLXBF_GIGE_TX_STATUS_DATA_FIFO_FULL)
@@ -143,7 +144,8 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
 	 * to be supported, that is TX consumer index wrapped from 0xFFFF
 	 * to 0 while TX completion index is still < 0xFFFF.
 	 */
-	for (; priv->prev_tx_ci != tx_ci; priv->prev_tx_ci++) {
+	for (entry = 0; priv->prev_tx_ci != tx_ci && entry < priv->tx_q_entries;
+	     priv->prev_tx_ci++, entry++) {
 		tx_wqe_index = priv->prev_tx_ci % priv->tx_q_entries;
 		/* Each TX WQE is 16 bytes. The 8 MSB store the 2KB TX
 		 * buffer address and the 8 LSB contain information
@@ -152,18 +154,25 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
 		tx_wqe_addr = priv->tx_wqe_base +
 			       (tx_wqe_index * MLXBF_GIGE_TX_WQE_SZ_QWORDS);
 
-		stats->tx_packets++;
-		stats->tx_bytes += MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr);
-
-		dma_unmap_single(priv->dev, *tx_wqe_addr,
-				 priv->tx_skb[tx_wqe_index]->len, DMA_TO_DEVICE);
-		dev_consume_skb_any(priv->tx_skb[tx_wqe_index]);
-		priv->tx_skb[tx_wqe_index] = NULL;
+		if (priv->tx_skb[tx_wqe_index]) {
+			stats->tx_packets++;
+			stats->tx_bytes += MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr);
+			dma_unmap_single(priv->dev, *tx_wqe_addr,
+					 priv->tx_skb[tx_wqe_index]->len, DMA_TO_DEVICE);
+			dev_consume_skb_any(priv->tx_skb[tx_wqe_index]);
+			priv->tx_skb[tx_wqe_index] = NULL;
 
-		/* Ensure completion of updates across all cores */
-		mb();
+			/* Ensure completion of updates across all cores */
+			mb();
+		}
 	}
 
+	/* At this point the completion index should be equal to the
+	 * consumer index. If they aren't then the driver is out of sync
+	 * from the HW's state and needs to be adjusted.
+	 */
+	priv->prev_tx_ci = tx_ci;
+
 	/* Since the TX ring was likely just drained, check if TX queue
 	 * had previously been stopped and now that there are TX buffers
 	 * available the TX queue can be awakened.
-- 
2.34.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic
  2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
  2026-11-20 20:24 ` [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX Chris Babroski
@ 2026-11-20 20:24 ` Chris Babroski
  1 sibling, 0 replies; 3+ messages in thread
From: Chris Babroski @ 2026-11-20 20:24 UTC (permalink / raw)
  To: netdev, linux-kernel
  Cc: andrew+netdev, davem, edumazet, kuba, pabeni, davthompson, cbabroski

After a kernel panic the MAC continues to receive packets into the RX
FIFO even though the driver can no longer process them. By default, the
MAC HW is designed to repeatedly send pause frames to its link partner
when the RX FIFO is full. The internal Ethernet switch on BlueField that
is connected to this port is also designed to stop traffic on all ports
when it receives a high number of continuous pause frames from one of
the ports.

What this means is that when the kernel panics on BlueField, it is
possible to reproduce a scenario under steady and high network load
where the network becomes inaccessible for other devices connected to
the 3-port switch, namely the OOB host and DPU-BMC.

Register a panic notifier at highest priority that disables MAC RX and
clears the RX FIFO to ensure that the MAC can no longer receive packets
or send pause frames.

Signed-off-by: Chris Babroski <cbabroski@nvidia.com>
---
 .../ethernet/mellanox/mlxbf_gige/mlxbf_gige.h |  2 +
 .../mellanox/mlxbf_gige/mlxbf_gige_main.c     | 39 +++++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
index e7777700ee18..c63bdfae4344 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
@@ -13,6 +13,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/irqreturn.h>
 #include <linux/netdevice.h>
+#include <linux/notifier.h>
 #include <linux/irq.h>
 #include <linux/phy.h>
 
@@ -130,6 +131,7 @@ struct mlxbf_gige {
 	u8 hw_version;
 	struct mlxbf_gige_mdio_gw *mdio_gw;
 	int prev_speed;
+	struct notifier_block panic_nb;
 };
 
 /* Rx Work Queue Element definitions */
diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
index d1f8a72cae53..76da71108345 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
@@ -12,6 +12,7 @@
 #include <linux/interrupt.h>
 #include <linux/iopoll.h>
 #include <linux/module.h>
+#include <linux/panic_notifier.h>
 #include <linux/phy.h>
 #include <linux/platform_device.h>
 #include <linux/rtnetlink.h>
@@ -127,6 +128,27 @@ static int mlxbf_gige_clean_port(struct mlxbf_gige *priv)
 	return err;
 }
 
+static void mlxbf_gige_disable_port(struct mlxbf_gige *priv)
+{
+	u64 control;
+
+	/* Disable RX DMA */
+	control = readq(priv->base + MLXBF_GIGE_RX_DMA);
+	control &= ~MLXBF_GIGE_RX_DMA_EN;
+	writeq(control, priv->base + MLXBF_GIGE_RX_DMA);
+
+	/* Disable port */
+	control = readq(priv->base + MLXBF_GIGE_CONTROL);
+	control &= ~MLXBF_GIGE_CONTROL_PORT_EN;
+	writeq(control, priv->base + MLXBF_GIGE_CONTROL);
+
+	/* Disable interrupts */
+	writeq(0, priv->base + MLXBF_GIGE_INT_EN);
+
+	/* Ensure completion of disable writes before cleaning the port */
+	mb();
+}
+
 static int mlxbf_gige_open(struct net_device *netdev)
 {
 	struct mlxbf_gige *priv = netdev_priv(netdev);
@@ -382,6 +404,17 @@ static struct mlxbf_gige_link_cfg mlxbf_gige_link_cfgs[] = {
 	}
 };
 
+static int mlxbf_gige_on_panic(struct notifier_block *nb,
+			       unsigned long action, void *data)
+{
+	struct mlxbf_gige *priv;
+
+	priv = container_of(nb, struct mlxbf_gige, panic_nb);
+	mlxbf_gige_disable_port(priv);
+	mlxbf_gige_clean_port(priv);
+	return NOTIFY_DONE;
+}
+
 static int mlxbf_gige_probe(struct platform_device *pdev)
 {
 	struct phy_device *phydev;
@@ -446,6 +479,10 @@ static int mlxbf_gige_probe(struct platform_device *pdev)
 	/* Write initial MAC address to hardware */
 	mlxbf_gige_initial_mac(priv);
 
+	priv->panic_nb.priority = INT_MAX;
+	priv->panic_nb.notifier_call = mlxbf_gige_on_panic;
+	atomic_notifier_chain_register(&panic_notifier_list, &priv->panic_nb);
+
 	err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
 	if (err) {
 		dev_err(&pdev->dev, "DMA configuration failed: %pe\n", ERR_PTR(err));
@@ -497,6 +534,7 @@ static int mlxbf_gige_probe(struct platform_device *pdev)
 	return 0;
 
 out:
+	atomic_notifier_chain_unregister(&panic_notifier_list, &priv->panic_nb);
 	mlxbf_gige_mdio_remove(priv);
 	return err;
 }
@@ -507,6 +545,7 @@ static void mlxbf_gige_remove(struct platform_device *pdev)
 
 	unregister_netdev(priv->netdev);
 	phy_disconnect(priv->netdev->phydev);
+	atomic_notifier_chain_unregister(&panic_notifier_list, &priv->panic_nb);
 	mlxbf_gige_mdio_remove(priv);
 	platform_set_drvdata(pdev, NULL);
 }
-- 
2.34.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-20 20:24 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic Chris Babroski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®