[PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Andrea della Porta]

From: Lukasz Raczylo <lukasz@raczylo.com>

The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
the TX queue.
While the exact root cause is not yet fully understood, it is likely
related to a hardware issue where a TSTART write to the NCR register
is missed, preventing the transmission from being kicked off.

Implement a timeout callback to handle TX queue stalls, triggering the
existing restart mechanism to recover.

Link: https://lore.kernel.org/all/20260514215459.36109-1-lukasz@raczylo.com/
Fixes: dc110d1b23564 ("net: cadence: macb: Add support for Raspberry Pi RP1 ethernet controller")
Signed-off-by: Lukasz Raczylo <lukasz@raczylo.com>
Co-developed-by: Steffen Jaeckel <sjaeckel@suse.de>
Signed-off-by: Steffen Jaeckel <sjaeckel@suse.de>
Co-developed-by: Andrea della Porta <andrea.porta@suse.com>
Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
---

CHANGES IN v2:

- dropped the rate-limited log message
- avoid incrementing tx_error as this is per packet

---
 drivers/net/ethernet/cadence/macb_main.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
index a12aa21244e83..fd282a1700fb9 100644
--- a/drivers/net/ethernet/cadence/macb_main.c
+++ b/drivers/net/ethernet/cadence/macb_main.c
@@ -4522,6 +4522,13 @@ static int macb_setup_tc(struct net_device *dev, enum tc_setup_type type,
 	}
 }
 
+static void macb_tx_timeout(struct net_device *dev, unsigned int q)
+{
+	struct macb *bp = netdev_priv(dev);
+
+	macb_tx_restart(&bp->queues[q]);
+}
+
 static const struct net_device_ops macb_netdev_ops = {
 	.ndo_open		= macb_open,
 	.ndo_stop		= macb_close,
@@ -4540,6 +4547,7 @@ static const struct net_device_ops macb_netdev_ops = {
 	.ndo_hwtstamp_set	= macb_hwtstamp_set,
 	.ndo_hwtstamp_get	= macb_hwtstamp_get,
 	.ndo_setup_tc		= macb_setup_tc,
+	.ndo_tx_timeout		= macb_tx_timeout,
 };
 
 /* Configure peripheral capabilities according to device tree
-- 
2.35.3

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Nicolai Buchwitz]

On 16.6.2026 15:23, Andrea della Porta wrote:
> From: Lukasz Raczylo <lukasz@raczylo.com>
> 
> The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> the TX queue.
> While the exact root cause is not yet fully understood, it is likely
> related to a hardware issue where a TSTART write to the NCR register
> is missed, preventing the transmission from being kicked off.
> 
> Implement a timeout callback to handle TX queue stalls, triggering the
> existing restart mechanism to recover.
> 
> Link: 
> https://lore.kernel.org/all/20260514215459.36109-1-lukasz@raczylo.com/
> Fixes: dc110d1b23564 ("net: cadence: macb: Add support for Raspberry Pi 
> RP1 ethernet controller")
> Signed-off-by: Lukasz Raczylo <lukasz@raczylo.com>
> Co-developed-by: Steffen Jaeckel <sjaeckel@suse.de>
> Signed-off-by: Steffen Jaeckel <sjaeckel@suse.de>
> Co-developed-by: Andrea della Porta <andrea.porta@suse.com>
> Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
> ---
> 
> CHANGES IN v2:
> 
> - dropped the rate-limited log message
> - avoid incrementing tx_error as this is per packet
> 
> ---
>  drivers/net/ethernet/cadence/macb_main.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/net/ethernet/cadence/macb_main.c 
> b/drivers/net/ethernet/cadence/macb_main.c
> index a12aa21244e83..fd282a1700fb9 100644
> --- a/drivers/net/ethernet/cadence/macb_main.c
> +++ b/drivers/net/ethernet/cadence/macb_main.c
> @@ -4522,6 +4522,13 @@ static int macb_setup_tc(struct net_device *dev, 
> enum tc_setup_type type,
>  	}
>  }
> 
> +static void macb_tx_timeout(struct net_device *dev, unsigned int q)
> +{
> +	struct macb *bp = netdev_priv(dev);
> +
> +	macb_tx_restart(&bp->queues[q]);
> +}
> +
>  static const struct net_device_ops macb_netdev_ops = {
>  	.ndo_open		= macb_open,
>  	.ndo_stop		= macb_close,
> @@ -4540,6 +4547,7 @@ static const struct net_device_ops 
> macb_netdev_ops = {
>  	.ndo_hwtstamp_set	= macb_hwtstamp_set,
>  	.ndo_hwtstamp_get	= macb_hwtstamp_get,
>  	.ndo_setup_tc		= macb_setup_tc,
> +	.ndo_tx_timeout		= macb_tx_timeout,
>  };
> 
>  /* Configure peripheral capabilities according to device tree

Reviewed-by: Nicolai Buchwitz <nb@tipi-net.de>

Thanks,
Nicolai

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Théo Lebrun]

Hello Andrea,

On Tue Jun 16, 2026 at 3:23 PM CEST, Andrea della Porta wrote:
> From: Lukasz Raczylo <lukasz@raczylo.com>
>
> The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> the TX queue.
> While the exact root cause is not yet fully understood, it is likely
> related to a hardware issue where a TSTART write to the NCR register
> is missed, preventing the transmission from being kicked off.
>
> Implement a timeout callback to handle TX queue stalls, triggering the
> existing restart mechanism to recover.
>
> Link: https://lore.kernel.org/all/20260514215459.36109-1-lukasz@raczylo.com/
> Fixes: dc110d1b23564 ("net: cadence: macb: Add support for Raspberry Pi RP1 ethernet controller")
> Signed-off-by: Lukasz Raczylo <lukasz@raczylo.com>
> Co-developed-by: Steffen Jaeckel <sjaeckel@suse.de>
> Signed-off-by: Steffen Jaeckel <sjaeckel@suse.de>
> Co-developed-by: Andrea della Porta <andrea.porta@suse.com>
> Signed-off-by: Andrea della Porta <andrea.porta@suse.com>

Thanks for this V2.

Reviewed-by: Théo Lebrun <theo.lebrun@bootlin.com>

Any news from the Raspberry Pi community about this bug investigation?

Thanks,

--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Andrea della Porta]

Hi Theo,

On 17:07 Tue 16 Jun     , Théo Lebrun wrote:
> Hello Andrea,
> 
> On Tue Jun 16, 2026 at 3:23 PM CEST, Andrea della Porta wrote:
> > From: Lukasz Raczylo <lukasz@raczylo.com>
> >
> > The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> > the TX queue.
> > While the exact root cause is not yet fully understood, it is likely
> > related to a hardware issue where a TSTART write to the NCR register
> > is missed, preventing the transmission from being kicked off.
> >
> > Implement a timeout callback to handle TX queue stalls, triggering the
> > existing restart mechanism to recover.
> >
> > Link: https://lore.kernel.org/all/20260514215459.36109-1-lukasz@raczylo.com/
> > Fixes: dc110d1b23564 ("net: cadence: macb: Add support for Raspberry Pi RP1 ethernet controller")
> > Signed-off-by: Lukasz Raczylo <lukasz@raczylo.com>
> > Co-developed-by: Steffen Jaeckel <sjaeckel@suse.de>
> > Signed-off-by: Steffen Jaeckel <sjaeckel@suse.de>
> > Co-developed-by: Andrea della Porta <andrea.porta@suse.com>
> > Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
> 
> Thanks for this V2.
> 
> Reviewed-by: Théo Lebrun <theo.lebrun@bootlin.com>
> 
> Any news from the Raspberry Pi community about this bug investigation?

Not from my side, unfortunately.

Regards,
Andrea

> 
> Thanks,
> 
> --
> Théo Lebrun, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> 

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Nicolai Buchwitz]

On 19.6.2026 09:17, Andrea della Porta wrote:

> [...]

>> Any news from the Raspberry Pi community about this bug investigation?
> 
> Not from my side, unfortunately.

If I remember it correctly, the downstream kernel carries earlier 
versions of Lukasz patches,
which he also submitted there. If time permits, I will run some tests 
with mainline kernel
on Pi5 + downstream kernel with reverted patches + only the upstream 
patches.

But realistically this won't happen before end of next week.

BR
Nicolai

> [...]

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Tue, 16 Jun 2026 15:23:03 +0200 you wrote:
> From: Lukasz Raczylo <lukasz@raczylo.com>
> 
> The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> the TX queue.
> While the exact root cause is not yet fully understood, it is likely
> related to a hardware issue where a TSTART write to the NCR register
> is missed, preventing the transmission from being kicked off.
> 
> [...]

Here is the summary with links:
  - [v2] net: macb: add TX stall timeout callback to recover from lost TSTART write
    https://git.kernel.org/netdev/net/c/e438ec3e9e95

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write

[Théo Lebrun]

Hello all,

On Tue Jun 16, 2026 at 3:23 PM CEST, Andrea della Porta wrote:
> From: Lukasz Raczylo <lukasz@raczylo.com>
>
> The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> the TX queue.
> While the exact root cause is not yet fully understood, it is likely
> related to a hardware issue where a TSTART write to the NCR register
> is missed, preventing the transmission from being kicked off.
>
> Implement a timeout callback to handle TX queue stalls, triggering the
> existing restart mechanism to recover.

Any news on this topic?

There was a guess that a "flush PCIe posted write after TSTART
doorbell" patch [0] could solve it but I was sceptical [1]. We never
landed that and only took the Tx timeout implementation. If it ever
triggers then the kernel log gets a "NETDEV WATCHDOG: ..." critical
line appended [2].

I got reminded because I came across on this wiki page [3] about the
issue, whose author is in Cc. As an aside, I landed there by testing out
the Marginalia search engine and queried "macb driver" (as one does).

Main discovery of the day: it happens on EyeQ5 & latest net/main
(46bc52d13594) as well. Thanks to Andrea for the quick reproducer.

	# udhcpc -i eth1
	...
	# iperf3 -c $IP -P10 -t3000
	...
	^C
	# dmesg | grep eth1
	[    2.182191] macb 2b00000.ethernet eth1: Cadence GEM rev 0x00070200 at 0x02b00000 irq 35 (00:28:f8:94:24:69)
	[   23.570335] macb 2b00000.ethernet eth1: PHY [2b00000.ethernet-ffffffff:0e] driver [Marvell 88E1510] (irq=POLL)
	[   23.570981] macb 2b00000.ethernet eth1: configuring for phy/rgmii-id link mode
	[   27.783260] macb 2b00000.ethernet eth1: Link is Up - 1Gbps/Full - flow control tx
	[   42.182370] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5088 ms
	[   51.158331] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5004 ms
	[   56.262381] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 10108 ms
	[   61.126317] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 14972 ms
	[   67.206322] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5004 ms

Note: the initial reproducer was `iperf3 -c $IP -P10 -t3000 -w4M`.
I can reproduce without -w4M but -P10 is required even though we reach
line rate with a single stream. This points to a race/mb issue in
macb_start_xmit?

I'm posting to see if anyone has theories. I'll be posting some race
fixes soon but they don't fix it (not surprising as they are unrelated).
I have many ideas but I need more time for testing; it'll be much easier
now with a reproduction setup.

[0]: https://lore.kernel.org/netdev/20260514215459.36109-2-lukasz@raczylo.com/
[1]: https://lore.kernel.org/netdev/DIK002QFFNBY.31C3KUX2SQC6W@bootlin.com/
[2]: https://elixir.bootlin.com/linux/v7.2.5/source/net/sched/sch_generic.c#L563-L569
[3]: https://dtype.org/wiki/Cm5_macb_network_hang

Thanks,

--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com