mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang
@ 2026-10-15 15:00 Chris Babroski
  2026-10-15 15:00 ` [PATCH net-next v1 1/5] net: lan743x: reset DMA channels that fail to stop Chris Babroski
                   ` (6 more replies)
  0 siblings, 7 replies; 14+ messages in thread
From: Chris Babroski @ 2026-10-15 15:00 UTC (permalink / raw)
  To: bryan.whitehead, UNGLinuxDriver, andrew+netdev, davem, edumazet,
	kuba, pabeni, netdev, linux-kernel, richardcochran, linux
  Cc: maxime.chevallier, Raju.Lakkaraju, cbabroski, davthompson

Background
----------
On a custom HW platform that uses the LAN7430 NIC we have been able to
reproduce a scenario where the SoC stops receiving TX completion
interrupts, appearing as a TX hang. After dumping and decoding the
LAN7430 registers we see that the TX channel reports that it is in
started state with no error, but TX_HEAD stops advancing. TX MSI-X stops
incrementing while RX interrupts continue to increment.

This scenario has most often been reproduced in large server cluster
environments, but it's also been seen locally during repeated SCP and
iperf testing.

The root cause of this issue is still unknown and we are continuing
debug. It was found that bringing the interface down and up recovers the
network. This issue has not been reproduced with the LAN7430 EVB
connected to a standard Linux server.

What this patch series does
---------------------------
This patch series attempts to implement automatic recovery of the
network interface after a TX hang is detected. It does not fix the
underlying root cause of the TX hang.

At a high level, the patches implement the following:
1. ndo_tx_timeout support to reset and recover the interface when the
   netdev TX watchdog triggers.
2. BQL support to decrease the amount of time it takes for the watchdog
   to trigger.
3. Stability improvements and error handling around the open/close paths
   that now get executed more frequently as part of the TX hang recovery
   flow.

Potential issues and concerns
-----------------------------
* One of the patches adds new lan743x_dmac_tx_reset and
  lan743x_dmac_rx_reset helper functions. There is no cleanup handling
  introduced if these return an error, only a new error log. This
  matches pre-existing behavior: if a reset timeout occurs then it is
  possible to DMA freed memory. The datasheet states that these resets
  are performed immediately in HW so I'm not sure if these will ever
  time out, but I'm open to suggestions on how we should handle an error
  in these cases.
* The TX timeout reset flow does a full lan743x_netdev_close followed by
  lan743x_netdev_open instead of individual TX channel reset. If the
  stall is happening inside the NIC DMA engine then the theory is that a
  full reset may be needed to recover. For internal use we also wanted
  to reduce risk by putting the NIC into a clean state after recovery
  with full reset. It would be great to have Microchip/LAN743x experts
  confirm the best reset flow.
* LLM review tools flagged several possible bugs during development
  that resulted in the inclusion of three "Fixes:" tagged patches in this
  series. They are included in this series with the ndo_tx_timeout
  support because lan743x_netdev_open and lan743x_netdev_close are now
  called more frequently as a result of TX hang recovery.

Testing
-------
Testing was performed on a 7.0 based kernel using the same driver code.
Please let me know if you would like me to run any other tests.

Tests:
* Verified the TX timeout watchdog triggers and recovers the network
  interface when TX hang is reproduced. This was done as part of
  repeated SCP loop testing and is also being tested in a server cluster
  environment. dmesg was inspected for "NETDEV WATCHDOG" to verify that
  the watchdog is being triggered. No errors were seen in dmesg after
  reset.
* Performance tests using iperf3 to measure RX and TX throughput show no
  noticeable difference with the patches applied.
* Repeated module load and unload cycles produce no errors in dmesg.
  Network appears to function as expected when module is loaded.
* Repeated ifconfig down and up cycles produce no errors in dmesg.
  Network appears to function as expected when interface is back up.

Chris Babroski (5):
  net: lan743x: reset DMA channels that fail to stop
  net: lan743x: start phylink last and stop it first
  net: lan743x: quiesce TX before freeing the rings
  net: lan743x: reset the device on TX timeout
  net: lan743x: add byte queue limits support

 drivers/net/ethernet/microchip/lan743x_main.c | 177 +++++++++++++++---
 drivers/net/ethernet/microchip/lan743x_main.h |   2 +
 2 files changed, 155 insertions(+), 24 deletions(-)

base-commit: 05b5019b01fb9443b09ae4ed34af1135a80a84c8
-- 
2.34.1


^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH net-next v1 4/5] net: lan743x: reset the device on TX timeout
@ 2026-10-19 16:34 netdev-bot+sashiko
  0 siblings, 0 replies; 14+ messages in thread
From: netdev-bot+sashiko @ 2026-10-19 16:34 UTC (permalink / raw)
  To: cbabroski
  Cc: bryan.whitehead, UNGLinuxDriver, andrew+netdev, davem, edumazet,
	kuba, pabeni, netdev, linux-kernel, richardcochran, linux,
	maxime.chevallier, Raju.Lakkaraju, davthompson

Thank you for your contribution! Sashiko AI review found 5 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 5 (1) · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] The commit message says lan743x_dmac_init() now returns the
  soft-reset poll error so that the work cannot reopen a controller…
- [Medium] Before this patch, lan743x_hardware_init() could never fail.
- [Medium] On boards that use the driver's fixed-link fallback (LAN7431
  or PCI11x1x with no DT node and no PHY on the MDIO bus), the reopen…
- [Medium] lan743x_reset_work() holds rtnl and keeps the device detached
  from netif_device_detach() until netif_device_attach().
- [Medium] lan743x_reset_work() runs the full lan743x_netdev_close() on a
  controller it already treats as wedged.

Pre-existing issues:
- [Medium] After the automatic TX-timeout reset, tx->ts_flags still
  contains TX_TS_FLAG_ONE_STEP_SYNC.

--

net: lan743x: reset the device on TX timeout

This adds an ndo_tx_timeout handler. The handler schedules a work item
that, under rtnl, detaches the netdev, closes it, soft-resets the DMA
controller, reopens it and reattaches it. It also makes
lan743x_dmac_init() return the soft-reset poll error, and routes
ndo_stop through a wrapper that skips the hardware close while the device
is detached.

> diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c
> index 8d256a80dac64..2545a846478d6 100644
> --- a/drivers/net/ethernet/microchip/lan743x_main.c
> +++ b/drivers/net/ethernet/microchip/lan743x_main.c
> @@ -1516,10 +1516,14 @@ static void lan743x_rfe_set_multicast(struct lan743x_adapter *adapter)
>  static int lan743x_dmac_init(struct lan743x_adapter *adapter)
>  {
>  	u32 data = 0;
> +	int ret;
>  
>  	lan743x_csr_write(adapter, DMAC_CMD, DMAC_CMD_SWR_);
> -	lan743x_csr_wait_for_bit(adapter, DMAC_CMD, DMAC_CMD_SWR_,
> -				 0, 1000, 20000, 100);
> +	ret = lan743x_csr_wait_for_bit(adapter, DMAC_CMD, DMAC_CMD_SWR_,
> +				       0, 1000, 20000, 100);
> +	if (ret)
> +		return ret;

[Severity: Medium]
The commit message only talks about probe here. However,
lan743x_hardware_init() is also called from lan743x_pm_resume().

Before this change lan743x_hardware_init() could not fail in practice,
so the resume error branch was dead code. Does this new return make that
branch reachable?

lan743x_pm_resume() {
    ...
	ret = lan743x_hardware_init(adapter, pdev);
	if (ret) {
		netif_err(adapter, probe, adapter->netdev,
			  "lan743x_hardware_init returned %d\n", ret);
		lan743x_pci_cleanup(adapter);
		return ret;
	}
    ...
}

If that branch runs, the netdev stays registered and detached. A later
unbind then runs:

lan743x_pcidev_remove()
  lan743x_full_cleanup()
    lan743x_hardware_cleanup()   <- INT_EN_CLR write to a disabled device
    lan743x_pci_cleanup()        <- second region release and disable

Would that release the PCI regions twice and trigger the "disabling
already-disabled device" warning?

> +
>  	switch (DEFAULT_DMA_DESCRIPTOR_SPACING) {
>  	case DMA_DESCRIPTOR_SPACING_16:
>  		data = DMAC_CFG_MAX_DSPACE_16_;

[ ... ]

> @@ -3431,6 +3441,61 @@ static int lan743x_netdev_open(struct net_device *netdev)

[ ... ]

> +static void lan743x_reset_work(struct work_struct *work)
> +{
> +	struct lan743x_adapter *adapter =
> +		container_of(work, struct lan743x_adapter, reset_work);
> +	struct net_device *netdev = adapter->netdev;
> +	int ret;
> +
> +	rtnl_lock();
> +
> +	if (!netif_running(netdev) || !netif_device_present(netdev))
> +		goto unlock;
> +
> +	netdev_warn(netdev, "resetting after TX timeout\n");
> +
> +	netif_device_detach(netdev);
> +	lan743x_netdev_close(netdev);

[Severity: Medium]
Here lan743x_netdev_close() runs on a controller that is already assumed
to be wedged. The controller-wide DMAC_CMD_SWR_ reset only happens later,
in lan743x_dmac_init().

In lan743x_tx_close(), suppose the channel does not stop and the
per-channel reset also times out. The error is only logged, and the ring
is freed anyway:

lan743x_tx_close() {
    ...
		ret = lan743x_dmac_tx_reset(adapter, tx->channel_number);
		if (ret)
			netdev_err(adapter->netdev,
				   "TX channel %d reset timed out\n",
				   tx->channel_number);
    ...
	lan743x_tx_release_all_descriptors(tx);
    ...
	lan743x_tx_ring_cleanup(tx);
}

lan743x_rx_close() does the same for the RX ring, the head write-back
buffer and the RX skbs.

If the engine is still running at that point, could it write head indices
or received frames into memory that was already freed and maybe reused?
Should the DMAC soft reset happen before the rings are torn down?

[Severity: Medium]
This is a pre-existing issue, not one introduced by this patch. The same
close/open sequence leaves the TX one-step state inconsistent, and with
this patch the watchdog can trigger that sequence automatically.

lan743x_ptp_open() calls lan743x_ptp_reset() and sets the RX filter back
to HWTSTAMP_FILTER_NONE. However, tx->ts_flags still has
TX_TS_FLAG_ONE_STEP_SYNC, and nothing turns one-step Sync insertion back
on in the hardware.

After the reset, lan743x_ptp_hwtstamp_get() still reports:

	if (tx->ts_flags & TX_TS_FLAG_ONE_STEP_SYNC)
		config->tx_type = HWTSTAMP_TX_ONESTEP_SYNC;

and lan743x_tx_xmit_frame() still sets ignore_sync for Sync frames:

		if (tx->ts_flags & TX_TS_FLAG_ONE_STEP_SYNC)
			ignore_sync = true;

Can Sync frames go out without a hardware-inserted origin timestamp until
the application sets up timestamping again?

Admin down/up and suspend/resume already have the same mismatch. The
commit message also documents it ("one-step sync insertion is disabled,
while the driver keeps its TX timestamping flags"). The PHC is
unregistered and registered again, so applications have to reconfigure.
Reconfiguring applies one-step mode again through
lan743x_ptp_hwtstamp_set().

> +
> +	ret = lan743x_dmac_init(adapter);
> +	if (ret) {
> +		netdev_err(netdev, "DMA controller reset failed: %d\n", ret);
> +		goto close;
> +	}

[Severity: Medium]
According to the commit message, lan743x_dmac_init() now returns the poll
error so the work does not reopen a controller whose reset timed out. It
also says the interface is taken down "so it can be brought back up
later".

lan743x_netdev_open() never calls lan743x_dmac_init(). That function is
only called from lan743x_hardware_init() (probe and resume) and from this
work.

So if the user runs "ip link set up" after this failure, does
lan743x_netdev_open() reopen the same controller that never came out of
reset?

The early return in lan743x_dmac_init() also skips the DMAC_CFG,
DMAC_COAL_CFG and DMAC_OBFF_CFG writes. Nothing on the open path writes
them again.

Should the open path retry the DMAC reset, or should the device be kept
from coming back up in this case?

> +
> +	ret = lan743x_netdev_open(netdev);
> +	if (ret) {
> +		netdev_err(netdev, "failed to reopen after reset: %d\n", ret);
> +		goto close;
> +	}

[Severity: Medium]
Some boards use the fixed-link fallback in lan743x_phylink_connect():
LAN7431 or PCI11x1x with no DT node and no PHY on the MDIO bus. Can this
reopen ever succeed on those boards?

On the first open, phylink_set_fixed_link() sets
pl->cfg_link_an_mode = MLO_AN_FIXED. Neither phylink_stop() nor
phylink_disconnect_phy() resets it. So on the reopen:

phylink_set_fixed_link() {
	if (pl->cfg_link_an_mode != MLO_AN_PHY || !state ||
	    !test_bit(PHYLINK_DISABLE_STOPPED, &pl->phylink_disable_state))
		return -EINVAL;
    ...
}

lan743x_phylink_connect() then logs "Could not set fixed link" and
lan743x_netdev_open() fails. The work jumps to close: and calls
dev_close(). Every later attempt to bring the interface up fails the same
way.

The fixed-link handling is existing code. On these boards, though, the
new recovery turns a stalled TX queue into an interface that cannot come
back up until the driver is probed again.

> +
> +	goto attach;
> +
> +close:
> +	dev_close(netdev);
> +attach:
> +	netif_device_attach(netdev);

[Severity: Medium]
While the device is detached under rtnl here, multicast list writers that
do not take rtnl can still change dev->mc. The IPv6 socket join path is
one example:

ipv6_sock_mc_join()
  igmp6_group_added()
    dev_mc_add()
      __dev_set_rx_mode()
	if (!netif_device_present(dev))
		return;

These changes never reach the hardware. After netif_device_attach(),
nothing reprograms the filter. lan743x_rfe_open() only programs RSS, and
lan743x_rfe_set_multicast() is only called from ndo_set_rx_mode.

The normal open path resyncs because __dev_open() calls dev_set_rx_mode()
after ndo_open. This path has nothing equivalent.

Can the RFE filter get out of sync with dev->mc? Groups joined during the
reset would not be received, and groups left would stay in the filter,
until the next rx_mode change. Would a call to dev_set_rx_mode() after
the reopen fix this?

> +unlock:
> +	rtnl_unlock();
> +}

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1792008404.git.cbabroski%40nvidia.com

^ permalink raw reply	[flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-10-19 19:49 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-15 15:00 [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang Chris Babroski
2026-10-15 15:00 ` [PATCH net-next v1 1/5] net: lan743x: reset DMA channels that fail to stop Chris Babroski
2026-10-19 10:18   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 2/5] net: lan743x: start phylink last and stop it first Chris Babroski
2026-10-19 10:13   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 3/5] net: lan743x: quiesce TX before freeing the rings Chris Babroski
2026-10-19 10:14   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 4/5] net: lan743x: reset the device on TX timeout Chris Babroski
2026-10-19 10:23   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 5/5] net: lan743x: add byte queue limits support Chris Babroski
2026-10-19 10:14   ` Nicolai Buchwitz
2026-10-19  9:47 ` [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang Nicolai Buchwitz
2026-10-19 19:49 ` Woojung.Huh
2026-10-19 16:34 [PATCH net-next v1 4/5] net: lan743x: reset the device on TX timeout netdev-bot+sashiko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®