Re: [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang

[Chris Babroski <cbabroski@nvidia.com>]

Hi Nicolai,

Thanks for taking a look at these patches; will respond to your other
comments soon.

> A TX channel that says started, no error, with TX_HEAD not moving is
> also what you see when the MAC stops draining the TX FIFO. When it
> hangs, is the FCT TX FIFO full?

Interestingly, we are consistently seeing FCT_TX_USED=0x34 when
reproducing the TX hang. We have FCT_TX_FIFO_END=0x27, default of 20kB,
so it is mostly empty when we reproduce. Is there anything special about
the 52 byte usage?

> Is the MAC in LPI, or paused by the link partner? Does it still happen
> with EEE and rx pause turned off?

In all repro cases so far the MAC does not appear to be in LPI and EEE
is disabled (MAC_CR=0x200C and EEE_EN is clear). I also tested with
"ethtool -A <if> autoneg off rx off tx off" and could still reproduce
the tx hang.

We have a PCIe switch connecting the NIC to our SoC and on the switch I
can see that the NIC's read requests drop to ~0 implying that it stops
fetching TX data and descriptors rather than the MAC stopping queue
drain.

> The other place to look is the PCIe link. We had a LAN7430 stall on a
> BCM2712 root complex that was also platform specific. The RC driver
> cleared an undocumented bit in its CLKREQ#/L1 control register. That
> one took the whole MMIO window down, so it is probably a different
> failure, but it took days of uptime to show up too. Do the Device Status
> or AER registers on the hung device show anything (UnsupReq, CmpltTO)?
> Is the link running with ASPM L1 or L1 substates on that platform?

I do think the PCIe link could be related; our current best guess is
that there is some issue between the NIC and PCIe switch and we have
tickets open with both vendors to help debug (along with setting up EVBs
to run comparison tests on).

UnsupReq and CmpltTO appear clean. ASPM and L1 substates are disabled;
the link stays in L0 at 2.5 GT/s x1.

Recently we found that after every reproduced hang we see that the NIC
sent a NAK to the switch, potentially in response to a CplD from switch
while reading TX data and descriptors, but I am not yet able to see the
rejected TLP or NAK reason with my setup.

We also found that disabling scatter gather makes this issue
significantly harder to reproduce. So it seems that a full TX descriptor
ring and the NIC sending a NAK while fetching TX data/descriptors
increases repro rate.

Best,
Chris