Re: [PATCH net-next v9 08/11] net: dsa: realtek: rtl8365mb: re-latch the family D SerDes

[Johan Alvarado <contact@c127.dev>]

Hi Mieczyslaw, Alvin, Oleg,

Thanks for holding off. I could reproduce the symptom here and I'm now
fairly sure the cause is on the SoC side, not in rtl8365mb.

Setup: Mercusys MR80X v2 (IPQ5018 + RTL8367S, family C), CPU port 6 at
2500base-x, booting from NAND with a real config in a reboot loop. A
boot counts as broken when CPU port receives nothing, or frames with FCS
errors, while both ends report 2.5Gbps/Full. Both flavours happen.

Results, same rtl8365mb code in all of them:
- OpenWrt 6.18 stack (stmmac dwmac-ipq5018 + the qca-uniphy PCS):
  18 of 51 boots broken.
- Older stack (qca-nss-dp + qca-ssdk, 6.12): 0 of 16.
- 6.18 stack, switch SerDes brought up only once the conduit is up
  (the OpenWrt ordering patch): 0 of 16.
- 6.18 stack, UNIPHY mode sequence moved to the GMAC probe, nothing
  changed in the switch driver: 0 of 32.

The bootloader hands the UNIPHY over in SGMII (1.25G line rate), and the
qca-uniphy PCS only programs it from pcs_config(), which on this board
runs at the first ndo_open of the conduit. The switch
brings its SerDes up in HSGMII at DSA setup, about two seconds earlier,
so its receiver trains against a transmitter at the wrong line rate.
I read the UNIPHY mode registers at probe and at each pcs_config() on
every boot: SGMII until the first open, SGMII+ after it, identical on
good and broken boots. About a third of the boots end up with a
receiver that does not recover when the far end later switches to
SGMII+.

Some checks that point at the switch receiver, not at the MAC/PCS:
- Broken boot: A full UNIPHY re-sequence on the SoC side does not cure
  it. Unbinding and binding the switch driver cures it every time.
- Switch SerDes brought up against an already programmed UNIPHY: 21 of
  21 good, even with a UNIPHY PLL reset one second later. The PLL
  reset on a settled link is harmless (12 of 12), so I don't think the
  10 s health check in patch 8 covers a real case.

So the "far end not up yet" in the commit message is, at least on this
SoC, "far end up at the wrong line rate". Once the far end is in the
right mode before the switch comes up, the receiver latches on the first
try and nothing needs to be re-latched. The two second gap in the
reported timeline (5.27 switch pcs_config, 7.72 eth0 configuring) is the
same window I see.

This is a family C board and I can't say whether the family D receiver
behaves the same. But the reported log is the same SoC with the same
OpenWrt stack, so the wrong-mode far end is present in that setup too.
Oleg, could you check this on the AX55 before the first ifup of the
conduit? A single register read tells:

  devmem 0x98218    # UNIPHY MISC2: 0x70 = SGMII, 0x50 = SGMII+

If it reads 0x70 there, patch 8 is working around the same thing.

The fix on the SoC side is one commit in the OpenWrt PCS driver, about
50 lines: run the mode sequence at GMAC probe, before the switch
probes. Rebased on current OpenWrt main, so it should apply to the
AX55 tree as is:

  https://github.com/c127dev/openwrt/commit/50e49862d2ba1abd0c93b64403c776af200323fb

Oleg, if the AX55 can take it, a reboot loop with that commit and
without patch 8 would settle this. I'll open the OpenWrt PR once that
is confirmed.

Neither dwmac-ipq5018 nor pcs-qca-uniphy is in mainline, so none of
this can be reproduced on a mainline kernel. That's why I'd rather not
have the workqueue, the extra locks and the reboot notifier in
rtl8365mb for it.

Best regards,
Johan Alvarado