dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

[Giuseppe Nespolino]

Hi Yao,

On a Slimbook EVO15-A8 (YT6801, 1f0a:6801, kernel 7.0.0-28 Ubuntu)
enp2s0 stops receiving after a long s2idle. 18-25 minutes reproduces
it; 5s and 30s cycles never did.
Link comes up at 1Gbps and the resume path logs nothing unusual.

Measured while broken:

- MAC is receiving: mmc_rx_framecount_gb climbing,
mmc_rx_fifo_overflow 2585. netdev rx_packets and the rx-0 MSI-X count
frozen.
- MSIX_PBA (BAR0+0x1300) = 0, so the device isn't asserting anything.
- CH0_STATUS = 0x4c5: RI/RBU/TI/TBU latched, NIS and AIS clear, with
NIE/AIE set in CH0_INTR_ENA (0xd041).
- All 512 RX descriptors in writeback format with sane lengths
(64-1518), OWN=0. The DMA received them; nothing ever consumed them.

Event bits latch and clear fine (W1C works, MGMT_INT_CTRL0 tracks it
as a mirror), but the summary stage never asserts and no MSI-X is
generated. RBU, the full ring and the FIFO overflow are all downstream
of that.

Ruled out with data: ASPM/remapping (PBA=0, MSI-X table intact, zero
IOMMU faults), wrapper registers lost across D3 (all identical to a
healthy baseline), unarmed ring, and MGMT_INT_CTRL0 needing an ack
(status field is read-only).

What recovers it: toggling SYS_RESET (0x152c bit31) plus
motorcomm_init(), then ip link down/up. down/up alone does not - rings
get rebuilt, IRQs re-requested, packets land in memory, still zero
interrupts. I haven't yet isolated the reset from motorcomm_init().

Which brings me to the question, since you know the chip: does
clearing OOB_WOL_CTRL_DIS actually disarm an OOB WOL engine that
already armed, or does that need the SYS_RESET? Your comment says OOB
WOL blocks DMA interrupt
delivery, which is exactly the symptom, and stmmac_pci_plat_suspend()
enables PCI wake unconditionally even with WoL off - it is off here.

I have a candidate fix (do the full motorcomm_reset() in
motorcomm_resume() instead of only deasserting the MDIO/PHY reset) and
I'm happy to send it, but if OOB WOL is the mechanism the right patch
is probably narrower. Caveat: I validated reset + full open, not reset
+ the stmmac_resume path, so a patched module still needs testing.

Happy to run any experiment on this hardware.

Giuseppe

Re: dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

[Giuseppe Nespolino]

Answering my own question, since it turned out to be testable without a
suspend at all.

On a healthy interface, writing 0 to OOB_WOL_CTRL (BAR0+0x1010) - clearing
the DIS bit, i.e. enabling OOB WOL - kills RX within a second:

  OOB=1 (healthy)                      ping   0% loss, rx-0 irq +39,
rx_packets +49
  OOB=0                                ping 100% loss, rx-0 irq  +0,
rx_packets  +0
  OOB=1 again                          ping 100% loss, rx-0 irq  +0,
rx_packets  +0
  SYS_RESET + init + ip link down/up   ping   0% loss, rx-0 irq +35,
rx_packets +41

So the DIS bit is one-way: writing it back reads 1 but does not disarm an
engine that already armed. Only the SYS_RESET recovers. That's a one-writel
reproducer of the symptom, no suspend needed.

It also explains why nothing driver-side helps after resume:
motorcomm_init() writes OOB_WOL_CTRL_DIS, the register reads 1, and it's a
no-op because the engine is already armed. Only the probe's
motorcomm_reset() tears it down.

Possible trigger: stmmac_pci_plat_suspend() calls
pci_wake_from_d3(pdev, true) unconditionally, even when WoL is off.

Caveat, because the two states are not identical: with OOB WOL enabled by
hand, CH0_STATUS reads 0x0 - fully quiesced, no events latched. In the
natural post-resume failure it was 0x4c5 with RI/RBU latched and packets
still landing in memory (RX descriptor OWN count dropping 512 -> 467). So
this proves OOB WOL blocks interrupt delivery and that the DIS bit is
irreversible, but not yet that OOB WOL is what arms across s2idle. I'll
check whether the engine is armed on the next natural occurrence.

Given this, the fix is probably narrower than what I suggested: either
don't enable PCI wake when WoL is off, or disarm OOB WOL properly on
resume - and if a reset is the only way to disarm it, that at least gives
the reset a documented reason. Tell me which direction you prefer and
I'll send a patch.

Giuseppe

Il giorno mer 12 ago 2026 alle ore 16:50 Giuseppe Nespolino
<g.nespolino@gmail.com> ha scritto:
>
> Hi Yao,
>
> On a Slimbook EVO15-A8 (YT6801, 1f0a:6801, kernel 7.0.0-28 Ubuntu)
> enp2s0 stops receiving after a long s2idle. 18-25 minutes reproduces
> it; 5s and 30s cycles never did.
> Link comes up at 1Gbps and the resume path logs nothing unusual.
>
> Measured while broken:
>
> - MAC is receiving: mmc_rx_framecount_gb climbing,
> mmc_rx_fifo_overflow 2585. netdev rx_packets and the rx-0 MSI-X count
> frozen.
> - MSIX_PBA (BAR0+0x1300) = 0, so the device isn't asserting anything.
> - CH0_STATUS = 0x4c5: RI/RBU/TI/TBU latched, NIS and AIS clear, with
> NIE/AIE set in CH0_INTR_ENA (0xd041).
> - All 512 RX descriptors in writeback format with sane lengths
> (64-1518), OWN=0. The DMA received them; nothing ever consumed them.
>
> Event bits latch and clear fine (W1C works, MGMT_INT_CTRL0 tracks it
> as a mirror), but the summary stage never asserts and no MSI-X is
> generated. RBU, the full ring and the FIFO overflow are all downstream
> of that.
>
> Ruled out with data: ASPM/remapping (PBA=0, MSI-X table intact, zero
> IOMMU faults), wrapper registers lost across D3 (all identical to a
> healthy baseline), unarmed ring, and MGMT_INT_CTRL0 needing an ack
> (status field is read-only).
>
> What recovers it: toggling SYS_RESET (0x152c bit31) plus
> motorcomm_init(), then ip link down/up. down/up alone does not - rings
> get rebuilt, IRQs re-requested, packets land in memory, still zero
> interrupts. I haven't yet isolated the reset from motorcomm_init().
>
> Which brings me to the question, since you know the chip: does
> clearing OOB_WOL_CTRL_DIS actually disarm an OOB WOL engine that
> already armed, or does that need the SYS_RESET? Your comment says OOB
> WOL blocks DMA interrupt
> delivery, which is exactly the symptom, and stmmac_pci_plat_suspend()
> enables PCI wake unconditionally even with WoL off - it is off here.
>
> I have a candidate fix (do the full motorcomm_reset() in
> motorcomm_resume() instead of only deasserting the MDIO/PHY reset) and
> I'm happy to send it, but if OOB WOL is the mechanism the right patch
> is probably narrower. Caveat: I validated reset + full open, not reset
> + the stmmac_resume path, so a patched module still needs testing.
>
> Happy to run any experiment on this hardware.
>
> Giuseppe

Re: dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

[Yao Zi]

On Wed, Aug 12, 2026 at 05:02:49PM +0200, Giuseppe Nespolino wrote:
> Answering my own question, since it turned out to be testable without a
> suspend at all.
> 
> On a healthy interface, writing 0 to OOB_WOL_CTRL (BAR0+0x1010) - clearing
> the DIS bit, i.e. enabling OOB WOL - kills RX within a second:
> 
>   OOB=1 (healthy)                      ping   0% loss, rx-0 irq +39,
> rx_packets +49
>   OOB=0                                ping 100% loss, rx-0 irq  +0,
> rx_packets  +0
>   OOB=1 again                          ping 100% loss, rx-0 irq  +0,
> rx_packets  +0
>   SYS_RESET + init + ip link down/up   ping   0% loss, rx-0 irq +35,
> rx_packets +41
> 
> So the DIS bit is one-way: writing it back reads 1 but does not disarm an
> engine that already armed. Only the SYS_RESET recovers. That's a one-writel
> reproducer of the symptom, no suspend needed.

This is quite unexpected. I found enabling OOB_WOL_CTRL blocks DMA
interrupts because it's the default state after resetting, without
clearing it, the MAC is non-operational.

> It also explains why nothing driver-side helps after resume:
> motorcomm_init() writes OOB_WOL_CTRL_DIS, the register reads 1, and it's a
> no-op because the engine is already armed. Only the probe's
> motorcomm_reset() tears it down.

This makes sense.

> Possible trigger: stmmac_pci_plat_suspend() calls
> pci_wake_from_d3(pdev, true) unconditionally, even when WoL is off.

Have you tested the idea? We always call pci_wake_from_d3(pdev, true) but
from your description, broken RX only happens after s2idle is active for
some time, right?

> Caveat, because the two states are not identical: with OOB WOL enabled by
> hand, CH0_STATUS reads 0x0 - fully quiesced, no events latched. In the
> natural post-resume failure it was 0x4c5 with RI/RBU latched and packets
> still landing in memory (RX descriptor OWN count dropping 512 -> 467). So
> this proves OOB WOL blocks interrupt delivery and that the DIS bit is
> irreversible, but not yet that OOB WOL is what arms across s2idle. I'll
> check whether the engine is armed on the next natural occurrence.

From my own testing, with OOB WOL enabled, both DMA TX and RX interrupts
aren't delivered. So TX behavior when RX is broken might indicate what
has happened.

> Given this, the fix is probably narrower than what I suggested: either
> don't enable PCI wake when WoL is off, or disarm OOB WOL properly on
> resume - and if a reset is the only way to disarm it, that at least gives
> the reset a documented reason. Tell me which direction you prefer and
> I'll send a patch.

I prefer the later, even disabling PCI wake works, we'll probably
re-visit this when adding WoL support. Sadly I took a brief look across
vendor's driver, and think there's no other way to disarm the OOB WOL.
Performing a SYS_RESET might be okay since the stmmac core restores
necessary hardware states.

Anyway, please try figuring out state of the DIS bit when the problem
occurs before sending the patch, which would be a strong reason to
perform a reset in the resume hook.

Best regards,
Yao Zi

> Giuseppe
> 
> Il giorno mer 12 ago 2026 alle ore 16:50 Giuseppe Nespolino
> <g.nespolino@gmail.com> ha scritto:
> >
> > Hi Yao,
> >
> > On a Slimbook EVO15-A8 (YT6801, 1f0a:6801, kernel 7.0.0-28 Ubuntu)
> > enp2s0 stops receiving after a long s2idle. 18-25 minutes reproduces
> > it; 5s and 30s cycles never did.
> > Link comes up at 1Gbps and the resume path logs nothing unusual.
> >
> > Measured while broken:
> >
> > - MAC is receiving: mmc_rx_framecount_gb climbing,
> > mmc_rx_fifo_overflow 2585. netdev rx_packets and the rx-0 MSI-X count
> > frozen.
> > - MSIX_PBA (BAR0+0x1300) = 0, so the device isn't asserting anything.
> > - CH0_STATUS = 0x4c5: RI/RBU/TI/TBU latched, NIS and AIS clear, with
> > NIE/AIE set in CH0_INTR_ENA (0xd041).
> > - All 512 RX descriptors in writeback format with sane lengths
> > (64-1518), OWN=0. The DMA received them; nothing ever consumed them.
> >
> > Event bits latch and clear fine (W1C works, MGMT_INT_CTRL0 tracks it
> > as a mirror), but the summary stage never asserts and no MSI-X is
> > generated. RBU, the full ring and the FIFO overflow are all downstream
> > of that.
> >
> > Ruled out with data: ASPM/remapping (PBA=0, MSI-X table intact, zero
> > IOMMU faults), wrapper registers lost across D3 (all identical to a
> > healthy baseline), unarmed ring, and MGMT_INT_CTRL0 needing an ack
> > (status field is read-only).
> >
> > What recovers it: toggling SYS_RESET (0x152c bit31) plus
> > motorcomm_init(), then ip link down/up. down/up alone does not - rings
> > get rebuilt, IRQs re-requested, packets land in memory, still zero
> > interrupts. I haven't yet isolated the reset from motorcomm_init().
> >
> > Which brings me to the question, since you know the chip: does
> > clearing OOB_WOL_CTRL_DIS actually disarm an OOB WOL engine that
> > already armed, or does that need the SYS_RESET? Your comment says OOB
> > WOL blocks DMA interrupt
> > delivery, which is exactly the symptom, and stmmac_pci_plat_suspend()
> > enables PCI wake unconditionally even with WoL off - it is off here.
> >
> > I have a candidate fix (do the full motorcomm_reset() in
> > motorcomm_resume() instead of only deasserting the MDIO/PHY reset) and
> > I'm happy to send it, but if OOB WOL is the mechanism the right patch
> > is probably narrower. Caveat: I validated reset + full open, not reset
> > + the stmmac_resume path, so a patched module still needs testing.
> >
> > Happy to run any experiment on this hardware.
> >
> > Giuseppe

Re: dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

[Giuseppe Nespolino]

On Wed, Aug 12, 2026 at 09:14:56PM +0000, Yao Zi wrote:
> This is quite unexpected. I found enabling OOB_WOL_CTRL blocks DMA
> interrupts because it's the default state after resetting, without
> clearing it, the MAC is non-operational.

That fits what I see, and it explains the asymmetry: the write only takes
effect in the window right after a reset. On an already-armed engine,
writing the DIS bit back does nothing, which is exactly what I measured.

> Have you tested the idea? We always call pci_wake_from_d3(pdev, true) but
> from your description, broken RX only happens after s2idle is active for
> some time, right?

Right, and it's a fair objection; the wake is armed on every suspend, so
it can't be what distinguishes a 30s cycle from an 18min one. I built and
loaded that change here but haven't been through a long suspend with it
yet, so I have no result either way yet.

> From my own testing, with OOB WOL enabled, both DMA TX and RX interrupts
> aren't delivered. So TX behavior when RX is broken might indicate what
> has happened.

I can't answer that from what I captured: between my two snapshots no
frames were transmitted (mmc_tx_framecount_gb stayed at 98), so the frozen
tx-0 vector count proves nothing. Worth noting that TX packets do leave
the interface while RX is broken, but that is expected even with TX
interrupts dead, since stmmac cleans the ring from the coalescing hrtimer.

I've instrumented for it. Next occurrence I'll force TX traffic and report
the tx-0 vector delta.

> Anyway, please try figuring out state of the DIS bit when the problem
> occurs before sending the patch, which would be a strong reason to
> perform a reset in the resume hook.

Already captured, from the last natural occurrence (s2idle 14:43 -> 15:01,
18 minutes, on AC):

  OOB_WOL_CTRL (BAR0+0x1010) while broken:  0x00000001
  OOB_WOL_CTRL healthy baseline:            0x00000001

Identical, and motorcomm_init() had already run at resume and written that
same value. So there is no software-visible state left to correct: the
register claims DIS is set while interrupts are not being delivered, and
nothing short of the reset recovers it.

I'll send the reset-in-resume patch. One design question bef
you want motorcomm_reset() followed by the eFuse settle delay and
motorcomm_init(), i.e. the probe sequence minus the MAC addr
would you rather keep resume lighter than that?

One disclosure, per Documentation/process/generated-content.rst: this
investigation was done with an AI coding assistant. It drove the
register-level diagnosis and the experiment design; the measurements are
all from this machine and I ran and verified them myself.

Thanks,
Giuseppe

Re: dwmac-motorcomm: RX dies after long s2idle, only wrapper SYS_RESET recovers

[Giuseppe Nespolino]

DISCLAIMER: most of what I write in this message comes from AI-driven
analysis, even though I manually tested the steps.

Follow-up, with the TX measurement I owed you, and a correction to
something I implied earlier.

TX first. The bug reproduced again today; I forced 20 raw Ethernet frames
out (AF_PACKET, EtherType 0x88b5, so no configuration change and no
dependency on a working RX):

  mmc_tx_framecount_gb   +20      frames really went out
  netdev tx_packets      +20
  tx-0 vector             +0
  rx-0 vector             +0

TX interrupts aren't delivered either, matching what you saw with OOB WOL
enabled. Register signature identical to the previous occurrence:
OOB_WOL_CTRL 0x1, MGMT_INT_CTRL0 0x11, MSI-X vector 0 unmasked, and all
512 RX descriptors written back with plausible lengths and OWN clear.

Now the correction: the failure is intermittent, and I had been assuming it
was deterministic for long suspends. With the pci_wake_from_d3() change
loaded I went through three long s2idle cycles and only the third broke:

  13h52m   lease 6s after resume    fine
  27min    lease 7s after resume    fine
  31min    no lease                 broken

The longest one was fine, so duration isn't a monotonic predictor either.
With n=3 I can't distinguish a partial effect of that change from plain
stochastic behaviour, so I'm claiming nothing for it - dropping it as a fix,
and happy to send it separately as a cleanup if you want it.

On the isolation I'd left open: motorcomm_init() on its own does not
recover the controller, not even followed by a full ip link down/up (tx-0
+0, rx-0 +0 throughout). The SYS_RESET is the necessary part.

But the reset alone isn't sufficient either, which answers the design
question I asked you, so you needn't. Right after the SYS_RESET:

  MGMT_INT_CTRL0   0x003f0000     mask field 0x3f, every source masked
  INT_MODERATION   0x00000000

reset + OOB_WOL_CTRL alone left interrupts dead; adding the
MGMT_INT_CTRL0 = 0 unmask and INT_MODERATION recovered it immediately
(tx-0 +16, rx-0 +220, DHCP completed). So the reset has to be followed by
the motorcomm_init() that resume already calls - it can neither replace it
nor be trimmed. What I can't separate is whether the OOB WOL clear or the
unmask is the load-bearing half, since the reset does both.

That makes the patch motorcomm_reset() plus the eFuse settle delay,
inserted before the existing motorcomm_init() call in motorcomm_resume().

Given the intermittency, waiting for natural occurrences is a poor way to
validate it. Instead I'll inject the fault deterministically: arm OOB WOL
by hand with the one writel, then suspend and resume, and check whether the
patched resume hook brings it back. On the current driver it stays dead, so
that gives a clean result per cycle instead of a coin flip. I'll keep
counting natural cycles in the background too.

Giuseppe