mt7921e hangs when hardware fails

mt7921e hangs when hardware fails

[Matthew Wilcox]

Something appears to have gone wrong with the mt7921e in my Framework
laptop, but my attempts to recover are hindered by the driver hanging.

lspci:

aa:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
	Subsystem: MEDIATEK Corp. Device e616
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupts: pin B disabled, MSI(X) routed to IRQ 207
	IOMMU group: 19
	Region 0: Memory at 6019200000 (64-bit, prefetchable) [size=1M]
	Region 2: Memory at 98000000 (64-bit, non-prefetchable) [size=32K]


Here's the relevant parts of my dmesg:

[   54.429097] mt7921e 0000:aa:00.0: enabling device (0000 -> 0002)
[   54.434064] mt7921e 0000:aa:00.0: ASIC revision: 79220010
[   54.532459] mt7921e 0000:aa:00.0: HW/SW Version: 0x8a108a10, Build Time: 2026
0724143257a
[   54.906027] mt7921e 0000:aa:00.0: WM Firmware Version: ____000000, Build Time
: 20260724143402
[   56.086928] mt7921e 0000:aa:00.0 wlp170s0: renamed from wlan0
[164937.892036] mt7921e 0000:aa:00.0: driver own failed
[164938.942093] mt7921e 0000:aa:00.0: driver own failed
[164939.995093] mt7921e 0000:aa:00.0: driver own failed
[164941.272075] mt7921e 0000:aa:00.0: Timeout for driver own
[164942.316076] mt7921e 0000:aa:00.0: driver own failed
(...)
[164962.212360] mt7921e 0000:aa:00.0: chip reset failed
[164962.212383] wlp170s0: Driver requested disconnection from AP 84:a0:6e:ee:3f:b7
[164965.388396] mt7921e 0000:aa:00.0: Message 00020001 (seq 4) timeout
[164968.464445] mt7921e 0000:aa:00.0: Message 00020001 (seq 6) timeout

(hundreds of similar lines)

So I tried to rmmod to see if reloading the module would fix anything.
Unfortunately, it hung like this:

$ sudo cat /proc/136298/stack
[<0>] __flush_work+0x315/0x4c0
[<0>] cancel_work_sync+0x5e/0x80
[<0>] mt7921_pci_remove+0xb0/0x190 [mt7921e]
[<0>] pci_device_remove+0x4a/0xb0
[<0>] device_release_driver_internal+0x19e/0x200
[<0>] driver_detach+0x48/0x90
[<0>] bus_remove_driver+0x78/0x110
[<0>] pci_unregister_driver+0x36/0xe0
[<0>] __do_sys_delete_module+0x1de/0x350
[<0>] do_syscall_64+0xe1/0x650
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

I think something's wrong with the hardware at this point because lspci
-vvv reported something funny in config space which has disappeared
after a power cycle.  Unfortunately I didn't record it, nor did I dump
lspci -xxxx.  I'll do that next time this happens.

My call to action here is that this driver should more gracefully cope
with hardware that's not responding properly.  Whatever work is being
waited for in __flush_work needs to timeout (I left it overnight and it
didn't timeout) or be cancellable.

After a reboot, it's working fine.  I'm going to try stressing the laptop
to see if it fails again, but wanted to get this report out first.

[PATCH wireless] wifi: mt76: mt7921e: fix unregister deadlock

[Devin Wittmayer]

Removing the driver can hang forever when the chip stops answering.

Removal disables NAPI and only then waits for the reset work, which a
failing chip keeps queuing. That reset disables NAPI again and never
returns.

Do what mt7925e does: mark the device removed once it is unregistered,
so no new reset is queued, and cancel the reset work before disabling
NAPI.

Fixes: 3d78c46423c6 ("wifi: mt76: mt7921e: stop chip reset worker in unregister hook")
Reported-by: Matthew Wilcox <willy@infradead.org>
Closes: https://lore.kernel.org/all/aujjJPq-Qth7UyMp@casper.infradead.org/
Signed-off-by: Devin Wittmayer <lucid_duck@justthetip.ca>
---
Reproduced on an MT7922 by putting the chip in D3hot under the driver:
the same driver own and chip reset failures, then rmmod waiting in the
same stack while the reset work is stuck disabling NAPI.

  dead chip, rmmod                  result
  6.18.52, in-tree                  hangs
  mt76 03dd150dbd36                 returns
    with this patch                 returns
    75673ab3c0e4 reverted           hangs
    reverted, with this patch       returns, 3 of 3

Since 75673ab3c0e4 the reset stops early when driver own fails, so
current mt76 gets through a fully dead chip. That commit is not in
v7.3, and any reset that gets as far as NAPI still hangs.

This needs the removed check in the reset helper from 79941e85f4b3. A
normal rmmod and a debugfs chip reset work as before.

Matthew, thanks for the stack trace.

 drivers/net/wireless/mediatek/mt76/mt7921/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c
index f87c44f2a8b6..1ff219005d14 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c
@@ -52,11 +52,12 @@ static void mt7921e_unregister_device(struct mt792x_dev *dev)
 
 	cancel_work_sync(&dev->init_work);
 	mt76_unregister_device(&dev->mt76);
+	set_bit(MT76_REMOVED, &dev->mphy.state);
+	cancel_work_sync(&dev->reset_work);
 	mt76_for_each_q_rx(&dev->mt76, i)
 		napi_disable(&dev->mt76.napi[i]);
 	cancel_delayed_work_sync(&pm->ps_work);
 	cancel_work_sync(&pm->wake_work);
-	cancel_work_sync(&dev->reset_work);
 
 	mt76_connac2_tx_token_put(&dev->mt76);
 	__mt792x_mcu_drv_pmctrl(dev);
@@ -460,7 +461,6 @@ static void mt7921_pci_remove(struct pci_dev *pdev)
 		device_init_wakeup(dev->mt76.dev, false);
 
 	mt7921e_unregister_device(dev);
-	set_bit(MT76_REMOVED, &mdev->phy.state);
 	devm_free_irq(&pdev->dev, pdev->irq, dev);
 	mt76_free_device(&dev->mt76);
 	pci_free_irq_vectors(pdev);
-- 
2.56.0