[PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[Julius Bairaktaris]

Firmware crash recovery on IPQ8074 resets a waitqueue under a sleeping
WMI sender (1/5), frees radios and ring memory still in use when it
fails (2/5, 3/5), sends on rings it has not rebuilt yet (4/5), and lets
cfg80211 commands crash the restarting firmware again (5/5). Patches 1
and 4 are v2 of the series linked below; 2, 3 and 5 are new.

Tested on IPQ8074 (WLAN.HK.2.9.0.1-02146) with the firmware assert from
debugfs. Without the series, every recovery reported list_del corruption
on the CE waitqueue (6 of 6), an RTS threshold loop during recovery
faulted in ath11k_hal_srng_access_begin() (3 of 3), and after a failed
recovery the next crash faulted in sta_info_hash_lookup() and rmmod in
ath11k_dp_rx_pdev_free() (1 of 1 each). With the series, 12 recoveries
(5 with the RTS loop, 2 with beacon updates, 2 after a failed recovery,
3 idle) showed no fault or list corruption, a failed recovery recovered
on the next crash, and rmmod and reload worked.

v2:
- 1/5: name ath11k_connect_pdev_htc_service() and ath11k_wmi_connect()
  (Baochen). Add Reviewed-by.
- 4/5: fix the recovery ordering instead of testing srng fields in the
  CE layer (Baochen). Drop the ath11k_ce_completed_send_next() hunk.
  Fixes: the first driver version.
- 2/5, 3/5, 5/5: new.

v1: https://lore.kernel.org/r/20260805192129.61113-1-julius@bairaktaris.de

Julius Bairaktaris (5):
  wifi: ath11k: initialise the CE descriptor waitqueue once per device
  wifi: ath11k: keep the radios when a firmware recovery fails
  wifi: ath11k: keep the srng allocations when a firmware recovery fails
  wifi: ath11k: clear CRASH_FLUSH only after recovery rebuilds the rings
  wifi: ath11k: hold back WMI commands while the firmware restarts

 drivers/net/wireless/ath/ath11k/core.c | 21 ++++++++++++---------
 drivers/net/wireless/ath/ath11k/wmi.c  |  5 ++++-
 2 files changed, 16 insertions(+), 10 deletions(-)


base-commit: cd026e68fdbf88b9cc522b76c7a9d92c9a41f709
-- 
2.53.0

[PATCH ath-next v2 1/5] wifi: ath11k: initialise the CE descriptor waitqueue once per device

[Julius Bairaktaris]

Firmware crash recovery reinitialises tx_ce_desc_wq in
ath11k_connect_pdev_htc_service(), reached through ath11k_core_start()
and ath11k_wmi_connect(), while a WMI command on a device without credit
flow can still sleep on it in ath11k_wmi_cmd_send(). finish_wait() then
unlinks the waiter from the reset list head:

  list_del corruption. prev->next should be ffffffc084293648, but was
  ffffff80075109a8. (prev=ffffff80075109a8)
  WARNING: CPU: 2 PID: 1782 at lib/list_debug.c:62
  __list_del_entry_valid_or_report+0x134/0x174
  CPU: 2 UID: 101 PID: 1782 Comm: hostapd
  Call trace:
   __list_del_entry_valid_or_report+0x134/0x174 (P)
   finish_wait+0x68/0xb8
   ath11k_wmi_cmd_send+0x150/0x2f4 [ath11k]
   ath11k_wmi_vdev_set_param_cmd+0x64/0xd8 [ath11k]
   ath11k_mac_op_bss_info_changed+0xe8/0xf6c [ath11k]
   drv_link_info_changed+0x128/0x180 [mac80211]
   ieee80211_link_info_change_notify+0xe0/0x100 [mac80211]
   ieee80211_change_bss+0x1bc/0x250 [mac80211]
   nl80211_set_bss+0x1a0/0x328 [cfg80211]

Initialise the queues once in ath11k_core_alloc().

Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146-QCAHKSWPL_SILICONZ-1

Fixes: f951380a6022 ("ath11k: Disabling credit flow for WMI path")
Reviewed-by: Baochen Qiang <baochen.qiang@oss.qualcomm.com>
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
---
 drivers/net/wireless/ath/ath11k/core.c | 5 +++++
 drivers/net/wireless/ath/ath11k/wmi.c  | 1 -
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c
index 766ae27306f43..8e4dab61a3009 100644
--- a/drivers/net/wireless/ath/ath11k/core.c
+++ b/drivers/net/wireless/ath/ath11k/core.c
@@ -2743,6 +2743,7 @@ struct ath11k_base *ath11k_core_alloc(struct device *dev, size_t priv_size,
 				      enum ath11k_bus bus)
 {
 	struct ath11k_base *ab;
+	int i;
 
 	ab = kzalloc(sizeof(*ab) + priv_size, GFP_KERNEL);
 	if (!ab)
@@ -2769,6 +2770,10 @@ struct ath11k_base *ath11k_core_alloc(struct device *dev, size_t priv_size,
 	INIT_LIST_HEAD(&ab->peers);
 	init_waitqueue_head(&ab->peer_mapping_wq);
 	init_waitqueue_head(&ab->wmi_ab.tx_credits_wq);
+
+	for (i = 0; i < ARRAY_SIZE(ab->wmi_ab.wmi); i++)
+		init_waitqueue_head(&ab->wmi_ab.wmi[i].tx_ce_desc_wq);
+
 	init_waitqueue_head(&ab->qmi.cold_boot_waitq);
 	INIT_WORK(&ab->restart_work, ath11k_core_restart);
 	INIT_WORK(&ab->update_11d_work, ath11k_update_11d);
diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c
index 08fd6795ed9e7..3d5b6605eace8 100644
--- a/drivers/net/wireless/ath/ath11k/wmi.c
+++ b/drivers/net/wireless/ath/ath11k/wmi.c
@@ -9101,7 +9101,6 @@ static int ath11k_connect_pdev_htc_service(struct ath11k_base *ab,
 	ab->wmi_ab.wmi_endpoint_id[pdev_idx] = conn_resp.eid;
 	ab->wmi_ab.wmi[pdev_idx].eid = conn_resp.eid;
 	ab->wmi_ab.max_msg_len[pdev_idx] = conn_resp.max_msg_len;
-	init_waitqueue_head(&ab->wmi_ab.wmi[pdev_idx].tx_ce_desc_wq);
 
 	return 0;
 }
-- 
2.53.0

[PATCH ath-next v2 2/5] wifi: ath11k: keep the radios when a firmware recovery fails

[Julius Bairaktaris]

On a firmware crash recovery ath11k_mac_allocate() keeps the registered
radios, but the error paths of ath11k_core_start() and
ath11k_core_qmi_firmware_ready() still call ath11k_mac_destroy(). A
failed recovery thus frees an ieee80211_hw that mac80211 still uses and
leaves ab->pdevs[i].ar NULL. A frame bridged to the radio then follows
the freed hw:

  Unable to handle kernel paging request at virtual address
  ffffff8316cfd480
  pc : sta_info_hash_lookup+0x4c/0xe4 [mac80211]
  Call trace:
   sta_info_hash_lookup+0x4c/0xe4 [mac80211] (P)
   sta_info_get_bss+0x1c/0x78 [mac80211]
   ieee80211_lookup_ra_sta+0x124/0x14c [mac80211]
   __ieee80211_subif_start_xmit+0xb8/0x3a0 [mac80211]
   ieee80211_subif_start_xmit+0x50/0x750 [mac80211]
   dev_hard_start_xmit+0x98/0x120

and driver removal dereferences the NULL ar:

  Unable to handle kernel access to user memory outside uaccess routines
  at virtual address 0000000000000000
  pc : ath11k_dp_rx_pdev_free+0x34/0x12c [ath11k]
  Call trace:
   ath11k_dp_rx_pdev_free+0x34/0x12c [ath11k] (P)
   ath11k_dp_pdev_free+0x4c/0x84 [ath11k]
   ath11k_core_deinit+0xdc/0x168 [ath11k]
   ath11k_ahb_remove+0x30/0x90 [ath11k_ahb]

Destroy the radios only when ATH11K_FLAG_REGISTERED is clear, as
ath11k_mac_allocate() does.

Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146-QCAHKSWPL_SILICONZ-1

Fixes: 0366f42640a4 ("ath11k: Move mac80211 hw allocation before wmi_init command")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
---
 drivers/net/wireless/ath/ath11k/core.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c
index 8e4dab61a3009..dd256853bab57 100644
--- a/drivers/net/wireless/ath/ath11k/core.c
+++ b/drivers/net/wireless/ath/ath11k/core.c
@@ -2231,7 +2231,8 @@ static int ath11k_core_start(struct ath11k_base *ab)
 err_reo_cleanup:
 	ath11k_dp_pdev_reo_cleanup(ab);
 err_mac_destroy:
-	ath11k_mac_destroy(ab);
+	if (!test_bit(ATH11K_FLAG_REGISTERED, &ab->dev_flags))
+		ath11k_mac_destroy(ab);
 err_hif_stop:
 	ath11k_hif_stop(ab);
 err_wmi_detach:
@@ -2315,7 +2316,8 @@ int ath11k_core_qmi_firmware_ready(struct ath11k_base *ab)
 
 err_core_stop:
 	ath11k_core_stop(ab);
-	ath11k_mac_destroy(ab);
+	if (!test_bit(ATH11K_FLAG_REGISTERED, &ab->dev_flags))
+		ath11k_mac_destroy(ab);
 err_dp_free:
 	ath11k_dp_free(ab);
 	mutex_unlock(&ab->core_lock);
-- 
2.53.0

[PATCH ath-next v2 3/5] wifi: ath11k: keep the srng allocations when a firmware recovery fails

[Julius Bairaktaris]

Since commit 32be3ca4cf78 ("wifi: ath11k: HAL SRNG: don't deinitialize
and re-initialize again") recovery zeroes the hal state instead of
freeing and reallocating it, but a failed recovery still calls
ath11k_hal_srng_deinit(). This frees the ring pointer buffers and the
ring configuration, which only probe allocates, while the rings still
point into them. Driver removal then walks the copy engine rings through
the freed buffers:

  ath11k c000000.wifi: failed to reconfigure driver on crash recovery
  ...
  Unable to handle kernel paging request at virtual address
  ffffffc080ee5090
  pc : ath11k_hal_srng_access_begin+0xc/0x60 [ath11k]
  Call trace:
   ath11k_hal_srng_access_begin+0xc/0x60 [ath11k] (P)
   ath11k_ce_cleanup_pipes+0x16c/0x4cc [ath11k]
   ath11k_ahb_stop+0x2c/0x3c [ath11k_ahb]
   ath11k_core_deinit+0xec/0x168 [ath11k]
   ath11k_ahb_remove+0x30/0x90 [ath11k_ahb]

Return the error and keep the allocations for ath11k_ahb_remove() and
ath11k_pci_remove() to free.

Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146-QCAHKSWPL_SILICONZ-1

Fixes: 32be3ca4cf78 ("wifi: ath11k: HAL SRNG: don't deinitialize and re-initialize again")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
---
 drivers/net/wireless/ath/ath11k/core.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c
index dd256853bab57..daa3751d75233 100644
--- a/drivers/net/wireless/ath/ath11k/core.c
+++ b/drivers/net/wireless/ath/ath11k/core.c
@@ -2352,15 +2352,11 @@ static int ath11k_core_reconfigure_on_crash(struct ath11k_base *ab)
 
 	ret = ath11k_core_qmi_firmware_ready(ab);
 	if (ret)
-		goto err_hal_srng_deinit;
+		return ret;
 
 	clear_bit(ATH11K_FLAG_RECOVERY, &ab->dev_flags);
 
 	return 0;
-
-err_hal_srng_deinit:
-	ath11k_hal_srng_deinit(ab);
-	return ret;
 }
 
 void ath11k_core_halt(struct ath11k *ar)
-- 
2.53.0

[PATCH ath-next v2 4/5] wifi: ath11k: clear CRASH_FLUSH only after recovery rebuilds the rings

[Julius Bairaktaris]

ath11k_core_reconfigure_on_crash() clears ATH11K_FLAG_CRASH_FLUSH right
after ath11k_hal_srng_clear() zeroes the rings, but
ath11k_ce_init_pipes() and ath11k_dp_alloc() rebuild them only later. A
WMI command sent in between passes the CRASH_FLUSH check in
ath11k_ce_send() and dereferences the NULL tail pointer of the cleared
ring:

  Unable to handle kernel read from unreadable memory at virtual address 0
  pc : ath11k_hal_srng_access_begin+0xc/0x60 [ath11k]
  lr : ath11k_ce_send+0x114/0x350 [ath11k]
  Call trace:
   ath11k_hal_srng_access_begin+0xc/0x60 [ath11k] (P)
   ath11k_htc_send+0x188/0x3c8 [ath11k]
   ath11k_wmi_cmd_send+0xc8/0x2f4 [ath11k]
   ath11k_wmi_send_peer_create_cmd+0x78/0xe0 [ath11k]
   ath11k_peer_create+0x138/0x500 [ath11k]

Clear the flag once the rings are rebuilt, before ath11k_core_start()
sends the first HTC message.

Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146-QCAHKSWPL_SILICONZ-1

Fixes: d5c65159f289 ("ath11k: driver for Qualcomm IEEE 802.11ax devices")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
---
 drivers/net/wireless/ath/ath11k/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c
index daa3751d75233..4666245aee36d 100644
--- a/drivers/net/wireless/ath/ath11k/core.c
+++ b/drivers/net/wireless/ath/ath11k/core.c
@@ -2294,6 +2294,8 @@ int ath11k_core_qmi_firmware_ready(struct ath11k_base *ab)
 		goto err_firmware_stop;
 	}
 
+	clear_bit(ATH11K_FLAG_CRASH_FLUSH, &ab->dev_flags);
+
 	if (ath11k_frame_mode == ATH11K_HW_TXRX_RAW)
 		set_bit(ATH11K_FLAG_RAW_MODE, &ab->dev_flags);
 
@@ -2348,8 +2350,6 @@ static int ath11k_core_reconfigure_on_crash(struct ath11k_base *ab)
 
 	ab->free_vdev_map = (1LL << (ab->num_radios * TARGET_NUM_VDEVS(ab))) - 1;
 
-	clear_bit(ATH11K_FLAG_CRASH_FLUSH, &ab->dev_flags);
-
 	ret = ath11k_core_qmi_firmware_ready(ab);
 	if (ret)
 		return ret;
-- 
2.53.0

[PATCH ath-next v2 5/5] wifi: ath11k: hold back WMI commands while the firmware restarts

[Julius Bairaktaris]

ath11k_core_restart() brings the firmware up outside mac80211's restart,
so cfg80211 operations keep sending WMI commands while the HTC services
reconnect and before the firmware reports ready. These commands crash
the restarting firmware again, and recovery then fails on the unified
ready timeout:

  ath11k c000000.wifi: failed to receive wmi unified ready event: -110
  ath11k c000000.wifi: failed to reconfigure driver on crash recovery

With iw setting the RTS threshold in a loop from the moment the firmware
stops, the firmware crashed 15, 15 and 11 times on three triggers
before recovery succeeded. With this patch it crashed once per trigger
(5 of 5).

Return -ESHUTDOWN for commands from outside the restart worker while
ATH11K_FLAG_RECOVERY is set. mac80211 replays the configuration when the
driver restarts the hardware.

Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146-QCAHKSWPL_SILICONZ-1

Fixes: d5c65159f289 ("ath11k: driver for Qualcomm IEEE 802.11ax devices")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
---
 drivers/net/wireless/ath/ath11k/wmi.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c
index 3d5b6605eace8..5c27e7057e6ad 100644
--- a/drivers/net/wireless/ath/ath11k/wmi.c
+++ b/drivers/net/wireless/ath/ath11k/wmi.c
@@ -312,6 +312,10 @@ int ath11k_wmi_cmd_send(struct ath11k_pdev_wmi *wmi, struct sk_buff *skb,
 
 	might_sleep();
 
+	if (test_bit(ATH11K_FLAG_RECOVERY, &ab->dev_flags) &&
+	    current_work() != &ab->restart_work)
+		return -ESHUTDOWN;
+
 	if (ab->hw_params.credit_flow) {
 		wait_event_timeout(wmi_ab->tx_credits_wq, ({
 			ret = ath11k_wmi_cmd_send_nowait(wmi, skb, cmd_id);
-- 
2.53.0

Re: [PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[Florian Maurer]

Hi Julius,

I have been chasing the same recovery problems on IPQ8074 (Xiaomi
AX3600, WLAN.HK.2.9.0.1-02146) in a Freifunk mesh network and ended up
with fixes very close to your 2/5-5/5, so I tested your series instead.

Setup: OpenWrt 25.12, kernel 6.12.112, your series backported to
backports-6.18.55 on top of the three ath-next IRQ fixes (a7ab5c835e42,
f7a74e131d3f, e56c9d429ba0), plus two fixes of mine that I posted on
top of your series [1], together with a Kconfig prerequisite (A: stop
TX before remoteproc resets WCSS; B: don't disable the ext IRQs twice
after a failed recovery). AP and mesh vifs on both radios, real
clients and mesh peers.

With your series + A:
- 10 simulated asserts (debugfs) while survey and station dumps are
  polled and downlink traffic runs: all recovered, no WARN/Oops.
- 10 real firmware asserts under load (5 AP vif, 5 mesh vif),
  triggered by an out-of-tree patch that sends WMI_PEER_FLUSH_TIDS_CMDID
  with all 32 TID bits set: all recovered, no WARN/Oops.
- Failed recovery (forced via a test-only knob) followed by another
  crash: the radios come back, but TX completions stop, TX fails with
  -12 and the firmware asserts again repeatedly, 2/2. That is what B
  fixes.

With your series + A + B:
- Failed recovery followed by another crash: recovered 3/3, which
  your 2/5 and 3/5 make possible in the first place.

One case is still open. It is not a regression of your series: before
3/5 the same failure freed the srng memory and crashed in the same
ath11k_ce_cleanup_pipes() path. If the firmware crashes again while
the restart work is bringing it up, the QMI firmware start can fail:

  remoteproc remoteproc0: remote processor cd00000.q6v5_wcss is now up
  remoteproc remoteproc0: crash detected in cd00000.q6v5_wcss: type fatal error
  ath11k c000000.wifi: fw_version 0x290b84a5 ...
  remoteproc remoteproc0: stopped remote processor cd00000.q6v5_wcss
  ath11k c000000.wifi: failed to send firmware start: -104
  ath11k c000000.wifi: failed to start firmware: -104
  ath11k c000000.wifi: failed to reconfigure driver on crash recovery

That recovery fails after ath11k_hal_srng_clear() but before
ath11k_ce_init_pipes(). The next recovery then tears down the CE pipes
on the cleared rings and hits a NULL pointer with BHs disabled (under
ce_lock):

  Unable to handle kernel read from unreadable memory at virtual address 0000000000000000
  pc : ath11k_hal_srng_access_begin+0xc/0x60 [ath11k]
  Kernel panic - not syncing: Oops: Fatal exception in interrupt

Decoded call chain (6.18 backport, the raw trace has wrong symbols
for the stripped module):

  ath11k_hal_srng_access_begin       hal.c:834
  ath11k_ce_poll_send_completed      ce.c:706
  ath11k_ce_cleanup_pipes            ce.c:884
  ath11k_core_reconfigure_on_crash   core.c:2403
  ath11k_core_restart                core.c:2574

Reproduced 2/2 by triggering a remoteproc crash 10-20 ms after "is now
up" (measured 11 and 21 ms), with your series and my two patches
applied. So 2/5 and 3/5 cover a recovery that fails once the rings are
set up again, but not a failure inside ath11k_core_start_firmware().

I haven't tried a fix yet. Since the CE-layer guard from v1 was
dropped, one option at the core level would be to remember whether the
CE pipes and DP were set up again since the last teardown, and to skip
that teardown in ath11k_core_reconfigure_on_crash() otherwise. I can
try that if you don't plan to cover it in a v3.

For the series:

Tested-by: Florian Maurer <fmaurer@disroot.org> # IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146 (backport to 6.18)

[1] https://lore.kernel.org/linux-wireless/cover.1793531790.git.fmaurer@disroot.org/

Regards
Florian

Re: [PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[Julius Bairaktaris]

Hi Florian,

Thanks for the testing and for your three patches.

> That recovery fails after ath11k_hal_srng_clear() but before
> ath11k_ce_init_pipes(). The next recovery then tears down the CE pipes
> on the cleared rings and hits a NULL pointer with BHs disabled (under
> ce_lock):

My v3 covers this case in 3/5 and 4/5. In 3/5,
ath11k_ce_completed_send_next() returns -EIO for a ring without a tail
pointer before it takes srng->lock, since ath11k_hal_srng_clear()
zeroes the lock with the ring, and ath11k_ce_send() tests CRASH_FLUSH
before it reaps. In 4/5, ath11k_core_qmi_firmware_ready() clears
CRASH_FLUSH once the rings are rebuilt and sets it again on its error
path before ath11k_dp_free().

> I can try that if you don't plan to cover it in a v3.

I will run your crash 10-20 ms after "is now up" against v3 on my
AX3600 and post it with your three patches on top as one series, with
your From: and Signed-off-by kept and your Tested-by on my five. Is
that fine with you?

Thanks,
Julius

Re: [PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[Florian Maurer]

Hi Julius,

On Sun, 2026-11-01 at 17:47 +0100, Julius Bairaktaris wrote:
> I will run your crash 10-20 ms after "is now up" against v3 on my
> AX3600 and post it with your three patches on top as one series, with
> your From: and Signed-off-by kept and your Tested-by on my five. Is
> that fine with you?
> 

Yes, sure - thanks! I'll also test this again as needed.

Thanks,
Florian