* [PATCH rtw-next v3 1/5] wifi: rtw88: free the reserved pages of an interface on firmware recovery
2026-11-17 17:09 [PATCH rtw-next v3 0/5] wifi: rtw88: fixes around the firmware recovery Mehmet Fide
@ 2026-11-17 17:09 ` Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 2/5] wifi: rtw88: report the frames purged from the TX report queue Mehmet Fide
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Mehmet Fide @ 2026-11-17 17:09 UTC (permalink / raw)
To: Ping-Ke Shih
Cc: Luka Gejak, Sascha Hauer, Zong-Zhe Yang, Tzu-En Huang,
Yan-Hsuan Chuang, Kalle Valo, Stanislaw Gruszka, Brian Norris,
linux-wireless, linux-kernel, mehmet.fide
From: Mehmet Fide <mehmet.fide@screeningeagle.com>
The recovery after a firmware crash resets the driver state of every
interface, and mac80211 then adds the interfaces again without removing
them first. rtw_ops_add_interface() starts with an empty reserved page
list, so the pages the interface had before the crash are lost: the
next page build takes them off the build list and nothing frees them.
kmemleak after a firmware crash with an AP interface:
unreferenced object 0xffff0000cbe22c60 (size 96):
comm "hostapd", pid 25545, jiffies 4296463067
backtrace (crc cbdc21c4):
kmemleak_alloc+0xcc/0xe4
__kmalloc_cache_noprof+0x234/0x3bc
rtw_add_rsvd_page+0x38/0xa8 [rtw88_core]
rtw_add_rsvd_page_bcn+0x30/0x58 [rtw88_core]
rtw_ops_add_interface+0x144/0x2cc [rtw88_core]
rtw_ops_change_interface+0x6c/0x80 [rtw88_core]
drv_change_interface+0x100/0x258 [mac80211]
Free the pages where the recovery resets the interface.
With the pages freed, a reserved page download before mac80211 adds the
interfaces again finds no page and warns "Should not have an empty
reserved page". The beacon update work queued by rtw_ops_set_tim() and
a beacon change through rtw_ops_bss_info_changed() can both get there:
the recovery drops the driver mutex in rtw_core_stop(), and mac80211
restarts the hardware only after the recovery. The recovery also clears
the port of every interface, and rtw_ops_add_interface() sets it again,
so download nothing and send no page locations while no port is in use.
Fixes: 5c831644e1f4 ("rtw88: handle and recover when firmware crash")
Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
---
Notes:
v3: no RTW_FLAG_RESTARTING test in the beacon update work. The page
download and the page location H2C do nothing while no port is in
use, which also covers a beacon change in the restart window and a
TIM update during the reconfiguration (Luka Gejak)
v2: the beacon update work does nothing while RTW_FLAG_RESTARTING is set
(Luka Gejak): rtw_ops_set_tim() can queue it between the reset and the
reconfiguration, where it found no page and warned
drivers/net/wireless/realtek/rtw88/fw.c | 6 ++++++
drivers/net/wireless/realtek/rtw88/main.c | 1 +
2 files changed, 7 insertions(+)
diff --git a/drivers/net/wireless/realtek/rtw88/fw.c b/drivers/net/wireless/realtek/rtw88/fw.c
index 945fedcd375b..c66dc9086378 100644
--- a/drivers/net/wireless/realtek/rtw88/fw.c
+++ b/drivers/net/wireless/realtek/rtw88/fw.c
@@ -1059,6 +1059,9 @@ void rtw_send_rsvd_page_h2c(struct rtw_dev *rtwdev)
u8 h2c_pkt[H2C_PKT_SIZE] = {0};
u8 location = 0;
+ if (bitmap_empty(rtwdev->hw_port, RTW_PORT_NUM))
+ return;
+
SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_RSVD_PAGE);
location = rtw_get_rsvd_page_location(rtwdev, RSVD_PROBE_RESP);
@@ -1765,6 +1768,9 @@ int rtw_fw_download_rsvd_page(struct rtw_dev *rtwdev)
u32 size;
int ret;
+ if (bitmap_empty(rtwdev->hw_port, RTW_PORT_NUM))
+ return 0;
+
buf = rtw_build_rsvd_page(rtwdev, &size);
if (!buf) {
rtw_err(rtwdev, "failed to build rsvd page pkt\n");
diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c
index f7708ad17b6f..adc54b063382 100644
--- a/drivers/net/wireless/realtek/rtw88/main.c
+++ b/drivers/net/wireless/realtek/rtw88/main.c
@@ -627,6 +627,7 @@ static void rtw_reset_vif_iter(void *data, u8 *mac, struct ieee80211_vif *vif)
rtw_bf_disassoc(rtwdev, vif, NULL);
rtw_vif_assoc_changed(rtwvif, NULL);
rtw_txq_cleanup(rtwdev, vif->txq);
+ rtw_remove_rsvd_page(rtwdev, rtwvif);
rtw_release_macid(rtwdev, rtwvif->mac_id);
}
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH rtw-next v3 2/5] wifi: rtw88: report the frames purged from the TX report queue
2026-11-17 17:09 [PATCH rtw-next v3 0/5] wifi: rtw88: fixes around the firmware recovery Mehmet Fide
2026-11-17 17:09 ` [PATCH rtw-next v3 1/5] wifi: rtw88: free the reserved pages of an interface on " Mehmet Fide
@ 2026-11-17 17:10 ` Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 3/5] wifi: rtw88: do not drop stations when collecting them for an iteration Mehmet Fide
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Mehmet Fide @ 2026-11-17 17:10 UTC (permalink / raw)
To: Ping-Ke Shih
Cc: Luka Gejak, Sascha Hauer, Zong-Zhe Yang, Tzu-En Huang,
Yan-Hsuan Chuang, Kalle Valo, Stanislaw Gruszka, Brian Norris,
linux-wireless, linux-kernel, mehmet.fide
From: Mehmet Fide <mehmet.fide@screeningeagle.com>
A frame whose sender asked for its TX status waits in the TX report
queue for the report of the firmware. When no report comes, the purge
timer frees the frames with skb_queue_purge(), and rtw_core_deinit()
does the same. mac80211 is not told: the sender gets no status, and the
copy mac80211 keeps for the status of a frame from user space stays in
its list until ieee80211_free_hw() warns about it. On an RTL8822CE with
hostapd, after a firmware crash and an unbind of the driver:
rtw88_8822ce 0000:01:00.0: failed to get tx report from firmware
[...]
Have pending ack frames!
WARNING: net/mac80211/main.c:1761 at ieee80211_free_ack_frame+0x4c/0x70 [mac80211], CPU#1: sh/24725
Call trace:
ieee80211_free_ack_frame+0x4c/0x70 [mac80211] (P)
idr_for_each+0x6c/0xf4
ieee80211_free_hw+0x30/0x8c [mac80211]
rtw_pci_remove+0xd8/0xf0 [rtw88_pci]
pci_device_remove+0x44/0xd4
Take the frames off the queue under its lock and hand them to
ieee80211_purge_tx_queue(), which reports each one as not sent.
Fixes: e3037485c68e ("rtw88: new Realtek 802.11ac driver")
Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
Reviewed-by: Luka Gejak <luka.gejak@linux.dev>
---
Notes:
v3: no change
v2: Reviewed-by Luka Gejak, no change
drivers/net/wireless/realtek/rtw88/main.c | 5 +----
drivers/net/wireless/realtek/rtw88/tx.c | 20 ++++++++++++++++----
drivers/net/wireless/realtek/rtw88/tx.h | 1 +
3 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c
index adc54b063382..ce65a1a80840 100644
--- a/drivers/net/wireless/realtek/rtw88/main.c
+++ b/drivers/net/wireless/realtek/rtw88/main.c
@@ -2225,7 +2225,6 @@ void rtw_core_deinit(struct rtw_dev *rtwdev)
struct rtw_fw_state *fw = &rtwdev->fw;
struct rtw_fw_state *wow_fw = &rtwdev->wow_fw;
struct rtw_rsvd_page *rsvd_pkt, *tmp;
- unsigned long flags;
rtw_wait_firmware_completion(rtwdev);
@@ -2237,9 +2236,7 @@ void rtw_core_deinit(struct rtw_dev *rtwdev)
destroy_workqueue(rtwdev->tx_wq);
timer_delete_sync(&rtwdev->tx_report.purge_timer);
- spin_lock_irqsave(&rtwdev->tx_report.q_lock, flags);
- skb_queue_purge(&rtwdev->tx_report.queue);
- spin_unlock_irqrestore(&rtwdev->tx_report.q_lock, flags);
+ rtw_tx_report_purge(rtwdev);
skb_queue_purge(&rtwdev->coex.queue);
skb_queue_purge(&rtwdev->c2h_queue);
diff --git a/drivers/net/wireless/realtek/rtw88/tx.c b/drivers/net/wireless/realtek/rtw88/tx.c
index 701ed7c4f3f7..87d993282b1c 100644
--- a/drivers/net/wireless/realtek/rtw88/tx.c
+++ b/drivers/net/wireless/realtek/rtw88/tx.c
@@ -176,21 +176,33 @@ static void rtw_tx_report_enable(struct rtw_dev *rtwdev,
pkt_info->report = true;
}
+void rtw_tx_report_purge(struct rtw_dev *rtwdev)
+{
+ struct rtw_tx_report *tx_report = &rtwdev->tx_report;
+ struct sk_buff_head purge;
+ unsigned long flags;
+
+ __skb_queue_head_init(&purge);
+
+ spin_lock_irqsave(&tx_report->q_lock, flags);
+ skb_queue_splice_init(&tx_report->queue, &purge);
+ spin_unlock_irqrestore(&tx_report->q_lock, flags);
+
+ ieee80211_purge_tx_queue(rtwdev->hw, &purge);
+}
+
void rtw_tx_report_purge_timer(struct timer_list *t)
{
struct rtw_dev *rtwdev = timer_container_of(rtwdev, t,
tx_report.purge_timer);
struct rtw_tx_report *tx_report = &rtwdev->tx_report;
- unsigned long flags;
if (skb_queue_len(&tx_report->queue) == 0)
return;
rtw_warn(rtwdev, "failed to get tx report from firmware\n");
- spin_lock_irqsave(&tx_report->q_lock, flags);
- skb_queue_purge(&tx_report->queue);
- spin_unlock_irqrestore(&tx_report->q_lock, flags);
+ rtw_tx_report_purge(rtwdev);
}
void rtw_tx_report_enqueue(struct rtw_dev *rtwdev, struct sk_buff *skb, u8 sn)
diff --git a/drivers/net/wireless/realtek/rtw88/tx.h b/drivers/net/wireless/realtek/rtw88/tx.h
index 168f48e87853..363f63525965 100644
--- a/drivers/net/wireless/realtek/rtw88/tx.h
+++ b/drivers/net/wireless/realtek/rtw88/tx.h
@@ -98,6 +98,7 @@ void rtw_tx_pkt_info_update(struct rtw_dev *rtwdev,
struct sk_buff *skb);
void rtw_tx_fill_tx_desc(struct rtw_dev *rtwdev,
struct rtw_tx_pkt_info *pkt_info, struct sk_buff *skb);
+void rtw_tx_report_purge(struct rtw_dev *rtwdev);
void rtw_tx_report_enqueue(struct rtw_dev *rtwdev, struct sk_buff *skb, u8 sn);
void rtw_tx_report_handle(struct rtw_dev *rtwdev, struct sk_buff *skb, int src);
void rtw_tx_rsvd_page_pkt_info_update(struct rtw_dev *rtwdev,
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH rtw-next v3 3/5] wifi: rtw88: do not drop stations when collecting them for an iteration
2026-11-17 17:09 [PATCH rtw-next v3 0/5] wifi: rtw88: fixes around the firmware recovery Mehmet Fide
2026-11-17 17:09 ` [PATCH rtw-next v3 1/5] wifi: rtw88: free the reserved pages of an interface on " Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 2/5] wifi: rtw88: report the frames purged from the TX report queue Mehmet Fide
@ 2026-11-17 17:10 ` Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 4/5] wifi: rtw88: reset the stations outside of RCU on firmware recovery Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 5/5] wifi: rtw88: clear the restart flag when the device is stopped Mehmet Fide
4 siblings, 0 replies; 6+ messages in thread
From: Mehmet Fide @ 2026-11-17 17:10 UTC (permalink / raw)
To: Ping-Ke Shih
Cc: Luka Gejak, Sascha Hauer, Zong-Zhe Yang, Tzu-En Huang,
Yan-Hsuan Chuang, Kalle Valo, Stanislaw Gruszka, Brian Norris,
linux-wireless, linux-kernel, mehmet.fide
From: Mehmet Fide <mehmet.fide@screeningeagle.com>
rtw_iterate_stas() collects the stations with an atomic iterator and
calls the iterator for them afterwards. It allocates an entry for each
station with GFP_ATOMIC and silently skips the station when that fails.
The firmware recovery resets the stations through it with the next
patch. A skipped station keeps its TX queues on rtwdev->txqs, and when
mac80211 adds it again, rtw_txq_init() reinitializes the list node of a
queue that is still linked there.
Link the stations through a list node in struct rtw_sta_info instead,
so the collection cannot fail. All callers hold rtwdev->mutex, which
keeps two collections apart, and no station iterator collects again.
Reported-by: Luka Gejak <luka.gejak@linux.dev>
Closes: https://lore.kernel.org/linux-wireless/4EE3D95E-988D-4514-9849-9546CD49D8E9@linux.dev/
Fixes: 78d5bf925f30 ("wifi: rtw88: iterate over vif/sta list non-atomically")
Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
---
Notes:
v3: new, recovery resets every station through rtw_iterate_stas() with
the next patch (Luka Gejak)
drivers/net/wireless/realtek/rtw88/main.h | 1 +
drivers/net/wireless/realtek/rtw88/util.c | 24 ++++++-----------------
2 files changed, 7 insertions(+), 18 deletions(-)
diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h
index 6ecbeb4ffe4b..6b26c517c738 100644
--- a/drivers/net/wireless/realtek/rtw88/main.h
+++ b/drivers/net/wireless/realtek/rtw88/main.h
@@ -792,6 +792,7 @@ struct rtw_sta_info {
struct cfg80211_bitrate_mask *mask;
struct work_struct rc_work;
+ struct list_head iter_list;
};
enum rtw_bfee_role {
diff --git a/drivers/net/wireless/realtek/rtw88/util.c b/drivers/net/wireless/realtek/rtw88/util.c
index fcd6eb1ab32a..3e8a520177d1 100644
--- a/drivers/net/wireless/realtek/rtw88/util.c
+++ b/drivers/net/wireless/realtek/rtw88/util.c
@@ -107,11 +107,6 @@ void rtw_desc_to_mcsrate(u16 rate, u8 *mcs, u8 *nss)
}
}
-struct rtw_stas_entry {
- struct list_head list;
- struct ieee80211_sta *sta;
-};
-
struct rtw_iter_stas_data {
struct rtw_dev *rtwdev;
struct list_head list;
@@ -120,14 +115,9 @@ struct rtw_iter_stas_data {
static void rtw_collect_sta_iter(void *data, struct ieee80211_sta *sta)
{
struct rtw_iter_stas_data *iter_stas = data;
- struct rtw_stas_entry *stas_entry;
+ struct rtw_sta_info *si = (struct rtw_sta_info *)sta->drv_priv;
- stas_entry = kmalloc_obj(*stas_entry, GFP_ATOMIC);
- if (!stas_entry)
- return;
-
- stas_entry->sta = sta;
- list_add_tail(&stas_entry->list, &iter_stas->list);
+ list_add_tail(&si->iter_list, &iter_stas->list);
}
void rtw_iterate_stas(struct rtw_dev *rtwdev,
@@ -136,7 +126,7 @@ void rtw_iterate_stas(struct rtw_dev *rtwdev,
void *data)
{
struct rtw_iter_stas_data iter_data;
- struct rtw_stas_entry *sta_entry, *tmp;
+ struct rtw_sta_info *si, *tmp;
/* &rtwdev->mutex makes sure no stations can be removed between
* collecting the stations and iterating over them.
@@ -149,11 +139,9 @@ void rtw_iterate_stas(struct rtw_dev *rtwdev,
ieee80211_iterate_stations_atomic(rtwdev->hw, rtw_collect_sta_iter,
&iter_data);
- list_for_each_entry_safe(sta_entry, tmp, &iter_data.list,
- list) {
- list_del_init(&sta_entry->list);
- iterator(data, sta_entry->sta);
- kfree(sta_entry);
+ list_for_each_entry_safe(si, tmp, &iter_data.list, iter_list) {
+ list_del_init(&si->iter_list);
+ iterator(data, si->sta);
}
}
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH rtw-next v3 4/5] wifi: rtw88: reset the stations outside of RCU on firmware recovery
2026-11-17 17:09 [PATCH rtw-next v3 0/5] wifi: rtw88: fixes around the firmware recovery Mehmet Fide
` (2 preceding siblings ...)
2026-11-17 17:10 ` [PATCH rtw-next v3 3/5] wifi: rtw88: do not drop stations when collecting them for an iteration Mehmet Fide
@ 2026-11-17 17:10 ` Mehmet Fide
2026-11-17 17:10 ` [PATCH rtw-next v3 5/5] wifi: rtw88: clear the restart flag when the device is stopped Mehmet Fide
4 siblings, 0 replies; 6+ messages in thread
From: Mehmet Fide @ 2026-11-17 17:10 UTC (permalink / raw)
To: Ping-Ke Shih
Cc: Luka Gejak, Sascha Hauer, Zong-Zhe Yang, Tzu-En Huang,
Yan-Hsuan Chuang, Kalle Valo, Stanislaw Gruszka, Brian Norris,
linux-wireless, linux-kernel, mehmet.fide
From: Mehmet Fide <mehmet.fide@screeningeagle.com>
The recovery after a firmware crash removes the driver state of every
station with rtw_sta_remove(), called from an atomic station iterator.
rtw_sta_remove() cancels the rate work of the station with
cancel_work_sync(), which can sleep. With a station associated:
BUG: sleeping function called from invalid context at kernel/workqueue.c:4722
in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 14, name: kworker/u16:1
RCU nest depth: 1, expected: 0
Workqueue: phy35 rtw_fw_recovery_work [rtw88_core]
Call trace:
__might_resched+0x13c/0x1f0
__might_sleep+0x48/0x7c
__cancel_work_sync+0xa8/0xb4
cancel_work_sync+0x14/0x20
rtw_sta_remove+0x34/0x10c [rtw88_core]
rtw_reset_sta_iter+0x20/0x48 [rtw88_core]
ieee80211_iterate_stations_atomic+0x80/0x144 [mac80211]
rtw_fw_recovery_work+0x1dc/0x2b8 [rtw88_core]
Use rtw_iterate_stas(), which collects the stations under RCU and calls
the iterator after it. The recovery holds the driver mutex it asks for.
Fixes: bcafcb959a57 ("wifi: rtw88: use work to update rate to avoid RCU warning")
Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
Reviewed-by: Luka Gejak <luka.gejak@linux.dev>
---
Notes:
v3: no change; the station skip Luka Gejak found is removed by the
previous patch
v2: Reviewed-by Luka Gejak, no change
drivers/net/wireless/realtek/rtw88/main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c
index ce65a1a80840..8e7ae747b900 100644
--- a/drivers/net/wireless/realtek/rtw88/main.c
+++ b/drivers/net/wireless/realtek/rtw88/main.c
@@ -664,7 +664,7 @@ static void __fw_recovery_work(struct rtw_dev *rtwdev)
WARN(1, "firmware crash, start reset and recover\n");
rtw_iterate_keys_atomic(rtwdev, NULL, rtw_reset_key_iter, rtwdev);
- rtw_iterate_stas_atomic(rtwdev, rtw_reset_sta_iter, rtwdev);
+ rtw_iterate_stas(rtwdev, rtw_reset_sta_iter, rtwdev);
rtw_iterate_vifs_atomic(rtwdev, rtw_reset_vif_iter, rtwdev);
bitmap_zero(rtwdev->hw_port, RTW_PORT_NUM);
rtw_enter_ips(rtwdev);
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH rtw-next v3 5/5] wifi: rtw88: clear the restart flag when the device is stopped
2026-11-17 17:09 [PATCH rtw-next v3 0/5] wifi: rtw88: fixes around the firmware recovery Mehmet Fide
` (3 preceding siblings ...)
2026-11-17 17:10 ` [PATCH rtw-next v3 4/5] wifi: rtw88: reset the stations outside of RCU on firmware recovery Mehmet Fide
@ 2026-11-17 17:10 ` Mehmet Fide
4 siblings, 0 replies; 6+ messages in thread
From: Mehmet Fide @ 2026-11-17 17:10 UTC (permalink / raw)
To: Ping-Ke Shih
Cc: Luka Gejak, Sascha Hauer, Zong-Zhe Yang, Tzu-En Huang,
Yan-Hsuan Chuang, Kalle Valo, Stanislaw Gruszka, Brian Norris,
linux-wireless, linux-kernel, mehmet.fide
From: Mehmet Fide <mehmet.fide@screeningeagle.com>
The firmware recovery sets RTW_FLAG_RESTARTING, and rtw_reconfig_complete()
clears it once mac80211 has restarted the hardware. If the last interface
goes down before the restart work runs, ieee80211_reconfig() finds no open
interface and does not call reconfig_complete(), so the flag is still set
when an interface comes up again. rtw_fw_recovery() then ignores every
later firmware crash, and writing the fw_crash debugfs file returns
-EINPROGRESS.
Clear the flag in rtw_ops_stop(). mac80211 does not stop the device while
it restarts it, so a stop leaves no restart to wait for.
Reported-by: Luka Gejak <luka.gejak@linux.dev>
Closes: https://lore.kernel.org/linux-wireless/5285DA11-651F-4286-A0FD-52A6051B3A88@linux.dev/
Fixes: 5c831644e1f4 ("rtw88: handle and recover when firmware crash")
Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
---
Notes:
v3: new (Luka Gejak)
drivers/net/wireless/realtek/rtw88/mac80211.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/net/wireless/realtek/rtw88/mac80211.c b/drivers/net/wireless/realtek/rtw88/mac80211.c
index 5e40cf06a9e3..10f1ae08f383 100644
--- a/drivers/net/wireless/realtek/rtw88/mac80211.c
+++ b/drivers/net/wireless/realtek/rtw88/mac80211.c
@@ -68,6 +68,7 @@ static void rtw_ops_stop(struct ieee80211_hw *hw, bool suspend)
mutex_lock(&rtwdev->mutex);
rtw_core_stop(rtwdev);
+ clear_bit(RTW_FLAG_RESTARTING, rtwdev->flags);
mutex_unlock(&rtwdev->mutex);
}
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread