From: Mani Tofigh via B4 Relay <devnull+mt4022.columbia.edu@kernel.org>
To: "David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Simon Horman <horms@kernel.org>,
Andrew Lunn <andrew+netdev@lunn.ch>,
Martin Karsten <mkarsten@uwaterloo.ca>, Joe Damato <joe@dama.to>,
Dragos Tatulea <dtatulea@nvidia.com>,
Kuniyuki Iwashima <kuniyu@google.com>,
Stanislav Fomichev <sdf.kernel@gmail.com>,
Sridhar Samudrala <sridhar.samudrala@intel.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>,
Wang Zhan <wang.zhan@smartx.com>,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
tz2294@columbia.edu, Mani Tofigh <mt4022@columbia.edu>
Subject: [PATCH net] net: cancel napi->timer after the grace period in netif_napi_del()
Date: Tue, 03 Nov 2026 13:19:16 -0500 [thread overview]
Message-ID: <20261103-napi-timer-cancel-v1-1-55c631ad5286@columbia.edu> (raw)
From: Mani Tofigh <mt4022@columbia.edu>
A napi can be freed with its timer still queued. napi_disable()
cancels napi->timer, but napi_complete_done(), busy_poll_stop() and
napi_suspend_irqs() start it without owning the napi and can do so
right after the cancel. netif_napi_del() only waits for that start to
finish:
Task A (epoll_wait) Task B (queue reset)
napi_suspend_irqs()
rcu_read_lock()
napi = napi_by_id(id)
napi_disable_locked(napi)
hrtimer_cancel(&napi->timer)
napi_hash_del(napi)
netif_napi_del_locked(napi)
synchronize_net()
/* waits for Task A */
hrtimer_start(&napi->timer)
rcu_read_unlock()
kfree(rq)
/* timer still queued */
With netdevsim and debug objects:
[ 3.073483] ODEBUG: free active (active state 0) object: 000000001504358f object type: hrtimer hint: napi_watchdog+0x0/0xa0
[ 3.073515] WARNING: lib/debugobjects.c:629 at debug_print_object+0x1a4/0x3b0, CPU#0: napitimer/66
[ 3.079603] CPU: 0 UID: 0 PID: 66 Comm: napitimer Tainted: G W 7.3.0-17087-gd865f2677c44 #1 PREEMPT
[ 3.084822] RIP: 0010:debug_print_object+0x1ae/0x3b0
...
[ 3.103822] Call Trace:
[ 3.104408] <TASK>
[ 3.105933] debug_check_no_obj_freed+0x496/0x5f0
[ 3.109988] kfree+0x33a/0x510
[ 3.110715] netdev_rx_queue_reconfig+0x357/0x580
[ 3.111780] netdev_rx_queue_restart+0x8f/0xc0
[ 3.113924] nsim_qreset_write+0x24d/0x350
[ 3.117832] full_proxy_write+0x109/0x1d0
[ 3.118741] vfs_write+0x1e3/0x1250
[ 3.124758] __x64_sys_pwrite64+0x18c/0x250
[ 3.129645] do_syscall_64+0xfb/0x590
[ 3.131385] entry_SYSCALL_64_after_hwframe+0x77/0x7f
Without debug objects, napi_watchdog() later runs on freed memory.
After napi_disable() and a grace period none of the three can reach the
napi anymore. Two need SCHED, which napi_disable() keeps, and
napi_suspend_irqs() needs the napi hash, which napi_disable() removed it
from. So cancel the timer again after the grace period that
netif_napi_del() already waits for, and in free_netdev(), where
unregister has already waited for one. With the fix, debug objects
don't report a freed active timer anymore.
This bug was identified and confirmed with the help of a deterministic
hypervisor currently under development. The C reproducer below however
hits it within about 1s.
Fixes: 6f8b12d661d0 ("net: napi: add hard irqs deferral feature")
Fixes: 3fcbecbdeb04 ("net: Add control functions for irq suspension")
Fixes: 58e2330bd455 ("net: napi: Avoid gro timer misfiring at end of busypoll")
Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
---
Drivers that call __netif_napi_del{,_locked}() and then their own
synchronize_net() (eg netdevsim on close, veth, virtio_net, bnxt and
gro_cells) need the same cancel. I can send that separately.
Reproducer: https://github.com/manitofigh/files/blob/main/linux-repros/napitimer.c
---
include/linux/netdevice.h | 9 +++++++++
net/core/dev.c | 5 ++++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e..ad3685caf814 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3013,8 +3013,13 @@ static inline void __netif_napi_del(struct napi_struct *napi)
static inline void netif_napi_del_locked(struct napi_struct *napi)
{
+ bool listed = test_bit(NAPI_STATE_LISTED, &napi->state);
+
__netif_napi_del_locked(napi);
synchronize_net();
+ /* napi->timer can be started until a grace period after napi_disable() */
+ if (listed)
+ hrtimer_cancel(&napi->timer);
}
/**
@@ -3025,8 +3030,12 @@ static inline void netif_napi_del_locked(struct napi_struct *napi)
*/
static inline void netif_napi_del(struct napi_struct *napi)
{
+ bool listed = test_bit(NAPI_STATE_LISTED, &napi->state);
+
__netif_napi_del(napi);
synchronize_net();
+ if (listed)
+ hrtimer_cancel(&napi->timer);
}
int netif_enable_cpu_rmap(struct net_device *dev, unsigned int num_irqs);
diff --git a/net/core/dev.c b/net/core/dev.c
index f32353b0467c..6d6384522a1b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -12450,9 +12450,12 @@ static void netdev_napi_exit(struct net_device *dev)
if (!list_empty(&dev->napi_list)) {
struct napi_struct *p, *n;
+ /* unregister has waited for a grace period after napi_disable() */
netdev_lock(dev);
- list_for_each_entry_safe(p, n, &dev->napi_list, dev_list)
+ list_for_each_entry_safe(p, n, &dev->napi_list, dev_list) {
+ hrtimer_cancel(&p->timer);
__netif_napi_del_locked(p);
+ }
netdev_unlock(dev);
synchronize_net();
---
base-commit: 647b81ddefc6364290e67bf33c89d54971f8995e
change-id: 20261103-napi-timer-cancel-fa0a285fd581
Best regards,
--
Mani Tofigh <mt4022@columbia.edu>
next reply other threads:[~2026-11-03 18:19 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-03 18:19 Mani Tofigh via B4 Relay [this message]
2026-11-03 18:37 ` Eric Dumazet
2026-11-04 0:55 ` Martin Karsten
2026-11-05 21:43 ` Mani Tofigh
2026-11-14 17:26 ` Martin Karsten
2026-11-05 21:40 ` Mani Tofigh
2026-11-06 9:22 ` netdev-bot+sashiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261103-napi-timer-cancel-v1-1-55c631ad5286@columbia.edu \
--to=devnull+mt4022.columbia.edu@kernel.org \
--cc=andrew+netdev@lunn.ch \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=dtatulea@nvidia.com \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=joe@dama.to \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mkarsten@uwaterloo.ca \
--cc=mt4022@columbia.edu \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sdf.kernel@gmail.com \
--cc=sridhar.samudrala@intel.com \
--cc=tariqt@nvidia.com \
--cc=tz2294@columbia.edu \
--cc=wang.zhan@smartx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®