* Re: [PATCH net v3] net: iterate online nodes in skb_defer_free_flush()
2026-09-16 0:34 [PATCH net v3] net: iterate online nodes in skb_defer_free_flush() Kris Pan
@ 2026-09-16 1:08 ` Eric Dumazet
2026-09-16 1:10 ` Kris Pan
0 siblings, 1 reply; 3+ messages in thread
From: Eric Dumazet @ 2026-09-16 1:08 UTC (permalink / raw)
To: Kris Pan
Cc: davem, kuba, pabeni, horms, atomasov, oliver.sang, netdev, linux-kernel
On Tue, Sep 15, 2026 at 5:37 PM Kris Pan <kris.pan@intel.com> wrote:
>
> skb_attempt_defer_free() only queues skbs on the current CPU's node
> (numa_node_id() of a running CPU), which is always online, so the
> flush loop never needs to visit nodes that are merely possible.
>
> for_each_node() walks node_possible_map. On machines where the
> possible map is much larger than the online map -- e.g. a POWER10
> LPAR with 32 possible but 1 online node -- the flush loop touches 31
> cold, always-empty per-node lists on every softirq pass.
>
> Use for_each_online_node() to iterate only node_online_map.
>
> Since the flush now skips offline nodes, also drain the per-node
> lists in dev_cpu_dead(). A skb is queued on the alloc CPU's list at
> the freeing CPU's node index; if that node is offlined before the
> alloc CPU flushes, the skb is stranded. Likewise, skbs queued on a
> CPU that later goes offline are never freed, since its softirq no
> longer runs.
>
> Loopback UDP throughput in a QEMU guest with 32 possible / 1 online
> nodes (bench_udp, 8 senders, interleaved runs) improves by ~5%.
>
> Fixes: 5628f3fe3b16 ("net: add NUMA awareness to skb_attempt_defer_free()")
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Suggested-by: Adrian Tomasov <atomasov@redhat.com>
> Suggested-by: Eric Dumazet <edumazet@google.com>
> Closes: https://lore.kernel.org/oe-lkp/202512112119.5b9829a-lkp@intel.com
> Signed-off-by: Kris Pan <kris.pan@intel.com>
> ---
pw-bot: cr
Missing atomic_long_set(&sdn->defer_count, 0) permanently wedges defer_count.
I suggest we fix the pre-existing bug in the net tree, then later in
net-next you send your one-liner patch?
(s/for_each_node/for_each_online_node)
My LLM (Gemini) came up with this (untested) patch.
commit 24094a55ca66a5507ea6eba395e469146ede7cc1
Author: Eric Dumazet <edumazet@google.com>
Date: Wed Sep 16 00:12:57 2026 +0000
net: flush skb_defer_nodes in dev_cpu_dead()
When a CPU goes offline, dev_cpu_dead() drains its softnet queues
(completion_queue, output_queue, poll_list, process_queue, and
input_pkt_queue), but leaves net_hotdata.skb_defer_nodes untouched.
If oldcpu goes offline while holding pending skbs in its
skb_defer_nodes lists (e.g. below the sysctl_skb_defer_max >> 1 IPI
threshold, or if the IPI races with CPU teardown), those skbs remain
stranded until oldcpu is brought back online. If any of these skbs
hold page_pool fragments, page_pool_destroy() will stall indefinitely
waiting for inflight pages to be returned when a netdev or driver is
torn down while oldcpu is offline.
Additionally, if smp_call_function_single_async() fails in
kick_defer_list_purge() because the target CPU went offline, reset
defer_ipi_scheduled to 0 so future IPI kicks are not blocked when the
CPU comes back online.
Also, if oldcpu was the last online CPU on its NUMA node, drain that
node's slot across all CPUs so no skbs deferred from that node remain
stranded on idle remote CPUs (or if the node itself is subsequently
offlined).
Finally, in skb_attempt_defer_free(), re-check cpu_online(cpu) and
whether the caller migrated CPUs after llist_add(), flushing the node
list if so, to close the preemption TOCTOU race against CPU/node
teardown.
Fixes: 68822bdf76f1 ("net: generalize skb freeing deferral to
per-cpu lists")
Fixes: 5628f3fe3b16 ("net: add NUMA awareness to skb_attempt_defer_free()")
Assisted-by: LLM
Signed-off-by: Eric Dumazet <edumazet@google.com>
diff --git a/net/core/dev.c b/net/core/dev.c
index ecfbd72d5d1a41d60e2276e9d78160c063fa347c..a3492d675138658f1f7bb4e6c48a306dbc3efb1b
100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5376,7 +5376,8 @@ void kick_defer_list_purge(unsigned int cpu)
backlog_unlock_irq_restore(sd, flags);
} else if (!cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) {
- smp_call_function_single_async(cpu, &sd->defer_csd);
+ if (smp_call_function_single_async(cpu, &sd->defer_csd))
+ WRITE_ONCE(sd->defer_ipi_scheduled, 0);
}
}
@@ -6900,25 +6901,35 @@ bool napi_complete_done(struct napi_struct *n,
int work_done)
}
EXPORT_SYMBOL(napi_complete_done);
-static void skb_defer_free_flush(void)
+static void __skb_defer_free_flush(struct skb_defer_node *sdn, int budget)
{
struct llist_node *free_list;
struct sk_buff *skb, *next;
+
+ if (llist_empty(&sdn->defer_list))
+ return;
+ atomic_long_set(&sdn->defer_count, 0);
+ free_list = llist_del_all(&sdn->defer_list);
+
+ llist_for_each_entry_safe(skb, next, free_list, ll_node) {
+ prefetch(next);
+ napi_consume_skb(skb, budget);
+ }
+}
+
+void skb_defer_node_flush(struct skb_defer_node *sdn)
+{
+ __skb_defer_free_flush(sdn, 0);
+}
+
+static void skb_defer_free_flush(void)
+{
struct skb_defer_node *sdn;
int node;
for_each_node(node) {
sdn = this_cpu_ptr(net_hotdata.skb_defer_nodes) + node;
-
- if (llist_empty(&sdn->defer_list))
- continue;
- atomic_long_set(&sdn->defer_count, 0);
- free_list = llist_del_all(&sdn->defer_list);
-
- llist_for_each_entry_safe(skb, next, free_list, ll_node) {
- prefetch(next);
- napi_consume_skb(skb, 1);
- }
+ __skb_defer_free_flush(sdn, 1);
}
}
@@ -12897,6 +12908,7 @@ static int dev_cpu_dead(unsigned int oldcpu)
struct sk_buff **list_skb;
struct sk_buff *skb;
unsigned int cpu;
+ int node;
struct softnet_data *sd, *oldsd, *remsd = NULL;
local_irq_disable();
@@ -12957,6 +12969,17 @@ static int dev_cpu_dead(unsigned int oldcpu)
rps_input_queue_head_incr(oldsd);
}
+ WRITE_ONCE(oldsd->defer_ipi_scheduled, 0);
+ for_each_node(node)
+ skb_defer_node_flush(per_cpu_ptr(net_hotdata.skb_defer_nodes,
+ oldcpu) + node);
+ node = cpu_to_node(oldcpu);
+ if (node_possible(node) && cpumask_empty(cpumask_of_node(node))) {
+ for_each_possible_cpu(cpu)
+
skb_defer_node_flush(per_cpu_ptr(net_hotdata.skb_defer_nodes,
+ cpu) + node);
+ }
+
return 0;
}
diff --git a/net/core/dev.h b/net/core/dev.h
index b757faead4d1a3e445e54d2f468c38c9e09b6762..04fb0e9a571e03f41cac110a0d1dd864d110b339
100644
--- a/net/core/dev.h
+++ b/net/core/dev.h
@@ -399,6 +399,8 @@ static inline void napi_assert_will_not_race(const
struct napi_struct *napi)
WARN_ON(READ_ONCE(napi->list_owner) != -1);
}
+struct skb_defer_node;
+void skb_defer_node_flush(struct skb_defer_node *sdn);
void kick_defer_list_purge(unsigned int cpu);
int dev_set_hwtstamp_phylib(struct net_device *dev,
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index a2d7b2fbc006d70e965c820b99d6eb4b3432e7c8..d8bd8ed2489e328d81aabde4114c0e878ebe0b63
100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -7332,8 +7332,8 @@ void skb_attempt_defer_free(struct sk_buff *skb)
struct skb_defer_node *sdn;
unsigned long defer_count;
unsigned int defer_max;
+ int cpu, my_cpu;
bool kick;
- int cpu;
if (static_branch_unlikely(&skb_defer_disable_key))
goto nodefer;
@@ -7343,7 +7343,8 @@ void skb_attempt_defer_free(struct sk_buff *skb)
goto nodefer;
cpu = skb->alloc_cpu;
- if (cpu == raw_smp_processor_id() ||
+ my_cpu = raw_smp_processor_id();
+ if (cpu == my_cpu ||
WARN_ON_ONCE(cpu >= nr_cpu_ids) ||
!cpu_online(cpu)) {
nodefer: kfree_skb_napi_cache(skb);
@@ -7354,7 +7355,7 @@ nodefer: kfree_skb_napi_cache(skb);
DEBUG_NET_WARN_ON_ONCE(skb->destructor);
DEBUG_NET_WARN_ON_ONCE(skb_nfct(skb));
- sdn = per_cpu_ptr(net_hotdata.skb_defer_nodes, cpu) + numa_node_id();
+ sdn = per_cpu_ptr(net_hotdata.skb_defer_nodes, cpu) +
cpu_to_node(my_cpu);
defer_max = READ_ONCE(net_hotdata.sysctl_skb_defer_max);
defer_count = atomic_long_inc_return(&sdn->defer_count);
@@ -7364,6 +7365,11 @@ nodefer: kfree_skb_napi_cache(skb);
llist_add(&skb->ll_node, &sdn->defer_list);
+ if (unlikely(!cpu_online(cpu) || my_cpu != raw_smp_processor_id())) {
+ skb_defer_node_flush(sdn);
+ return;
+ }
+
/* Send an IPI every time queue reaches half capacity. */
kick = (defer_count - 1) == (defer_max >> 1);
^ permalink raw reply [flat|nested] 3+ messages in thread