mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown
@ 2026-11-14 19:54 pavankumaryalagada
  2026-11-15 16:09 ` Julian Anastasov
  0 siblings, 1 reply; 2+ messages in thread
From: pavankumaryalagada @ 2026-11-14 19:54 UTC (permalink / raw)
  To: horms, ja
  Cc: pablo, fw, phil, davem, edumazet, kuba, pabeni, netdev,
	lvs-devel, netfilter-devel, coreteam, linux-kernel, skhan,
	Yalagada Pavan Kumar, syzbot+84e27adb95fd2a1fe9bb, stable

From: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>

During network namespace cleanup, IPVS releases cached destinations
using RCU callbacks. However, ip6_route_net_exit() destroys the
per-CPU destination counters before these callbacks complete.
The RCU callbacks call dst_release(), which updates the destroyed
counters and causes a NULL pointer dereference in
percpu_counter_add_batch().

Call rcu_barrier() at the end of __ip_vs_cleanup_batch() to make
sure all pending IPVS RCU callbacks finish before the destination
counters are destroyed.

Reported-by: syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com
Link: https://lore.kernel.org/all/6af795e2.98e60839.1e023a.00d4.GAE@google.com/T/
Closes: https://syzkaller.appspot.com/bug?extid=84e27adb95fd2a1fe9bb
Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
Cc: stable@vger.kernel.org
Signed-off-by: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
---
 net/netfilter/ipvs/ip_vs_core.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 8a3c880fdb26..9c456efe13c7 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -2553,6 +2553,11 @@ static void __net_exit __ip_vs_cleanup_batch(struct list_head *net_list)
 		IP_VS_DBG(2, "ipvs netns %d released\n", ipvs->gen);
 		net->ipvs = NULL;
 	}
+
+	/*
+	 * Wait for ip_vs_dest_dst_rcu_free() callbacks before dst_ops teardown
+	 */
+	rcu_barrier();
 }
 
 static void __net_exit __ip_vs_dev_cleanup_batch(struct list_head *net_list)
-- 
2.43.0


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown
  2026-11-14 19:54 [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown pavankumaryalagada
@ 2026-11-15 16:09 ` Julian Anastasov
  0 siblings, 0 replies; 2+ messages in thread
From: Julian Anastasov @ 2026-11-15 16:09 UTC (permalink / raw)
  To: Yalagada Pavan Kumar
  Cc: Simon Horman, pablo, fw, phil, David S. Miller, edumazet, kuba,
	pabeni, netdev, lvs-devel, netfilter-devel, coreteam,
	linux-kernel, skhan, syzbot+84e27adb95fd2a1fe9bb, stable,
	Wei Wang


	Hello,

	Adding Wei Wang to the CC list...

On Sun, 15 Nov 2026, pavankumaryalagada@gmail.com wrote:

> From: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
> 
> During network namespace cleanup, IPVS releases cached destinations
> using RCU callbacks. However, ip6_route_net_exit() destroys the
> per-CPU destination counters before these callbacks complete.
> The RCU callbacks call dst_release(), which updates the destroyed
> counters and causes a NULL pointer dereference in
> percpu_counter_add_batch().
> 
> Call rcu_barrier() at the end of __ip_vs_cleanup_batch() to make
> sure all pending IPVS RCU callbacks finish before the destination
> counters are destroyed.

	What you see is a result of more fundamental problem.
IPVS is supposed to catch all NETDEV_DOWN events coming from the
pernet device phase which come before the pernet subsys phase.
So, on cleanup_net IPVS is supposed to run its subsys cleanup
with no traffic, no devs, no cached dsts, no netfilter hooks, etc.

	But in my tests free_fib_info_rcu -> fib_nh_release ->
fib_nh_common_release -> rt_fibinfo_free_cpus calls
dst_dev_put() where the dst device that IPVS listens for is replaced
with a blackhole_netdev. This happens before the NETDEV_DOWN
event and as result IPVS misses the event because dst->dev is
already replaced and does not match. When such dst_cache
remains for the pernet subsys cleanup you see these crashes
because IPVS has another problem that it does not call
dst_release immediately but from RCU callback.

	I now created a change to call dst_release immediately
but the problem with the blackholing before NETDEV_DOWN remains.
It can hurt other event listeners.

	My tests were with tools/testing/selftests/net/netfilter/ipvs.sh
where I modify it to run only the NAT test. What I see is veth12
to be blackholed before its NETDEV_DOWN event and ip_vs_forget_dev()
missed it. Trying to find more information...

> 
> Reported-by: syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com
> Link: https://lore.kernel.org/all/6af795e2.98e60839.1e023a.00d4.GAE@google.com/T/
> Closes: https://syzkaller.appspot.com/bug?extid=84e27adb95fd2a1fe9bb
> Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
> ---
>  net/netfilter/ipvs/ip_vs_core.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index 8a3c880fdb26..9c456efe13c7 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -2553,6 +2553,11 @@ static void __net_exit __ip_vs_cleanup_batch(struct list_head *net_list)
>  		IP_VS_DBG(2, "ipvs netns %d released\n", ipvs->gen);
>  		net->ipvs = NULL;
>  	}
> +
> +	/*
> +	 * Wait for ip_vs_dest_dst_rcu_free() callbacks before dst_ops teardown
> +	 */
> +	rcu_barrier();
>  }
>  
>  static void __net_exit __ip_vs_dev_cleanup_batch(struct list_head *net_list)
> -- 
> 2.43.0

Regards

--
Julian Anastasov <ja@ssi.bg>


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-15 16:10 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-14 19:54 [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown pavankumaryalagada
2026-11-15 16:09 ` Julian Anastasov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®