From: Julian Anastasov <ja@ssi.bg>
To: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
Cc: Simon Horman <horms@verge.net.au>,
pablo@netfilter.org, fw@strlen.de, phil@nwl.cc,
"David S. Miller" <davem@davemloft.net>,
edumazet@kernel.org, kuba@kernel.org, pabeni@redhat.com,
netdev@vger.kernel.org, lvs-devel@vger.kernel.org,
netfilter-devel@vger.kernel.org, coreteam@netfilter.org,
linux-kernel <linux-kernel@vger.kernel.org>,
skhan@linuxfoundation.org,
syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com,
stable@vger.kernel.org, Wei Wang <weiwan@google.com>
Subject: Re: [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown
Date: Sun, 15 Nov 2026 18:09:56 +0200 (EET) [thread overview]
Message-ID: <8473d38d-2d50-ad77-18c1-29bbdda56616@ssi.bg> (raw)
In-Reply-To: <20261114195410.96503-1-pavankumaryalagada@gmail.com>
Hello,
Adding Wei Wang to the CC list...
On Sun, 15 Nov 2026, pavankumaryalagada@gmail.com wrote:
> From: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
>
> During network namespace cleanup, IPVS releases cached destinations
> using RCU callbacks. However, ip6_route_net_exit() destroys the
> per-CPU destination counters before these callbacks complete.
> The RCU callbacks call dst_release(), which updates the destroyed
> counters and causes a NULL pointer dereference in
> percpu_counter_add_batch().
>
> Call rcu_barrier() at the end of __ip_vs_cleanup_batch() to make
> sure all pending IPVS RCU callbacks finish before the destination
> counters are destroyed.
What you see is a result of more fundamental problem.
IPVS is supposed to catch all NETDEV_DOWN events coming from the
pernet device phase which come before the pernet subsys phase.
So, on cleanup_net IPVS is supposed to run its subsys cleanup
with no traffic, no devs, no cached dsts, no netfilter hooks, etc.
But in my tests free_fib_info_rcu -> fib_nh_release ->
fib_nh_common_release -> rt_fibinfo_free_cpus calls
dst_dev_put() where the dst device that IPVS listens for is replaced
with a blackhole_netdev. This happens before the NETDEV_DOWN
event and as result IPVS misses the event because dst->dev is
already replaced and does not match. When such dst_cache
remains for the pernet subsys cleanup you see these crashes
because IPVS has another problem that it does not call
dst_release immediately but from RCU callback.
I now created a change to call dst_release immediately
but the problem with the blackholing before NETDEV_DOWN remains.
It can hurt other event listeners.
My tests were with tools/testing/selftests/net/netfilter/ipvs.sh
where I modify it to run only the NAT test. What I see is veth12
to be blackholed before its NETDEV_DOWN event and ip_vs_forget_dev()
missed it. Trying to find more information...
>
> Reported-by: syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com
> Link: https://lore.kernel.org/all/6af795e2.98e60839.1e023a.00d4.GAE@google.com/T/
> Closes: https://syzkaller.appspot.com/bug?extid=84e27adb95fd2a1fe9bb
> Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
> ---
> net/netfilter/ipvs/ip_vs_core.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index 8a3c880fdb26..9c456efe13c7 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -2553,6 +2553,11 @@ static void __net_exit __ip_vs_cleanup_batch(struct list_head *net_list)
> IP_VS_DBG(2, "ipvs netns %d released\n", ipvs->gen);
> net->ipvs = NULL;
> }
> +
> + /*
> + * Wait for ip_vs_dest_dst_rcu_free() callbacks before dst_ops teardown
> + */
> + rcu_barrier();
> }
>
> static void __net_exit __ip_vs_dev_cleanup_batch(struct list_head *net_list)
> --
> 2.43.0
Regards
--
Julian Anastasov <ja@ssi.bg>
prev parent reply other threads:[~2026-11-15 16:10 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 19:54 pavankumaryalagada
2026-11-15 16:09 ` Julian Anastasov [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8473d38d-2d50-ad77-18c1-29bbdda56616@ssi.bg \
--to=ja@ssi.bg \
--cc=coreteam@netfilter.org \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=fw@strlen.de \
--cc=horms@verge.net.au \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lvs-devel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pablo@netfilter.org \
--cc=pavankumaryalagada@gmail.com \
--cc=phil@nwl.cc \
--cc=skhan@linuxfoundation.org \
--cc=stable@vger.kernel.org \
--cc=syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com \
--cc=weiwan@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®