mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Julian Anastasov <ja@ssi.bg>
To: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
Cc: Simon Horman <horms@verge.net.au>,
	pablo@netfilter.org, fw@strlen.de, phil@nwl.cc,
	"David S. Miller" <davem@davemloft.net>,
	edumazet@kernel.org, kuba@kernel.org, pabeni@redhat.com,
	netdev@vger.kernel.org, lvs-devel@vger.kernel.org,
	netfilter-devel@vger.kernel.org, coreteam@netfilter.org,
	linux-kernel <linux-kernel@vger.kernel.org>,
	skhan@linuxfoundation.org,
	syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com,
	stable@vger.kernel.org, Wei Wang <weiwan@google.com>
Subject: Re: [PATCH net] ipvs: wait for RCU callbacks before dst_ops teardown
Date: Sun, 15 Nov 2026 18:09:56 +0200 (EET)	[thread overview]
Message-ID: <8473d38d-2d50-ad77-18c1-29bbdda56616@ssi.bg> (raw)
In-Reply-To: <20261114195410.96503-1-pavankumaryalagada@gmail.com>


	Hello,

	Adding Wei Wang to the CC list...

On Sun, 15 Nov 2026, pavankumaryalagada@gmail.com wrote:

> From: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
> 
> During network namespace cleanup, IPVS releases cached destinations
> using RCU callbacks. However, ip6_route_net_exit() destroys the
> per-CPU destination counters before these callbacks complete.
> The RCU callbacks call dst_release(), which updates the destroyed
> counters and causes a NULL pointer dereference in
> percpu_counter_add_batch().
> 
> Call rcu_barrier() at the end of __ip_vs_cleanup_batch() to make
> sure all pending IPVS RCU callbacks finish before the destination
> counters are destroyed.

	What you see is a result of more fundamental problem.
IPVS is supposed to catch all NETDEV_DOWN events coming from the
pernet device phase which come before the pernet subsys phase.
So, on cleanup_net IPVS is supposed to run its subsys cleanup
with no traffic, no devs, no cached dsts, no netfilter hooks, etc.

	But in my tests free_fib_info_rcu -> fib_nh_release ->
fib_nh_common_release -> rt_fibinfo_free_cpus calls
dst_dev_put() where the dst device that IPVS listens for is replaced
with a blackhole_netdev. This happens before the NETDEV_DOWN
event and as result IPVS misses the event because dst->dev is
already replaced and does not match. When such dst_cache
remains for the pernet subsys cleanup you see these crashes
because IPVS has another problem that it does not call
dst_release immediately but from RCU callback.

	I now created a change to call dst_release immediately
but the problem with the blackholing before NETDEV_DOWN remains.
It can hurt other event listeners.

	My tests were with tools/testing/selftests/net/netfilter/ipvs.sh
where I modify it to run only the NAT test. What I see is veth12
to be blackholed before its NETDEV_DOWN event and ip_vs_forget_dev()
missed it. Trying to find more information...

> 
> Reported-by: syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com
> Link: https://lore.kernel.org/all/6af795e2.98e60839.1e023a.00d4.GAE@google.com/T/
> Closes: https://syzkaller.appspot.com/bug?extid=84e27adb95fd2a1fe9bb
> Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yalagada Pavan Kumar <pavankumaryalagada@gmail.com>
> ---
>  net/netfilter/ipvs/ip_vs_core.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index 8a3c880fdb26..9c456efe13c7 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -2553,6 +2553,11 @@ static void __net_exit __ip_vs_cleanup_batch(struct list_head *net_list)
>  		IP_VS_DBG(2, "ipvs netns %d released\n", ipvs->gen);
>  		net->ipvs = NULL;
>  	}
> +
> +	/*
> +	 * Wait for ip_vs_dest_dst_rcu_free() callbacks before dst_ops teardown
> +	 */
> +	rcu_barrier();
>  }
>  
>  static void __net_exit __ip_vs_dev_cleanup_batch(struct list_head *net_list)
> -- 
> 2.43.0

Regards

--
Julian Anastasov <ja@ssi.bg>


      reply	other threads:[~2026-11-15 16:10 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-14 19:54 pavankumaryalagada
2026-11-15 16:09 ` Julian Anastasov [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8473d38d-2d50-ad77-18c1-29bbdda56616@ssi.bg \
    --to=ja@ssi.bg \
    --cc=coreteam@netfilter.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=fw@strlen.de \
    --cc=horms@verge.net.au \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lvs-devel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=pablo@netfilter.org \
    --cc=pavankumaryalagada@gmail.com \
    --cc=phil@nwl.cc \
    --cc=skhan@linuxfoundation.org \
    --cc=stable@vger.kernel.org \
    --cc=syzbot+84e27adb95fd2a1fe9bb@syzkaller.appspotmail.com \
    --cc=weiwan@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®