mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Florian Fainelli <florian.fainelli@broadcom.com>
To: netdev-bot+sashiko@kernel.org
Cc: netdev@vger.kernel.org, rafal@milecki.pl,
	bcm-kernel-feedback-list@broadcom.com, andrew+netdev@lunn.ch,
	davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	pabeni@redhat.com, robh@kernel.org, krzk+dt@kernel.org,
	conor+dt@kernel.org, horms@kernel.org, corbet@lwn.net,
	jonas.gorski@gmail.com, olteanv@gmail.com, pablo@netfilter.org,
	fw@strlen.de, phil@nwl.cc, devicetree@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
	netfilter-devel@vger.kernel.org, coreteam@netfilter.org
Subject: Re: [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes
Date: Tue, 17 Nov 2026 15:57:23 -0800	[thread overview]
Message-ID: <85be0d5e-918c-48cc-999a-3193154ecfad@broadcom.com> (raw)
In-Reply-To: <179495815956.3136.7417407587662864290@kernel.org>

On 11/17/26 15:29, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 6 potential
> issue(s) to consider.
> 
> Critical: 0 · High: 0 · Medium: 4 · Low: 2
> 
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
> 
> - [Medium] Moving IPS_OFFLOAD_BIT out of flow_offload_teardown() means
>    the lockless check in nf_flow_table_extend_ct_timeout() can no longer…
> - [Medium] This patch reverses the central change of the commit in its
>    Fixes: tag but does not say so, and it does not mention the later…
> - [Medium] The patch's ownership argument ('For hardware flows, GC
>    reaches this point only after hardware deletion is complete ...…
> - [Medium] The patch now releases IPS_OFFLOAD (re-admission) in
>    flow_offload_del() once NF_FLOW_HW_DEAD is set, and the commit…
> - [Low] The new KUnit test nf_flow_test_hardware_retirement() and the
>    NF_FLOW_TABLE_KUNIT_TEST help text say they check that hash removal…
> - [Low] The new Documentation/networking/nf_flowtable.rst text (and the
>    commit message) list the visible side effects of keeping IPS_OFFLOAD…
> 
> --
> 
> netfilter: flowtable: retain offload ownership until deletion completes
> 
> This patch moves the clearing of IPS_OFFLOAD_BIT from
> flow_offload_teardown() to flow_offload_del(), after both hash entries have
> been removed. The goal is that a replacement flow can't be admitted on the
> same conntrack while the old owner's cleanup is still pending. It also
> documents the transient status and adds KUnit tests for the ownership
> lifetime.
> 
>> diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst
>> index d757c21c10f2d..b9b4444357758 100644
>> --- a/Documentation/networking/nf_flowtable.rst
>> +++ b/Documentation/networking/nf_flowtable.rst
>> @@ -205,6 +205,19 @@ refers to the software offload mode, so there is a distinction between [OFFLOAD]
>>   which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers
>>   to the hardware offload datapath being used by the flow.
>>   
>> +During teardown, software fastpath lookups immediately stop matching the
>> +flow. Packets reaching software follow the classic forwarding path, while
>> +hardware deletion may still be pending. The OFFLOAD status
>> +bit remains set until hash removal and asynchronous hardware deletion are
>> +complete, preventing a new owner from racing the old owner's cleanup. As a
>> +result, conntrack dumps can temporarily retain [OFFLOAD] or [HW_OFFLOAD]
>> +and omit the TCP state after a FIN or RST has returned to the slow path.
> 
> [Severity: Low]
> Is this list of side effects complete? Several in-tree users of
> IPS_OFFLOAD_BIT now see the bit set for the whole GC and async DESTROY
> window.
> 
> ctnetlink_dump_info() skips the timeout as well as the protoinfo:
> 
> net/netfilter/nf_conntrack_netlink.c:ctnetlink_dump_info() {
>      ...
> 	if (!test_bit(IPS_OFFLOAD_BIT, &ct->status) &&
> 	    (ctnetlink_dump_timeout(skb, ct, false) < 0 ||
> 	     ctnetlink_dump_protoinfo(skb, ct, false) < 0))
>      ...
> }
> 
> ct_seq_show() hides the expires field in the same way. As a result, the
> repaired timeout mentioned in the next paragraph can't be seen during this
> interval.
> 
> nf_nat_used_tuple_harder() also reads the bit as "entry is offloaded":
> 
> net/netfilter/nf_nat_core.c:nf_nat_used_tuple_harder() {
>      ...
> 	static const unsigned long flags_offload = IPS_OFFLOAD | IPS_HW_OFFLOAD;
>      ...
> 	if (nf_ct_kill(ct))
> 		taken = flags & flags_offload;
>      ...
> }
> 
> A torn-down TCP entry in CLOSE or TIME_WAIT that has gone back to the slow
> path would now be evicted but still reported as taken.
> 
> The UAPI comment for IPS_OFFLOAD_BIT in nf_conntrack_common.h still says
> "Conntrack has been offloaded to flow table", but this text redefines the
> bit as ownership pending cleanup.
> 
> Should these effects be documented here too?
> 
> [ ... ]
> 
>> diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c
>> index 2aa2647ed9df0..4c82097cd50bc 100644
>> --- a/net/netfilter/nf_flow_table_core.c
>> +++ b/net/netfilter/nf_flow_table_core.c
>> @@ -393,12 +393,15 @@ static void flow_offload_del(struct nf_flowtable *flow_table,
>>   	rhashtable_remove_fast(&flow_table->rhashtable,
>>   			       &flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].node,
>>   			       nf_flow_offload_rhash_params);
>> +	/* Release admission only after both hashes and async hardware state
>> +	 * have gone. An older owner must not clear its replacement's status.
>> +	 */
>> +	clear_bit_unlock(IPS_OFFLOAD_BIT, &flow->ct->status);
> 
> [Severity: Medium]
> The commit message says "For hardware flows, GC reaches this point only
> after hardware deletion is complete" and "Holding the admission bit
> prevents overlapping owners".
> 
> Can a datapath reader still queue a FLOW_CLS_REPLACE for this flow after
> its DESTROY has completed?
> 
> flow_offload_lookup() checks NF_FLOW_TEARDOWN once. The later
> flow_offload_refresh() only looks at NF_FLOW_CLOSING and NF_FLOW_HW:
> 
> net/netfilter/nf_flow_table_core.c:flow_offload_refresh() {
>      ...
> 	if (likely(!nf_flowtable_hw_offload(flow_table)) ||
> 	    test_bit(NF_FLOW_CLOSING, &flow->flags))
> 		return;
> 
> 	if (test_bit(NF_FLOW_HW, &flow->flags))
> 		nf_flow_offload_refresh(flow_table, flow);
> }
> 
> nf_flow_offload_work_alloc() is gated only by NF_FLOW_PENDING.
> flow_offload_work_handler() clears that bit after flow_offload_work_del()
> has set NF_FLOW_HW_DEAD. One possible sequence:
> 
> reader                      gc / offload work
> flow_offload_lookup()
>    TEARDOWN not set
>                              flow_offload_teardown()
>                              nf_flow_offload_del()
>                              flow_offload_work_del()
>                                set_bit(NF_FLOW_HW_DEAD)
>                              NF_FLOW_PENDING cleared
> flow_offload_refresh()
>    nf_flow_offload_refresh()
>      REPLACE queued
>                              flow_offload_work_add()
>                                set_bit(IPS_HW_OFFLOAD_BIT)
>                              flow_offload_del()
>                                clear_bit_unlock(IPS_OFFLOAD_BIT)
>                                flow_offload_free()
> 
> After this, the re-installed hardware rule is keyed by a freed cookie, and
> IPS_HW_OFFLOAD stays set on the ct. nft_flow_offload_eval() or act_ct can
> then admit a replacement owner through test_and_set_bit(IPS_OFFLOAD_BIT).
> 
> With PREEMPT_RCU or PREEMPT_RT, the reader's RCU section could span
> flow_offload_del(). Could the queued REPLACE work then use the flow after
> flow_offload_free_rcu() has freed it?
> 
> The lookup/refresh window existed before this patch, but the new ownership
> argument depends on it being closed. Later patches in the series don't
> touch flow_offload_refresh() or nf_flow_offload_refresh().
> 
> [Severity: Medium]
> Does NF_FLOW_HW_DEAD really mean the device removed the rule?
> flow_offload_work_del() sets it unconditionally:
> 
> net/netfilter/nf_flow_table_offload.c:flow_offload_work_del() {
> 	clear_bit(IPS_HW_OFFLOAD_BIT, &offload->flow->ct->status);
> 	flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_ORIGINAL);
> 	if (test_bit(NF_FLOW_HW_BIDIRECTIONAL, &offload->flow->flags))
> 		flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_REPLY);
> 	set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags);
> }
> 
> flow_offload_tuple_del() throws away the return value of
> nf_flow_offload_tuple(..., FLOW_CLS_DESTROY, ...), and
> nf_flow_offload_tuple() skips callbacks that return errors.
> 
> A driver's DESTROY can fail and leave the rule programmed, for example
> bnxt_tc_del_flow() after a firmware HWRM failure. In that case
> flow_offload_del() still releases IPS_OFFLOAD and frees the flow, and the
> flow's tuplehash[dir].tuple is the driver cookie.
> 
> Could a replacement owner then program a second rule for the same tuple
> while the stale rule, with the old route or NAT, stays in hardware with no
> way to delete it?
> 
> The ignored DESTROY result predates this patch. However, the ownership
> serialization described in the commit message depends on deletion having
> completed.
> 
>>   	flow_offload_free(flow);
>>   }
>>   
>>   void flow_offload_teardown(struct flow_offload *flow)
>>   {
>> -	clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status);
>>   	if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags))
>>   		flow_offload_fixup_ct(flow);
>>   }
> 
> [Severity: Medium]
> Without this clear_bit(), can the flowtable GC overwrite the timeout that
> flow_offload_fixup_ct() just repaired?
> 
> nf_flow_offload_gc_step() samples NF_FLOW_TEARDOWN once and doesn't check
> it again before calling nf_flow_table_extend_ct_timeout(). The only guard
> left in the extend helper is the IPS_OFFLOAD_BIT re-test:
> 
> net/netfilter/nf_flow_table_core.c:nf_flow_table_extend_ct_timeout() {
> 	u32 ct_timeout = READ_ONCE(ct->timeout);
>      ...
> 	if (!refcount_inc_not_zero(&ct->ct_general.use))
> 		return;
> 
> 	/* load ct->status after refcount increase */
> 	smp_acquire__after_ctrl_dep();
> 
> 	if (nf_ct_is_confirmed(ct) &&
> 	    test_bit(IPS_OFFLOAD_BIT, &ct->status)) {
>      ...
> 			cmpxchg(&ct->timeout, ct_timeout, new_timeout);
>      ...
> }
> 
> That re-test now always passes until flow_offload_del() runs. Consider:
> 
> CPU0                                CPU1
> nf_flow_offload_gc_step()
>    teardown = false
>                                      flow_offload_teardown()
>                                        set NF_FLOW_TEARDOWN
>                                        flow_offload_fixup_ct()
>                                          nf_ct_refresh()
> nf_flow_table_extend_ct_timeout()
>    READ_ONCE(ct->timeout)
>      /* repaired, below 5 min */
>    test_bit(IPS_OFFLOAD_BIT)
>      /* still set */
>    cmpxchg(&ct->timeout, ct_timeout,
>            now + NF_CT_DAY)
> 
> CPU1 can be the datapath after a failed dst check or a missing outdev, act_ct
> on FIN/RST, or nf_flow_table_cleanup() on netdev events.
> 
> The refcount, smp_acquire__after_ctrl_dep() and re-test sequence from
> commit 03428ca5cee9 depends on teardown clearing the bit before
> flow_offload_fixup_ct() repairs ct->timeout.
> 
> flow_offload_del() doesn't repair the timeout again. A UDP flow with no
> further packets would therefore keep its conntrack entry for NF_CT_DAY
> instead of seconds or minutes.
> 
> Doesn't this contradict "TCP state and timeout repair run at first
> teardown" in the commit message? Later patches in the series leave
> flow_offload_teardown(), nf_flow_offload_gc_step() and
> nf_flow_table_extend_ct_timeout() unchanged.
> 
> [Severity: Medium]
> This undoes the main change of the commit named in the Fixes: tag:
> 
>      Fixes: e5eaac2beb54 ("netfilter: flowtable: fix TCP flow teardown")
> 
> e5eaac2beb54 moved this clear_bit() from flow_offload_del() into
> flow_offload_teardown(). The reason was that conntrack gc could race to
> undo the packet path's timeout adjustment, leaving the entry with the
> one-day offload timeout.
> 
> Moving it back looks safe on mainline only because commit 03428ca5cee9
> ("netfilter: conntrack: rework offload nf_conn timeout extension logic")
> removed this block from gc_worker():
> 
> 	if (test_bit(IPS_OFFLOAD_BIT, &tmp->status)) {
> 		nf_ct_offload_timeout(tmp);
> 		...
> 		continue;
> 	}
> 
> 03428ca5cee9 has no Fixes: or stable tag. Suppose this patch is backported
> to a stable tree that has e5eaac2beb54 but not 03428ca5cee9.
> 
> Wouldn't gc_worker() then reset ct->timeout to now + NF_CT_DAY during the
> whole window from teardown to flow_offload_del()? flow_offload_del() would
> then clear the bit without another repair. For closed TCP and expired UDP
> entries, that brings back the bug e5eaac2beb54 fixed.
> 
> Could the commit message say that it reverts part of e5eaac2beb54 and
> depends on 03428ca5cee9?
> 
> [ ... ]
> 
>> diff --git a/net/netfilter/nf_flow_table_core_test.c b/net/netfilter/nf_flow_table_core_test.c
>> new file mode 100644
>> index 0000000000000..378a1c60b50b9
>> --- /dev/null
>> +++ b/net/netfilter/nf_flow_table_core_test.c
>> @@ -0,0 +1,86 @@
> 
> [ ... ]
> 
>> +	flow_offload_teardown(flow);
>> +	/* nft_flow_offload must not admit another owner during async DESTROY. */
>> +	KUNIT_EXPECT_TRUE(test, test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status));
>> +	flow_offload_teardown(flow);
>> +	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
>> +	/* Model completed hardware deletion before the GC removes the hashes. */
>> +	clear_bit(IPS_HW_OFFLOAD_BIT, &ct->status);
>> +	set_bit(NF_FLOW_HW_DEAD, &flow->flags);
>> +	rcu_read_lock();
>> +	flow_offload_del(&table, flow);
> 
> [Severity: Low]
> Does this test cover the ordering it describes? The Kconfig help says
> "Require hash and asynchronous hardware deletion before re-enrolment".
> However, the only code that makes deletion wait for hardware removal is in
> nf_flow_offload_gc_step():
> 
> 	if (test_bit(NF_FLOW_HW, &flow->flags)) {
> 		if (!test_bit(NF_FLOW_HW_DYING, &flow->flags))
> 			nf_flow_offload_del(flow_table, flow);
> 		else if (test_bit(NF_FLOW_HW_DEAD, &flow->flags))
> 			flow_offload_del(flow_table, flow);
> 
> This test sets NF_FLOW_HW_DEAD by hand and calls flow_offload_del()
> directly, and flow_offload_del() reads neither flag. If the
> HW_DYING/HW_DEAD gating were removed or broken, neither case would fail.
> 
> Both cases also use IPPROTO_ICMP, so flow_offload_fixup_ct() returns
> immediately. The TCP state and timeout repair at first teardown is never
> exercised. nf_flow_test_teardown_repeat() only checks that
> flow_offload_teardown() no longer touches IPS_OFFLOAD_BIT.
> 
> Later in the series the Kconfig help is reworded to "Test flowtable
> ownership during asynchronous hardware deletion", so the gap seems to
> remain in the final state.
> 

[Medium] extend_ct_timeout() racing with teardown

Agreed. With IPS_OFFLOAD kept until flow_offload_del(), the re-test of
the bit in nf_flow_table_extend_ct_timeout() no longer detects a 
teardown that already repaired ct->timeout. v2 passes the flow to the 
helper and checks NF_FLOW_TEARDOWN after sampling ct->timeout (teardown 
sets the flag before repairing the timeout), before the cmpxchg. A KUnit 
case covers it.

[Medium] Reverts part of e5eaac2beb54, depends on 03428ca5cee9

Agreed. The commit message now says this moves the clear_bit() back to 
flow_offload_del(), that this is only safe since 03428ca5cee9 removed 
the IPS_OFFLOAD handling from gc_worker(), and that the patch must not 
be backported to trees without it.

[Medium] "Hardware deletion is complete"

Agreed on the wording. GC reaches flow_offload_del() once the DESTROY
work has run (NF_FLOW_HW_DEAD), not once the device is known to have 
removed the rule, since the DESTROY result is not propagated. v2 words 
it that way. Making DESTROY failures visible is a separate, pre-existing 
limitation of the hardware offload code and not addressed here.

[Medium] Refresh queued after DESTROY

The lookup/refresh window in flow_offload_refresh() predates this patch
and is independent of when IPS_OFFLOAD is released, so I'd rather not
change it here. I will look at it separately.

[Low] Documentation and KUnit

v2 documents the other effects of the bit remaining set (ctnetlink and
/proc omit the timeout as well as the TCP state, NAT does not reuse the 
tuple), makes the retirement test go through nf_flow_offload_gc_step() 
so the HW_DYING/HW_DEAD gating is covered, and describes in Kconfig only 
what the tests check. The TCP/UDP timeout repair itself is still not
covered by them.

Pre-existing issues from the first review

The use-after-free in flow_offload_work_handler() and the missing GRE
timeout fixup in flow_offload_fixup_ct() are not introduced by this
patch. I will follow up on them separately.

pw-bot: cr
-- 
Florian

  reply	other threads:[~2026-11-17 23:57 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 22:25 [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 1/9] net: ethernet: bgmac: validate RX length against the DMA packet capacity Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-17 23:43     ` Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 2/9] netfilter: flowtable: reject non-host IPv4 packets before lookup Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-17 23:51     ` Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-17 23:57     ` Florian Fainelli [this message]
2026-11-16 22:25 ` [PATCH net-next 4/9] dt-bindings: net: describe the Northstar flow accelerator Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 5/9] net: dsa: add Broadcom FA metadata and IPv4 exception paths Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 6/9] net: flow_offload: allow drivers to validate and invalidate cached paths Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 7/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 8/9] net: add KUnit coverage for FA and flowtable path validation Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 9/9] docs: net: document Northstar FA setup and NAT qualification Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:38 ` [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload netdev-bot+sinfo
2026-11-16 22:45   ` Florian Fainelli

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=85be0d5e-918c-48cc-999a-3193154ecfad@broadcom.com \
    --to=florian.fainelli@broadcom.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=bcm-kernel-feedback-list@broadcom.com \
    --cc=conor+dt@kernel.org \
    --cc=corbet@lwn.net \
    --cc=coreteam@netfilter.org \
    --cc=davem@davemloft.net \
    --cc=devicetree@vger.kernel.org \
    --cc=edumazet@kernel.org \
    --cc=fw@strlen.de \
    --cc=horms@kernel.org \
    --cc=jonas.gorski@gmail.com \
    --cc=krzk+dt@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev-bot+sashiko@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=olteanv@gmail.com \
    --cc=pabeni@redhat.com \
    --cc=pablo@netfilter.org \
    --cc=phil@nwl.cc \
    --cc=rafal@milecki.pl \
    --cc=robh@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®