From: Florian Fainelli <florian.fainelli@broadcom.com>
To: netdev-bot+sashiko@kernel.org
Cc: netdev@vger.kernel.org, rafal@milecki.pl,
bcm-kernel-feedback-list@broadcom.com, andrew+netdev@lunn.ch,
davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
pabeni@redhat.com, robh@kernel.org, krzk+dt@kernel.org,
conor+dt@kernel.org, horms@kernel.org, corbet@lwn.net,
jonas.gorski@gmail.com, olteanv@gmail.com, pablo@netfilter.org,
fw@strlen.de, phil@nwl.cc, devicetree@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
netfilter-devel@vger.kernel.org, coreteam@netfilter.org
Subject: Re: [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes
Date: Tue, 17 Nov 2026 15:57:23 -0800 [thread overview]
Message-ID: <85be0d5e-918c-48cc-999a-3193154ecfad@broadcom.com> (raw)
In-Reply-To: <179495815956.3136.7417407587662864290@kernel.org>
On 11/17/26 15:29, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 6 potential
> issue(s) to consider.
>
> Critical: 0 · High: 0 · Medium: 4 · Low: 2
>
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
>
> - [Medium] Moving IPS_OFFLOAD_BIT out of flow_offload_teardown() means
> the lockless check in nf_flow_table_extend_ct_timeout() can no longer…
> - [Medium] This patch reverses the central change of the commit in its
> Fixes: tag but does not say so, and it does not mention the later…
> - [Medium] The patch's ownership argument ('For hardware flows, GC
> reaches this point only after hardware deletion is complete ...…
> - [Medium] The patch now releases IPS_OFFLOAD (re-admission) in
> flow_offload_del() once NF_FLOW_HW_DEAD is set, and the commit…
> - [Low] The new KUnit test nf_flow_test_hardware_retirement() and the
> NF_FLOW_TABLE_KUNIT_TEST help text say they check that hash removal…
> - [Low] The new Documentation/networking/nf_flowtable.rst text (and the
> commit message) list the visible side effects of keeping IPS_OFFLOAD…
>
> --
>
> netfilter: flowtable: retain offload ownership until deletion completes
>
> This patch moves the clearing of IPS_OFFLOAD_BIT from
> flow_offload_teardown() to flow_offload_del(), after both hash entries have
> been removed. The goal is that a replacement flow can't be admitted on the
> same conntrack while the old owner's cleanup is still pending. It also
> documents the transient status and adds KUnit tests for the ownership
> lifetime.
>
>> diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst
>> index d757c21c10f2d..b9b4444357758 100644
>> --- a/Documentation/networking/nf_flowtable.rst
>> +++ b/Documentation/networking/nf_flowtable.rst
>> @@ -205,6 +205,19 @@ refers to the software offload mode, so there is a distinction between [OFFLOAD]
>> which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers
>> to the hardware offload datapath being used by the flow.
>>
>> +During teardown, software fastpath lookups immediately stop matching the
>> +flow. Packets reaching software follow the classic forwarding path, while
>> +hardware deletion may still be pending. The OFFLOAD status
>> +bit remains set until hash removal and asynchronous hardware deletion are
>> +complete, preventing a new owner from racing the old owner's cleanup. As a
>> +result, conntrack dumps can temporarily retain [OFFLOAD] or [HW_OFFLOAD]
>> +and omit the TCP state after a FIN or RST has returned to the slow path.
>
> [Severity: Low]
> Is this list of side effects complete? Several in-tree users of
> IPS_OFFLOAD_BIT now see the bit set for the whole GC and async DESTROY
> window.
>
> ctnetlink_dump_info() skips the timeout as well as the protoinfo:
>
> net/netfilter/nf_conntrack_netlink.c:ctnetlink_dump_info() {
> ...
> if (!test_bit(IPS_OFFLOAD_BIT, &ct->status) &&
> (ctnetlink_dump_timeout(skb, ct, false) < 0 ||
> ctnetlink_dump_protoinfo(skb, ct, false) < 0))
> ...
> }
>
> ct_seq_show() hides the expires field in the same way. As a result, the
> repaired timeout mentioned in the next paragraph can't be seen during this
> interval.
>
> nf_nat_used_tuple_harder() also reads the bit as "entry is offloaded":
>
> net/netfilter/nf_nat_core.c:nf_nat_used_tuple_harder() {
> ...
> static const unsigned long flags_offload = IPS_OFFLOAD | IPS_HW_OFFLOAD;
> ...
> if (nf_ct_kill(ct))
> taken = flags & flags_offload;
> ...
> }
>
> A torn-down TCP entry in CLOSE or TIME_WAIT that has gone back to the slow
> path would now be evicted but still reported as taken.
>
> The UAPI comment for IPS_OFFLOAD_BIT in nf_conntrack_common.h still says
> "Conntrack has been offloaded to flow table", but this text redefines the
> bit as ownership pending cleanup.
>
> Should these effects be documented here too?
>
> [ ... ]
>
>> diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c
>> index 2aa2647ed9df0..4c82097cd50bc 100644
>> --- a/net/netfilter/nf_flow_table_core.c
>> +++ b/net/netfilter/nf_flow_table_core.c
>> @@ -393,12 +393,15 @@ static void flow_offload_del(struct nf_flowtable *flow_table,
>> rhashtable_remove_fast(&flow_table->rhashtable,
>> &flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].node,
>> nf_flow_offload_rhash_params);
>> + /* Release admission only after both hashes and async hardware state
>> + * have gone. An older owner must not clear its replacement's status.
>> + */
>> + clear_bit_unlock(IPS_OFFLOAD_BIT, &flow->ct->status);
>
> [Severity: Medium]
> The commit message says "For hardware flows, GC reaches this point only
> after hardware deletion is complete" and "Holding the admission bit
> prevents overlapping owners".
>
> Can a datapath reader still queue a FLOW_CLS_REPLACE for this flow after
> its DESTROY has completed?
>
> flow_offload_lookup() checks NF_FLOW_TEARDOWN once. The later
> flow_offload_refresh() only looks at NF_FLOW_CLOSING and NF_FLOW_HW:
>
> net/netfilter/nf_flow_table_core.c:flow_offload_refresh() {
> ...
> if (likely(!nf_flowtable_hw_offload(flow_table)) ||
> test_bit(NF_FLOW_CLOSING, &flow->flags))
> return;
>
> if (test_bit(NF_FLOW_HW, &flow->flags))
> nf_flow_offload_refresh(flow_table, flow);
> }
>
> nf_flow_offload_work_alloc() is gated only by NF_FLOW_PENDING.
> flow_offload_work_handler() clears that bit after flow_offload_work_del()
> has set NF_FLOW_HW_DEAD. One possible sequence:
>
> reader gc / offload work
> flow_offload_lookup()
> TEARDOWN not set
> flow_offload_teardown()
> nf_flow_offload_del()
> flow_offload_work_del()
> set_bit(NF_FLOW_HW_DEAD)
> NF_FLOW_PENDING cleared
> flow_offload_refresh()
> nf_flow_offload_refresh()
> REPLACE queued
> flow_offload_work_add()
> set_bit(IPS_HW_OFFLOAD_BIT)
> flow_offload_del()
> clear_bit_unlock(IPS_OFFLOAD_BIT)
> flow_offload_free()
>
> After this, the re-installed hardware rule is keyed by a freed cookie, and
> IPS_HW_OFFLOAD stays set on the ct. nft_flow_offload_eval() or act_ct can
> then admit a replacement owner through test_and_set_bit(IPS_OFFLOAD_BIT).
>
> With PREEMPT_RCU or PREEMPT_RT, the reader's RCU section could span
> flow_offload_del(). Could the queued REPLACE work then use the flow after
> flow_offload_free_rcu() has freed it?
>
> The lookup/refresh window existed before this patch, but the new ownership
> argument depends on it being closed. Later patches in the series don't
> touch flow_offload_refresh() or nf_flow_offload_refresh().
>
> [Severity: Medium]
> Does NF_FLOW_HW_DEAD really mean the device removed the rule?
> flow_offload_work_del() sets it unconditionally:
>
> net/netfilter/nf_flow_table_offload.c:flow_offload_work_del() {
> clear_bit(IPS_HW_OFFLOAD_BIT, &offload->flow->ct->status);
> flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_ORIGINAL);
> if (test_bit(NF_FLOW_HW_BIDIRECTIONAL, &offload->flow->flags))
> flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_REPLY);
> set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags);
> }
>
> flow_offload_tuple_del() throws away the return value of
> nf_flow_offload_tuple(..., FLOW_CLS_DESTROY, ...), and
> nf_flow_offload_tuple() skips callbacks that return errors.
>
> A driver's DESTROY can fail and leave the rule programmed, for example
> bnxt_tc_del_flow() after a firmware HWRM failure. In that case
> flow_offload_del() still releases IPS_OFFLOAD and frees the flow, and the
> flow's tuplehash[dir].tuple is the driver cookie.
>
> Could a replacement owner then program a second rule for the same tuple
> while the stale rule, with the old route or NAT, stays in hardware with no
> way to delete it?
>
> The ignored DESTROY result predates this patch. However, the ownership
> serialization described in the commit message depends on deletion having
> completed.
>
>> flow_offload_free(flow);
>> }
>>
>> void flow_offload_teardown(struct flow_offload *flow)
>> {
>> - clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status);
>> if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags))
>> flow_offload_fixup_ct(flow);
>> }
>
> [Severity: Medium]
> Without this clear_bit(), can the flowtable GC overwrite the timeout that
> flow_offload_fixup_ct() just repaired?
>
> nf_flow_offload_gc_step() samples NF_FLOW_TEARDOWN once and doesn't check
> it again before calling nf_flow_table_extend_ct_timeout(). The only guard
> left in the extend helper is the IPS_OFFLOAD_BIT re-test:
>
> net/netfilter/nf_flow_table_core.c:nf_flow_table_extend_ct_timeout() {
> u32 ct_timeout = READ_ONCE(ct->timeout);
> ...
> if (!refcount_inc_not_zero(&ct->ct_general.use))
> return;
>
> /* load ct->status after refcount increase */
> smp_acquire__after_ctrl_dep();
>
> if (nf_ct_is_confirmed(ct) &&
> test_bit(IPS_OFFLOAD_BIT, &ct->status)) {
> ...
> cmpxchg(&ct->timeout, ct_timeout, new_timeout);
> ...
> }
>
> That re-test now always passes until flow_offload_del() runs. Consider:
>
> CPU0 CPU1
> nf_flow_offload_gc_step()
> teardown = false
> flow_offload_teardown()
> set NF_FLOW_TEARDOWN
> flow_offload_fixup_ct()
> nf_ct_refresh()
> nf_flow_table_extend_ct_timeout()
> READ_ONCE(ct->timeout)
> /* repaired, below 5 min */
> test_bit(IPS_OFFLOAD_BIT)
> /* still set */
> cmpxchg(&ct->timeout, ct_timeout,
> now + NF_CT_DAY)
>
> CPU1 can be the datapath after a failed dst check or a missing outdev, act_ct
> on FIN/RST, or nf_flow_table_cleanup() on netdev events.
>
> The refcount, smp_acquire__after_ctrl_dep() and re-test sequence from
> commit 03428ca5cee9 depends on teardown clearing the bit before
> flow_offload_fixup_ct() repairs ct->timeout.
>
> flow_offload_del() doesn't repair the timeout again. A UDP flow with no
> further packets would therefore keep its conntrack entry for NF_CT_DAY
> instead of seconds or minutes.
>
> Doesn't this contradict "TCP state and timeout repair run at first
> teardown" in the commit message? Later patches in the series leave
> flow_offload_teardown(), nf_flow_offload_gc_step() and
> nf_flow_table_extend_ct_timeout() unchanged.
>
> [Severity: Medium]
> This undoes the main change of the commit named in the Fixes: tag:
>
> Fixes: e5eaac2beb54 ("netfilter: flowtable: fix TCP flow teardown")
>
> e5eaac2beb54 moved this clear_bit() from flow_offload_del() into
> flow_offload_teardown(). The reason was that conntrack gc could race to
> undo the packet path's timeout adjustment, leaving the entry with the
> one-day offload timeout.
>
> Moving it back looks safe on mainline only because commit 03428ca5cee9
> ("netfilter: conntrack: rework offload nf_conn timeout extension logic")
> removed this block from gc_worker():
>
> if (test_bit(IPS_OFFLOAD_BIT, &tmp->status)) {
> nf_ct_offload_timeout(tmp);
> ...
> continue;
> }
>
> 03428ca5cee9 has no Fixes: or stable tag. Suppose this patch is backported
> to a stable tree that has e5eaac2beb54 but not 03428ca5cee9.
>
> Wouldn't gc_worker() then reset ct->timeout to now + NF_CT_DAY during the
> whole window from teardown to flow_offload_del()? flow_offload_del() would
> then clear the bit without another repair. For closed TCP and expired UDP
> entries, that brings back the bug e5eaac2beb54 fixed.
>
> Could the commit message say that it reverts part of e5eaac2beb54 and
> depends on 03428ca5cee9?
>
> [ ... ]
>
>> diff --git a/net/netfilter/nf_flow_table_core_test.c b/net/netfilter/nf_flow_table_core_test.c
>> new file mode 100644
>> index 0000000000000..378a1c60b50b9
>> --- /dev/null
>> +++ b/net/netfilter/nf_flow_table_core_test.c
>> @@ -0,0 +1,86 @@
>
> [ ... ]
>
>> + flow_offload_teardown(flow);
>> + /* nft_flow_offload must not admit another owner during async DESTROY. */
>> + KUNIT_EXPECT_TRUE(test, test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status));
>> + flow_offload_teardown(flow);
>> + KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
>> + /* Model completed hardware deletion before the GC removes the hashes. */
>> + clear_bit(IPS_HW_OFFLOAD_BIT, &ct->status);
>> + set_bit(NF_FLOW_HW_DEAD, &flow->flags);
>> + rcu_read_lock();
>> + flow_offload_del(&table, flow);
>
> [Severity: Low]
> Does this test cover the ordering it describes? The Kconfig help says
> "Require hash and asynchronous hardware deletion before re-enrolment".
> However, the only code that makes deletion wait for hardware removal is in
> nf_flow_offload_gc_step():
>
> if (test_bit(NF_FLOW_HW, &flow->flags)) {
> if (!test_bit(NF_FLOW_HW_DYING, &flow->flags))
> nf_flow_offload_del(flow_table, flow);
> else if (test_bit(NF_FLOW_HW_DEAD, &flow->flags))
> flow_offload_del(flow_table, flow);
>
> This test sets NF_FLOW_HW_DEAD by hand and calls flow_offload_del()
> directly, and flow_offload_del() reads neither flag. If the
> HW_DYING/HW_DEAD gating were removed or broken, neither case would fail.
>
> Both cases also use IPPROTO_ICMP, so flow_offload_fixup_ct() returns
> immediately. The TCP state and timeout repair at first teardown is never
> exercised. nf_flow_test_teardown_repeat() only checks that
> flow_offload_teardown() no longer touches IPS_OFFLOAD_BIT.
>
> Later in the series the Kconfig help is reworded to "Test flowtable
> ownership during asynchronous hardware deletion", so the gap seems to
> remain in the final state.
>
[Medium] extend_ct_timeout() racing with teardown
Agreed. With IPS_OFFLOAD kept until flow_offload_del(), the re-test of
the bit in nf_flow_table_extend_ct_timeout() no longer detects a
teardown that already repaired ct->timeout. v2 passes the flow to the
helper and checks NF_FLOW_TEARDOWN after sampling ct->timeout (teardown
sets the flag before repairing the timeout), before the cmpxchg. A KUnit
case covers it.
[Medium] Reverts part of e5eaac2beb54, depends on 03428ca5cee9
Agreed. The commit message now says this moves the clear_bit() back to
flow_offload_del(), that this is only safe since 03428ca5cee9 removed
the IPS_OFFLOAD handling from gc_worker(), and that the patch must not
be backported to trees without it.
[Medium] "Hardware deletion is complete"
Agreed on the wording. GC reaches flow_offload_del() once the DESTROY
work has run (NF_FLOW_HW_DEAD), not once the device is known to have
removed the rule, since the DESTROY result is not propagated. v2 words
it that way. Making DESTROY failures visible is a separate, pre-existing
limitation of the hardware offload code and not addressed here.
[Medium] Refresh queued after DESTROY
The lookup/refresh window in flow_offload_refresh() predates this patch
and is independent of when IPS_OFFLOAD is released, so I'd rather not
change it here. I will look at it separately.
[Low] Documentation and KUnit
v2 documents the other effects of the bit remaining set (ctnetlink and
/proc omit the timeout as well as the TCP state, NAT does not reuse the
tuple), makes the retirement test go through nf_flow_offload_gc_step()
so the HW_DYING/HW_DEAD gating is covered, and describes in Kconfig only
what the tests check. The TCP/UDP timeout repair itself is still not
covered by them.
Pre-existing issues from the first review
The use-after-free in flow_offload_work_handler() and the missing GRE
timeout fixup in flow_offload_fixup_ct() are not introduced by this
patch. I will follow up on them separately.
pw-bot: cr
--
Florian
next prev parent reply other threads:[~2026-11-17 23:57 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 22:25 [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 1/9] net: ethernet: bgmac: validate RX length against the DMA packet capacity Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-17 23:43 ` Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 2/9] netfilter: flowtable: reject non-host IPv4 packets before lookup Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-17 23:51 ` Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-17 23:57 ` Florian Fainelli [this message]
2026-11-16 22:25 ` [PATCH net-next 4/9] dt-bindings: net: describe the Northstar flow accelerator Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 5/9] net: dsa: add Broadcom FA metadata and IPv4 exception paths Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 6/9] net: flow_offload: allow drivers to validate and invalidate cached paths Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 7/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 8/9] net: add KUnit coverage for FA and flowtable path validation Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 9/9] docs: net: document Northstar FA setup and NAT qualification Florian Fainelli
2026-11-17 23:29 ` netdev-bot+sashiko
2026-11-16 22:38 ` [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload netdev-bot+sinfo
2026-11-16 22:45 ` Florian Fainelli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=85be0d5e-918c-48cc-999a-3193154ecfad@broadcom.com \
--to=florian.fainelli@broadcom.com \
--cc=andrew+netdev@lunn.ch \
--cc=bcm-kernel-feedback-list@broadcom.com \
--cc=conor+dt@kernel.org \
--cc=corbet@lwn.net \
--cc=coreteam@netfilter.org \
--cc=davem@davemloft.net \
--cc=devicetree@vger.kernel.org \
--cc=edumazet@kernel.org \
--cc=fw@strlen.de \
--cc=horms@kernel.org \
--cc=jonas.gorski@gmail.com \
--cc=krzk+dt@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev-bot+sashiko@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
--cc=olteanv@gmail.com \
--cc=pabeni@redhat.com \
--cc=pablo@netfilter.org \
--cc=phil@nwl.cc \
--cc=rafal@milecki.pl \
--cc=robh@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®