mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement
@ 2026-11-18 13:54 Jingxuan Xie
  2026-11-18 23:10 ` Willem de Bruijn
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Jingxuan Xie @ 2026-11-18 13:54 UTC (permalink / raw)
  To: netdev
  Cc: Willem de Bruijn, 韩少阳,
	David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Simon Horman, Sebastian Andrzej Siewior, Clark Williams,
	Steven Rostedt, John Ogness, linux-kernel

TPACKET_V3 retires a full or expired block while holding
sk_receive_queue.lock. prb_retire_current_block() takes
blk_fill_in_prog_lock for write to wait for in-flight skb_copy_bits()
before publishing the block.

On PREEMPT_RT a read lock can be held by a task that has been
preempted or is running on another CPU. The packet path or the
retirement timer then blocks on the write lock while holding
sk_receive_queue.lock, and the receive path resumes only after the
copy completes. This was observed as a receive stall of hundreds of
milliseconds in a PREEMPT_RT deployment; manual inspection found the
same unconditional write_lock() in both the full-block and the timer
retirement paths.

Try to take the write lock without blocking on PREEMPT_RT. If a copy
is still in progress, leave the current block and its metadata
untouched:

- the retirement timer re-arms itself and retries after the configured
  interval;

- packet allocation returns NULL, so tpacket_rcv() drops this capture
  record and accounts it in tp_drops, while the original packet remains
  in the network stack;

- a later packet can retire the block once the copier has finished.

Do not dispatch the next block while the current block is still open.
The existing lock order and completion ordering are preserved, and no
new state or worker is introduced. The trade-off is a higher capture
drop rate while a copy is in progress; original traffic is unaffected.

Non-PREEMPT_RT kernels keep the unconditional write_lock(): there
read_lock() disables preemption, so the writer only waits for the copy
to finish, not for a scheduled-out task to run again.

The stall was reproduced on QEMU arm64 SMP PREEMPT_RT by delaying a
copier after frame reservation: without this change the next sender
blocks until the copier is released, with this change only that
capture record is dropped.

Fixes: 632ca50f2cbd ("af_packet: TPACKET_V3: replace busy-wait loop")
Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>
---
Changes in v6:
- Repost without the linux-rt-devel copy: the sender's mail gateway
  duplicated that copy with a confidentiality footer and lost the
  patch's tabs, so it did not apply. No code change.

Changes in v5:
- Rebase onto net/main; no functional change, so the v4 Reviewed-by
  still applies.

Changes in v4:
- Limit the write_trylock() switch to PREEMPT_RT; non-PREEMPT_RT
  kernels keep the blocking write lock (Willem).
- Leave the existing indentation of prb_retire_current_block()
  unchanged (Willem).
- Add the discovery and reproduction information requested by
  netdev-bot.

v3: https://lore.kernel.org/r/20261103092541.273847-1-jingxuanxie@deeproute.ai
v4: https://lore.kernel.org/r/20261110072221.24701-1-jingxuanxie@deeproute.ai
v5: https://lore.kernel.org/r/20261113023809.24730-1-jingxuanxie@deeproute.ai

Testing:
- arm64 W=1 af_packet.o built with PREEMPT_RT=y and PREEMPT_RT=n.
- QEMU arm64 SMP PREEMPT_RT, 3 boots: psock_tpacket and psock_fanout
  exit 0 (the fanout queue-length warning also appears on the
  baseline); with a copier delayed in skb_copy_bits() while holding
  blk_fill_in_prog_lock, a second sender returns in 0.7-1.8 ms and only
  its capture record is dropped; the retirement timer retries and
  publishes the block after the copier is released; close/unmap during
  the delay does not hang; no lockdep or atomic-sleep splats.
- QEMU arm64 SMP PREEMPT (no RT) with the same delay: the second
  sender blocks until the copier is released (503 ms) and no capture
  record is dropped, matching the unpatched kernel.
 net/packet/af_packet.c | 40 ++++++++++++++++++++++++++++++----------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 7322256..e0ebd06 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -200,7 +200,7 @@ static void packet_increment_head(struct packet_ring_buffer *buff);
 static int prb_curr_blk_in_use(struct tpacket_block_desc *);
 static void *prb_dispatch_next_block(struct tpacket_kbdq_core *,
 			struct packet_sock *);
-static void prb_retire_current_block(struct tpacket_kbdq_core *,
+static bool prb_retire_current_block(struct tpacket_kbdq_core *,
 		struct packet_sock *, unsigned int status);
 static int prb_queue_frozen(struct tpacket_kbdq_core *);
 static void prb_open_block(struct tpacket_kbdq_core *,
@@ -677,6 +677,24 @@ static void init_prb_bdqc(struct packet_sock *po,
 	prb_open_block(p1, pbd);
 }
 
+/*
+ * Wait for an in-flight skb_copy_bits() before retiring the current block.
+ * On PREEMPT_RT the read lock can be held by a task that has been preempted
+ * or is running on another CPU, so do not block while holding
+ * sk_receive_queue.lock; return false and let the caller retry.
+ */
+static bool prb_wait_for_copy_bits(struct tpacket_kbdq_core *pkc)
+{
+	if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
+		if (!write_trylock(&pkc->blk_fill_in_prog_lock))
+			return false;
+	} else {
+		write_lock(&pkc->blk_fill_in_prog_lock);
+	}
+	write_unlock(&pkc->blk_fill_in_prog_lock);
+	return true;
+}
+
 /*
  * With a 1MB block-size, on a 1Gbps line, it will take
  * i) ~8 ms to fill a block + ii) memcpy etc.
@@ -711,9 +729,9 @@ static enum hrtimer_restart prb_retire_rx_blk_timer_expired(struct hrtimer *t)
 	 *
 	 */
 	if (BLOCK_NUM_PKTS(pbd)) {
-		/* Waiting for skb_copy_bits to finish... */
-		write_lock(&pkc->blk_fill_in_prog_lock);
-		write_unlock(&pkc->blk_fill_in_prog_lock);
+		/* Retry later if a packet copy is still in progress. */
+		if (!prb_wait_for_copy_bits(pkc))
+			goto refresh_timer;
 	}
 
 	if (!frozen) {
@@ -738,6 +756,7 @@ static enum hrtimer_restart prb_retire_rx_blk_timer_expired(struct hrtimer *t)
 		}
 	}
 
+refresh_timer:
 	hrtimer_forward_now(&pkc->retire_blk_timer, pkc->interval_ktime);
 	spin_unlock(&po->sk.sk_receive_queue.lock);
 	return HRTIMER_RESTART;
@@ -946,7 +965,7 @@ static void *prb_dispatch_next_block(struct tpacket_kbdq_core *pkc,
 	return (void *)pkc->nxt_offset;
 }
 
-static void prb_retire_current_block(struct tpacket_kbdq_core *pkc,
+static bool prb_retire_current_block(struct tpacket_kbdq_core *pkc,
 		struct packet_sock *po, unsigned int status)
 {
 	struct tpacket_block_desc *pbd = GET_CURR_PBLOCK_DESC_FROM_CORE(pkc);
@@ -963,13 +982,13 @@ static void prb_retire_current_block(struct tpacket_kbdq_core *pkc,
 		 * the timer-handler already handled this case.
 		 */
 		if (!(status & TP_STATUS_BLK_TMO)) {
-			/* Waiting for skb_copy_bits to finish... */
-			write_lock(&pkc->blk_fill_in_prog_lock);
-			write_unlock(&pkc->blk_fill_in_prog_lock);
+			/* Do not wait for copies while holding the queue lock. */
+			if (!prb_wait_for_copy_bits(pkc))
+				return false;
 		}
 		prb_close_block(pkc, pbd, po, status);
-		return;
 	}
+	return true;
 }
 
 static int prb_curr_blk_in_use(struct tpacket_block_desc *pbd)
@@ -1097,7 +1116,8 @@ static void *__packet_lookup_frame_in_block(struct packet_sock *po,
 	}
 
 	/* Ok, close the current block */
-	prb_retire_current_block(pkc, po, 0);
+	if (!prb_retire_current_block(pkc, po, 0))
+		return NULL;
 
 	/* Now, try to dispatch the next block */
 	curr = (char *)prb_dispatch_next_block(pkc, po);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement
  2026-11-18 13:54 [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement Jingxuan Xie
@ 2026-11-18 23:10 ` Willem de Bruijn
  2026-11-20 18:40 ` patchwork-bot+netdevbpf
  2026-11-22 14:04 ` netdev-bot+sashiko
  2 siblings, 0 replies; 4+ messages in thread
From: Willem de Bruijn @ 2026-11-18 23:10 UTC (permalink / raw)
  To: Jingxuan Xie, netdev
  Cc: Willem de Bruijn, 韩少阳,
	David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Simon Horman, Sebastian Andrzej Siewior, Clark Williams,
	Steven Rostedt, John Ogness, linux-kernel

Jingxuan Xie wrote:
> TPACKET_V3 retires a full or expired block while holding
> sk_receive_queue.lock. prb_retire_current_block() takes
> blk_fill_in_prog_lock for write to wait for in-flight skb_copy_bits()
> before publishing the block.
> 
> On PREEMPT_RT a read lock can be held by a task that has been
> preempted or is running on another CPU. The packet path or the
> retirement timer then blocks on the write lock while holding
> sk_receive_queue.lock, and the receive path resumes only after the
> copy completes. This was observed as a receive stall of hundreds of
> milliseconds in a PREEMPT_RT deployment; manual inspection found the
> same unconditional write_lock() in both the full-block and the timer
> retirement paths.
> 
> Try to take the write lock without blocking on PREEMPT_RT. If a copy
> is still in progress, leave the current block and its metadata
> untouched:
> 
> - the retirement timer re-arms itself and retries after the configured
>   interval;
> 
> - packet allocation returns NULL, so tpacket_rcv() drops this capture
>   record and accounts it in tp_drops, while the original packet remains
>   in the network stack;
> 
> - a later packet can retire the block once the copier has finished.
> 
> Do not dispatch the next block while the current block is still open.
> The existing lock order and completion ordering are preserved, and no
> new state or worker is introduced. The trade-off is a higher capture
> drop rate while a copy is in progress; original traffic is unaffected.
> 
> Non-PREEMPT_RT kernels keep the unconditional write_lock(): there
> read_lock() disables preemption, so the writer only waits for the copy
> to finish, not for a scheduled-out task to run again.
> 
> The stall was reproduced on QEMU arm64 SMP PREEMPT_RT by delaying a
> copier after frame reservation: without this change the next sender
> blocks until the copier is released, with this change only that
> capture record is dropped.
> 
> Fixes: 632ca50f2cbd ("af_packet: TPACKET_V3: replace busy-wait loop")
> Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>

Reviewed-by: Willem de Bruijn <willemb@google.com>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement
  2026-11-18 13:54 [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement Jingxuan Xie
  2026-11-18 23:10 ` Willem de Bruijn
@ 2026-11-20 18:40 ` patchwork-bot+netdevbpf
  2026-11-22 14:04 ` netdev-bot+sashiko
  2 siblings, 0 replies; 4+ messages in thread
From: patchwork-bot+netdevbpf @ 2026-11-20 18:40 UTC (permalink / raw)
  To: Jingxuan Xie
  Cc: netdev, willemdebruijn.kernel, shaoyanghan, davem, edumazet,
	kuba, pabeni, horms, bigeasy, clrkwllms, rostedt, john.ogness,
	linux-kernel

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Wed, 18 Nov 2026 13:54:43 +0000 you wrote:
> TPACKET_V3 retires a full or expired block while holding
> sk_receive_queue.lock. prb_retire_current_block() takes
> blk_fill_in_prog_lock for write to wait for in-flight skb_copy_bits()
> before publishing the block.
> 
> On PREEMPT_RT a read lock can be held by a task that has been
> preempted or is running on another CPU. The packet path or the
> retirement timer then blocks on the write lock while holding
> sk_receive_queue.lock, and the receive path resumes only after the
> copy completes. This was observed as a receive stall of hundreds of
> milliseconds in a PREEMPT_RT deployment; manual inspection found the
> same unconditional write_lock() in both the full-block and the timer
> retirement paths.
> 
> [...]

Here is the summary with links:
  - [net,v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement
    https://git.kernel.org/netdev/net/c/d7f2d9b6c7d0

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement
  2026-11-18 13:54 [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement Jingxuan Xie
  2026-11-18 23:10 ` Willem de Bruijn
  2026-11-20 18:40 ` patchwork-bot+netdevbpf
@ 2026-11-22 14:04 ` netdev-bot+sashiko
  2 siblings, 0 replies; 4+ messages in thread
From: netdev-bot+sashiko @ 2026-11-22 14:04 UTC (permalink / raw)
  To: jingxuanxie
  Cc: netdev, willemdebruijn.kernel, shaoyanghan, davem, edumazet,
	kuba, pabeni, horms, bigeasy, clrkwllms, rostedt, john.ogness,
	linux-kernel

Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 2

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] On PREEMPT_RT, __packet_lookup_frame_in_block() can now return
  NULL while the current block is still TP_STATUS_KERNEL.
- [Low] The patch changelog says a review tag carries over, but the tag
  block does not contain one.
- [Low] The new comments describe only the PREEMPT_RT behaviour, and one
  of them is wrong for the default (non-RT) build.

--

af_packet: avoid waiting for TPACKET_V3 copies at retirement

On PREEMPT_RT, this patch uses write_trylock() on blk_fill_in_prog_lock
when retiring a TPACKET_V3 block. The packet path and the retirement
hrtimer then no longer block on an in-flight skb_copy_bits() while
holding sk_receive_queue.lock. If the trylock fails, the timer re-arms
and the packet path drops the capture record, while non-RT kernels keep
the blocking write_lock().

> Fixes: 632ca50f2cbd ("af_packet: TPACKET_V3: replace busy-wait loop")
> Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>
>
> ---
> Changes in v6:

[ ... ]

> Changes in v5:
> - Rebase onto net/main; no functional change, so the v4 Reviewed-by
>   still applies.

[Severity: Low]
The tag block has only the Fixes: and Signed-off-by: lines and no
Reviewed-by. Was the v4 Reviewed-by lost when v6 was reposted? If so,
could it be added back? If not, could the changelog note be corrected?

> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 73222563fb704..e0ebd0627509f 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c

[ ... ]

> @@ -677,6 +677,24 @@ static void init_prb_bdqc(struct packet_sock *po,
>  	prb_open_block(p1, pbd);
>  }
>  
> +/*
> + * Wait for an in-flight skb_copy_bits() before retiring the current block.
> + * On PREEMPT_RT the read lock can be held by a task that has been preempted
> + * or is running on another CPU, so do not block while holding
> + * sk_receive_queue.lock; return false and let the caller retry.
> + */
> +static bool prb_wait_for_copy_bits(struct tpacket_kbdq_core *pkc)
> +{
> +	if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
> +		if (!write_trylock(&pkc->blk_fill_in_prog_lock))
> +			return false;
> +	} else {
> +		write_lock(&pkc->blk_fill_in_prog_lock);
> +	}
> +	write_unlock(&pkc->blk_fill_in_prog_lock);
> +	return true;
> +}
> +

[ ... ]

> @@ -963,13 +982,13 @@ static void prb_retire_current_block(struct tpacket_kbdq_core *pkc,
>  		 * the timer-handler already handled this case.
>  		 */
>  		if (!(status & TP_STATUS_BLK_TMO)) {
> -			/* Waiting for skb_copy_bits to finish... */
> -			write_lock(&pkc->blk_fill_in_prog_lock);
> -			write_unlock(&pkc->blk_fill_in_prog_lock);
> +			/* Do not wait for copies while holding the queue lock. */
> +			if (!prb_wait_for_copy_bits(pkc))
> +				return false;

[Severity: Low]
This isn't a bug, but does this comment hold for non-RT builds?

There, prb_wait_for_copy_bits() takes the else branch and blocks in
write_lock(&pkc->blk_fill_in_prog_lock). Both callers hold
sk_receive_queue.lock at that point: tpacket_rcv() takes it before
packet_current_rx_frame(), and prb_retire_rx_blk_timer_expired() takes
it on entry.

The helper's comment also says "return false and let the caller retry".
Only the hrtimer path retries, through goto refresh_timer. On the packet
path, __packet_lookup_frame_in_block() returns NULL and tpacket_rcv()
goes to drop_n_account, so that record is never retried.

The name prb_wait_for_copy_bits() also doesn't fit the RT case, where
it doesn't wait.

Could the comments, and maybe the helper name, cover both
configurations?

>  		}
>  		prb_close_block(pkc, pbd, po, status);
> -		return;
>  	}
> +	return true;
>  }

[ ... ]

> @@ -1097,7 +1116,8 @@ static void *__packet_lookup_frame_in_block(struct packet_sock *po,
>  	}
>  
>  	/* Ok, close the current block */
> -	prb_retire_current_block(pkc, po, 0);
> +	if (!prb_retire_current_block(pkc, po, 0))
> +		return NULL;

[Severity: Medium]
On PREEMPT_RT, can this new NULL return break fanout rollover? At this
point the current block is full, but it stays TP_STATUS_KERNEL because
write_trylock() failed.

__tpacket_v3_has_room() only checks the block status:

net/packet/af_packet.c:__tpacket_v3_has_room() {
	...
	idx = READ_ONCE(po->rx_ring.prb_bdqc.kactive_blk_num);
	...
	return prb_lookup_block(po, &po->rx_ring, idx, TP_STATUS_KERNEL);
}

So packet_rcv_has_room() still returns ROOM_NORMAL or ROOM_LOW for this
socket, and PACKET_SOCK_PRESSURE is not set. fanout_demux_rollover()
then keeps sending packets to this socket:

	if (try_self) {
		room = packet_rcv_has_room(po, skb);
		if (room == ROOM_NORMAL ||
		    (room == ROOM_LOW && !fanout_flow_is_huge(po, skb)))
			return idx;
		...

With try_self=false, it can also choose this socket as a rollover
target.

Each of those packets then takes this path:

tpacket_rcv()
  __packet_lookup_frame_in_block()
    returns NULL
  goto drop_n_account

The packet is counted in tp_drops instead of going to a sibling socket
with space. This lasts as long as the copier holds the read lock, which
the commit message puts at hundreds of milliseconds.

Before this patch, the room check and the allocator agreed in this
state, because the allocator blocked and then succeeded.

The commit message accepts a higher capture drop rate. Should it also
say that PACKET_FANOUT_ROLLOVER and PACKET_FANOUT_FLAG_ROLLOVER no
longer move packets away from this socket in this state?

>  
>  	/* Now, try to dispatch the next block */
>  	curr = (char *)prb_dispatch_next_block(pkc, po);

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261118135443.24767-1-jingxuanxie%40deeproute.ai

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-22 14:04 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-18 13:54 [PATCH net v6] af_packet: avoid waiting for TPACKET_V3 copies at retirement Jingxuan Xie
2026-11-18 23:10 ` Willem de Bruijn
2026-11-20 18:40 ` patchwork-bot+netdevbpf
2026-11-22 14:04 ` netdev-bot+sashiko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®