From: Simon Schippers <simon.schippers@tu-dortmund.de>
To: willemdebruijn.kernel@gmail.com, jasowang@redhat.com,
andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com,
kuba@kernel.org, pabeni@redhat.com, mst@redhat.com,
eperezma@redhat.com, jon@nutanix.com, tim.gebauer@tu-dortmund.de,
simon.schippers@tu-dortmund.de, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
virtualization@lists.linux.dev
Subject: [PATCH net-next v6 3/8] tun/tap: add synchronized ring produce/consume with queue management
Date: Thu, 20 Nov 2025 16:29:08 +0100 [thread overview]
Message-ID: <20251120152914.1127975-4-simon.schippers@tu-dortmund.de> (raw)
In-Reply-To: <20251120152914.1127975-1-simon.schippers@tu-dortmund.de>
Implement new ring buffer produce and consume functions for tun and tap
drivers that provide lockless producer-consumer synchronization and
netdev queue management to prevent ptr_ring tail drop and permanent
starvation.
- tun_ring_produce(): Produces packets to the ptr_ring with proper memory
barriers and proactively stops the netdev queue when the ring is about
to become full.
- __tun_ring_consume() / __tap_ring_consume(): Internal consume functions
that check if the netdev queue was stopped due to a full ring, and wake
it when space becomes available. Uses memory barriers to ensure proper
ordering between producer and consumer.
- tun_ring_consume() / tap_ring_consume(): Wrapper functions that acquire
the consumer lock before calling the internal consume functions.
Key features:
- Proactive queue stopping using __ptr_ring_full_next() to stop the queue
before it becomes completely full.
- Not stopping the queue when the ptr_ring is full already, because if
the consumer empties all entries in the meantime, stopping the queue
would cause permanent starvation.
- Conditional queue waking using __ptr_ring_consume_created_space() to
wake the queue only when space is actually created in the ring.
- Prevents permanent starvation by ensuring the queue is also woken when
the ring becomes empty, which can happen when racing the producer.
NB: __always_unused on unused functions, to be removed later in the
series to not break bisectability.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Co-developed by: Jon Kohler <jon@nutanix.com>
Signed-off-by: Jon Kohler <jon@nutanix.com>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
drivers/net/tap.c | 63 +++++++++++++++++++++++++++++
drivers/net/tun.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 164 insertions(+)
diff --git a/drivers/net/tap.c b/drivers/net/tap.c
index 1197f245e873..c370a02789eb 100644
--- a/drivers/net/tap.c
+++ b/drivers/net/tap.c
@@ -753,6 +753,69 @@ static ssize_t tap_put_user(struct tap_queue *q,
return ret ? ret : total;
}
+/*
+ * Consume a packet from the transmit ring. Callers must hold
+ * the consumer_lock of the ptr_ring. If the ring was full and the
+ * queue was stopped, this may wake up the queue if space is created.
+ */
+static void *__tap_ring_consume(struct tap_queue *q)
+{
+ struct ptr_ring *ring = &q->ring;
+ struct netdev_queue *txq;
+ struct net_device *dev;
+ bool stopped;
+ void *ptr;
+
+ ptr = __ptr_ring_peek(ring);
+ if (!ptr)
+ return ptr;
+
+ /* Paired with smp_wmb() in the ring producer path. Ensures we
+ * see any updated netdev queue state caused by a full ring.
+ * Needed for proper synchronization between the ring and the
+ * netdev queue.
+ */
+ smp_rmb();
+ rcu_read_lock();
+ dev = rcu_dereference(q->tap)->dev;
+ txq = netdev_get_tx_queue(dev, q->queue_index);
+ stopped = netif_tx_queue_stopped(txq);
+
+ /* Ensures the read for a stopped queue completes before the
+ * discard, so that we don't miss the window to wake the queue if
+ * needed.
+ */
+ smp_rmb();
+ __ptr_ring_discard_one(ring);
+
+ /* If the queue was stopped (meaning the producer couldn't have
+ * inserted new entries just now), and we have actually created
+ * space in the ring, or the ring is now empty (due to a race
+ * with the producer), then it is now safe to wake the queue.
+ */
+ if (unlikely(stopped &&
+ (__ptr_ring_consume_created_space(ring) ||
+ __ptr_ring_empty(ring)))) {
+ /* Paired with smp_rmb() in tun_ring_produce. */
+ smp_wmb();
+ netif_tx_wake_queue(txq);
+ }
+ rcu_read_unlock();
+
+ return ptr;
+}
+
+static __always_unused void *tap_ring_consume(struct tap_queue *q)
+{
+ void *ptr;
+
+ spin_lock(&q->ring.consumer_lock);
+ ptr = __tap_ring_consume(q);
+ spin_unlock(&q->ring.consumer_lock);
+
+ return ptr;
+}
+
static ssize_t tap_do_read(struct tap_queue *q,
struct iov_iter *to,
int noblock, struct sk_buff *skb)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 8192740357a0..3b9d8d406ff5 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -999,6 +999,107 @@ static unsigned int run_ebpf_filter(struct tun_struct *tun,
return len;
}
+/* Produce a packet into the transmit ring. If the ring becomes full, the
+ * netdev queue is stopped until the consumer wakes it again.
+ */
+static __always_unused int tun_ring_produce(struct ptr_ring *ring,
+ struct netdev_queue *queue,
+ struct sk_buff *skb)
+{
+ int ret;
+
+ spin_lock(&ring->producer_lock);
+
+ /* Pairs with smp_wmb() in __tun_ring_consume/__tap_ring_consume.
+ * Ensures that freed space by the consumer is visible.
+ */
+ smp_rmb();
+
+ /* Do not stop the netdev queue if the ptr_ring is full already.
+ * The consumer could empty out the ptr_ring in the meantime
+ * without noticing the stopped netdev queue, resulting in a
+ * stopped netdev queue and an empty ptr_ring. In this case the
+ * netdev queue would stay stopped forever.
+ */
+ if (unlikely(!__ptr_ring_full(ring) &&
+ __ptr_ring_full_next(ring)))
+ netif_tx_stop_queue(queue);
+
+ /* Note: __ptr_ring_produce has an internal smp_wmb() to synchronize the
+ * state with the consumer. This ensures that after adding an entry to
+ * the ring, any stopped queue state is visible to the consumer after
+ * dequeueing.
+ */
+ ret = __ptr_ring_produce(ring, skb);
+
+ spin_unlock(&ring->producer_lock);
+
+ return ret;
+}
+
+/*
+ * Consume a packet from the transmit ring. Callers must hold
+ * the consumer_lock of the ptr_ring. If the ring was full and the
+ * queue was stopped, this may wake up the queue if space is created.
+ */
+static void *__tun_ring_consume(struct tun_file *tfile)
+{
+ struct ptr_ring *ring = &tfile->tx_ring;
+ struct netdev_queue *txq;
+ struct net_device *dev;
+ bool stopped;
+ void *ptr;
+
+ ptr = __ptr_ring_peek(ring);
+ if (!ptr)
+ return ptr;
+
+ /* Paired with smp_wmb() in the ring producer path. Ensures we
+ * see any updated netdev queue state caused by a full ring.
+ * Needed for proper synchronization between the ring and the
+ * netdev queue.
+ */
+ smp_rmb();
+ rcu_read_lock();
+ dev = rcu_dereference(tfile->tun)->dev;
+ txq = netdev_get_tx_queue(dev, tfile->queue_index);
+ stopped = netif_tx_queue_stopped(txq);
+
+ /* Ensures the read for a stopped queue completes before the
+ * discard, so that we don't miss the window to wake the queue if
+ * needed.
+ */
+ smp_rmb();
+ __ptr_ring_discard_one(ring);
+
+ /* If the queue was stopped (meaning the producer couldn't have
+ * inserted new entries just now), and we have actually created
+ * space in the ring, or the ring is now empty (due to a race
+ * with the producer), then it is now safe to wake the queue.
+ */
+ if (unlikely(stopped &&
+ (__ptr_ring_consume_created_space(ring) ||
+ __ptr_ring_empty(ring)))) {
+ /* Paired with smp_rmb() in tun_ring_produce. */
+ smp_wmb();
+ netif_tx_wake_queue(txq);
+ }
+ rcu_read_unlock();
+
+ return ptr;
+}
+
+static void __always_unused *tun_ring_consume(struct tun_file *tfile)
+{
+ void *ptr;
+
+ spin_lock(&tfile->tx_ring.consumer_lock);
+ ptr = __tun_ring_consume(tfile);
+ spin_unlock(&tfile->tx_ring.consumer_lock);
+
+ return ptr;
+}
+
/* Net device start xmit */
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
--
2.43.0
next prev parent reply other threads:[~2025-11-20 15:30 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-20 15:29 [PATCH net-next v6 0/8] tun/tap & vhost-net: netdev queue flow control to avoid ptr_ring tail drop Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 1/8] ptr_ring: add __ptr_ring_full_next() to predict imminent fullness Simon Schippers
2025-11-25 14:54 ` Michael S. Tsirkin
2025-11-20 15:29 ` [PATCH net-next v6 2/8] ptr_ring: add helper to check if consume created space Simon Schippers
2025-11-25 15:01 ` Michael S. Tsirkin
2025-11-25 16:12 ` Simon Schippers
2025-11-25 17:18 ` Michael S. Tsirkin
2025-11-30 18:16 ` Willem de Bruijn
2025-11-20 15:29 ` Simon Schippers [this message]
2025-11-25 16:54 ` [PATCH net-next v6 3/8] tun/tap: add synchronized ring produce/consume with queue management Michael S. Tsirkin
2025-11-26 9:23 ` Simon Schippers
2025-11-26 15:25 ` Michael S. Tsirkin
2025-11-26 16:04 ` Simon Schippers
2025-11-26 18:16 ` Michael S. Tsirkin
2025-11-20 15:29 ` [PATCH net-next v6 4/8] tun/tap: add batched ring consume function Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 5/8] tun/tap: add uncomsume function for returning entries to ring Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 6/8] tun/tap: add helper functions to check file type Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 7/8] tun/tap & vhost-net: use {tun|tap}_ring_{consume|produce} to avoid tail drops Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 7/8] tun/tap/vhost: " Simon Schippers
2025-11-20 15:29 ` [PATCH net-next v6 8/8] tun/tap: drop get ring exports Simon Schippers
2025-11-21 6:19 ` [PATCH net-next v6 0/8] tun/tap & vhost-net: netdev queue flow control to avoid ptr_ring tail drop Jason Wang
2025-11-21 9:22 ` Simon Schippers
2025-11-24 1:04 ` Jason Wang
2025-11-24 9:19 ` Simon Schippers
2025-11-25 1:34 ` Jason Wang
2025-11-25 14:04 ` Simon Schippers
2025-11-26 6:19 ` Jason Wang
2025-11-26 7:15 ` Michael S. Tsirkin
2025-11-26 9:24 ` Simon Schippers
2025-11-21 9:18 ` [syzbot ci] " syzbot ci
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251120152914.1127975-4-simon.schippers@tu-dortmund.de \
--to=simon.schippers@tu-dortmund.de \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eperezma@redhat.com \
--cc=jasowang@redhat.com \
--cc=jon@nutanix.com \
--cc=kuba@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=tim.gebauer@tu-dortmund.de \
--cc=virtualization@lists.linux.dev \
--cc=willemdebruijn.kernel@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®