* [PATCH net v2 1/9] gve: increment work_done for XDP and error packets
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 2/9] gve: fix XSK buffer leak when rings are stopped Joshua Washington
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
The GVE RX NAPI will continue polling as long as
1) there are packets to be processed, and
2) less than NAPI budget SKBs (denoted in GVE by work_done) have been
passed up to the kernel.
However, GVE does not account for all of the packets that don't create
SKBs, namely error packets and XDP packets.
This can result in XDP programs that scarcely return XDP_PASS failing to
exit the NAPI poll as long as the NIC is DMA'ing packets, possibly
processing the entire RX ring before returning from the NAPI.
This has 3 negative implications:
1) XDP RX path can run much longer than is desirable, hogging CPU
resources.
2) If XDP_PASS is never returned, the work_done never increases beyond
0, which can lead to scheduling delays due to missed chances to
reschedule the NAPI.
3) In AF_XDP zero-copy, XSK_TX occurs after the RX poll. If the RX poll
takes a long time, it will delay TX, leading to degraded performance.
Ensure every packet is accounted for in work_done by incrementing
work_done before checking for the existence of a SKB.
Fixes: 293b49361f91 ("gve: add XDP DROP and PASS support for DQ")
Cc: stable@vger.kernel.org
Reviewed-by: Tim Hostetler <thostet@google.com>
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
v2:
- corrected stat counting for packets relative to work_done
---
drivers/net/ethernet/google/gve/gve_rx_dqo.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index 5cf242b28557..c3f4a76b0fac 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -932,6 +932,10 @@ static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
if (xdp_act != XDP_PASS) {
gve_xdp_done_dqo(priv, rx, &gve_xdp.xdp, xprog, xdp_act,
buf_state);
+ u64_stats_update_begin(&rx->statss);
+ rx->rpackets++;
+ rx->rbytes += compl_desc->packet_len;
+ u64_stats_update_end(&rx->statss);
return 0;
}
@@ -1090,6 +1094,7 @@ int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
struct gve_rx_ring *rx;
struct gve_priv *priv;
u64 xdp_redirects;
+ u32 rx_packets = 0;
u32 work_done = 0;
u64 bytes = 0;
u64 xdp_txs;
@@ -1150,13 +1155,14 @@ int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
/* Free running counter of completed descriptors */
rx->cnt++;
- if (!rx->ctx.skb_head)
- continue;
-
if (!compl_desc->end_of_packet)
continue;
work_done++;
+
+ if (!rx->ctx.skb_head)
+ continue;
+
pkt_bytes = rx->ctx.skb_head->len;
/* The ethernet header (first ETH_HLEN bytes) is snipped off
* by eth_type_trans.
@@ -1164,6 +1170,9 @@ int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
if (skb_headlen(rx->ctx.skb_head))
pkt_bytes += ETH_HLEN;
+ rx_packets++;
+ bytes += pkt_bytes;
+
/* gve_rx_complete_skb() will consume skb if successful */
if (gve_rx_complete_skb(rx, napi, compl_desc, feat) != 0) {
gve_rx_free_skb(napi, rx);
@@ -1173,7 +1182,6 @@ int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
continue;
}
- bytes += pkt_bytes;
rx->ctx.skb_head = NULL;
rx->ctx.skb_tail = NULL;
}
@@ -1187,7 +1195,7 @@ int gve_rx_poll_dqo(struct gve_notify_block *block, int budget)
gve_rx_post_buffers_dqo(rx);
u64_stats_update_begin(&rx->statss);
- rx->rpackets += work_done;
+ rx->rpackets += rx_packets;
rx->rbytes += bytes;
u64_stats_update_end(&rx->statss);
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 2/9] gve: fix XSK buffer leak when rings are stopped
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 1/9] gve: increment work_done for XDP and error packets Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 3/9] gve: fix XSK buffer leak on error descriptor Joshua Washington
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
GVE does not free XSK buffers when resetting ring state as a part of
stopping queues. This causes all XSK buffers which are posted to the
NIC to be leaked.
Free XSK buffers attached to an allocated buf_state when stopping rings.
Fixes: c1fffc5d66a7 ("gve: implement DQO RX datapath and control path for AF_XDP zero-copy")
Cc: stable@vger.kernel.org
Reviewed-by: Tim Hostetler <thostet@google.com>
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
drivers/net/ethernet/google/gve/gve_rx_dqo.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index c3f4a76b0fac..3a88b7e98b16 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -113,6 +113,12 @@ static void gve_rx_reset_ring_dqo(struct gve_priv *priv, int idx)
gve_free_to_page_pool(rx, bs, false);
else
gve_free_qpl_page_dqo(bs);
+
+ if (gve_buf_state_is_allocated(rx, bs) &&
+ bs->xsk_buff) {
+ xsk_buff_free(bs->xsk_buff);
+ bs->xsk_buff = NULL;
+ }
}
}
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 3/9] gve: fix XSK buffer leak on error descriptor
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 1/9] gve: increment work_done for XDP and error packets Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 2/9] gve: fix XSK buffer leak when rings are stopped Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 4/9] gve: don't register xsk pool on pre-existing queues in RDA mode Joshua Washington
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
When the error bit is set in the RX completion descriptor, the buf_state
and its attached buffer should be freed. In the case of AF_XDP ZC, the
XSK buffer was not freed, leading to a leak.
Fixes: c1fffc5d66a7 ("gve: implement DQO RX datapath and control path for AF_XDP zero-copy")
Cc: stable@vger.kernel.org
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Reviewed-by: Tim Hostetler <thostet@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
drivers/net/ethernet/google/gve/gve_rx_dqo.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index 3a88b7e98b16..c1e97e11ff35 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -842,7 +842,12 @@ static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
}
if (unlikely(compl_desc->rx_error)) {
- gve_free_buffer(rx, buf_state);
+ if (buf_state->xsk_buff) {
+ xsk_buff_free(buf_state->xsk_buff);
+ gve_free_buf_state(rx, buf_state);
+ } else {
+ gve_free_buffer(rx, buf_state);
+ }
return -EINVAL;
}
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 4/9] gve: don't register xsk pool on pre-existing queues in RDA mode
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (2 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 3/9] gve: fix XSK buffer leak on error descriptor Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 5/9] gve: fix napi_disable deadlock when attempting to disable XSK pools Joshua Washington
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
When XSK pools are enabled after an XDP program has already been loaded,
XSK pools are registered on pre-existing queues before queues are
re-created with the XSK pool fully registered in DQ RDA mode.
This can lead to a race condition between the RX NAPI and the control
plane thread wherein a pre-existing queue sees the live XSK pool and
attempts to use recycled buffers not backed by XSK buffs for AF_XDP ZC
traffic. This causes the following kernel panic to occur when attempting
to DMA map a NULL XSK buffer:
BUG: kernel NULL pointer dereference, address: 0000000000000050
...
RIP: 0010:gve_rx_post_buffers_dqo+0x99/0x190 [gve]
...
Call Trace:
<TASK>
gve_rx_poll_dqo+0x4d9/0xf10 [gve]
gve_napi_poll_dqo+0x76/0x170 [gve]
__napi_poll+0x28/0x160
net_rx_action+0x2a0/0x350
handle_softirqs+0xd4/0x280
? sort_range+0x20/0x20
run_ksoftirqd+0x2d/0x40
smpboot_thread_fn+0xd5/0x1d0
kthread+0xd7/0x100
? kthread_complete_and_exit+0x20/0x20
ret_from_fork+0x1f/0x30
</TASK>
The XSK pool should only be registered with current queues if XSK
buffers are allocated on-the-fly, as is the case in QPL mode.
Fixes: c1fffc5d66a7 ("gve: implement DQO RX datapath and control path for AF_XDP zero-copy")
Cc: stable@vger.kernel.org
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Reviewed-by: Tim Hostetler <thostet@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
drivers/net/ethernet/google/gve/gve_main.c | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 9cc343a16271..b9bcdc7619b2 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1654,20 +1654,18 @@ static int gve_xsk_pool_enable(struct net_device *dev,
if (!priv->xdp_prog || !netif_running(dev))
return 0;
- err = gve_reg_xsk_pool(priv, dev, pool, qid);
- if (err)
- goto err_xsk_pool_dma_mapped;
-
- /* Stop and start RDA queues to repost buffers. */
- if (!gve_is_qpl(priv)) {
+ if (gve_is_qpl(priv)) {
+ err = gve_reg_xsk_pool(priv, dev, pool, qid);
+ if (err)
+ goto err_xsk_pool_dma_mapped;
+ } else {
+ /* Stop and start RDA queues to repost buffers. */
err = gve_configure_rings_xdp(priv, priv->rx_cfg.num_queues);
if (err)
- goto err_xsk_pool_registered;
+ goto err_xsk_pool_dma_mapped;
}
return 0;
-err_xsk_pool_registered:
- gve_unreg_xsk_pool(priv, qid);
err_xsk_pool_dma_mapped:
clear_bit(qid, priv->xsk_pools);
xsk_pool_dma_unmap(pool,
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 5/9] gve: fix napi_disable deadlock when attempting to disable XSK pools
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (3 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 4/9] gve: don't register xsk pool on pre-existing queues in RDA mode Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 6/9] gve: fix NULL dereference from premature XSK pool DMA unmap Joshua Washington
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
When disabling XSK pools, GVE calls the unlocked versions of
napi_disable and napi_enable. However, the netdev lock has already been
acquired before ndo_bpf is called because GVE supports queue management
ops. Calling the unlocked versions of napi_disable/enable results in a
deadlock when attempting to disable XSK pools, as the thread attempts to
re-acquire a lock it already holds.
Update the NAPI calls to use the locked versions.
Fixes: 606048cbd834 ("net: designate XSK pool pointers in queues as "ops protected"")
Cc: stable@vger.kernel.org
Reviewed-by: Harshitha Ramamurthy <hramamurthy@google.com>
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
drivers/net/ethernet/google/gve/gve_main.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index b9bcdc7619b2..3712ff364cbd 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1706,17 +1706,17 @@ static int gve_xsk_pool_disable(struct net_device *dev,
}
napi_rx = &priv->ntfy_blocks[priv->rx[qid].ntfy_id].napi;
- napi_disable(napi_rx); /* make sure current rx poll is done */
+ napi_disable_locked(napi_rx); /* make sure current rx poll is done */
tx_qid = gve_xdp_tx_queue_id(priv, qid);
napi_tx = &priv->ntfy_blocks[priv->tx[tx_qid].ntfy_id].napi;
- napi_disable(napi_tx); /* make sure current tx poll is done */
+ napi_disable_locked(napi_tx); /* make sure current tx poll is done */
gve_unreg_xsk_pool(priv, qid);
smp_mb(); /* Make sure it is visible to the workers on datapath */
- napi_enable(napi_rx);
- napi_enable(napi_tx);
+ napi_enable_locked(napi_rx);
+ napi_enable_locked(napi_tx);
if (gve_is_gqi(priv)) {
if (gve_rx_work_pending(&priv->rx[qid]))
napi_schedule(napi_rx);
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 6/9] gve: fix NULL dereference from premature XSK pool DMA unmap
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (4 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 5/9] gve: fix napi_disable deadlock when attempting to disable XSK pools Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 7/9] gve: disable NAPI when registering XSK pools in QPL mode Joshua Washington
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
To ensure that XSK pools are DMA unmapped in all scenarios, GVE performs
the unmapping before validating if the interface is up and early
returning.
However, if rings are up, this introduces a race between the RX NAPI and
the control plane. As part of DMA unmapping the XSK pool, the kernel
sets pool->dev to NULL. Because xsk_buff_dma_sync_for_cpu() relies on
pool->dev, this results in a kernel panic:
BUG: kernel NULL pointer dereference, address: 000000000000030c
...
RIP: 0010:gve_rx_poll_dqo+0x2e2/0x13b0 [gve]
...
Call Trace:
<IRQ>
gve_napi_poll_dqo+0x88/0x170 [gve]
__napi_poll+0x30/0x210
net_rx_action+0x210/0x410
? dst_destroy_rcu+0x12/0x20
handle_softirqs+0xe4/0x310
__irq_exit_rcu+0x10e/0x130
irq_exit_rcu+0xe/0x20
common_interrupt+0xb6/0xe0
</IRQ>
Leave the XSK pool DMA mapped until after rings are guaranteed to no
longer rely on the pool.
Fixes: d57ae093c887 ("gve: deduplicate xdp info and xsk pool registration logic")
Cc: stable@vger.kernel.org
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
drivers/net/ethernet/google/gve/gve_main.c | 25 ++++++++++++----------
1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 3712ff364cbd..49ae2b8c6a27 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1681,22 +1681,18 @@ static int gve_xsk_pool_disable(struct net_device *dev,
struct napi_struct *napi_rx;
struct napi_struct *napi_tx;
struct xsk_buff_pool *pool;
+ int err = 0;
int tx_qid;
- int err;
- if (qid >= priv->rx_cfg.num_queues)
- return -EINVAL;
+ if (qid >= priv->rx_cfg.num_queues) {
+ err = -EINVAL;
+ goto unmap_and_return;
+ }
clear_bit(qid, priv->xsk_pools);
- pool = xsk_get_pool_from_qid(dev, qid);
- if (pool)
- xsk_pool_dma_unmap(pool,
- DMA_ATTR_SKIP_CPU_SYNC |
- DMA_ATTR_WEAK_ORDERING);
-
if (!netif_running(dev) || !priv->tx_cfg.num_xdp_queues)
- return 0;
+ goto unmap_and_return;
/* Stop and start RDA queues to repost buffers. */
if (!gve_is_qpl(priv) && priv->xdp_prog) {
@@ -1725,7 +1721,14 @@ static int gve_xsk_pool_disable(struct net_device *dev,
napi_schedule(napi_tx);
}
- return 0;
+unmap_and_return:
+ pool = xsk_get_pool_from_qid(dev, qid);
+ if (pool)
+ xsk_pool_dma_unmap(pool,
+ DMA_ATTR_SKIP_CPU_SYNC |
+ DMA_ATTR_WEAK_ORDERING);
+
+ return err;
}
static int gve_xsk_wakeup(struct net_device *dev, u32 queue_id, u32 flags)
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 7/9] gve: disable NAPI when registering XSK pools in QPL mode
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (5 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 6/9] gve: fix NULL dereference from premature XSK pool DMA unmap Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 8/9] gve: ensure XDP mem model is registered when disabling XSK pools Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 9/9] gve: prevent XDP frame leak and corruption during DQO TX cleanup Joshua Washington
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
As a result of QPL modes not posting XSK buffers directly to the NIC,
they do not need to recreate queues or re-post DMA buffers.
However, traffic should still be quiesced because both the driver and
the XDP redirect stack must have the same knowledge about whether a
given packet is being processed with AF_XDP zero-copy enabled or not.
GVE in QPL mode does not current respect this, which could lead to a
race condition between packet processing and the XSK_BUFF_POOL memory
model registration.
Quiesce traffic by disabling the NAPI while the XSK_BUFF_POOL memory
model is being registered with the kernel.
Fixes: fd8e40321a12 ("gve: Add AF_XDP zero-copy support for GQI-QPL format")
Cc: stable@vger.kernel.org
Reviewed-by: Harshitha Ramamurthy <hramamurthy@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
v2:
- newly introduced
---
drivers/net/ethernet/google/gve/gve_main.c | 41 ++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 49ae2b8c6a27..f2bd4011de23 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1626,6 +1626,40 @@ static int gve_xdp_xmit(struct net_device *dev, int n,
return -EOPNOTSUPP;
}
+static void gve_disable_xsk_napis(struct gve_priv *priv, u16 qid)
+{
+ struct napi_struct *napi_rx, *napi_tx;
+ u16 tx_qid;
+
+ napi_rx = &priv->ntfy_blocks[priv->rx[qid].ntfy_id].napi;
+ napi_disable_locked(napi_rx);
+
+ tx_qid = gve_xdp_tx_queue_id(priv, qid);
+ napi_tx = &priv->ntfy_blocks[priv->tx[tx_qid].ntfy_id].napi;
+ napi_disable_locked(napi_tx);
+}
+
+static void gve_enable_xsk_napis(struct gve_priv *priv, u16 qid)
+{
+ struct napi_struct *napi_rx, *napi_tx;
+ u16 tx_qid;
+
+ napi_rx = &priv->ntfy_blocks[priv->rx[qid].ntfy_id].napi;
+ napi_enable_locked(napi_rx);
+
+ tx_qid = gve_xdp_tx_queue_id(priv, qid);
+ napi_tx = &priv->ntfy_blocks[priv->tx[tx_qid].ntfy_id].napi;
+ napi_enable_locked(napi_tx);
+
+ if (gve_is_gqi(priv)) {
+ if (gve_rx_work_pending(&priv->rx[qid]))
+ napi_schedule(napi_rx);
+
+ if (gve_tx_clean_pending(priv, &priv->tx[tx_qid]))
+ napi_schedule(napi_tx);
+ }
+}
+
static int gve_xsk_pool_enable(struct net_device *dev,
struct xsk_buff_pool *pool,
u16 qid)
@@ -1655,7 +1689,14 @@ static int gve_xsk_pool_enable(struct net_device *dev,
return 0;
if (gve_is_qpl(priv)) {
+ gve_disable_xsk_napis(priv, qid);
+
err = gve_reg_xsk_pool(priv, dev, pool, qid);
+ /* Make sure it is visible to the workers on datapath */
+ smp_mb();
+
+ gve_enable_xsk_napis(priv, qid);
+
if (err)
goto err_xsk_pool_dma_mapped;
} else {
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 8/9] gve: ensure XDP mem model is registered when disabling XSK pools
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (6 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 7/9] gve: disable NAPI when registering XSK pools in QPL mode Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
2026-09-22 19:45 ` [PATCH net v2 9/9] gve: prevent XDP frame leak and corruption during DQO TX cleanup Joshua Washington
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
When disabling XSK pools, QPL and RDA modes have different behaviors,
but they are both incorrect in that the function returns just after
unregistering the memory model.
RDA mode performs an internally consistent re-configuration of rings,
making the extra logic, including the XSK pool unregistration,
unnecessary. However, it is possible for the reconfiguration to fail due
to memory allocation. Instead of freeing and re-allocating ring memory,
stop the rings and reinitialize the ring state. That way, failure to
stop the rings would result in a safer device reset, making it safe to
unregister the pool regardless of the error condition when stopping
queues. This change involves a bit of refactoring in the TX
initialization path, introducing new methods to reset ring state when
stopping the rings.
QPL mode, which does not need to reconfigure rings due to not posting
XSK umem to the hardware ring, simply misses registering the RXQ XDP
info with the MEM_TYPE_PAGE_SHARED memory model.
Make a best-effort attempt to register memory model in both cases.
Because memory model registration can fail and xp_release_deferred
cannot, the XSK pool must be unregistered and DMA-unmapped regardless of
whether memory model registration succeeds. In both cases, there should
be a guarantee against the device DMA'ing into freed memory, however.
Fixes: 077f7153fd25 ("gve: merge xdp and xsk registration")
Cc: stable@vger.kernel.org
Reviewed-by: Harshitha Ramamurthy <hramamurthy@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
v2:
- Newly introduced
---
drivers/net/ethernet/google/gve/gve_main.c | 77 +++++++++-----
drivers/net/ethernet/google/gve/gve_tx_dqo.c | 101 ++++++++++++++-----
2 files changed, 126 insertions(+), 52 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index f2bd4011de23..787d311ff99b 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1715,15 +1715,56 @@ static int gve_xsk_pool_enable(struct net_device *dev,
return err;
}
+static int gve_unreg_xsk_pool_live(struct gve_priv *priv,
+ struct net_device *dev, u16 qid)
+{
+ struct gve_rx_ring *rx;
+ int err;
+
+ rx = &priv->rx[qid];
+ gve_disable_xsk_napis(priv, qid);
+
+ gve_unreg_xsk_pool(priv, qid);
+ if (gve_is_qpl(priv))
+ err = xdp_rxq_info_reg_mem_model(&rx->xdp_rxq,
+ MEM_TYPE_PAGE_SHARED,
+ NULL);
+ else
+ err = xdp_rxq_info_reg_mem_model(&rx->xdp_rxq,
+ MEM_TYPE_PAGE_POOL,
+ rx->dqo.page_pool);
+ if (err)
+ netdev_warn(dev,
+ "Failed to register memory model after unregistering XSK pool");
+
+ smp_mb(); /* Make sure it is visible to the workers on datapath */
+
+ gve_enable_xsk_napis(priv, qid);
+
+ return err;
+}
+
+static int gve_restart_rings(struct gve_priv *priv)
+{
+ struct gve_tx_alloc_rings_cfg tx_alloc_cfg = {0};
+ struct gve_rx_alloc_rings_cfg rx_alloc_cfg = {0};
+ int err;
+
+ gve_get_curr_alloc_cfgs(priv, &tx_alloc_cfg, &rx_alloc_cfg);
+ err = gve_queues_stop(priv);
+ if (err)
+ return err;
+
+ err = gve_queues_start(priv, &tx_alloc_cfg, &rx_alloc_cfg);
+ return err;
+}
+
static int gve_xsk_pool_disable(struct net_device *dev,
u16 qid)
{
struct gve_priv *priv = netdev_priv(dev);
- struct napi_struct *napi_rx;
- struct napi_struct *napi_tx;
struct xsk_buff_pool *pool;
int err = 0;
- int tx_qid;
if (qid >= priv->rx_cfg.num_queues) {
err = -EINVAL;
@@ -1735,31 +1776,11 @@ static int gve_xsk_pool_disable(struct net_device *dev,
if (!netif_running(dev) || !priv->tx_cfg.num_xdp_queues)
goto unmap_and_return;
- /* Stop and start RDA queues to repost buffers. */
- if (!gve_is_qpl(priv) && priv->xdp_prog) {
- err = gve_configure_rings_xdp(priv, priv->rx_cfg.num_queues);
- if (err)
- return err;
- }
-
- napi_rx = &priv->ntfy_blocks[priv->rx[qid].ntfy_id].napi;
- napi_disable_locked(napi_rx); /* make sure current rx poll is done */
-
- tx_qid = gve_xdp_tx_queue_id(priv, qid);
- napi_tx = &priv->ntfy_blocks[priv->tx[tx_qid].ntfy_id].napi;
- napi_disable_locked(napi_tx); /* make sure current tx poll is done */
-
- gve_unreg_xsk_pool(priv, qid);
- smp_mb(); /* Make sure it is visible to the workers on datapath */
-
- napi_enable_locked(napi_rx);
- napi_enable_locked(napi_tx);
- if (gve_is_gqi(priv)) {
- if (gve_rx_work_pending(&priv->rx[qid]))
- napi_schedule(napi_rx);
-
- if (gve_tx_clean_pending(priv, &priv->tx[tx_qid]))
- napi_schedule(napi_tx);
+ if (gve_is_qpl(priv)) {
+ err = gve_unreg_xsk_pool_live(priv, dev, qid);
+ } else {
+ /* Stop and start RDA queues to repost buffers. */
+ err = gve_restart_rings(priv);
}
unmap_and_return:
diff --git a/drivers/net/ethernet/google/gve/gve_tx_dqo.c b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
index 80ab0a449ff5..78f946ae7264 100644
--- a/drivers/net/ethernet/google/gve/gve_tx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
@@ -208,6 +208,78 @@ static void gve_tx_clean_pending_packets(struct gve_tx_ring *tx)
}
}
+static void gve_tx_init_ring_state_dqo(struct gve_tx_ring *tx)
+{
+ int i;
+
+ atomic_set_release(&tx->dqo_compl.hw_tx_head, 0);
+
+ /* Set up linked list of pending packets */
+ for (i = 0; i < tx->dqo.num_pending_packets - 1; i++)
+ tx->dqo.pending_packets[i].next = i + 1;
+
+ tx->dqo.pending_packets[tx->dqo.num_pending_packets - 1].next = -1;
+ atomic_set_release(&tx->dqo_compl.free_pending_packets, -1);
+
+ tx->dqo_compl.miss_completions.head = -1;
+ tx->dqo_compl.miss_completions.tail = -1;
+ tx->dqo_compl.timed_out_completions.head = -1;
+ tx->dqo_compl.timed_out_completions.tail = -1;
+
+ /* Generate free TX buf list */
+ if (tx->dqo.tx_qpl_buf_next) {
+ for (i = 0; i < tx->dqo.num_tx_qpl_bufs - 1; i++)
+ tx->dqo.tx_qpl_buf_next[i] = i + 1;
+ tx->dqo.tx_qpl_buf_next[tx->dqo.num_tx_qpl_bufs - 1] = -1;
+
+ atomic_set_release(&tx->dqo_compl.free_tx_qpl_buf_head, -1);
+ atomic_set_release(&tx->dqo_compl.free_tx_qpl_buf_cnt, 0);
+ }
+}
+
+static void gve_tx_reset_ring_dqo(struct gve_tx_ring *tx)
+{
+ size_t size;
+
+ /* Reset dqo_tx fields. */
+ tx->dqo_tx.head = 0;
+ tx->dqo_tx.tail = 0;
+ tx->dqo_tx.last_re_idx = 0;
+ tx->dqo_tx.posted_packet_desc_cnt = 0;
+ tx->dqo_tx.completed_packet_desc_cnt = 0;
+ tx->dqo_tx.free_pending_packets = 0;
+
+ /* Reset dqo_compl fields. */
+ tx->dqo_compl.head = 0;
+ tx->dqo_compl.cur_gen_bit = 0;
+ tx->dqo_compl.xsk_reorder_queue_head = 0;
+ tx->dqo_compl.xsk_reorder_queue_tail = 0;
+
+ if (tx->dqo.xsk_reorder_queue) {
+ size = (tx->dqo.complq_mask + 1) *
+ sizeof(*tx->dqo.xsk_reorder_queue);
+ memset(tx->dqo.xsk_reorder_queue, 0, size);
+ atomic_set(&tx->dqo_tx.xsk_reorder_queue_tail, 0);
+ }
+
+ size = sizeof(tx->dqo.tx_ring[0]) * (tx->mask + 1);
+ memset(tx->dqo.tx_ring, 0, size);
+
+ size = sizeof(tx->dqo.compl_ring[0]) * (tx->dqo.complq_mask + 1);
+ memset(tx->dqo.compl_ring, 0, size);
+
+ memset(tx->q_resources, 0, sizeof(*tx->q_resources));
+
+ if (tx->dqo.tx_qpl_buf_next) {
+ tx->dqo_tx.free_tx_qpl_buf_head = 0;
+ size = sizeof(tx->dqo.tx_qpl_buf_next[0]) *
+ tx->dqo.num_tx_qpl_bufs;
+ memset(tx->dqo.tx_qpl_buf_next, 0, size);
+ }
+
+ gve_tx_init_ring_state_dqo(tx);
+}
+
void gve_tx_stop_ring_dqo(struct gve_priv *priv, int idx)
{
int ntfy_idx = gve_tx_idx_to_ntfy(priv, idx);
@@ -222,6 +294,7 @@ void gve_tx_stop_ring_dqo(struct gve_priv *priv, int idx)
netdev_tx_reset_queue(tx->netdev_txq);
gve_tx_clean_pending_packets(tx);
gve_tx_remove_from_block(priv, idx);
+ gve_tx_reset_ring_dqo(tx);
}
static void gve_tx_free_ring_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
@@ -270,11 +343,10 @@ static void gve_tx_free_ring_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
netif_dbg(priv, drv, priv->dev, "freed tx queue %d\n", idx);
}
-static int gve_tx_qpl_buf_init(struct gve_tx_ring *tx)
+static int gve_tx_qpl_buf_list_alloc(struct gve_tx_ring *tx)
{
int num_tx_qpl_bufs = GVE_TX_BUFS_PER_PAGE_DQO *
tx->dqo.qpl->num_entries;
- int i;
tx->dqo.tx_qpl_buf_next = kvzalloc_objs(tx->dqo.tx_qpl_buf_next[0],
num_tx_qpl_bufs);
@@ -282,13 +354,6 @@ static int gve_tx_qpl_buf_init(struct gve_tx_ring *tx)
return -ENOMEM;
tx->dqo.num_tx_qpl_bufs = num_tx_qpl_bufs;
-
- /* Generate free TX buf list */
- for (i = 0; i < num_tx_qpl_bufs - 1; i++)
- tx->dqo.tx_qpl_buf_next[i] = i + 1;
- tx->dqo.tx_qpl_buf_next[num_tx_qpl_bufs - 1] = -1;
-
- atomic_set_release(&tx->dqo_compl.free_tx_qpl_buf_head, -1);
return 0;
}
@@ -304,6 +369,7 @@ void gve_tx_start_ring_dqo(struct gve_priv *priv, int idx)
gve_add_napi(priv, ntfy_idx, gve_napi_poll_dqo);
}
+
static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
struct gve_tx_alloc_rings_cfg *cfg,
struct gve_tx_ring *tx,
@@ -313,13 +379,11 @@ static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
int num_pending_packets;
size_t bytes;
u32 qpl_id;
- int i;
memset(tx, 0, sizeof(*tx));
tx->q_num = idx;
tx->dev = hdev;
spin_lock_init(&tx->dqo_tx.xdp_lock);
- atomic_set_release(&tx->dqo_compl.hw_tx_head, 0);
/* Queue sizes must be a power of 2 */
tx->mask = cfg->ring_size - 1;
@@ -350,13 +414,6 @@ static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
if (!tx->dqo.pending_packets)
goto err;
- /* Set up linked list of pending packets */
- for (i = 0; i < tx->dqo.num_pending_packets - 1; i++)
- tx->dqo.pending_packets[i].next = i + 1;
-
- tx->dqo.pending_packets[tx->dqo.num_pending_packets - 1].next = -1;
- atomic_set_release(&tx->dqo_compl.free_pending_packets, -1);
-
/* Only alloc xsk pool for XDP queues */
if (idx >= cfg->qcfg->num_queues && cfg->num_xdp_rings) {
tx->dqo.xsk_reorder_queue =
@@ -367,11 +424,6 @@ static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
goto err;
}
- tx->dqo_compl.miss_completions.head = -1;
- tx->dqo_compl.miss_completions.tail = -1;
- tx->dqo_compl.timed_out_completions.head = -1;
- tx->dqo_compl.timed_out_completions.tail = -1;
-
bytes = sizeof(tx->dqo.tx_ring[0]) * (tx->mask + 1);
tx->dqo.tx_ring = dma_alloc_coherent(hdev, bytes, &tx->bus, GFP_KERNEL);
if (!tx->dqo.tx_ring)
@@ -397,10 +449,11 @@ static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
if (!tx->dqo.qpl)
goto err;
- if (gve_tx_qpl_buf_init(tx))
+ if (gve_tx_qpl_buf_list_alloc(tx))
goto err;
}
+ gve_tx_init_ring_state_dqo(tx);
return 0;
err:
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH net v2 9/9] gve: prevent XDP frame leak and corruption during DQO TX cleanup
2026-09-22 19:45 [PATCH net v2 0/9] gve: various XDP fixes Joshua Washington
` (7 preceding siblings ...)
2026-09-22 19:45 ` [PATCH net v2 8/9] gve: ensure XDP mem model is registered when disabling XSK pools Joshua Washington
@ 2026-09-22 19:45 ` Joshua Washington
8 siblings, 0 replies; 10+ messages in thread
From: Joshua Washington @ 2026-09-22 19:45 UTC (permalink / raw)
To: netdev
Cc: Joshua Washington, Harshitha Ramamurthy, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Stanislav Fomichev, Jordan Rhee,
Willem de Bruijn, Tim Hostetler, Ankit Garg, Eddie Phillips,
Praveen Kaligineedi, Jeroen de Borst, linux-kernel, bpf, stable
From: Eddie Phillips <eddiephillips@google.com>
When tearing down a DQO TX ring or processing miss completions,
pending packets were assumed to be SKBs. If an XDP frame was pending
during cleanup or timeout, cur_state->skb was accessed on an xdpf
union pointer or the XDP frame was leaked without calling
xdp_return_frame().
Refactor gve_tx_clean_pending_packets(), gve_handle_miss_completion(),
and remove_miss_completions() to switch on pending_packet->type.
Fixes: d8a8ca14c937 ("gve: add XDP_TX and XDP_REDIRECT support for DQ RDA")
Cc: stable@vger.kernel.org
Signed-off-by: Eddie Phillips <eddiephillips@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
---
v2:
- Newly introduced.
---
drivers/net/ethernet/google/gve/gve_tx_dqo.c | 155 +++++++++++++------
1 file changed, 104 insertions(+), 51 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_tx_dqo.c b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
index 78f946ae7264..e8481b993beb 100644
--- a/drivers/net/ethernet/google/gve/gve_tx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
@@ -186,6 +186,49 @@ static void gve_unmap_packet(struct device *dev,
pkt->num_bufs = 0;
}
+static struct gve_tx_pending_packet_dqo *
+gve_xsk_reorder_queue_head(struct gve_tx_ring *tx)
+{
+ u32 head = tx->dqo_compl.xsk_reorder_queue_head;
+
+ if (head == tx->dqo_compl.xsk_reorder_queue_tail) {
+ tx->dqo_compl.xsk_reorder_queue_tail =
+ atomic_read_acquire(&tx->dqo_tx.xsk_reorder_queue_tail);
+
+ if (head == tx->dqo_compl.xsk_reorder_queue_tail)
+ return NULL;
+ }
+
+ return &tx->dqo.pending_packets[tx->dqo.xsk_reorder_queue[head]];
+}
+
+static void gve_xsk_reorder_queue_pop_dqo(struct gve_tx_ring *tx)
+{
+ tx->dqo_compl.xsk_reorder_queue_head++;
+ tx->dqo_compl.xsk_reorder_queue_head &= tx->dqo.complq_mask;
+}
+
+static void gve_tx_process_xsk_completions(struct gve_tx_ring *tx)
+{
+ u32 num_xsks = 0;
+
+ while (true) {
+ struct gve_tx_pending_packet_dqo *pending_packet =
+ gve_xsk_reorder_queue_head(tx);
+
+ if (!pending_packet ||
+ pending_packet->state != GVE_PACKET_STATE_XSK_COMPLETE)
+ break;
+
+ num_xsks++;
+ gve_xsk_reorder_queue_pop_dqo(tx);
+ gve_free_pending_packet(tx, pending_packet);
+ }
+
+ if (num_xsks)
+ xsk_tx_completed(tx->xsk_pool, num_xsks);
+}
+
/* gve_tx_free_desc - Cleans up all pending tx requests and buffers.
*/
static void gve_tx_clean_pending_packets(struct gve_tx_ring *tx)
@@ -201,11 +244,30 @@ static void gve_tx_clean_pending_packets(struct gve_tx_ring *tx)
else
gve_unmap_packet(tx->dev, cur_state);
- if (cur_state->skb) {
- dev_consume_skb_any(cur_state->skb);
- cur_state->skb = NULL;
+ switch (cur_state->type) {
+ case GVE_TX_PENDING_PACKET_DQO_SKB:
+ if (cur_state->skb) {
+ dev_consume_skb_any(cur_state->skb);
+ cur_state->skb = NULL;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XDP_FRAME:
+ if (cur_state->xdpf) {
+ xdp_return_frame(cur_state->xdpf);
+ cur_state->xdpf = NULL;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XSK:
+ cur_state->state = GVE_PACKET_STATE_XSK_COMPLETE;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
}
}
+
+ if (tx->xsk_pool)
+ gve_tx_process_xsk_completions(tx);
}
static void gve_tx_init_ring_state_dqo(struct gve_tx_ring *tx)
@@ -1089,28 +1151,6 @@ static void gve_xsk_reorder_queue_push_dqo(struct gve_tx_ring *tx,
atomic_set_release(&tx->dqo_tx.xsk_reorder_queue_tail, tail);
}
-static struct gve_tx_pending_packet_dqo *
-gve_xsk_reorder_queue_head(struct gve_tx_ring *tx)
-{
- u32 head = tx->dqo_compl.xsk_reorder_queue_head;
-
- if (head == tx->dqo_compl.xsk_reorder_queue_tail) {
- tx->dqo_compl.xsk_reorder_queue_tail =
- atomic_read_acquire(&tx->dqo_tx.xsk_reorder_queue_tail);
-
- if (head == tx->dqo_compl.xsk_reorder_queue_tail)
- return NULL;
- }
-
- return &tx->dqo.pending_packets[tx->dqo.xsk_reorder_queue[head]];
-}
-
-static void gve_xsk_reorder_queue_pop_dqo(struct gve_tx_ring *tx)
-{
- tx->dqo_compl.xsk_reorder_queue_head++;
- tx->dqo_compl.xsk_reorder_queue_head &= tx->dqo.complq_mask;
-}
-
/* Transmit a given skb and ring the doorbell. */
netdev_tx_t gve_tx_dqo(struct sk_buff *skb, struct net_device *dev)
{
@@ -1347,8 +1387,25 @@ static void gve_handle_miss_completion(struct gve_priv *priv,
secs_to_jiffies(GVE_REINJECT_COMPL_TIMEOUT);
add_to_list(tx, &tx->dqo_compl.miss_completions, pending_packet);
- *bytes += pending_packet->skb->len;
- (*pkts)++;
+ switch (pending_packet->type) {
+ case GVE_TX_PENDING_PACKET_DQO_SKB:
+ if (pending_packet->skb) {
+ *bytes += pending_packet->skb->len;
+ (*pkts)++;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XDP_FRAME:
+ if (pending_packet->xdpf) {
+ *bytes += pending_packet->xdpf->len;
+ (*pkts)++;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XSK:
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
}
static void remove_miss_completions(struct gve_priv *priv,
@@ -1377,9 +1434,26 @@ static void remove_miss_completions(struct gve_priv *priv,
else
gve_unmap_packet(tx->dev, pending_packet);
- /* This indicates the packet was dropped. */
- dev_kfree_skb_any(pending_packet->skb);
- pending_packet->skb = NULL;
+ switch (pending_packet->type) {
+ case GVE_TX_PENDING_PACKET_DQO_SKB:
+ if (pending_packet->skb) {
+ /* This indicates the packet was dropped. */
+ dev_kfree_skb_any(pending_packet->skb);
+ pending_packet->skb = NULL;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XDP_FRAME:
+ if (pending_packet->xdpf) {
+ xdp_return_frame(pending_packet->xdpf);
+ pending_packet->xdpf = NULL;
+ }
+ break;
+ case GVE_TX_PENDING_PACKET_DQO_XSK:
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
u64_stats_update_begin(&tx->statss);
tx->dropped_pkt++;
@@ -1426,27 +1500,6 @@ static void remove_timed_out_completions(struct gve_priv *priv,
}
}
-static void gve_tx_process_xsk_completions(struct gve_tx_ring *tx)
-{
- u32 num_xsks = 0;
-
- while (true) {
- struct gve_tx_pending_packet_dqo *pending_packet =
- gve_xsk_reorder_queue_head(tx);
-
- if (!pending_packet ||
- pending_packet->state != GVE_PACKET_STATE_XSK_COMPLETE)
- break;
-
- num_xsks++;
- gve_xsk_reorder_queue_pop_dqo(tx);
- gve_free_pending_packet(tx, pending_packet);
- }
-
- if (num_xsks)
- xsk_tx_completed(tx->xsk_pool, num_xsks);
-}
-
int gve_clean_tx_done_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
struct napi_struct *napi)
{
--
2.55.0.1082.g2b9226bbc0-goog
^ permalink raw reply [flat|nested] 10+ messages in thread