* [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem
@ 2026-11-17 16:46 Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 1/7] bpf: Keep XDP helpers within readable data Björn Töpel
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
Hi!
I had a long chat on page-pool providers with Magnus and Maciej
earlier this week, and the topic of "XDP on unreadable providers" came
up. Magnus suggested building on the xdp.frags mode, to support this.
Here goes...
Devmem and io_uring zero-copy RX place packet payload in memory that the
kernel cannot read through the normal packet APIs. Header split keeps
the headers in ordinary kernel memory. Today, the core rejects XDP and
these memory providers on the same device, in either attach order.
A program that checks headers should not need to copy the payload or
turn off zero-copy receive. Keep the payload in fragments and run a
native XDP program with frags support. Frags support means that the
program accepts a packet whose data_end does not cover the whole
packet. It does not give the program permission to read provider data.
Kuba added XDP_FLAGS_FRAGS_UNREADABLE, which already records this kind
of payload. xdp_buff_add_frag() sets it for net_iov fragments, and skb
construction copies it to skb->unreadable. Drivers use this path even
without an XDP program. The series uses the same flag to stop helpers
from reading or writing provider fragments. Header access still works.
XDP_PASS keeps the payload for the normal receive path. XDP_DROP frees
it without reading it. Frame redirects and AF_XDP copies reject a
buffer while the flag is set. The driver must also reject XDP_TX:
provider payload is mapped for receive, not transmit. Tail trimming
can remove the payload without reading it; after the last fragment is
removed, the remaining header buffer has no unreadable fragments.
Only drivers that declare xdp_unreadable support accept the combination.
fbnic is the real driver in this series. Its XDP_TX path drops packets
while unreadable fragments remain. netdevsim tests the attachment
rules, but it does not test packet access to provider memory.
Generic XDP remains rejected. Its receive path first copies or
linearizes nonlinear skbs, which fails when it reaches provider data.
Even if that step were skipped, skb-to-XDP conversion does not carry
the unreadable flag, and generic transmit and redirect use skb paths
outside the native driver's XDP_TX guard. Supporting generic XDP needs
separate handling; admitting it here would not be enough.
Björn
Björn Töpel (7):
bpf: Keep XDP helpers within readable data
xdp: Reject frames with unreadable fragments
net: Admit native XDP on provider queues
eth: fbnic: Run XDP on provider queues
netdevsim: Run native XDP next to devmem
selftests: net: Test XDP with devmem bindings
selftests: net: Test XDP on io_uring RX
Documentation/networking/netmem.rst | 8 +
MAINTAINERS | 2 +
drivers/net/ethernet/meta/fbnic/fbnic_txrx.c | 4 +-
drivers/net/netdevsim/netdev.c | 2 +
include/linux/filter.h | 9 +-
include/net/netdev_queues.h | 6 +
include/net/xdp.h | 20 +-
kernel/bpf/helpers.c | 4 +-
net/core/dev.c | 28 +-
net/core/dev.h | 1 +
net/core/filter.c | 37 ++-
net/core/netdev_rx_queue.c | 2 +-
net/xdp/xsk.c | 5 +
.../selftests/drivers/net/hw/.gitignore | 1 +
.../testing/selftests/drivers/net/hw/Makefile | 10 +-
tools/testing/selftests/drivers/net/hw/config | 5 +
.../selftests/drivers/net/hw/iou-zcrx.c | 242 +++++++++++++++++-
.../selftests/drivers/net/hw/iou-zcrx.py | 108 +++++++-
.../selftests/drivers/net/hw/iou_zcrx.bpf.c | 111 ++++++++
.../selftests/drivers/net/hw/iou_zcrx.h | 34 +++
tools/testing/selftests/net/nl_netdev.py | 54 +++-
21 files changed, 646 insertions(+), 47 deletions(-)
create mode 100644 tools/testing/selftests/drivers/net/hw/iou_zcrx.bpf.c
create mode 100644 tools/testing/selftests/drivers/net/hw/iou_zcrx.h
base-commit: f23dcd8055612eb37c666c67fe0cfe38b3c65021
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 1/7] bpf: Keep XDP helpers within readable data
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 2/7] xdp: Reject frames with unreadable fragments Björn Töpel
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
Devmem and io_uring zero-copy RX put payload in fragments that the
kernel cannot read. XDP helpers must not treat these fragments as
normal pages.
Reject operations that read or write these fragments. Access to the
linear part still works. Tail trimming does not read the payload;
clear the unreadable flag after it removes every fragment.
Drivers currently use these buffers to build skbs without running XDP.
These checks are needed before native XDP can run on provider queues.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
include/linux/filter.h | 9 +++++----
include/net/xdp.h | 12 ++++++++++++
kernel/bpf/helpers.c | 4 +++-
net/core/filter.c | 37 +++++++++++++++++++++++++------------
4 files changed, 45 insertions(+), 17 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4f0cd67caccb..5c2f57ec3940 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1994,8 +1994,8 @@ int __bpf_skb_store_bytes(struct sk_buff *skb, u32 offset, const void *from,
int __bpf_xdp_load_bytes(struct xdp_buff *xdp, u32 offset, void *buf, u32 len);
int __bpf_xdp_store_bytes(struct xdp_buff *xdp, u32 offset, void *buf, u32 len);
void *bpf_xdp_pointer(struct xdp_buff *xdp, u32 offset, u32 len);
-void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
- void *buf, unsigned long len, bool flush);
+int bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
+ void *buf, unsigned long len, bool flush);
int __bpf_skb_meta_store_bytes(struct sk_buff *skb, u32 offset,
const void *from, u32 len, u64 flags);
void *bpf_skb_meta_pointer(struct sk_buff *skb, u32 offset);
@@ -2029,9 +2029,10 @@ static inline void *bpf_xdp_pointer(struct xdp_buff *xdp, u32 offset, u32 len)
return NULL;
}
-static inline void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off, void *buf,
- unsigned long len, bool flush)
+static inline int bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
+ void *buf, unsigned long len, bool flush)
{
+ return -EOPNOTSUPP;
}
static inline int __bpf_skb_meta_store_bytes(struct sk_buff *skb, u32 offset,
diff --git a/include/net/xdp.h b/include/net/xdp.h
index 07231adfb5f8..16693cf9653e 100644
--- a/include/net/xdp.h
+++ b/include/net/xdp.h
@@ -136,6 +136,18 @@ static __always_inline void xdp_buff_set_frag_unreadable(struct xdp_buff *xdp)
xdp->flags |= XDP_FLAGS_FRAGS_UNREADABLE;
}
+static __always_inline void
+xdp_buff_clear_frag_unreadable(struct xdp_buff *xdp)
+{
+ xdp->flags &= ~XDP_FLAGS_FRAGS_UNREADABLE;
+}
+
+static __always_inline bool
+xdp_buff_is_frag_unreadable(const struct xdp_buff *xdp)
+{
+ return !!(xdp->flags & XDP_FLAGS_FRAGS_UNREADABLE);
+}
+
static __always_inline u32 xdp_buff_get_skb_flags(const struct xdp_buff *xdp)
{
return xdp->flags;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 6788489a2608..8c9e4640bcfe 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3140,7 +3140,9 @@ __bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
if (!buffer__nullable)
return NULL;
- bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
+ if (bpf_xdp_copy_buf(ptr->data, ptr->offset + offset,
+ buffer__nullable, len, false))
+ return NULL;
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
diff --git a/net/core/filter.c b/net/core/filter.c
index f42ecfb5841f..9a55f61fbcdf 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -4269,8 +4269,8 @@ static const struct bpf_func_proto bpf_xdp_adjust_head_proto = {
.arg2_type = ARG_ANYTHING,
};
-void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
- void *buf, unsigned long len, bool flush)
+int bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
+ void *buf, unsigned long len, bool flush)
{
unsigned long ptr_len, ptr_off = 0;
skb_frag_t *next_frag, *end_frag;
@@ -4282,9 +4282,12 @@ void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
src = flush ? buf : xdp->data + off;
dst = flush ? xdp->data + off : buf;
memcpy(dst, src, len);
- return;
+ return 0;
}
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp)))
+ return -EFAULT;
+
sinfo = xdp_get_shared_info_from_buff(xdp);
end_frag = &sinfo->frags[sinfo->nr_frags];
next_frag = &sinfo->frags[0];
@@ -4314,6 +4317,8 @@ void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
ptr_len = skb_frag_size(next_frag);
next_frag++;
}
+
+ return 0;
}
void *bpf_xdp_pointer(struct xdp_buff *xdp, u32 offset, u32 len)
@@ -4332,6 +4337,9 @@ void *bpf_xdp_pointer(struct xdp_buff *xdp, u32 offset, u32 len)
if (likely(offset < size)) /* linear area */
goto out;
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp)))
+ return ERR_PTR(-EFAULT);
+
sinfo = xdp_get_shared_info_from_buff(xdp);
offset -= size;
for (i = 0; i < sinfo->nr_frags; i++) { /* paged area */
@@ -4358,10 +4366,9 @@ BPF_CALL_4(bpf_xdp_load_bytes, struct xdp_buff *, xdp, u32, offset,
return PTR_ERR(ptr);
if (!ptr)
- bpf_xdp_copy_buf(xdp, offset, buf, len, false);
- else
- memmove(buf, ptr, len);
+ return bpf_xdp_copy_buf(xdp, offset, buf, len, false);
+ memmove(buf, ptr, len);
return 0;
}
@@ -4390,11 +4397,9 @@ BPF_CALL_4(bpf_xdp_store_bytes, struct xdp_buff *, xdp, u32, offset,
return PTR_ERR(ptr);
if (!ptr)
- bpf_xdp_copy_buf(xdp, offset, buf, len, true);
- else
- /* through a dynptr the source may be an overlapping slice */
- memmove(ptr, buf, len);
+ return bpf_xdp_copy_buf(xdp, offset, buf, len, true);
+ memmove(ptr, buf, len);
return 0;
}
@@ -4422,6 +4427,8 @@ static int bpf_xdp_frags_increase_tail(struct xdp_buff *xdp, int offset)
if (!rxq->frag_size || rxq->frag_size > xdp->frame_sz)
return -EOPNOTSUPP;
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp)))
+ return -EFAULT;
tailroom = rxq->frag_size - skb_frag_size(frag) -
skb_frag_off(frag) % rxq->frag_size;
@@ -4503,6 +4510,7 @@ static int bpf_xdp_frags_shrink_tail(struct xdp_buff *xdp, int offset)
if (unlikely(!sinfo->nr_frags)) {
xdp_buff_clear_frags_flag(xdp);
xdp_buff_clear_frag_pfmemalloc(xdp);
+ xdp_buff_clear_frag_unreadable(xdp);
xdp->data_end -= offset;
}
@@ -5355,8 +5363,7 @@ static unsigned long bpf_xdp_copy(void *dst, const void *ctx,
{
struct xdp_buff *xdp = (struct xdp_buff *)ctx;
- bpf_xdp_copy_buf(xdp, off, dst, len, false);
- return 0;
+ return bpf_xdp_copy_buf(xdp, off, dst, len, false) ? len : 0;
}
BPF_CALL_5(bpf_xdp_event_output, struct xdp_buff *, xdp, struct bpf_map *, map,
@@ -5369,6 +5376,9 @@ BPF_CALL_5(bpf_xdp_event_output, struct xdp_buff *, xdp, struct bpf_map *, map,
if (unlikely(!xdp || xdp_size > xdp_get_buff_len(xdp)))
return -EFAULT;
+ if (unlikely(xdp_size > xdp->data_end - xdp->data &&
+ xdp_buff_is_frag_unreadable(xdp)))
+ return -EFAULT;
return bpf_event_output(map, flags, meta, meta_size, xdp,
xdp_size, bpf_xdp_copy);
@@ -12797,6 +12807,7 @@ __bpf_kfunc int bpf_sock_ops_enable_tx_tstamp(struct bpf_sock_ops_kern *skops,
* Return:
* * %0 - success
* * %-EINVAL - invalid len
+ * * %-EFAULT - the kernel cannot read the fragments
*/
__bpf_kfunc int bpf_xdp_pull_data(struct xdp_md *x, u32 len)
{
@@ -12812,6 +12823,8 @@ __bpf_kfunc int bpf_xdp_pull_data(struct xdp_md *x, u32 len)
if (unlikely(len > xdp_get_buff_len(xdp)))
return -EINVAL;
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp)))
+ return -EFAULT;
start = xdp_data_meta_unsupported(xdp) ? xdp->data : xdp->data_meta;
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 2/7] xdp: Reject frames with unreadable fragments
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 1/7] bpf: Keep XDP helpers within readable data Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 3/7] net: Admit native XDP on provider queues Björn Töpel
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
An xdp_frame can be sent to another device, processed on another CPU,
or copied into an AF_XDP socket. These paths do not support provider
fragments that the kernel cannot read or remap for transmit.
Reject conversion of such buffers to xdp_frame and reject AF_XDP
copies. A failed redirect leaves buffer ownership with the driver,
which drops and frees the packet.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
include/net/xdp.h | 3 +++
net/xdp/xsk.c | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/include/net/xdp.h b/include/net/xdp.h
index 16693cf9653e..20d173fe163c 100644
--- a/include/net/xdp.h
+++ b/include/net/xdp.h
@@ -432,6 +432,9 @@ int xdp_update_frame_from_buff(const struct xdp_buff *xdp,
return -ENOSPC;
}
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp)))
+ return -EFAULT;
+
xdp_frame->data = xdp->data;
xdp_frame->len = xdp->data_end - xdp->data;
xdp_frame->headroom = headroom - sizeof(*xdp_frame);
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index f22f9a2cdbbe..25b95ec19334 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -489,6 +489,11 @@ static int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
return xsk_rcv_zc(xs, xdp, len);
}
+ if (unlikely(xdp_buff_is_frag_unreadable(xdp))) {
+ xs->rx_dropped++;
+ return -EFAULT;
+ }
+
err = __xsk_rcv(xs, xdp, len);
if (!err)
xdp_return_buff(xdp);
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 3/7] net: Admit native XDP on provider queues
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 1/7] bpf: Keep XDP helpers within readable data Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 2/7] xdp: Reject frames with unreadable fragments Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 4/7] eth: fbnic: Run " Björn Töpel
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
XDP and memory providers are currently rejected on the same device.
Header-split RX keeps headers in normal memory, so a program can
filter packets without reading the provider payload.
Keep this restriction unless the driver declares xdp_unreadable
support. Admit native XDP in either attach order on these drivers.
Existing header-split checks require frags-aware programs.
The driver must mark provider fragments and reject XDP_TX while those
fragments remain. The core rejects helper access and frame redirects.
Keep generic and hardware-offloaded XDP rejected. Generic XDP builds
an xdp_buff from an skb without carrying the unreadable flag. Its
transmit and redirect paths also bypass the native driver guard.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
Documentation/networking/netmem.rst | 8 ++++++++
include/net/netdev_queues.h | 6 ++++++
include/net/xdp.h | 5 ++---
net/core/dev.c | 28 ++++++++++++++++++++++++++--
net/core/dev.h | 1 +
net/core/netdev_rx_queue.c | 2 +-
6 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/Documentation/networking/netmem.rst b/Documentation/networking/netmem.rst
index 6871dcdd700c..1ac94fc9d667 100644
--- a/Documentation/networking/netmem.rst
+++ b/Documentation/networking/netmem.rst
@@ -131,6 +131,14 @@ Driver RX Requirements
circulation times, such as when userspace holds a reference in zerocopy
scenarios.
+8. XDP programs run next to unreadable memory providers only if the driver
+ sets netdev_queue_mgmt_ops.xdp_unreadable, and only in native mode. The
+ programs see the headers in the linear part, and helpers that reach
+ unreadable fragments fail. Attach payload fragments with
+ xdp_buff_add_frag(), which marks unreadable ones, and drop XDP_TX of a
+ buffer for which xdp_buff_is_frag_unreadable() is true: the payload is
+ mapped for receive only. Redirects of such a buffer fail in the core.
+
Driver TX Requirements
======================
diff --git a/include/net/netdev_queues.h b/include/net/netdev_queues.h
index c5335e935b7d..4f6702b284b4 100644
--- a/include/net/netdev_queues.h
+++ b/include/net/netdev_queues.h
@@ -199,6 +199,11 @@ enum {
*
* @supported_params: Bitmask of supported parameters, see QCFG_*.
*
+ * @xdp_unreadable: Native XDP programs may run on queues with unreadable
+ * memory providers. The driver builds XDP buffers with
+ * xdp_buff_add_frag(), and drops XDP_TX of a buffer for
+ * which xdp_buff_is_frag_unreadable() is true.
+ *
* Note that @ndo_queue_mem_alloc and @ndo_queue_mem_free may be called while
* the interface is closed. @ndo_queue_start and @ndo_queue_stop will only
* be called for an interface which is open.
@@ -229,6 +234,7 @@ struct netdev_queue_mgmt_ops {
struct netlink_ext_ack *extack);
unsigned int supported_params;
+ bool xdp_unreadable;
};
void netdev_queue_config(struct net_device *dev, int rxq,
diff --git a/include/net/xdp.h b/include/net/xdp.h
index 20d173fe163c..167f733a0bb0 100644
--- a/include/net/xdp.h
+++ b/include/net/xdp.h
@@ -76,9 +76,8 @@ enum xdp_buff_flags {
XDP_FLAGS_FRAGS_PF_MEMALLOC = BIT(1), /* xdp paged memory is under
* pressure
*/
- /* frags have unreadable mem, this can't be true for real XDP packets,
- * but drivers may use XDP helpers to construct Rx pkt state even when
- * XDP program is not attached.
+ /* frags have unreadable mem: XDP programs reach only the linear part,
+ * and the buffer cannot become an xdp_frame
*/
XDP_FLAGS_FRAGS_UNREADABLE = BIT(2),
};
diff --git a/net/core/dev.c b/net/core/dev.c
index a230ecce9327..f0c3d56d5b83 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -10575,6 +10575,27 @@ u8 dev_xdp_sb_prog_count(struct net_device *dev)
return count;
}
+/* Only native programs run next to unreadable memory providers. Drivers must
+ * mark unreadable fragments and reject XDP_TX while they remain attached.
+ * Tcp-data-split, which these providers need, also needs frags support.
+ */
+static bool netif_xdp_mp_mode_ok(const struct net_device *dev,
+ enum bpf_xdp_mode mode)
+{
+ return mode == XDP_MODE_DRV && dev->queue_mgmt_ops->xdp_unreadable;
+}
+
+/* Can the XDP programs of @dev run next to a new memory provider? */
+bool netif_xdp_mp_ok(struct net_device *dev)
+{
+ int i;
+
+ for (i = 0; i < __MAX_XDP_MODE; i++)
+ if (dev_xdp_prog(dev, i) && !netif_xdp_mp_mode_ok(dev, i))
+ return false;
+ return true;
+}
+
int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
{
if (!dev->netdev_ops->ndo_bpf)
@@ -10588,7 +10609,9 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
return -EBUSY;
}
- if (dev_get_min_mp_channel_count(dev)) {
+ if (dev_get_min_mp_channel_count(dev) &&
+ (bpf->command != XDP_SETUP_PROG ||
+ !netif_xdp_mp_mode_ok(dev, XDP_MODE_DRV))) {
NL_SET_ERR_MSG(bpf->extack, "unable to propagate XDP to device using memory provider");
return -EBUSY;
}
@@ -10664,7 +10687,8 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
return -EBUSY;
}
- if (dev_get_min_mp_channel_count(dev)) {
+ if (dev_get_min_mp_channel_count(dev) &&
+ !netif_xdp_mp_mode_ok(dev, mode)) {
NL_SET_ERR_MSG(extack, "unable to install XDP to device using memory provider");
return -EBUSY;
}
diff --git a/net/core/dev.h b/net/core/dev.h
index 4e9e8cd15697..a7036302f120 100644
--- a/net/core/dev.h
+++ b/net/core/dev.h
@@ -118,6 +118,7 @@ int netdev_queue_config_revalidate(struct net_device *dev,
bool netif_rxq_has_mp(struct net_device *dev, unsigned int rxq_idx);
bool netif_rxq_is_leased(struct net_device *dev, unsigned int rxq_idx);
bool netif_is_queue_leasee(const struct net_device *dev);
+bool netif_xdp_mp_ok(struct net_device *dev);
void __netif_mp_uninstall_rxq(struct netdev_rx_queue *rxq,
const struct pp_memory_provider_params *p);
diff --git a/net/core/netdev_rx_queue.c b/net/core/netdev_rx_queue.c
index a7261fedd5e7..dab5107e31f8 100644
--- a/net/core/netdev_rx_queue.c
+++ b/net/core/netdev_rx_queue.c
@@ -204,7 +204,7 @@ static int __netif_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx,
NL_SET_ERR_MSG(extack, "hds-thresh is not zero");
return -EINVAL;
}
- if (dev_xdp_prog_count(dev)) {
+ if (!netif_xdp_mp_ok(dev)) {
NL_SET_ERR_MSG(extack, "unable to custom memory provider to device with XDP program attached");
return -EEXIST;
}
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 4/7] eth: fbnic: Run XDP on provider queues
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
` (2 preceding siblings ...)
2026-11-17 16:46 ` [PATCH net-next 3/7] net: Admit native XDP on provider queues Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 5/7] netdevsim: Run native XDP next to devmem Björn Töpel
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
fbnic keeps headers in normal pages and attaches provider payload as
XDP fragments. The payload is mapped only for receive and must not be
sent with XDP_TX.
Drop XDP_TX while unreadable fragments remain and declare native XDP
support on provider queues. Normal page-backed packets keep their
existing behavior.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
drivers/net/ethernet/meta/fbnic/fbnic_txrx.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c b/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
index 07d4b2e92362..52f461bdc62a 100644
--- a/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
+++ b/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
@@ -1130,7 +1130,8 @@ static long fbnic_pkt_tx(struct fbnic_napi_vector *nv,
frag = &shinfo->frags[0];
}
- if (fbnic_desc_unused(ring) < nsegs) {
+ if (unlikely(xdp_buff_is_frag_unreadable(&pkt->buff)) ||
+ fbnic_desc_unused(ring) < nsegs) {
u64_stats_update_begin(&ring->stats.syncp);
ring->stats.dropped++;
u64_stats_update_end(&ring->stats.syncp);
@@ -3124,4 +3125,5 @@ const struct netdev_queue_mgmt_ops fbnic_queue_mgmt_ops = {
.ndo_default_qcfg = fbnic_default_qcfg,
.ndo_validate_qcfg = fbnic_validate_qcfg,
.supported_params = QCFG_RX_PAGE_SIZE,
+ .xdp_unreadable = true,
};
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 5/7] netdevsim: Run native XDP next to devmem
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
` (3 preceding siblings ...)
2026-11-17 16:46 ` [PATCH net-next 4/7] eth: fbnic: Run " Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 6/7] selftests: net: Test XDP with devmem bindings Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 7/7] selftests: net: Test XDP on io_uring RX Björn Töpel
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
The attachment rules need a test driver that accepts native XDP and
devmem bindings together. netdevsim runs XDP on ordinary skbs sent by
its peer, not on provider payload.
Declare native XDP support alongside devmem bindings so selftests can
check both attach orders. This tests configuration, not access to
provider data.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
drivers/net/netdevsim/netdev.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c
index 0372a9bd63bb..3d1657d24f60 100644
--- a/drivers/net/netdevsim/netdev.c
+++ b/drivers/net/netdevsim/netdev.c
@@ -880,6 +880,8 @@ static const struct netdev_queue_mgmt_ops nsim_queue_mgmt_ops = {
.ndo_queue_mem_free = nsim_queue_mem_free,
.ndo_queue_start = nsim_queue_start,
.ndo_queue_stop = nsim_queue_stop,
+ /* XDP runs on the skbs of the peer, never on provider memory */
+ .xdp_unreadable = true,
};
static ssize_t
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 6/7] selftests: net: Test XDP with devmem bindings
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
` (4 preceding siblings ...)
2026-11-17 16:46 ` [PATCH net-next 5/7] netdevsim: Run native XDP next to devmem Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 7/7] selftests: net: Test XDP on io_uring RX Björn Töpel
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
The XDP and devmem attachment rules must work in either order. A
native program with frags support should coexist with a devmem RX
binding; generic and single-buffer programs must remain rejected.
Test these rules on netdevsim with header split enabled. The test does
not send packets with provider fragments.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
tools/testing/selftests/net/nl_netdev.py | 54 +++++++++++++++++++++++-
1 file changed, 52 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/nl_netdev.py b/tools/testing/selftests/net/nl_netdev.py
index dd0321c0b463..15326fb66362 100755
--- a/tools/testing/selftests/net/nl_netdev.py
+++ b/tools/testing/selftests/net/nl_netdev.py
@@ -13,7 +13,8 @@ import struct
from os import system
from lib.py import ksft_run, ksft_exit, KsftFailEx, KsftSkipEx
from lib.py import ksft_eq, ksft_ge, ksft_ne, ksft_raises, ksft_busy_wait
-from lib.py import EthtoolFamily, NetdevFamily, NetdevSimDev, NlError, defer, ip
+from lib.py import CmdExitFailure, EthtoolFamily, NetdevFamily, NetdevSimDev
+from lib.py import NlError, defer, ip
def empty_check(nf) -> None:
@@ -483,6 +484,54 @@ def devmem_bind_tx_unregister_check(_nf) -> None:
nf_priv.close()
+def devmem_xdp_check(_nf) -> None:
+ """Native XDP with frags runs next to a devmem binding, in either order."""
+ prog = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ "lib", "xdp_dummy.bpf.o")
+ if not os.path.exists(prog):
+ raise KsftSkipEx(f"{prog} is not built")
+
+ dmabuf_fd = _create_udmabuf()
+ defer(os.close, dmabuf_fd)
+ nf_priv = NetdevFamily()
+ defer(nf_priv.close)
+ nsimdev = NetdevSimDev(queue_count=2)
+ defer(nsimdev.remove)
+ nsim = nsimdev.nsims[0]
+ dev = nsim.ifname
+
+ ip(f"link set dev {dev} up")
+ with EthtoolFamily() as ef:
+ ef.rings_set({"header": {"dev-index": nsim.ifindex},
+ "tcp-data-split": "enabled"})
+
+ # A generic program keeps the provider away
+ ip(f"link set dev {dev} xdpgeneric obj {prog} sec xdp.frags")
+ with ksft_raises(NlError) as cm:
+ nf_priv.bind_rx({"ifindex": nsim.ifindex, "fd": dmabuf_fd,
+ "queues": [{"id": 1, "type": "rx"}]})
+ ksft_eq(cm.exception.nl_msg.error, -errno.EEXIST)
+ ip(f"link set dev {dev} xdpgeneric off")
+
+ # A native program first, then the provider
+ ip(f"link set dev {dev} xdpdrv obj {prog} sec xdp.frags")
+ _devmem_bind(nf_priv.bind_rx, {
+ "ifindex": nsim.ifindex,
+ "fd": dmabuf_fd,
+ "queues": [{"id": 1, "type": "rx"}],
+ })
+ ip(f"link set dev {dev} xdpdrv off")
+
+ # The provider first: native programs with frags only. Header split
+ # already refuses single-buffer programs.
+ ip(f"link set dev {dev} xdpdrv obj {prog} sec xdp.frags")
+ ip(f"link set dev {dev} xdpdrv off")
+ with ksft_raises(CmdExitFailure):
+ ip(f"link set dev {dev} xdpdrv obj {prog} sec xdp")
+ with ksft_raises(CmdExitFailure):
+ ip(f"link set dev {dev} xdpgeneric obj {prog} sec xdp.frags")
+
+
def main() -> None:
""" Ksft boiler plate main """
nf = NetdevFamily()
@@ -498,7 +547,8 @@ def main() -> None:
page_pool_ifindex_leak_check,
page_pool_stats_ifindex_check,
devmem_bind_rx_unregister_check,
- devmem_bind_tx_unregister_check
+ devmem_bind_tx_unregister_check,
+ devmem_xdp_check
],
args=(nf, ))
ksft_exit()
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net-next 7/7] selftests: net: Test XDP on io_uring RX
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
` (5 preceding siblings ...)
2026-11-17 16:46 ` [PATCH net-next 6/7] selftests: net: Test XDP with devmem bindings Björn Töpel
@ 2026-11-17 16:46 ` Björn Töpel
6 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2026-11-17 16:46 UTC (permalink / raw)
To: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jonathan Corbet, Alexander Duyck, Andrew Lunn,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi,
Jesper Dangaard Brouer, Stanislav Fomichev, Magnus Karlsson,
Maciej Fijalkowski, Pavel Begunkov, netdev, bpf
Cc: Björn Töpel, Simon Horman, Shuah Khan, Randy Dunlap,
kernel-team, John Fastabend, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Weiming Shi, Nikolay Aleksandrov, David Wei, Bobby Eshleman,
linux-doc, linux-kernel, linux-kselftest, io-uring, Mina Almasry
Native XDP must not access io_uring payload through normal packet
helpers. Compare ordinary page-backed RX with an io_uring RX queue.
Check header, fragment and boundary access, and verify the received
bytes through recv() and RECV_ZC.
Probe support by binding io_uring first and attaching a frags-aware
XDP_PASS program. Skip unsupported combinations, but fail later
attach, verifier and data-path checks. Require non-linear packets on
the selected queue. Use XDP links so the test cannot replace another
program, and restore header split, RSS and the flow rule.
An AI review found that ignoring io_uring setup errors could turn them
into capability skips. A process-local seccomp filter returning EPERM
reproduced this. The new XDP test now treats setup errors as failures.
Signed-off-by: Björn Töpel <bjorn@kernel.org>
---
MAINTAINERS | 2 +
.../selftests/drivers/net/hw/.gitignore | 1 +
.../testing/selftests/drivers/net/hw/Makefile | 10 +-
tools/testing/selftests/drivers/net/hw/config | 5 +
.../selftests/drivers/net/hw/iou-zcrx.c | 242 +++++++++++++++++-
.../selftests/drivers/net/hw/iou-zcrx.py | 108 +++++++-
.../selftests/drivers/net/hw/iou_zcrx.bpf.c | 111 ++++++++
.../selftests/drivers/net/hw/iou_zcrx.h | 34 +++
8 files changed, 492 insertions(+), 21 deletions(-)
create mode 100644 tools/testing/selftests/drivers/net/hw/iou_zcrx.bpf.c
create mode 100644 tools/testing/selftests/drivers/net/hw/iou_zcrx.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 32e218c0fe68..ea383dd81668 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -14076,6 +14076,8 @@ T: git https://github.com/isilence/linux.git zcrx/for-next
T: git git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git
S: Maintained
F: io_uring/zcrx.*
+F: tools/testing/selftests/drivers/net/hw/iou-zcrx.*
+F: tools/testing/selftests/drivers/net/hw/iou_zcrx.*
IO_URING BPF-OPS
M: Pavel Begunkov <asml.silence@gmail.com>
diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
index 911a9bfeaf41..085aa8b17afc 100644
--- a/tools/testing/selftests/drivers/net/hw/.gitignore
+++ b/tools/testing/selftests/drivers/net/hw/.gitignore
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: GPL-2.0-only
iou-zcrx
+iou_zcrx.skel.h
ncdevmem
tls_hw_offload
toeplitz
diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile
index 429cb545a4f0..d7a1fea08ea7 100644
--- a/tools/testing/selftests/drivers/net/hw/Makefile
+++ b/tools/testing/selftests/drivers/net/hw/Makefile
@@ -10,7 +10,8 @@ HAS_IOURING_ZCRX := $(shell \
echo -e '#include <liburing.h>\n' \
'void *func = (void *)io_uring_register_ifq;\n' \
'int main() {return 0;}' | \
- $(CC) -luring -x c - -o /dev/null 2>&1 && echo y)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(USERCFLAGS) $(LDFLAGS) \
+ $(USERLDFLAGS) -x c - -o /dev/null -luring 2>&1 && echo y)
ifeq ($(HAS_IOURING_ZCRX),y)
COND_GEN_FILES += iou-zcrx
@@ -98,5 +99,10 @@ include ../../../net/ynl.mk
include ../../../net/bpf.mk
ifeq ($(HAS_IOURING_ZCRX),y)
-$(OUTPUT)/iou-zcrx: LDLIBS += -luring -lpthread
+$(OUTPUT)/iou-zcrx: CFLAGS += -I$(OUTPUT) -I$(SCRATCH_DIR)/include
+$(OUTPUT)/iou-zcrx: LDLIBS += -luring -lpthread -lelf -lz
+$(OUTPUT)/iou-zcrx: LOCAL_HDRS += iou_zcrx.h $(OUTPUT)/iou_zcrx.skel.h
+$(OUTPUT)/iou-zcrx: iou_zcrx.h $(OUTPUT)/iou_zcrx.skel.h $(BPFOBJ)
endif
+
+$(OUTPUT)/iou_zcrx.bpf.o: iou_zcrx.h $(selfdir)/bpf/bpf_kfuncs.h
diff --git a/tools/testing/selftests/drivers/net/hw/config b/tools/testing/selftests/drivers/net/hw/config
index 081155bf7847..ef77543b567f 100644
--- a/tools/testing/selftests/drivers/net/hw/config
+++ b/tools/testing/selftests/drivers/net/hw/config
@@ -1,4 +1,6 @@
+CONFIG_BPF_JIT=y
CONFIG_BPF_SYSCALL=y
+CONFIG_DEBUG_INFO_BTF=y
CONFIG_FAIL_FUNCTION=y
CONFIG_FAULT_INJECTION=y
CONFIG_FAULT_INJECTION_DEBUG_FS=y
@@ -9,6 +11,7 @@ CONFIG_INET6_ESP_OFFLOAD=y
CONFIG_INET_ESP=y
CONFIG_INET_ESP_OFFLOAD=y
CONFIG_IO_URING=y
+CONFIG_IO_URING_ZCRX=y
CONFIG_IPV6=y
CONFIG_IPV6_GRE=y
CONFIG_IPV6_SIT=y
@@ -20,7 +23,9 @@ CONFIG_NET_IPGRE=y
CONFIG_NET_IPGRE_DEMUX=y
CONFIG_NET_IPIP=y
CONFIG_NETKIT=y
+CONFIG_NET_RX_BUSY_POLL=y
CONFIG_NET_SCH_INGRESS=y
+CONFIG_PAGE_POOL=y
CONFIG_SYNC_FILE=y
CONFIG_TLS=y
CONFIG_TLS_DEVICE=y
diff --git a/tools/testing/selftests/drivers/net/hw/iou-zcrx.c b/tools/testing/selftests/drivers/net/hw/iou-zcrx.c
index 91b025a8771e..f7e909fd6840 100644
--- a/tools/testing/selftests/drivers/net/hw/iou-zcrx.c
+++ b/tools/testing/selftests/drivers/net/hw/iou-zcrx.c
@@ -16,6 +16,7 @@
#include <linux/mman.h>
#include <linux/errqueue.h>
#include <linux/if_packet.h>
+#include <linux/if_link.h>
#include <linux/ipv6.h>
#include <linux/socket.h>
#include <linux/sockios.h>
@@ -37,9 +38,13 @@
#include <sys/un.h>
#include <sys/wait.h>
+#include <bpf/bpf.h>
#include <liburing.h>
#include <ynl.h>
+#include "kselftest.h"
#include "netdev-user.h"
+#include "iou_zcrx.h"
+#include "iou_zcrx.skel.h"
#define SKIP_CODE 42
@@ -89,7 +94,11 @@ static struct sockaddr_in6 cfg_addr;
static unsigned int cfg_rx_buf_len;
static bool cfg_dry_run;
static int cfg_num_threads = 1;
+static bool cfg_xdp;
+static bool cfg_pages;
+static struct iou_zcrx *xdp_skel;
+static int xdp_link = -1;
static char *payload;
#define MAX_CONNS_PER_THREAD 64
@@ -154,6 +163,139 @@ static inline size_t get_refill_ring_size(unsigned int rq_entries)
return ALIGN_UP(size, page_size);
}
+static int probe_xdp(void)
+{
+ const struct bpf_insn insns[] = {
+ {
+ .code = BPF_ALU64 | BPF_MOV | BPF_K,
+ .dst_reg = BPF_REG_0,
+ .imm = XDP_PASS,
+ },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, load_opts,
+ .prog_flags = BPF_F_XDP_HAS_FRAGS);
+ LIBBPF_OPTS(bpf_link_create_opts, link_opts,
+ .flags = XDP_FLAGS_DRV_MODE);
+ unsigned int ifindex;
+ int prog_fd, link_fd;
+
+ ifindex = if_nametoindex(cfg_ifname);
+ if (!ifindex) {
+ error(0, errno, "if_nametoindex()");
+ return EXIT_FAILURE;
+ }
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_XDP, "zcrx_probe", "GPL",
+ insns, ARRAY_SIZE(insns), &load_opts);
+ if (prog_fd < 0) {
+ error(0, -prog_fd, "load XDP probe");
+ return EXIT_FAILURE;
+ }
+
+ /* The io_uring queue is already registered; never replace a program. */
+ link_fd = bpf_link_create(prog_fd, ifindex, BPF_XDP, &link_opts);
+ close(prog_fd);
+ if (link_fd >= 0) {
+ close(link_fd);
+ return 0;
+ }
+ if (link_fd == -EBUSY || link_fd == -EOPNOTSUPP) {
+ printf("Native XDP with ZCRX is not supported: %s\n",
+ strerror(-link_fd));
+ return SKIP_CODE;
+ }
+ error(0, -link_fd, "attach XDP probe");
+ return EXIT_FAILURE;
+}
+
+static void setup_xdp(void)
+{
+ LIBBPF_OPTS(bpf_link_create_opts, opts,
+ .flags = XDP_FLAGS_DRV_MODE);
+ unsigned int ifindex;
+ int prog_fd;
+
+ ifindex = if_nametoindex(cfg_ifname);
+ if (!ifindex)
+ error(1, errno, "if_nametoindex()");
+ xdp_skel = iou_zcrx__open();
+ if (!xdp_skel)
+ error(1, errno, "iou_zcrx__open()");
+ xdp_skel->bss->port = cfg_port;
+ xdp_skel->bss->queue_id = cfg_queue_id;
+ xdp_skel->bss->unreadable = !cfg_pages;
+ if (iou_zcrx__load(xdp_skel))
+ error(1, errno, "iou_zcrx__load()");
+
+ /* A link cannot replace another program and closes with the process. */
+ prog_fd = bpf_program__fd(xdp_skel->progs.iou_zcrx);
+ xdp_link = bpf_link_create(prog_fd, ifindex, BPF_XDP, &opts);
+ if (xdp_link < 0)
+ error(1, errno, "attach native XDP");
+}
+
+static void check_xdp(void)
+{
+ static const char * const locations[] = {
+ "header", "fragment", "boundary",
+ };
+ static const char * const ops[] = {
+ "load", "store", "dynptr read", "dynptr write",
+ "slice", "slice rdwr",
+ };
+ struct xdp_zcrx_stats sum = {}, *all, *s;
+ int ncpus, cpu, loc, op;
+ __u32 key = 0;
+ bool failed;
+
+ close(xdp_link);
+ xdp_link = -1;
+ ncpus = libbpf_num_possible_cpus();
+ if (ncpus < 0)
+ error(1, 0, "libbpf_num_possible_cpus(): %d", ncpus);
+ all = calloc(ncpus, sizeof(*all));
+ if (!all)
+ error(1, errno, "calloc XDP statistics");
+ if (bpf_map_lookup_elem(bpf_map__fd(xdp_skel->maps.stats), &key, all))
+ error(1, errno, "read XDP statistics");
+
+ for (cpu = 0; cpu < ncpus; cpu++) {
+ s = &all[cpu];
+ sum.packets += s->packets;
+ sum.tested += s->tested;
+ sum.wrong_queue += s->wrong_queue;
+ sum.bad_headers += s->bad_headers;
+ sum.linear_bytes += s->linear_bytes;
+ sum.frag_bytes += s->frag_bytes;
+ for (loc = 0; loc < XDP_ZCRX_LOCATIONS; loc++) {
+ for (op = 0; op < XDP_ZCRX_OPS; op++)
+ sum.errors[loc][op] += s->errors[loc][op];
+ }
+ }
+ free(all);
+
+ printf("XDP %s: queue=%d packets=%llu tested=%llu\n",
+ cfg_pages ? "pages" : "io_uring", cfg_queue_id,
+ sum.packets, sum.tested);
+ printf("XDP geometry: linear=%llu frags=%llu\n",
+ sum.linear_bytes, sum.frag_bytes);
+ printf("XDP errors: wrong_queue=%llu bad_headers=%llu\n",
+ sum.wrong_queue, sum.bad_headers);
+ failed = !sum.tested || sum.wrong_queue || sum.bad_headers;
+ for (loc = 0; loc < XDP_ZCRX_LOCATIONS; loc++) {
+ for (op = 0; op < XDP_ZCRX_OPS; op++) {
+ if (!sum.errors[loc][op])
+ continue;
+ fprintf(stderr, "%s %s: %llu failures\n",
+ locations[loc], ops[op], sum.errors[loc][op]);
+ failed = true;
+ }
+ }
+ iou_zcrx__destroy(xdp_skel);
+ if (failed)
+ error(1, 0, "XDP access checks failed");
+}
+
static void setup_zcrx(struct thread_ctx *ctx)
{
unsigned int rq_entries = AREA_SIZE / page_size;
@@ -217,6 +359,10 @@ static void setup_zcrx(struct thread_ctx *ctx)
};
ret = io_uring_register_ifq(&ctx->ring, (void *)®);
+ if (cfg_xdp && cfg_dry_run && ret == -EOPNOTSUPP) {
+ puts("Interface-bound ZCRX is not supported");
+ exit(SKIP_CODE);
+ }
if (cfg_rx_buf_len && (ret == -EINVAL || ret == -EOPNOTSUPP ||
ret == -ERANGE)) {
printf("Large chunks are not supported %i\n", ret);
@@ -258,6 +404,34 @@ static void add_recvzc_oneshot(struct thread_ctx *ctx, int conn_idx, size_t len)
sqe->user_data = conn_idx;
}
+static void verify_data(struct thread_ctx *ctx, int conn_idx,
+ const char *data, size_t len)
+{
+ size_t offset = ctx->received[conn_idx];
+ size_t i;
+
+ for (i = 0; i < len; i++) {
+ if (data[i] != payload[offset + i])
+ error(1, 0, "payload mismatch at %zu", i);
+ }
+ ctx->received[conn_idx] += len;
+}
+
+static void recv_pages(struct thread_ctx *ctx)
+{
+ char buf[8192];
+ ssize_t ret;
+
+ while ((ret = recv(ctx->connfds[0], buf, sizeof(buf), 0)) > 0) {
+ if ((size_t)ret > (size_t)cfg_send_size - ctx->received[0])
+ error(1, 0, "received too much data");
+ verify_data(ctx, 0, buf, ret);
+ }
+ if (ret < 0)
+ error(1, errno, "recv()");
+ ctx->nr_conns--;
+}
+
static void process_recvzc(struct thread_ctx *ctx, struct io_uring_cqe *cqe,
int conn_idx)
{
@@ -267,7 +441,6 @@ static void process_recvzc(struct thread_ctx *ctx, struct io_uring_cqe *cqe,
uint64_t mask;
char *data;
ssize_t n;
- int i;
if (cqe->res == 0 && cqe->flags == 0 &&
ctx->oneshot_recvs[conn_idx] == 0) {
@@ -295,11 +468,7 @@ static void process_recvzc(struct thread_ctx *ctx, struct io_uring_cqe *cqe,
mask = (1ULL << IORING_ZCRX_AREA_SHIFT) - 1;
data = (char *)ctx->area_ptr + (rcqe->off & mask);
- for (i = 0; i < n; i++) {
- if (*(data + i) != payload[(ctx->received[conn_idx] + i)])
- error(1, 0, "payload mismatch at %d", i);
- }
- ctx->received[conn_idx] += n;
+ verify_data(ctx, conn_idx, data, n);
rqe = &ctx->rq_ring.rqes[(ctx->rq_ring.rq_tail & rq_mask)];
rqe->off = (rcqe->off & ~IORING_ZCRX_AREA_MASK) | ctx->area_token;
@@ -328,7 +497,10 @@ static void *server_worker(void *arg)
struct thread_ctx *ctx = arg;
uint64_t tstop;
int nr_conns;
- int i;
+ int ret, i;
+
+ if (cfg_pages)
+ goto ready;
params.flags |= IORING_SETUP_COOP_TASKRUN;
params.flags |= IORING_SETUP_SINGLE_ISSUER;
@@ -338,16 +510,23 @@ static void *server_worker(void *arg)
params.flags |= IORING_SETUP_CQSIZE;
params.cq_entries = AREA_SIZE / page_size;
- io_uring_queue_init_params(512, &ctx->ring, ¶ms);
+ ret = io_uring_queue_init_params(512, &ctx->ring, ¶ms);
+ if (cfg_xdp && ret)
+ error(1, 0, "io_uring_queue_init_params(): %d", ret);
setup_zcrx(ctx);
if (cfg_dry_run)
return NULL;
+ready:
pthread_barrier_wait(ctx->setup_done);
pthread_barrier_wait(ctx->dispatch_done);
nr_conns = ctx->nr_conns;
+ if (cfg_pages) {
+ recv_pages(ctx);
+ goto check_received;
+ }
for (i = 0; i < ctx->nr_conns; i++) {
if (cfg_oneshot) {
@@ -366,6 +545,7 @@ static void *server_worker(void *arg)
error(1, 0, "test failed: %d connections incomplete",
ctx->nr_conns);
+check_received:
for (i = 0; i < nr_conns; i++) {
if (cfg_oneshot) {
if (!ctx->received[i])
@@ -374,8 +554,11 @@ static void *server_worker(void *arg)
error(1, 0, "connection %d received %zu of %d bytes",
i, ctx->received[i], cfg_send_size);
}
+ close(ctx->connfds[i]);
}
-
+ if (cfg_xdp)
+ printf("Verified %zu bytes through %s\n", ctx->received[0],
+ cfg_pages ? "recv" : "RECV_ZC");
return NULL;
}
@@ -434,11 +617,14 @@ static void run_server(void)
{
pthread_barrier_t setup_done, dispatch_done;
int total_conns, accepted = 0, connfd;
+ int fd, ret, enable, i, result = 0;
struct thread_ctx *ctxs;
- int fd, ret, enable, i;
unsigned int ifindex;
pthread_t *threads;
+ if (cfg_xdp && !cfg_dry_run)
+ setup_xdp();
+
ctxs = calloc(cfg_num_threads, sizeof(*ctxs));
threads = calloc(cfg_num_threads, sizeof(*threads));
if (!ctxs || !threads)
@@ -514,11 +700,26 @@ static void run_server(void)
for (i = 0; i < cfg_num_threads; i++)
pthread_join(threads[i], NULL);
+ if (cfg_xdp) {
+ if (cfg_dry_run)
+ result = probe_xdp();
+ else
+ check_xdp();
+ }
+ if (!cfg_pages) {
+ for (i = 0; i < cfg_num_threads; i++) {
+ io_uring_queue_exit(&ctxs[i].ring);
+ munmap(ctxs[i].ring_ptr, ctxs[i].ring_size);
+ munmap(ctxs[i].area_ptr, AREA_SIZE);
+ }
+ }
pthread_barrier_destroy(&setup_done);
pthread_barrier_destroy(&dispatch_done);
close(fd);
free(threads);
free(ctxs);
+ if (result)
+ exit(result);
}
static void *client_worker(void *arg)
@@ -574,8 +775,9 @@ static void run_client(void)
static void usage(const char *filepath)
{
- error(1, 0, "Usage: %s (-4|-6) (-s|-c) -h<server_ip> -p<port> "
- "-l<payload_size> -i<ifname> -q<rxq_id> -t<num_threads>",
+ error(1, 0, "Usage: %s (-s|-c) -h<server_ip> -p<port>\n"
+ " -l<payload_size> -i<ifname> -q<rxq_id>\n"
+ " -t<num_threads> -z<total_size> -X (XDP) -P (pages)",
filepath);
}
@@ -594,7 +796,7 @@ static void parse_opts(int argc, char **argv)
usage(argv[0]);
cfg_payload_len = max_payload_len;
- while ((c = getopt(argc, argv, "sch:p:l:i:q:o:z:x:dt:")) != -1) {
+ while ((c = getopt(argc, argv, "sch:p:l:i:q:o:z:x:dt:XP")) != -1) {
switch (c) {
case 's':
if (cfg_client)
@@ -638,9 +840,23 @@ static void parse_opts(int argc, char **argv)
case 't':
cfg_num_threads = strtoul(optarg, NULL, 0);
break;
+ case 'X':
+ cfg_xdp = true;
+ break;
+ case 'P':
+ cfg_pages = true;
+ break;
}
}
+ if ((cfg_xdp || cfg_pages) &&
+ (!cfg_server || cfg_num_threads != 1 || cfg_oneshot ||
+ !cfg_ifname || cfg_queue_id < 0 || (cfg_pages && cfg_dry_run)))
+ error(1, 0, "-X/-P need one server connection and an RX queue");
+ if ((cfg_xdp || cfg_pages) &&
+ (cfg_send_size <= 0 || cfg_send_size > SEND_SIZE))
+ error(1, 0, "-X/-P: invalid total size");
+
if (cfg_server && addr)
error(1, 0, "Receiver cannot have -h specified");
diff --git a/tools/testing/selftests/drivers/net/hw/iou-zcrx.py b/tools/testing/selftests/drivers/net/hw/iou-zcrx.py
index 5047fdb56005..e6b765c523ed 100755
--- a/tools/testing/selftests/drivers/net/hw/iou-zcrx.py
+++ b/tools/testing/selftests/drivers/net/hw/iou-zcrx.py
@@ -1,24 +1,34 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
+import errno
import re
import resource
import time
+from enum import Enum
from os import path
from lib.py import ksft_run, ksft_exit, KsftSkipEx, ksft_variants, KsftNamedVariant
+from lib.py import ksft_disruptive, ksft_pr, KsftFailEx
from lib.py import NetDrvEpEnv
-from lib.py import bkg, cmd, defer, ethtool, rand_port, wait_port_listen
-from lib.py import EthtoolFamily, NetdevFamily
+from lib.py import bkg, cmd, defer, ethtool, ip, rand_port, wait_port_listen
+from lib.py import EthtoolFamily, NetdevFamily, NlError
SKIP_CODE = 42
-def mp_clear_wait(cfg):
- """Wait for io_uring memory providers to clear from all device queues."""
+class RxMode(Enum):
+ """Receive modes used by the XDP access test."""
+ PAGES = '-P'
+ ZCRX = ''
+
+
+def mp_clear_wait(cfg, queue_id=None):
+ """Wait for io_uring memory providers to clear from device queues."""
deadline = time.time() + 5
while time.time() < deadline:
queues = cfg.netnl.queue_get({'ifindex': cfg.ifindex}, dump=True)
- if not any('io-uring' in q for q in queues):
+ if not any('io-uring' in q for q in queues
+ if queue_id is None or q['id'] == queue_id):
return
time.sleep(0.1)
raise TimeoutError("Timed out waiting for memory provider to clear")
@@ -243,6 +253,91 @@ def test_zcrx_large_chunks(cfg) -> None:
cmd(tx_cmd, host=cfg.remote)
+def xdp_setup(cfg):
+ """Reserve one RX queue without changing unrelated RSS contexts."""
+ cfg.require_ipver('6')
+ dev = cfg.netnl.dev_get({'ifindex': cfg.ifindex})
+ if not {'basic', 'rx-sg'} <= set(dev.get('xdp-features', [])):
+ raise KsftSkipEx('Device lacks native multi-buffer XDP')
+ if 'xdp' in ip(f"-d link show dev {cfg.ifname}", json=True)[0]:
+ raise KsftSkipEx('An XDP program is already attached')
+
+ queues = cfg.netnl.queue_get({'ifindex': cfg.ifindex}, dump=True)
+ if any(key in q for q in queues
+ for key in ('io-uring', 'dmabuf', 'xsk', 'lease')):
+ raise KsftSkipEx('Device has an active queue provider or lease')
+
+ header = {'dev-index': cfg.ifindex}
+ try:
+ channels = cfg.ethnl.channels_get({'header': header})
+ rings = cfg.ethnl.rings_get({'header': header})
+ indir = cfg.ethnl.rss_get({'header': header})['indir']
+ except NlError as error:
+ if error.error != errno.EOPNOTSUPP:
+ raise
+ raise KsftSkipEx('Device lacks channels, rings or RSS support') from error
+ count = channels.get('combined-count', 0)
+ if count < 2:
+ raise KsftSkipEx('Test requires at least two combined channels')
+ if rings.get('tcp-data-split', 'unknown') == 'unknown':
+ raise KsftSkipEx('Device lacks TCP header/data split')
+
+ _require_ntuple(cfg)
+ cfg.target = count - 1
+ old_rings = {'header': header,
+ 'tcp-data-split': rings['tcp-data-split']}
+ new_rings = {'header': header, 'tcp-data-split': 'enabled'}
+ if 'hds-thresh' in rings:
+ old_rings['hds-thresh'] = rings['hds-thresh']
+ new_rings['hds-thresh'] = 0
+ cfg.ethnl.rings_set(new_rings)
+ defer(cfg.ethnl.rings_set, old_rings)
+
+ cfg.ethnl.rss_set({'header': header,
+ 'indir': [q if q != cfg.target else 0 for q in indir]})
+ defer(cfg.ethnl.rss_set, {'header': header, 'indir': indir})
+ flow_rule_id = set_flow_rule(cfg)
+ defer(ethtool, f"-N {cfg.ifname} delete {flow_rule_id}")
+ defer(mp_clear_wait, cfg, cfg.target)
+
+
+def xdp_exchange(cfg, mode):
+ """Receive verified TCP data with the XDP access checks enabled."""
+ size = 64 * 1024
+ rx_cmd = (f"{cfg.bin_local} -s -X -p {cfg.port} -i {cfg.ifname} "
+ f"-q {cfg.target} -z {size} {mode.value}")
+ tx_cmd = (f"{cfg.bin_remote} -c -h {cfg.addr_v['6']} -p {cfg.port} "
+ f"-l 4096 -z {size}")
+ with bkg(rx_cmd, exit_wait=True) as receiver:
+ wait_port_listen(cfg.port, proto='tcp')
+ queue = cfg.netnl.queue_get({'ifindex': cfg.ifindex, 'type': 'rx',
+ 'id': cfg.target})
+ if ('io-uring' in queue) != (mode is RxMode.ZCRX):
+ raise KsftFailEx(f'Unexpected RX queue binding: {queue}')
+ ksft_pr(f'RX queue: {queue}')
+ cmd(tx_cmd, host=cfg.remote)
+ ksft_pr(receiver.stdout.strip())
+
+
+@ksft_disruptive
+def test_zcrx_xdp(cfg):
+ """Compare page-backed and io_uring-backed native XDP fragments."""
+ xdp_setup(cfg)
+
+ # Bind ZCRX first, then probe native XDP with a harmless frags program.
+ probe = cmd(f"{cfg.bin_local} -s -d -X -i {cfg.ifname} "
+ f"-q {cfg.target} -p {cfg.port}", fail=False)
+ if probe.ret == SKIP_CODE:
+ raise KsftSkipEx(probe.stdout.strip())
+ if probe.ret:
+ raise KsftFailEx(str(probe))
+ mp_clear_wait(cfg, cfg.target)
+
+ # After a successful probe, both attach orders must work.
+ xdp_exchange(cfg, RxMode.PAGES)
+ xdp_exchange(cfg, RxMode.ZCRX)
+
+
def main() -> None:
with NetDrvEpEnv(__file__) as cfg:
cfg.bin_local = path.abspath(path.dirname(__file__) + "/../../../drivers/net/hw/iou-zcrx")
@@ -252,7 +347,8 @@ def main() -> None:
cfg.netnl = NetdevFamily()
cfg.port = rand_port()
ksft_run(globs=globals(), cases=[test_zcrx, test_zcrx_oneshot,
- test_zcrx_large_chunks], args=(cfg, ))
+ test_zcrx_large_chunks,
+ test_zcrx_xdp], args=(cfg, ))
ksft_exit()
diff --git a/tools/testing/selftests/drivers/net/hw/iou_zcrx.bpf.c b/tools/testing/selftests/drivers/net/hw/iou_zcrx.bpf.c
new file mode 100644
index 000000000000..4d22ca8b87b0
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/iou_zcrx.bpf.c
@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/bpf.h>
+#include <linux/errno.h>
+#include <linux/if_ether.h>
+#include <linux/in.h>
+#include <linux/ipv6.h>
+#include <linux/tcp.h>
+#include <stdbool.h>
+#include <bpf/bpf_endian.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_kfuncs.h"
+#include "iou_zcrx.h"
+
+__u32 port;
+__u32 queue_id;
+int unreadable;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, struct xdp_zcrx_stats);
+} stats SEC(".maps");
+
+static __always_inline void
+check_access(struct xdp_md *ctx, const struct bpf_dynptr *ptr,
+ struct xdp_zcrx_stats *s, enum xdp_zcrx_location loc,
+ __u32 offset, int expected)
+{
+ __u32 value = 0;
+ int ret;
+
+ ret = bpf_xdp_load_bytes(ctx, offset, &value, sizeof(value));
+ s->errors[loc][XDP_ZCRX_LOAD] += ret != expected;
+ /* Successful stores put the original bytes back. */
+ ret = bpf_xdp_store_bytes(ctx, offset, &value, sizeof(value));
+ s->errors[loc][XDP_ZCRX_STORE] += ret != expected;
+
+ ret = bpf_dynptr_read(&value, sizeof(value), ptr, offset, 0);
+ s->errors[loc][XDP_ZCRX_DYN_READ] += ret != expected;
+ ret = bpf_dynptr_write(ptr, offset, &value, sizeof(value), 0);
+ s->errors[loc][XDP_ZCRX_DYN_WRITE] += ret != expected;
+
+ ret = bpf_dynptr_slice(ptr, offset, &value, sizeof(value)) ?
+ 0 : -EFAULT;
+ s->errors[loc][XDP_ZCRX_SLICE] += ret != expected;
+ ret = bpf_dynptr_slice_rdwr(ptr, offset, &value, sizeof(value)) ?
+ 0 : -EFAULT;
+ s->errors[loc][XDP_ZCRX_SLICE_RDWR] += ret != expected;
+}
+
+SEC("xdp.frags")
+int iou_zcrx(struct xdp_md *ctx)
+{
+ void *end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ struct ipv6hdr *ip6 = data + ETH_HLEN;
+ struct tcphdr *tcp = (void *)(ip6 + 1);
+ struct ethhdr *eth = data;
+ __u32 linear, total, hdr_len, wire_len;
+ struct xdp_zcrx_stats *s;
+ struct bpf_dynptr ptr;
+ __u32 key = 0;
+ int expected;
+
+ /* The traffic generator sends IPv6/TCP without extension headers. */
+ if ((void *)(tcp + 1) > end || eth->h_proto != bpf_htons(ETH_P_IPV6) ||
+ ip6->nexthdr != IPPROTO_TCP || tcp->dest != bpf_htons(port))
+ return XDP_PASS;
+
+ hdr_len = ETH_HLEN + sizeof(*ip6) + tcp->doff * 4;
+ wire_len = ETH_HLEN + sizeof(*ip6) + bpf_ntohs(ip6->payload_len);
+ if (wire_len <= hdr_len)
+ return XDP_PASS;
+
+ s = bpf_map_lookup_elem(&stats, &key);
+ if (!s)
+ return XDP_PASS;
+ s->packets++;
+ if (ctx->rx_queue_index != queue_id) {
+ s->wrong_queue++;
+ return XDP_PASS;
+ }
+
+ linear = end - data;
+ total = bpf_xdp_get_buff_len(ctx);
+ if (tcp->doff < sizeof(*tcp) / 4 || hdr_len > linear ||
+ wire_len > total || linear > 0xffff - sizeof(__u32)) {
+ s->bad_headers++;
+ return XDP_PASS;
+ }
+ /* Both accesses must be in the packet, not Ethernet padding. */
+ if (linear + sizeof(__u32) > wire_len)
+ return XDP_PASS;
+
+ if (bpf_dynptr_from_xdp(ctx, 0, &ptr)) {
+ s->bad_headers++;
+ return XDP_PASS;
+ }
+ s->tested++;
+ s->linear_bytes += linear;
+ s->frag_bytes += total - linear;
+ expected = unreadable ? -EFAULT : 0;
+ check_access(ctx, &ptr, s, XDP_ZCRX_HEADER,
+ ETH_HLEN + sizeof(*ip6), 0);
+ check_access(ctx, &ptr, s, XDP_ZCRX_FRAG, linear, expected);
+ check_access(ctx, &ptr, s, XDP_ZCRX_BOUNDARY, linear - 2, expected);
+ return XDP_PASS;
+}
+
+char LICENSE[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/drivers/net/hw/iou_zcrx.h b/tools/testing/selftests/drivers/net/hw/iou_zcrx.h
new file mode 100644
index 000000000000..c9749bc9a0a6
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/iou_zcrx.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef IOU_ZCRX_XDP_H
+#define IOU_ZCRX_XDP_H
+
+#include <linux/types.h>
+
+enum xdp_zcrx_location {
+ XDP_ZCRX_HEADER,
+ XDP_ZCRX_FRAG,
+ XDP_ZCRX_BOUNDARY,
+ XDP_ZCRX_LOCATIONS,
+};
+
+enum xdp_zcrx_op {
+ XDP_ZCRX_LOAD,
+ XDP_ZCRX_STORE,
+ XDP_ZCRX_DYN_READ,
+ XDP_ZCRX_DYN_WRITE,
+ XDP_ZCRX_SLICE,
+ XDP_ZCRX_SLICE_RDWR,
+ XDP_ZCRX_OPS,
+};
+
+struct xdp_zcrx_stats {
+ __u64 packets;
+ __u64 tested;
+ __u64 wrong_queue;
+ __u64 bad_headers;
+ __u64 linear_bytes;
+ __u64 frag_bytes;
+ __u64 errors[XDP_ZCRX_LOCATIONS][XDP_ZCRX_OPS];
+};
+
+#endif
--
2.56.1
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-17 16:48 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 16:46 [PATCH net-next 0/7] XDP with io_uring zero-copy RX and devmem Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 1/7] bpf: Keep XDP helpers within readable data Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 2/7] xdp: Reject frames with unreadable fragments Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 3/7] net: Admit native XDP on provider queues Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 4/7] eth: fbnic: Run " Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 5/7] netdevsim: Run native XDP next to devmem Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 6/7] selftests: net: Test XDP with devmem bindings Björn Töpel
2026-11-17 16:46 ` [PATCH net-next 7/7] selftests: net: Test XDP on io_uring RX Björn Töpel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®