Re: [PATCH net-next 6/7] net: devmem: size TX net_iovs from the dmabuf sg table

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 11/02, Mina Almasry wrote:
> On Mon, Nov 2, 2026 at 11:14 AM Stanislav Fomichev <sdf.kernel@gmail.com> wrote:
> >
> > PAGE_SIZE TX net_iovs cap devmem skbs at MAX_SKB_FRAGS pages (~68KB),
> > which makes BIG TCP ineffective.
> >
> > TX net_iovs only need to be DMA contiguous. Use the largest power of 2,
> > up to 2MB, which aligns every DMA segment of the dmabuf:
> > - 2MB hugepage-backed udmabufs get 2MB net_iovs
> > - no UAPI change, TX addresses are dmabuf offsets
> > - skbs are still bounded by TCP's size_goal, only frags get larger
> >
> > Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
> > ---
> >  net/core/devmem.c      | 28 ++++++++++++++++++++++++++--
> >  net/core/netdev-genl.c |  2 +-
> >  2 files changed, 27 insertions(+), 3 deletions(-)
> >
> > diff --git a/net/core/devmem.c b/net/core/devmem.c
> > index 19c039ed2cb2..98c9594dbf99 100644
> > --- a/net/core/devmem.c
> > +++ b/net/core/devmem.c
> > @@ -181,6 +181,20 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,
> >         return err;
> >  }
> >
> > +static bool net_devmem_sgt_aligned(struct sg_table *sgt, size_t size)
> > +{
> > +       struct scatterlist *sg;
> > +       unsigned int i;
> > +
> > +       for_each_sgtable_dma_sg(sgt, sg, i) {
> > +               if (!IS_ALIGNED(sg_dma_address(sg), size) ||
> > +                   !IS_ALIGNED(sg_dma_len(sg), size))
> > +                       return false;
> > +       }
> > +       return true;
> > +}
> > +
> > +/* niov_shift == 0 picks the net_iov size from the dmabuf, TX only */
> >  struct net_devmem_dmabuf_binding *
> >  net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> >                        struct device *dma_dev,
> > @@ -191,11 +205,11 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> >                        struct netlink_ext_ack *extack)
> >  {
> >         struct net_devmem_dmabuf_binding *binding;
> > -       size_t niov_size = 1UL << niov_shift;
> >         static u32 id_alloc_next;
> >         struct scatterlist *sg;
> >         struct dma_buf *dmabuf;
> >         unsigned int sg_idx;
> > +       size_t niov_size;
> >         size_t niov_idx;
> >         size_t i;
> >         long err;
> > @@ -221,7 +235,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> >         binding->tx_max_frag_size = dev->netmem_tx_max_frag_size;
> >         write_pnet(&binding->net, sock_net(nl_sk));
> >         binding->uid = sk_uid(nl_sk);
> > -       binding->niov_shift = niov_shift;
> >         xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC);
> >
> >         err = percpu_ref_init(&binding->ref,
> > @@ -266,6 +279,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> >                 goto err_unmap;
> >         }
> >
> > +       if (!niov_shift) {
> > +               niov_size = SZ_2M;
> > +               while (niov_size > PAGE_SIZE &&
> > +                      (!IS_ALIGNED(dmabuf->size, niov_size) ||
> > +                       !net_devmem_sgt_aligned(binding->sgt, niov_size)))
> > +                       niov_size /= 2;
> > +               niov_shift = ilog2(niov_size);
> > +       }
> > +       binding->niov_shift = niov_shift;
> > +       niov_size = 1UL << niov_shift;
> > +
> 
> How come we're not checknig the largest supported size of the netdev here?
> 
> Also, maybe should clarify in the cl with the largest supported size,
> is it actually only powers of 2 up to that power of 2? Or all sizes up
> to that (not-necassirly) power of 2 size?

I think these are a bit orthogonal, no? We can have niovs larger than
the HW can tx. If that's the case, we'll just sub-divide when we build
the skb. If we limit to the same chunk the device can handle here,
we might use 2 frags to send single chunk worth of data more often (offset
doesn't have to be aligned to the chunk size).


Assuming device can only send 16K, current patch:

[       2MB        ] 2BM niov
     [ 16K ]         sending 16K from that 2MB will use one chunk


Limiting chunks to device's chunk size:

[ 16K ][ 16K ]       16K niov
    [ 16K ]          sending 16K from offset 8K will have to use 2 frags

(obviously, the same thing can happen with 2MB chunks, but less often)

LMK if it does not sound convincing.