[RFC PATCH 0/3] io_uring/rsrc: reduce node allocation cost on sparse file table installs

[Uzair Beg <uzairbeg11@gmail.com>]

Chengfeng Lin reported an 11.6% per-install slowdown in MSG_RING SEND_FD
fixed-file installation, bisected to 7029acd8a950 ("io_uring/rsrc: get rid
of per-ring io_rsrc_node list"):

  https://lore.kernel.org/io-uring/CANGjgdmt0FQ=offsdfn+wEaDxbOFoAa6bi92X_vEo4S6aCZ56A@mail.gmail.com/

That commit is not being questioned here. It removed a real serialisation
cost. The side effect is that each fixed-file install now allocates its own
io_rsrc_node, and on a first fill of a sparse table every one of those is
an allocator miss: io_alloc_cache_init() only allocates the pointer array,
io_alloc_cache_put() on the free path is the only thing that populates it,
and io_reset_rsrc_node() returns early on a NULL slot so nothing is freed
during a first fill. With IO_ALLOC_CACHE_MAX at 128, warming the existing
cache cannot cover a 4,096-slot fill.

We spent some time isolating where the per-install cost actually lives.
All timing below is Chengfeng's, on a bare-metal i7-12700KF with a pinned
core, performance governor, turbo off and fresh boots per point; the
evidence tree with reproducers and full logs is linked at the end.

  hypothesis                          result
  ----------------------------------  ---------------------------------
  slab merging defeats locality       slab_nomerge: no change
  SLAB_ACCOUNT on the new cache       +8.5% cost; removed (memcg
                                      accounting the old path never did)
  allocator call overhead             bulk refill, 32x fewer calls:
                                      +0.007%, path verified by probe
  fresh slab page creation            slab primed, 0 new pages: +0.9%
  per-object SLUB allocation path     prefill removes it: -9.5%

So the cost is the per-object allocation itself, not calls or pages, and
the only way to take it off the install path is to not allocate there.

This series does that in three steps:

  0001  dedicated kmem_cache for io_rsrc_node. Neutral on its own; exists
        so 0002/0003 can use kmem_cache_alloc_bulk().
  0002  bulk refill of the per-ring cache on a miss. Also neutral on the
        reported workload, for the reason above; kept for the machinery.
  0003  when a sparse table of N slots is registered, grow the per-ring
        cache to min(N, 4096) and bulk-fill it.

Measured on the actual series (v6.18-rc4, 6146a0f1dfae):

  workload                            unpatched   0001+2    +0003
  ----------------------------------  ---------  --------  --------
  reported 4,096-slot first fill,     119.423    118.151   107.731   -9.8%
    ns/install
  same-ring remove+refill, 4,096      712.451    710.384   578.365   -18.8%
    files, us
  one-shot register through fill,     518.387    520.663   526.873   +1.6%
    4,096 slots, us
  register 4096 / install 64, us       20.361     21.193    84.407   worse

The second row was not predicted: the enlarged cache retains nodes released
by FILES_UPDATE, so steady-state churn stops allocating entirely.

The last two rows are the cost, stated plainly. Prefill moves the
allocation work to registration rather than removing it, so a program
that registers once and fills once sees no total saving, and a program
that registers many slots and installs few pays for nodes it never uses
(up to 4096 x 32 bytes plus a 32 KiB pointer array, freed at ring
teardown).

Whether that trade is acceptable as default behaviour is the question this
RFC raises. An alternative would be an opt-in registration flag, so a
program that intends to fill the table can say so and nobody else pays. I
am happy to rework 0003 into that shape if it is the preferred one.

Testing: builds and boots on io_uring-6.18; the file table, rsrc and
msg_ring liburing tests pass on the patched kernel. The full runtests
suite was not run to completion on my build VM (the networking tests take
its interface down); the targeted set was. Chengfeng ran the alloc_cache
code through allocation-failure, limit and cleanup cases under
ASan/UBSan/LeakSanitizer, and verified the 8,192-slot case prefills 4,096
and falls through to bulk for the remainder.

Evidence, reproducers and diagnostic patches:
  https://github.com/lcf0399/linux-regression-evidence/tree/af2bf8eaae547d940c64ae8dcd1803baf31f2139/io-uring-msg-ring-send-fd-install

Uzair Beg (3):
  io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cache
  io_uring/rsrc: bulk refill the node cache on allocation miss
  io_uring/rsrc: prefill the node cache when a file table is registered
    empty

 include/linux/io_uring_types.h |  1 +
 io_uring/alloc_cache.c         | 75 ++++++++++++++++++++++++++++++++--
 io_uring/alloc_cache.h         | 11 ++++-
 io_uring/io_uring.c            |  5 +++
 io_uring/io_uring.h            |  1 +
 io_uring/rsrc.c                |  6 +++
 6 files changed, 94 insertions(+), 5 deletions(-)

-- 
2.43.0