[PATCH 0/4] io_uring: avoid req->flags store-to-load forwarding failures

[Caleb Sander Mateos <csander@purestorage.com>]

clang likes to set/clear req->flags bits using the narrowest possible or/and
instructions (e.g. an 8-bit or if all flag bits are in the same byte). This
causes store-to-load forwarding failures on subsequent wider req->flags loads.

For example, clang compiles the req->flags |= REQ_F_ASYNC_DATA in
io_uring_cmd_prep() -> io_uring_alloc_cached_async_data() on x86_64 into an
orb $0x40, 0x4a(%rdi) instruction, since REQ_F_ASYNC_DATA = 0x40 << 16 and
offsetof(struct io_kiocb, flags) = 0x48. The subsequent 64-bit req->flags load
for unlikely(req->flags & (IO_REQ_LINK_FLAGS | REQ_F_FORCE_ASYNC | REQ_F_FAIL))
in io_submit_sqe() is extremely hot because it can't be served from the store
buffer and must wait for the REQ_F_ASYNC_DATA set to hit the L1 cache. On a
thread issuing ublk and NVMe passthrough commands, that one req->flags load
consumes around 1% of all CPU cycles.

Introduce io_req_set_flags() and io_req_clear_flags() helpers to set req->flags
using volatile writes so clang won't emit a store to only part of the field.
The downside of volatile writes is that consecutive req->flags updates must
store to req->flags multiple times. Where possible, convert these updates to
pass multiple flags to io_req_set_flags(), io_req_clear_flags(), or an extra
helper io_req_switch_flags() which can do both. An extra store would still be
much cheaper than a store-to-load forwarding stall.

Alternatively, -mllvm -combiner-reduce-load-op-store-width=false could be used
to disable the clang "optimization". It seemed ugly to refer to such an internal
optimization pass, and I wasn't sure what other impacts this flag might have
(even if restricted to io_uring). But I'm happy to implement that approach if
it's preferred.

Caleb Sander Mateos (4):
  io_uring: return io_req_flags_t from io_slot_flags()
  io_uring: add multishot_flags constant in io_wq_submit_work()
  io_uring: add io_req_{set,clear,switch}_flags()
  io_uring: use io_req_{set,clear,switch}_flags()

 include/linux/io_uring.h | 35 +++++++++++++++++++++++++-----
 io_uring/advise.c        |  4 ++--
 io_uring/cancel.c        |  2 +-
 io_uring/dma-buf.h       |  4 ++--
 io_uring/filetable.h     |  2 +-
 io_uring/fs.c            | 25 +++++++++-------------
 io_uring/futex.c         |  4 ++--
 io_uring/io_uring.c      | 41 +++++++++++++++++------------------
 io_uring/io_uring.h      |  8 +++----
 io_uring/kbuf.c          | 31 +++++++++++++++------------
 io_uring/kbuf.h          |  4 ++--
 io_uring/msg_ring.c      |  4 ++--
 io_uring/net.c           | 46 ++++++++++++++++++++--------------------
 io_uring/nop.c           |  2 +-
 io_uring/openclose.c     |  6 +++---
 io_uring/poll.c          | 20 +++++++++--------
 io_uring/refs.h          |  4 ++--
 io_uring/rsrc.c          | 10 ++++-----
 io_uring/rw.c            | 28 ++++++++++++------------
 io_uring/splice.c        |  4 ++--
 io_uring/statx.c         |  3 +--
 io_uring/sync.c          |  6 +++---
 io_uring/timeout.c       |  8 +++----
 io_uring/truncate.c      |  2 +-
 io_uring/uring_cmd.c     |  8 +++----
 io_uring/xattr.c         |  8 +++----
 26 files changed, 170 insertions(+), 149 deletions(-)


base-commit: ddcc50dd41d331dc66e0e5793492c3fa09941ff9
-- 
2.56.1