mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Xiubo Li via B4 Relay <devnull+xiubo.li.clyso.com@kernel.org>
To: Ilya Dryomov <idryomov@gmail.com>,
	Alex Markuze <amarkuze@redhat.com>,
	 Viacheslav Dubeyko <slava@dubeyko.com>
Cc: ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org,
	 Xiubo Li <xiubo.li@clyso.com>
Subject: [PATCH v7 0/5] ceph: reduce mdsc->mutex contention in the cephfs kclient
Date: Fri, 20 Nov 2026 09:11:08 +0200	[thread overview]
Message-ID: <20261120-ceph-mdsc-mutex-optimization-v7-0-a4eee70e0038@clyso.com> (raw)

This series reduces mdsc->mutex hold times from hundreds of
microseconds to tens of microseconds on the hot request-submit and
reply-handling paths.

The approach is incremental:
  1. Convert oldest_tid to atomic64_t so that __prepare_send_request()
     and __send_request() no longer need the mutex.
  2. Replace the request_tree rbtree with an xarray for O(1) lookups
     and internally-locked iteration.
  3. Add a dedicated wait_list_lock spinlock so wait-list operations
     no longer depend on the global mutex.
  4. Move mdsc->mutex acquisition inside __do_request(), then release
     it during the send phase (message construction and path walking),
     leaving only the brief setup/teardown under the lock.
  5. Narrow the mutex scope in replay_unsafe_requests() similarly.

Tested with concurrent readdir + stat on a 5000-file directory
(32 threads).  bpftrace measurements show:

                 before         after
  __do_request   354-2327 us    10-68 us    (34x)
  handle_reply   51-416 us     10-32 us    (13x)
  submit_request 89-211 us     10-41 us    (5x)

Benchmark results from test_i_caps (aggregate cache test):

  | Test                | Before   | After    | Improvement |
  |---------------------|----------|----------|-------------|
  | stat storm ST       | 692k/s   | 1,420k/s | +105% (2.1x)|
  | stat storm MT       | 722k/s   | 1,373k/s | +90%  (1.9x)|
  | open/close          | 403k/s   | 524k/s   | +30%        |
  | stat hot (cache)    | 353k/s   | 540k/s   | +53%        |
  | readdir             | 877/s    | 832/s    | ~0%         |
  | Total time          | 195s     | 121s     | -38%  (1.6x)|

The 2x stat throughput gain comes from __do_request() no longer
holding mdsc->mutex during the __send_request() phase, so dentry
path walking and message encoding in create_request_message()
run outside the lock.

No functional changes intended.

Testing for 24+ hours:

The runtime validation ran on a vstart cluster (3 active + 3 standby
MDS) with a 32-thread metadata load generator and a kernel built with
CONFIG_PROVE_LOCKING, CONFIG_DEBUG_LIST and CONFIG_DETECT_HUNG_TASK:

- fsstress + dbench smoke runs: clean.
- r1, single-MDS failover churn: 108 failovers across 4 runs, every
  session reconnected, no lost wakeups, no hangs.
- r2, double "ceph mds fail" chaos: ~90 double-fail cycles across
  2 runs, zero hangs, zero kernel warnings, clean teardown.
- r3, double-fail plus victim processes SIGKILLed every 50-400 ms
  (5084 kills in one run) to exercise the request abort path: zero
  warnings, zero list-corruption reports, zero refcount issues.
- r4 (double mds fail + evict churn + 8 victim processes killed at
  200-2000 ms): one 32-minute run FAILED -- a worker was stuck in
  truncate for ~15 minutes at shutdown, the join oracle's first hit
  across all runs.  Live-kernel forensics (drgn against /proc/kcore)
  showed the wedge is unrelated to this series: the double-fail had
  dropped the inode's dirty caps in the session-loss path (caps.c)
  without cleaning up the page wrbuffer refs, so the ceph_inode_work()
  kworker spun forever in __ceph_do_pending_vmtruncate()'s flush
  branch -- with no dirty caps, ceph_writepages_start() returns
  -ENODATA without writing, the refs never drain, and the loop holds
  i_truncate_mutex indefinitely, starving concurrent truncates (and,
  via setattr's inode lock and vfs_unlink()'s target-inode lock,
  unlinks).  Every link of that chain is in caps.c/inode.c/addr.c/VFS
  code untouched by these patches; it is a pre-existing consistency
  bug (dirty caps dropped without wrbuffer-ref cleanup on session
  loss, plus an unbounded flush loop) to be fixed separately.  (The
  r4 evict trigger was skipped in this run as debugfs was not
  mounted; session eviction itself had been exercised in earlier
  testing.)

No lockdep reports, list corruption, refcount warnings, or hung tasks
across the whole campaign.  The campaign also exposed two pre-existing
mainline bugs (a NULL oldest-snapc dereference in the writeback path
and a request abort-path use-after-free), both reproducible without
this series on a vanilla 7.2 kernel and fixed separately.

The whole test code could be find in:
https://tracker.ceph.com/issues/80086
Or
https://tracker.ceph.com/issues/80084

Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
---
Changes in v7:
- Patches 1-3 are identical to v6.
- patch 4: add the missing braces around the s_unsafe loop in
  replay_unsafe_requests().  Only r_attempts++ was inside the loop,
  and __send_request() ran once after it on a request pointer derived
  from the list head, so a reconnect with patch 4 alone replayed no
  unsafe request and touched a bogus pointer.  Patch 5 rewrites the
  function, so the series result is unchanged; this keeps patch 4
  bisectable.  Found by -Wmisleading-indentation.
- patch 5:
  - replay_unsafe_requests() no longer skips a request whose
    CEPH_MDS_R_DISPATCHING is held.  The owner may already have queued
    the request message on the connection that send_mds_reconnect()
    resets, and nothing would send it again, so the caller waited
    forever.  The replay now waits for the owner to release the bit,
    rescans, and replays the request itself.  The owner never takes
    session->s_mutex in its send window, so the wait cannot deadlock.
    (Alex Markuze)
  - Add release_dispatch() and use it on every DISPATCHING release
    path.  The replay flags its wait with the new
    CEPH_MDS_R_REPLAY_WAIT bit, and release_dispatch() calls
    wake_up_bit() only when that bit is set, so the common release
    stays one atomic clear plus a plain test_bit() under mdsc->mutex,
    without a lookup in the kernel-wide bit_wait_table.
  - __do_request() and replay_unsafe_requests() copy r_attempts and
    r_num_fwd into the new r_send_attempts and r_send_num_fwd under
    mdsc->mutex, and __prepare_send_request() encodes num_retry and
    num_fwd, and checks for retry overflow, from the copies.
    cleanup_session_requests() and handle_forward() may rewrite the
    originals while the message is built, which encoded num_retry as
    255 and ext_num_retry as 0xffffffff.  (Alex Markuze)
  - __do_request() resets r_resend_mds before dropping mdsc->mutex, so
    that a forward arriving in the send window is not undone.
- Link to v6: https://patch.msgid.link/20260829-ceph-mdsc-mutex-optimization-v6-0-466936ccbd9d@clyso.com

Changes in v6:
- Only patch 5/5 is changed; patches 1-4 are identical to v5.
- Replace the ad-hoc r_attempts dispatch gate with an explicit
  ownership protocol.  __do_request() and replay_unsafe_requests()
  claim CEPH_MDS_R_DISPATCHING under mdsc->mutex before the unlocked
  prepare/send window and release it on exit, so only one context ever
  rebuilds and sends a request's message.  A claim consumes any
  pending CEPH_MDS_R_RESEND.
- cleanup_session_requests() and handle_forward() set
  CEPH_MDS_R_RESEND under mdsc->mutex when they see DISPATCHING held,
  instead of relying on r_attempts == 0 to trigger a resend.  Fixes
  the v5 race where cleanup_session_requests() could zero r_attempts
  while a sender was between the mutex release and __send_request(),
  letting a second __do_request() in and double ceph_msg_put() the
  request message.
- On release, the owner consumes a pending CEPH_MDS_R_RESEND
  and re-dispatches the request through __do_request() (restart label)
  as long as it is still registered in the request xarray.
- __wake_requests() now takes mdsc->mutex and splices the wait
  list under it, pinning a reference on each request in the same
  critical section as the list_del_init().  This fixes the v5 lockless
  walk of the spliced list against concurrent list_del_init() by the
  park paths, and serializes the SESSION_OPEN wake in handle_session()
  against the park decision in __do_request().
- Collector exclusivity — kick_requests(), __wake_requests()
  and replay_unsafe_requests() skip any request whose r_aux_item is
  non-empty (checked under mdsc->mutex), so two concurrent collectors
  can no longer add the same node to their local lists.
- The re-park paths in __do_request() now list_del_init() the
  request before list_add()ing it onto a wait list, so re-parking is
  idempotent (needed now that a request can re-enter __do_request()
  through the RESEND restart loop).
- __do_request() refuses to dispatch GOT_UNSAFE requests,
  clearing any pending RESEND/DISPATCHING bits, so an unsafe request
  is never parked on a wait list.  Together with the exclusivity
  predicate this guarantees a request is never both wait-listed and on
  session->s_unsafe, and replay_unsafe_requests() remains the only
  replayer of unsafe requests.
- replay_unsafe_requests() claims DISPATCHING in both collect
  loops, and after __send_request() consumes a pending RESEND and
  re-dispatches through __do_request().
- send_mds_reconnect() drops snap_rwsem before calling
  __wake_requests(), which now takes mdsc->mutex internally (lock
  order).
- Link to v5: https://patch.msgid.link/20260818-ceph-mdsc-mutex-optimization-v5-0-7d335a3a1d0b@clyso.com

Changes in v5:
- patch 4: add INIT_LIST_HEAD(&req->r_aux_item) in ceph_mdsc_create_request().
- patch 5: keep requests on session->s_unsafe during replay; use r_aux_item
  only as a walk list.
- Link to v4: https://patch.msgid.link/20260812-ceph-mdsc-mutex-optimization-v4-0-fca3b7462f94@clyso.com

Changes in v4:
- Fix ref leak in __register_request() xa_store error path
- Fix r_attempts data race: move r_attempts++ from
  __prepare_send_request() (lockless) into __do_request() and
  replay_unsafe_requests() under mdsc->mutex.  Adjust the retry
  overflow check accordingly.
- Fix kick_requests() list corruption: detach r_wait from the local
  kick_list before calling __do_request().
- Fix collect-then-replay list-node races in both kick_requests()
  and replay_unsafe_requests(): introduce r_aux_item, a dedicated
  list_head for temporary local list iteration, so that concurrent
  __unregister_request() cannot corrupt the iterator.
- Link to v3: https://patch.msgid.link/20260811-ceph-mdsc-mutex-optimization-v3-0-d031114419f4@clyso.com

Changes in v3:
- Restrict CEPH_FS to 64BIT to prevent xarray index truncation of
  u64 transaction IDs on 32-bit platforms, instead of the previous
  #if BITS_PER_LONG guard.  There are no 32-bit users.
  The survey: https://lore.kernel.org/ceph-devel/CAOJNxR+XiUxR1GNUgd8T18x2KH8kstWtpxwL-mwwE0i_qospJA@mail.gmail.com/T/#t
- Fix two plain reads of oldest_tid in the writer paths to use
  READ_ONCE() for consistency with the lockless read side.
- Replace the original replay_unsafe_requests() change with a
  proper collect-then-replay pattern: unsafe list entries and
  matching old xarray entries are gathered under the mutex with a
  reference taken, then replayed outside it.  Taking a reference
  ensures a concurrent reply handler cannot free an entry out
  from under the local-list iterator.
- Rewrite all commit messages as descriptive prose, focusing on
  the problem and the approach rather than enumerating modified
  functions.
- Link to v2: https://patch.msgid.link/20260715-ceph-mdsc-mutex-optimization-v2-0-90e81b726724@clyso.com

Changes in v2:
- Add xa_store() error handling and bail out on failure in the submit path
- Guard xarray conversion with BITS_PER_LONG==64, fall back to rbtree on 32-bit
- Fix missing mutex_unlock on early-return path in __do_request()
- Move mutex_unlock before __wake_requests() and kick_requests() calls to
  avoid recursive lock acquisition
- Pin requests with ceph_mdsc_get_request() across lockless __send_request()
  in replay_unsafe_requests() to prevent use-after-free
- Keep mdsc->mutex held on 32-bit for rb_first()/rb_next() iteration in
  replay_unsafe_requests()
- Drop stale "called under mdsc->mutex" comment on __wake_requests()
- Add benchmark results from test_i_caps (2.1x stat throughput improvement)
- Link to v1: https://patch.msgid.link/20260713-ceph-mdsc-mutex-optimization-v1-0-9ae5ac135c34@clyso.com

To: Ilya Dryomov <idryomov@gmail.com>
To: Alex Markuze <amarkuze@redhat.com>
To: Viacheslav Dubeyko <slava@dubeyko.com>
Cc: ceph-devel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

---
Xiubo Li (5):
      ceph: use READ_ONCE/WRITE_ONCE for oldest_tid
      ceph: replace the request_tree rbtree with an xarray keyed by r_tid
      ceph: add wait_list_lock for wait-list serialization
      ceph: move mdsc->mutex into __do_request()
      ceph: narrow mdsc->mutex scope in replay_unsafe_requests

 fs/ceph/Kconfig      |   1 +
 fs/ceph/debugfs.c    |   6 +-
 fs/ceph/mds_client.c | 666 ++++++++++++++++++++++++++++++++++++++++-----------
 fs/ceph/mds_client.h |  56 ++++-
 4 files changed, 579 insertions(+), 150 deletions(-)
---
base-commit: dee30ce1286a0d18b14545ecac345e4cf4a80511
change-id: 20260713-ceph-mdsc-mutex-optimization-7e74ab6bbc8b

Best regards,
--  
Xiubo Li <xiubo.li@clyso.com>



             reply	other threads:[~2026-11-20  7:11 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  7:11 Xiubo Li via B4 Relay [this message]
2026-11-20  7:11 ` [PATCH v7 1/5] ceph: use READ_ONCE/WRITE_ONCE for oldest_tid Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 2/5] ceph: replace the request_tree rbtree with an xarray keyed by r_tid Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 3/5] ceph: add wait_list_lock for wait-list serialization Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 4/5] ceph: move mdsc->mutex into __do_request() Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 5/5] ceph: narrow mdsc->mutex scope in replay_unsafe_requests Xiubo Li via B4 Relay

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120-ceph-mdsc-mutex-optimization-v7-0-a4eee70e0038@clyso.com \
    --to=devnull+xiubo.li.clyso.com@kernel.org \
    --cc=amarkuze@redhat.com \
    --cc=ceph-devel@vger.kernel.org \
    --cc=idryomov@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=slava@dubeyko.com \
    --cc=xiubo.li@clyso.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®