mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Xiubo Li via B4 Relay <devnull+xiubo.li.clyso.com@kernel.org>
To: Ilya Dryomov <idryomov@gmail.com>,
	Alex Markuze <amarkuze@redhat.com>,
	 Viacheslav Dubeyko <slava@dubeyko.com>
Cc: ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org,
	 Xiubo Li <xiubo.li@clyso.com>
Subject: [PATCH v7 5/5] ceph: narrow mdsc->mutex scope in replay_unsafe_requests
Date: Fri, 20 Nov 2026 09:11:13 +0200	[thread overview]
Message-ID: <20261120-ceph-mdsc-mutex-optimization-v7-5-a4eee70e0038@clyso.com> (raw)
In-Reply-To: <20261120-ceph-mdsc-mutex-optimization-v7-0-a4eee70e0038@clyso.com>

From: Xiubo Li <xiubo.li@clyso.com>

Currently replay_unsafe_requests() holds mdsc->mutex across the
entire function, including the __send_request() calls.  Since
__send_request() is lockless and the async cap-release helper
schedules deferred work, neither needs the mutex.

Collect the unsafe-list entries and the matching old xarray
entries into local lists under mdsc->mutex, taking a reference on
each, then replay them outside the mutex.  Taking a reference
ensures a concurrent reply handler can complete and unregister a
request without invalidating the local list or the iterator.

Unsafe requests remain on session->s_unsafe; r_aux_item serves
only as a walk-list link.  This keeps them tracked as unsafe
until the MDS replies, so a later reconnect can replay them again
and cleanup_session_requests() can still abort them on session
teardown.

A dispatch owner in __do_request() holds CEPH_MDS_R_DISPATCHING
across its unlocked send window.  If send_mds_reconnect() resets the
session connection inside that window, the message the owner has
already queued is dropped, so replay_unsafe_requests() must not skip
such a request: it waits for the owner to release the bit, then
rescans and replays the request itself.  The owner never takes
session->s_mutex in the send window, so waiting with it held cannot
deadlock.

release_dispatch() runs for every dispatched request with mdsc->mutex
held, so it must not pay for that rare wait.  The replay sets
CEPH_MDS_R_REPLAY_WAIT and issues smp_mb__after_atomic() before it
waits; release_dispatch() clears DISPATCHING, issues
smp_mb__after_atomic(), and calls wake_up_bit() only when it sees
REPLAY_WAIT set, clearing it with test_and_clear_bit().  Either the
waiter sees DISPATCHING clear or the releaser sees REPLAY_WAIT set, so
no wake-up is lost, and the common release is one atomic clear plus a
plain test_bit(), without the lookup in the kernel-wide bit_wait_table
that wake_up_bit() does.  The releaser clears REPLAY_WAIT, so two
waiters on one request cannot clear it under each other.

cleanup_session_requests() and handle_forward() may rewrite
r_attempts and r_num_fwd under mdsc->mutex while the owner encodes
the message without it, which would put a retry count of 255 (or
0xffffffff) into the request.  The owner therefore copies both into
r_send_attempts and r_send_num_fwd under mdsc->mutex, and
__prepare_send_request() encodes from the copies.  r_resend_mds is
also reset before the mutex is dropped, so that a forward arriving in
the window is not undone.

Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
---
 fs/ceph/mds_client.c | 464 +++++++++++++++++++++++++++++++++++++++++++--------
 fs/ceph/mds_client.h |  51 +++++-
 2 files changed, 447 insertions(+), 68 deletions(-)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 6d4182cfa70a..d4699d0e4539 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -1925,12 +1925,24 @@ static void cleanup_session_requests(struct ceph_mds_client *mdsc,
 			mapping_set_error(req->r_unsafe_dir->i_mapping, -EIO);
 		__unregister_request(mdsc, req);
 	}
-	/* zero r_attempts, so kick_requests() will re-send requests */
+	/*
+	 * Zero r_attempts so that the following kick_requests() will
+	 * re-send the request.  If a dispatch owner is currently in the
+	 * send window (CEPH_MDS_R_DISPATCHING set), a concurrent
+	 * kick_requests() could not pick the request up and the resend
+	 * request would be lost; set CEPH_MDS_R_RESEND instead and let
+	 * the owner re-evaluate the request when it releases dispatch
+	 * ownership.
+	 */
 	idx = 0;
 	xa_for_each(&mdsc->request_tree, idx, req) {
 		if (req->r_session &&
-		    req->r_session->s_mds == session->s_mds)
+		    req->r_session->s_mds == session->s_mds) {
 			req->r_attempts = 0;
+			if (test_bit(CEPH_MDS_R_DISPATCHING,
+				     &req->r_req_flags))
+				set_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+		}
 	}
 	mutex_unlock(&mdsc->mutex);
 }
@@ -3526,6 +3538,8 @@ static int __prepare_send_request(struct ceph_mds_session *session,
 	struct ceph_mds_request_head *nhead;
 	struct ceph_msg *msg;
 	int flags = 0, old_max_retry;
+	int attempts = req->r_send_attempts;
+	int num_fwd = req->r_send_num_fwd;
 	bool old_version = !test_bit(CEPHFS_FEATURE_32BITS_RETRY_FWD,
 				     &session->s_features);
 
@@ -3535,16 +3549,19 @@ static int __prepare_send_request(struct ceph_mds_session *session,
 	 * so we limit to retry at most 256 times.
 	 *
 	 * r_attempts was already incremented under mdsc->mutex by the
-	 * caller (__do_request or replay_unsafe_requests), so the
-	 * actual retry count is r_attempts - 1 and we skip the check
-	 * on the first dispatch (r_attempts == 1).
+	 * caller (__do_request or replay_unsafe_requests), which also
+	 * copied it to r_send_attempts, so the actual retry count is
+	 * attempts - 1 and we skip the check on the first dispatch
+	 * (attempts == 1).  Use the copy: r_attempts itself may be
+	 * reset concurrently by cleanup_session_requests() or
+	 * handle_forward() while this runs without mdsc->mutex.
 	 */
-	if (req->r_attempts > 1) {
+	if (attempts > 1) {
 		old_max_retry = sizeof_field(struct ceph_mds_request_head,
 					     num_retry);
 		old_max_retry = 1 << (old_max_retry * BITS_PER_BYTE);
-		if ((old_version && (req->r_attempts - 1) >= old_max_retry) ||
-		    ((uint32_t)(req->r_attempts - 1) >= U32_MAX)) {
+		if ((old_version && (attempts - 1) >= old_max_retry) ||
+		    ((uint32_t)(attempts - 1) >= U32_MAX)) {
 			pr_warn_ratelimited_client(cl, "request tid %llu seq overflow\n",
 						   req->r_tid);
 			return -EMULTIHOP;
@@ -3560,7 +3577,7 @@ static int __prepare_send_request(struct ceph_mds_session *session,
 			req->r_sent_on_mseq = -1;
 	}
 	doutc(cl, "%p tid %lld %s (attempt %d)\n", req, req->r_tid,
-	      ceph_mds_op_name(req->r_op), req->r_attempts);
+	      ceph_mds_op_name(req->r_op), attempts);
 
 	if (test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags)) {
 		void *p;
@@ -3582,10 +3599,10 @@ static int __prepare_send_request(struct ceph_mds_session *session,
 		if (req->r_target_inode)
 			lhead->ino = cpu_to_le64(ceph_ino(req->r_target_inode));
 
-		lhead->num_retry = req->r_attempts - 1;
+		lhead->num_retry = attempts - 1;
 		if (!old_version) {
 			nhead = (struct ceph_mds_request_head*)msg->front.iov_base;
-			nhead->ext_num_retry = cpu_to_le32(req->r_attempts - 1);
+			nhead->ext_num_retry = cpu_to_le32(attempts - 1);
 		}
 
 		/* remove cap/dentry releases from message */
@@ -3620,12 +3637,12 @@ static int __prepare_send_request(struct ceph_mds_session *session,
 	if (req->r_parent)
 		flags |= CEPH_MDS_FLAG_WANT_DENTRY;
 	lhead->flags = cpu_to_le32(flags);
-	lhead->num_fwd = req->r_num_fwd;
-	lhead->num_retry = req->r_attempts - 1;
+	lhead->num_fwd = num_fwd;
+	lhead->num_retry = attempts - 1;
 	if (!old_version) {
 		nhead = (struct ceph_mds_request_head*)msg->front.iov_base;
-		nhead->ext_num_fwd = cpu_to_le32(req->r_num_fwd);
-		nhead->ext_num_retry = cpu_to_le32(req->r_attempts - 1);
+		nhead->ext_num_fwd = cpu_to_le32(num_fwd);
+		nhead->ext_num_retry = cpu_to_le32(attempts - 1);
 	}
 
 	doutc(cl, " r_parent = %p\n", req->r_parent);
@@ -3649,6 +3666,28 @@ static int __send_request(struct ceph_mds_session *session,
 	return err;
 }
 
+/*
+ * Release dispatch ownership of @req and wake up anyone waiting for
+ * it: replay_unsafe_requests() waits for an in-flight dispatch to
+ * finish before it replays the request itself.
+ */
+static void release_dispatch(struct ceph_mds_request *req)
+{
+	clear_bit_unlock(CEPH_MDS_R_DISPATCHING, &req->r_req_flags);
+	/*
+	 * Pairs with smp_mb__after_atomic() in replay_unsafe_requests():
+	 * either the waiter sees DISPATCHING clear, or this sees
+	 * REPLAY_WAIT set and wakes it up.  The plain test_bit() keeps the
+	 * common, no-waiter path free of atomic read-modify-writes and of
+	 * the shared bit_wait_table.  test_and_clear_bit() wakes every
+	 * waiter at once, since wait_on_bit() waiters are not exclusive.
+	 */
+	smp_mb__after_atomic();
+	if (test_bit(CEPH_MDS_R_REPLAY_WAIT, &req->r_req_flags) &&
+	    test_and_clear_bit(CEPH_MDS_R_REPLAY_WAIT, &req->r_req_flags))
+		wake_up_bit(&req->r_req_flags, CEPH_MDS_R_DISPATCHING);
+}
+
 /*
  * send request, or put it on the appropriate wait list.
  */
@@ -3661,37 +3700,58 @@ static void __do_request(struct ceph_mds_client *mdsc,
 	int err = 0;
 	bool random;
 
+restart:
+	/* re-entry from the resend loop below: reset per-pass state */
+	session = NULL;
+	err = 0;
 	mutex_lock(&mdsc->mutex);
 
 	/*
-	 * r_attempts is bumped under mdsc->mutex just before the mutex
-	 * is dropped to send the request, and it is only ever written
-	 * back to 0 by the forward handler or cleanup_session_requests()
-	 * (both under the mutex) before re-dispatching through a new
-	 * __do_request() call.  Consequently, r_attempts > 0 at this
-	 * point always means another __do_request() instance has already
-	 * passed the point of no return for this request, i.e. a racing
-	 * kick_requests() or __wake_requests() picked up the same
-	 * request from the xarray or a wait list and is about to send
-	 * it.  Bail out to prevent a double dispatch.
+	 * r_attempts only counts protocol send attempts now.  Dispatch
+	 * ownership is claimed separately via CEPH_MDS_R_DISPATCHING
+	 * under mdsc->mutex and held across the unlocked prepare/send
+	 * window: only one context may rebuild and send req->r_request
+	 * at a time.  A racing kick_requests() or __wake_requests()
+	 * that sees the claim set simply drops the request here.
 	 *
-	 * Note: kick_requests() also filters on r_attempts > 0 during
-	 * its collection pass, but that is a one-time snapshot taken
-	 * under the mutex.  Between that snapshot and the actual
-	 * __do_request() call the mutex is dropped and re-acquired, so
-	 * the protection is not atomic — this per-request gate closes
-	 * the remaining window.
+	 * cleanup_session_requests() and handle_forward() may zero
+	 * r_attempts while the owner is in flight; instead of racing,
+	 * they set CEPH_MDS_R_RESEND and the owner consumes it when
+	 * releasing ownership below.
 	 */
-	if (req->r_attempts > 0) {
+	if (test_and_set_bit(CEPH_MDS_R_DISPATCHING, &req->r_req_flags)) {
 		mutex_unlock(&mdsc->mutex);
 		return;
 	}
+	/*
+	 * Claiming dispatch ownership supersedes any pending resend
+	 * request: the dispatch about to happen below is the
+	 * re-evaluation.  RESEND set later, while the send window is
+	 * open, is consumed by the release path at the bottom.
+	 */
+	clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
 
 	if (req->r_err || test_bit(CEPH_MDS_R_GOT_RESULT, &req->r_req_flags)) {
 		if (test_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags))
 			__unregister_request(mdsc, req);
-		mutex_unlock(&mdsc->mutex);
-		return;
+		/*
+		 * The request is done (or dead) and must not be sent
+		 * again; the dispatch ownership claimed above is simply
+		 * released, consuming any pending resend request with it.
+		 */
+		clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+		release_dispatch(req);
+		goto no_dispatch;
+	}
+	if (test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags)) {
+		/*
+		 * Unsafe requests are replayed only by
+		 * replay_unsafe_requests() during MDS reconnect, never
+		 * through the normal dispatch path.
+		 */
+		clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+		release_dispatch(req);
+		goto no_dispatch;
 	}
 
 	if (READ_ONCE(mdsc->fsc->mount_state) == CEPH_MOUNT_FENCE_IO) {
@@ -3721,10 +3781,10 @@ static void __do_request(struct ceph_mds_client *mdsc,
 			trace_ceph_mdsc_suspend_request(mdsc, session, req,
 							ceph_mdsc_suspend_reason_no_mdsmap);
 			spin_lock(&mdsc->wait_list_lock);
+			list_del_init(&req->r_wait);
 			list_add(&req->r_wait, &mdsc->waiting_for_map);
 			spin_unlock(&mdsc->wait_list_lock);
-			mutex_unlock(&mdsc->mutex);
-			return;
+			goto out_session;
 		}
 		if (!(mdsc->fsc->mount_options->flags &
 		      CEPH_MOUNT_OPT_MOUNTWAIT) &&
@@ -3747,10 +3807,10 @@ static void __do_request(struct ceph_mds_client *mdsc,
 		trace_ceph_mdsc_suspend_request(mdsc, session, req,
 						ceph_mdsc_suspend_reason_no_active_mds);
 		spin_lock(&mdsc->wait_list_lock);
+		list_del_init(&req->r_wait);
 		list_add(&req->r_wait, &mdsc->waiting_for_map);
 		spin_unlock(&mdsc->wait_list_lock);
-		mutex_unlock(&mdsc->mutex);
-		return;
+		goto out_session;
 	}
 
 	/* get, open session */
@@ -3798,6 +3858,7 @@ static void __do_request(struct ceph_mds_client *mdsc,
 				trace_ceph_mdsc_suspend_request(mdsc, session, req,
 								ceph_mdsc_suspend_reason_rejected);
 				spin_lock(&mdsc->wait_list_lock);
+				list_del_init(&req->r_wait);
 				list_add(&req->r_wait, &mdsc->waiting_for_map);
 				spin_unlock(&mdsc->wait_list_lock);
 			} else {
@@ -3818,17 +3879,26 @@ static void __do_request(struct ceph_mds_client *mdsc,
 		trace_ceph_mdsc_suspend_request(mdsc, session, req,
 						ceph_mdsc_suspend_reason_session);
 		spin_lock(&mdsc->wait_list_lock);
+		list_del_init(&req->r_wait);
 		list_add(&req->r_wait, &session->s_waiting);
 		spin_unlock(&mdsc->wait_list_lock);
 		goto out_session;
 	}
 
+	/*
+	 * Everything that handle_forward() and cleanup_session_requests()
+	 * may rewrite under mdsc->mutex while the send window is open is
+	 * read or reset here, before the mutex is dropped: a forward
+	 * that arrives in the window sets a new r_resend_mds and
+	 * CEPH_MDS_R_RESEND, and must not be undone by this dispatch.
+	 */
 	req->r_attempts++;
+	req->r_send_attempts = req->r_attempts;
+	req->r_send_num_fwd = req->r_num_fwd;
+	req->r_resend_mds = -1;   /* forget any previous mds hint */
 	mutex_unlock(&mdsc->mutex);
 
 	/* send request */
-	req->r_resend_mds = -1;   /* forget any previous mds hint */
-
 	if (req->r_request_started == 0)   /* note request start time */
 		req->r_request_started = jiffies;
 
@@ -3840,7 +3910,8 @@ static void __do_request(struct ceph_mds_client *mdsc,
 	 *
 	 * And then the auth cap will be changed.
 	 */
-	if (test_bit(CEPH_MDS_R_ASYNC, &req->r_req_flags) && req->r_num_fwd) {
+	if (test_bit(CEPH_MDS_R_ASYNC, &req->r_req_flags) &&
+	    req->r_send_num_fwd) {
 		struct ceph_dentry_info *di = ceph_dentry(req->r_dentry);
 		struct ceph_inode_info *ci;
 		struct ceph_cap *cap;
@@ -3905,6 +3976,20 @@ static void __do_request(struct ceph_mds_client *mdsc,
 		complete_request(mdsc, req);
 		__unregister_request(mdsc, req);
 	}
+	/*
+	 * Release dispatch ownership.  If a session teardown or a
+	 * forward observed this request while the mutex was dropped
+	 * above, it set CEPH_MDS_R_RESEND under the mutex: consume it
+	 * here and re-evaluate, unless the request has already been
+	 * unregistered (error path above, or a racing reply).
+	 */
+	release_dispatch(req);
+	if (test_and_clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags) &&
+	    xa_load(&mdsc->request_tree, req->r_tid) == req) {
+		mutex_unlock(&mdsc->mutex);
+		goto restart;
+	}
+no_dispatch:
 	mutex_unlock(&mdsc->mutex);
 	return;
 }
@@ -3913,22 +3998,68 @@ static void __wake_requests(struct ceph_mds_client *mdsc,
 			    struct list_head *head)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
-	struct ceph_mds_request *req;
-	LIST_HEAD(tmp_list);
+	struct ceph_mds_request *req, *nreq;
+	LIST_HEAD(wake_list);
 
+	/*
+	 * Serialize the splice against the park decision in
+	 * __do_request(): both take mdsc->mutex, so a request cannot
+	 * be parked on @head after we have drained it, and everything
+	 * parked before we take the mutex is moved here.
+	 */
+	mutex_lock(&mdsc->mutex);
 	spin_lock(&mdsc->wait_list_lock);
-	list_splice_init(head, &tmp_list);
+	list_for_each_entry_safe(req, nreq, head, r_wait) {
+		/*
+		 * A non-empty r_aux_item means the request is already
+		 * owned by another dispatch collector (kick_requests()
+		 * or replay_unsafe_requests()); let it be dispatched by
+		 * that collector.  No stale waiter can result: a
+		 * kick-owned request was already delinked from r_wait
+		 * at claim time, and a replay-owned request is never on
+		 * a wait list (s_unsafe requests carry GOT_UNSAFE,
+		 * which __do_request() refuses to park, and replay's
+		 * old-request scan only takes r_attempts > 0 while
+		 * parked requests always have r_attempts == 0).
+		 */
+		if (!list_empty(&req->r_aux_item))
+			continue;
+		list_del_init(&req->r_wait);
+		/*
+		 * Pin the request in the same critical section that
+		 * delinks it from the wait list: once r_wait is
+		 * delinked, no list still owns it, so the reference
+		 * keeps the object alive while it is dispatched.
+		 */
+		ceph_mdsc_get_request(req);
+		list_add_tail(&req->r_aux_item, &wake_list);
+	}
 	spin_unlock(&mdsc->wait_list_lock);
+	mutex_unlock(&mdsc->mutex);
+
+	/*
+	 * Dispatch without holding the locks, but pop each node under
+	 * mdsc->mutex: r_aux_item doubles as the collector-ownership
+	 * predicate, so every access to it (collector list_empty()
+	 * checks and these pops) must be serialized by the mutex to
+	 * avoid a real data race on the node.
+	 */
+	mutex_lock(&mdsc->mutex);
+	while (!list_empty(&wake_list)) {
+		req = list_first_entry(&wake_list, struct ceph_mds_request,
+				       r_aux_item);
+		list_del_init(&req->r_aux_item);
+		mutex_unlock(&mdsc->mutex);
 
-	while (!list_empty(&tmp_list)) {
-		req = list_entry(tmp_list.next,
-				 struct ceph_mds_request, r_wait);
-		list_del_init(&req->r_wait);
 		doutc(cl, " wake request %p tid %llu\n", req,
 		      req->r_tid);
 		trace_ceph_mdsc_resume_request(mdsc, req);
 		__do_request(mdsc, req);
+		ceph_mdsc_put_request(req);
+
+		mutex_lock(&mdsc->mutex);
 	}
+	mutex_unlock(&mdsc->mutex);
 }
 
 /*
@@ -3938,7 +4069,7 @@ static void __wake_requests(struct ceph_mds_client *mdsc,
 static void kick_requests(struct ceph_mds_client *mdsc, int mds)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
-	struct ceph_mds_request *req, *nreq;
+	struct ceph_mds_request *req;
 	unsigned long idx;
 	LIST_HEAD(kick_list);
 
@@ -3958,19 +4089,42 @@ static void kick_requests(struct ceph_mds_client *mdsc, int mds)
 			spin_lock(&mdsc->wait_list_lock);
 			list_del_init(&req->r_wait);
 			spin_unlock(&mdsc->wait_list_lock);
+			/*
+			 * A non-empty r_aux_item means the request is
+			 * already owned by another dispatch collector
+			 * (__wake_requests() or replay_unsafe_requests());
+			 * never queue the same node twice.
+			 */
+			if (!list_empty(&req->r_aux_item)) {
+				ceph_mdsc_put_request(req);
+				continue;
+			}
 			list_add_tail(&req->r_aux_item, &kick_list);
 		}
 	}
 	mutex_unlock(&mdsc->mutex);
 
 	/* replay without the mutex */
-	list_for_each_entry_safe(req, nreq, &kick_list, r_aux_item) {
+	/*
+	 * Same as __wake_requests(): r_aux_item is the collector-ownership
+	 * predicate, so pops are done under mdsc->mutex; the dispatch
+	 * itself runs outside the locks.
+	 */
+	mutex_lock(&mdsc->mutex);
+	while (!list_empty(&kick_list)) {
+		req = list_first_entry(&kick_list, struct ceph_mds_request,
+				       r_aux_item);
+		list_del_init(&req->r_aux_item);
+		mutex_unlock(&mdsc->mutex);
+
 		doutc(cl, " kicking tid %llu\n", req->r_tid);
 		trace_ceph_mdsc_resume_request(mdsc, req);
-		list_del_init(&req->r_aux_item);
 		__do_request(mdsc, req);
 		ceph_mdsc_put_request(req);
+
+		mutex_lock(&mdsc->mutex);
 	}
+	mutex_unlock(&mdsc->mutex);
 }
 
 int ceph_mdsc_submit_request(struct ceph_mds_client *mdsc, struct inode *dir,
@@ -4423,6 +4577,15 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 		req->r_num_fwd = fwd_seq;
 		req->r_resend_mds = next_mds;
 		put_request_session(req);
+		/*
+		 * If a dispatch owner is in the send window, the
+		 * __do_request() below cannot claim the request and the
+		 * forward would be lost; set CEPH_MDS_R_RESEND so that
+		 * the owner re-evaluates the request when it releases
+		 * dispatch ownership.
+		 */
+		if (test_bit(CEPH_MDS_R_DISPATCHING, &req->r_req_flags))
+			set_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
 	}
 	mutex_unlock(&mdsc->mutex);
 
@@ -4804,21 +4967,86 @@ void ceph_mdsc_release_dir_caps_async(struct ceph_mds_request *req)
 	}
 }
 
+enum {
+	REPLAY_SKIP,		/* already queued by a collector */
+	REPLAY_QUEUED,		/* claimed and queued on the replay list */
+	REPLAY_BUSY,		/* a dispatch owner is in its send window */
+};
+
+/*
+ * Claim dispatch ownership of @req for a replay and queue it on @list.
+ * Called under mdsc->mutex.
+ */
+static int replay_claim_request(struct ceph_mds_client *mdsc,
+				struct ceph_mds_request *req,
+				struct list_head *list)
+{
+	lockdep_assert_held(&mdsc->mutex);
+
+	/*
+	 * A non-empty r_aux_item means the request is already owned by
+	 * a dispatch collector: either this replay queued it on an
+	 * earlier pass, or kick_requests()/__wake_requests() hold it and
+	 * will run it through __do_request(), which parks it on
+	 * s_waiting while the session is reconnecting.
+	 */
+	if (!list_empty(&req->r_aux_item))
+		return REPLAY_SKIP;
+	/*
+	 * Replay sends __send_request() directly, so it must claim
+	 * dispatch ownership to exclude a concurrent __do_request()
+	 * from the send window.  If another context holds it, the
+	 * caller waits for it to be released and rescans: skipping the
+	 * request would lose it when that dispatch had already queued
+	 * its message on the connection this reconnect has just reset.
+	 */
+	if (test_and_set_bit(CEPH_MDS_R_DISPATCHING, &req->r_req_flags))
+		return REPLAY_BUSY;
+	/* the replay send below supersedes any pending resend */
+	clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+	ceph_mdsc_get_request(req);
+	req->r_attempts++;
+	req->r_send_attempts = req->r_attempts;
+	req->r_send_num_fwd = req->r_num_fwd;
+	list_add_tail(&req->r_aux_item, list);
+	return REPLAY_QUEUED;
+}
+
 /*
  * called under session->mutex.
  */
 static void replay_unsafe_requests(struct ceph_mds_client *mdsc,
 				   struct ceph_mds_session *session)
 {
-	struct ceph_mds_request *req, *nreq;
+	struct ceph_mds_request *req, *nreq, *busy;
 	unsigned long idx;
+	LIST_HEAD(unsafe_list);
+	LIST_HEAD(old_list);
 
 	doutc(mdsc->fsc->client, "mds%d\n", session->s_mds);
 
+	/*
+	 * Collect unsafe and old requests under mdsc->mutex, then
+	 * replay them without it: __send_request() is lockless and
+	 * ceph_mdsc_release_dir_caps_async() schedules work.
+	 */
 	mutex_lock(&mdsc->mutex);
-	list_for_each_entry_safe(req, nreq, &session->s_unsafe, r_unsafe_item) {
-		req->r_attempts++;
-		__send_request(session, req, true);
+rescan:
+	busy = NULL;
+	list_for_each_entry_safe(req, nreq, &session->s_unsafe,
+				 r_unsafe_item) {
+		/*
+		 * Keep the request on s_unsafe: r_aux_item is only a
+		 * walk list.  The request must stay tracked as unsafe
+		 * until the MDS replies, so that a later reconnect can
+		 * replay it again and cleanup_session_requests() can
+		 * still abort it on session teardown.
+		 */
+		if (replay_claim_request(mdsc, req, &unsafe_list) ==
+		    REPLAY_BUSY) {
+			busy = req;
+			break;
+		}
 	}
 
 	/*
@@ -4826,20 +5054,121 @@ static void replay_unsafe_requests(struct ceph_mds_client *mdsc,
 	 * can process completed request in clientreplay stage.
 	 */
 	idx = 0;
-	xa_for_each(&mdsc->request_tree, idx, req) {
-		if (test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags))
-			continue;
-		if (req->r_attempts == 0)
-			continue; /* only old requests */
-		if (!req->r_session)
-			continue;
-		if (req->r_session->s_mds != session->s_mds)
-			continue;
+	if (!busy) {
+		xa_for_each(&mdsc->request_tree, idx, req) {
+			if (test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags))
+				continue;
+			if (req->r_attempts == 0)
+				continue; /* only old requests */
+			if (!req->r_session)
+				continue;
+			if (req->r_session->s_mds != session->s_mds)
+				continue;
+			if (replay_claim_request(mdsc, req, &old_list) ==
+			    REPLAY_BUSY) {
+				busy = req;
+				break;
+			}
+		}
+	}
 
-		ceph_mdsc_release_dir_caps_async(req);
+	if (busy) {
+		/*
+		 * Wait for the owner to leave its send window, then scan
+		 * again: the request may have been answered, forwarded,
+		 * parked or unregistered meanwhile, and the lists may
+		 * have changed while mdsc->mutex was dropped.  Requests
+		 * already claimed above stay queued (their r_aux_item is
+		 * set, so the rescan skips them).  The owner does not
+		 * take session->s_mutex in its send window, so waiting
+		 * here with it held cannot deadlock.  A later
+		 * __do_request() sees CEPH_MDS_SESSION_RECONNECTING and
+		 * only parks the request on s_waiting.
+		 */
+		ceph_mdsc_get_request(busy);
+		mutex_unlock(&mdsc->mutex);
+		doutc(mdsc->fsc->client, "mds%d waiting for tid %llu\n",
+		      session->s_mds, busy->r_tid);
+		set_bit(CEPH_MDS_R_REPLAY_WAIT, &busy->r_req_flags);
+		/*
+		 * Ask release_dispatch() for a wake-up before testing the
+		 * bit; pairs with smp_mb__after_atomic() there.  A stale
+		 * REPLAY_WAIT, left when the bit was already clear, only
+		 * costs the next release one spurious wake_up_bit().
+		 */
+		smp_mb__after_atomic();
+		wait_on_bit(&busy->r_req_flags, CEPH_MDS_R_DISPATCHING,
+			    TASK_UNINTERRUPTIBLE);
+		ceph_mdsc_put_request(busy);
+		mutex_lock(&mdsc->mutex);
+		goto rescan;
+	}
+
+	mutex_unlock(&mdsc->mutex);
+
+	/*
+	 * Same as __wake_requests(): r_aux_item is the
+	 * collector-ownership predicate, so every access to it (the
+	 * list_empty() checks in the collectors above and these pops)
+	 * is serialized by mdsc->mutex.  The send itself runs outside
+	 * the locks.
+	 */
+
+	/* replay unsafe requests */
+	mutex_lock(&mdsc->mutex);
+	while (!list_empty(&unsafe_list)) {
+		bool resend;
+
+		req = list_first_entry(&unsafe_list, struct ceph_mds_request,
+				       r_aux_item);
+		list_del_init(&req->r_aux_item);
+		mutex_unlock(&mdsc->mutex);
 
-		req->r_attempts++;
 		__send_request(session, req, true);
+
+		mutex_lock(&mdsc->mutex);
+		release_dispatch(req);
+		resend = test_and_clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+		mutex_unlock(&mdsc->mutex);
+		/*
+		 * A teardown or forward may have requested a re-evaluation
+		 * while this replay send was in flight.  Re-dispatch
+		 * through the normal path, which claims dispatch ownership
+		 * itself: if the request has become unsafe meanwhile,
+		 * __do_request() sees GOT_UNSAFE and bails out, leaving
+		 * the request queued for the next replay; otherwise it is
+		 * sent again normally.
+		 */
+		if (resend)
+			__do_request(mdsc, req);
+		ceph_mdsc_put_request(req);
+
+		mutex_lock(&mdsc->mutex);
+	}
+	mutex_unlock(&mdsc->mutex);
+
+	/* replay old requests */
+	mutex_lock(&mdsc->mutex);
+	while (!list_empty(&old_list)) {
+		bool resend;
+
+		req = list_first_entry(&old_list, struct ceph_mds_request,
+				       r_aux_item);
+		list_del_init(&req->r_aux_item);
+		mutex_unlock(&mdsc->mutex);
+
+		ceph_mdsc_release_dir_caps_async(req);
+		__send_request(session, req, true);
+
+		mutex_lock(&mdsc->mutex);
+		release_dispatch(req);
+		resend = test_and_clear_bit(CEPH_MDS_R_RESEND, &req->r_req_flags);
+		mutex_unlock(&mdsc->mutex);
+		if (resend)
+			__do_request(mdsc, req);
+		ceph_mdsc_put_request(req);
+
+		mutex_lock(&mdsc->mutex);
 	}
 	mutex_unlock(&mdsc->mutex);
 }
@@ -5399,9 +5728,10 @@ static int send_mds_reconnect(struct ceph_mds_client *mdsc,
 
 	mutex_unlock(&session->s_mutex);
 
+	up_read(&mdsc->snap_rwsem);
+
 	__wake_requests(mdsc, &session->s_waiting);
 
-	up_read(&mdsc->snap_rwsem);
 	ceph_pagelist_release(recon_state.pagelist);
 	return 0;
 
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 11059678284d..914b3c03fdb0 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -359,6 +359,39 @@ struct ceph_mds_request {
 #define CEPH_MDS_R_PARENT_LOCKED	(7) /* is r_parent->i_rwsem wlocked? */
 #define CEPH_MDS_R_ASYNC		(8) /* async request */
 #define CEPH_MDS_R_FSCRYPT_FILE		(9) /* must marshal fscrypt_file field */
+/*
+ * A dispatch owner has claimed the right to rebuild and send
+ * req->r_request (__do_request() past the send gate, or
+ * replay_unsafe_requests()).  Set and cleared under mdsc->mutex;
+ * held across the unlocked send window, cleared by the owner when it
+ * re-acquires the mutex and releases ownership.  Release it with
+ * release_dispatch(): replay_unsafe_requests() waits for it with
+ * wait_on_bit() instead of skipping a request in flight, and flags
+ * the wait with CEPH_MDS_R_REPLAY_WAIT.
+ */
+#define CEPH_MDS_R_DISPATCHING		(10)
+/*
+ * Another context observed a session/request state change while a
+ * dispatch owner was in flight and wants a re-evaluation.  It is a
+ * request to re-evaluate, not a guarantee that a resend is still
+ * needed.
+ *
+ * If RESEND is set while DISPATCHING is held, the current dispatcher
+ * consumes it on release and redispatches.
+ * If RESEND is set after DISPATCHING has been cleared, the producer
+ * must itself trigger a redispatch: cleanup_session_requests() is
+ * followed by kick_requests(), while handle_forward() calls
+ * __do_request() directly.
+ */
+#define CEPH_MDS_R_RESEND		(11)
+/*
+ * replay_unsafe_requests() is waiting for the dispatch owner to clear
+ * CEPH_MDS_R_DISPATCHING.  Lets release_dispatch() skip wake_up_bit(),
+ * and its lookup in the shared bit_wait_table, in the common case where
+ * nobody waits.  Set by the waiter, cleared by the releaser that wakes
+ * it up.
+ */
+#define CEPH_MDS_R_REPLAY_WAIT		(12)
 	unsigned long	r_req_flags;
 
 	struct mutex r_fill_mutex;
@@ -420,6 +453,16 @@ struct ceph_mds_request {
 	int               r_attempts;   /* resend attempts */
 	int               r_num_fwd;    /* number of forward attempts */
 	int               r_resend_mds; /* mds to resend to next, if any*/
+	/*
+	 * Copies of r_attempts and r_num_fwd taken under mdsc->mutex by
+	 * the dispatch owner when it claims a send.  The owner encodes
+	 * the message from these in the unlocked send window, while
+	 * cleanup_session_requests() and handle_forward() may rewrite
+	 * r_attempts and r_num_fwd under mdsc->mutex.  Written and read
+	 * only by the CEPH_MDS_R_DISPATCHING owner.
+	 */
+	int               r_send_attempts;
+	int               r_send_num_fwd;
 	u32               r_sent_on_mseq; /* cap mseq request was sent at*/
 	u64		  r_deleg_ino;
 
@@ -428,7 +471,13 @@ struct ceph_mds_request {
 	struct completion r_safe_completion;
 	ceph_mds_request_callback_t r_callback;
 	struct list_head  r_unsafe_item;  /* per-session unsafe list item */
-	struct list_head  r_aux_item;     /* auxiliary local list item */
+	/*
+	 * Auxiliary walk list for the dispatch collectors; doubles as
+	 * the collector-ownership token.  INIT_LIST_HEAD() is done
+	 * before publication; afterwards, all accesses are serialized
+	 * by mdsc->mutex.
+	 */
+	struct list_head  r_aux_item;
 
 	long long	  r_dir_release_cnt;
 	long long	  r_dir_ordered_cnt;

-- 
2.53.0



      parent reply	other threads:[~2026-11-20  7:11 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  7:11 [PATCH v7 0/5] ceph: reduce mdsc->mutex contention in the cephfs kclient Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 1/5] ceph: use READ_ONCE/WRITE_ONCE for oldest_tid Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 2/5] ceph: replace the request_tree rbtree with an xarray keyed by r_tid Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 3/5] ceph: add wait_list_lock for wait-list serialization Xiubo Li via B4 Relay
2026-11-20  7:11 ` [PATCH v7 4/5] ceph: move mdsc->mutex into __do_request() Xiubo Li via B4 Relay
2026-11-20  7:11 ` Xiubo Li via B4 Relay [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120-ceph-mdsc-mutex-optimization-v7-5-a4eee70e0038@clyso.com \
    --to=devnull+xiubo.li.clyso.com@kernel.org \
    --cc=amarkuze@redhat.com \
    --cc=ceph-devel@vger.kernel.org \
    --cc=idryomov@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=slava@dubeyko.com \
    --cc=xiubo.li@clyso.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®