mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption
@ 2026-11-20  7:34 Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li,
	Viacheslav Dubeyko, Sebastian Andrzej Siewior, linux-kernel

When the Ceph daemons become unreachable for a long time (seen during
Rook-Ceph rolling upgrades with active CephFS I/O), the kernel client
leaves sync(2), syncfs(2) and flock() stuck in D state indefinitely,
keeps backing off on a purely local connect() error, and can miss the
mdsmap update that carries the new MDS address.

This v2 is rebased on next-20261119 and reworked after the v1 review
and further investigation.  The patches:

  1    libceph: fixed 100ms retry instead of exponential backoff when
       connect() fails locally with EADDRNOTAVAIL
  2-4  bound the sync(2)/syncfs(2) and lock-interrupt waits by
       mount_timeout; mount_timeout=0 keeps the current wait-forever
       behaviour, and nothing queued is dropped
  5    default r_timeout of MDS requests to mount_timeout (unchanged
       from v1)
  6    request a full mdsmap when an MDS keeps faulting or a session
       goes HUNG
  7    don't escalate the monitor hunt backoff on EADDRNOTAVAIL

Patch 2 is built on top of the wait_caps_flush() diagnostic loop from
ebbbab66bd74 ("ceph: add diagnostic timeout loop to wait_caps_flush()")
and keeps it.  I know that patch deliberately left the wait unbounded
and that b52861897be8 ("ceph: add client reset state machine and
session teardown") added a manual escape hatch.  Patches 2-5 are meant
for the case where nobody is around to trigger that reset, with
mount_timeout as the admin-controlled bound already used for umount.
If you'd rather keep sync unbounded, patches 2-5 can be dropped without
affecting the rest.

Changes since v1:
- Dropped "ceph: make ceph_start_io_write() killable" and "ceph: make
  remaining I/O lock functions killable": already upstream as
  fa073039466f ("ceph: make ceph_start_io_*() killable").
- Dropped "ceph: add timeout to caps wait in __ceph_get_caps()": covered
  by 50958bb928ba ("ceph: fix hanging __ceph_get_caps() with stale
  mds_wanted").
- Dropped "libceph: force host network namespace for kernel CephFS
  mounts", as discussed with Ilya.  The root cause was the Rook CSI holder
  pod's network namespace being torn down during the Rook upgrade.
- Dropped "libceph: reset source address on persistent EADDRNOTAVAIL".
  ceph_tcp_connect() never binds to msgr->inst.addr, so blanking that
  address cannot influence source address selection; the patch had
  no effect on the error it was meant to fix.
- Dropped "ceph: fix race condition in cleanup_session_requests()".
  Slava, sorry, I have to withdraw this one despite your Reviewed-by:
  unregistering requests that have no unsafe reply yet removes them
  from request_tree, so kick_requests() can no longer resend them after
  an MDS failover (mds_peer_reset(), check_new_map()), and the caller
  in ceph_mdsc_wait_request() is never completed.  The sync hang it
  targeted is now bounded by patch 2.
- Patch 1 (Slava): the flag now lives in struct ceph_connection
  instead of the v1/v2 union members, so the ceph_msgr2() checks and
  the local variable are gone.  The delay is msecs_to_jiffies(100)
  rather than HZ/10.  The point is that it no longer grows, which
  BASE_DELAY_INTERVAL would also give.  I'm happy to switch to that if
  you prefer one constant fewer.  Dropped the Fixes: tag, since this is
  a behaviour change rather than a regression fix.
- Patches 2-3 (Slava): ceph_mdsc_sync() returns 0 again on shutdown
  instead of -EIO.  The MDS waits share a single mount_timeout deadline
  instead of one per request, and so do the OSD waits.  Timeout
  messages are now dout()s.  On mount_timeout values: it is already range
  checked at parse time and is the admin's bound for the existing
  umount and wait_requests() waits.  0 still means forever, so I
  didn't add another clamp.
- Patch 4 (Slava): dout() instead of pr_warn(), if/else-if style, and
  the result of wait_for_completion_killable_timeout() is kept in a
  long (v1 truncated it to int).
- Patch 6 (Slava): s_con_failures is a u8 that wraps back to 0 at the
  threshold, with no local variable or modulo arithmetic.  The new
  messages are dout()s, the existing "mds%d hung" pr_info() is kept,
  and the threshold of 10 is explained (about a minute of messenger
  backoff).
- Patch 7 rewritten: v1 called reopen_session() from mon_fault() while
  hunting.  ceph_con_open() resets con->delay and queues immediately,
  so with an instant EADDRNOTAVAIL that spun the monitor connection
  with no delay at all.  It now only keeps hunt_mult from growing.
- Fixed a wrong Fixes: tag (the cleanup_session_requests() patch
  pointed at a non-existent commit) and dropped a reference to a local
  commit id.

Answering Ilya's questions on the earlier standalone EADDRNOTAVAIL
patch: the cluster has 2 monitors and, as you said, the client talks to
one at a time.  The 36 minutes were mostly the external condition
itself (the dead holder-pod namespace); patches 1 and 7 only remove
the extra wait once that condition clears.  Sebastian asked whether
this is PREEMPT_RT specific.  It isn't: none of these paths depend on
PREEMPT_RT.

Built with W=1 and sparse, and bisected build by build, on
next-20261119.  The v1 testing was done on 6.12.y; v2 has only been
build-tested, not run against a live cluster.

I used an LLM to help rebase the series, review the code and draft
the changelogs.  I have reviewed every change and take responsibility
for it.

v1: https://lore.kernel.org/all/20260312081619.40854-1-ionut.nechita@windriver.com/

Ionut Nechita (7):
  libceph: don't back off exponentially on EADDRNOTAVAIL
  ceph: bound the ceph_mdsc_sync() wait by mount_timeout
  ceph: bound the ceph_osdc_sync() wait by mount_timeout
  ceph: bound the safe-completion wait in
    ceph_lock_wait_for_completion()
  ceph: set default timeout for MDS requests
  ceph: request a full mdsmap when an MDS stays unreachable
  libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL

 fs/ceph/locks.c                 |  15 +++-
 fs/ceph/mds_client.c            | 133 +++++++++++++++++++++++++++-----
 fs/ceph/mds_client.h            |   4 +-
 fs/ceph/super.c                 |   9 ++-
 include/linux/ceph/messenger.h  |   7 ++
 include/linux/ceph/osd_client.h |   2 +-
 net/ceph/messenger.c            |  15 +++-
 net/ceph/mon_client.c           |   8 ++
 net/ceph/osd_client.c           |  24 +++++-
 9 files changed, 182 insertions(+), 35 deletions(-)


base-commit: adcad7f4e6332cff3d6d7165f41584899f598534
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

When kernel_connect() fails with EADDRNOTAVAIL, the error is local:
no route or no usable source address for the peer, and nothing was
sent on the wire.  con_fault() nevertheless treats it like any other
failure and doubles con->delay up to MAX_DELAY_INTERVAL (15s), so once
the local condition clears the client may sit idle for up to 15s
before trying again.

Since a failed attempt costs nothing and never reaches the peer,
retry at a fixed 100ms instead.  Track the condition in struct
ceph_connection so that it doesn't depend on which protocol (msgr1 or
msgr2) is in use, and rate-limit the message.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 include/linux/ceph/messenger.h |  7 +++++++
 net/ceph/messenger.c           | 15 ++++++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 6aa4c6478c9f6..344496967b5c5 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -321,6 +321,12 @@ struct ceph_msg {
 /* ceph connection fault delay defaults, for exponential backoff */
 #define BASE_DELAY_INTERVAL	(HZ / 4)
 #define MAX_DELAY_INTERVAL	(15 * HZ)
+/*
+ * Fixed retry delay used instead of exponential backoff when connect()
+ * fails locally with EADDRNOTAVAIL.  Such an attempt never reaches the
+ * peer, so backing off protects nobody and only delays recovery.
+ */
+#define ADDRNOTAVAIL_DELAY	msecs_to_jiffies(100)
 
 struct ceph_connection_v1_info {
 	struct kvec out_kvec[8],         /* sending header/footer data */
@@ -507,6 +513,7 @@ struct ceph_connection {
 
 	struct delayed_work work;	    /* send|recv work */
 	unsigned long       delay;          /* current delay interval */
+	bool addr_notavail;  /* last connect() failed with EADDRNOTAVAIL */
 
 	union {
 		struct ceph_connection_v1_info v1;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index cc7db8c61db37..3c893bc0f7d8b 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -467,8 +467,13 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		     ceph_pr_addr(&con->peer_addr),
 		     sock->sk->sk_state);
 	} else if (ret < 0) {
-		pr_err("connect %s error %d\n",
-		       ceph_pr_addr(&con->peer_addr), ret);
+		if (ret == -EADDRNOTAVAIL)
+			pr_warn_ratelimited("connect %s: address not available, will retry\n",
+					    ceph_pr_addr(&con->peer_addr));
+		else
+			pr_err("connect %s error %d\n",
+			       ceph_pr_addr(&con->peer_addr), ret);
+		con->addr_notavail = ret == -EADDRNOTAVAIL;
 		sock_release(sock);
 		return ret;
 	}
@@ -477,6 +482,7 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		tcp_sock_set_nodelay(sock->sk);
 
 	con->sock = sock;
+	con->addr_notavail = false;
 	return 0;
 }
 
@@ -610,6 +616,7 @@ void ceph_con_open(struct ceph_connection *con,
 
 	memcpy(&con->peer_addr, addr, sizeof(*addr));
 	con->delay = 0;      /* reset backoff memory */
+	con->addr_notavail = false;
 	mutex_unlock(&con->mutex);
 	queue_con(con);
 }
@@ -1644,7 +1651,9 @@ static void con_fault(struct ceph_connection *con)
 	} else {
 		/* retry after a delay. */
 		con->state = CEPH_CON_S_PREOPEN;
-		if (!con->delay) {
+		if (con->addr_notavail) {
+			con->delay = ADDRNOTAVAIL_DELAY;
+		} else if (!con->delay) {
 			con->delay = BASE_DELAY_INTERVAL;
 		} else if (con->delay < MAX_DELAY_INTERVAL) {
 			con->delay *= 2;
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

When the MDS is unreachable, sync(2) and syncfs(2) block forever in
ceph_mdsc_sync(): flush_mdlog_and_wait_mdsc_unsafe_requests() waits
for r_safe_completion and wait_caps_flush() waits for the cap flushes
without any bound.  The diagnostic loop added to wait_caps_flush()
makes the stall visible, but the task still can't make progress and
keeps tripping the hung task detector:

  INFO: task sync:514917 blocked for more than 983 seconds.
  Call Trace:
   ceph_mdsc_sync+0x4d6/0x5a0 [ceph]
   ceph_sync_fs+0x31/0x130 [ceph]
   iterate_supers+0x97/0x100
   ksys_sync+0x32/0xb0

Bound both waits by a single mount_timeout deadline, like
wait_requests() and ceph_mdsc_close_sessions() already do, and
return -ETIMEDOUT from ceph_sync_fs().  Nothing is discarded: unsafe
requests and dirty caps stay queued and are flushed once the MDS is
reachable again.  mount_timeout=0 keeps the current wait-forever
behaviour.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 84 +++++++++++++++++++++++++++++++++-----------
 fs/ceph/mds_client.h |  2 +-
 fs/ceph/super.c      |  5 +--
 3 files changed, 67 insertions(+), 24 deletions(-)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 3791fc627714c..d02fe1e34527b 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2520,11 +2520,24 @@ static void dump_cap_flushes(struct ceph_mds_client *mdsc, u64 want_tid)
 }
 
 /*
- * Wait for all cap flushes through @want_flush_tid to complete.
- * Periodically dumps pending cap flush state for diagnostics.
+ * Time left until the sync deadline, or MAX_SCHEDULE_TIMEOUT if
+ * mount_timeout=0 (wait forever).
  */
-static void wait_caps_flush(struct ceph_mds_client *mdsc,
-			    u64 want_flush_tid)
+static unsigned long sync_time_left(struct ceph_mds_client *mdsc,
+				    unsigned long deadline)
+{
+	if (!mdsc->fsc->client->options->mount_timeout)
+		return MAX_SCHEDULE_TIMEOUT;
+	return time_before(jiffies, deadline) ? deadline - jiffies : 0;
+}
+
+/*
+ * Wait for all cap flushes through @want_flush_tid to complete or for
+ * @deadline to pass.  Periodically dumps pending cap flush state for
+ * diagnostics.
+ */
+static int wait_caps_flush(struct ceph_mds_client *mdsc,
+			   u64 want_flush_tid, unsigned long deadline)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
 	int i = 0;
@@ -2532,22 +2545,31 @@ static void wait_caps_flush(struct ceph_mds_client *mdsc,
 
 	doutc(cl, "want %llu\n", want_flush_tid);
 
-	do {
+	while (true) {
 		/* 60 * HZ fits in a long on all supported architectures. */
 		ret = wait_event_timeout(mdsc->cap_flushing_wq,
 			   check_caps_flush(mdsc, want_flush_tid),
-			   CEPH_CAP_FLUSH_WAIT_TIMEOUT_SEC * HZ);
-		if (ret == 0) {
-			if (i < CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
-				dump_cap_flushes(mdsc, want_flush_tid);
-			else if (i == CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
-				pr_info_client(cl,
-					       "still waiting for cap flushes; suppressing further dumps\n");
-			i++;
+			   min_t(unsigned long, sync_time_left(mdsc, deadline),
+				 CEPH_CAP_FLUSH_WAIT_TIMEOUT_SEC * HZ));
+		if (ret)
+			break;
+
+		if (i < CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
+			dump_cap_flushes(mdsc, want_flush_tid);
+		else if (i == CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
+			pr_info_client(cl,
+				       "still waiting for cap flushes; suppressing further dumps\n");
+		i++;
+
+		if (!sync_time_left(mdsc, deadline)) {
+			doutc(cl, "timed out waiting for flush tid %llu\n",
+			      want_flush_tid);
+			return -ETIMEDOUT;
 		}
-	} while (ret == 0);
+	}
 
 	doutc(cl, "ok, flushed thru %llu\n", want_flush_tid);
+	return 0;
 }
 
 /*
@@ -6737,13 +6759,15 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)
 /*
  * flush the mdlog and wait for all write mds requests to flush.
  */
-static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
-						 u64 want_tid)
+static int flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
+						      u64 want_tid,
+						      unsigned long deadline)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
 	struct ceph_mds_request *req = NULL, *nextreq;
 	struct ceph_mds_session *last_session = NULL;
 	struct rb_node *n;
+	unsigned long left;
 
 	mutex_lock(&mdsc->mutex);
 	doutc(cl, "want %lld\n", want_tid);
@@ -6782,7 +6806,18 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
 			}
 			doutc(cl, "wait on %llu (want %llu)\n",
 			      req->r_tid, want_tid);
-			wait_for_completion(&req->r_safe_completion);
+			left = sync_time_left(mdsc, deadline);
+			left = wait_for_completion_timeout(&req->r_safe_completion,
+							   left);
+			if (!left) {
+				doutc(cl, "timed out waiting for tid %llu\n",
+				      req->r_tid);
+				ceph_mdsc_put_request(req);
+				if (nextreq)
+					ceph_mdsc_put_request(nextreq);
+				ceph_put_mds_session(last_session);
+				return -ETIMEDOUT;
+			}
 
 			mutex_lock(&mdsc->mutex);
 			ceph_mdsc_put_request(req);
@@ -6800,15 +6835,18 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
 	mutex_unlock(&mdsc->mutex);
 	ceph_put_mds_session(last_session);
 	doutc(cl, "done\n");
+	return 0;
 }
 
-void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
+int ceph_mdsc_sync(struct ceph_mds_client *mdsc)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
+	unsigned long deadline = jiffies + cl->options->mount_timeout;
 	u64 want_tid, want_flush;
+	int ret;
 
 	if (READ_ONCE(mdsc->fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN)
-		return;
+		return 0;
 
 	doutc(cl, "sync\n");
 	mutex_lock(&mdsc->mutex);
@@ -6829,8 +6867,12 @@ void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
 
 	doutc(cl, "sync want tid %lld flush_seq %lld\n", want_tid, want_flush);
 
-	flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
-	wait_caps_flush(mdsc, want_flush);
+	ret = flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid,
+							deadline);
+	if (ret)
+		return ret;
+
+	return wait_caps_flush(mdsc, want_flush, deadline);
 }
 
 /*
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 4749438d94e1f..31abbb3b3bdb3 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -635,7 +635,7 @@ extern void ceph_mdsc_close_sessions(struct ceph_mds_client *mdsc);
 extern void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc);
 extern void ceph_mdsc_destroy(struct ceph_fs_client *fsc);
 
-extern void ceph_mdsc_sync(struct ceph_mds_client *mdsc);
+extern int ceph_mdsc_sync(struct ceph_mds_client *mdsc);
 
 extern void ceph_invalidate_dir_request(struct ceph_mds_request *req);
 extern int ceph_alloc_readdir_reply_buffer(struct ceph_mds_request *req,
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index 7b5ef76e430d7..7936b76aa3cc0 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -126,6 +126,7 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 {
 	struct ceph_fs_client *fsc = ceph_sb_to_fs_client(sb);
 	struct ceph_client *cl = fsc->client;
+	int ret;
 
 	if (!wait) {
 		doutc(cl, "(non-blocking)\n");
@@ -137,9 +138,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 
 	doutc(cl, "(blocking)\n");
 	ceph_osdc_sync(&fsc->client->osdc);
-	ceph_mdsc_sync(fsc->mdsc);
+	ret = ceph_mdsc_sync(fsc->mdsc);
 	doutc(cl, "(blocking) done\n");
-	return 0;
+	return ret;
 }
 
 /*
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() wait by mount_timeout
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

ceph_sync_fs() calls ceph_osdc_sync() before ceph_mdsc_sync(), and
ceph_osdc_sync() waits for every in-flight write with a plain
wait_for_completion().  With an OSD unreachable, sync(2) therefore
hangs there and never gets to the bounded MDS wait:

  ceph_osdc_sync [libceph]
  ceph_sync_fs [ceph]
  iterate_supers
  ksys_sync

Bound the wait by mount_timeout in the same way and propagate
-ETIMEDOUT.  The requests are not cancelled and complete once the OSD
is reachable again.  mount_timeout=0 keeps waiting forever.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/super.c                 |  4 +++-
 include/linux/ceph/osd_client.h |  2 +-
 net/ceph/osd_client.c           | 24 ++++++++++++++++++++++--
 3 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index 7936b76aa3cc0..166f6eb21f0af 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -137,7 +137,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 	}
 
 	doutc(cl, "(blocking)\n");
-	ceph_osdc_sync(&fsc->client->osdc);
+	ret = ceph_osdc_sync(&fsc->client->osdc);
+	if (ret)
+		return ret;
 	ret = ceph_mdsc_sync(fsc->mdsc);
 	doutc(cl, "(blocking) done\n");
 	return ret;
diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
index 7349ca09a227a..674dd9221d657 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -579,7 +579,7 @@ void ceph_osdc_start_request(struct ceph_osd_client *osdc,
 extern void ceph_osdc_cancel_request(struct ceph_osd_request *req);
 extern int ceph_osdc_wait_request(struct ceph_osd_client *osdc,
 				  struct ceph_osd_request *req);
-extern void ceph_osdc_sync(struct ceph_osd_client *osdc);
+extern int ceph_osdc_sync(struct ceph_osd_client *osdc);
 
 extern void ceph_osdc_flush_notifies(struct ceph_osd_client *osdc);
 void ceph_osdc_maybe_request_map(struct ceph_osd_client *osdc);
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 4479ec7a932b3..417374c95dc14 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -4715,11 +4715,17 @@ EXPORT_SYMBOL(ceph_osdc_wait_request);
 
 /*
  * sync - wait for all in-flight requests to flush.  avoid starvation.
+ *
+ * Gives up with -ETIMEDOUT after mount_timeout (0 means wait forever),
+ * leaving the requests in flight.
  */
-void ceph_osdc_sync(struct ceph_osd_client *osdc)
+int ceph_osdc_sync(struct ceph_osd_client *osdc)
 {
+	unsigned long timeout = osdc->client->options->mount_timeout;
+	unsigned long deadline = jiffies + timeout;
 	struct rb_node *n, *p;
 	u64 last_tid = atomic64_read(&osdc->last_tid);
+	unsigned long left;
 
 again:
 	down_read(&osdc->lock);
@@ -4742,7 +4748,20 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
 			up_read(&osdc->lock);
 			dout("%s waiting on req %p tid %llu last_tid %llu\n",
 			     __func__, req, req->r_tid, last_tid);
-			wait_for_completion(&req->r_completion);
+			if (!timeout)
+				left = MAX_SCHEDULE_TIMEOUT;
+			else if (time_before(jiffies, deadline))
+				left = deadline - jiffies;
+			else
+				left = 0;
+			left = wait_for_completion_timeout(&req->r_completion,
+							   left);
+			if (!left) {
+				dout("%s timed out on req %p tid %llu\n",
+				     __func__, req, req->r_tid);
+				ceph_osdc_put_request(req);
+				return -ETIMEDOUT;
+			}
 			ceph_osdc_put_request(req);
 			goto again;
 		}
@@ -4752,6 +4771,7 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
 
 	up_read(&osdc->lock);
 	dout("%s done last_tid %llu\n", __func__, last_tid);
+	return 0;
 }
 EXPORT_SYMBOL(ceph_osdc_sync);
 
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion()
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
                   ` (2 preceding siblings ...)
  2026-11-20  7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

When a blocking lock request is interrupted,
ceph_lock_wait_for_completion() sends an unlock request to cancel it
and then waits for the original request's r_safe_completion with no
bound.  If the MDS goes away at
that point the task hangs in D state:

  INFO: task flock:12345 blocked for more than 122 seconds.
  Call Trace:
   wait_for_completion_killable
   ceph_lock_wait_for_completion
   ceph_flock

Bound the wait by mount_timeout and return -ETIMEDOUT; the original
request has already been marked CEPH_MDS_R_ABORTED at this point.
mount_timeout=0 keeps waiting forever.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/locks.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index 677221bd64e01..b4faf112b3f84 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -158,9 +158,11 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
                                          struct ceph_mds_request *req)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
+	unsigned long timeout = ceph_timeout_jiffies(cl->options->mount_timeout);
 	struct ceph_mds_request *intr_req;
 	struct inode *inode = req->r_inode;
 	int err, lock_type;
+	long left;
 
 	BUG_ON(req->r_op != CEPH_MDS_OP_SETFILELOCK);
 	if (req->r_args.filelock_change.rule == CEPH_LOCK_FCNTL)
@@ -219,10 +221,15 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
 	if (err && err != -ERESTARTSYS)
 		return err;
 
-	err = wait_for_completion_killable(&req->r_safe_completion);
-	if (err)
-		return err;
-
+	left = wait_for_completion_killable_timeout(&req->r_safe_completion,
+						    timeout);
+	if (left < 0) {
+		return left;
+	} else if (left == 0) {
+		doutc(cl, "request %llu safe completion timed out\n",
+		      req->r_tid);
+		return -ETIMEDOUT;
+	}
 	return 0;
 }
 
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 5/7] ceph: set default timeout for MDS requests
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
                   ` (3 preceding siblings ...)
  2026-11-20  7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li,
	linux-kernel, Viacheslav Dubeyko

MDS requests created via ceph_mdsc_create_request() have r_timeout
initialized to 0 (from kmem_cache_zalloc). When r_timeout is 0,
ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, causing
ceph_mdsc_wait_request() to wait indefinitely.

This causes hung task warnings when MDS becomes unavailable during
operations like setattr or truncate:

  INFO: task dd:12345 blocked for more than 122 seconds.
  Call Trace:
    ceph_mdsc_wait_request+0x...
    ceph_mdsc_do_request+0x...
    __ceph_setattr+0x...

Only the mount path in super.c explicitly sets r_timeout to
mount_timeout. All other MDS requests (setattr, lookup, mkdir,
etc.) use the default 0 value, making them wait forever.

Fix this by initializing r_timeout to mount_timeout in
ceph_mdsc_create_request(). This ensures all MDS requests have
a reasonable timeout and will fail with -ETIMEDOUT rather than
hanging indefinitely.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
---
 fs/ceph/mds_client.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index d02fe1e34527b..4460f614050e4 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2852,6 +2852,7 @@ ceph_mdsc_create_request(struct ceph_mds_client *mdsc, int op, int mode)
 	mutex_init(&req->r_fill_mutex);
 	req->r_mdsc = mdsc;
 	req->r_started = jiffies;
+	req->r_timeout = mdsc->fsc->client->options->mount_timeout;
 	req->r_start_latency = ktime_get();
 	req->r_resend_mds = -1;
 	INIT_LIST_HEAD(&req->r_unsafe_dir_item);
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
                   ` (4 preceding siblings ...)
  2026-11-20  7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  2026-11-20  7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

The client subscribes to incremental mdsmap updates starting at
current_epoch + 1.  If an MDS restarts at a different address while
the monitor session is also down (e.g. during a rolling upgrade in a
containerized deployment), the client can miss that update and keep
reconnecting to the stale address indefinitely.  This was observed
with the mdsmap stuck at epoch 53 while the cluster was at epoch 68,
the session HUNG and monc reporting "have mdsmap 53 want 54+".

Nothing currently reacts to this: mds_con_ops has no .fault callback,
and a session going HUNG doesn't touch the subscription either.

Add a .fault callback that asks for a full mdsmap (start epoch 0) on
every MDS_CON_FAIL_REFRESH_MDSMAP consecutive faults, resetting the
count on any session message, and do the same when a session is
marked HUNG, which covers a connection that went to STANDBY and thus
never faults.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 48 ++++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/mds_client.h |  2 ++
 2 files changed, 50 insertions(+)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 4460f614050e4..3c7ecf3147e6f 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -72,6 +72,13 @@ static void ceph_mdsc_reset_workfn(struct work_struct *work);
 
 static const struct ceph_connection_operations mds_con_ops;
 
+/*
+ * Consecutive MDS connection faults after which we ask the monitor for
+ * a full mdsmap.  With the messenger backoff (250ms doubling up to 15s)
+ * ten faults take about a minute, comparable to mount_timeout.
+ */
+#define MDS_CON_FAIL_REFRESH_MDSMAP	10
+
 static void ceph_metric_bind_session(struct ceph_mds_client *mdsc,
 				     struct ceph_mds_session *session)
 {
@@ -4651,6 +4658,9 @@ static void handle_session(struct ceph_mds_session *session,
 	      ceph_session_op_name(op), session,
 	      ceph_session_state_name(session->s_state), seq);
 
+	/* the connection works, see mds_fault() */
+	session->s_con_failures = 0;
+
 	if (session->s_state == CEPH_MDS_SESSION_HUNG) {
 		session->s_state = CEPH_MDS_SESSION_OPEN;
 		pr_info_client(cl, "mds%d came back\n", session->s_mds);
@@ -6285,6 +6295,18 @@ static void maybe_recover_session(struct ceph_mds_client *mdsc)
 	ceph_force_reconnect(fsc->sb);
 }
 
+/*
+ * Subscribe to the mdsmap from epoch 0 so that the monitor sends the
+ * current full map rather than waiting for an incremental update.
+ */
+static void request_full_mdsmap(struct ceph_mds_client *mdsc)
+{
+	struct ceph_mon_client *monc = &mdsc->fsc->client->monc;
+
+	if (ceph_monc_want_map(monc, CEPH_SUB_MDSMAP, 0, true))
+		ceph_monc_renew_subs(monc);
+}
+
 bool check_session_state(struct ceph_mds_session *s)
 {
 	struct ceph_client *cl = s->s_mdsc->fsc->client;
@@ -6294,6 +6316,12 @@ bool check_session_state(struct ceph_mds_session *s)
 		if (s->s_ttl && time_after(jiffies, s->s_ttl)) {
 			s->s_state = CEPH_MDS_SESSION_HUNG;
 			pr_info_client(cl, "mds%d hung\n", s->s_mds);
+			/*
+			 * The MDS may have come back at a different
+			 * address while the connection sat in STANDBY,
+			 * which doesn't fault.
+			 */
+			request_full_mdsmap(s->s_mdsc);
 		}
 		break;
 	case CEPH_MDS_SESSION_CLOSING:
@@ -7525,12 +7553,32 @@ static int mds_check_message_signature(struct ceph_msg *msg)
        return ceph_auth_check_message_signature(auth, msg);
 }
 
+/*
+ * If the MDS keeps failing to connect, our mdsmap may be stale (e.g. the
+ * MDS restarted at a different address and the incremental update was
+ * missed while the monitor session was down).  Periodically ask for a
+ * full map.
+ */
+static void mds_fault(struct ceph_connection *con)
+{
+	struct ceph_mds_session *s = con->private;
+
+	if (++s->s_con_failures < MDS_CON_FAIL_REFRESH_MDSMAP)
+		return;
+
+	s->s_con_failures = 0;
+	doutc(s->s_mdsc->fsc->client,
+	      "mds%d keeps faulting, requesting full mdsmap\n", s->s_mds);
+	request_full_mdsmap(s->s_mdsc);
+}
+
 static const struct ceph_connection_operations mds_con_ops = {
 	.get = mds_get_con,
 	.put = mds_put_con,
 	.alloc_msg = mds_alloc_msg,
 	.dispatch = mds_dispatch,
 	.peer_reset = mds_peer_reset,
+	.fault = mds_fault,
 	.get_authorizer = mds_get_authorizer,
 	.add_authorizer_challenge = mds_add_authorizer_challenge,
 	.verify_authorizer_reply = mds_verify_authorizer_reply,
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 31abbb3b3bdb3..46e13446afcc7 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -301,6 +301,8 @@ struct ceph_mds_session {
 	struct list_head  s_unsafe;   /* unsafe requests */
 	struct xarray	  s_delegated_inos;
 	atomic_t	  s_num_deleg_inos;
+
+	u8		  s_con_failures; /* consecutive connection faults */
 };
 
 /*
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL
  2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
                   ` (5 preceding siblings ...)
  2026-11-20  7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
@ 2026-11-20  7:34 ` Ionut Nechita
  6 siblings, 0 replies; 8+ messages in thread
From: Ionut Nechita @ 2026-11-20  7:34 UTC (permalink / raw)
  To: ceph-devel
  Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel

While hunting, every reopen_session() doubles hunt_mult up to
CEPH_MONC_HUNT_MAX_MULT, so the client ends up spending 30s on each
monitor before moving to the next one.  The backoff is there to avoid
hammering monitors, but when connect() fails locally with
EADDRNOTAVAIL nothing reaches the monitor and another monitor may
well be routable.

Reset hunt_mult when the monitor connection faults with
EADDRNOTAVAIL so that hunting keeps going at the base interval.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 net/ceph/mon_client.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c
index 2f0181b2566dc..bceeaa3690145 100644
--- a/net/ceph/mon_client.c
+++ b/net/ceph/mon_client.c
@@ -1568,6 +1568,14 @@ static void mon_fault(struct ceph_connection *con)
 			__schedule_delayed(monc);
 		} else {
 			dout("%s already hunting\n", __func__);
+			/*
+			 * A local EADDRNOTAVAIL never reached the
+			 * monitor, so there is nothing to back off from.
+			 * Keep moving on to the next monitor at the
+			 * base hunt interval.
+			 */
+			if (con->addr_notavail)
+				monc->hunt_mult = 1;
 		}
 	}
 	mutex_unlock(&monc->mutex);
-- 
2.56.0


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-20  7:35 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20  7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®