* [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
When kernel_connect() fails with EADDRNOTAVAIL, the error is local:
no route or no usable source address for the peer, and nothing was
sent on the wire. con_fault() nevertheless treats it like any other
failure and doubles con->delay up to MAX_DELAY_INTERVAL (15s), so once
the local condition clears the client may sit idle for up to 15s
before trying again.
Since a failed attempt costs nothing and never reaches the peer,
retry at a fixed 100ms instead. Track the condition in struct
ceph_connection so that it doesn't depend on which protocol (msgr1 or
msgr2) is in use, and rate-limit the message.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
include/linux/ceph/messenger.h | 7 +++++++
net/ceph/messenger.c | 15 ++++++++++++---
2 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 6aa4c6478c9f6..344496967b5c5 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -321,6 +321,12 @@ struct ceph_msg {
/* ceph connection fault delay defaults, for exponential backoff */
#define BASE_DELAY_INTERVAL (HZ / 4)
#define MAX_DELAY_INTERVAL (15 * HZ)
+/*
+ * Fixed retry delay used instead of exponential backoff when connect()
+ * fails locally with EADDRNOTAVAIL. Such an attempt never reaches the
+ * peer, so backing off protects nobody and only delays recovery.
+ */
+#define ADDRNOTAVAIL_DELAY msecs_to_jiffies(100)
struct ceph_connection_v1_info {
struct kvec out_kvec[8], /* sending header/footer data */
@@ -507,6 +513,7 @@ struct ceph_connection {
struct delayed_work work; /* send|recv work */
unsigned long delay; /* current delay interval */
+ bool addr_notavail; /* last connect() failed with EADDRNOTAVAIL */
union {
struct ceph_connection_v1_info v1;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index cc7db8c61db37..3c893bc0f7d8b 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -467,8 +467,13 @@ int ceph_tcp_connect(struct ceph_connection *con)
ceph_pr_addr(&con->peer_addr),
sock->sk->sk_state);
} else if (ret < 0) {
- pr_err("connect %s error %d\n",
- ceph_pr_addr(&con->peer_addr), ret);
+ if (ret == -EADDRNOTAVAIL)
+ pr_warn_ratelimited("connect %s: address not available, will retry\n",
+ ceph_pr_addr(&con->peer_addr));
+ else
+ pr_err("connect %s error %d\n",
+ ceph_pr_addr(&con->peer_addr), ret);
+ con->addr_notavail = ret == -EADDRNOTAVAIL;
sock_release(sock);
return ret;
}
@@ -477,6 +482,7 @@ int ceph_tcp_connect(struct ceph_connection *con)
tcp_sock_set_nodelay(sock->sk);
con->sock = sock;
+ con->addr_notavail = false;
return 0;
}
@@ -610,6 +616,7 @@ void ceph_con_open(struct ceph_connection *con,
memcpy(&con->peer_addr, addr, sizeof(*addr));
con->delay = 0; /* reset backoff memory */
+ con->addr_notavail = false;
mutex_unlock(&con->mutex);
queue_con(con);
}
@@ -1644,7 +1651,9 @@ static void con_fault(struct ceph_connection *con)
} else {
/* retry after a delay. */
con->state = CEPH_CON_S_PREOPEN;
- if (!con->delay) {
+ if (con->addr_notavail) {
+ con->delay = ADDRNOTAVAIL_DELAY;
+ } else if (!con->delay) {
con->delay = BASE_DELAY_INTERVAL;
} else if (con->delay < MAX_DELAY_INTERVAL) {
con->delay *= 2;
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
When the MDS is unreachable, sync(2) and syncfs(2) block forever in
ceph_mdsc_sync(): flush_mdlog_and_wait_mdsc_unsafe_requests() waits
for r_safe_completion and wait_caps_flush() waits for the cap flushes
without any bound. The diagnostic loop added to wait_caps_flush()
makes the stall visible, but the task still can't make progress and
keeps tripping the hung task detector:
INFO: task sync:514917 blocked for more than 983 seconds.
Call Trace:
ceph_mdsc_sync+0x4d6/0x5a0 [ceph]
ceph_sync_fs+0x31/0x130 [ceph]
iterate_supers+0x97/0x100
ksys_sync+0x32/0xb0
Bound both waits by a single mount_timeout deadline, like
wait_requests() and ceph_mdsc_close_sessions() already do, and
return -ETIMEDOUT from ceph_sync_fs(). Nothing is discarded: unsafe
requests and dirty caps stay queued and are flushed once the MDS is
reachable again. mount_timeout=0 keeps the current wait-forever
behaviour.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
fs/ceph/mds_client.c | 84 +++++++++++++++++++++++++++++++++-----------
fs/ceph/mds_client.h | 2 +-
fs/ceph/super.c | 5 +--
3 files changed, 67 insertions(+), 24 deletions(-)
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 3791fc627714c..d02fe1e34527b 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2520,11 +2520,24 @@ static void dump_cap_flushes(struct ceph_mds_client *mdsc, u64 want_tid)
}
/*
- * Wait for all cap flushes through @want_flush_tid to complete.
- * Periodically dumps pending cap flush state for diagnostics.
+ * Time left until the sync deadline, or MAX_SCHEDULE_TIMEOUT if
+ * mount_timeout=0 (wait forever).
*/
-static void wait_caps_flush(struct ceph_mds_client *mdsc,
- u64 want_flush_tid)
+static unsigned long sync_time_left(struct ceph_mds_client *mdsc,
+ unsigned long deadline)
+{
+ if (!mdsc->fsc->client->options->mount_timeout)
+ return MAX_SCHEDULE_TIMEOUT;
+ return time_before(jiffies, deadline) ? deadline - jiffies : 0;
+}
+
+/*
+ * Wait for all cap flushes through @want_flush_tid to complete or for
+ * @deadline to pass. Periodically dumps pending cap flush state for
+ * diagnostics.
+ */
+static int wait_caps_flush(struct ceph_mds_client *mdsc,
+ u64 want_flush_tid, unsigned long deadline)
{
struct ceph_client *cl = mdsc->fsc->client;
int i = 0;
@@ -2532,22 +2545,31 @@ static void wait_caps_flush(struct ceph_mds_client *mdsc,
doutc(cl, "want %llu\n", want_flush_tid);
- do {
+ while (true) {
/* 60 * HZ fits in a long on all supported architectures. */
ret = wait_event_timeout(mdsc->cap_flushing_wq,
check_caps_flush(mdsc, want_flush_tid),
- CEPH_CAP_FLUSH_WAIT_TIMEOUT_SEC * HZ);
- if (ret == 0) {
- if (i < CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
- dump_cap_flushes(mdsc, want_flush_tid);
- else if (i == CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
- pr_info_client(cl,
- "still waiting for cap flushes; suppressing further dumps\n");
- i++;
+ min_t(unsigned long, sync_time_left(mdsc, deadline),
+ CEPH_CAP_FLUSH_WAIT_TIMEOUT_SEC * HZ));
+ if (ret)
+ break;
+
+ if (i < CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
+ dump_cap_flushes(mdsc, want_flush_tid);
+ else if (i == CEPH_CAP_FLUSH_MAX_DUMP_ITERS)
+ pr_info_client(cl,
+ "still waiting for cap flushes; suppressing further dumps\n");
+ i++;
+
+ if (!sync_time_left(mdsc, deadline)) {
+ doutc(cl, "timed out waiting for flush tid %llu\n",
+ want_flush_tid);
+ return -ETIMEDOUT;
}
- } while (ret == 0);
+ }
doutc(cl, "ok, flushed thru %llu\n", want_flush_tid);
+ return 0;
}
/*
@@ -6737,13 +6759,15 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)
/*
* flush the mdlog and wait for all write mds requests to flush.
*/
-static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
- u64 want_tid)
+static int flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
+ u64 want_tid,
+ unsigned long deadline)
{
struct ceph_client *cl = mdsc->fsc->client;
struct ceph_mds_request *req = NULL, *nextreq;
struct ceph_mds_session *last_session = NULL;
struct rb_node *n;
+ unsigned long left;
mutex_lock(&mdsc->mutex);
doutc(cl, "want %lld\n", want_tid);
@@ -6782,7 +6806,18 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
}
doutc(cl, "wait on %llu (want %llu)\n",
req->r_tid, want_tid);
- wait_for_completion(&req->r_safe_completion);
+ left = sync_time_left(mdsc, deadline);
+ left = wait_for_completion_timeout(&req->r_safe_completion,
+ left);
+ if (!left) {
+ doutc(cl, "timed out waiting for tid %llu\n",
+ req->r_tid);
+ ceph_mdsc_put_request(req);
+ if (nextreq)
+ ceph_mdsc_put_request(nextreq);
+ ceph_put_mds_session(last_session);
+ return -ETIMEDOUT;
+ }
mutex_lock(&mdsc->mutex);
ceph_mdsc_put_request(req);
@@ -6800,15 +6835,18 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
mutex_unlock(&mdsc->mutex);
ceph_put_mds_session(last_session);
doutc(cl, "done\n");
+ return 0;
}
-void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
+int ceph_mdsc_sync(struct ceph_mds_client *mdsc)
{
struct ceph_client *cl = mdsc->fsc->client;
+ unsigned long deadline = jiffies + cl->options->mount_timeout;
u64 want_tid, want_flush;
+ int ret;
if (READ_ONCE(mdsc->fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN)
- return;
+ return 0;
doutc(cl, "sync\n");
mutex_lock(&mdsc->mutex);
@@ -6829,8 +6867,12 @@ void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
doutc(cl, "sync want tid %lld flush_seq %lld\n", want_tid, want_flush);
- flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
- wait_caps_flush(mdsc, want_flush);
+ ret = flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid,
+ deadline);
+ if (ret)
+ return ret;
+
+ return wait_caps_flush(mdsc, want_flush, deadline);
}
/*
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 4749438d94e1f..31abbb3b3bdb3 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -635,7 +635,7 @@ extern void ceph_mdsc_close_sessions(struct ceph_mds_client *mdsc);
extern void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc);
extern void ceph_mdsc_destroy(struct ceph_fs_client *fsc);
-extern void ceph_mdsc_sync(struct ceph_mds_client *mdsc);
+extern int ceph_mdsc_sync(struct ceph_mds_client *mdsc);
extern void ceph_invalidate_dir_request(struct ceph_mds_request *req);
extern int ceph_alloc_readdir_reply_buffer(struct ceph_mds_request *req,
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index 7b5ef76e430d7..7936b76aa3cc0 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -126,6 +126,7 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
{
struct ceph_fs_client *fsc = ceph_sb_to_fs_client(sb);
struct ceph_client *cl = fsc->client;
+ int ret;
if (!wait) {
doutc(cl, "(non-blocking)\n");
@@ -137,9 +138,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
doutc(cl, "(blocking)\n");
ceph_osdc_sync(&fsc->client->osdc);
- ceph_mdsc_sync(fsc->mdsc);
+ ret = ceph_mdsc_sync(fsc->mdsc);
doutc(cl, "(blocking) done\n");
- return 0;
+ return ret;
}
/*
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() wait by mount_timeout
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
ceph_sync_fs() calls ceph_osdc_sync() before ceph_mdsc_sync(), and
ceph_osdc_sync() waits for every in-flight write with a plain
wait_for_completion(). With an OSD unreachable, sync(2) therefore
hangs there and never gets to the bounded MDS wait:
ceph_osdc_sync [libceph]
ceph_sync_fs [ceph]
iterate_supers
ksys_sync
Bound the wait by mount_timeout in the same way and propagate
-ETIMEDOUT. The requests are not cancelled and complete once the OSD
is reachable again. mount_timeout=0 keeps waiting forever.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
fs/ceph/super.c | 4 +++-
include/linux/ceph/osd_client.h | 2 +-
net/ceph/osd_client.c | 24 ++++++++++++++++++++++--
3 files changed, 26 insertions(+), 4 deletions(-)
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index 7936b76aa3cc0..166f6eb21f0af 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -137,7 +137,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
}
doutc(cl, "(blocking)\n");
- ceph_osdc_sync(&fsc->client->osdc);
+ ret = ceph_osdc_sync(&fsc->client->osdc);
+ if (ret)
+ return ret;
ret = ceph_mdsc_sync(fsc->mdsc);
doutc(cl, "(blocking) done\n");
return ret;
diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
index 7349ca09a227a..674dd9221d657 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -579,7 +579,7 @@ void ceph_osdc_start_request(struct ceph_osd_client *osdc,
extern void ceph_osdc_cancel_request(struct ceph_osd_request *req);
extern int ceph_osdc_wait_request(struct ceph_osd_client *osdc,
struct ceph_osd_request *req);
-extern void ceph_osdc_sync(struct ceph_osd_client *osdc);
+extern int ceph_osdc_sync(struct ceph_osd_client *osdc);
extern void ceph_osdc_flush_notifies(struct ceph_osd_client *osdc);
void ceph_osdc_maybe_request_map(struct ceph_osd_client *osdc);
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 4479ec7a932b3..417374c95dc14 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -4715,11 +4715,17 @@ EXPORT_SYMBOL(ceph_osdc_wait_request);
/*
* sync - wait for all in-flight requests to flush. avoid starvation.
+ *
+ * Gives up with -ETIMEDOUT after mount_timeout (0 means wait forever),
+ * leaving the requests in flight.
*/
-void ceph_osdc_sync(struct ceph_osd_client *osdc)
+int ceph_osdc_sync(struct ceph_osd_client *osdc)
{
+ unsigned long timeout = osdc->client->options->mount_timeout;
+ unsigned long deadline = jiffies + timeout;
struct rb_node *n, *p;
u64 last_tid = atomic64_read(&osdc->last_tid);
+ unsigned long left;
again:
down_read(&osdc->lock);
@@ -4742,7 +4748,20 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
up_read(&osdc->lock);
dout("%s waiting on req %p tid %llu last_tid %llu\n",
__func__, req, req->r_tid, last_tid);
- wait_for_completion(&req->r_completion);
+ if (!timeout)
+ left = MAX_SCHEDULE_TIMEOUT;
+ else if (time_before(jiffies, deadline))
+ left = deadline - jiffies;
+ else
+ left = 0;
+ left = wait_for_completion_timeout(&req->r_completion,
+ left);
+ if (!left) {
+ dout("%s timed out on req %p tid %llu\n",
+ __func__, req, req->r_tid);
+ ceph_osdc_put_request(req);
+ return -ETIMEDOUT;
+ }
ceph_osdc_put_request(req);
goto again;
}
@@ -4752,6 +4771,7 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
up_read(&osdc->lock);
dout("%s done last_tid %llu\n", __func__, last_tid);
+ return 0;
}
EXPORT_SYMBOL(ceph_osdc_sync);
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion()
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
` (2 preceding siblings ...)
2026-11-20 7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
When a blocking lock request is interrupted,
ceph_lock_wait_for_completion() sends an unlock request to cancel it
and then waits for the original request's r_safe_completion with no
bound. If the MDS goes away at
that point the task hangs in D state:
INFO: task flock:12345 blocked for more than 122 seconds.
Call Trace:
wait_for_completion_killable
ceph_lock_wait_for_completion
ceph_flock
Bound the wait by mount_timeout and return -ETIMEDOUT; the original
request has already been marked CEPH_MDS_R_ABORTED at this point.
mount_timeout=0 keeps waiting forever.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
fs/ceph/locks.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index 677221bd64e01..b4faf112b3f84 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -158,9 +158,11 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
struct ceph_mds_request *req)
{
struct ceph_client *cl = mdsc->fsc->client;
+ unsigned long timeout = ceph_timeout_jiffies(cl->options->mount_timeout);
struct ceph_mds_request *intr_req;
struct inode *inode = req->r_inode;
int err, lock_type;
+ long left;
BUG_ON(req->r_op != CEPH_MDS_OP_SETFILELOCK);
if (req->r_args.filelock_change.rule == CEPH_LOCK_FCNTL)
@@ -219,10 +221,15 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
if (err && err != -ERESTARTSYS)
return err;
- err = wait_for_completion_killable(&req->r_safe_completion);
- if (err)
- return err;
-
+ left = wait_for_completion_killable_timeout(&req->r_safe_completion,
+ timeout);
+ if (left < 0) {
+ return left;
+ } else if (left == 0) {
+ doutc(cl, "request %llu safe completion timed out\n",
+ req->r_tid);
+ return -ETIMEDOUT;
+ }
return 0;
}
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 5/7] ceph: set default timeout for MDS requests
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
` (3 preceding siblings ...)
2026-11-20 7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li,
linux-kernel, Viacheslav Dubeyko
MDS requests created via ceph_mdsc_create_request() have r_timeout
initialized to 0 (from kmem_cache_zalloc). When r_timeout is 0,
ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, causing
ceph_mdsc_wait_request() to wait indefinitely.
This causes hung task warnings when MDS becomes unavailable during
operations like setattr or truncate:
INFO: task dd:12345 blocked for more than 122 seconds.
Call Trace:
ceph_mdsc_wait_request+0x...
ceph_mdsc_do_request+0x...
__ceph_setattr+0x...
Only the mount path in super.c explicitly sets r_timeout to
mount_timeout. All other MDS requests (setattr, lookup, mkdir,
etc.) use the default 0 value, making them wait forever.
Fix this by initializing r_timeout to mount_timeout in
ceph_mdsc_create_request(). This ensures all MDS requests have
a reasonable timeout and will fail with -ETIMEDOUT rather than
hanging indefinitely.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
---
fs/ceph/mds_client.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index d02fe1e34527b..4460f614050e4 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2852,6 +2852,7 @@ ceph_mdsc_create_request(struct ceph_mds_client *mdsc, int op, int mode)
mutex_init(&req->r_fill_mutex);
req->r_mdsc = mdsc;
req->r_started = jiffies;
+ req->r_timeout = mdsc->fsc->client->options->mount_timeout;
req->r_start_latency = ktime_get();
req->r_resend_mds = -1;
INIT_LIST_HEAD(&req->r_unsafe_dir_item);
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
` (4 preceding siblings ...)
2026-11-20 7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita
2026-11-22 20:27 ` [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Alex Markuze
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
The client subscribes to incremental mdsmap updates starting at
current_epoch + 1. If an MDS restarts at a different address while
the monitor session is also down (e.g. during a rolling upgrade in a
containerized deployment), the client can miss that update and keep
reconnecting to the stale address indefinitely. This was observed
with the mdsmap stuck at epoch 53 while the cluster was at epoch 68,
the session HUNG and monc reporting "have mdsmap 53 want 54+".
Nothing currently reacts to this: mds_con_ops has no .fault callback,
and a session going HUNG doesn't touch the subscription either.
Add a .fault callback that asks for a full mdsmap (start epoch 0) on
every MDS_CON_FAIL_REFRESH_MDSMAP consecutive faults, resetting the
count on any session message, and do the same when a session is
marked HUNG, which covers a connection that went to STANDBY and thus
never faults.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
fs/ceph/mds_client.c | 48 ++++++++++++++++++++++++++++++++++++++++++++
fs/ceph/mds_client.h | 2 ++
2 files changed, 50 insertions(+)
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 4460f614050e4..3c7ecf3147e6f 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -72,6 +72,13 @@ static void ceph_mdsc_reset_workfn(struct work_struct *work);
static const struct ceph_connection_operations mds_con_ops;
+/*
+ * Consecutive MDS connection faults after which we ask the monitor for
+ * a full mdsmap. With the messenger backoff (250ms doubling up to 15s)
+ * ten faults take about a minute, comparable to mount_timeout.
+ */
+#define MDS_CON_FAIL_REFRESH_MDSMAP 10
+
static void ceph_metric_bind_session(struct ceph_mds_client *mdsc,
struct ceph_mds_session *session)
{
@@ -4651,6 +4658,9 @@ static void handle_session(struct ceph_mds_session *session,
ceph_session_op_name(op), session,
ceph_session_state_name(session->s_state), seq);
+ /* the connection works, see mds_fault() */
+ session->s_con_failures = 0;
+
if (session->s_state == CEPH_MDS_SESSION_HUNG) {
session->s_state = CEPH_MDS_SESSION_OPEN;
pr_info_client(cl, "mds%d came back\n", session->s_mds);
@@ -6285,6 +6295,18 @@ static void maybe_recover_session(struct ceph_mds_client *mdsc)
ceph_force_reconnect(fsc->sb);
}
+/*
+ * Subscribe to the mdsmap from epoch 0 so that the monitor sends the
+ * current full map rather than waiting for an incremental update.
+ */
+static void request_full_mdsmap(struct ceph_mds_client *mdsc)
+{
+ struct ceph_mon_client *monc = &mdsc->fsc->client->monc;
+
+ if (ceph_monc_want_map(monc, CEPH_SUB_MDSMAP, 0, true))
+ ceph_monc_renew_subs(monc);
+}
+
bool check_session_state(struct ceph_mds_session *s)
{
struct ceph_client *cl = s->s_mdsc->fsc->client;
@@ -6294,6 +6316,12 @@ bool check_session_state(struct ceph_mds_session *s)
if (s->s_ttl && time_after(jiffies, s->s_ttl)) {
s->s_state = CEPH_MDS_SESSION_HUNG;
pr_info_client(cl, "mds%d hung\n", s->s_mds);
+ /*
+ * The MDS may have come back at a different
+ * address while the connection sat in STANDBY,
+ * which doesn't fault.
+ */
+ request_full_mdsmap(s->s_mdsc);
}
break;
case CEPH_MDS_SESSION_CLOSING:
@@ -7525,12 +7553,32 @@ static int mds_check_message_signature(struct ceph_msg *msg)
return ceph_auth_check_message_signature(auth, msg);
}
+/*
+ * If the MDS keeps failing to connect, our mdsmap may be stale (e.g. the
+ * MDS restarted at a different address and the incremental update was
+ * missed while the monitor session was down). Periodically ask for a
+ * full map.
+ */
+static void mds_fault(struct ceph_connection *con)
+{
+ struct ceph_mds_session *s = con->private;
+
+ if (++s->s_con_failures < MDS_CON_FAIL_REFRESH_MDSMAP)
+ return;
+
+ s->s_con_failures = 0;
+ doutc(s->s_mdsc->fsc->client,
+ "mds%d keeps faulting, requesting full mdsmap\n", s->s_mds);
+ request_full_mdsmap(s->s_mdsc);
+}
+
static const struct ceph_connection_operations mds_con_ops = {
.get = mds_get_con,
.put = mds_put_con,
.alloc_msg = mds_alloc_msg,
.dispatch = mds_dispatch,
.peer_reset = mds_peer_reset,
+ .fault = mds_fault,
.get_authorizer = mds_get_authorizer,
.add_authorizer_challenge = mds_add_authorizer_challenge,
.verify_authorizer_reply = mds_verify_authorizer_reply,
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 31abbb3b3bdb3..46e13446afcc7 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -301,6 +301,8 @@ struct ceph_mds_session {
struct list_head s_unsafe; /* unsafe requests */
struct xarray s_delegated_inos;
atomic_t s_num_deleg_inos;
+
+ u8 s_con_failures; /* consecutive connection faults */
};
/*
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
` (5 preceding siblings ...)
2026-11-20 7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
@ 2026-11-20 7:34 ` Ionut Nechita
2026-11-22 20:27 ` [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Alex Markuze
7 siblings, 0 replies; 9+ messages in thread
From: Ionut Nechita @ 2026-11-20 7:34 UTC (permalink / raw)
To: ceph-devel
Cc: Ilya Dryomov, Alex Markuze, Viacheslav Dubeyko, Xiubo Li, linux-kernel
While hunting, every reopen_session() doubles hunt_mult up to
CEPH_MONC_HUNT_MAX_MULT, so the client ends up spending 30s on each
monitor before moving to the next one. The backoff is there to avoid
hammering monitors, but when connect() fails locally with
EADDRNOTAVAIL nothing reaches the monitor and another monitor may
well be routable.
Reset hunt_mult when the monitor connection faults with
EADDRNOTAVAIL so that hunting keeps going at the base interval.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
net/ceph/mon_client.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c
index 2f0181b2566dc..bceeaa3690145 100644
--- a/net/ceph/mon_client.c
+++ b/net/ceph/mon_client.c
@@ -1568,6 +1568,14 @@ static void mon_fault(struct ceph_connection *con)
__schedule_delayed(monc);
} else {
dout("%s already hunting\n", __func__);
+ /*
+ * A local EADDRNOTAVAIL never reached the
+ * monitor, so there is nothing to back off from.
+ * Keep moving on to the next monitor at the
+ * base hunt interval.
+ */
+ if (con->addr_notavail)
+ monc->hunt_mult = 1;
}
}
mutex_unlock(&monc->mutex);
--
2.56.0
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption
2026-11-20 7:34 [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption Ionut Nechita
` (6 preceding siblings ...)
2026-11-20 7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita
@ 2026-11-22 20:27 ` Alex Markuze
7 siblings, 0 replies; 9+ messages in thread
From: Alex Markuze @ 2026-11-22 20:27 UTC (permalink / raw)
To: Ionut Nechita
Cc: Ilya Dryomov, Viacheslav Dubeyko, Viacheslav Dubeyko, Xiubo Li,
Sebastian Andrzej Siewior, ceph-devel, linux-kernel
Hi Ionut,
The timeouts in 2/7 and 3/7 can make ceph_sync_fs() return while
requests are still outstanding. syncfs() can report the error, but
global sync() ignores the sync_fs return value. It can therefore return
normally while an unsafe metadata request is still waiting for its safe
reply. Both paths call sync_fs with wait=1, so returning an error from
that callback is not enough for global sync(). Please preserve global
sync's completion guarantee when adding these timeouts.
The default timeout in 5/7 also applies to the interrupt request for a
blocking lock. On the v2 base, if the MDS has already granted the lock,
the interrupt does not remove that grant. If the grant reply is delayed
and the interrupt request times out, the syscall can return an error
without installing the granted lock locally. The MDS can still hold the
lock and block other clients. The underlying grant-versus-interrupt race
predates this series; the default timeout adds another way to return
before the replies arrive. Please exclude interrupt requests from the
new default timeout.
Thanks,
Alex
^ permalink raw reply [flat|nested] 9+ messages in thread