From: Ionut Nechita <ionut.nechita@windriver.com>
To: ceph-devel@vger.kernel.org
Cc: Ilya Dryomov <idryomov@gmail.com>,
Alex Markuze <amarkuze@redhat.com>,
Viacheslav Dubeyko <slava@dubeyko.com>,
Xiubo Li <xiubo.li@clyso.com>,
Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption
Date: Fri, 20 Nov 2026 09:34:16 +0200 [thread overview]
Message-ID: <20261120073431.53636-1-ionut.nechita@windriver.com> (raw)
When the Ceph daemons become unreachable for a long time (seen during
Rook-Ceph rolling upgrades with active CephFS I/O), the kernel client
leaves sync(2), syncfs(2) and flock() stuck in D state indefinitely,
keeps backing off on a purely local connect() error, and can miss the
mdsmap update that carries the new MDS address.
This v2 is rebased on next-20261119 and reworked after the v1 review
and further investigation. The patches:
1 libceph: fixed 100ms retry instead of exponential backoff when
connect() fails locally with EADDRNOTAVAIL
2-4 bound the sync(2)/syncfs(2) and lock-interrupt waits by
mount_timeout; mount_timeout=0 keeps the current wait-forever
behaviour, and nothing queued is dropped
5 default r_timeout of MDS requests to mount_timeout (unchanged
from v1)
6 request a full mdsmap when an MDS keeps faulting or a session
goes HUNG
7 don't escalate the monitor hunt backoff on EADDRNOTAVAIL
Patch 2 is built on top of the wait_caps_flush() diagnostic loop from
ebbbab66bd74 ("ceph: add diagnostic timeout loop to wait_caps_flush()")
and keeps it. I know that patch deliberately left the wait unbounded
and that b52861897be8 ("ceph: add client reset state machine and
session teardown") added a manual escape hatch. Patches 2-5 are meant
for the case where nobody is around to trigger that reset, with
mount_timeout as the admin-controlled bound already used for umount.
If you'd rather keep sync unbounded, patches 2-5 can be dropped without
affecting the rest.
Changes since v1:
- Dropped "ceph: make ceph_start_io_write() killable" and "ceph: make
remaining I/O lock functions killable": already upstream as
fa073039466f ("ceph: make ceph_start_io_*() killable").
- Dropped "ceph: add timeout to caps wait in __ceph_get_caps()": covered
by 50958bb928ba ("ceph: fix hanging __ceph_get_caps() with stale
mds_wanted").
- Dropped "libceph: force host network namespace for kernel CephFS
mounts", as discussed with Ilya. The root cause was the Rook CSI holder
pod's network namespace being torn down during the Rook upgrade.
- Dropped "libceph: reset source address on persistent EADDRNOTAVAIL".
ceph_tcp_connect() never binds to msgr->inst.addr, so blanking that
address cannot influence source address selection; the patch had
no effect on the error it was meant to fix.
- Dropped "ceph: fix race condition in cleanup_session_requests()".
Slava, sorry, I have to withdraw this one despite your Reviewed-by:
unregistering requests that have no unsafe reply yet removes them
from request_tree, so kick_requests() can no longer resend them after
an MDS failover (mds_peer_reset(), check_new_map()), and the caller
in ceph_mdsc_wait_request() is never completed. The sync hang it
targeted is now bounded by patch 2.
- Patch 1 (Slava): the flag now lives in struct ceph_connection
instead of the v1/v2 union members, so the ceph_msgr2() checks and
the local variable are gone. The delay is msecs_to_jiffies(100)
rather than HZ/10. The point is that it no longer grows, which
BASE_DELAY_INTERVAL would also give. I'm happy to switch to that if
you prefer one constant fewer. Dropped the Fixes: tag, since this is
a behaviour change rather than a regression fix.
- Patches 2-3 (Slava): ceph_mdsc_sync() returns 0 again on shutdown
instead of -EIO. The MDS waits share a single mount_timeout deadline
instead of one per request, and so do the OSD waits. Timeout
messages are now dout()s. On mount_timeout values: it is already range
checked at parse time and is the admin's bound for the existing
umount and wait_requests() waits. 0 still means forever, so I
didn't add another clamp.
- Patch 4 (Slava): dout() instead of pr_warn(), if/else-if style, and
the result of wait_for_completion_killable_timeout() is kept in a
long (v1 truncated it to int).
- Patch 6 (Slava): s_con_failures is a u8 that wraps back to 0 at the
threshold, with no local variable or modulo arithmetic. The new
messages are dout()s, the existing "mds%d hung" pr_info() is kept,
and the threshold of 10 is explained (about a minute of messenger
backoff).
- Patch 7 rewritten: v1 called reopen_session() from mon_fault() while
hunting. ceph_con_open() resets con->delay and queues immediately,
so with an instant EADDRNOTAVAIL that spun the monitor connection
with no delay at all. It now only keeps hunt_mult from growing.
- Fixed a wrong Fixes: tag (the cleanup_session_requests() patch
pointed at a non-existent commit) and dropped a reference to a local
commit id.
Answering Ilya's questions on the earlier standalone EADDRNOTAVAIL
patch: the cluster has 2 monitors and, as you said, the client talks to
one at a time. The 36 minutes were mostly the external condition
itself (the dead holder-pod namespace); patches 1 and 7 only remove
the extra wait once that condition clears. Sebastian asked whether
this is PREEMPT_RT specific. It isn't: none of these paths depend on
PREEMPT_RT.
Built with W=1 and sparse, and bisected build by build, on
next-20261119. The v1 testing was done on 6.12.y; v2 has only been
build-tested, not run against a live cluster.
I used an LLM to help rebase the series, review the code and draft
the changelogs. I have reviewed every change and take responsibility
for it.
v1: https://lore.kernel.org/all/20260312081619.40854-1-ionut.nechita@windriver.com/
Ionut Nechita (7):
libceph: don't back off exponentially on EADDRNOTAVAIL
ceph: bound the ceph_mdsc_sync() wait by mount_timeout
ceph: bound the ceph_osdc_sync() wait by mount_timeout
ceph: bound the safe-completion wait in
ceph_lock_wait_for_completion()
ceph: set default timeout for MDS requests
ceph: request a full mdsmap when an MDS stays unreachable
libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL
fs/ceph/locks.c | 15 +++-
fs/ceph/mds_client.c | 133 +++++++++++++++++++++++++++-----
fs/ceph/mds_client.h | 4 +-
fs/ceph/super.c | 9 ++-
include/linux/ceph/messenger.h | 7 ++
include/linux/ceph/osd_client.h | 2 +-
net/ceph/messenger.c | 15 +++-
net/ceph/mon_client.c | 8 ++
net/ceph/osd_client.c | 24 +++++-
9 files changed, 182 insertions(+), 35 deletions(-)
base-commit: adcad7f4e6332cff3d6d7165f41584899f598534
--
2.56.0
next reply other threads:[~2026-11-20 7:35 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 7:34 Ionut Nechita [this message]
2026-11-20 7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
2026-11-20 7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120073431.53636-1-ionut.nechita@windriver.com \
--to=ionut.nechita@windriver.com \
--cc=Slava.Dubeyko@ibm.com \
--cc=amarkuze@redhat.com \
--cc=bigeasy@linutronix.de \
--cc=ceph-devel@vger.kernel.org \
--cc=idryomov@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=slava@dubeyko.com \
--cc=xiubo.li@clyso.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®