mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ionut Nechita <ionut.nechita@windriver.com>
To: ceph-devel@vger.kernel.org
Cc: Ilya Dryomov <idryomov@gmail.com>,
	Alex Markuze <amarkuze@redhat.com>,
	Viacheslav Dubeyko <slava@dubeyko.com>,
	Xiubo Li <xiubo.li@clyso.com>,
	Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>,
	Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/7] ceph/libceph: bound sync waits and speed up recovery after network disruption
Date: Fri, 20 Nov 2026 09:34:16 +0200	[thread overview]
Message-ID: <20261120073431.53636-1-ionut.nechita@windriver.com> (raw)

When the Ceph daemons become unreachable for a long time (seen during
Rook-Ceph rolling upgrades with active CephFS I/O), the kernel client
leaves sync(2), syncfs(2) and flock() stuck in D state indefinitely,
keeps backing off on a purely local connect() error, and can miss the
mdsmap update that carries the new MDS address.

This v2 is rebased on next-20261119 and reworked after the v1 review
and further investigation.  The patches:

  1    libceph: fixed 100ms retry instead of exponential backoff when
       connect() fails locally with EADDRNOTAVAIL
  2-4  bound the sync(2)/syncfs(2) and lock-interrupt waits by
       mount_timeout; mount_timeout=0 keeps the current wait-forever
       behaviour, and nothing queued is dropped
  5    default r_timeout of MDS requests to mount_timeout (unchanged
       from v1)
  6    request a full mdsmap when an MDS keeps faulting or a session
       goes HUNG
  7    don't escalate the monitor hunt backoff on EADDRNOTAVAIL

Patch 2 is built on top of the wait_caps_flush() diagnostic loop from
ebbbab66bd74 ("ceph: add diagnostic timeout loop to wait_caps_flush()")
and keeps it.  I know that patch deliberately left the wait unbounded
and that b52861897be8 ("ceph: add client reset state machine and
session teardown") added a manual escape hatch.  Patches 2-5 are meant
for the case where nobody is around to trigger that reset, with
mount_timeout as the admin-controlled bound already used for umount.
If you'd rather keep sync unbounded, patches 2-5 can be dropped without
affecting the rest.

Changes since v1:
- Dropped "ceph: make ceph_start_io_write() killable" and "ceph: make
  remaining I/O lock functions killable": already upstream as
  fa073039466f ("ceph: make ceph_start_io_*() killable").
- Dropped "ceph: add timeout to caps wait in __ceph_get_caps()": covered
  by 50958bb928ba ("ceph: fix hanging __ceph_get_caps() with stale
  mds_wanted").
- Dropped "libceph: force host network namespace for kernel CephFS
  mounts", as discussed with Ilya.  The root cause was the Rook CSI holder
  pod's network namespace being torn down during the Rook upgrade.
- Dropped "libceph: reset source address on persistent EADDRNOTAVAIL".
  ceph_tcp_connect() never binds to msgr->inst.addr, so blanking that
  address cannot influence source address selection; the patch had
  no effect on the error it was meant to fix.
- Dropped "ceph: fix race condition in cleanup_session_requests()".
  Slava, sorry, I have to withdraw this one despite your Reviewed-by:
  unregistering requests that have no unsafe reply yet removes them
  from request_tree, so kick_requests() can no longer resend them after
  an MDS failover (mds_peer_reset(), check_new_map()), and the caller
  in ceph_mdsc_wait_request() is never completed.  The sync hang it
  targeted is now bounded by patch 2.
- Patch 1 (Slava): the flag now lives in struct ceph_connection
  instead of the v1/v2 union members, so the ceph_msgr2() checks and
  the local variable are gone.  The delay is msecs_to_jiffies(100)
  rather than HZ/10.  The point is that it no longer grows, which
  BASE_DELAY_INTERVAL would also give.  I'm happy to switch to that if
  you prefer one constant fewer.  Dropped the Fixes: tag, since this is
  a behaviour change rather than a regression fix.
- Patches 2-3 (Slava): ceph_mdsc_sync() returns 0 again on shutdown
  instead of -EIO.  The MDS waits share a single mount_timeout deadline
  instead of one per request, and so do the OSD waits.  Timeout
  messages are now dout()s.  On mount_timeout values: it is already range
  checked at parse time and is the admin's bound for the existing
  umount and wait_requests() waits.  0 still means forever, so I
  didn't add another clamp.
- Patch 4 (Slava): dout() instead of pr_warn(), if/else-if style, and
  the result of wait_for_completion_killable_timeout() is kept in a
  long (v1 truncated it to int).
- Patch 6 (Slava): s_con_failures is a u8 that wraps back to 0 at the
  threshold, with no local variable or modulo arithmetic.  The new
  messages are dout()s, the existing "mds%d hung" pr_info() is kept,
  and the threshold of 10 is explained (about a minute of messenger
  backoff).
- Patch 7 rewritten: v1 called reopen_session() from mon_fault() while
  hunting.  ceph_con_open() resets con->delay and queues immediately,
  so with an instant EADDRNOTAVAIL that spun the monitor connection
  with no delay at all.  It now only keeps hunt_mult from growing.
- Fixed a wrong Fixes: tag (the cleanup_session_requests() patch
  pointed at a non-existent commit) and dropped a reference to a local
  commit id.

Answering Ilya's questions on the earlier standalone EADDRNOTAVAIL
patch: the cluster has 2 monitors and, as you said, the client talks to
one at a time.  The 36 minutes were mostly the external condition
itself (the dead holder-pod namespace); patches 1 and 7 only remove
the extra wait once that condition clears.  Sebastian asked whether
this is PREEMPT_RT specific.  It isn't: none of these paths depend on
PREEMPT_RT.

Built with W=1 and sparse, and bisected build by build, on
next-20261119.  The v1 testing was done on 6.12.y; v2 has only been
build-tested, not run against a live cluster.

I used an LLM to help rebase the series, review the code and draft
the changelogs.  I have reviewed every change and take responsibility
for it.

v1: https://lore.kernel.org/all/20260312081619.40854-1-ionut.nechita@windriver.com/

Ionut Nechita (7):
  libceph: don't back off exponentially on EADDRNOTAVAIL
  ceph: bound the ceph_mdsc_sync() wait by mount_timeout
  ceph: bound the ceph_osdc_sync() wait by mount_timeout
  ceph: bound the safe-completion wait in
    ceph_lock_wait_for_completion()
  ceph: set default timeout for MDS requests
  ceph: request a full mdsmap when an MDS stays unreachable
  libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL

 fs/ceph/locks.c                 |  15 +++-
 fs/ceph/mds_client.c            | 133 +++++++++++++++++++++++++++-----
 fs/ceph/mds_client.h            |   4 +-
 fs/ceph/super.c                 |   9 ++-
 include/linux/ceph/messenger.h  |   7 ++
 include/linux/ceph/osd_client.h |   2 +-
 net/ceph/messenger.c            |  15 +++-
 net/ceph/mon_client.c           |   8 ++
 net/ceph/osd_client.c           |  24 +++++-
 9 files changed, 182 insertions(+), 35 deletions(-)


base-commit: adcad7f4e6332cff3d6d7165f41584899f598534
-- 
2.56.0


             reply	other threads:[~2026-11-20  7:35 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  7:34 Ionut Nechita [this message]
2026-11-20  7:34 ` [PATCH v2 1/7] libceph: don't back off exponentially on EADDRNOTAVAIL Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 2/7] ceph: bound the ceph_mdsc_sync() wait by mount_timeout Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 3/7] ceph: bound the ceph_osdc_sync() " Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 4/7] ceph: bound the safe-completion wait in ceph_lock_wait_for_completion() Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 5/7] ceph: set default timeout for MDS requests Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 6/7] ceph: request a full mdsmap when an MDS stays unreachable Ionut Nechita
2026-11-20  7:34 ` [PATCH v2 7/7] libceph: don't escalate monitor hunt backoff on EADDRNOTAVAIL Ionut Nechita

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120073431.53636-1-ionut.nechita@windriver.com \
    --to=ionut.nechita@windriver.com \
    --cc=Slava.Dubeyko@ibm.com \
    --cc=amarkuze@redhat.com \
    --cc=bigeasy@linutronix.de \
    --cc=ceph-devel@vger.kernel.org \
    --cc=idryomov@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=slava@dubeyko.com \
    --cc=xiubo.li@clyso.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®