[PATCH v1 00/13] ceph/libceph: fix hung tasks and connection recovery during network disruptions

[PATCH v1 00/13] ceph/libceph: fix hung tasks and connection recovery during network disruptions

[Ionut Nechita (Wind River)]

During Rook-Ceph rolling upgrades (e.g., Ceph 18.2.2 -> 18.2.5) with
active CephFS workloads (ReadWriteMany PVCs with continuous I/O), the
kernel CephFS client encounters multiple cascading failures that leave
the filesystem completely unresponsive:

1. Persistent EADDRNOTAVAIL (-99) on all connections

   When monitor/MDS/OSD pods restart during the upgrade, they receive
   new IP addresses from the CNI plugin. The kernel client's cached
   source address (learned via process_hello/process_banner from the
   initial connection) may become invalid -- e.g., a Calico-assigned
   pod address that was removed, with a blackhole route installed for
   the old address range. All subsequent kernel_connect() calls fail
   with EADDRNOTAVAIL at ip6_dst_lookup_flow() before even sending a
   TCP SYN.

   The existing exponential backoff (250ms -> 15s) compounds with
   the monitor hunt backoff (3s * hunt_mult, up to 30s), making
   recovery take 30+ minutes even after the network issue resolves.

   Observed: ~470 failed connect attempts over ~36 minutes, with the
   client sitting idle for up to 15s between attempts.

2. Indefinite hangs in sync() path

   ceph_mdsc_sync() and ceph_osdc_sync() use wait_event() and
   wait_for_completion() with no timeout. When MDS/OSD connections
   are down, sync tasks block indefinitely in D state, triggering
   hung_task warnings that escalate: 122s, 245s, 368s, ... 983s+.

   Stack traces show:
     ceph_mdsc_sync -> wait_caps_flush (indefinite wait_event)
     ceph_mdsc_sync -> flush_mdlog_and_wait_mdsc_unsafe_requests
                       (indefinite wait_for_completion)
     ceph_osdc_sync -> wait_for_completion (indefinite)

3. Stale mdsmap causing permanent MDS reconnection failure

   The kernel client caches the mdsmap and subscribes for incremental
   updates (start=current_epoch+1). During the upgrade, if the monitor
   subscription was lost (also affected by EADDRNOTAVAIL), the client
   never receives updated maps. The mdsmap was observed stuck at
   epoch 53 while the cluster had progressed to epoch 90. The client
   retried connections to the old MDS address indefinitely.

   Two scenarios lead to this:
   a) Active connection failures: mds_con_ops had no .fault callback,
      so the MDS client was never notified
   b) Silent connection death: messenger enters STANDBY, session
      transitions to HUNG via TTL, but no mdsmap refresh is triggered

4. I/O operations hung in unkillable D state

   ceph_start_io_write() and related I/O lock functions use
   inode_dio_wait() and wait_on_inode_writeback() which are
   TASK_UNINTERRUPTIBLE. During MDS failover, these block indefinitely
   and cannot be killed, accumulating D-state processes.

   Test results (20 iterations of MDS kill during active I/O):
   12 passed, 8 failed with hung tasks in ceph_start_io_write,
   __ceph_get_caps, ceph_fsync.

5. Wrong network namespace captured at mount time

   ceph_messenger_init() captures current->nsproxy->net_ns. In CSI
   environments, mount() may be invoked from a pod namespace that
   lacks routes to Ceph monitors, causing permanent EADDRNOTAVAIL.

This series addresses all five issues:

Connection layer (patches 1, 11-12):
  - Bypass exponential backoff for EADDRNOTAVAIL, use fixed 100ms retry
  - After 30 consecutive EADDRNOTAVAIL failures, reset the cached source
    address to blank so process_hello() re-learns it from the monitor
  - Force immediate monitor reconnect during persistent EADDRNOTAVAIL,
    reset hunt_mult to prevent accumulated backoff

Sync path timeouts (patches 2-3, 5-7):
  - Add mount_timeout-based timeouts to all indefinite waits in
    the sync path: wait_caps_flush(), ceph_osdc_sync(),
    flush_mdlog_and_wait_mdsc_unsafe_requests(),
    ceph_lock_wait_for_completion(), __ceph_get_caps()
  - Set a default timeout for MDS requests
  - On timeout, pending operations are NOT discarded -- they remain in
    memory and complete when connectivity is restored

Race condition fix (patch 4):
  - Fix race in cleanup_session_requests() where the request list can
    be modified concurrently during MDS reconnection

I/O killability (patches 8-9):
  - Make ceph_start_io_write() and related I/O lock functions killable
    (TASK_KILLABLE instead of TASK_UNINTERRUPTIBLE)

MDS map refresh (patch 10):
  - Add .fault callback to mds_con_ops to detect persistent MDS
    connection failures
  - Force fresh mdsmap subscription (start=0) after 10 consecutive
    failures or when a session becomes HUNG
  - Reset failure counter on successful session message

Network namespace (patch 13):
  - Always use init_net in ceph_messenger_init() instead of the
    caller's namespace
  - This is the final piece that ensures mon, mds, and osd
    connections all use the host network after the upgrade,
    allowing the client to successfully reconnect to all Ceph
    daemons regardless of which namespace triggered the mount

Tested on kernel 6.12.x with Rook-Ceph (Ceph Reef 18.2.5), IPv6-only
cluster, during rolling upgrades with active CephFS workloads. The
patches resolve all five failure modes described above.

Ionut Nechita (13):
  libceph: handle EADDRNOTAVAIL more gracefully
  ceph: add timeout protection to ceph_mdsc_sync() path
  ceph: add timeout protection to ceph_osdc_sync() path
  ceph: fix race condition in cleanup_session_requests()
  ceph: add timeout protection to ceph_lock_wait_for_completion()
  ceph: set default timeout for MDS requests
  ceph: add timeout to caps wait in __ceph_get_caps()
  ceph: make ceph_start_io_write() killable
  ceph: make remaining I/O lock functions killable
  ceph: force mdsmap refresh on persistent MDS connection failures
  libceph: reset source address on persistent EADDRNOTAVAIL
  libceph: force monitor reconnect on persistent EADDRNOTAVAIL
  libceph: force host network namespace for kernel CephFS mounts

 fs/ceph/caps.c                  |  16 +++-
 fs/ceph/file.c                  |  34 ++++++--
 fs/ceph/io.c                    |  37 +++++---
 fs/ceph/io.h                    |   6 +-
 fs/ceph/locks.c                 |  14 ++-
 fs/ceph/mds_client.c            | 148 +++++++++++++++++++++++++++++---
 fs/ceph/mds_client.h            |   4 +-
 fs/ceph/super.c                 |   9 +-
 include/linux/ceph/messenger.h  |  31 +++++++
 include/linux/ceph/osd_client.h |   2 +-
 net/ceph/messenger.c            | 133 +++++++++++++++++++++++++++-
 net/ceph/messenger_v1.c         |   7 ++
 net/ceph/messenger_v2.c         |  12 +++
 net/ceph/mon_client.c           |  39 ++++++++-
 net/ceph/osd_client.c           |  15 +++-
 15 files changed, 457 insertions(+), 50 deletions(-)

base-commit: 8a243ecde1f6447b8e237f2c1c67c0bb67d16d67
--
2.53.0

[PATCH v1 01/13] libceph: handle EADDRNOTAVAIL more gracefully

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When connecting to Ceph monitors/OSDs, kernel_connect() may return
-EADDRNOTAVAIL if the source address is unavailable. This occurs
during:
- IPv6 Duplicate Address Detection (DAD)
- IPv4/IPv6 interface state changes (link up/down events)
- Address removal or reconfiguration on the interface
- Network namespace transitions in containerized environments
- CNI reconfigurations during containerized rolling upgrades
Currently, libceph treats EADDRNOTAVAIL like any other connection error
and enters exponential backoff (BASE_DELAY_INTERVAL 250ms doubling up
to MAX_DELAY_INTERVAL 15s). Additionally, the monitor client has its
own hunt-level backoff (CEPH_MONC_HUNT_INTERVAL 3s * hunt_mult, where
hunt_mult doubles up to 10x = 30s max). These two backoff mechanisms
compound: at steady state each monitor gets ~30 seconds of attempts
with connection-level delays up to 15s, and the round-trip through
all monitors takes ~60 seconds.
In production testing (6.12.0-1-rt-amd64, Dell PowerEdge
R720, IPv6-only Ceph cluster with 2 monitors), the EADDRNOTAVAIL
condition persisted for ~36 minutes during a rolling upgrade:
  13:20:52 - mon0 session lost, hunting begins, first error -99
  13:57:03 - mon0 session finally re-established
  ~470 failed connect attempts across both monitors
  sync task blocked for 983+ seconds, triggering hung task warnings:
    "INFO: task sync:514917 blocked for more than 122 seconds"
    ...repeated at 245s, 368s, 491s, 614s, 737s, 860s, 983s
The duration of EADDRNOTAVAIL varies by environment: it can be brief
(simple DAD, 1-2s) or prolonged (complex network reconfiguration
during rolling upgrades, minutes). In both cases, the key issue is
that exponential backoff up to 15s wastes time once the address
becomes available -- the client may sit idle for up to 15 seconds
before attempting to reconnect.
This patch bypasses the exponential backoff for EADDRNOTAVAIL by
using a fixed short retry interval (ADDRNOTAVAIL_DELAY, HZ/10 =
100ms). This ensures reconnection happens within 100ms of the address
becoming available, rather than waiting up to 15 seconds.
Implementation:
- Detect EADDRNOTAVAIL in ceph_tcp_connect() for both IPv4 and IPv6
- Signal the condition to con_fault() via an addr_notavail flag
  (per-protocol: v1 and v2)
- In con_fault(), use ADDRNOTAVAIL_DELAY instead of exponential
  backoff when the flag is set
- Clear the flag on successful connection and when reopening
- Use pr_warn_ratelimited() instead of pr_err() for this case
The fast retry is appropriate because each attempt is inexpensive
(kernel_connect() fails immediately when the address is unavailable)
and quick recovery is critical for storage availability.
Fixes: 60bf8bf8815e ("libceph: fix msgr backoff")
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 include/linux/ceph/messenger.h | 11 +++++++
 net/ceph/messenger.c           | 55 ++++++++++++++++++++++++++++++++--
 2 files changed, 63 insertions(+), 3 deletions(-)

diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 1717cc57cdacd..730a754353aed 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -320,6 +320,13 @@ struct ceph_msg {
 /* ceph connection fault delay defaults, for exponential backoff */
 #define BASE_DELAY_INTERVAL	(HZ / 4)
 #define MAX_DELAY_INTERVAL	(15 * HZ)
+/*
+ * Shorter retry delay for EADDRNOTAVAIL. This error typically indicates
+ * a transient condition (IPv6 DAD in progress, address reconfiguration,
+ * temporary route issue) that resolves in 1-2 seconds. Fast retries
+ * allow quick recovery without exponential backoff delays.
+ */
+#define ADDRNOTAVAIL_DELAY	(HZ / 10)
 
 struct ceph_connection_v1_info {
 	struct kvec out_kvec[8],         /* sending header/footer data */
@@ -360,6 +367,8 @@ struct ceph_connection_v1_info {
 	u32 connect_seq;      /* identify the most recent connection
 				 attempt for this session */
 	u32 peer_global_seq;  /* peer's global seq for this connection */
+
+	bool addr_notavail;  /* address not available (transient) */
 };
 
 #define CEPH_CRC_LEN			4
@@ -430,6 +439,8 @@ struct ceph_connection_v2_info {
 
 	int con_mode;  /* CEPH_CON_MODE_* */
 
+	bool addr_notavail;  /* address not available (transient) */
+
 	void *conn_bufs[16];
 	int conn_buf_cnt;
 	int data_len_remain;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 9f6d860411cbd..c40c7c332e7f4 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -466,8 +466,22 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		     ceph_pr_addr(&con->peer_addr),
 		     sock->sk->sk_state);
 	} else if (ret < 0) {
-		pr_err("connect %s error %d\n",
-		       ceph_pr_addr(&con->peer_addr), ret);
+		if (ret == -EADDRNOTAVAIL) {
+			/*
+			 * Address not yet available - could be IPv6 DAD in
+			 * progress, address reconfiguration, or temporary
+			 * route issue. Use shorter delay.
+			 */
+			pr_warn_ratelimited("connect %s: address not available (DAD/route issue?), will retry\n",
+					    ceph_pr_addr(&con->peer_addr));
+			if (ceph_msgr2(from_msgr(con->msgr)))
+				con->v2.addr_notavail = true;
+			else
+				con->v1.addr_notavail = true;
+		} else {
+			pr_err("connect %s error %d\n",
+			       ceph_pr_addr(&con->peer_addr), ret);
+		}
 		sock_release(sock);
 		return ret;
 	}
@@ -476,6 +490,13 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		tcp_sock_set_nodelay(sock->sk);
 
 	con->sock = sock;
+
+	/* Clear addr_notavail flag on successful connection */
+	if (ceph_msgr2(from_msgr(con->msgr)))
+		con->v2.addr_notavail = false;
+	else
+		con->v1.addr_notavail = false;
+
 	return 0;
 }
 
@@ -609,6 +630,13 @@ void ceph_con_open(struct ceph_connection *con,
 
 	memcpy(&con->peer_addr, addr, sizeof(*addr));
 	con->delay = 0;      /* reset backoff memory */
+
+	/* Clear addr_notavail flag when opening/reopening connection */
+	if (ceph_msgr2(from_msgr(con->msgr)))
+		con->v2.addr_notavail = false;
+	else
+		con->v1.addr_notavail = false;
+
 	mutex_unlock(&con->mutex);
 	queue_con(con);
 }
@@ -1613,6 +1641,8 @@ static void ceph_con_workfn(struct work_struct *work)
  */
 static void con_fault(struct ceph_connection *con)
 {
+	bool addr_issue = false;
+
 	dout("fault %p state %d to peer %s\n",
 	     con, con->state, ceph_pr_addr(&con->peer_addr));
 
@@ -1620,6 +1650,19 @@ static void con_fault(struct ceph_connection *con)
 		ceph_pr_addr(&con->peer_addr), con->error_msg);
 	con->error_msg = NULL;
 
+	/* Check and reset addr_notavail flag if set */
+	if (ceph_msgr2(from_msgr(con->msgr))) {
+		if (con->v2.addr_notavail) {
+			addr_issue = true;
+			con->v2.addr_notavail = false;
+		}
+	} else {
+		if (con->v1.addr_notavail) {
+			addr_issue = true;
+			con->v1.addr_notavail = false;
+		}
+	}
+
 	WARN_ON(con->state == CEPH_CON_S_STANDBY ||
 		con->state == CEPH_CON_S_CLOSED);
 
@@ -1644,7 +1687,13 @@ static void con_fault(struct ceph_connection *con)
 	} else {
 		/* retry after a delay. */
 		con->state = CEPH_CON_S_PREOPEN;
-		if (!con->delay) {
+		if (addr_issue) {
+			/*
+			 * Address not available - use shorter delay as this
+			 * is often a transient condition.
+			 */
+			con->delay = ADDRNOTAVAIL_DELAY;
+		} else if (!con->delay) {
 			con->delay = BASE_DELAY_INTERVAL;
 		} else if (con->delay < MAX_DELAY_INTERVAL) {
 			con->delay *= 2;
-- 
2.53.0

[PATCH v1 02/13] ceph: add timeout protection to ceph_mdsc_sync() path

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When Ceph MDS becomes unreachable (e.g., due to IPv6 EADDRNOTAVAIL
during DAD or network transitions), the sync syscall can block
indefinitely in ceph_mdsc_sync(). The hung_task detector fires
repeatedly (122s, 245s, 368s... up to 983+ seconds) with traces like:
  INFO: task sync:12345 blocked for more than 122 seconds.
  Call Trace:
    ceph_mdsc_sync+0x4d6/0x5a0 [ceph]
    ceph_sync_fs+0x31/0x130 [ceph]
    iterate_supers+0x97/0x100
    ksys_sync+0x32/0xb0
Three functions in the MDS sync path use indefinite waits:
1. wait_caps_flush() uses wait_event() with no timeout
2. flush_mdlog_and_wait_mdsc_unsafe_requests() uses
   wait_for_completion() with no timeout
3. ceph_mdsc_sync() returns void, cannot propagate errors
This is particularly problematic in containerized environments with
PREEMPT_RT kernels where Ceph storage pods undergo rolling updates
and IPv6 network reconfigurations cause temporary MDS unavailability.
Fix this by adding mount_timeout-based timeouts (default 60s) to the
blocking waits, following the existing pattern used by wait_requests()
and ceph_mdsc_close_sessions() in the same file:
- wait_caps_flush(): use wait_event_timeout() with mount_timeout
- flush_mdlog_and_wait_mdsc_unsafe_requests(): use
  wait_for_completion_timeout() with mount_timeout
- ceph_mdsc_sync(): change return type to int, propagate -ETIMEDOUT
- ceph_sync_fs(): propagate error from ceph_mdsc_sync() to VFS
On timeout, dirty caps and pending requests are NOT discarded - they
remain in memory and are re-synced when MDS reconnects. The timeout
simply unblocks the calling task. If mount_timeout is set to 0,
ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, preserving the
original infinite-wait behavior.
Real-world impact: In production logs showing 'task sync blocked for
more than 983 seconds', this patch limits the block to mount_timeout
(60s default), returning -ETIMEDOUT to the VFS layer instead of
hanging indefinitely.
Fixes: 1b2ba3c5616e ("ceph: flush the mdlog for filesystem sync")
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 50 ++++++++++++++++++++++++++++++++++----------
 fs/ceph/mds_client.h |  2 +-
 fs/ceph/super.c      |  5 +++--
 3 files changed, 43 insertions(+), 14 deletions(-)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index df89d45f33a1f..37899464101f7 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2296,17 +2296,26 @@ static int check_caps_flush(struct ceph_mds_client *mdsc,
  *
  * returns true if we've flushed through want_flush_tid
  */
-static void wait_caps_flush(struct ceph_mds_client *mdsc,
-			    u64 want_flush_tid)
+static int wait_caps_flush(struct ceph_mds_client *mdsc,
+			   u64 want_flush_tid)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
+	struct ceph_options *opts = mdsc->fsc->client->options;
+	long ret;
 
 	doutc(cl, "want %llu\n", want_flush_tid);
 
-	wait_event(mdsc->cap_flushing_wq,
-		   check_caps_flush(mdsc, want_flush_tid));
+	ret = wait_event_timeout(mdsc->cap_flushing_wq,
+				 check_caps_flush(mdsc, want_flush_tid),
+				 ceph_timeout_jiffies(opts->mount_timeout));
+	if (!ret) {
+		pr_warn_client(cl, "cap flush timeout waiting for tid %llu\n",
+			       want_flush_tid);
+		return -ETIMEDOUT;
+	}
 
 	doutc(cl, "ok, flushed thru %llu\n", want_flush_tid);
+	return 0;
 }
 
 /*
@@ -5838,13 +5847,15 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)
 /*
  * flush the mdlog and wait for all write mds requests to flush.
  */
-static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
-						 u64 want_tid)
+static int flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
+						      u64 want_tid)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
+	struct ceph_options *opts = mdsc->fsc->client->options;
 	struct ceph_mds_request *req = NULL, *nextreq;
 	struct ceph_mds_session *last_session = NULL;
 	struct rb_node *n;
+	unsigned long left;
 
 	mutex_lock(&mdsc->mutex);
 	doutc(cl, "want %lld\n", want_tid);
@@ -5883,7 +5894,19 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
 			}
 			doutc(cl, "wait on %llu (want %llu)\n",
 			      req->r_tid, want_tid);
-			wait_for_completion(&req->r_safe_completion);
+			left = wait_for_completion_timeout(
+					&req->r_safe_completion,
+					ceph_timeout_jiffies(opts->mount_timeout));
+			if (!left) {
+				pr_warn_client(cl,
+					       "flush mdlog request tid %llu timed out\n",
+					       req->r_tid);
+				ceph_mdsc_put_request(req);
+				if (nextreq)
+					ceph_mdsc_put_request(nextreq);
+				ceph_put_mds_session(last_session);
+				return -ETIMEDOUT;
+			}
 
 			mutex_lock(&mdsc->mutex);
 			ceph_mdsc_put_request(req);
@@ -5901,15 +5924,17 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
 	mutex_unlock(&mdsc->mutex);
 	ceph_put_mds_session(last_session);
 	doutc(cl, "done\n");
+	return 0;
 }
 
-void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
+int ceph_mdsc_sync(struct ceph_mds_client *mdsc)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
 	u64 want_tid, want_flush;
+	int ret;
 
 	if (READ_ONCE(mdsc->fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN)
-		return;
+		return -EIO;
 
 	doutc(cl, "sync\n");
 	mutex_lock(&mdsc->mutex);
@@ -5930,8 +5955,11 @@ void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
 
 	doutc(cl, "sync want tid %lld flush_seq %lld\n", want_tid, want_flush);
 
-	flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
-	wait_caps_flush(mdsc, want_flush);
+	ret = flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
+	if (ret)
+		return ret;
+
+	return wait_caps_flush(mdsc, want_flush);
 }
 
 /*
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 0a602080d8ef6..695c5a9c94026 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -564,7 +564,7 @@ extern void ceph_mdsc_close_sessions(struct ceph_mds_client *mdsc);
 extern void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc);
 extern void ceph_mdsc_destroy(struct ceph_fs_client *fsc);
 
-extern void ceph_mdsc_sync(struct ceph_mds_client *mdsc);
+extern int ceph_mdsc_sync(struct ceph_mds_client *mdsc);
 
 extern void ceph_invalidate_dir_request(struct ceph_mds_request *req);
 extern int ceph_alloc_readdir_reply_buffer(struct ceph_mds_request *req,
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index b61074b377ac5..b52960402d68e 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -122,6 +122,7 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 {
 	struct ceph_fs_client *fsc = ceph_sb_to_fs_client(sb);
 	struct ceph_client *cl = fsc->client;
+	int ret;
 
 	if (!wait) {
 		doutc(cl, "(non-blocking)\n");
@@ -133,9 +134,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 
 	doutc(cl, "(blocking)\n");
 	ceph_osdc_sync(&fsc->client->osdc);
-	ceph_mdsc_sync(fsc->mdsc);
+	ret = ceph_mdsc_sync(fsc->mdsc);
 	doutc(cl, "(blocking) done\n");
-	return 0;
+	return ret;
 }
 
 /*
-- 
2.53.0

[PATCH v1 03/13] ceph: add timeout protection to ceph_osdc_sync() path

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When a Ceph OSD becomes unreachable (e.g., due to IPv6 EADDRNOTAVAIL
during DAD or network transitions), the sync syscall can block
indefinitely in ceph_osdc_sync(). This function iterates over all
in-flight write requests and calls wait_for_completion() with no
timeout on each one. The hung_task detector fires repeatedly with
stack traces showing:
  ceph_osdc_sync [libceph]
  ceph_sync_fs [ceph]
  iterate_supers
  ksys_sync
Since ceph_osdc_sync() is called before ceph_mdsc_sync() in
ceph_sync_fs(), an OSD hang prevents the MDS timeout protection
from commit e789e5252fda ("ceph: add timeout protection to
ceph_mdsc_sync() path") from ever being reached.
This is particularly problematic in containerized environments with
PREEMPT_RT kernels where Ceph storage pods undergo rolling updates
and IPv6 network reconfigurations cause temporary OSD unavailability.
Fix this by adding mount_timeout-based timeout to the blocking wait,
following the existing pattern used by wait_request_timeout() in the
same file:
- ceph_osdc_sync(): use wait_for_completion_timeout() with
  mount_timeout instead of indefinite wait_for_completion()
- Change return type from void to int, return -ETIMEDOUT on timeout
- ceph_sync_fs(): propagate OSD sync error, short-circuit before
  MDS sync on failure
On timeout, pending OSD requests are NOT cancelled - they remain
in-flight and complete when the OSD reconnects. The timeout simply
unblocks the calling task. If mount_timeout is set to 0,
ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, preserving the
original infinite-wait behavior.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/super.c                 |  4 +++-
 include/linux/ceph/osd_client.h |  2 +-
 net/ceph/osd_client.c           | 15 +++++++++++++--
 3 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index b52960402d68e..6f4ee457c1b52 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -133,7 +133,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
 	}
 
 	doutc(cl, "(blocking)\n");
-	ceph_osdc_sync(&fsc->client->osdc);
+	ret = ceph_osdc_sync(&fsc->client->osdc);
+	if (ret)
+		return ret;
 	ret = ceph_mdsc_sync(fsc->mdsc);
 	doutc(cl, "(blocking) done\n");
 	return ret;
diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
index d7941478158cd..871827e2dd983 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -587,7 +587,7 @@ void ceph_osdc_start_request(struct ceph_osd_client *osdc,
 extern void ceph_osdc_cancel_request(struct ceph_osd_request *req);
 extern int ceph_osdc_wait_request(struct ceph_osd_client *osdc,
 				  struct ceph_osd_request *req);
-extern void ceph_osdc_sync(struct ceph_osd_client *osdc);
+extern int ceph_osdc_sync(struct ceph_osd_client *osdc);
 
 extern void ceph_osdc_flush_notifies(struct ceph_osd_client *osdc);
 void ceph_osdc_maybe_request_map(struct ceph_osd_client *osdc);
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index abac770bc0b4c..7d5e4a078fb10 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -4734,10 +4734,13 @@ EXPORT_SYMBOL(ceph_osdc_wait_request);
 /*
  * sync - wait for all in-flight requests to flush.  avoid starvation.
  */
-void ceph_osdc_sync(struct ceph_osd_client *osdc)
+int ceph_osdc_sync(struct ceph_osd_client *osdc)
 {
+	struct ceph_options *opts = osdc->client->options;
+	unsigned long timeout = ceph_timeout_jiffies(opts->mount_timeout);
 	struct rb_node *n, *p;
 	u64 last_tid = atomic64_read(&osdc->last_tid);
+	unsigned long left;
 
 again:
 	down_read(&osdc->lock);
@@ -4760,7 +4763,14 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
 			up_read(&osdc->lock);
 			dout("%s waiting on req %p tid %llu last_tid %llu\n",
 			     __func__, req, req->r_tid, last_tid);
-			wait_for_completion(&req->r_completion);
+			left = wait_for_completion_timeout(&req->r_completion,
+							   timeout);
+			if (!left) {
+				pr_warn("ceph: osd sync request tid %llu timed out\n",
+					req->r_tid);
+				ceph_osdc_put_request(req);
+				return -ETIMEDOUT;
+			}
 			ceph_osdc_put_request(req);
 			goto again;
 		}
@@ -4770,6 +4780,7 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
 
 	up_read(&osdc->lock);
 	dout("%s done last_tid %llu\n", __func__, last_tid);
+	return 0;
 }
 EXPORT_SYMBOL(ceph_osdc_sync);
 
-- 
2.53.0

[PATCH v1 04/13] ceph: fix race condition in cleanup_session_requests()

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When an MDS session is closed or reset, cleanup_session_requests()
only unregisters requests that are on the session's s_unsafe list.
However, requests are only added to s_unsafe after receiving an
"unsafe" reply from the MDS.
This creates a race condition: if a write request has been sent
but the MDS becomes unavailable before sending the unsafe reply,
the request will:
  - Have r_session set (points to the failed session)
  - Be in the request_tree
  - NOT be on s_unsafe list
  - Never have r_safe_completion signaled
Meanwhile, flush_mdlog_and_wait_mdsc_unsafe_requests() iterates
the request_tree looking for write requests with r_session set,
and waits on r_safe_completion for each one. Since the request
is not on s_unsafe, cleanup_session_requests() won't unregister
it, and the completion is never signaled - causing an indefinite
hang.
This was observed in production when running xfstests generic/013
in a loop, with stack traces showing:
  INFO: task fsstress:14466 blocked for more than 122 seconds.
  Call Trace:
    wait_for_completion+0x14a/0x340
    ceph_mdsc_sync+0x4b4/0xe80
    ceph_sync_fs+0xa0/0x4c0
    sync_filesystem+0x182/0x240
Fix this by extending cleanup_session_requests() to also unregister
requests that:
  - Belong to the closing session (r_session->s_mds matches)
  - Have NOT received an unsafe reply (CEPH_MDS_R_GOT_UNSAFE not set)
  - Have NOT received a safe reply (CEPH_MDS_R_GOT_SAFE not set)
These are requests that were in-flight when the session failed and
will never complete. Unregistering them signals r_safe_completion,
unblocking any waiters.
Requests that received an unsafe reply but not yet a safe reply
are already on s_unsafe and handled by the existing code. For
these, we preserve the original behavior of resetting r_attempts
to allow re-sending when the session reconnects.
Fixes: e3ec8d689cf4 ("ceph: clean up unsafe requests when reconnecting is denied")
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 37899464101f7..45abddd7f317e 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -1792,6 +1792,8 @@ static void cleanup_session_requests(struct ceph_mds_client *mdsc,
 
 	doutc(cl, "mds%d\n", session->s_mds);
 	mutex_lock(&mdsc->mutex);
+
+	/* First, handle requests on the unsafe list */
 	while (!list_empty(&session->s_unsafe)) {
 		req = list_first_entry(&session->s_unsafe,
 				       struct ceph_mds_request, r_unsafe_item);
@@ -1803,14 +1805,30 @@ static void cleanup_session_requests(struct ceph_mds_client *mdsc,
 			mapping_set_error(req->r_unsafe_dir->i_mapping, -EIO);
 		__unregister_request(mdsc, req);
 	}
-	/* zero r_attempts, so kick_requests() will re-send requests */
+
+	/*
+	 * Iterate through all pending requests for this session.
+	 * Requests that haven't received an unsafe reply yet will never
+	 * complete on this session - unregister them to signal waiters.
+	 * Requests that got unsafe but not safe are handled above via
+	 * s_unsafe list; for any remaining, reset r_attempts to allow
+	 * re-sending when session reconnects.
+	 */
 	p = rb_first(&mdsc->request_tree);
 	while (p) {
 		req = rb_entry(p, struct ceph_mds_request, r_node);
 		p = rb_next(p);
 		if (req->r_session &&
-		    req->r_session->s_mds == session->s_mds)
-			req->r_attempts = 0;
+		    req->r_session->s_mds == session->s_mds) {
+			if (!test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags) &&
+			    !test_bit(CEPH_MDS_R_GOT_SAFE, &req->r_req_flags)) {
+				doutc(cl, " dropping pending request %llu\n",
+				      req->r_tid);
+				__unregister_request(mdsc, req);
+			} else {
+				req->r_attempts = 0;
+			}
+		}
 	}
 	mutex_unlock(&mdsc->mutex);
 }
-- 
2.53.0

[PATCH v1 05/13] ceph: add timeout protection to ceph_lock_wait_for_completion()

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When a file lock operation is interrupted and an unlock request is
sent to cancel it, ceph_lock_wait_for_completion() waits indefinitely
for r_safe_completion using wait_for_completion_killable().
If the MDS becomes unreachable after the unlock request is sent,
this wait will block indefinitely, causing hung task warnings:
  INFO: task flock:12345 blocked for more than 122 seconds.
  Call Trace:
    wait_for_completion_killable+0x...
    ceph_lock_wait_for_completion+0x...
    ceph_flock+0x...
This is similar to the issue fixed in ceph_mdsc_sync() where
indefinite waits on r_safe_completion can hang when MDS is
unavailable.
Fix this by using wait_for_completion_killable_timeout() with
mount_timeout instead of the indefinite wait. On timeout, return
-ETIMEDOUT to the caller. The lock state remains consistent because:
1. If the unlock succeeded on MDS, the lock is released
2. If the unlock didn't reach MDS, the original lock request
   was already aborted (CEPH_MDS_R_ABORTED set), so MDS will
   clean it up on reconnect
This follows the same timeout pattern used throughout the ceph
client for MDS operations.
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/locks.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index ebf4ac0055ddc..55dd99460b81a 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -160,6 +160,8 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
                                          struct ceph_mds_request *req)
 {
 	struct ceph_client *cl = mdsc->fsc->client;
+	struct ceph_options *opts = mdsc->fsc->client->options;
+	unsigned long timeout = ceph_timeout_jiffies(opts->mount_timeout);
 	struct ceph_mds_request *intr_req;
 	struct inode *inode = req->r_inode;
 	int err, lock_type;
@@ -221,7 +223,17 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
 	if (err && err != -ERESTARTSYS)
 		return err;
 
-	wait_for_completion_killable(&req->r_safe_completion);
+	err = wait_for_completion_killable_timeout(&req->r_safe_completion,
+						   timeout);
+	if (err == -ERESTARTSYS) {
+		/* Interrupted again, just return the error */
+		return err;
+	}
+	if (err == 0) {
+		pr_warn_client(cl, "lock request tid %llu safe completion timed out\n",
+			       req->r_tid);
+		return -ETIMEDOUT;
+	}
 	return 0;
 }
 
-- 
2.53.0

[PATCH v1 06/13] ceph: set default timeout for MDS requests

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

MDS requests created via ceph_mdsc_create_request() have r_timeout
initialized to 0 (from kmem_cache_zalloc). When r_timeout is 0,
ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, causing
ceph_mdsc_wait_request() to wait indefinitely.

This causes hung task warnings when MDS becomes unavailable during
operations like setattr or truncate:

  INFO: task dd:12345 blocked for more than 122 seconds.
  Call Trace:
    ceph_mdsc_wait_request+0x...
    ceph_mdsc_do_request+0x...
    __ceph_setattr+0x...

Only the mount path in super.c explicitly sets r_timeout to
mount_timeout. All other MDS requests (setattr, lookup, mkdir,
etc.) use the default 0 value, making them wait forever.

Fix this by initializing r_timeout to mount_timeout in
ceph_mdsc_create_request(). This ensures all MDS requests have
a reasonable timeout and will fail with -ETIMEDOUT rather than
hanging indefinitely.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 45abddd7f317e..ac86225595b5f 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2613,6 +2613,7 @@ ceph_mdsc_create_request(struct ceph_mds_client *mdsc, int op, int mode)
 	mutex_init(&req->r_fill_mutex);
 	req->r_mdsc = mdsc;
 	req->r_started = jiffies;
+	req->r_timeout = mdsc->fsc->client->options->mount_timeout;
 	req->r_start_latency = ktime_get();
 	req->r_resend_mds = -1;
 	INIT_LIST_HEAD(&req->r_unsafe_dir_item);
-- 
2.53.0

[PATCH v1 07/13] ceph: add timeout to caps wait in __ceph_get_caps()

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When waiting for caps in __ceph_get_caps(), the code uses
wait_woken() with MAX_SCHEDULE_TIMEOUT, which can block
indefinitely if the MDS is unavailable or slow to grant caps
during reconnection.

This causes hung task warnings when MDS fails over:

  INFO: task dd:12345 blocked for more than 122 seconds.
  Call Trace:
    __ceph_get_caps+0x...
    ceph_write_iter+0x...

During MDS failover, caps may be revoked or delayed while the
client reconnects. Processes waiting for caps block indefinitely,
also holding i_rwsem which blocks other I/O operations on the
same inode, causing a cascade of blocked processes.

Fix this by using wait_woken() with mount_timeout instead of
MAX_SCHEDULE_TIMEOUT. On timeout, return -ETIMEDOUT to allow
the caller to handle the situation appropriately.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/caps.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c
index bed34fc11c919..c88e10a634e5c 100644
--- a/fs/ceph/caps.c
+++ b/fs/ceph/caps.c
@@ -3055,7 +3055,10 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 {
 	struct ceph_inode_info *ci = ceph_inode(inode);
 	struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+	struct ceph_client *cl = fsc->client;
+	unsigned long timeout = ceph_timeout_jiffies(cl->options->mount_timeout);
 	int ret, _got, flags;
+	bool warned = false;
 
 	ret = ceph_pool_perm_check(inode, need);
 	if (ret < 0)
@@ -3104,7 +3107,18 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 					ret = -ERESTARTSYS;
 					break;
 				}
-				wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
+				if (!wait_woken(&wait, TASK_INTERRUPTIBLE, timeout)) {
+					if (!warned) {
+						pr_warn_ratelimited_client(cl,
+							"%p %llx.%llx caps wait timed out (need %s want %s)\n",
+							inode, ceph_vinop(inode),
+							ceph_cap_string(need),
+							ceph_cap_string(want));
+						warned = true;
+					}
+					ret = -ETIMEDOUT;
+					break;
+				}
 			}
 
 			remove_wait_queue(&ci->i_cap_wq, &wait);
-- 
2.53.0

[PATCH v1 08/13] ceph: make ceph_start_io_write() killable

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When multiple processes write to the same file and one of them is
blocked waiting for MDS/OSD response (e.g., during MDS failover),
other processes block indefinitely on down_write(&inode->i_rwsem)
in ceph_start_io_write().

This causes hung task warnings:

  INFO: task dd:12345 blocked for more than 122 seconds.
  Call Trace:
    ceph_start_io_write+0x...
    ceph_write_iter+0x...

The i_rwsem is held by a process doing fsync/writeback that is
waiting for MDS or OSD response. Other writers queue up on the
rwsem and block indefinitely.

Fix this by using down_write_killable() instead of down_write().
This allows blocked processes to be killed with SIGKILL, preventing
indefinite hangs. The function now returns an error code that
callers must check.

Update ceph_write_iter() to handle the new error return from
ceph_start_io_write().

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/file.c | 9 +++++++--
 fs/ceph/io.c   | 9 +++++++--
 fs/ceph/io.h   | 2 +-
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 6587c2d5af1e0..01e4f31b1f2f3 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -2359,8 +2359,13 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
 retry_snap:
 	if (direct_lock)
 		ceph_start_io_direct(inode);
-	else
-		ceph_start_io_write(inode);
+	else {
+		err = ceph_start_io_write(inode);
+		if (err) {
+			ceph_free_cap_flush(prealloc_cf);
+			return err;
+		}
+	}
 
 	if (iocb->ki_flags & IOCB_APPEND) {
 		err = ceph_do_getattr(inode, CEPH_STAT_CAP_SIZE, false);
diff --git a/fs/ceph/io.c b/fs/ceph/io.c
index c456509b31c3f..f9ac89ec1d6a1 100644
--- a/fs/ceph/io.c
+++ b/fs/ceph/io.c
@@ -83,11 +83,16 @@ ceph_end_io_read(struct inode *inode)
  * Declare that a buffered write operation is about to start, and ensure
  * that we block all direct I/O.
  */
-void
+int
 ceph_start_io_write(struct inode *inode)
 {
-	down_write(&inode->i_rwsem);
+	int ret;
+
+	ret = down_write_killable(&inode->i_rwsem);
+	if (ret)
+		return ret;
 	ceph_block_o_direct(ceph_inode(inode), inode);
+	return 0;
 }
 
 /**
diff --git a/fs/ceph/io.h b/fs/ceph/io.h
index fa594cd77348a..94ce176df9997 100644
--- a/fs/ceph/io.h
+++ b/fs/ceph/io.h
@@ -4,7 +4,7 @@
 
 void ceph_start_io_read(struct inode *inode);
 void ceph_end_io_read(struct inode *inode);
-void ceph_start_io_write(struct inode *inode);
+int ceph_start_io_write(struct inode *inode);
 void ceph_end_io_write(struct inode *inode);
 void ceph_start_io_direct(struct inode *inode);
 void ceph_end_io_direct(struct inode *inode);
-- 
2.53.0

[PATCH v1 09/13] ceph: make remaining I/O lock functions killable

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

Following the same pattern as ceph_start_io_write(), make
ceph_start_io_read() and ceph_start_io_direct() killable to
prevent indefinite hangs when waiting for i_rwsem during
MDS/OSD unavailability.

This completes the killable lock conversion for all ceph I/O
start functions, allowing blocked processes to be terminated
with SIGKILL instead of hanging indefinitely.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/file.c | 27 +++++++++++++++++++--------
 fs/ceph/io.c   | 28 ++++++++++++++++++++--------
 fs/ceph/io.h   |  4 ++--
 3 files changed, 41 insertions(+), 18 deletions(-)

diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 01e4f31b1f2f3..c828552d51920 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -2122,10 +2122,15 @@ static ssize_t ceph_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	if (ceph_inode_is_shutdown(inode))
 		return -ESTALE;
 
-	if (direct_lock)
-		ceph_start_io_direct(inode);
-	else
-		ceph_start_io_read(inode);
+	if (direct_lock) {
+		ret = ceph_start_io_direct(inode);
+		if (ret)
+			return ret;
+	} else {
+		ret = ceph_start_io_read(inode);
+		if (ret)
+			return ret;
+	}
 
 	if (!(fi->flags & CEPH_F_SYNC) && !direct_lock)
 		want |= CEPH_CAP_FILE_CACHE;
@@ -2278,7 +2283,9 @@ static ssize_t ceph_splice_read(struct file *in, loff_t *ppos,
 	    (fi->flags & CEPH_F_SYNC))
 		return copy_splice_read(in, ppos, pipe, len, flags);
 
-	ceph_start_io_read(inode);
+	ret = ceph_start_io_read(inode);
+	if (ret)
+		return ret;
 
 	want = CEPH_CAP_FILE_CACHE;
 	if (fi->fmode & CEPH_FILE_MODE_LAZY)
@@ -2357,9 +2364,13 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		direct_lock = true;
 
 retry_snap:
-	if (direct_lock)
-		ceph_start_io_direct(inode);
-	else {
+	if (direct_lock) {
+		err = ceph_start_io_direct(inode);
+		if (err) {
+			ceph_free_cap_flush(prealloc_cf);
+			return err;
+		}
+	} else {
 		err = ceph_start_io_write(inode);
 		if (err) {
 			ceph_free_cap_flush(prealloc_cf);
diff --git a/fs/ceph/io.c b/fs/ceph/io.c
index f9ac89ec1d6a1..7bd57de2d9681 100644
--- a/fs/ceph/io.c
+++ b/fs/ceph/io.c
@@ -47,20 +47,26 @@ static void ceph_block_o_direct(struct ceph_inode_info *ci, struct inode *inode)
  * Note that buffered writes and truncates both take a write lock on
  * inode->i_rwsem, meaning that those are serialised w.r.t. the reads.
  */
-void
+int
 ceph_start_io_read(struct inode *inode)
 {
 	struct ceph_inode_info *ci = ceph_inode(inode);
+	int ret;
 
 	/* Be an optimist! */
-	down_read(&inode->i_rwsem);
+	ret = down_read_killable(&inode->i_rwsem);
+	if (ret)
+		return ret;
 	if (!(READ_ONCE(ci->i_ceph_flags) & CEPH_I_ODIRECT))
-		return;
+		return 0;
 	up_read(&inode->i_rwsem);
 	/* Slow path.... */
-	down_write(&inode->i_rwsem);
+	ret = down_write_killable(&inode->i_rwsem);
+	if (ret)
+		return ret;
 	ceph_block_o_direct(ci, inode);
 	downgrade_write(&inode->i_rwsem);
+	return 0;
 }
 
 /**
@@ -138,20 +144,26 @@ static void ceph_block_buffered(struct ceph_inode_info *ci, struct inode *inode)
  * Note that buffered writes and truncates both take a write lock on
  * inode->i_rwsem, meaning that those are serialised w.r.t. O_DIRECT.
  */
-void
+int
 ceph_start_io_direct(struct inode *inode)
 {
 	struct ceph_inode_info *ci = ceph_inode(inode);
+	int ret;
 
 	/* Be an optimist! */
-	down_read(&inode->i_rwsem);
+	ret = down_read_killable(&inode->i_rwsem);
+	if (ret)
+		return ret;
 	if (READ_ONCE(ci->i_ceph_flags) & CEPH_I_ODIRECT)
-		return;
+		return 0;
 	up_read(&inode->i_rwsem);
 	/* Slow path.... */
-	down_write(&inode->i_rwsem);
+	ret = down_write_killable(&inode->i_rwsem);
+	if (ret)
+		return ret;
 	ceph_block_buffered(ci, inode);
 	downgrade_write(&inode->i_rwsem);
+	return 0;
 }
 
 /**
diff --git a/fs/ceph/io.h b/fs/ceph/io.h
index 94ce176df9997..9432b8b607650 100644
--- a/fs/ceph/io.h
+++ b/fs/ceph/io.h
@@ -2,11 +2,11 @@
 #ifndef _FS_CEPH_IO_H
 #define _FS_CEPH_IO_H
 
-void ceph_start_io_read(struct inode *inode);
+int ceph_start_io_read(struct inode *inode);
 void ceph_end_io_read(struct inode *inode);
 int ceph_start_io_write(struct inode *inode);
 void ceph_end_io_write(struct inode *inode);
-void ceph_start_io_direct(struct inode *inode);
+int ceph_start_io_direct(struct inode *inode);
 void ceph_end_io_direct(struct inode *inode);
 
 #endif /* FS_CEPH_IO_H */
-- 
2.53.0

[PATCH v1 10/13] ceph: force mdsmap refresh on persistent MDS connection failures

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

During rolling upgrades in containerized environments (e.g.,
rook-ceph in containerized environments), MDS daemons are restarted and
may receive new IP addresses from the CNI plugin. The kernel CephFS
client (libceph) maintains a cached mdsmap with the old MDS address
and attempts to reconnect indefinitely.

The monitor client subscribes to mdsmap updates with
start=current_epoch+1, expecting the monitor to push new maps.
However, if the monitor connection was also disrupted during the
upgrade (e.g., due to EADDRNOTAVAIL from IPv6 DAD), the subscription
may not be properly re-established, leaving the client with a stale
mdsmap.

This results in a deadlock:
- The kernel client retries connecting to the old MDS address forever
- The MDS connection has no .fault callback, so the MDS client is
  never notified of persistent connection failures
- The stale mdsmap is never refreshed because the client believes
  its subscription is active
- New pod mounts via CSI hang in ContainerCreating state
- The rook-ceph upgrade cannot complete

Observed in production (kernel 6.12.0-1-rt-amd64,
Ceph Reef 18.2.2->18.2.5 upgrade):
  - mdsmap stuck at epoch 53 while cluster was at epoch 68
  - MDS session state: hung
  - monc showed: have mdsmap 53 want 54+
  - MDS address changed from dead:beef::...eb75 to dead:beef::...bc76
  - Client kept retrying on old address for 30+ minutes

Fix this by:
1. Adding a .fault callback to the MDS connection operations
   (mds_con_ops) so the MDS client is notified when connections fail
2. Tracking consecutive connection failures per MDS session via a
   new s_con_failures counter
3. When failures exceed MDS_CON_FAIL_REFRESH_MDSMAP (10 consecutive
   failures, ~2.5-15 seconds depending on backoff), forcing a fresh
   mdsmap subscription with start=0 to get the complete current map
4. Resetting the failure counter when a session message is
   successfully received (in handle_session)

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 fs/ceph/mds_client.c | 73 ++++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/mds_client.h |  2 ++
 2 files changed, 75 insertions(+)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index ac86225595b5f..0e766880056c0 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -66,6 +66,12 @@ static void ceph_cap_release_work(struct work_struct *work);
 static void ceph_cap_reclaim_work(struct work_struct *work);
 
 static const struct ceph_connection_operations mds_con_ops;
+/*
+ * Number of consecutive MDS connection failures before forcing
+ * a fresh mdsmap subscription. This handles stale mdsmap scenarios
+ * during rolling upgrades where MDS addresses change.
+ */
+#define MDS_CON_FAIL_REFRESH_MDSMAP	10
 
 
 /*
@@ -997,6 +1003,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,
 	s->s_mdsc = mdsc;
 	s->s_mds = mds;
 	s->s_state = CEPH_MDS_SESSION_NEW;
+	s->s_con_failures = 0;
 	mutex_init(&s->s_mutex);
 
 	ceph_con_init(&s->s_con, s, &mds_con_ops, &mdsc->fsc->client->msgr);
@@ -4341,6 +4348,9 @@ static void handle_session(struct ceph_mds_session *session,
 	      ceph_session_op_name(op), session,
 	      ceph_session_state_name(session->s_state), seq);
 
+	/* Reset connection failure counter on successful session message */
+	session->s_con_failures = 0;
+
 	if (session->s_state == CEPH_MDS_SESSION_HUNG) {
 		session->s_state = CEPH_MDS_SESSION_OPEN;
 		pr_info_client(cl, "mds%d came back\n", session->s_mds);
@@ -5427,6 +5437,22 @@ bool check_session_state(struct ceph_mds_session *s)
 		if (s->s_ttl && time_after(jiffies, s->s_ttl)) {
 			s->s_state = CEPH_MDS_SESSION_HUNG;
 			pr_info_client(cl, "mds%d hung\n", s->s_mds);
+
+			/*
+			 * Force a fresh mdsmap subscription when a session
+			 * becomes hung. The MDS may have restarted with a
+			 * new address during a rolling upgrade, and the
+			 * connection may have entered STANDBY state (no
+			 * .fault callback) rather than generating connect
+			 * errors. Requesting mdsmap from epoch 0 ensures
+			 * we get the current map with updated addresses.
+			 */
+			pr_warn_client(cl,
+				"mds%d hung, requesting fresh mdsmap\n",
+				s->s_mds);
+			if (ceph_monc_want_map(&s->s_mdsc->fsc->client->monc,
+					       CEPH_SUB_MDSMAP, 0, true))
+				ceph_monc_renew_subs(&s->s_mdsc->fsc->client->monc);
 		}
 		break;
 	case CEPH_MDS_SESSION_CLOSING:
@@ -6528,12 +6554,59 @@ static int mds_check_message_signature(struct ceph_msg *msg)
        return ceph_auth_check_message_signature(auth, msg);
 }
 
+/*
+ * Handle MDS connection fault.
+ *
+ * Track consecutive connection failures and force a fresh mdsmap
+ * subscription when failures exceed the threshold. This handles the
+ * case where the MDS address has changed (e.g., during a rolling
+ * upgrade) but the client has a stale mdsmap and keeps retrying
+ * on the old address.
+ */
+static void mds_fault(struct ceph_connection *con)
+{
+	struct ceph_mds_session *s = con->private;
+	struct ceph_mds_client *mdsc = s->s_mdsc;
+	struct ceph_client *cl = mdsc->fsc->client;
+	int failures;
+
+	failures = ++s->s_con_failures;
+
+	if (failures == MDS_CON_FAIL_REFRESH_MDSMAP) {
+		pr_warn_client(cl,
+			"mds%d connection failed %d times, requesting fresh mdsmap\n",
+			s->s_mds, failures);
+
+		/*
+		 * Force a fresh mdsmap subscription by requesting from
+		 * epoch 0. This ensures we get the complete current map
+		 * with up-to-date MDS addresses, rather than waiting for
+		 * an incremental update that may never arrive if our
+		 * subscription was lost during a monitor reconnection.
+		 */
+		if (ceph_monc_want_map(&mdsc->fsc->client->monc,
+				       CEPH_SUB_MDSMAP, 0, true))
+			ceph_monc_renew_subs(&mdsc->fsc->client->monc);
+	} else if (failures > MDS_CON_FAIL_REFRESH_MDSMAP &&
+		   failures % MDS_CON_FAIL_REFRESH_MDSMAP == 0) {
+		/*
+		 * Periodically retry the fresh mdsmap request in case
+		 * the previous one was lost or the monitor was also
+		 * temporarily unavailable.
+		 */
+		if (ceph_monc_want_map(&mdsc->fsc->client->monc,
+				       CEPH_SUB_MDSMAP, 0, true))
+			ceph_monc_renew_subs(&mdsc->fsc->client->monc);
+	}
+}
+
 static const struct ceph_connection_operations mds_con_ops = {
 	.get = mds_get_con,
 	.put = mds_put_con,
 	.alloc_msg = mds_alloc_msg,
 	.dispatch = mds_dispatch,
 	.peer_reset = mds_peer_reset,
+	.fault = mds_fault,
 	.get_authorizer = mds_get_authorizer,
 	.add_authorizer_challenge = mds_add_authorizer_challenge,
 	.verify_authorizer_reply = mds_verify_authorizer_reply,
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 695c5a9c94026..44585b1cb4485 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -251,6 +251,8 @@ struct ceph_mds_session {
 	struct list_head  s_waiting;  /* waiting requests */
 	struct list_head  s_unsafe;   /* unsafe requests */
 	struct xarray	  s_delegated_inos;
+
+	int		  s_con_failures; /* consecutive connection failures */
 };
 
 /*
-- 
2.53.0

[PATCH v1 11/13] libceph: reset source address on persistent EADDRNOTAVAIL

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

In containerized environments (e.g., Rook-Ceph with
Calico CNI), the kernel CephFS client's source address
(msgr->inst.addr) is learned from the first successful monitor
connection via process_hello(). If the initial connection was made
through a transient CNI pod address (e.g., a Calico-assigned
dead:beef::... address from a CSI plugin pod), that address is
stored permanently in inst.addr.

When the pod is later rescheduled or the CNI reconfigures networking,
the original pod address is removed and Calico installs a blackhole
route for the old address range. All subsequent kernel socket
connections fail with EADDRNOTAVAIL at ip6_dst_lookup_flow() before
even sending a TCP SYN, because the IPv6 source address selection
finds the blackhole route for the old address range.

This creates a permanent deadlock:
- All connections (mon, mds, osd) fail with EADDRNOTAVAIL
- The client cannot reach any monitor to re-learn its address
- inst.addr is never blank again (set once, never cleared)
- The only recovery is force-unmounting and remounting

Fix this by tracking consecutive EADDRNOTAVAIL failures across all
connections using an atomic counter in struct ceph_messenger. After
ADDRNOTAVAIL_RESET_THRESHOLD (30) consecutive failures (~3 seconds
at 100ms retry interval), reset inst.addr.in_addr to zero (blank)
while preserving the nonce and type. This allows process_hello()
(msgr2) or process_banner() (msgr1) to re-learn the source address
from the next successful monitor connection, which will use the
current stable host address instead of the defunct pod address.

The counter is reset to zero when:
- A TCP connection succeeds (in ceph_tcp_connect)
- The address is successfully re-learned (in process_hello/
  process_banner)

Observed in production (kernel 6.12.0-1-rt-amd64, Ceph Reef
18.2.2->18.2.5 upgrade, IPv6-only cluster):
  - Client instance: client.55136 [dead:beef::a2bf:c94c:345d:bc66]:0
  - Address dead:beef::a2bf:c94c:345d:bc66 was a Calico pod address
  - After pod reschedule: blackhole dead:beef::a2bf:c94c:345d:bc40/122
  - All connections stuck in EADDRNOTAVAIL loop for 16+ hours
  - After force-unmount + remount: new client got stable host address
    [aefd::2b93:d245:fd09:127e]:0 and worked immediately

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 include/linux/ceph/messenger.h | 20 +++++++++++++
 net/ceph/messenger.c           | 51 ++++++++++++++++++++++++++++++++++
 net/ceph/messenger_v1.c        |  7 +++++
 net/ceph/messenger_v2.c        | 12 ++++++++
 4 files changed, 90 insertions(+)

diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 730a754353aed..d8f7946d85a68 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -113,6 +113,17 @@ struct ceph_messenger {
 	 */
 	u32 global_seq;
 	spinlock_t global_seq_lock;
+
+	/*
+	 * Track consecutive EADDRNOTAVAIL failures across all
+	 * connections. When this exceeds a threshold, the client's
+	 * inst.addr is reset to blank so that process_hello() will
+	 * re-learn the source address from the next successful
+	 * monitor connection. This handles the case where the
+	 * original source address was a transient CNI pod address
+	 * that no longer exists.
+	 */
+	atomic_t addr_notavail_count;
 };
 
 enum ceph_msg_data_type {
@@ -328,6 +339,15 @@ struct ceph_msg {
  */
 #define ADDRNOTAVAIL_DELAY	(HZ / 10)
 
+/*
+ * Number of consecutive EADDRNOTAVAIL failures (across all connections)
+ * before resetting the messenger's source address. At ~100ms per retry,
+ * 30 failures means ~3 seconds of persistent EADDRNOTAVAIL before we
+ * conclude the source address is permanently gone (e.g., a CNI pod
+ * address that was removed) and needs to be re-learned.
+ */
+#define ADDRNOTAVAIL_RESET_THRESHOLD	30
+
 struct ceph_connection_v1_info {
 	struct kvec out_kvec[8],         /* sending header/footer data */
 		*out_kvec_cur;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index c40c7c332e7f4..8165e6a8fe092 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -497,6 +497,10 @@ int ceph_tcp_connect(struct ceph_connection *con)
 	else
 		con->v1.addr_notavail = false;
 
+	/* Reset the persistent EADDRNOTAVAIL counter on success */
+	if (atomic_read(&con->msgr->addr_notavail_count) > 0)
+		atomic_set(&con->msgr->addr_notavail_count, 0);
+
 	return 0;
 }
 
@@ -1663,6 +1667,52 @@ static void con_fault(struct ceph_connection *con)
 		}
 	}
 
+	/*
+	 * Track persistent EADDRNOTAVAIL across all connections.
+	 * If the source address stored in msgr->inst.addr is no longer
+	 * valid (e.g., it was a transient CNI pod address that has been
+	 * removed), all connections will fail with EADDRNOTAVAIL at
+	 * ip6_dst_lookup_flow() before even sending a SYN.
+	 *
+	 * After ADDRNOTAVAIL_RESET_THRESHOLD consecutive failures,
+	 * reset inst.addr to blank so that process_hello() will
+	 * re-learn the source address from the next successful
+	 * monitor connection. The nonce is preserved.
+	 */
+	if (addr_issue) {
+		int count = atomic_inc_return(&con->msgr->addr_notavail_count);
+
+		if (count == ADDRNOTAVAIL_RESET_THRESHOLD) {
+			struct ceph_entity_addr *my_addr =
+				&con->msgr->inst.addr;
+
+			pr_warn("libceph: %d consecutive EADDRNOTAVAIL errors, resetting source address %s (will re-learn from monitor)\n",
+				count, ceph_pr_addr(my_addr));
+
+			/*
+			 * Zero out the address portion of in_addr but
+			 * preserve ss_family, nonce, and type so the
+			 * client identity is maintained and debug output
+			 * remains readable. process_hello() checks
+			 * ceph_addr_is_blank() and will fill in the new
+			 * address from the monitor's addr_for_me response.
+			 *
+			 * We preserve ss_family so that ceph_pr_addr()
+			 * shows e.g. "[::]:0" instead of
+			 * "(unknown sockaddr family 0)".
+			 */
+			{
+				sa_family_t family =
+					get_unaligned(&my_addr->in_addr.ss_family);
+				memset(&my_addr->in_addr, 0,
+				       sizeof(my_addr->in_addr));
+				put_unaligned(family,
+					      &my_addr->in_addr.ss_family);
+			}
+			ceph_encode_my_addr(con->msgr);
+		}
+	}
+
 	WARN_ON(con->state == CEPH_CON_S_STANDBY ||
 		con->state == CEPH_CON_S_CLOSED);
 
@@ -1740,6 +1790,7 @@ void ceph_messenger_init(struct ceph_messenger *msgr,
 	ceph_encode_my_addr(msgr);
 
 	atomic_set(&msgr->stopping, 0);
+	atomic_set(&msgr->addr_notavail_count, 0);
 	write_pnet(&msgr->net, get_net(current->nsproxy->net_ns));
 
 	dout("%s %p\n", __func__, msgr);
diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c
index 0cb61c76b9b87..4f3868f296c06 100644
--- a/net/ceph/messenger_v1.c
+++ b/net/ceph/messenger_v1.c
@@ -736,6 +736,13 @@ static int process_banner(struct ceph_connection *con)
 		ceph_encode_my_addr(con->msgr);
 		dout("process_banner learned my addr is %s\n",
 		     ceph_pr_addr(my_addr));
+
+		if (atomic_read(&con->msgr->addr_notavail_count) > 0) {
+			pr_info("libceph: re-learned source address %s from peer %s\n",
+				ceph_pr_addr(my_addr),
+				ceph_pr_addr(&con->peer_addr));
+			atomic_set(&con->msgr->addr_notavail_count, 0);
+		}
 	}
 
 	return 0;
diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
index bd608ffa06279..12ad9f571dcca 100644
--- a/net/ceph/messenger_v2.c
+++ b/net/ceph/messenger_v2.c
@@ -2260,6 +2260,18 @@ static int process_hello(struct ceph_connection *con, void *p, void *end)
 		dout("%s con %p set my addr %s, as seen by peer %s\n",
 		     __func__, con, ceph_pr_addr(my_addr),
 		     ceph_pr_addr(&con->peer_addr));
+
+		/*
+		 * If we re-learned the address after a reset due to
+		 * persistent EADDRNOTAVAIL, log it and clear the
+		 * failure counter.
+		 */
+		if (atomic_read(&con->msgr->addr_notavail_count) > 0) {
+			pr_info("libceph: re-learned source address %s from monitor %s\n",
+				ceph_pr_addr(my_addr),
+				ceph_pr_addr(&con->peer_addr));
+			atomic_set(&con->msgr->addr_notavail_count, 0);
+		}
 	} else {
 		dout("%s con %p my addr already set %s\n",
 		     __func__, con, ceph_pr_addr(my_addr));
-- 
2.53.0

[PATCH v1 12/13] libceph: force monitor reconnect on persistent EADDRNOTAVAIL

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When the kernel CephFS client experiences persistent EADDRNOTAVAIL
errors (e.g., because the original source address was a transient
CNI pod address that no longer exists), the monitor client may get
stuck retrying the same monitor indefinitely while in hunting mode.
The mon_fault() handler currently ignores faults when already
hunting, assuming delayed_work() will handle the retry. However,
delayed_work() simply calls reopen_session() which may pick the
same monitor again, creating an infinite loop of failed connection
attempts to the same target.

Additionally, when EADDRNOTAVAIL is persistent across all monitors,
the hunt_mult backoff grows exponentially, causing increasingly
long delays between reconnection attempts. Once the network issue
resolves (e.g., route cache expires, new address becomes available),
the client may take minutes to recover due to the accumulated
backoff.

Fix this by modifying mon_fault() to force a reopen_session() even
when already hunting, if the messenger's addr_notavail_count
indicates persistent address failures. This ensures the client
tries a different monitor on each fault rather than waiting for
the delayed_work timer. Also reset hunt_mult to 1 when forcing
a reconnect due to EADDRNOTAVAIL, so that once the network issue
resolves, the client recovers quickly without accumulated backoff
delays.

Also add a safety check in delayed_work(): if addr_notavail_count
exceeds the reset threshold and we're hunting, reset hunt_mult to
prevent accumulated backoff from delaying recovery.

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 net/ceph/mon_client.c | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c
index ab66b599ac479..6e3d314fbf2b2 100644
--- a/net/ceph/mon_client.c
+++ b/net/ceph/mon_client.c
@@ -1084,6 +1084,7 @@ static void delayed_work(struct work_struct *work)
 {
 	struct ceph_mon_client *monc =
 		container_of(work, struct ceph_mon_client, delayed_work.work);
+	int notavail_count;
 
 	mutex_lock(&monc->mutex);
 	dout("%s mon%d\n", __func__, monc->cur_mon);
@@ -1094,6 +1095,22 @@ static void delayed_work(struct work_struct *work)
 	if (monc->hunting) {
 		dout("%s continuing hunt\n", __func__);
 		reopen_session(monc);
+
+		/*
+		 * If we're hunting and EADDRNOTAVAIL has been persistent,
+		 * reset the backoff multiplier so we recover quickly once
+		 * the network issue resolves. Without this, hunt_mult can
+		 * grow large during extended EADDRNOTAVAIL periods, causing
+		 * the client to take minutes to reconnect even after the
+		 * underlying issue is fixed.
+		 */
+		notavail_count =
+			atomic_read(&monc->client->msgr.addr_notavail_count);
+		if (notavail_count >= ADDRNOTAVAIL_RESET_THRESHOLD) {
+			dout("%s addr_notavail_count %d, resetting hunt_mult\n",
+			     __func__, notavail_count);
+			monc->hunt_mult = 1;
+		}
 	} else {
 		int is_auth = ceph_auth_is_authenticated(monc->auth);
 
@@ -1554,6 +1571,7 @@ static struct ceph_msg *mon_alloc_msg(struct ceph_connection *con,
 static void mon_fault(struct ceph_connection *con)
 {
 	struct ceph_mon_client *monc = con->private;
+	int notavail_count;
 
 	mutex_lock(&monc->mutex);
 	dout("%s mon%d\n", __func__, monc->cur_mon);
@@ -1563,7 +1581,26 @@ static void mon_fault(struct ceph_connection *con)
 			reopen_session(monc);
 			__schedule_delayed(monc);
 		} else {
-			dout("%s already hunting\n", __func__);
+			/*
+			 * Already hunting. Normally we just wait for
+			 * delayed_work() to retry. But if EADDRNOTAVAIL
+			 * is persistent, force an immediate reconnect to
+			 * a different monitor. This avoids getting stuck
+			 * retrying the same monitor that keeps failing.
+			 * Also reset hunt_mult so we don't accumulate
+			 * excessive backoff during the outage.
+			 */
+			notavail_count =
+				atomic_read(&con->msgr->addr_notavail_count);
+			if (notavail_count > 0) {
+				dout("%s addr_notavail %d, forcing reopen\n",
+				     __func__, notavail_count);
+				monc->hunt_mult = 1;
+				reopen_session(monc);
+				__schedule_delayed(monc);
+			} else {
+				dout("%s already hunting\n", __func__);
+			}
 		}
 	}
 	mutex_unlock(&monc->mutex);
-- 
2.53.0

[PATCH v1 13/13] libceph: force host network namespace for kernel CephFS mounts

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

In containerized environments (e.g., Rook-Ceph CSI with
forcecephkernelclient=true), the mount() syscall
for kernel CephFS may be invoked from a pod's network namespace
instead of the host namespace. This happens despite the CSI node
plugin (csi-cephfsplugin) running with hostNetwork: true, due to
race conditions during kubelet restart or pod scheduling.

ceph_messenger_init() captures current->nsproxy->net_ns at mount
time and uses it for all subsequent socket operations. When a pod
NS is captured, all kernel ceph sockets (mon, mds, osd) are
created in that namespace, which typically lacks routes to the
Ceph monitors (e.g., fd04:: ClusterIP addresses).
This causes permanent EADDRNOTAVAIL (-99) on every connection
attempt at ip6_dst_lookup_flow(), with no possibility of recovery
short of force-unmount and remount from the correct namespace.

Root cause confirmed via kprobe tracing on ip6_dst_lookup_flow:
the net pointer passed to the routing lookup was the pod's
net_ns (0xff367a0125dd5780) instead of init_net
(0xffffffffbda76940). The pod NS had no route for fd04::/64
(monitor ClusterIP range), while userspace python connect() from
the same host succeeded because it ran in host NS.

Fix this by always using init_net (the host network namespace)
in ceph_messenger_init(). The kernel CephFS client inherently
requires host-level network access to reach Ceph monitors, OSDs,
and MDS daemons. Using the caller's namespace was inherited from
generic socket patterns but is incorrect for a kernel filesystem
client that must survive beyond the lifetime of the mounting
process and its network namespace.

A warning is logged when a mount from a non-init namespace is
detected, to aid debugging.

Observed in production (kernel 6.12.0-1-rt-amd64, Ceph Reef
18.2.5, IPv6-only cluster, ceph-csi v3.13.1):
  - Fresh boot of compute-0, ceph-csi mounts CephFS via kernel
  - All monitor connections fail with EADDRNOTAVAIL immediately
  - kprobe confirms wrong net_ns in ip6_dst_lookup_flow
  - Workaround: umount -l + systemctl restart kubelet
  - After restart: mount captures host NS, works immediately

Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 net/ceph/messenger.c | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 8165e6a8fe092..a2e8ea6d339c9 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -1791,7 +1791,32 @@ void ceph_messenger_init(struct ceph_messenger *msgr,
 
 	atomic_set(&msgr->stopping, 0);
 	atomic_set(&msgr->addr_notavail_count, 0);
-	write_pnet(&msgr->net, get_net(current->nsproxy->net_ns));
+
+	/*
+	 * Use the initial (host) network namespace instead of the
+	 * caller's current namespace. In containerized environments
+	 * (e.g., Rook-Ceph CSI with forcecephkernelclient=true), the
+	 * mount() syscall may be invoked from a pod's network namespace
+	 * even when the CSI plugin runs with hostNetwork: true (race
+	 * conditions during kubelet restart, pod scheduling, etc.).
+	 *
+	 * If the pod NS is captured here, all kernel ceph sockets will
+	 * be created in that NS, which typically lacks routes to the
+	 * Ceph monitors (e.g., fd04:: ClusterIP addresses). This causes
+	 * permanent EADDRNOTAVAIL on every connection attempt with no
+	 * possibility of recovery short of force-unmount + remount.
+	 *
+	 * The kernel CephFS client always needs host-level network
+	 * access to reach Ceph monitors, OSDs, and MDS daemons, so
+	 * using init_net is the correct choice. The previous behavior
+	 * of capturing current->nsproxy->net_ns was inherited from
+	 * generic socket code but is wrong for a kernel filesystem
+	 * client that must survive beyond the lifetime of the mounting
+	 * process's network namespace.
+	 */
+	if (current->nsproxy->net_ns != &init_net)
+		pr_warn("libceph: mount from non-init network namespace detected, using host namespace instead\n");
+	write_pnet(&msgr->net, get_net(&init_net));
 
 	dout("%s %p\n", __func__, msgr);
 }
-- 
2.53.0

Re: [PATCH v1 01/13] libceph: handle EADDRNOTAVAIL more gracefully

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When connecting to Ceph monitors/OSDs, kernel_connect() may return
> -EADDRNOTAVAIL if the source address is unavailable. This occurs
> during:
> - IPv6 Duplicate Address Detection (DAD)
> - IPv4/IPv6 interface state changes (link up/down events)
> - Address removal or reconfiguration on the interface
> - Network namespace transitions in containerized environments
> - CNI reconfigurations during containerized rolling upgrades
> Currently, libceph treats EADDRNOTAVAIL like any other connection error
> and enters exponential backoff (BASE_DELAY_INTERVAL 250ms doubling up
> to MAX_DELAY_INTERVAL 15s). Additionally, the monitor client has its
> own hunt-level backoff (CEPH_MONC_HUNT_INTERVAL 3s * hunt_mult, where
> hunt_mult doubles up to 10x = 30s max). These two backoff mechanisms
> compound: at steady state each monitor gets ~30 seconds of attempts
> with connection-level delays up to 15s, and the round-trip through
> all monitors takes ~60 seconds.
> In production testing (6.12.0-1-rt-amd64, Dell PowerEdge
> R720, IPv6-only Ceph cluster with 2 monitors), the EADDRNOTAVAIL
> condition persisted for ~36 minutes during a rolling upgrade:
>   13:20:52 - mon0 session lost, hunting begins, first error -99
>   13:57:03 - mon0 session finally re-established
>   ~470 failed connect attempts across both monitors
>   sync task blocked for 983+ seconds, triggering hung task warnings:
>     "INFO: task sync:514917 blocked for more than 122 seconds"
>     ...repeated at 245s, 368s, 491s, 614s, 737s, 860s, 983s
> The duration of EADDRNOTAVAIL varies by environment: it can be brief
> (simple DAD, 1-2s) or prolonged (complex network reconfiguration
> during rolling upgrades, minutes). In both cases, the key issue is
> that exponential backoff up to 15s wastes time once the address
> becomes available -- the client may sit idle for up to 15 seconds
> before attempting to reconnect.
> This patch bypasses the exponential backoff for EADDRNOTAVAIL by
> using a fixed short retry interval (ADDRNOTAVAIL_DELAY, HZ/10 =
> 100ms). This ensures reconnection happens within 100ms of the address

As far as I know, HZ depends on frequency. So, HZ/10 is not necessary 100ms. Am
I right here?

> becoming available, rather than waiting up to 15 seconds.
> Implementation:
> - Detect EADDRNOTAVAIL in ceph_tcp_connect() for both IPv4 and IPv6
> - Signal the condition to con_fault() via an addr_notavail flag
>   (per-protocol: v1 and v2)
> - In con_fault(), use ADDRNOTAVAIL_DELAY instead of exponential
>   backoff when the flag is set
> - Clear the flag on successful connection and when reopening
> - Use pr_warn_ratelimited() instead of pr_err() for this case
> The fast retry is appropriate because each attempt is inexpensive
> (kernel_connect() fails immediately when the address is unavailable)
> and quick recovery is critical for storage availability.
> Fixes: 60bf8bf8815e ("libceph: fix msgr backoff")
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  include/linux/ceph/messenger.h | 11 +++++++
>  net/ceph/messenger.c           | 55 ++++++++++++++++++++++++++++++++--
>  2 files changed, 63 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
> index 1717cc57cdacd..730a754353aed 100644
> --- a/include/linux/ceph/messenger.h
> +++ b/include/linux/ceph/messenger.h
> @@ -320,6 +320,13 @@ struct ceph_msg {
>  /* ceph connection fault delay defaults, for exponential backoff */
>  #define BASE_DELAY_INTERVAL	(HZ / 4)
>  #define MAX_DELAY_INTERVAL	(15 * HZ)
> +/*
> + * Shorter retry delay for EADDRNOTAVAIL. This error typically indicates
> + * a transient condition (IPv6 DAD in progress, address reconfiguration,
> + * temporary route issue) that resolves in 1-2 seconds. Fast retries
> + * allow quick recovery without exponential backoff delays.
> + */
> +#define ADDRNOTAVAIL_DELAY	(HZ / 10)

What's wrong with BASE_DELAY_INTERVAL? I don't see big difference between HZ/4
and HZ/10.

>  
>  struct ceph_connection_v1_info {
>  	struct kvec out_kvec[8],         /* sending header/footer data */
> @@ -360,6 +367,8 @@ struct ceph_connection_v1_info {
>  	u32 connect_seq;      /* identify the most recent connection
>  				 attempt for this session */
>  	u32 peer_global_seq;  /* peer's global seq for this connection */
> +
> +	bool addr_notavail;  /* address not available (transient) */

You've introduced the same field for v1 and v2. But why you haven't used the
struct ceph_connection? In this case, you don't need to use
ceph_msgr2(from_msgr(con->msgr)) everywhere.

>  };
>  
>  #define CEPH_CRC_LEN			4
> @@ -430,6 +439,8 @@ struct ceph_connection_v2_info {
>  
>  	int con_mode;  /* CEPH_CON_MODE_* */
>  
> +	bool addr_notavail;  /* address not available (transient) */
> +
>  	void *conn_bufs[16];
>  	int conn_buf_cnt;
>  	int data_len_remain;
> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> index 9f6d860411cbd..c40c7c332e7f4 100644
> --- a/net/ceph/messenger.c
> +++ b/net/ceph/messenger.c
> @@ -466,8 +466,22 @@ int ceph_tcp_connect(struct ceph_connection *con)
>  		     ceph_pr_addr(&con->peer_addr),
>  		     sock->sk->sk_state);
>  	} else if (ret < 0) {
> -		pr_err("connect %s error %d\n",
> -		       ceph_pr_addr(&con->peer_addr), ret);
> +		if (ret == -EADDRNOTAVAIL) {
> +			/*
> +			 * Address not yet available - could be IPv6 DAD in
> +			 * progress, address reconfiguration, or temporary
> +			 * route issue. Use shorter delay.
> +			 */
> +			pr_warn_ratelimited("connect %s: address not available (DAD/route issue?), will retry\n",
> +					    ceph_pr_addr(&con->peer_addr));
> +			if (ceph_msgr2(from_msgr(con->msgr)))
> +				con->v2.addr_notavail = true;
> +			else
> +				con->v1.addr_notavail = true;
> +		} else {
> +			pr_err("connect %s error %d\n",
> +			       ceph_pr_addr(&con->peer_addr), ret);
> +		}
>  		sock_release(sock);
>  		return ret;
>  	}
> @@ -476,6 +490,13 @@ int ceph_tcp_connect(struct ceph_connection *con)
>  		tcp_sock_set_nodelay(sock->sk);
>  
>  	con->sock = sock;
> +
> +	/* Clear addr_notavail flag on successful connection */
> +	if (ceph_msgr2(from_msgr(con->msgr)))
> +		con->v2.addr_notavail = false;
> +	else
> +		con->v1.addr_notavail = false;
> +
>  	return 0;
>  }
>  
> @@ -609,6 +630,13 @@ void ceph_con_open(struct ceph_connection *con,
>  
>  	memcpy(&con->peer_addr, addr, sizeof(*addr));
>  	con->delay = 0;      /* reset backoff memory */
> +
> +	/* Clear addr_notavail flag when opening/reopening connection */
> +	if (ceph_msgr2(from_msgr(con->msgr)))
> +		con->v2.addr_notavail = false;
> +	else
> +		con->v1.addr_notavail = false;
> +
>  	mutex_unlock(&con->mutex);
>  	queue_con(con);
>  }
> @@ -1613,6 +1641,8 @@ static void ceph_con_workfn(struct work_struct *work)
>   */
>  static void con_fault(struct ceph_connection *con)
>  {
> +	bool addr_issue = false;

What's the point to introduce this local variable? Why don't use con-
>v2.addr_notavail where you are using addr_issue? Any particular reason for
this?

> +
>  	dout("fault %p state %d to peer %s\n",
>  	     con, con->state, ceph_pr_addr(&con->peer_addr));
>  
> @@ -1620,6 +1650,19 @@ static void con_fault(struct ceph_connection *con)
>  		ceph_pr_addr(&con->peer_addr), con->error_msg);
>  	con->error_msg = NULL;
>  
> +	/* Check and reset addr_notavail flag if set */
> +	if (ceph_msgr2(from_msgr(con->msgr))) {
> +		if (con->v2.addr_notavail) {
> +			addr_issue = true;
> +			con->v2.addr_notavail = false;
> +		}
> +	} else {
> +		if (con->v1.addr_notavail) {
> +			addr_issue = true;
> +			con->v1.addr_notavail = false;
> +		}
> +	}
> +
>  	WARN_ON(con->state == CEPH_CON_S_STANDBY ||
>  		con->state == CEPH_CON_S_CLOSED);
>  
> @@ -1644,7 +1687,13 @@ static void con_fault(struct ceph_connection *con)
>  	} else {
>  		/* retry after a delay. */
>  		con->state = CEPH_CON_S_PREOPEN;
> -		if (!con->delay) {
> +		if (addr_issue) {
> +			/*
> +			 * Address not available - use shorter delay as this
> +			 * is often a transient condition.
> +			 */
> +			con->delay = ADDRNOTAVAIL_DELAY;

So, the main point of introducing con->v2/1.addr_notavail is to set this delay.
I am not sure that there is big difference between HZ/4 and HZ/10. Do we really
need to change the delay here?

Thanks,
Slava.

> +		} else if (!con->delay) {
>  			con->delay = BASE_DELAY_INTERVAL;
>  		} else if (con->delay < MAX_DELAY_INTERVAL) {
>  			con->delay *= 2;

Re: [PATCH v1 02/13] ceph: add timeout protection to ceph_mdsc_sync() path

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When Ceph MDS becomes unreachable (e.g., due to IPv6 EADDRNOTAVAIL
> during DAD or network transitions), the sync syscall can block
> indefinitely in ceph_mdsc_sync(). The hung_task detector fires
> repeatedly (122s, 245s, 368s... up to 983+ seconds) with traces like:
>   INFO: task sync:12345 blocked for more than 122 seconds.
>   Call Trace:
>     ceph_mdsc_sync+0x4d6/0x5a0 [ceph]
>     ceph_sync_fs+0x31/0x130 [ceph]
>     iterate_supers+0x97/0x100
>     ksys_sync+0x32/0xb0
> Three functions in the MDS sync path use indefinite waits:
> 1. wait_caps_flush() uses wait_event() with no timeout
> 2. flush_mdlog_and_wait_mdsc_unsafe_requests() uses
>    wait_for_completion() with no timeout
> 3. ceph_mdsc_sync() returns void, cannot propagate errors
> This is particularly problematic in containerized environments with
> PREEMPT_RT kernels where Ceph storage pods undergo rolling updates
> and IPv6 network reconfigurations cause temporary MDS unavailability.
> Fix this by adding mount_timeout-based timeouts (default 60s) to the
> blocking waits, following the existing pattern used by wait_requests()
> and ceph_mdsc_close_sessions() in the same file:
> - wait_caps_flush(): use wait_event_timeout() with mount_timeout
> - flush_mdlog_and_wait_mdsc_unsafe_requests(): use
>   wait_for_completion_timeout() with mount_timeout
> - ceph_mdsc_sync(): change return type to int, propagate -ETIMEDOUT
> - ceph_sync_fs(): propagate error from ceph_mdsc_sync() to VFS
> On timeout, dirty caps and pending requests are NOT discarded - they
> remain in memory and are re-synced when MDS reconnects. The timeout
> simply unblocks the calling task. If mount_timeout is set to 0,
> ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, preserving the
> original infinite-wait behavior.
> Real-world impact: In production logs showing 'task sync blocked for
> more than 983 seconds', this patch limits the block to mount_timeout
> (60s default), returning -ETIMEDOUT to the VFS layer instead of
> hanging indefinitely.
> Fixes: 1b2ba3c5616e ("ceph: flush the mdlog for filesystem sync")
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/mds_client.c | 50 ++++++++++++++++++++++++++++++++++----------
>  fs/ceph/mds_client.h |  2 +-
>  fs/ceph/super.c      |  5 +++--
>  3 files changed, 43 insertions(+), 14 deletions(-)
> 
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index df89d45f33a1f..37899464101f7 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -2296,17 +2296,26 @@ static int check_caps_flush(struct ceph_mds_client *mdsc,
>   *
>   * returns true if we've flushed through want_flush_tid
>   */
> -static void wait_caps_flush(struct ceph_mds_client *mdsc,
> -			    u64 want_flush_tid)
> +static int wait_caps_flush(struct ceph_mds_client *mdsc,
> +			   u64 want_flush_tid)
>  {
>  	struct ceph_client *cl = mdsc->fsc->client;
> +	struct ceph_options *opts = mdsc->fsc->client->options;
> +	long ret;
>  
>  	doutc(cl, "want %llu\n", want_flush_tid);
>  
> -	wait_event(mdsc->cap_flushing_wq,
> -		   check_caps_flush(mdsc, want_flush_tid));
> +	ret = wait_event_timeout(mdsc->cap_flushing_wq,
> +				 check_caps_flush(mdsc, want_flush_tid),
> +				 ceph_timeout_jiffies(opts->mount_timeout));

Technically speaking, opts->mount_timeout is configurable option and it can be
defined long enough. Finally, you could see the same issue even with your
solution. Maybe, we need to have some check of opts->mount_timeout?

> +	if (!ret) {
> +		pr_warn_client(cl, "cap flush timeout waiting for tid %llu\n",
> +			       want_flush_tid);

Now we will have these messages instead of "process has been blocked" messages.
:) Do we really need to inform about this? Maybe, debug message here?

> +		return -ETIMEDOUT;
> +	}
>  
>  	doutc(cl, "ok, flushed thru %llu\n", want_flush_tid);
> +	return 0;
>  }
>  
>  /*
> @@ -5838,13 +5847,15 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)
>  /*
>   * flush the mdlog and wait for all write mds requests to flush.
>   */
> -static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
> -						 u64 want_tid)
> +static int flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *mdsc,
> +						      u64 want_tid)
>  {
>  	struct ceph_client *cl = mdsc->fsc->client;
> +	struct ceph_options *opts = mdsc->fsc->client->options;
>  	struct ceph_mds_request *req = NULL, *nextreq;
>  	struct ceph_mds_session *last_session = NULL;
>  	struct rb_node *n;
> +	unsigned long left;
>  
>  	mutex_lock(&mdsc->mutex);
>  	doutc(cl, "want %lld\n", want_tid);
> @@ -5883,7 +5894,19 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
>  			}
>  			doutc(cl, "wait on %llu (want %llu)\n",
>  			      req->r_tid, want_tid);
> -			wait_for_completion(&req->r_safe_completion);
> +			left = wait_for_completion_timeout(
> +					&req->r_safe_completion,
> +					ceph_timeout_jiffies(opts->mount_timeout));
> +			if (!left) {
> +				pr_warn_client(cl,
> +					       "flush mdlog request tid %llu timed out\n",
> +					       req->r_tid);
> +				ceph_mdsc_put_request(req);
> +				if (nextreq)
> +					ceph_mdsc_put_request(nextreq);
> +				ceph_put_mds_session(last_session);
> +				return -ETIMEDOUT;

The same concerns here.

> +			}
>  
>  			mutex_lock(&mdsc->mutex);
>  			ceph_mdsc_put_request(req);
> @@ -5901,15 +5924,17 @@ static void flush_mdlog_and_wait_mdsc_unsafe_requests(struct ceph_mds_client *md
>  	mutex_unlock(&mdsc->mutex);
>  	ceph_put_mds_session(last_session);
>  	doutc(cl, "done\n");
> +	return 0;
>  }
>  
> -void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
> +int ceph_mdsc_sync(struct ceph_mds_client *mdsc)
>  {
>  	struct ceph_client *cl = mdsc->fsc->client;
>  	u64 want_tid, want_flush;
> +	int ret;
>  
>  	if (READ_ONCE(mdsc->fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN)
> -		return;
> +		return -EIO;

Why -EIO here? As far as I can follow, we will retry the sync operation. Am I
correct? So, it's not I/O failure yet.

Thanks,
Slava.

>  
>  	doutc(cl, "sync\n");
>  	mutex_lock(&mdsc->mutex);
> @@ -5930,8 +5955,11 @@ void ceph_mdsc_sync(struct ceph_mds_client *mdsc)
>  
>  	doutc(cl, "sync want tid %lld flush_seq %lld\n", want_tid, want_flush);
>  
> -	flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
> -	wait_caps_flush(mdsc, want_flush);
> +	ret = flush_mdlog_and_wait_mdsc_unsafe_requests(mdsc, want_tid);
> +	if (ret)
> +		return ret;
> +
> +	return wait_caps_flush(mdsc, want_flush);
>  }
>  
>  /*
> diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
> index 0a602080d8ef6..695c5a9c94026 100644
> --- a/fs/ceph/mds_client.h
> +++ b/fs/ceph/mds_client.h
> @@ -564,7 +564,7 @@ extern void ceph_mdsc_close_sessions(struct ceph_mds_client *mdsc);
>  extern void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc);
>  extern void ceph_mdsc_destroy(struct ceph_fs_client *fsc);
>  
> -extern void ceph_mdsc_sync(struct ceph_mds_client *mdsc);
> +extern int ceph_mdsc_sync(struct ceph_mds_client *mdsc);
>  
>  extern void ceph_invalidate_dir_request(struct ceph_mds_request *req);
>  extern int ceph_alloc_readdir_reply_buffer(struct ceph_mds_request *req,
> diff --git a/fs/ceph/super.c b/fs/ceph/super.c
> index b61074b377ac5..b52960402d68e 100644
> --- a/fs/ceph/super.c
> +++ b/fs/ceph/super.c
> @@ -122,6 +122,7 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
>  {
>  	struct ceph_fs_client *fsc = ceph_sb_to_fs_client(sb);
>  	struct ceph_client *cl = fsc->client;
> +	int ret;
>  
>  	if (!wait) {
>  		doutc(cl, "(non-blocking)\n");
> @@ -133,9 +134,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
>  
>  	doutc(cl, "(blocking)\n");
>  	ceph_osdc_sync(&fsc->client->osdc);
> -	ceph_mdsc_sync(fsc->mdsc);
> +	ret = ceph_mdsc_sync(fsc->mdsc);
>  	doutc(cl, "(blocking) done\n");
> -	return 0;
> +	return ret;
>  }
>  
>  /*

Re: [PATCH v1 03/13] ceph: add timeout protection to ceph_osdc_sync() path

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When a Ceph OSD becomes unreachable (e.g., due to IPv6 EADDRNOTAVAIL
> during DAD or network transitions), the sync syscall can block
> indefinitely in ceph_osdc_sync(). This function iterates over all
> in-flight write requests and calls wait_for_completion() with no
> timeout on each one. The hung_task detector fires repeatedly with
> stack traces showing:
>   ceph_osdc_sync [libceph]
>   ceph_sync_fs [ceph]
>   iterate_supers
>   ksys_sync
> Since ceph_osdc_sync() is called before ceph_mdsc_sync() in
> ceph_sync_fs(), an OSD hang prevents the MDS timeout protection
> from commit e789e5252fda ("ceph: add timeout protection to
> ceph_mdsc_sync() path") from ever being reached.
> This is particularly problematic in containerized environments with
> PREEMPT_RT kernels where Ceph storage pods undergo rolling updates
> and IPv6 network reconfigurations cause temporary OSD unavailability.
> Fix this by adding mount_timeout-based timeout to the blocking wait,
> following the existing pattern used by wait_request_timeout() in the
> same file:
> - ceph_osdc_sync(): use wait_for_completion_timeout() with
>   mount_timeout instead of indefinite wait_for_completion()
> - Change return type from void to int, return -ETIMEDOUT on timeout
> - ceph_sync_fs(): propagate OSD sync error, short-circuit before
>   MDS sync on failure
> On timeout, pending OSD requests are NOT cancelled - they remain
> in-flight and complete when the OSD reconnects. The timeout simply
> unblocks the calling task. If mount_timeout is set to 0,
> ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, preserving the
> original infinite-wait behavior.
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/super.c                 |  4 +++-
>  include/linux/ceph/osd_client.h |  2 +-
>  net/ceph/osd_client.c           | 15 +++++++++++++--
>  3 files changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/ceph/super.c b/fs/ceph/super.c
> index b52960402d68e..6f4ee457c1b52 100644
> --- a/fs/ceph/super.c
> +++ b/fs/ceph/super.c
> @@ -133,7 +133,9 @@ static int ceph_sync_fs(struct super_block *sb, int wait)
>  	}
>  
>  	doutc(cl, "(blocking)\n");
> -	ceph_osdc_sync(&fsc->client->osdc);
> +	ret = ceph_osdc_sync(&fsc->client->osdc);
> +	if (ret)
> +		return ret;
>  	ret = ceph_mdsc_sync(fsc->mdsc);
>  	doutc(cl, "(blocking) done\n");
>  	return ret;
> diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
> index d7941478158cd..871827e2dd983 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -587,7 +587,7 @@ void ceph_osdc_start_request(struct ceph_osd_client *osdc,
>  extern void ceph_osdc_cancel_request(struct ceph_osd_request *req);
>  extern int ceph_osdc_wait_request(struct ceph_osd_client *osdc,
>  				  struct ceph_osd_request *req);
> -extern void ceph_osdc_sync(struct ceph_osd_client *osdc);
> +extern int ceph_osdc_sync(struct ceph_osd_client *osdc);
>  
>  extern void ceph_osdc_flush_notifies(struct ceph_osd_client *osdc);
>  void ceph_osdc_maybe_request_map(struct ceph_osd_client *osdc);
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index abac770bc0b4c..7d5e4a078fb10 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -4734,10 +4734,13 @@ EXPORT_SYMBOL(ceph_osdc_wait_request);
>  /*
>   * sync - wait for all in-flight requests to flush.  avoid starvation.
>   */
> -void ceph_osdc_sync(struct ceph_osd_client *osdc)
> +int ceph_osdc_sync(struct ceph_osd_client *osdc)
>  {
> +	struct ceph_options *opts = osdc->client->options;
> +	unsigned long timeout = ceph_timeout_jiffies(opts->mount_timeout);

The opts->mount_timeout could be configured unreasonably.

>  	struct rb_node *n, *p;
>  	u64 last_tid = atomic64_read(&osdc->last_tid);
> +	unsigned long left;
>  
>  again:
>  	down_read(&osdc->lock);
> @@ -4760,7 +4763,14 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
>  			up_read(&osdc->lock);
>  			dout("%s waiting on req %p tid %llu last_tid %llu\n",
>  			     __func__, req, req->r_tid, last_tid);
> -			wait_for_completion(&req->r_completion);
> +			left = wait_for_completion_timeout(&req->r_completion,
> +							   timeout);
> +			if (!left) {
> +				pr_warn("ceph: osd sync request tid %llu timed out\n",
> +					req->r_tid);

I am not sure about necessity to send this message into syslog. Maybe, debug
output here? My point here, if we simply postpone some operation that will be
executed lately, then should we inform about elapsed timeout. Let's imagine that
opts->mount_timeout will be really short, then we will have bunch of messages in
the system log.

Thanks,
Slava.

> +				ceph_osdc_put_request(req);
> +				return -ETIMEDOUT;
> +			}
>  			ceph_osdc_put_request(req);
>  			goto again;
>  		}
> @@ -4770,6 +4780,7 @@ void ceph_osdc_sync(struct ceph_osd_client *osdc)
>  
>  	up_read(&osdc->lock);
>  	dout("%s done last_tid %llu\n", __func__, last_tid);
> +	return 0;
>  }
>  EXPORT_SYMBOL(ceph_osdc_sync);
>  

Re: [PATCH v1 04/13] ceph: fix race condition in cleanup_session_requests()

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When an MDS session is closed or reset, cleanup_session_requests()
> only unregisters requests that are on the session's s_unsafe list.
> However, requests are only added to s_unsafe after receiving an
> "unsafe" reply from the MDS.
> This creates a race condition: if a write request has been sent
> but the MDS becomes unavailable before sending the unsafe reply,
> the request will:
>   - Have r_session set (points to the failed session)
>   - Be in the request_tree
>   - NOT be on s_unsafe list
>   - Never have r_safe_completion signaled
> Meanwhile, flush_mdlog_and_wait_mdsc_unsafe_requests() iterates
> the request_tree looking for write requests with r_session set,
> and waits on r_safe_completion for each one. Since the request
> is not on s_unsafe, cleanup_session_requests() won't unregister
> it, and the completion is never signaled - causing an indefinite
> hang.
> This was observed in production when running xfstests generic/013
> in a loop, with stack traces showing:
>   INFO: task fsstress:14466 blocked for more than 122 seconds.
>   Call Trace:
>     wait_for_completion+0x14a/0x340
>     ceph_mdsc_sync+0x4b4/0xe80
>     ceph_sync_fs+0xa0/0x4c0
>     sync_filesystem+0x182/0x240
> Fix this by extending cleanup_session_requests() to also unregister
> requests that:
>   - Belong to the closing session (r_session->s_mds matches)
>   - Have NOT received an unsafe reply (CEPH_MDS_R_GOT_UNSAFE not set)
>   - Have NOT received a safe reply (CEPH_MDS_R_GOT_SAFE not set)
> These are requests that were in-flight when the session failed and
> will never complete. Unregistering them signals r_safe_completion,
> unblocking any waiters.
> Requests that received an unsafe reply but not yet a safe reply
> are already on s_unsafe and handled by the existing code. For
> these, we preserve the original behavior of resetting r_attempts
> to allow re-sending when the session reconnects.
> Fixes: e3ec8d689cf4 ("ceph: clean up unsafe requests when reconnecting is denied")
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/mds_client.c | 24 +++++++++++++++++++++---
>  1 file changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index 37899464101f7..45abddd7f317e 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -1792,6 +1792,8 @@ static void cleanup_session_requests(struct ceph_mds_client *mdsc,
>  
>  	doutc(cl, "mds%d\n", session->s_mds);
>  	mutex_lock(&mdsc->mutex);
> +
> +	/* First, handle requests on the unsafe list */
>  	while (!list_empty(&session->s_unsafe)) {
>  		req = list_first_entry(&session->s_unsafe,
>  				       struct ceph_mds_request, r_unsafe_item);
> @@ -1803,14 +1805,30 @@ static void cleanup_session_requests(struct ceph_mds_client *mdsc,
>  			mapping_set_error(req->r_unsafe_dir->i_mapping, -EIO);
>  		__unregister_request(mdsc, req);
>  	}
> -	/* zero r_attempts, so kick_requests() will re-send requests */
> +
> +	/*
> +	 * Iterate through all pending requests for this session.
> +	 * Requests that haven't received an unsafe reply yet will never
> +	 * complete on this session - unregister them to signal waiters.
> +	 * Requests that got unsafe but not safe are handled above via
> +	 * s_unsafe list; for any remaining, reset r_attempts to allow
> +	 * re-sending when session reconnects.
> +	 */
>  	p = rb_first(&mdsc->request_tree);
>  	while (p) {
>  		req = rb_entry(p, struct ceph_mds_request, r_node);
>  		p = rb_next(p);
>  		if (req->r_session &&
> -		    req->r_session->s_mds == session->s_mds)
> -			req->r_attempts = 0;
> +		    req->r_session->s_mds == session->s_mds) {
> +			if (!test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags) &&
> +			    !test_bit(CEPH_MDS_R_GOT_SAFE, &req->r_req_flags)) {
> +				doutc(cl, " dropping pending request %llu\n",
> +				      req->r_tid);
> +				__unregister_request(mdsc, req);
> +			} else {
> +				req->r_attempts = 0;
> +			}
> +		}
>  	}
>  	mutex_unlock(&mdsc->mutex);
>  }

Nice fix.

Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>

Thanks,
Slava.

Re: [PATCH v1 05/13] ceph: add timeout protection to ceph_lock_wait_for_completion()

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When a file lock operation is interrupted and an unlock request is
> sent to cancel it, ceph_lock_wait_for_completion() waits indefinitely
> for r_safe_completion using wait_for_completion_killable().
> If the MDS becomes unreachable after the unlock request is sent,
> this wait will block indefinitely, causing hung task warnings:
>   INFO: task flock:12345 blocked for more than 122 seconds.
>   Call Trace:
>     wait_for_completion_killable+0x...
>     ceph_lock_wait_for_completion+0x...
>     ceph_flock+0x...
> This is similar to the issue fixed in ceph_mdsc_sync() where
> indefinite waits on r_safe_completion can hang when MDS is
> unavailable.
> Fix this by using wait_for_completion_killable_timeout() with
> mount_timeout instead of the indefinite wait. On timeout, return
> -ETIMEDOUT to the caller. The lock state remains consistent because:
> 1. If the unlock succeeded on MDS, the lock is released
> 2. If the unlock didn't reach MDS, the original lock request
>    was already aborted (CEPH_MDS_R_ABORTED set), so MDS will
>    clean it up on reconnect
> This follows the same timeout pattern used throughout the ceph
> client for MDS operations.
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/locks.c | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
> index ebf4ac0055ddc..55dd99460b81a 100644
> --- a/fs/ceph/locks.c
> +++ b/fs/ceph/locks.c
> @@ -160,6 +160,8 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
>                                           struct ceph_mds_request *req)
>  {
>  	struct ceph_client *cl = mdsc->fsc->client;
> +	struct ceph_options *opts = mdsc->fsc->client->options;
> +	unsigned long timeout = ceph_timeout_jiffies(opts->mount_timeout);

The opts->mount_timeout could be configured unreasonably. Should we do something
about it?

>  	struct ceph_mds_request *intr_req;
>  	struct inode *inode = req->r_inode;
>  	int err, lock_type;
> @@ -221,7 +223,17 @@ static int ceph_lock_wait_for_completion(struct ceph_mds_client *mdsc,
>  	if (err && err != -ERESTARTSYS)
>  		return err;
>  
> -	wait_for_completion_killable(&req->r_safe_completion);
> +	err = wait_for_completion_killable_timeout(&req->r_safe_completion,
> +						   timeout);
> +	if (err == -ERESTARTSYS) {

Interesting... You didn't check this in other patches. Why? :)

> +		/* Interrupted again, just return the error */
> +		return err;
> +	}
> +	if (err == 0) {
> +		pr_warn_client(cl, "lock request tid %llu safe completion timed out\n",
> +			       req->r_tid);

The same concern about sending warning into system log here.

> +		return -ETIMEDOUT;
> +	}

Maybe, some style cleanup here:

if (err == -ERESTARTSYS) {
  <logic_1>
} else if (err == 0) {
  <logic_2>
}

Thanks,
Slava.

>  	return 0;
>  }
>  

Re: [PATCH v1 06/13] ceph: set default timeout for MDS requests

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> MDS requests created via ceph_mdsc_create_request() have r_timeout
> initialized to 0 (from kmem_cache_zalloc). When r_timeout is 0,
> ceph_timeout_jiffies() returns MAX_SCHEDULE_TIMEOUT, causing
> ceph_mdsc_wait_request() to wait indefinitely.
> 
> This causes hung task warnings when MDS becomes unavailable during
> operations like setattr or truncate:
> 
>   INFO: task dd:12345 blocked for more than 122 seconds.
>   Call Trace:
>     ceph_mdsc_wait_request+0x...
>     ceph_mdsc_do_request+0x...
>     __ceph_setattr+0x...
> 
> Only the mount path in super.c explicitly sets r_timeout to
> mount_timeout. All other MDS requests (setattr, lookup, mkdir,
> etc.) use the default 0 value, making them wait forever.
> 
> Fix this by initializing r_timeout to mount_timeout in
> ceph_mdsc_create_request(). This ensures all MDS requests have
> a reasonable timeout and will fail with -ETIMEDOUT rather than
> hanging indefinitely.
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/mds_client.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index 45abddd7f317e..ac86225595b5f 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -2613,6 +2613,7 @@ ceph_mdsc_create_request(struct ceph_mds_client *mdsc, int op, int mode)
>  	mutex_init(&req->r_fill_mutex);
>  	req->r_mdsc = mdsc;
>  	req->r_started = jiffies;
> +	req->r_timeout = mdsc->fsc->client->options->mount_timeout;
>  	req->r_start_latency = ktime_get();
>  	req->r_resend_mds = -1;
>  	INIT_LIST_HEAD(&req->r_unsafe_dir_item);

I like this fix. Really nice. But, maybe, we should check the mount_timeout
value.

Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>

Thanks,
Slava.

Re: [PATCH v1 07/13] ceph: add timeout to caps wait in __ceph_get_caps()

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When waiting for caps in __ceph_get_caps(), the code uses
> wait_woken() with MAX_SCHEDULE_TIMEOUT, which can block
> indefinitely if the MDS is unavailable or slow to grant caps
> during reconnection.
> 
> This causes hung task warnings when MDS fails over:
> 
>   INFO: task dd:12345 blocked for more than 122 seconds.
>   Call Trace:
>     __ceph_get_caps+0x...
>     ceph_write_iter+0x...
> 
> During MDS failover, caps may be revoked or delayed while the
> client reconnects. Processes waiting for caps block indefinitely,
> also holding i_rwsem which blocks other I/O operations on the
> same inode, causing a cascade of blocked processes.
> 
> Fix this by using wait_woken() with mount_timeout instead of
> MAX_SCHEDULE_TIMEOUT. On timeout, return -ETIMEDOUT to allow
> the caller to handle the situation appropriately.
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/caps.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c
> index bed34fc11c919..c88e10a634e5c 100644
> --- a/fs/ceph/caps.c
> +++ b/fs/ceph/caps.c
> @@ -3055,7 +3055,10 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
>  {
>  	struct ceph_inode_info *ci = ceph_inode(inode);
>  	struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
> +	struct ceph_client *cl = fsc->client;
> +	unsigned long timeout = ceph_timeout_jiffies(cl->options->mount_timeout);

The same concern about timeout value. :)

>  	int ret, _got, flags;
> +	bool warned = false;

Technically speaking, you are trying to create pr_warn_once_client() here.
Probably, we need to prefer this one instead of pr_warn_ratelimited_client().
However, maybe, we need to have debug output instead. This warned variable
completely not necessary here.

Thanks,
Slava.

>  
>  	ret = ceph_pool_perm_check(inode, need);
>  	if (ret < 0)
> @@ -3104,7 +3107,18 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
>  					ret = -ERESTARTSYS;
>  					break;
>  				}
> -				wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
> +				if (!wait_woken(&wait, TASK_INTERRUPTIBLE, timeout)) {
> +					if (!warned) {
> +						pr_warn_ratelimited_client(cl,
> +							"%p %llx.%llx caps wait timed out (need %s want %s)\n",
> +							inode, ceph_vinop(inode),
> +							ceph_cap_string(need),
> +							ceph_cap_string(want));
> +						warned = true;
> +					}
> +					ret = -ETIMEDOUT;
> +					break;
> +				}
>  			}
>  
>  			remove_wait_queue(&ci->i_cap_wq, &wait);

Re: [PATCH v1 08/13] ceph: make ceph_start_io_write() killable

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When multiple processes write to the same file and one of them is
> blocked waiting for MDS/OSD response (e.g., during MDS failover),
> other processes block indefinitely on down_write(&inode->i_rwsem)
> in ceph_start_io_write().
> 
> This causes hung task warnings:
> 
>   INFO: task dd:12345 blocked for more than 122 seconds.
>   Call Trace:
>     ceph_start_io_write+0x...
>     ceph_write_iter+0x...
> 
> The i_rwsem is held by a process doing fsync/writeback that is
> waiting for MDS or OSD response. Other writers queue up on the
> rwsem and block indefinitely.
> 
> Fix this by using down_write_killable() instead of down_write().
> This allows blocked processes to be killed with SIGKILL, preventing
> indefinite hangs. The function now returns an error code that
> callers must check.
> 
> Update ceph_write_iter() to handle the new error return from
> ceph_start_io_write().
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/file.c | 9 +++++++--
>  fs/ceph/io.c   | 9 +++++++--
>  fs/ceph/io.h   | 2 +-
>  3 files changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 6587c2d5af1e0..01e4f31b1f2f3 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -2359,8 +2359,13 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
>  retry_snap:
>  	if (direct_lock)
>  		ceph_start_io_direct(inode);
> -	else
> -		ceph_start_io_write(inode);
> +	else {
> +		err = ceph_start_io_write(inode);
> +		if (err) {
> +			ceph_free_cap_flush(prealloc_cf);
> +			return err;
> +		}
> +	}
>  
>  	if (iocb->ki_flags & IOCB_APPEND) {
>  		err = ceph_do_getattr(inode, CEPH_STAT_CAP_SIZE, false);
> diff --git a/fs/ceph/io.c b/fs/ceph/io.c
> index c456509b31c3f..f9ac89ec1d6a1 100644
> --- a/fs/ceph/io.c
> +++ b/fs/ceph/io.c
> @@ -83,11 +83,16 @@ ceph_end_io_read(struct inode *inode)
>   * Declare that a buffered write operation is about to start, and ensure
>   * that we block all direct I/O.
>   */
> -void
> +int
>  ceph_start_io_write(struct inode *inode)
>  {
> -	down_write(&inode->i_rwsem);
> +	int ret;
> +
> +	ret = down_write_killable(&inode->i_rwsem);
> +	if (ret)
> +		return ret;
>  	ceph_block_o_direct(ceph_inode(inode), inode);
> +	return 0;
>  }

Which kernel version do you have? Because, we have this for v.7.0.0-rc3 [1]:

/**
 * ceph_start_io_write - declare the file is being used for buffered writes
 * @inode: file inode
 *
 * Declare that a buffered write operation is about to start, and ensure
 * that we block all direct I/O.
 */
int ceph_start_io_write(struct inode *inode)
{
	int err = down_write_killable(&inode->i_rwsem);
	if (!err)
		ceph_block_o_direct(ceph_inode(inode), inode);
	return err;
}

Thanks,
Slava.

>  
>  /**
> diff --git a/fs/ceph/io.h b/fs/ceph/io.h
> index fa594cd77348a..94ce176df9997 100644
> --- a/fs/ceph/io.h
> +++ b/fs/ceph/io.h
> @@ -4,7 +4,7 @@
>  
>  void ceph_start_io_read(struct inode *inode);
>  void ceph_end_io_read(struct inode *inode);
> -void ceph_start_io_write(struct inode *inode);
> +int ceph_start_io_write(struct inode *inode);
>  void ceph_end_io_write(struct inode *inode);
>  void ceph_start_io_direct(struct inode *inode);
>  void ceph_end_io_direct(struct inode *inode);

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/fs/ceph/io.c#L110

Re: [PATCH v1 09/13] ceph: make remaining I/O lock functions killable

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> Following the same pattern as ceph_start_io_write(), make
> ceph_start_io_read() and ceph_start_io_direct() killable to
> prevent indefinite hangs when waiting for i_rwsem during
> MDS/OSD unavailability.
> 
> This completes the killable lock conversion for all ceph I/O
> start functions, allowing blocked processes to be terminated
> with SIGKILL instead of hanging indefinitely.
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/file.c | 27 +++++++++++++++++++--------
>  fs/ceph/io.c   | 28 ++++++++++++++++++++--------
>  fs/ceph/io.h   |  4 ++--
>  3 files changed, 41 insertions(+), 18 deletions(-)
> 
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 01e4f31b1f2f3..c828552d51920 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -2122,10 +2122,15 @@ static ssize_t ceph_read_iter(struct kiocb *iocb, struct iov_iter *to)
>  	if (ceph_inode_is_shutdown(inode))
>  		return -ESTALE;
>  
> -	if (direct_lock)
> -		ceph_start_io_direct(inode);
> -	else
> -		ceph_start_io_read(inode);
> +	if (direct_lock) {
> +		ret = ceph_start_io_direct(inode);
> +		if (ret)
> +			return ret;
> +	} else {
> +		ret = ceph_start_io_read(inode);
> +		if (ret)
> +			return ret;
> +	}
>  
>  	if (!(fi->flags & CEPH_F_SYNC) && !direct_lock)
>  		want |= CEPH_CAP_FILE_CACHE;
> @@ -2278,7 +2283,9 @@ static ssize_t ceph_splice_read(struct file *in, loff_t *ppos,
>  	    (fi->flags & CEPH_F_SYNC))
>  		return copy_splice_read(in, ppos, pipe, len, flags);
>  
> -	ceph_start_io_read(inode);
> +	ret = ceph_start_io_read(inode);
> +	if (ret)
> +		return ret;
>  
>  	want = CEPH_CAP_FILE_CACHE;
>  	if (fi->fmode & CEPH_FILE_MODE_LAZY)
> @@ -2357,9 +2364,13 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
>  		direct_lock = true;
>  
>  retry_snap:
> -	if (direct_lock)
> -		ceph_start_io_direct(inode);
> -	else {
> +	if (direct_lock) {
> +		err = ceph_start_io_direct(inode);
> +		if (err) {
> +			ceph_free_cap_flush(prealloc_cf);
> +			return err;
> +		}
> +	} else {
>  		err = ceph_start_io_write(inode);
>  		if (err) {
>  			ceph_free_cap_flush(prealloc_cf);
> diff --git a/fs/ceph/io.c b/fs/ceph/io.c
> index f9ac89ec1d6a1..7bd57de2d9681 100644
> --- a/fs/ceph/io.c
> +++ b/fs/ceph/io.c
> @@ -47,20 +47,26 @@ static void ceph_block_o_direct(struct ceph_inode_info *ci, struct inode *inode)
>   * Note that buffered writes and truncates both take a write lock on
>   * inode->i_rwsem, meaning that those are serialised w.r.t. the reads.
>   */
> -void
> +int
>  ceph_start_io_read(struct inode *inode)

Which kernel version do you have? I can see down_read_killable() already
available in ceph_start_io_read() for v.7.0.0-rc3 [1].

Thanks,
Slava.

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/fs/ceph/io.c#L59

>  {
>  	struct ceph_inode_info *ci = ceph_inode(inode);
> +	int ret;
>  
>  	/* Be an optimist! */
> -	down_read(&inode->i_rwsem);
> +	ret = down_read_killable(&inode->i_rwsem);
> +	if (ret)
> +		return ret;
>  	if (!(READ_ONCE(ci->i_ceph_flags) & CEPH_I_ODIRECT))
> -		return;
> +		return 0;
>  	up_read(&inode->i_rwsem);
>  	/* Slow path.... */
> -	down_write(&inode->i_rwsem);
> +	ret = down_write_killable(&inode->i_rwsem);
> +	if (ret)
> +		return ret;
>  	ceph_block_o_direct(ci, inode);
>  	downgrade_write(&inode->i_rwsem);
> +	return 0;
>  }
>  
>  /**
> @@ -138,20 +144,26 @@ static void ceph_block_buffered(struct ceph_inode_info *ci, struct inode *inode)
>   * Note that buffered writes and truncates both take a write lock on
>   * inode->i_rwsem, meaning that those are serialised w.r.t. O_DIRECT.
>   */
> -void
> +int
>  ceph_start_io_direct(struct inode *inode)
>  {
>  	struct ceph_inode_info *ci = ceph_inode(inode);
> +	int ret;
>  
>  	/* Be an optimist! */
> -	down_read(&inode->i_rwsem);
> +	ret = down_read_killable(&inode->i_rwsem);
> +	if (ret)
> +		return ret;
>  	if (READ_ONCE(ci->i_ceph_flags) & CEPH_I_ODIRECT)
> -		return;
> +		return 0;
>  	up_read(&inode->i_rwsem);
>  	/* Slow path.... */
> -	down_write(&inode->i_rwsem);
> +	ret = down_write_killable(&inode->i_rwsem);
> +	if (ret)
> +		return ret;
>  	ceph_block_buffered(ci, inode);
>  	downgrade_write(&inode->i_rwsem);
> +	return 0;
>  }
>  
>  /**
> diff --git a/fs/ceph/io.h b/fs/ceph/io.h
> index 94ce176df9997..9432b8b607650 100644
> --- a/fs/ceph/io.h
> +++ b/fs/ceph/io.h
> @@ -2,11 +2,11 @@
>  #ifndef _FS_CEPH_IO_H
>  #define _FS_CEPH_IO_H
>  
> -void ceph_start_io_read(struct inode *inode);
> +int ceph_start_io_read(struct inode *inode);
>  void ceph_end_io_read(struct inode *inode);
>  int ceph_start_io_write(struct inode *inode);
>  void ceph_end_io_write(struct inode *inode);
> -void ceph_start_io_direct(struct inode *inode);
> +int ceph_start_io_direct(struct inode *inode);
>  void ceph_end_io_direct(struct inode *inode);
>  
>  #endif /* FS_CEPH_IO_H */

Re: [PATCH v1 08/13] ceph: make ceph_start_io_write() killable

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

Hi Slava,

Thanks for pointing this out.

My patch series is based on v6.12.57 (stable/LTS), where
ceph_start_io_write() still uses the non-killable down_write().

I see that upstream v7.0-rc3 already has this change. I will take
this into account and adapt the series for 6.18 LTS and 7.0+ as
well, dropping patches that are already upstream.

Thanks,
Ionut

Re: [PATCH v1 10/13] ceph: force mdsmap refresh on persistent MDS connection failures

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> During rolling upgrades in containerized environments (e.g.,
> rook-ceph in containerized environments), MDS daemons are restarted and
> may receive new IP addresses from the CNI plugin. The kernel CephFS
> client (libceph) maintains a cached mdsmap with the old MDS address
> and attempts to reconnect indefinitely.
> 
> The monitor client subscribes to mdsmap updates with
> start=current_epoch+1, expecting the monitor to push new maps.
> However, if the monitor connection was also disrupted during the
> upgrade (e.g., due to EADDRNOTAVAIL from IPv6 DAD), the subscription
> may not be properly re-established, leaving the client with a stale
> mdsmap.
> 
> This results in a deadlock:
> - The kernel client retries connecting to the old MDS address forever
> - The MDS connection has no .fault callback, so the MDS client is
>   never notified of persistent connection failures
> - The stale mdsmap is never refreshed because the client believes
>   its subscription is active
> - New pod mounts via CSI hang in ContainerCreating state
> - The rook-ceph upgrade cannot complete
> 
> Observed in production (kernel 6.12.0-1-rt-amd64,
> Ceph Reef 18.2.2->18.2.5 upgrade):
>   - mdsmap stuck at epoch 53 while cluster was at epoch 68
>   - MDS session state: hung
>   - monc showed: have mdsmap 53 want 54+
>   - MDS address changed from dead:beef::...eb75 to dead:beef::...bc76
>   - Client kept retrying on old address for 30+ minutes
> 
> Fix this by:
> 1. Adding a .fault callback to the MDS connection operations
>    (mds_con_ops) so the MDS client is notified when connections fail
> 2. Tracking consecutive connection failures per MDS session via a
>    new s_con_failures counter
> 3. When failures exceed MDS_CON_FAIL_REFRESH_MDSMAP (10 consecutive
>    failures, ~2.5-15 seconds depending on backoff), forcing a fresh
>    mdsmap subscription with start=0 to get the complete current map
> 4. Resetting the failure counter when a session message is
>    successfully received (in handle_session)
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  fs/ceph/mds_client.c | 73 ++++++++++++++++++++++++++++++++++++++++++++
>  fs/ceph/mds_client.h |  2 ++
>  2 files changed, 75 insertions(+)
> 
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index ac86225595b5f..0e766880056c0 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -66,6 +66,12 @@ static void ceph_cap_release_work(struct work_struct *work);
>  static void ceph_cap_reclaim_work(struct work_struct *work);
>  
>  static const struct ceph_connection_operations mds_con_ops;
> +/*
> + * Number of consecutive MDS connection failures before forcing
> + * a fresh mdsmap subscription. This handles stale mdsmap scenarios
> + * during rolling upgrades where MDS addresses change.
> + */
> +#define MDS_CON_FAIL_REFRESH_MDSMAP	10

Why exactly 10? Any thoughts?

>  
>  
>  /*
> @@ -997,6 +1003,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,
>  	s->s_mdsc = mdsc;
>  	s->s_mds = mds;
>  	s->s_state = CEPH_MDS_SESSION_NEW;
> +	s->s_con_failures = 0;
>  	mutex_init(&s->s_mutex);
>  
>  	ceph_con_init(&s->s_con, s, &mds_con_ops, &mdsc->fsc->client->msgr);
> @@ -4341,6 +4348,9 @@ static void handle_session(struct ceph_mds_session *session,
>  	      ceph_session_op_name(op), session,
>  	      ceph_session_state_name(session->s_state), seq);
>  
> +	/* Reset connection failure counter on successful session message */
> +	session->s_con_failures = 0;
> +
>  	if (session->s_state == CEPH_MDS_SESSION_HUNG) {
>  		session->s_state = CEPH_MDS_SESSION_OPEN;
>  		pr_info_client(cl, "mds%d came back\n", session->s_mds);
> @@ -5427,6 +5437,22 @@ bool check_session_state(struct ceph_mds_session *s)
>  		if (s->s_ttl && time_after(jiffies, s->s_ttl)) {
>  			s->s_state = CEPH_MDS_SESSION_HUNG;
>  			pr_info_client(cl, "mds%d hung\n", s->s_mds);

Do we need this message now?

> +
> +			/*
> +			 * Force a fresh mdsmap subscription when a session
> +			 * becomes hung. The MDS may have restarted with a
> +			 * new address during a rolling upgrade, and the
> +			 * connection may have entered STANDBY state (no
> +			 * .fault callback) rather than generating connect
> +			 * errors. Requesting mdsmap from epoch 0 ensures
> +			 * we get the current map with updated addresses.
> +			 */
> +			pr_warn_client(cl,
> +				"mds%d hung, requesting fresh mdsmap\n",
> +				s->s_mds);

Maybe, pr_info_client()?

> +			if (ceph_monc_want_map(&s->s_mdsc->fsc->client->monc,
> +					       CEPH_SUB_MDSMAP, 0, true))
> +				ceph_monc_renew_subs(&s->s_mdsc->fsc->client->monc);
>  		}
>  		break;
>  	case CEPH_MDS_SESSION_CLOSING:
> @@ -6528,12 +6554,59 @@ static int mds_check_message_signature(struct ceph_msg *msg)
>         return ceph_auth_check_message_signature(auth, msg);
>  }
>  
> +/*
> + * Handle MDS connection fault.
> + *
> + * Track consecutive connection failures and force a fresh mdsmap
> + * subscription when failures exceed the threshold. This handles the
> + * case where the MDS address has changed (e.g., during a rolling
> + * upgrade) but the client has a stale mdsmap and keeps retrying
> + * on the old address.
> + */
> +static void mds_fault(struct ceph_connection *con)
> +{
> +	struct ceph_mds_session *s = con->private;
> +	struct ceph_mds_client *mdsc = s->s_mdsc;
> +	struct ceph_client *cl = mdsc->fsc->client;
> +	int failures;

Why do you need failures local variable? You can use s->s_con_failures
everywhere.

> +
> +	failures = ++s->s_con_failures;
> +
> +	if (failures == MDS_CON_FAIL_REFRESH_MDSMAP) {

Why not simply failures >= MDS_CON_FAIL_REFRESH_MDSMAP?

> +		pr_warn_client(cl,
> +			"mds%d connection failed %d times, requesting fresh mdsmap\n",
> +			s->s_mds, failures);

Do we need this message in system log? Maybe, debug output instead?

> +
> +		/*
> +		 * Force a fresh mdsmap subscription by requesting from
> +		 * epoch 0. This ensures we get the complete current map
> +		 * with up-to-date MDS addresses, rather than waiting for
> +		 * an incremental update that may never arrive if our
> +		 * subscription was lost during a monitor reconnection.
> +		 */
> +		if (ceph_monc_want_map(&mdsc->fsc->client->monc,
> +				       CEPH_SUB_MDSMAP, 0, true))
> +			ceph_monc_renew_subs(&mdsc->fsc->client->monc);
> +	} else if (failures > MDS_CON_FAIL_REFRESH_MDSMAP &&
> +		   failures % MDS_CON_FAIL_REFRESH_MDSMAP == 0) {

I don't follow this arithmetics. Why do you need this case at all?

> +		/*
> +		 * Periodically retry the fresh mdsmap request in case
> +		 * the previous one was lost or the monitor was also
> +		 * temporarily unavailable.
> +		 */
> +		if (ceph_monc_want_map(&mdsc->fsc->client->monc,
> +				       CEPH_SUB_MDSMAP, 0, true))
> +			ceph_monc_renew_subs(&mdsc->fsc->client->monc);
> +	}
> +}
> +
>  static const struct ceph_connection_operations mds_con_ops = {
>  	.get = mds_get_con,
>  	.put = mds_put_con,
>  	.alloc_msg = mds_alloc_msg,
>  	.dispatch = mds_dispatch,
>  	.peer_reset = mds_peer_reset,
> +	.fault = mds_fault,
>  	.get_authorizer = mds_get_authorizer,
>  	.add_authorizer_challenge = mds_add_authorizer_challenge,
>  	.verify_authorizer_reply = mds_verify_authorizer_reply,
> diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
> index 695c5a9c94026..44585b1cb4485 100644
> --- a/fs/ceph/mds_client.h
> +++ b/fs/ceph/mds_client.h
> @@ -251,6 +251,8 @@ struct ceph_mds_session {
>  	struct list_head  s_waiting;  /* waiting requests */
>  	struct list_head  s_unsafe;   /* unsafe requests */
>  	struct xarray	  s_delegated_inos;
> +
> +	int		  s_con_failures; /* consecutive connection failures */

Why do you need int data type? The upper threshold is declared as 10. Why not
u8, then ?

Thanks,
Slava.

>  };
>  
>  /*

Re: [PATCH v1 11/13] libceph: reset source address on persistent EADDRNOTAVAIL

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 10:16 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> In containerized environments (e.g., Rook-Ceph with
> Calico CNI), the kernel CephFS client's source address
> (msgr->inst.addr) is learned from the first successful monitor
> connection via process_hello(). If the initial connection was made
> through a transient CNI pod address (e.g., a Calico-assigned
> dead:beef::... address from a CSI plugin pod), that address is
> stored permanently in inst.addr.
> 
> When the pod is later rescheduled or the CNI reconfigures networking,
> the original pod address is removed and Calico installs a blackhole
> route for the old address range. All subsequent kernel socket
> connections fail with EADDRNOTAVAIL at ip6_dst_lookup_flow() before
> even sending a TCP SYN, because the IPv6 source address selection
> finds the blackhole route for the old address range.
> 
> This creates a permanent deadlock:
> - All connections (mon, mds, osd) fail with EADDRNOTAVAIL
> - The client cannot reach any monitor to re-learn its address
> - inst.addr is never blank again (set once, never cleared)
> - The only recovery is force-unmounting and remounting
> 
> Fix this by tracking consecutive EADDRNOTAVAIL failures across all
> connections using an atomic counter in struct ceph_messenger. After
> ADDRNOTAVAIL_RESET_THRESHOLD (30) consecutive failures (~3 seconds
> at 100ms retry interval), reset inst.addr.in_addr to zero (blank)
> while preserving the nonce and type. This allows process_hello()
> (msgr2) or process_banner() (msgr1) to re-learn the source address
> from the next successful monitor connection, which will use the
> current stable host address instead of the defunct pod address.
> 
> The counter is reset to zero when:
> - A TCP connection succeeds (in ceph_tcp_connect)
> - The address is successfully re-learned (in process_hello/
>   process_banner)
> 
> Observed in production (kernel 6.12.0-1-rt-amd64, Ceph Reef
> 18.2.2->18.2.5 upgrade, IPv6-only cluster):
>   - Client instance: client.55136 [dead:beef::a2bf:c94c:345d:bc66]:0
>   - Address dead:beef::a2bf:c94c:345d:bc66 was a Calico pod address
>   - After pod reschedule: blackhole dead:beef::a2bf:c94c:345d:bc40/122
>   - All connections stuck in EADDRNOTAVAIL loop for 16+ hours
>   - After force-unmount + remount: new client got stable host address
>     [aefd::2b93:d245:fd09:127e]:0 and worked immediately
> 
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  include/linux/ceph/messenger.h | 20 +++++++++++++
>  net/ceph/messenger.c           | 51 ++++++++++++++++++++++++++++++++++
>  net/ceph/messenger_v1.c        |  7 +++++
>  net/ceph/messenger_v2.c        | 12 ++++++++
>  4 files changed, 90 insertions(+)
> 
> diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
> index 730a754353aed..d8f7946d85a68 100644
> --- a/include/linux/ceph/messenger.h
> +++ b/include/linux/ceph/messenger.h
> @@ -113,6 +113,17 @@ struct ceph_messenger {
>  	 */
>  	u32 global_seq;
>  	spinlock_t global_seq_lock;
> +
> +	/*
> +	 * Track consecutive EADDRNOTAVAIL failures across all
> +	 * connections. When this exceeds a threshold, the client's
> +	 * inst.addr is reset to blank so that process_hello() will
> +	 * re-learn the source address from the next successful
> +	 * monitor connection. This handles the case where the
> +	 * original source address was a transient CNI pod address
> +	 * that no longer exists.
> +	 */
> +	atomic_t addr_notavail_count;

Would atomic_t be enough? Do we need to consider atomic64_t? Any thoughts?

>  };
>  
>  enum ceph_msg_data_type {
> @@ -328,6 +339,15 @@ struct ceph_msg {
>   */
>  #define ADDRNOTAVAIL_DELAY	(HZ / 10)

You already introduce likewise constant. Why do you need introduce another one?
And why HZ/10 namely?

>  
> +/*
> + * Number of consecutive EADDRNOTAVAIL failures (across all connections)
> + * before resetting the messenger's source address. At ~100ms per retry,
> + * 30 failures means ~3 seconds of persistent EADDRNOTAVAIL before we
> + * conclude the source address is permanently gone (e.g., a CNI pod
> + * address that was removed) and needs to be re-learned.
> + */
> +#define ADDRNOTAVAIL_RESET_THRESHOLD	30

I am not completely sure that this math sounds reasonably well. :)

> +
>  struct ceph_connection_v1_info {
>  	struct kvec out_kvec[8],         /* sending header/footer data */
>  		*out_kvec_cur;
> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> index c40c7c332e7f4..8165e6a8fe092 100644
> --- a/net/ceph/messenger.c
> +++ b/net/ceph/messenger.c
> @@ -497,6 +497,10 @@ int ceph_tcp_connect(struct ceph_connection *con)
>  	else
>  		con->v1.addr_notavail = false;
>  
> +	/* Reset the persistent EADDRNOTAVAIL counter on success */
> +	if (atomic_read(&con->msgr->addr_notavail_count) > 0)

Why do you check the addr_notavail_count value? If it is success, then who cares
what this value has before.

> +		atomic_set(&con->msgr->addr_notavail_count, 0);
> +
>  	return 0;
>  }
>  
> @@ -1663,6 +1667,52 @@ static void con_fault(struct ceph_connection *con)
>  		}
>  	}
>  
> +	/*
> +	 * Track persistent EADDRNOTAVAIL across all connections.
> +	 * If the source address stored in msgr->inst.addr is no longer
> +	 * valid (e.g., it was a transient CNI pod address that has been
> +	 * removed), all connections will fail with EADDRNOTAVAIL at
> +	 * ip6_dst_lookup_flow() before even sending a SYN.
> +	 *
> +	 * After ADDRNOTAVAIL_RESET_THRESHOLD consecutive failures,
> +	 * reset inst.addr to blank so that process_hello() will
> +	 * re-learn the source address from the next successful
> +	 * monitor connection. The nonce is preserved.
> +	 */
> +	if (addr_issue) {

Where addr_issue has been declared and initialized?

> +		int count = atomic_inc_return(&con->msgr->addr_notavail_count);
> +
> +		if (count == ADDRNOTAVAIL_RESET_THRESHOLD) {

Why not count >= ADDRNOTAVAIL_RESET_THRESHOLD?

> +			struct ceph_entity_addr *my_addr =
> +				&con->msgr->inst.addr;

It looks like you need to introduce a static inline function if you need to play
likewise tricks.

> +
> +			pr_warn("libceph: %d consecutive EADDRNOTAVAIL errors, resetting source address %s (will re-learn from monitor)\n",
> +				count, ceph_pr_addr(my_addr));
> +
> +			/*
> +			 * Zero out the address portion of in_addr but
> +			 * preserve ss_family, nonce, and type so the
> +			 * client identity is maintained and debug output
> +			 * remains readable. process_hello() checks
> +			 * ceph_addr_is_blank() and will fill in the new
> +			 * address from the monitor's addr_for_me response.
> +			 *
> +			 * We preserve ss_family so that ceph_pr_addr()
> +			 * shows e.g. "[::]:0" instead of
> +			 * "(unknown sockaddr family 0)".
> +			 */
> +			{

It is another reason of necessity of static inline function.

> +				sa_family_t family =
> +					get_unaligned(&my_addr->in_addr.ss_family);
> +				memset(&my_addr->in_addr, 0,
> +				       sizeof(my_addr->in_addr));
> +				put_unaligned(family,
> +					      &my_addr->in_addr.ss_family);

Do we have any existing function for this?

> +			}
> +			ceph_encode_my_addr(con->msgr);
> +		}
> +	}
> +
>  	WARN_ON(con->state == CEPH_CON_S_STANDBY ||
>  		con->state == CEPH_CON_S_CLOSED);
>  
> @@ -1740,6 +1790,7 @@ void ceph_messenger_init(struct ceph_messenger *msgr,
>  	ceph_encode_my_addr(msgr);
>  
>  	atomic_set(&msgr->stopping, 0);
> +	atomic_set(&msgr->addr_notavail_count, 0);
>  	write_pnet(&msgr->net, get_net(current->nsproxy->net_ns));
>  
>  	dout("%s %p\n", __func__, msgr);
> diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c
> index 0cb61c76b9b87..4f3868f296c06 100644
> --- a/net/ceph/messenger_v1.c
> +++ b/net/ceph/messenger_v1.c
> @@ -736,6 +736,13 @@ static int process_banner(struct ceph_connection *con)
>  		ceph_encode_my_addr(con->msgr);
>  		dout("process_banner learned my addr is %s\n",
>  		     ceph_pr_addr(my_addr));
> +
> +		if (atomic_read(&con->msgr->addr_notavail_count) > 0) {
> +			pr_info("libceph: re-learned source address %s from peer %s\n",
> +				ceph_pr_addr(my_addr),
> +				ceph_pr_addr(&con->peer_addr));

Do we need to inform about this? I assume we could have bunch of messages in
system log.

> +			atomic_set(&con->msgr->addr_notavail_count, 0);
> +		}
>  	}
>  
>  	return 0;
> diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
> index bd608ffa06279..12ad9f571dcca 100644
> --- a/net/ceph/messenger_v2.c
> +++ b/net/ceph/messenger_v2.c
> @@ -2260,6 +2260,18 @@ static int process_hello(struct ceph_connection *con, void *p, void *end)
>  		dout("%s con %p set my addr %s, as seen by peer %s\n",
>  		     __func__, con, ceph_pr_addr(my_addr),
>  		     ceph_pr_addr(&con->peer_addr));
> +
> +		/*
> +		 * If we re-learned the address after a reset due to
> +		 * persistent EADDRNOTAVAIL, log it and clear the
> +		 * failure counter.
> +		 */
> +		if (atomic_read(&con->msgr->addr_notavail_count) > 0) {
> +			pr_info("libceph: re-learned source address %s from monitor %s\n",
> +				ceph_pr_addr(my_addr),
> +				ceph_pr_addr(&con->peer_addr));

The same question here.

Thanks,
Slava.

> +			atomic_set(&con->msgr->addr_notavail_count, 0);
> +		}
>  	} else {
>  		dout("%s con %p my addr already set %s\n",
>  		     __func__, con, ceph_pr_addr(my_addr));

RE: [PATCH v1 08/13] ceph: make ceph_start_io_write() killable

[Viacheslav Dubeyko]

On Thu, 2026-03-12 at 22:45 +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> Hi Slava,
> 
> Thanks for pointing this out.
> 
> My patch series is based on v6.12.57 (stable/LTS), where
> ceph_start_io_write() still uses the non-killable down_write().
> 
> I see that upstream v7.0-rc3 already has this change. I will take
> this into account and adapt the series for 6.18 LTS and 7.0+ as
> well, dropping patches that are already upstream.
> 
> 

Sounds great! :)

I think, maybe, you need to consider to use namely 100ms as timeout instead of
HZ/10.

Thanks,
Slava.

Re: [PATCH v1 13/13] libceph: force host network namespace for kernel CephFS mounts

[Ilya Dryomov]

On Thu, Mar 12, 2026 at 9:17 AM Ionut Nechita (Wind River)
<ionut.nechita@windriver.com> wrote:
>
> From: Ionut Nechita <ionut.nechita@windriver.com>
>
> In containerized environments (e.g., Rook-Ceph CSI with
> forcecephkernelclient=true), the mount() syscall
> for kernel CephFS may be invoked from a pod's network namespace
> instead of the host namespace. This happens despite the CSI node
> plugin (csi-cephfsplugin) running with hostNetwork: true, due to
> race conditions during kubelet restart or pod scheduling.

Hi Ionut,

Can you elaborate on these race conditions?  It sounds like a bug or
a misconfiguration in the orchestration in userspace that this patch is
trying to work around in the kernel client.

>
> ceph_messenger_init() captures current->nsproxy->net_ns at mount
> time and uses it for all subsequent socket operations. When a pod
> NS is captured, all kernel ceph sockets (mon, mds, osd) are
> created in that namespace, which typically lacks routes to the
> Ceph monitors (e.g., fd04:: ClusterIP addresses).
> This causes permanent EADDRNOTAVAIL (-99) on every connection
> attempt at ip6_dst_lookup_flow(), with no possibility of recovery
> short of force-unmount and remount from the correct namespace.

What network provider (in the sense of [1]) are you using?

>
> Root cause confirmed via kprobe tracing on ip6_dst_lookup_flow:
> the net pointer passed to the routing lookup was the pod's
> net_ns (0xff367a0125dd5780) instead of init_net
> (0xffffffffbda76940). The pod NS had no route for fd04::/64
> (monitor ClusterIP range), while userspace python connect() from
> the same host succeeded because it ran in host NS.
>
> Fix this by always using init_net (the host network namespace)
> in ceph_messenger_init(). The kernel CephFS client inherently
> requires host-level network access to reach Ceph monitors, OSDs,
> and MDS daemons. Using the caller's namespace was inherited from
> generic socket patterns but is incorrect for a kernel filesystem

This behavior wasn't inherited but actually introduced as a feature in
commit [2] at someone's request.  Prior to that change attempting to
mount a CephFS filesytem or map an RBD image from anywhere but init_net
produced an error, see commit [3].

I'm going to challenge your "incorrect for a kernel filesystem" claim
because NFS, SMB/CIFS, AFS and likely other network filesystem clients
in the kernel behave the same way.  Mounts outside of init_net are
allowed with the mounting process network namespace getting captured
and used when creating sockets.

> client that must survive beyond the lifetime of the mounting
> process and its network namespace.

Network namespaces are reference counted and CephFS grabs a reference
for the namespace it's mounted in.  The namespace should persist for as
long as the CephFS mount persists even if the mounting process goes
away: another process should be able to enter that namespace, etc.  The
namespace can of course get wedged by the orchestration tearing down
the relevant virtual network devices prematurely, but it's a separate
issue.

>
> A warning is logged when a mount from a non-init namespace is
> detected, to aid debugging.
>
> Observed in production (kernel 6.12.0-1-rt-amd64, Ceph Reef
> 18.2.5, IPv6-only cluster, ceph-csi v3.13.1):
>   - Fresh boot of compute-0, ceph-csi mounts CephFS via kernel
>   - All monitor connections fail with EADDRNOTAVAIL immediately
>   - kprobe confirms wrong net_ns in ip6_dst_lookup_flow
>   - Workaround: umount -l + systemctl restart kubelet
>   - After restart: mount captures host NS, works immediately

[1] https://github.com/rook/rook/blob/master/Documentation/CRDs/Cluster/network-providers.md
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=757856d2b9568a701df9ea6a4be68effbb9d6f44
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eea553c21fbfa486978c82525ee8256239d4f921

Thanks,

                Ilya

Re: [PATCH v1 13/13] libceph: force host network namespace for kernel CephFS mounts

[Ionut Nechita (Wind River)]

Hi Ilya,

Thank you for the detailed feedback and the historical context around
commits 757856d2 and eea553c2 -- I wasn't aware that namespace-aware
mounting was an intentional feature.

With the full series applied (patches 1-13), the Rook-Ceph rolling
upgrade scenario (e.g., Ceph 18.2.2 -> 18.2.5) with active CephFS
workloads completes successfully. The connection recovery, sync
timeouts, and mdsmap refresh patches address the core issues.

I agree that patch 13 is a force-impact change and can be seen as a
workaround for what is likely a race condition in the CSI/kubelet
orchestration layer. I'll drop it from v2 of this series.

I'd like to collaborate to better understand the namespace interaction
with CephFS in containerized environments. I'll gather more details  
about the specific race condition and share them both here and in the
Ceph tracker bug I opened:

  https://tracker.ceph.com/issues/74897

Regarding your questions:
- The network provider is Calico (IPv6-only cluster)
- The race condition occurs during kubelet restart when ceph-csi  
  issues mount() -- in some cases the mount syscall appears 
  to execute in the context of a pod namespace rather than 
  the host namespace, though I need to investigate 
  further to provide a proper reproducer

Thanks again for the review.

Best regards,
Ionut

Re: [PATCH v1 13/13] libceph: force host network namespace for kernel CephFS mounts

[Ionut Nechita (Wind River)]

Hi Ilya,

Following up with the additional data I promised. I reproduced the
issue on a fresh cluster and have concrete evidence of the namespace
problem.

Environment (4-node cluster, 2 controllers + 2 workers):

  $ kubectl get nodes
  NAME           STATUS   ROLES           KERNEL-VERSION
  compute-0      Ready    <none>          6.12.0-1-rt-amd64
  compute-1      Ready    <none>          6.12.0-1-rt-amd64
  controller-0   Ready    control-plane   6.12.0-1-amd64
  controller-1   Ready    control-plane   6.12.0-1-amd64

  - OS: Debian GNU/Linux 11 (bullseye)
  - Container runtime: containerd 1.7.27
  - Kubernetes: v1.29.2
  - Rook: v1.16.6
  - ceph-csi: v3.13.1
  - Ceph: 18.2.5 (Reef)
  - Network: Calico + Multus, IPv6-only
  - Pod CIDR: dead:beef::/64 (Calico, vxlanMode: Never)
  - Service CIDR: fd04::/112
  - CSI_FORCE_CEPHFS_KERNEL_CLIENT: true
  - CSI_ENABLE_HOST_NETWORK: true

The scenario is a Rook-Ceph rolling upgrade (Ceph 18.2.2 -> 18.2.5).
During the upgrade, Rook recreates the CSI DaemonSet pods and various
Ceph daemon pods (MON, MDS, OSD). Kubelet then needs to remount
CephFS volumes for workload pods on the node.

After the upgrade, the kernel ceph client is stuck with permanent
EADDRNOTAVAIL (-99) on all monitor connections:

  libceph: connect (1)[fd04::652b]:6789 error -99
  libceph: mon0 (1)[fd04::652b]:6789 connect error

The monitors are Kubernetes ClusterIP services:

  rook-ceph-mon-a  ClusterIP  fd04::652b  6789/TCP,3300/TCP
  rook-ceph-mon-b  ClusterIP  fd04::c0e7  6789/TCP,3300/TCP
  rook-ceph-mon-c  ClusterIP  fd04::1981  6789/TCP,3300/TCP

Here is the key evidence. The kernel ceph client debugfs status shows:

  $ cat /sys/kernel/debug/ceph/*/status
  instance: client.374328 (3)[dead:beef::a2bf:c94c:345d:bc66]:0

The source address dead:beef::a2bf:c94c:345d:bc66 is from the Calico
pod CIDR (dead:beef::/64). This address does NOT belong to any
currently running pod on the node. I enumerated all active CNI
namespaces:

  $ for ns in $(ip netns list | awk '{print $1}'); do
      ip netns exec $ns ip -6 addr show | grep dead:beef
    done

  ...bc6d  kube-sriov-cni-ds
  ...bc70  stx-centos
  ...bc73  rook-ceph-mon-a
  ...bc74  rook-ceph-crashcollector
  ...bc75  rook-ceph-exporter
  ...bc76  rook-ceph-mgr-c
  ...bc78  rook-ceph-osd-0

Address ...bc66 is not present in any existing namespace. The pod
that owned it was destroyed during the upgrade, and Calico removed
its veth interfaces during CNI cleanup.

Meanwhile, the CSI plugin pod is correctly in the host namespace:

  $ kubectl exec csi-cephfsplugin-gdrqr -c csi-cephfsplugin \
      -- readlink /proc/1/ns/net
  net:[4026531840]

  $ readlink /proc/1/ns/net   # on host
  net:[4026531840]

And from host userspace, connecting to the same ClusterIP monitors
works fine (goes through kube-proxy iptables DNAT):

  $ python3 -c "import socket; s=socket.socket(socket.AF_INET6, \
      socket.SOCK_STREAM); s.connect(('fd04::652b', 6789)); print('OK')"
  OK

But ping6 from host fails (ICMP not NAT'd by kube-proxy):

  $ ping6 -c1 fd04::652b
  From fdff:719a:bf60:4008::46e icmp_seq=1 Destination unreachable: No route

So the situation is:
  1. The kernel ceph client captured a pod network namespace at mount
     time (source address from dead:beef::/64 proves this)
  2. That pod was later destroyed during the upgrade
  3. Calico tore down the veth interfaces in that namespace
  4. The namespace persists (ref-counted by ceph) but has no
     interfaces or routes -- it is a zombie namespace
  5. All kernel ceph connect() calls fail with EADDRNOTAVAIL
  6. No recovery is possible without force-unmount + remount

As you noted, this is the "orchestration tearing down the relevant
virtual network devices prematurely" scenario. The namespace is kept
alive by the ceph reference, but it becomes non-functional.

I'm still investigating exactly how mount.ceph ends up in a pod
namespace despite the CSI plugin having hostNetwork: true. I have a
monitoring script set up to capture the namespace of mount.ceph
processes during the next upgrade attempt. I suspect it happens
during the brief window when the old CSI pod is terminated and the
new one is not yet ready, but kubelet still attempts to mount
volumes. I'll follow up with that data.

I've also filed this on the Ceph tracker:
  https://tracker.ceph.com/issues/74897

Thanks,
Ionut

Re: [PATCH v1 13/13] libceph: force host network namespace for kernel CephFS mounts

[Ionut Nechita (Wind River)]

Hi Ilya,

I've identified the root cause. You were right -- this is an
orchestration issue, not a kernel bug.

The problem is caused by the Rook "holder pod" mechanism in
Rook v1.13.7 (used in our older release). Here is the full picture:

In Rook v1.13.7, when Multus is present or CSI_ENABLE_HOST_NETWORK
is false, Rook deploys a "csi-cephfsplugin-holder" DaemonSet. This
holder pod does NOT have hostNetwork: true -- it runs in a Calico
pod network namespace. Its purpose is to expose its network namespace
via a symlink:

  ln -s /proc/$$/ns/net /var/lib/kubelet/plugins/<driver>/<ns>.net.ns

Ceph-CSI then uses this network namespace when performing kernel
mounts. The holder pod template even has a comment:

  "This pod is not expected to be updated nor restarted unless
   the node reboots."

And uses updateStrategy: OnDelete to prevent rolling updates.

The condition for enabling holder pods (controller.go:206):

  holderEnabled := !csiHostNetworkEnabled || cluster.Spec.Network.IsMultus()

Our cluster uses Calico + Multus, so holderEnabled is always true
regardless of CSI_ENABLE_HOST_NETWORK.

During the upgrade from Rook v1.13.7 to v1.16.6, the new Rook
version sets holderEnabled = false unconditionally and deletes the
holder DaemonSets. When the holder pod is deleted, Calico tears
down the veth interfaces in its network namespace. The kernel ceph
client still holds a reference to that namespace, but it no longer
has any network interfaces or routes, resulting in permanent
EADDRNOTAVAIL (-99).

Evidence from the live reproduction:

  Kernel ceph client status:
    instance: client.74244 (3)[dead:beef::a2bf:c94c:345d:bc6f]:0

  The holder pod on compute-0 had the same address:
    csi-cephfsplugin-holder-rook-ceph-dpnbl  dead:beef::a2bf:c94c:345d:bc6f

  After upgrade, the address ...bc6f is not present in any active
  CNI namespace -- the holder pod was deleted and Calico cleaned up
  the veth.

  dmesg shows the session was initially established successfully
  (at boot time, from the holder pod namespace), then lost when
  the holder pod was destroyed during upgrade:

    [  204.515008] libceph: mon0 session established
    [  959.829581] libceph: mon0 session lost, hunting for new mon
    [  959.829698] libceph: connect error -99  (permanent)

Version details:
  Old release (stx.10): Rook v1.13.7, ceph-csi v3.10.2, Ceph v18.2.2
  New release (stx.11): Rook v1.16.6, ceph-csi v3.13.1, Ceph v18.2.5

The new release (Rook v1.16.6) eliminates holder pods entirely and
performs kernel mounts directly from the csi-cephfsplugin DaemonSet,
which has hostNetwork: true. After the upgrade completes and the
stale mount is cleared (umount -l + kubelet restart), new mounts
work correctly from the host namespace.

So to summarize: this was not a kernel bug. The kernel ceph client
correctly captured the network namespace of the mounting process
(the holder pod), as designed. The problem was that the orchestration
(Rook upgrade) destroyed the holder pod and its network namespace
while the kernel mount was still active.

I'll drop patch 13 from the series as previously agreed. Thank you
for pushing me to investigate this properly.

I've also updated the Ceph tracker:
  https://tracker.ceph.com/issues/74897

Thanks,
Ionut