From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B47653E5561 for ; Wed, 15 Apr 2026 17:01:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776272510; cv=none; b=echbRhZ46/4yRy0l8RXmNJuI2rhzPXn28cR7jm7PwbgSohOeINdBAKVk69OU0i0mmk+zMUfxktJmp6jgbJin8o4OWjc1Ui0y/uYeViRz9sxqXWp6oTZWICzBqRANJP6uGEoGVTTZfxRv2hvEbNTwhbugFJ/7NqOIvpo+n+NcfIc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776272510; c=relaxed/simple; bh=EjpLUZD65b4tabg+yzSnnYbDXhagBAIufBacfc+cllk=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=eEQaW7GILTdBktW402tUIZ+AAR11ipIVuWT1PRqTkpzxKcM6Nku60iXdWNIz9eo7CB1dgTBWw0I/rSdqq3eQPEeGTO673Hm8xa3yBjr0rU42ztfbSXYJgg4Da4lD9OGFUtOLr7kp1+fciPMbh3n6JRuDgYSdKqcSqQDCfBh7hL4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Bmxmkdsl; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=DYpbekav; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Bmxmkdsl"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="DYpbekav" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1776272507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=H4wsvwRJbqgcLtEjgPuYkCxpD7ZHOSfOGTS4tTJGptA=; b=BmxmkdsllwA1vIGx3E7wbkv3D88Pe+T0SoCJWm1HOdMBi78hnIS0wkd+jo1VgEvo53Oq44 wtNdvacVl5Ntb18d54SS3wstjA51XgsKXcuhXyouJ89ynNn5NXzdZJ4O7YPpgelko5SKxV p4Ar6tKZ1IbrNxQRTN1sKuza80Y7Sg8= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-zFwyOdvwPd-0hGg9mE3fgA-1; Wed, 15 Apr 2026 13:01:45 -0400 X-MC-Unique: zFwyOdvwPd-0hGg9mE3fgA-1 X-Mimecast-MFC-AGG-ID: zFwyOdvwPd-0hGg9mE3fgA_1776272505 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-50d8ed08aa4so146689691cf.3 for ; Wed, 15 Apr 2026 10:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1776272505; x=1776877305; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=H4wsvwRJbqgcLtEjgPuYkCxpD7ZHOSfOGTS4tTJGptA=; b=DYpbekavkYP3br4Y+6f8A7sf8JVZyYDLxD0zbGt+zQ80UywWBfvrVf7rld8Nvu9peB aii2rf141fHlZIKMNf8z+OBdVd69Es6uOwh41a4O5xntqhGH0XfQQzC2AFlyVrYHkRl1 jPFEsfO+n7WV1p7MJObxnuTSXKkKnvl60XgjrlsXwfLHrDYsEvKDICssUoSnr0Y+Aumh ZDI3sZTLinlZ6tVuh5hl1SdenxPEbTKRlV0T7GCaVhQliF4xYu/l/ucni9PUQ8KNyUS4 42wevNNmZcSTbHhqvQ4Bvhr3ldm45P4VljhIfLRgMY/smYAdeOoRk0sPkGZUSIE/z7+w 4zzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776272505; x=1776877305; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=H4wsvwRJbqgcLtEjgPuYkCxpD7ZHOSfOGTS4tTJGptA=; b=emJ+FKCzDVCZBeN1NLU0d2Y1jQxqFZNm5ymZc5xOTD9eEhdmRF6jyW0QZyopGNdnxU /TLXVi67/hgxNjsJaat5cQdyJ6/rPYLzhUojYhzxtt657H55Jns7w/2mO7ByVNK1tmOE 7ZPmqzRm2ffRJM/GRZLJQJEwmtz/e8J7yRm2/5z5viyFQecKVwycEA28YFqjG+qZzFVI W7xzqpZ4FFT5i8VG5RFGDIfLEgweMh1lcy9DJ4imAyuv7qgMdadlH797q5w3TNFZbBZk AP4pEACyOdtSH1dsaswOFdLdeL30BQ/v3NVHrcP4yN3RfvcvSvd1hnuavwburJMZzhVY 05/g== X-Gm-Message-State: AOJu0YwUD+YM9pCb+DnhnUQg2+f4rdiL/VjKq1b1FFBsX3iJBwkwuQmx EYTF+W+dOd5KyQIaMwvPLYY6i1AF+JG7LMDLiO1ZmZvBEvvmLZLKOpZXcJfg6TVjY4bX7iAb/kR 0z30KxOR/Su7OKxWEXvV5aHw7BojgA25pSYFN/xkbQAqSIWX+t04bbMNT/U7VTqOZbA== X-Gm-Gg: AeBDietjlwnJI27P2d5WubwYFQF6KeM1eTbs4Eomt0YXlcjBjzIGrUuPCxrwKkhmVoz fYASuGG1s3qS+6VUJPWgxa5kk4Js6mszJY8D0dh5Xqw/VslP5jKxXYJVngJA/lZabiopVBgsghZ 6khMwyVnAy7IU34Fxw1bIARzm8YHlGkEsFl0WAmqY6RuVWUH1OWouef9SLTva5RjLKJIXCq957I +l8T0kTaxoB3HXFDdjQ/WQADivd3S6lOZ0wtiASFttoKVqysMOksITdjzIc0O2FScR+A4s0FJSm zJIJDqEPqJZyY+zoNQJ97xQw/kCVvzVRJ6+F99lUaTrTx6GEy1qYL9I6dOX0T1waa0MMNJxQwJF NBEOmyPBo6EUy3T0VeXgdTOhu2z7MQe+n4UCbbWT1RTAiixyFGN7VuKOY/5QCZpihkw== X-Received: by 2002:ac8:5ac4:0:b0:50d:e298:8f1c with SMTP id d75a77b69052e-50defc74506mr255459711cf.32.1776272493321; Wed, 15 Apr 2026 10:01:33 -0700 (PDT) X-Received: by 2002:ac8:5ac4:0:b0:50d:e298:8f1c with SMTP id d75a77b69052e-50defc74506mr255421761cf.32.1776272458487; Wed, 15 Apr 2026 10:00:58 -0700 (PDT) Received: from cluster.. (4f.55.790d.ip4.static.sl-reverse.com. [13.121.85.79]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-50e1af9dc5fsm16817841cf.16.2026.04.15.10.00.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Apr 2026 10:00:57 -0700 (PDT) From: Alex Markuze To: ceph-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, idryomov@gmail.com, vdubeyko@redhat.com, Alex Markuze Subject: [PATCH v2 0/7] ceph: manual client session reset Date: Wed, 15 Apr 2026 17:00:36 +0000 Message-Id: <20260415170043.3882912-1-amarkuze@redhat.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This series adds operator-initiated manual client session reset for CephFS, providing a controlled escape hatch for client/MDS stalemates in which caps, locks, or unsafe metadata state stop making forward progress. Motivation ---------- When a CephFS client enters a stalemate with the MDS -- stuck cap flushes, hung file locks, or unsafe requests that cannot be journaled -- the only current recovery options are client eviction from the MDS side or a full client node restart. Both are disruptive and can cascade to other workloads on the same node. Manual reset gives the operator a targeted tool: block new metadata work, attempt a bounded best-effort drain of dirty client state while sessions are still alive, then tear sessions down and let new requests re-open fresh sessions. State that cannot drain (the stuck state causing the stalemate) is force-dropped -- that is the point of the reset. Design ------ The reset is triggered via debugfs: echo "reason" > /sys/kernel/debug/ceph//reset/trigger cat /sys/kernel/debug/ceph//reset/status The state machine tracks four phases: IDLE -> QUIESCING -> DRAINING -> TEARDOWN -> IDLE QUIESCING is set synchronously by schedule_reset() before the workqueue item is dispatched. This provides immediate request gating from the caller's context -- new metadata requests and file-lock acquisitions block the moment the operator triggers the reset, with no race window between scheduling and the work function starting. All non-IDLE phases block callers on blocked_wq; the hot path adds only a single READ_ONCE per request. The drain phase (bounded at 5 seconds) flushes the MDS journal, dirty caps, and pending cap releases. Non-stuck state drains in milliseconds; stuck state times out and is force-dropped during teardown. The session teardown follows the established check_new_map() forced-close pattern: unregister sessions under mdsc->mutex, then clean up caps and requests under s->s_mutex. Reconnect is not attempted because the MDS only accepts CLIENT_RECONNECT during its own RECONNECT phase after restart, not from an active client. A SESSION_REQUEST_CLOSE is sent to each MDS before local teardown so the MDS can release server-side state promptly rather than waiting for session_autoclose timeout. Blocked callers are released when reset completes and observe the final result. The wait path verifies the phase is still IDLE under the lock after wakeup, looping back if a new reset was scheduled in the interim. Patch breakdown --------------- Prep / cleanup: 1. Convert all CEPH_I_* inode flags to named bit-position constants and switch all flag modifications to atomic bitops (set_bit, clear_bit, test_and_clear_bit). The previous code mixed lockless atomics with non-atomic read-modify-write on the same unsigned long, which is a correctness hazard. Flag reads under i_ceph_lock that only test lock-serialised flags retain bitmask tests. 2. Fix a __force endian cast in reconnect_caps_cb() to use the proper cpu_to_le32() macro and the new test_bit() accessor. Hardening / diagnostics: 3. Harden send_mds_reconnect() with error return, early bailout for closed/rejected/unregistered sessions, state restoration on transient failure. Rewrite mds_peer_reset() to handle active-MDS (past RECONNECT phase) by tearing the session down locally. 4. Convert wait_caps_flush() to a diagnostic timeout loop that periodically dumps pending flush state, improving observability for reset-drain stalls and existing sync/writeback hangs. Core feature: 5. Add the reset state machine, request gating, session teardown work function, scheduling, and destroy-path coordination. 6. Add the debugfs trigger/status interface and four tracepoints (schedule, complete, blocked, unblocked). Testing: 7. Add kselftest-integrated shell tests: stress testing with concurrent I/O and random-interval reset injection, targeted corner cases (overlapping resets, dirty data across reset, stale locks, unmount during reset), a five-stage validation wrapper with per-stage timeouts, and a post-run data integrity checker. All 5 validation stages pass on a real CephFS cluster. Changes since v1 ----------------- - Patch 1 now converts ALL flag modifications to atomic bitops, eliminating the mixed atomic/non-atomic RMW hazard on the shared i_ceph_flags field. Removed mask defines for flags that are only accessed via the _BIT form. Fixed Co-authored-by -> Co-developed-by. - Rewrote mds_peer_reset() to handle active-MDS state correctly instead of sending a doomed reconnect (patch 3). - Added state restoration in send_mds_reconnect() failure path to prevent sessions stranding in RECONNECTING (patch 3). - Added early bailout checks in send_mds_reconnect() for closed/rejected/unregistered sessions (patch 3). - Added diagnostic timeout loop to wait_caps_flush() (patch 4). - Patch 5 commit message now documents the QUIESCING phase purpose (synchronous request gating before async work dispatch). - Fixed TOCTOU in ceph_mdsc_wait_for_reset(): the phase is now re-verified under the lock after wakeup, with a deadline-based retry loop if a new reset was scheduled in the interim (patch 5). - Added tracepoints for reset lifecycle events (patch 6). - Added selftest validation harness with kselftest integration (patch 7). - Rebased onto v7.0. Alex Markuze (7): ceph: convert inode flags to named bit positions and atomic bitops ceph: use proper endian conversion for flock_len in reconnect ceph: harden send_mds_reconnect and handle active-MDS peer reset ceph: add diagnostic timeout loop to wait_caps_flush() ceph: add client reset state machine and session teardown ceph: add manual reset debugfs control and tracepoints ceph: add manual reset selftests and validation harness MAINTAINERS | 1 + fs/ceph/addr.c | 16 +- fs/ceph/caps.c | 29 +- fs/ceph/debugfs.c | 104 +++ fs/ceph/file.c | 12 +- fs/ceph/inode.c | 5 +- fs/ceph/locks.c | 38 +- fs/ceph/mds_client.c | 656 ++++++++++++++++- fs/ceph/mds_client.h | 44 +- fs/ceph/snap.c | 2 +- fs/ceph/super.h | 73 +- fs/ceph/xattr.c | 2 +- include/trace/events/ceph.h | 63 ++ tools/testing/selftests/Makefile | 1 + .../selftests/filesystems/ceph/Makefile | 7 + .../selftests/filesystems/ceph/README.md | 84 +++ .../filesystems/ceph/reset_corner_cases.sh | 646 ++++++++++++++++ .../filesystems/ceph/reset_stress.sh | 694 ++++++++++++++++++ .../filesystems/ceph/run_validation.sh | 350 +++++++++ .../selftests/filesystems/ceph/settings | 1 + .../filesystems/ceph/validate_consistency.py | 297 ++++++++ 21 files changed, 3025 insertions(+), 100 deletions(-) create mode 100644 tools/testing/selftests/filesystems/ceph/Makefile create mode 100644 tools/testing/selftests/filesystems/ceph/README.md create mode 100755 tools/testing/selftests/filesystems/ceph/reset_corner_cases.sh create mode 100755 tools/testing/selftests/filesystems/ceph/reset_stress.sh create mode 100755 tools/testing/selftests/filesystems/ceph/run_validation.sh create mode 100644 tools/testing/selftests/filesystems/ceph/settings create mode 100755 tools/testing/selftests/filesystems/ceph/validate_consistency.py -- 2.34.1