From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEF7B15746F for ; Wed, 29 Apr 2026 12:52:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777467156; cv=none; b=d3jtmaCDGkEJqNjzv71xRlfNJgcLQFPq7bavPbQWls8AciLbY1pTHVt0u4A3Xf8xR1bmarT2n3RZOR0MbJAp3yB4NYmyn0+wJoejOlCiq1f/UHqpf+dJhcuVAEpRD8jJBQRJIOGNusaVbD17L815HdRP2HpVjkbSc6jdIyr7GUs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777467156; c=relaxed/simple; bh=gL/jSOB0IbbkKEWXnBLxDu/IoPYG7MLarbK3QKLi5pE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=THDgcaaUggHByXOdrVxkVb1mmMO8gtu6nG6jj59t33gMtuSy8qNeySOeRvTBgrIkffs+9lSQt97HOnVj+mWTwbVS0s065jpbTeS8rF+kGtHQYQXeUDO7YGe83F0kcXAGiAVrh27eXHAno0bg6CUXjeXTTGpOAYsQWyZWP5t/OrQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=P/6/vUdR; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=pneEepzA; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P/6/vUdR"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="pneEepzA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777467153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3wPDtDVExZ0W3N9cH87fjGtCkBCDVUlOYUQfuZUKnE4=; b=P/6/vUdRrzA02mGwWgpTJh60lMyGyxlnecFKLtfSWvtrr9u3tmfai/mDxRKD0DW33/wzZ1 yE7WQLTxQfCszqvHOzXdT1lKorUgVhcSlN7/3DB9OtNb9ykPHyicr8m+YTxwNqnshzNz2X VnNBbUkKq+EzJ1eul2Ksbf67bzvWgoo= Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-681-aPTeYlyONIqVKXVK_YRuzg-1; Wed, 29 Apr 2026 08:52:31 -0400 X-MC-Unique: aPTeYlyONIqVKXVK_YRuzg-1 X-Mimecast-MFC-AGG-ID: aPTeYlyONIqVKXVK_YRuzg_1777467151 Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-b844098869cso975374766b.2 for ; Wed, 29 Apr 2026 05:52:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1777467150; x=1778071950; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3wPDtDVExZ0W3N9cH87fjGtCkBCDVUlOYUQfuZUKnE4=; b=pneEepzAbSvU/c3xaoksQsDJoqkEBmW2cZzZt61f5nksxQYxsPlnABw/s6H+2MPvv4 g4uERlEgp0n2EhTLYWy1334tlBmc1YCDkEKHFKaRZQhlY9Clke1YrzGLTmMP6CJLMu6Z YwacyQngU4ZpHBDI0SP29DL4DWDryS+W+oWUQwPpUoepQM6Dz2j+7Wovi5pro9o+6AVv ov/UIN14lSojq2B/Stgmr+OyRALThCf+Hg7K+JmIumTSL8u2pZYnYsQPqNr9W95ekivY bDeKmPdApKujFY9JZ4gmtyDW2VdD/KMJLeQe95DDQjl9pRrGTjvXk+Ioiwu2mGp7C93Q XdMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777467150; x=1778071950; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3wPDtDVExZ0W3N9cH87fjGtCkBCDVUlOYUQfuZUKnE4=; b=ZDSLN6Ag6fl9zXm0lB0473GVlC7Qm4XPVwKZjpN3YDeChsmVcjLz+39V/WMK0WmKBV nwpT8Kuc/WA4KHr1A2eR72U6IO9h2OYkNC2uNj/hILBfuz60pJ2TqrgfoYSP/p5Txtve 6z9Yfj9qe2lCG5+LoVesy/cRJI4I9ZMa4kUgQlgE87VCJiXm2BQhCnd9Jb6JQ8qEo8dP bZIR7YSOkDbsIALsLNzi8mM/I2RKLKHIF4JUmZA0YHk1n6tXWnL2lOa6VQLA8Y1xg/A4 h2hOwE17h3RFfI3w6LB1mkwygeMwWPWgUfU+u9Z5UhJmmjn/uKen9TpXyd3iOosfCOvL TFcw== X-Gm-Message-State: AOJu0YwGwyRKl0RT0MpYnLg2LbpEwFPDldHgMvKgZtnWxSTIUdnG1aQD lRoElAxT6eaGxvfvCqOCyW4WTQsrvRLGlY4yeoDynTm/nBpF1M/+ifFt/1CZsl1knZjrlImncMB GVbJCnt7I9kBwrhye/Q+wwjn1raJdZKDm3EaZ2h/6djF5IfSVCsULZ7dzD3CczrbnRg== X-Gm-Gg: AeBDietOYHNX2bNfkaVY33ENzI6Tn5Do1uozWSh47SFchgntLyUw1RB09Bk1sMm3+Pv hHyPLl2/wTQLHAQKW1C0WZ2ezG5CMNJ5RkchdquTh6FiktYW2B30E+nTVeJFw93gDSaQtmAJ5bR pVk46vuRPI0LuYajrfhCUEGxabx9VQaL6V+4QnfOK977WyOBGAjcOk+n2wQZtI5nSOiLQSq9RjF DyJrQ2xA7+XUp1LWKYNGN4Effx9gyC6zpx8T5xhZWHoAA0q2x0RbGaNRG/XL2abc4ilfa85Sm6e 7MCNL5r1aaUqvcLTHTwTQXdnYxjEXEeM0WFleE7mJvwJWsp51qoiy0ZeGB1w/0IKeWi+DuCvGoy YZEU4mdDLobfz310Mp4mvcEuedncb59xaOY4//dMrZ/3ZFe7DkNLlrgnSkMKqKty04Q== X-Received: by 2002:a17:907:c783:b0:bab:f5c7:23ca with SMTP id a640c23a62f3a-bb93f04382emr222520966b.38.1777467149925; Wed, 29 Apr 2026 05:52:29 -0700 (PDT) X-Received: by 2002:a17:907:c783:b0:bab:f5c7:23ca with SMTP id a640c23a62f3a-bb93f04382emr222518266b.38.1777467149191; Wed, 29 Apr 2026 05:52:29 -0700 (PDT) Received: from cluster.. (4f.55.790d.ip4.static.sl-reverse.com. [13.121.85.79]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-67b22166a6esm680526a12.25.2026.04.29.05.52.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Apr 2026 05:52:28 -0700 (PDT) From: Alex Markuze To: ceph-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, idryomov@gmail.com, vdubeyko@redhat.com, Alex Markuze Subject: [PATCH v3 08/11] selftests: ceph: add reset stress test Date: Wed, 29 Apr 2026 12:52:03 +0000 Message-Id: <20260429125206.1512203-9-amarkuze@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260429125206.1512203-1-amarkuze@redhat.com> References: <20260429125206.1512203-1-amarkuze@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Add a single-client stress test for the CephFS manual session reset feature. The test runs concurrent I/O workers alongside periodic reset injection, then validates data integrity via validate_consistency.py. Supports four profiles (baseline, moderate, aggressive, soak) with configurable duration, reset interval, and worker counts. Signed-off-by: Alex Markuze --- .../filesystems/ceph/reset_stress.sh | 694 ++++++++++++++++++ 1 file changed, 694 insertions(+) create mode 100755 tools/testing/selftests/filesystems/ceph/reset_stress.sh diff --git a/tools/testing/selftests/filesystems/ceph/reset_stress.sh b/tools/testing/selftests/filesystems/ceph/reset_stress.sh new file mode 100755 index 000000000000..c503c75a5f7a --- /dev/null +++ b/tools/testing/selftests/filesystems/ceph/reset_stress.sh @@ -0,0 +1,694 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# CephFS reset stress test: +# - Runs concurrent I/O and rename workloads +# - Triggers random client resets through debugfs +# - Validates consistency and recovery behavior + +set -euo pipefail + +KSFT_SKIP=4 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# kselftest auto-detect: when invoked with no arguments (e.g. by +# "make run_tests"), find a CephFS mount automatically or skip. +if [[ $# -eq 0 ]]; then + MOUNT_POINT="$(findmnt -t ceph -n -o TARGET 2>/dev/null | head -1)" + if [[ -z "$MOUNT_POINT" ]]; then + echo "SKIP: No CephFS mount found and --mount-point not specified" + exit "$KSFT_SKIP" + fi + exec "$0" --mount-point "$MOUNT_POINT" +fi + +PROFILE="moderate" +DURATION_SEC="" +COOLDOWN_SEC=20 +FILE_COUNT=64 +IO_WORKERS="" +RENAME_WORKERS="" +MOUNT_POINT="" +OUT_DIR="" +CLIENT_ID="" +DEBUGFS_ROOT="/sys/kernel/debug/ceph" +SLO_SECONDS=30 +EXPECT_RESET=1 +DMESG_CMD="" +SUDO="" + +RESET_MIN_SEC=5 +RESET_MAX_SEC=15 + +RUN_ID="$(date +%Y%m%d-%H%M%S)" +WORKLOAD_FLAG="" +RESET_FLAG="" +DATA_DIR="" + +IO_LOG="" +RENAME_LOG="" +RESET_LOG="" +STATUS_LOG="" +STATUS_BEFORE="" +STATUS_FINAL="" +DMESG_LOG="" +SUMMARY_LOG="" +REPORT_JSON="" + +RESET_PID=0 +STATUS_PID=0 +declare -a IO_WORKER_PIDS=() +declare -a RENAME_WORKER_PIDS=() + +usage() +{ + cat < [options] + +Required: + --mount-point PATH CephFS mount point to test under + +Options: + --profile NAME baseline|moderate|aggressive|soak (default: moderate) + --duration-sec N Override profile runtime in seconds + --cooldown-sec N Workload drain time after injector stop (default: 20) + --file-count N Number of logical files (default: 64) + --io-workers N Number of concurrent I/O workers (profile default) + --rename-workers N Number of concurrent rename workers (profile default) + --out-dir PATH Artifact directory (default: /tmp/ceph_reset_stress_) + --client-id ID Ceph debugfs client id; auto-detect if one client exists + --debugfs-root PATH Debugfs Ceph root (default: /sys/kernel/debug/ceph) + --slo-seconds N Max allowed post-reset stall window (default: 30) + --no-reset Disable reset injector (baseline mode helper) + --help Show this message + +Examples: + $0 --mount-point /mnt/cephfs --profile moderate + $0 --mount-point /mnt/cephfs --profile aggressive --duration-sec 300 + $0 --mount-point /mnt/cephfs --profile baseline --no-reset +EOF +} + +now_ms() +{ + date +%s%3N +} + +set_profile_defaults() +{ + case "$PROFILE" in + baseline) + RESET_MIN_SEC=0 + RESET_MAX_SEC=0 + EXPECT_RESET=0 + : "${DURATION_SEC:=600}" + : "${IO_WORKERS:=1}" + : "${RENAME_WORKERS:=1}" + ;; + moderate) + RESET_MIN_SEC=5 + RESET_MAX_SEC=15 + : "${DURATION_SEC:=900}" + : "${IO_WORKERS:=2}" + : "${RENAME_WORKERS:=1}" + ;; + aggressive) + RESET_MIN_SEC=1 + RESET_MAX_SEC=5 + : "${DURATION_SEC:=900}" + : "${IO_WORKERS:=4}" + : "${RENAME_WORKERS:=2}" + ;; + soak) + RESET_MIN_SEC=5 + RESET_MAX_SEC=15 + : "${DURATION_SEC:=3600}" + : "${IO_WORKERS:=2}" + : "${RENAME_WORKERS:=1}" + ;; + *) + echo "Unknown profile: $PROFILE" >&2 + exit 2 + ;; + esac +} + +log_summary() +{ + local msg="$1" + printf '[%s] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$msg" | tee -a "$SUMMARY_LOG" +} + +discover_client_id() +{ + local candidates=() + local entry + + if [[ -n "$CLIENT_ID" ]]; then + if ! $SUDO test -d "$DEBUGFS_ROOT/$CLIENT_ID/reset"; then + echo "SKIP: reset debugfs not found for client-id=$CLIENT_ID" >&2 + exit "$KSFT_SKIP" + fi + return 0 + fi + + if ! $SUDO test -d "$DEBUGFS_ROOT"; then + echo "SKIP: Debugfs root not found: $DEBUGFS_ROOT" >&2 + exit "$KSFT_SKIP" + fi + + while IFS= read -r entry; do + $SUDO test -d "$DEBUGFS_ROOT/$entry/reset" || continue + $SUDO test -w "$DEBUGFS_ROOT/$entry/reset/trigger" || continue + candidates+=("$entry") + done < <($SUDO ls -1 "$DEBUGFS_ROOT" 2>/dev/null || true) + + if [[ ${#candidates[@]} -eq 1 ]]; then + CLIENT_ID="${candidates[0]}" + return 0 + fi + + if [[ ${#candidates[@]} -eq 0 ]]; then + echo "SKIP: No writable Ceph reset interface found under $DEBUGFS_ROOT" >&2 + exit "$KSFT_SKIP" + fi + + echo "SKIP: Multiple Ceph clients found (${candidates[*]}). Use --client-id." >&2 + exit "$KSFT_SKIP" +} + +init_dataset() +{ + local i + mkdir -p "$DATA_DIR/A" "$DATA_DIR/B" + + for ((i = 0; i < FILE_COUNT; i++)); do + printf 'seed logical_id=%05d ts_ms=%s\n' "$i" "$(now_ms)" > "$DATA_DIR/A/file_$(printf '%05d' "$i")" + done +} + +io_worker() +{ + set +e + local worker_id="$1" + local seq=0 + local id + local relpath + local abspath + local payload + local hash + local ts + + while [[ -f "$WORKLOAD_FLAG" ]]; do + id="$(printf '%05d' $((RANDOM % FILE_COUNT)))" + if [[ -f "$DATA_DIR/A/file_$id" ]]; then + relpath="A/file_$id" + elif [[ -f "$DATA_DIR/B/file_$id" ]]; then + relpath="B/file_$id" + else + sleep 0.02 + continue + fi + + abspath="$DATA_DIR/$relpath" + alt_relpath="" + if [[ "$relpath" == A/* ]]; then + alt_relpath="B/file_$id" + else + alt_relpath="A/file_$id" + fi + alt_abspath="$DATA_DIR/$alt_relpath" + payload="worker=${worker_id} io_seq=${seq} id=${id} ts_ms=$(now_ms)" + result="$( + python3 - "$abspath" "$alt_abspath" "$payload" <<'PY' +import hashlib +import os +import sys + +path = sys.argv[1] +alt_path = sys.argv[2] +payload = sys.argv[3] + +try: + fd = os.open(path, os.O_RDWR | os.O_APPEND) + actual = path +except FileNotFoundError: + try: + fd = os.open(alt_path, os.O_RDWR | os.O_APPEND) + actual = alt_path + except FileNotFoundError: + sys.exit(1) + +try: + os.write(fd, (payload + "\n").encode()) + os.fsync(fd) + os.lseek(fd, 0, os.SEEK_SET) + digest = hashlib.sha256() + while True: + chunk = os.read(fd, 1 << 20) + if not chunk: + break + digest.update(chunk) + print(actual + " " + digest.hexdigest()) +finally: + os.close(fd) +PY + )" || { + sleep 0.02 + continue + } + + actual_abspath="${result%% *}" + hash="${result#* }" + if [[ "$actual_abspath" == "$alt_abspath" ]]; then + relpath="$alt_relpath" + fi + + ts="$(now_ms)" + printf '%s,%s,%s,%s,%s\n' "$ts" "$seq" "$id" "$relpath" "$hash" >> "$IO_LOG" + seq=$((seq + 1)) + sleep 0.02 + done +} + +rename_worker() +{ + set +e + local worker_id="$1" + local seq=0 + local id + local src_rel + local dst_rel + local rc + local ts + + while [[ -f "$WORKLOAD_FLAG" ]]; do + id="$(printf '%05d' $((RANDOM % FILE_COUNT)))" + + if [[ -f "$DATA_DIR/A/file_$id" ]]; then + src_rel="A/file_$id" + dst_rel="B/file_$id" + elif [[ -f "$DATA_DIR/B/file_$id" ]]; then + src_rel="B/file_$id" + dst_rel="A/file_$id" + else + sleep 0.02 + continue + fi + + ts="$(now_ms)" + if mv -T "$DATA_DIR/$src_rel" "$DATA_DIR/$dst_rel" 2>/dev/null; then + rc=0 + else + rc=$? + fi + printf '%s,%s,%s,%s,%s,%s,%s\n' "$ts" "$worker_id" "$seq" "$id" "$src_rel" "$dst_rel" "$rc" >> "$RENAME_LOG" + seq=$((seq + 1)) + sleep 0.02 + done +} + +random_sleep_seconds() +{ + local min_sec="$1" + local max_sec="$2" + local wait_sec + local span + + span=$((max_sec - min_sec + 1)) + wait_sec=$((min_sec + RANDOM % span)) + sleep "$wait_sec" +} + +reset_injector() +{ + set +e + local trigger_path="$1" + local seq=0 + local ts + local reason + local rc + + while [[ -f "$RESET_FLAG" ]]; do + random_sleep_seconds "$RESET_MIN_SEC" "$RESET_MAX_SEC" + [[ -f "$RESET_FLAG" ]] || break + + ts="$(now_ms)" + reason="stress_${seq}_${ts}" + if echo "$reason" | $SUDO tee "$trigger_path" > /dev/null 2>&1; then + rc=0 + else + rc=$? + fi + printf '%s,%s,%s,%s\n' "$ts" "$seq" "$reason" "$rc" >> "$RESET_LOG" + seq=$((seq + 1)) + done +} + +status_sampler() +{ + set +e + local status_path="$1" + local ts + local kv_line + + while [[ -f "$WORKLOAD_FLAG" || -f "$RESET_FLAG" ]]; do + ts="$(now_ms)" + if $SUDO test -r "$status_path"; then + kv_line="$($SUDO awk -F': ' 'NF>=2 {gsub(/ /, "", $1); gsub(/ /, "", $2); printf "%s=%s;", $1, $2}' "$status_path")" + printf '%s,%s\n' "$ts" "$kv_line" >> "$STATUS_LOG" + fi + sleep 1 + done +} + +stop_pid_with_timeout() +{ + local pid="$1" + local name="$2" + local timeout="$3" + local waited=0 + + if [[ "$pid" -le 0 ]]; then + return 0 + fi + + while kill -0 "$pid" 2>/dev/null; do + if (( waited >= timeout )); then + log_summary "Timeout waiting for $name (pid=$pid), sending SIGTERM/SIGKILL" + kill -TERM "$pid" 2>/dev/null || true + sleep 1 + kill -KILL "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + return 1 + fi + sleep 1 + waited=$((waited + 1)) + done + + wait "$pid" 2>/dev/null || true + return 0 +} + +detect_privileges() +{ + if [[ -r "$DEBUGFS_ROOT" ]]; then + SUDO="" + elif sudo -n true 2>/dev/null; then + SUDO="sudo" + else + echo "WARNING: $DEBUGFS_ROOT is not readable and passwordless sudo is not available" >&2 + echo "WARNING: reset injection, debugfs status checks, and dmesg capture will not work" >&2 + fi + + if $SUDO dmesg > /dev/null 2>&1; then + DMESG_CMD="$SUDO dmesg" + else + DMESG_CMD="" + echo "WARNING: dmesg is not accessible; kernel errors (hung tasks) will not be detected" >&2 + fi +} + +check_dmesg() +{ + local start_epoch="$1" + + if [[ -z "$DMESG_CMD" ]]; then + return 0 + fi + + if ! $DMESG_CMD --since "@$start_epoch" > "$DMESG_LOG" 2>/dev/null; then + if ! $DMESG_CMD > "$DMESG_LOG" 2>/dev/null; then + log_summary "WARNING: dmesg capture failed unexpectedly" + return 0 + fi + log_summary "dmesg --since unsupported; captured full dmesg" + fi + + if grep -qi "hung task" "$DMESG_LOG" 2>/dev/null; then + log_summary "ERROR: kernel log contains 'hung task' during test window" + return 1 + fi + + return 0 +} + +cleanup() +{ + rm -f "$WORKLOAD_FLAG" "$RESET_FLAG" + local pid + for pid in "${IO_WORKER_PIDS[@]}" "${RENAME_WORKER_PIDS[@]}" "$RESET_PID" "$STATUS_PID"; do + [[ "$pid" -gt 0 ]] 2>/dev/null && kill "$pid" 2>/dev/null || true + done + wait 2>/dev/null || true +} + +parse_args() +{ + while [[ $# -gt 0 ]]; do + case "$1" in + --mount-point) + MOUNT_POINT="$2" + shift 2 + ;; + --profile) + PROFILE="$2" + shift 2 + ;; + --duration-sec) + DURATION_SEC="$2" + shift 2 + ;; + --cooldown-sec) + COOLDOWN_SEC="$2" + shift 2 + ;; + --file-count) + FILE_COUNT="$2" + shift 2 + ;; + --io-workers) + IO_WORKERS="$2" + shift 2 + ;; + --rename-workers) + RENAME_WORKERS="$2" + shift 2 + ;; + --out-dir) + OUT_DIR="$2" + shift 2 + ;; + --client-id) + CLIENT_ID="$2" + shift 2 + ;; + --debugfs-root) + DEBUGFS_ROOT="$2" + shift 2 + ;; + --slo-seconds) + SLO_SECONDS="$2" + shift 2 + ;; + --no-reset) + EXPECT_RESET=0 + shift + ;; + --help|-h) + usage + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + usage + exit 2 + ;; + esac + done +} + +main() +{ + local start_epoch + local trigger_path="" + local status_path="" + local final_rc=0 + local reset_enabled=0 + local i + + parse_args "$@" + + if [[ -z "$MOUNT_POINT" ]]; then + echo "--mount-point is required" >&2 + usage + exit 2 + fi + + if [[ ! -d "$MOUNT_POINT" ]]; then + echo "SKIP: Mount point does not exist: $MOUNT_POINT" >&2 + exit "$KSFT_SKIP" + fi + + if ! touch "$MOUNT_POINT/.ceph_reset_test_probe" 2>/dev/null; then + echo "SKIP: Mount point is not writable: $MOUNT_POINT" >&2 + exit "$KSFT_SKIP" + fi + rm -f "$MOUNT_POINT/.ceph_reset_test_probe" + + if ! command -v python3 > /dev/null 2>&1; then + echo "SKIP: python3 is required but not found in PATH" >&2 + exit "$KSFT_SKIP" + fi + + if ! stat -f -c '%T' "$MOUNT_POINT" 2>/dev/null | grep -qi ceph; then + echo "WARNING: $MOUNT_POINT does not appear to be a CephFS mount" >&2 + fi + + detect_privileges + + set_profile_defaults + if [[ "$EXPECT_RESET" -eq 0 ]]; then + PROFILE="baseline" + RESET_MIN_SEC=0 + RESET_MAX_SEC=0 + fi + + if ! [[ "$IO_WORKERS" =~ ^[0-9]+$ && "$RENAME_WORKERS" =~ ^[0-9]+$ ]]; then + echo "io-workers and rename-workers must be integers" >&2 + exit 2 + fi + + if [[ "$IO_WORKERS" -le 0 || "$RENAME_WORKERS" -le 0 ]]; then + echo "io-workers and rename-workers must be > 0" >&2 + exit 2 + fi + + if [[ -z "$OUT_DIR" ]]; then + OUT_DIR="/tmp/ceph_reset_stress_${RUN_ID}" + fi + mkdir -p "$OUT_DIR" + + WORKLOAD_FLAG="$OUT_DIR/workload.running" + RESET_FLAG="$OUT_DIR/reset.running" + + DATA_DIR="$MOUNT_POINT/ceph_reset_stress_${RUN_ID}" + mkdir -p "$DATA_DIR" + + IO_LOG="$OUT_DIR/io.log" + RENAME_LOG="$OUT_DIR/rename.log" + RESET_LOG="$OUT_DIR/reset.log" + STATUS_LOG="$OUT_DIR/status.log" + STATUS_BEFORE="$OUT_DIR/reset_status.before" + STATUS_FINAL="$OUT_DIR/reset_status.final" + DMESG_LOG="$OUT_DIR/dmesg.log" + SUMMARY_LOG="$OUT_DIR/summary.log" + REPORT_JSON="$OUT_DIR/validator_report.json" + + : > "$IO_LOG" + : > "$RENAME_LOG" + : > "$RESET_LOG" + : > "$STATUS_LOG" + : > "$SUMMARY_LOG" + + start_epoch="$(date +%s)" + + log_summary "Starting Ceph reset stress test" + log_summary "Profile=$PROFILE duration=${DURATION_SEC}s cooldown=${COOLDOWN_SEC}s file_count=${FILE_COUNT} io_workers=${IO_WORKERS} rename_workers=${RENAME_WORKERS}" + [[ -n "$SUDO" ]] && log_summary "Using sudo for privileged operations" + [[ -z "$DMESG_CMD" ]] && log_summary "WARNING: dmesg not available; hung task detection disabled" + log_summary "Artifacts=$OUT_DIR" + log_summary "Data dir=$DATA_DIR" + + init_dataset + + if [[ "$EXPECT_RESET" -eq 1 ]]; then + discover_client_id + trigger_path="$DEBUGFS_ROOT/$CLIENT_ID/reset/trigger" + status_path="$DEBUGFS_ROOT/$CLIENT_ID/reset/status" + if ! $SUDO test -w "$trigger_path"; then + echo "SKIP: Reset trigger is not writable: $trigger_path" >&2 + exit "$KSFT_SKIP" + fi + if ! $SUDO test -r "$status_path"; then + echo "SKIP: Reset status is not readable: $status_path" >&2 + exit "$KSFT_SKIP" + fi + $SUDO cat "$status_path" > "$STATUS_BEFORE" || true + reset_enabled=1 + log_summary "Using ceph client id: $CLIENT_ID" + fi + + trap cleanup EXIT INT TERM + + touch "$WORKLOAD_FLAG" + for ((i = 0; i < IO_WORKERS; i++)); do + io_worker "$i" & + IO_WORKER_PIDS+=("$!") + done + + for ((i = 0; i < RENAME_WORKERS; i++)); do + rename_worker "$i" & + RENAME_WORKER_PIDS+=("$!") + done + + if [[ "$reset_enabled" -eq 1 ]]; then + touch "$RESET_FLAG" + reset_injector "$trigger_path" & + RESET_PID=$! + + status_sampler "$status_path" & + STATUS_PID=$! + fi + + sleep "$DURATION_SEC" + + if [[ "$reset_enabled" -eq 1 ]]; then + rm -f "$RESET_FLAG" + stop_pid_with_timeout "$RESET_PID" "reset_injector" 20 || final_rc=1 + log_summary "Injector stopped; entering cooldown=${COOLDOWN_SEC}s" + fi + + sleep "$COOLDOWN_SEC" + + rm -f "$WORKLOAD_FLAG" + for i in "${!IO_WORKER_PIDS[@]}"; do + stop_pid_with_timeout "${IO_WORKER_PIDS[$i]}" "io_worker[$i]" 20 || final_rc=1 + done + for i in "${!RENAME_WORKER_PIDS[@]}"; do + stop_pid_with_timeout "${RENAME_WORKER_PIDS[$i]}" "rename_worker[$i]" 20 || final_rc=1 + done + + if [[ "$reset_enabled" -eq 1 ]]; then + stop_pid_with_timeout "$STATUS_PID" "status_sampler" 10 || final_rc=1 + $SUDO cat "$status_path" > "$STATUS_FINAL" || true + fi + + if ! check_dmesg "$start_epoch"; then + final_rc=1 + fi + + if ! python3 "$SCRIPT_DIR/validate_consistency.py" \ + --data-dir "$DATA_DIR" \ + --file-count "$FILE_COUNT" \ + --io-log "$IO_LOG" \ + --rename-log "$RENAME_LOG" \ + --reset-log "$RESET_LOG" \ + --status-final "$STATUS_FINAL" \ + --slo-seconds "$SLO_SECONDS" \ + --report-json "$REPORT_JSON" \ + $( [[ "$reset_enabled" -eq 1 ]] && echo "--expect-reset" ); then + final_rc=1 + fi + + if [[ "$final_rc" -eq 0 ]]; then + log_summary "PASS: stress run completed successfully" + else + log_summary "FAIL: stress run detected one or more failures" + fi + + log_summary "Artifacts available in: $OUT_DIR" + exit "$final_rc" +} + +main "$@" -- 2.34.1