From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEF2A3A1E72 for ; Thu, 1 Jan 2026 00:04:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767225853; cv=none; b=K7PBFI04sp4PQy+AW4H3A6T6Tla55PclmPndvDLRq/aOLi7wcgB3eH2834vPrqOJSQ794+U21/Khazm8lNmyPlTlRxvULf7NSdTJhm2DnvMcM6NaBSG1kR4fFYIvGFdqI2aIgayFd00gks+kCgrsxxzMwUfxtUZLPKwHd3YCLZQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767225853; c=relaxed/simple; bh=QB88Zg0Qg2Y4jfrIC6M0MynkrTSZjj7Q939VqbS8vwI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CKDfQfOPQfSHSZmAmuJW7NkPl0u7DLqtlpUd1rUr/e7mmUc17v9xVYuN5ArP7A9Aa7gzSyzehPDvyZO66JsCj4LPCxcoEFANRNSpLuYldMr18QlLtutaGPDfTrb/USSjMGFgu1klDpDN52Xc5lnbf/O6MXhgxPl3ScJvovcNK9Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=EcF3/wuA; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="EcF3/wuA" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-7b8bbf16b71so11194358b3a.2 for ; Wed, 31 Dec 2025 16:04:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1767225851; x=1767830651; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=bW+D11PmZg+BW3Ns46JZJEkxh91q6adf7Xop1mv+6C8=; b=EcF3/wuAymtHTqylKEccw7OdnWy2HLz737pYGqffr6aBKTEMYdbXqKIBRfnfo3Acqw 6yZgz2tNMJhP7j40Ci3OzOMDsTG95n9Vup1Ba5XPXyzuZa6r/XmlFjv3bL2KSAuB3CdH uf1iQ0cCqq411AqG3N4s4/7wCrBh8elkjSuCpEOar6Z1aY8+0eJMNtzY25+/NFgAgDd7 fHDWTxUpZzUfvf8+y5B9CeMGcsrKVyJJJrnFeiiB0GK/HUlq4tviNb1u930ELNIhzT9Y x9VGlz1XCcoIDey+evuA5KPxRSuybRXzKr9li6WUiSs7S68PqZWz/SmggOu1dayyiE25 IOTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767225851; x=1767830651; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bW+D11PmZg+BW3Ns46JZJEkxh91q6adf7Xop1mv+6C8=; b=w5JTR6gAAtQSYSwM0HN/G0E/f1wAmNPde2BlZjt8J20Ci9Ek0EHdPdui/MQujByjXJ yAbsBKCIEQz4ejh64IL589qC5NxedWPJKEUVmYk0JlM/AGUiO+LKlPX7Lo+n886e7aAT DjEOMzyhOcQ+nZEvs3mRNykKI9zpW8J1QwnG2waJhFkgfN7nUjsFFcz4tLReIrIj/QOh QzJFwXwe27Lln6FXdtBhb5h5TlWhuajRfwrMNWjOIw9alFNBZdOf/uLno2feRya1KNwB 8Cz0UAZfqzGJ+v/opDviCeVBBgvsQCjcHRv3l+J7+XoxsCFeJ7cHoAfPXPOLZyD0Ie2u uwEQ== X-Forwarded-Encrypted: i=1; AJvYcCW/17RbprVmZD6UU4LFGIPWbVgrr3nKPIcDRM0QDuZfEzyBRRDSDcu8KsQersYcIlyD/2aN7LW2vyQDMj0=@vger.kernel.org X-Gm-Message-State: AOJu0YxHs/L1NfnDP0/EZB1vHQ3eCJaKWh5ksDYX3g6wDKzgy5GqMp1s MLdfDEJn3UrbVOGvdx7LUWK+z12bVKF8Y30K29VVSOIgxK10GK+i/MbMjvC3a25QAJE= X-Gm-Gg: AY/fxX7xdPjQTO0Z7bCKX7FeJpzft7Rv9udyXKHx6kumlMRLZFhRNwf+MO1AOaQrFxz 6/WmJnzOe6YUjIemf3ijY3CXq1Nh5Kdv35YFS0tr2dzp8klkUnS5D4b4KQ6N1nWq3dvdUM7f94w A0/0k60jyote3eJpu7SvKq+tBIkgquFvsKgkHtbY6nAnFzUIHgpF4+1c8oJqUKzv9N6pSduqLLJ xduiJO43SBS0MCQKnCM5WniY0gdrchxZ+cdMu5vS1sOmB7bTPbUgJk1dCQTLe/YJ4MpdXBnJ5A2 5H5lBeqZkcDmr8tqmN8OBzdZRgWSr+ob5Ma5cnkJef8KXKhhmhuShAy7yhCs3GLO2hOFvSRFSDl HG4MUzOGe6vXaRPeE2uz7xlRbplAlbsdcol9HWh+so49d2qj4cHtTIQ6Ou/nbrG/7zRAgvV661V tCtmoqX4F/v8IPigRi+4HZYJDHV8B4M+k= X-Google-Smtp-Source: AGHT+IEuZfbioQuYzhI3YjMZrFOHh5dyQiwCSIzXX9mc086QtZB5nMrGCUFYrTJZvLPrV/MfJAab8Q== X-Received: by 2002:a05:7023:903:b0:119:e569:fbb2 with SMTP id a92af1059eb24-12172300472mr38435939c88.33.1767225850646; Wed, 31 Dec 2025 16:04:10 -0800 (PST) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with UTF8SMTPSA id a92af1059eb24-1217253c0c6sm160274051c88.12.2025.12.31.16.04.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Dec 2025 16:04:10 -0800 (PST) Date: Wed, 31 Dec 2025 16:04:09 -0800 From: Mohamed Khalfella To: Randy Jennings Cc: Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Aaron Dailey , John Meneghini , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error Message-ID: <20260101000409.GR3864520-mkhalfella@purestorage.com> References: <20251126021250.2583630-1-mkhalfella@purestorage.com> <20251126021250.2583630-11-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Thu 2025-12-18 18:06:02 -0800, Randy Jennings wrote: > On Tue, Nov 25, 2025 at 6:13 PM Mohamed Khalfella > wrote: > > > > An alive nvme controller that hits an error now will move to RECOVERING > > state instead of RESETTING state. In RECOVERING state ctrl->err_work > > will attempt to use cross-controller recovery to terminate inflight IOs > > on the controller. If CCR succeeds, then switch to RESETTING state and > > continue error recovery as usuall by tearing down controller and attempt > > reconnecting to target. If CCR fails, then the behavior of recovery > "usuall" -> "usual" > "attempt reconnecting" -> "attempting to reconnect" > > it would read better with "the" added: > "tearing down the controller" > "reconnect to the target" Updated as suggested. > > > depends on whether CQT is supported or not. If CQT is supported, switch > > to time-based recovery by holding inflight IOs until it is safe for them > > to be retried. If CQT is not supported proceed to retry requests > > immediately, as the code currently does. > > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c > > > +static int nvme_tcp_recover_ctrl(struct nvme_ctrl *ctrl) > > > + dev_info(ctrl->device, > > + "CCR failed, switch to time-based recovery, timeout = %ums\n", > > + jiffies_to_msecs(rem)); > > + set_bit(NVME_CTRL_RECOVERED, &ctrl->flags); > > + queue_delayed_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work, rem); > > + return -EAGAIN; > I see how setting this bit before the delayed work executes works > to complete recovery, but it is kindof weird that the bit is called > RECOVERED. I do not have a better name. TIME_BASED_RECOVERY? > RECOVERY_WAIT? Agree. It does look weird. If we agree to add two states FENCING and FENCED then the flag might not be needed. > > > static void nvme_tcp_error_recovery_work(struct work_struct *work) > > { > > - struct nvme_tcp_ctrl *tcp_ctrl = container_of(work, > > + struct nvme_tcp_ctrl *tcp_ctrl = container_of(to_delayed_work(work), > > struct nvme_tcp_ctrl, err_work); > > struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; > > > > + if (nvme_ctrl_state(ctrl) == NVME_CTRL_RECOVERING) { > > + if (nvme_tcp_recover_ctrl(ctrl)) > > + return; > > + } > > + > > if (nvme_tcp_key_revoke_needed(ctrl)) > > nvme_auth_revoke_tls_key(ctrl); > > nvme_stop_keep_alive(ctrl); > The state of the controller should not be LIVE while waiting for > recovery, so I do not think we will succeed in sending keep alives, > but I think this should move to before (or inside of) > nvme_tcp_recover_ctrl(). This is correct, no keepalive traffic will be sent in RECOVERING state. If we split fencing work from existing error recovery work then this should removed. I think we are going in that direction. > > Sincerely, > Randy Jennings