From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f48.google.com (mail-dl1-f48.google.com [74.125.82.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30F4C39B491 for ; Wed, 4 Feb 2026 23:24:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770247499; cv=none; b=hN7rhu4JpoAQBUFNe/LyH/JBBziyWFTdjzqbOZey4Z8jPg9468VUi9K6gOtfHWp9035qOZqgl3cQg/ASyxy2wdDiFkL4WB4ygP8d1yMGURm9bVUZL/n2dwQI5r8/IqXTJBL/+iV/iQqhUS8MGGBEJ4Gz67PewPMSLDLq/Qf3IOg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770247499; c=relaxed/simple; bh=Fnavj7d3kDSrMCNWpqeObxo6M0qtOyVY6Ja8Ew5MgVE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gZEiRRoAAczS2CJWUhShgv2m0M8UuJ5vkKruBHVKJs6Eh9yhDS62Srj5VM/A5Rcgeg+vtQAxtBShv6YEcm3lPhc9cHvtlydoslcPl2gW9hVQ0KFTwMTUbtydDkOf8RrGyAhgjxAvkW+VNHDg6VjKdIBX4Jju/EMw1D/eNahkZd0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=OerSVMjX; arc=none smtp.client-ip=74.125.82.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="OerSVMjX" Received: by mail-dl1-f48.google.com with SMTP id a92af1059eb24-126ea4e9694so789582c88.1 for ; Wed, 04 Feb 2026 15:24:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1770247498; x=1770852298; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=UFXDmM2WsnEcrtt2/f8D7d2IPqt9PDasWFuSkn4PWxo=; b=OerSVMjX0jzfkk7t7wv1RRUXl0Bctd2sZJaBG2fr/uiUpaD3CfekO0UhngRlWRnsAa E35ZQ7LnjFnoHoMwW8EIbjKsyASrd8d4m2BBCTmD3iC9OselRo6SSTzAYe2c/lO7sxNf IfxBYqzgj0QE502CBD6BSM+SHxf1Nx+4l4WapO3Kjp884O0BfLmc+p/Pn7HDSXs9LpP/ ip7RdT8IB84SQv0rv2iqtIBY7of3ficYskNQB2/hCR4NgkSmkrAyWh5cmuKXE2zrTIQV Af1ClE7fpiDlSUhIM5NxjCeuXVm3bWOI7DUx8kKa5hgNrJ6xMbIfKboyVyA+3drQwT/a /nLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770247498; x=1770852298; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UFXDmM2WsnEcrtt2/f8D7d2IPqt9PDasWFuSkn4PWxo=; b=hvJPWOrqGxlwvOCy3eg1pENnO1KbZ2iSFvRn+F/K8j4FmA4qf74dZNbdOuI0NuimHb 5HCTp++yL0Z7QUUsJYmarS36J5oI/DAIjm0I6Y/L1rTM2mVn7eku/zi19sHDK4ZjcsJH QcKrTto4O0OPkRdxxmn/caM1j3R6p58K3nHsHonBwWHjxDeb/oYJ9blZ297dISZhKWWC /KgFqYZ1EQCTPZkLZhDeKTzCzt3wOGGUrypigolXrxTmEPqKuHhUCywilYSE3EK2pN7E OBeJj7u2rKDbWCu0S/EhG1F7R8FcXVeXyfUlXkSt3pU6m+Sv0O0WiwvSgZhmhDGJSH3L zvEA== X-Forwarded-Encrypted: i=1; AJvYcCWpyF9/JLDrYtejH42FNX53UpTzZX5fOhiBBMvOgVadDAmPkVRlNPNoaBSQtrunlAP6qeRjLE6G2/iRC48=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2xDhweSUljO8DAIJHZQzQAjo/s7/Pmc6jQVRBDVwP9pfTMS9p gceWzLWAuYDtx3LXREpROH06KxBLAN/6/9BNW6ARnk6AdxsGM5p6EM89xuX+V1ne3JQ= X-Gm-Gg: AZuq6aIzU/ZzqPT812MSc+yN3OZ/XsQsZ0X3EcE7kadJPSe3b+5pv7VIzCr1vZ5WQnA Bb5vglyLBxpJ738x+dUkqY/KeeY5PA4HNUKpHmSYdEl7Pk+JtEUr5pSXpHaeodChnBSpu9Oi0rg 7lV1wGIrf6OR+eR3Pni/bhtp+3RIb7zVXThb+BAuYWFFPwptsNu3xyrJ98oLW8HdZbpYFQK/bES IpMBBWpzBBOzWIzwwai3lcPpyMtMUlcnGbUoFBn2axU5dfsMeZNp26P8INoagudBcWOPVixXn2A Izi8sW6nQpoq21d68sl1HV5ngqEQKWirxbQsyRqlB7Wg8pnRR7gpzGyGt4CZVIiPFdK3s0cENex F2ZSIaXh9gSGA6HexBNSBiAJT50I0R3e5VvKjkGzxZMEUiHWTvsjOJDM7PlKT78SN0ZrmuOAPM1 I/B6m1iVCXWyNjuiSK4p460h049WLSL3c= X-Received: by 2002:a05:7022:6ba1:b0:11b:b064:f606 with SMTP id a92af1059eb24-126f47bb0afmr1925009c88.26.1770247497991; Wed, 04 Feb 2026 15:24:57 -0800 (PST) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with UTF8SMTPSA id a92af1059eb24-126f4e04297sm3040651c88.3.2026.02.04.15.24.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Feb 2026 15:24:57 -0800 (PST) Date: Wed, 4 Feb 2026 15:24:56 -0800 From: Mohamed Khalfella To: Hannes Reinecke Cc: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 08/14] nvme: Implement cross-controller reset recovery Message-ID: <20260204232456.GM3729-mkhalfella@purestorage.com> References: <20260130223531.2478849-1-mkhalfella@purestorage.com> <20260130223531.2478849-9-mkhalfella@purestorage.com> <528a39a9-f087-461f-9f5b-c4b5821396b6@suse.de> <20260203200048.GE3729-mkhalfella@purestorage.com> <481562cd-0444-49db-8755-29436bec02de@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <481562cd-0444-49db-8755-29436bec02de@suse.de> On Wed 2026-02-04 02:10:48 +0100, Hannes Reinecke wrote: > On 2/3/26 21:00, Mohamed Khalfella wrote: > > On Tue 2026-02-03 06:19:51 +0100, Hannes Reinecke wrote: > >> On 1/30/26 23:34, Mohamed Khalfella wrote: > [ .. ] > >>> + timeout = nvme_fence_timeout_ms(ictrl); > >>> + dev_info(ictrl->device, "attempting CCR, timeout %lums\n", timeout); > >>> + > >>> + now = jiffies; > >>> + deadline = now + msecs_to_jiffies(timeout); > >>> + while (time_before(now, deadline)) { > >>> + sctrl = nvme_find_ctrl_ccr(ictrl, min_cntlid); > >>> + if (!sctrl) { > >>> + /* CCR failed, switch to time-based recovery */ > >>> + return deadline - now; > >>> + } > >>> + > >>> + ret = nvme_issue_wait_ccr(sctrl, ictrl); > >>> + if (!ret) { > >>> + dev_info(ictrl->device, "CCR succeeded using %s\n", > >>> + dev_name(sctrl->device)); > >>> + nvme_put_ctrl_ccr(sctrl); > >>> + return 0; > >>> + } > >>> + > >>> + /* CCR failed, try another path */ > >>> + min_cntlid = sctrl->cntlid + 1; > >>> + nvme_put_ctrl_ccr(sctrl); > >>> + now = jiffies; > >>> + } > >> > >> That will spin until 'deadline' is reached if 'nvme_issue_wait_ccr()' > >> returns an error. _And_ if the CCR itself runs into a timeout we would > >> never have tried another path (which could have succeeded). > > > > True. We can do one thing at a time in CCR time budget. Either wait for > > CCR to succeed or give up early and try another path. It is a trade off. > > > Yes. But I guess my point here is that we should differentiate between > 'CCR failed to be sent' and 'CCR completed with error'. > The logic above treats both the same. > > >> > >> I'd rather rework this loop to open-code 'issue_and_wait()' in the loop, > >> and only switch to the next controller if the submission of CCR failed. > >> Once that is done we can 'just' wait for completion, as a failure there > >> will be after KATO timeout anyway and any subsequent CCR would be pointless. > > > > If I understood this correctly then we will stick with the first sctrl > > that accepts the CCR command. We wait for CCR to complete and give up on > > fencing ictrl if CCR operation fails or times out. Did I get this correctly? > > > Yes. > If a CCR could be send but the controller failed to process it something > very odd is ongoing, and it's extremely questionable whether a CCR to > another controller would be succeeding. That's why I would switch to the > next available controller if we could not _send_ the CCR, but would > rather wait for KATO if CCR processing returned an error. > > But the main point is that CCR is a way to _shorten_ the interval > (until KATO timeout) until we can start retrying commands. > If the controller ran into an error during CCR processing chances > are that quite some time has elapsed already, and we might as well > wait for KATO instead of retrying with yet another CCR. Got it. I updated the code to do that. > > Cheers, > > Hannes > -- > Dr. Hannes Reinecke Kernel Storage Architect > hare@suse.de +49 911 74053 688 > SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg > HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich