From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27F1B23AB8D for ; Wed, 4 Feb 2026 01:11:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770167465; cv=none; b=gOtvabavvEV7EykoaX9zTxh0Jfin7F0KpzG5HpQMNC/CKF5yD8HsYJ0fTtlGYh8Rof6VpQgEBj1YLxmll4byS6S/lYNVZ7RxXp8V58dvm6qd0kV9A8s3Av8SEI/55023bFc7AvOis9Agm80eD/2O48h8k8NZYVJmYClZzzuGnL0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770167465; c=relaxed/simple; bh=5CGXTffyKQYN2EQSuovGSuo5tBaEjWYyicWygZU7jaY=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=mPXpFpf8pfPOIfMzbZBzENcNpmJI/pYjXPSJ0xybBCe55+bkUp02Lf/61Hg9Qr5VzkIBV/V6ICPIumfF1qoUncE2jRJaSs8MdztFAXNbbXiQXD5g3VxorZi9y9c8IIPML9uYitWmlePeuU4svHCL/A69xnpLv4y4ervQVk1Q5co= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=SZOdm9yP; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=4CrIHP5X; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=KmlWaGj5; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=Nxy78gIf; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="SZOdm9yP"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="4CrIHP5X"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="KmlWaGj5"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="Nxy78gIf" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 25A9C5BCC6; Wed, 4 Feb 2026 01:11:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1770167461; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BN+nd2sukPj4ejAn84TBHhk2uc/tPxCtdCFrVp1KgMQ=; b=SZOdm9yPQ3LA4Cl5n0XN+Zmqhi0piGegfzQZJ4v8h9Th0r2EN3tkIFn4csv0tiRvOddgG/ e0oCgZAQYJMgFysy3OwgC15Iq2slmD8sKcuFDDBFYff61t8CbUk91zS2hnA1X6C5g2W2bY 4QTe8J6OyNtOqGPTDXSPwVsuwd+hIHk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1770167461; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BN+nd2sukPj4ejAn84TBHhk2uc/tPxCtdCFrVp1KgMQ=; b=4CrIHP5X/SKpMFjXMFb2zgZXE5cjCoNM50Eo+lrgLlibZwBl2+nCK0RCSxadXkcFj4uOxI PZyqTR3smQ9kDNCg== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1770167460; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BN+nd2sukPj4ejAn84TBHhk2uc/tPxCtdCFrVp1KgMQ=; b=KmlWaGj5oozLR0S/LmT/4qTqici50iduPDGx5lhgnzKiT7IDogjccAjQfH3kajlvy8Ua8l dC9p+yuCudhYqalvlMNXNStIenOIpYakgddfGiIjwmH9iqYDWdZxwiW765gl9/Pbr5E2pq Igd+oknrpXo1kw0k8u65r+olyg282Nc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1770167460; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BN+nd2sukPj4ejAn84TBHhk2uc/tPxCtdCFrVp1KgMQ=; b=Nxy78gIfiUgP6rsk5HY2Rr6o04mGIPb7s6dqU5dqUcHs8nfU8kkxpJXsnsmmtxK8IdyvR6 StE/jI0msNiDCsBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8CF153EA63; Wed, 4 Feb 2026 01:10:55 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id GQTmD5+cgmn3bQAAD6G6ig (envelope-from ); Wed, 04 Feb 2026 01:10:55 +0000 Message-ID: <481562cd-0444-49db-8755-29436bec02de@suse.de> Date: Wed, 4 Feb 2026 02:10:48 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 08/14] nvme: Implement cross-controller reset recovery To: Mohamed Khalfella Cc: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org References: <20260130223531.2478849-1-mkhalfella@purestorage.com> <20260130223531.2478849-9-mkhalfella@purestorage.com> <528a39a9-f087-461f-9f5b-c4b5821396b6@suse.de> <20260203200048.GE3729-mkhalfella@purestorage.com> Content-Language: en-US From: Hannes Reinecke In-Reply-To: <20260203200048.GE3729-mkhalfella@purestorage.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.30 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; URIBL_BLOCKED(0.00)[suse.de:mid,suse.de:email,imap1.dmz-prg2.suse.org:helo]; MIME_TRACE(0.00)[0:+]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_RATELIMITED(0.00)[rspamd.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; FREEMAIL_CC(0.00)[broadcom.com,gmail.com,nvidia.com,lst.de,kernel.dk,kernel.org,grimberg.me,purestorage.com,lists.infradead.org,vger.kernel.org]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email,imap1.dmz-prg2.suse.org:helo] X-Spam-Flag: NO X-Spam-Score: -4.30 X-Spam-Level: On 2/3/26 21:00, Mohamed Khalfella wrote: > On Tue 2026-02-03 06:19:51 +0100, Hannes Reinecke wrote: >> On 1/30/26 23:34, Mohamed Khalfella wrote: [ .. ] >>> + timeout = nvme_fence_timeout_ms(ictrl); >>> + dev_info(ictrl->device, "attempting CCR, timeout %lums\n", timeout); >>> + >>> + now = jiffies; >>> + deadline = now + msecs_to_jiffies(timeout); >>> + while (time_before(now, deadline)) { >>> + sctrl = nvme_find_ctrl_ccr(ictrl, min_cntlid); >>> + if (!sctrl) { >>> + /* CCR failed, switch to time-based recovery */ >>> + return deadline - now; >>> + } >>> + >>> + ret = nvme_issue_wait_ccr(sctrl, ictrl); >>> + if (!ret) { >>> + dev_info(ictrl->device, "CCR succeeded using %s\n", >>> + dev_name(sctrl->device)); >>> + nvme_put_ctrl_ccr(sctrl); >>> + return 0; >>> + } >>> + >>> + /* CCR failed, try another path */ >>> + min_cntlid = sctrl->cntlid + 1; >>> + nvme_put_ctrl_ccr(sctrl); >>> + now = jiffies; >>> + } >> >> That will spin until 'deadline' is reached if 'nvme_issue_wait_ccr()' >> returns an error. _And_ if the CCR itself runs into a timeout we would >> never have tried another path (which could have succeeded). > > True. We can do one thing at a time in CCR time budget. Either wait for > CCR to succeed or give up early and try another path. It is a trade off. > Yes. But I guess my point here is that we should differentiate between 'CCR failed to be sent' and 'CCR completed with error'. The logic above treats both the same. >> >> I'd rather rework this loop to open-code 'issue_and_wait()' in the loop, >> and only switch to the next controller if the submission of CCR failed. >> Once that is done we can 'just' wait for completion, as a failure there >> will be after KATO timeout anyway and any subsequent CCR would be pointless. > > If I understood this correctly then we will stick with the first sctrl > that accepts the CCR command. We wait for CCR to complete and give up on > fencing ictrl if CCR operation fails or times out. Did I get this correctly? > Yes. If a CCR could be send but the controller failed to process it something very odd is ongoing, and it's extremely questionable whether a CCR to another controller would be succeeding. That's why I would switch to the next available controller if we could not _send_ the CCR, but would rather wait for KATO if CCR processing returned an error. But the main point is that CCR is a way to _shorten_ the interval (until KATO timeout) until we can start retrying commands. If the controller ran into an error during CCR processing chances are that quite some time has elapsed already, and we might as well wait for KATO instead of retrying with yet another CCR. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich