From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-vk1-f170.google.com (mail-vk1-f170.google.com [209.85.221.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4FB23446A5 for ; Wed, 11 Feb 2026 03:44:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=209.85.221.170 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770781475; cv=pass; b=ZMN11+9ZHtI4o5hyHELs5LdOJ1t1km8cO+ElteEsAuNqhJx4PIRKoiuib7Ql/Xtz3CPUV6u6GTpDjv8+ZId22sAqFBin/zZNiAKQYWPBEmyzX36FU92abmK2nS8wm800DE4MQV7IEiFJLKhxF+BkZAiWC5kvJ5+KIhhwg8puZ+w= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770781475; c=relaxed/simple; bh=Uk3oOZvUm5ZxGsguhQj+llJBjauC9cXvjNA+Al734aE=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=iJESBLRLPIYSoveEQsFhzbgedSbQ6YS/Lzp+hjdMjZ05rPEWmajIXDVqjCnZX8qCrQzyqxIY2o5OXahexbzIWS+JAKG1XiZ8jXeqkH7TG6sxRAuI4/uxNnplRb8aF2USj2Ihavx017L+T55Iw8aNTcRJZoFyE8xdJgsJjVKGhqk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=A9+MHxJu; arc=pass smtp.client-ip=209.85.221.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="A9+MHxJu" Received: by mail-vk1-f170.google.com with SMTP id 71dfb90a1353d-56637565faaso1244813e0c.1 for ; Tue, 10 Feb 2026 19:44:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1770781472; cv=none; d=google.com; s=arc-20240605; b=Iw0yqrlZMqEyaRkEQ9XjEavzTwUQsiCerqKHO7h+mSaC7uv/ua8YImgmL57q0Pzm+Q RFUh07zZwzZi2vUCUsUfb9AyMv6My823NanHvj1BlYbmyFRuK0wF5V6nWh12NvaYjGTF dJARw7pn9ualYNCt/FcwF8zHJAYUjqTUeTYY2Vob3U+mtY9oJtXJ6v3bT3AScQJl7gB5 rkms3qqda9ByIBGqMk3L8WvyLwS66ll4izFq3teXLhf4eslsgFM7SliedhsWfEkXv/nY g4dpNFj7UUJ9tKFB4sJ8zPJldcB2MKIS69vszC6kiI6kKRSq/wsW1WL24cU+DWOCKf9H v6XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=bkfps1sGjuiHIHlEXIXeU8rsUSF5ZRZi+4yZM9t5aRg=; fh=whKVu4zbO5cdeug3yzLz9tdbWQKzBVzJI63wA09/CGo=; b=CCJw1MMYB+mVx6Hw0VK/W0zS2VYa+7ryTouHJ4V6SA/R52qBduaoMt/SiwM31Mg0sp yxqASImd0wPCNfimBnqaAirIZBqn0sWjqc/gd5UFzjiTMonTW4MQpiDLT29t2yGRMSCa vRbpreYoktLbWidm9FC/Q2ZRxTvY+t1PZIAj4KSiAHZB1wa4fu2U0CvwdlxBa8v3iA2x sr1gtUBe5WH2w5+BxwMTBoSSVpECI3Mm2DIfjNJfQxlWrlnbwgh9mji8av8x7lz+PVp4 OvXaZ9d8wWjrZm5oLWsq05VoiGbqibXIk5dgucxwtyli5f9bAQI0Fem5XheigX919iog eT5w==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1770781472; x=1771386272; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=bkfps1sGjuiHIHlEXIXeU8rsUSF5ZRZi+4yZM9t5aRg=; b=A9+MHxJuaGJFq8zRO3qIPuiO7E7/ZSCTrxMaIibi1e+rvBScgUojSAt2MF6ne9sR5J QBpVxctS+Me3w3zY8B1vn0SyWFYtG07xEDdMaVY89H8JgJB30d7X27TG1fvK3LLlC/CC tPU0cOhAyT6SPNS1B/QNy6I2LMxfkxvuwDL7i2uttjjzOsjsMA+7AzyaHPB5LAKmchos R/cUZi43hbzuzCcDVUBbBIKhq42GAwfWMzb+R/Bp6NHdxKOUwnGfsqwuQ7Wb2oaqOPzT D/pRZOpu3UwO8FdQR+ANSxM0k8yMhrlLgrbQJcVDxcYX/v28JZFKq/dRTgCEZOzPgzQZ pqPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770781472; x=1771386272; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bkfps1sGjuiHIHlEXIXeU8rsUSF5ZRZi+4yZM9t5aRg=; b=UsnaAaMx1aRDB1B1E28wL+nypOdxeLIokxRd2MLOx3E7RNeFwXxX7VTSCRaAz4mVbi fkH4UAj578O/CHZiXQ9buyp7w1mQHR0lFUNqtBZV1my2VrRN3WoGO2D5WAyS3OkXWBhb WSsoT5iKFKDZhiJ3mA3g6VE4qjsfLfbseVxFXLm/LAaxpqpNQm5IRU0d+mhuoB7xepri cf1g71O2AnBLBAri61geI1MNS2Omyj9lkVl2x/QXbkVwXMTdls9GsePRW1+0ZF2HWLOY 4cZh4noeUWc1Szieozn1lQwnjiGJeYJMygu2qd92HmlmMK6jHqvyw4iw+1kJytrzKbX3 rhlw== X-Forwarded-Encrypted: i=1; AJvYcCVpWO7tqHKmtTTRhn9jus5Qdoa5obwawVGnUV31zG+2kSf1dTdjAah/Kp7PA6DDrvXzQnaUUrzRcUZfr2U=@vger.kernel.org X-Gm-Message-State: AOJu0Yz14ENnc+iabWyumyStmHEF8imn8aufnN4Ry63t+eVD4kz+6nfa 1SuQlGKwSdDZtV42bBjhVpibUwoN46wllUweyMijmRElNy+pjeJPdWdKNNeWA9fEwhwo950hwb+ U111SoHBk+HtjXe0KWAFDIFQubh72w6Oa/fD33VYJNw== X-Gm-Gg: AZuq6aL6YzlEKb7z5pprcDLyym+8ucNNqbkkta9AbCJXDjc2+bW53ACZa/WdRtgZkoP Pe5u0zajO2xj18jV4lK77/gu3OKeX+IqmUwHas6Cwp/JfnBb/dOa2NMjMOXT2JAShI+Z1wZe7pX mglZMLtCLkH1kwrmplPV1KBnxdbv8pjEQJJUdoTxrXCLOltjxvc4taGliILLRZ2sO+TTzQnX2Ni rw2jgCVKQ/FTvReZECR/A7wCPLjqFNhEDidPNnnOkpCvD3/63r8dtN3ZVOsjVTZfHZ2p6rpiRWE IFrT7vs= X-Received: by 2002:a05:6102:304e:b0:5f5:11d5:70fc with SMTP id ada2fe7eead31-5fae8a2590cmr6091828137.2.1770781472580; Tue, 10 Feb 2026 19:44:32 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260130223531.2478849-1-mkhalfella@purestorage.com> <20260130223531.2478849-9-mkhalfella@purestorage.com> <528a39a9-f087-461f-9f5b-c4b5821396b6@suse.de> <20260203200048.GE3729-mkhalfella@purestorage.com> <481562cd-0444-49db-8755-29436bec02de@suse.de> <20260204232456.GM3729-mkhalfella@purestorage.com> In-Reply-To: <20260204232456.GM3729-mkhalfella@purestorage.com> From: Randy Jennings Date: Tue, 10 Feb 2026 19:44:20 -0800 X-Gm-Features: AZwV_QjB1bjkiieHDQ482L0acS6wH-EjdjIfY69EtULgkTEqI55bpBcCdFjUJLc Message-ID: Subject: Re: [PATCH v2 08/14] nvme: Implement cross-controller reset recovery To: Mohamed Khalfella Cc: Hannes Reinecke , Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Aaron Dailey , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 4, 2026 at 3:24=E2=80=AFPM Mohamed Khalfella wrote: > > On Wed 2026-02-04 02:10:48 +0100, Hannes Reinecke wrote: > > On 2/3/26 21:00, Mohamed Khalfella wrote: > > > On Tue 2026-02-03 06:19:51 +0100, Hannes Reinecke wrote: > > >> On 1/30/26 23:34, Mohamed Khalfella wrote: > > [ .. ] > > >>> + timeout =3D nvme_fence_timeout_ms(ictrl); > > >>> + dev_info(ictrl->device, "attempting CCR, timeout %lums\n", timeou= t); > > >>> + > > >>> + now =3D jiffies; > > >>> + deadline =3D now + msecs_to_jiffies(timeout); > > >>> + while (time_before(now, deadline)) { > > >>> + sctrl =3D nvme_find_ctrl_ccr(ictrl, min_cntlid); > > >>> + if (!sctrl) { > > >>> + /* CCR failed, switch to time-based recovery */ > > >>> + return deadline - now; > > >>> + } > > >>> + > > >>> + ret =3D nvme_issue_wait_ccr(sctrl, ictrl); > > >>> + if (!ret) { > > >>> + dev_info(ictrl->device, "CCR succeeded using %s\n= ", > > >>> + dev_name(sctrl->device)); > > >>> + nvme_put_ctrl_ccr(sctrl); > > >>> + return 0; > > >>> + } > > >>> + > > >>> + /* CCR failed, try another path */ > > >>> + min_cntlid =3D sctrl->cntlid + 1; > > >>> + nvme_put_ctrl_ccr(sctrl); > > >>> + now =3D jiffies; > > >>> + } > > >> > > >> That will spin until 'deadline' is reached if 'nvme_issue_wait_ccr()= ' > > >> returns an error. _And_ if the CCR itself runs into a timeout we wou= ld > > >> never have tried another path (which could have succeeded). > > > > > > True. We can do one thing at a time in CCR time budget. Either wait f= or > > > CCR to succeed or give up early and try another path. It is a trade o= ff. > > > > > Yes. But I guess my point here is that we should differentiate between > > 'CCR failed to be sent' and 'CCR completed with error'. > > The logic above treats both the same. > > > > >> > > >> I'd rather rework this loop to open-code 'issue_and_wait()' in the l= oop, > > >> and only switch to the next controller if the submission of CCR fail= ed. > > >> Once that is done we can 'just' wait for completion, as a failure th= ere > > >> will be after KATO timeout anyway and any subsequent CCR would be po= intless. > > > > > > If I understood this correctly then we will stick with the first sctr= l > > > that accepts the CCR command. We wait for CCR to complete and give up= on > > > fencing ictrl if CCR operation fails or times out. Did I get this cor= rectly? > > > > > Yes. > > If a CCR could be send but the controller failed to process it somethin= g > > very odd is ongoing, and it's extremely questionable whether a CCR to > > another controller would be succeeding. That's why I would switch to th= e > > next available controller if we could not _send_ the CCR, but would > > rather wait for KATO if CCR processing returned an error. > > > > But the main point is that CCR is a way to _shorten_ the interval > > (until KATO timeout) until we can start retrying commands. > > If the controller ran into an error during CCR processing chances > > are that quite some time has elapsed already, and we might as well > > wait for KATO instead of retrying with yet another CCR. > > Got it. I updated the code to do that. It is not true that CCR failing means something odd is going on. In a tightly-coupled storage HA pair, hopefully, all the NVMe controllers will be able to figure out the status of the other NVMe controllers. However, I know of multiple systems (one of which I care about) where the NVMe controllers may have no way of figuring out the state of some other NVMe controllers. In that case, the log page entry indicates that the CCR might succeed on some other NVMe controller (and in these systems, I expect they would not be able to be particularly specific about which one). Very little time will elapse for that to happen. It is important for those systems to have a retry on another NVMe controller. Sincerely, Randy Jennings