From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 676F129CEB for ; Thu, 13 Nov 2025 20:04:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763064270; cv=none; b=fj4jFz2nLpPlKGS/pydXGdl97E2zmZw9dNLrDCi0uJjilljlfoTxawtJ1r3lFXSsvoNFEMUsTQcy63r4dNAWm90uzOzGb3e5nusSw+Z4zXCZu76WTzhV15ZS4v/847SieNHb3FT/eMKJM2GtF88JiS5piWMG21go+KTR04bZWhQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763064270; c=relaxed/simple; bh=kOIdcG5l5q8f1fqAk7n+AxnjDIxkOxGAnR1mUTt0teY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=JjhnfInXKdIj1aX14K3XfERePwc1cu7CRdhvpG1zMQp78iOHADILcleJlPm2YA0H9FYgvp+qjoXyuTK34C8OJKcnp52Ez3VfUCgSgduIUNkPqvhCiToFpe5IM+7w+mH4Ql6971YcykRULZIJlxrKIe/FmEQ3IO6AmvyQJMAMjQc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mo5XgosM; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mo5XgosM" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-87c13813464so16953636d6.1 for ; Thu, 13 Nov 2025 12:04:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763064267; x=1763669067; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Tu/gd2+Qo/3wWM34T9vyVaIbJm5PzirrTBwTyzzXvE=; b=mo5XgosMeGioMkFFRfquSYQ4/yyZk8SOsiEbOXYmg1BPwwz1UqefpXyRcimKTgMESG ONIlyU7SCAPIiuj6C8E17zWJItlRFBeSbNtIVEVxXmmqXRLdr14SMTG/jItbAmaPZdQn uv7p9t9UoO2ZzYm+VlN0a2siQwUafli+lByy2gAzijeADezTYg6EBC/0ooUUqOkYfAza xtzeHNhyU+7WAJCybVqs7JGyLNVtrRSiEmTU4AKe0VwG1ky66FsyNbKtPfTnu6kusiVc OWKKdu0Oq6EwA3wMEUs0H91tda522hWQIoPeTmzC6u40l4a5GCMSPg9TqgcraPffDN+o xlKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763064267; x=1763669067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3Tu/gd2+Qo/3wWM34T9vyVaIbJm5PzirrTBwTyzzXvE=; b=B+vH5Pip+NCchwyWZgnOzZALABq7v0s5R+2XehQJbLCqIu3Eo+80s11zIn+cuShqGA dyBMyQEW7OYNycPSdov+ihn1cEWd7juLFPHKhwxnfJHwEIDR6us8DGZDe5xdmMfNxEkm /u99TzXbhken8dvLqVVh3tKZuBtqMWj5h2IJ+xzdwB7KASPTxduOb5mjHQuhEqvebjr2 b4cBYOtAapxtp3HzftWqzoh9ao8nJKjTqpKruOXqGq01a3QOpHurACQWSFJs0SoHShI8 9Ktauu+hYdGxmNHtuqwTupU1APSlC8tzx8HurHEsBLpAocCCs2kL8mtnKT6kuELa7QN0 B1Gw== X-Forwarded-Encrypted: i=1; AJvYcCWC/+5m9O+BCeL41DmiljfuLhYbWIV2zmwDGqYES6BIqV0A/CqPxDfecJAobbjnhEhzDrDgjI4eqxzXNOo=@vger.kernel.org X-Gm-Message-State: AOJu0YzF9xrcfKS+FOBLjYNPaL04xyAfoveDaRrehp3IZd1LL/byyBGC lhjx+BXKaZE4b6PjKZP2he74ESXSiL6KzrCsEIUSNij207Hp0cmXMv6X X-Gm-Gg: ASbGnctFNLODWH25Le7mu7vlHjIAwvEHNqmrYT67V6vqBahdH9jwRCLY+TVANdfA0Rl 2lRPFqpovPNEkhRc1v/FG+NwpYcctMtXkvIAmvEcbz9LXMzrGAkyD097Hm3xEu+t6nejccA37NM 5sm/1J4G2rUzyPX1gEJ6wzoy0HqKPlW9/FsXTK+TUeCjoq+0WZ6ijmahp4+dll/lKcthQk7uCA9 dptP1wbw+rKW/eh3ZXuCCz/dLMq7YOZBvOF3s/xxI8UTHP07cF0P7daVH3AC26dMMd1HJAAt5Q9 XM62Eo0oIAMFaeb3LbHbmI9/RWzF9zWo0Pnc52WFi4nwPoEECiksbu4oqkprBm3gEMIqaoDRZud TJXTCSAZ40NBnYR3HZyJPv+paU4xH33w63os24rNjZN+3GfxCH/1Y9YwjypqOZj57S0rbw/Hlsx fBW6StPLD4gRj2JzFkedMdkbNUnnUTmCrMpgZWAjyQ X-Google-Smtp-Source: AGHT+IGrx0FrDHr5X0Tz6kLKlVGflXMJORlOkolbSkhTGNZbsFYBJY1W9gQiUPtZK8T5GerMtFv9rw== X-Received: by 2002:a05:6214:2a8c:b0:882:5124:24b2 with SMTP id 6a1803df08f44-8829263b776mr6217716d6.32.1763064267020; Thu, 13 Nov 2025 12:04:27 -0800 (PST) Received: from sydsfwlab01-x2.apac.dell.com ([143.166.81.254]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-882863923fcsm17995966d6.25.2025.11.13.12.04.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Nov 2025 12:04:26 -0800 (PST) From: Stuart Hayes To: mkhalfella@purestorage.com Cc: adailey@purestorage.com, hare@suse.de, hch@lst.de, jmeneghi@redhat.com, jrani@purestorage.com, kbusch@kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, randyj@purestorage.com, sagi@grimberg.me, wagi@kernel.org, Shai.Dagieli@dell.com Subject: Re: [RFC PATCH v1 0/7] nvme-tcp: Implement TP4129 (KATO Corrections and Clarifications) Date: Thu, 13 Nov 2025 14:04:07 -0600 Message-Id: <20251113200407.398321-1-stuart.w.hayes@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20250324174909.3919131-1-mkhalfella@purestorage.com> References: <20250324174909.3919131-1-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2025-03-24 17:48 Mohamed Khalfella mkhalfella@purestorage.com wrote > Hello, > > RFC patchset implementing TP4129 (KATO Corrections and Clarifications) > for nvme-tcp. nvme-tcp was choosen as an example to demonstrate the > approach taken in the patchset. Other fabric transports, nvme-rdma and > nvme-fc, will be added including the feedback received in this RFC. > > TP4129 requires nvme controller to not immediately cancel inflight > requests when connection is lost between initiator and target. > Instead, inflight requests need to be held for a duration that is long > enough to allow target to learn about connection loss and quiesce > pending commands. Only then pending requests on the initiator side can > be canceled and possibly retried safely on another path. The main issue > TP4129 tries to address is ABA corruption that could happen if inflight > requests are tried immediately on another path. > > Requests hold timeout has two components: > > - KATO timeout is the time sufficient for target to learn about the > connection loss. It depends on whether command based or traffic based > keepalive is used. As per TP4129 the timeout is supposed to be 3 x KATO > for traffic based keepalive and 2 * KATO for command based keepalive. > > - CQT is the time needed by target controller to quiesce in flight nvme > commands after the controller learns about connection loss. > > On controller reset or delete cancel inflight requests if controller was > disabled correctly. Otherwise, hold the requests until it is safe to be > released. > > Jyoti Rani (1): > nvme-core: Read CQT wait from identify controller response > > Mohamed Khalfella (6): > nvmef: Add nvmef_req_hold_timeout_ms() to calculate kato request hold > time > nvme-tcp: Move freeing tagset out of nvme_tcp_teardown_io_queues() > nvme-tcp: Move freeing admin tagset out of > nvme_tcp_teardown_admin_queue() > nvme-tcp: Split nvme_tcp_teardown_io_queues() into two functions > nvme-core: Add support for holding inflight requests > nvme-tcp: Do not immediately cancel inflight requests during recovery > > drivers/nvme/host/core.c | 62 +++++++++++++++++++++++++++++++++++++ > drivers/nvme/host/fabrics.h | 7 +++++ > drivers/nvme/host/nvme.h | 5 +++ > drivers/nvme/host/tcp.c | 56 +++++++++++++++++++++------------ > include/linux/nvme.h | 4 ++- > 5 files changed, 114 insertions(+), 20 deletions(-) > > -- > 2.48.1 Verified the patch fixes the Data integrity issue for which TP4129 was written. Tested-by: Shai Dagieli (Shai.Dagieli@dell.com) Summary: The test included two parts: 1. Part 1: testing with original kernel without the patch to verify the test can consistently reproduce the data integrity issue. a. Result: the Data integrity was recreated. 2. Part 2: testing with a kernel version with the patch. a. Results: i. The data integrity issued was not recreated with the patch. ii. The messages seen on the storage log and dmesg are as expected based on the test error injection and the patch expected behavior and messages. 3. Conclusion: the patch fixes the data integrity issue. 4. Additional comments: a. TP4129 defines an extended delay before an IO for which the host did not receive a ‘command completion’ can be retried on a different path. Based on the test logs, the extended delay is applied in other cases where the IO is retried on another path, even when the host received command completion. For example, the extended delay before retry was seen when the status returned in the Command Completion was ‘status code type = 3h’ (Path Related Status) & ‘Status code = 00h’ (Internal Path Error). Applying the extended delay in cases where it is not required increases the potential performance impact of this patch and should be fixed. i. Recommendation: Modify the patch to limit the extended delay to the relevant cases: IO retry, on a different path, when the host DID NOT receive a Command Completion status. Execution overview - Test ran continuously for almost 24 hours with periodic SDT disruptions and host I/O validation. Findings - No DIs or I/O errors observed in Vdbench error logs (errorlog.html remained clean). - No host panics or abnormal behavior were reported. - dmesg shows expected TP4129 recovery messages, for example: [18:52:28] nvme nvme2: holding inflight requests for 15000ms [18:52:43] nvme nvme2: releasing inflight requests [18:52:43] nvme_cancel_request: 123 callbacks suppressed [18:52:43] nvme nvme2: Cancelling I/O ...