From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=pass header.oldest-pass=0 smtp.remote-ip=100.103.45.18 ARC-Seal: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=pass; t=1793097541; b=S2Z50BtL1TFY90sjWxWjRFGbm7PsIqRsrLNdQI83HAyFP1Vha/gmxYCLlxayZLsIy65P O2fxrR/LSqnLDqPZ76cVZhOpDkg9mPkSTHHHp6WpdepjsvGW842/7THns/Fh+x3i5jnfE dGwoqXxOzDxtUmmS5Oswc/hg5CvON+1URWlNXhYrwWSl/sbnsu5b+svBhdjpkwXlFmbGU la2lo2jjzRyEiXny12r4a5126E7h5BUZhFDYXXOIHZ3D4csGtds+ylcn/WZ2F512st/j4 ma9u2m7H4SHCdAfZaijiVnrbTjGofvOKzrYV0L20DDpwGC/AjX9sPi5cZ//6+cBLIfA== ARC-Message-Signature: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793097541; h=Received:DKIM-Signature:Received:From:Date:Subject:MIME-Version: Content-Type:Content-Transfer-Encoding:Message-Id:References: In-Reply-To:To:Cc:X-Mailer:X-Developer-Signature:X-Developer-Key: X-Endpoint-Received:X-Original-From:Reply-To; bh=5xCZMUf31kMXTmH3yaFMZ+E8GqAm2dXo9oDp4R3rRdY=; b=OV22/SJOv9o5Sj+6iMABlCK2YGmJJeNtynWcGyJeT/bcHIec7jHwgN6aVp13K8/iXJZu txr37ij0sAwuTuXOn3cX0OWFXLoG3tbeefhwJMCtmWIV+8chYXaTtrnVr0ka4xBM9Hdge 0FJDLpYY8VnM7IdQG9G12vNKWygQMMBdZcK0/O3fYvcap5eJEAahX6LKxsYnSILcMa2cV zWXJ6JNNbNLBhHm108o20s2ZCH4spCCD/RX9wfMap+Rk5H0ZfFlCbXcGaUzpe2sBE0eyn 44wkUb1alABqdbCSvTD1jvD/N30OWirTiR8KWJekHF5aP5ctlvKzsCrxdzCK8jiyreg== ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=b3zm2Pjp; arc=pass header.oldest-pass=0 smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08ACF60A2610 for ; Tue, 27 Oct 2026 10:39:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=b3zm2Pjp Authentication-Results: smtp.kernel.org; arc=none smtp.remote-ip=100.80.115.77 ARC-Seal: i=1; d=kernel.org; s=arc20260519; a=rsa-sha256; cv=none; t=1793097540; b=29ZBQY4REtDyLQVHSrWGa3ZtzZHVJVyxReqmQtW+wBfjlSBr7M9ybDrVvWoloW+JqRc7 ZLsk0Ra90lattXleu0BvdwTC5I+AdjG7KYcq78wzan1tJZUFeby1Ls8AdpP9gnq5srCTl DQz2BO/JDBL2zCNYfTuOFOS+aCS1yfzIzbVe870uDX7GXsP6mcjAp9dIzyIFZ279P4Lg1 UPM9c6Z/3feOmJQMURQmcqpX8LsWunVV3Brrk+SmvfP59sRiH++5eK/jgboh4zDUPQ1Qp lni8t8GQlmjk+nQ/0yuiLJduwPq9mG9DvfpVUtN57xYvF4q+EdCxn7aS2ddyjjjO2bw== ARC-Message-Signature: i=1; d=kernel.org; s=arc20260519; a=rsa-sha256; c=relaxed/relaxed; t=1793097540; h=DKIM-Signature:Received:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Mailer:X-Developer-Signature:X-Developer-Key:X-Endpoint-Received: X-Original-From:Reply-To; bh=5xCZMUf31kMXTmH3yaFMZ+E8GqAm2dXo9oDp4R3rRdY=; b=BTdNgHri005+FyvdwmU7BksPtLrz4/LZXVvOc9XJM36WtfvJpd7Ch+7+/TZYuN8SNz2+ 39Ue6moEIPps68+SlxvGvbfqfpDRI87g5N2wrh+zxEzDLg1iKES8CwFtzNilUmL5LYd6C FZvzt7NKkz/CkRH10kOarL7/BHjbKNkXmrf3IWtvogvWLKxnu/CgBk2o9QO87T3BttQMo RRz04kQxn4DTPSJqv5k4hTdccL61tuRs2WbBA8lhl6/rRYp9iMw30Lp/4xazXgjFhmcRe jUc3rMq4mBG5HGKg6qiwn65iT426rAd8CHJj9S/DtSdbsH9R1xLrr2kgYJ9AlZjlMqw== ARC-Authentication-Results: i=1; smtp.kernel.org; arc=none smtp.remote-ip=100.80.115.77 Received: from lkml.kernel.org (nexus.subspace.kernel.org [100.80.115.77]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.kernel.org (Postfix) with ESMTPS id C8FF51F031E1; Tue, 27 Oct 2026 10:38:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1793097540; bh=5xCZMUf31kMXTmH3yaFMZ+E8GqAm2dXo9oDp4R3rRdY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To; b=b3zm2Pjp62GRVAFoRvjGgk4zTGhHao/C0GrT09dKFS7N3WKciawFDfFqSBdz6MgkU tTMI29amA58y21KUFZ9sGHhsOWXLU4U4lEoPfn0uDgIDSlNjk4UcK6JUdF0xRd+PLT teo+nfC+Wy0Kg57noAn1KlyxeJsmywN3PUt6PS1RfLG+5q++1DJMqGpJtD/85FFz+z T5Qt0ap4PZpa8BFAXi6BYiG1MnHfWT+gKCxzmIdF+bAoiRR+iGKzTo8zqWBXQyBJGS kXxitRrq8nf+qA2S2j8QZ0+an8ZFcgguYd/u/ZfImCPY+JNtWxg4c2Ve8B+Q1KGUu5 tPFbxVFrh7S1g== Received: from nexus.subspace.kernel.org (localhost [IPv6:::1]) by lkml.kernel.org (Postfix) with ESMTP id 3ED9445093; Tue, 27 Oct 2026 10:38:59 +0000 (UTC) From: Martin Belanger via B4 Relay Date: Tue, 27 Oct 2026 06:38:53 -0400 Subject: [PATCH v2 2/2] nvme-tcp: skip TLS control records while waiting for the ICResp X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261027-nvme-tcp-ctrlrec-fixes-v2-2-9656395ebcf2@gmail.com> References: <20261027-nvme-tcp-ctrlrec-fixes-v2-0-9656395ebcf2@gmail.com> In-Reply-To: <20261027-nvme-tcp-ctrlrec-fixes-v2-0-9656395ebcf2@gmail.com> To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Hannes Reinecke Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Martin Belanger X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1793097538; l=6992; i=martin.nvme.dev@gmail.com; s=20261014; h=from:subject:message-id; bh=6vhImiUHY3qOfKIw4v4VLuZ5CbCy/ZozsM7ByGzixOY=; b=6SS0XhMJnZGPOJUztcGQwlfnhSnykB3DMAdhB14L1EYuWo3KD4povtuxNK3ncTljSplWu2VeC QpU5Lqx3JIbAAmnJGAdqqy7ltgb3wRSoWs8ob3eOMKJJF9t2Ihpywxd X-Developer-Key: i=martin.nvme.dev@gmail.com; a=ed25519; pk=4K+XXetGK35w2LzRRddd/bbHawZU222BN+YZRuSpN+k= X-Endpoint-Received: by B4 Relay for martin.nvme.dev@gmail.com/20261014 with auth_id=1176 X-Original-From: Martin Belanger Reply-To: martin.nvme.dev@gmail.com From: Martin Belanger A peer may send a NewSessionTicket as soon as the handshake completes, ahead of the ICResp. nvme_tcp_init_connection() fails the connection on any record that is not application data, so such a target cannot be reached at all. An OpenSSL-based target does this by default. RFC 9846 section 4.7.1 requires a client that does not support session resumption to ignore a NewSessionTicket. Discard such a record and read again, up to a fixed number of records. Report an alert, and fail the connection for any other record type. The record type is now checked before the length. A control record shorter than the ICResp was reported as -ECONNRESET, which hid the real cause. Fixes: 2837966ab2a8 ("nvme-tcp: control message handling for recvmsg()") Assisted-by: LLM Reviewed-by: Hannes Reinecke Signed-off-by: Martin Belanger --- drivers/nvme/host/tcp.c | 162 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 140 insertions(+), 22 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 0231b95fe4e4..2410223b1500 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -55,6 +55,12 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)"); #endif +/* read a TLS control record in chunks of this size */ +#define NVME_TCP_TLS_REC_CHUNK 256 + +/* TLS records read while waiting for the ICResp, the ICResp included */ +#define NVME_TCP_ICRESP_MAX_RECORDS 8 + static int nvme_tcp_cpu_queues[NR_CPUS]; static DEFINE_SPINLOCK(nvme_tcp_cpu_queues_lock); @@ -1547,12 +1553,143 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) mutex_destroy(&queue->pf_cache_lock); } +static int nvme_tcp_recv_tls_alert(struct nvme_tcp_queue *queue, + struct msghdr *msg, int len) +{ + struct sock *sk = queue->sock->sk; + u8 level, description; + + if (len < 2) { + dev_err(queue->ctrl->ctrl.device, + "queue %d: truncated TLS alert\n", + nvme_tcp_queue_id(queue)); + return -ENOTCONN; + } + + /* tls_alert_recv() reads through the iterator */ + iov_iter_revert(&msg->msg_iter, len); + tls_alert_recv(sk, msg, &level, &description); + + if (level == TLS_ALERT_LEVEL_FATAL) { + dev_err(queue->ctrl->ctrl.device, + "queue %d: fatal TLS alert %u\n", + nvme_tcp_queue_id(queue), description); + return -ENOTCONN; + } + + /* a peer shutting down normally is not worth a warning */ + if (description == TLS_ALERT_DESC_CLOSE_NOTIFY) + dev_dbg(queue->ctrl->ctrl.device, + "queue %d: TLS close_notify\n", + nvme_tcp_queue_id(queue)); + else + dev_warn(queue->ctrl->ctrl.device, + "queue %d: TLS alert %u\n", + nvme_tcp_queue_id(queue), description); + + return 0; +} + +/* + * Discard a TLS record that arrived in place of the ICResp. A peer may send + * a NewSessionTicket as soon as the handshake completes. + * + * Returns 0 when the record was discarded and the ICResp can be read, or a + * negative errno if the connection cannot continue. + */ +static int nvme_tcp_skip_tls_record(struct nvme_tcp_queue *queue, + struct msghdr *msg, u8 ctype, int len) +{ + u8 buf[NVME_TCP_TLS_REC_CHUNK]; + int ret; + + switch (ctype) { + case TLS_RECORD_TYPE_HANDSHAKE: + break; + case TLS_RECORD_TYPE_ALERT: + ret = nvme_tcp_recv_tls_alert(queue, msg, len); + return ret ? : -ENOTCONN; + default: + pr_err("queue %d: unhandled TLS record %u\n", + nvme_tcp_queue_id(queue), ctype); + return -ENOTCONN; + } + + /* the ICResp is in a later record, drain this one first */ + while (!(msg->msg_flags & MSG_EOR)) { + struct kvec iov = { + .iov_base = buf, + .iov_len = sizeof(buf), + }; + char cbuf[CMSG_LEN(sizeof(char))] = {}; + /* kTLS fails a control record read without a cmsg buffer */ + struct msghdr drain = { + .msg_control = cbuf, + .msg_controllen = sizeof(cbuf), + }; + + ret = kernel_recvmsg(queue->sock, &drain, &iov, 1, sizeof(buf), + 0); + if (ret <= 0) { + pr_warn("queue %d: failed to drain TLS record: %d\n", + nvme_tcp_queue_id(queue), ret); + return ret ? : -ECONNRESET; + } + msg->msg_flags = drain.msg_flags; + } + + return 0; +} + +/* + * Receive the ICResp. Over TLS, skip the records that a peer may send + * ahead of it. + * + * Returns the length received, or a negative errno. + */ +static int nvme_tcp_recv_icresp(struct nvme_tcp_queue *queue, + struct nvme_tcp_icresp_pdu *icresp) +{ + char cbuf[CMSG_LEN(sizeof(char))]; + struct msghdr msg; + struct kvec iov; + u8 ctype; + int i, ret; + + for (i = 0; i < NVME_TCP_ICRESP_MAX_RECORDS; i++) { + memset(&msg, 0, sizeof(msg)); + memset(cbuf, 0, sizeof(cbuf)); + iov.iov_base = icresp; + iov.iov_len = sizeof(*icresp); + if (nvme_tcp_queue_tls(queue)) { + msg.msg_control = cbuf; + msg.msg_controllen = sizeof(cbuf); + } + msg.msg_flags = MSG_WAITALL; + ret = kernel_recvmsg(queue->sock, &msg, &iov, 1, + iov.iov_len, msg.msg_flags); + if (ret <= 0 || !nvme_tcp_queue_tls(queue)) + return ret; + + ctype = tls_get_record_type(queue->sock->sk, + (struct cmsghdr *)cbuf); + if (ctype == TLS_RECORD_TYPE_DATA) + return ret; + + ret = nvme_tcp_skip_tls_record(queue, &msg, ctype, ret); + if (ret < 0) + return ret; + } + + pr_err("queue %d: too many TLS records before the ICResp\n", + nvme_tcp_queue_id(queue)); + return -EPROTO; +} + static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) { struct nvme_tcp_icreq_pdu *icreq; struct nvme_tcp_icresp_pdu *icresp; - char cbuf[CMSG_LEN(sizeof(char))] = {}; - u8 ctype; struct msghdr msg = {}; struct kvec iov; bool ctrl_hdgst, ctrl_ddgst; @@ -1590,16 +1727,7 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) goto free_icresp; } - memset(&msg, 0, sizeof(msg)); - iov.iov_base = icresp; - iov.iov_len = sizeof(*icresp); - if (nvme_tcp_queue_tls(queue)) { - msg.msg_control = cbuf; - msg.msg_controllen = sizeof(cbuf); - } - msg.msg_flags = MSG_WAITALL; - ret = kernel_recvmsg(queue->sock, &msg, &iov, 1, - iov.iov_len, msg.msg_flags); + ret = nvme_tcp_recv_icresp(queue, icresp); if (ret >= 0 && ret < sizeof(*icresp)) ret = -ECONNRESET; if (ret < 0) { @@ -1607,16 +1735,6 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) nvme_tcp_queue_id(queue), ret); goto free_icresp; } - ret = -ENOTCONN; - if (nvme_tcp_queue_tls(queue)) { - ctype = tls_get_record_type(queue->sock->sk, - (struct cmsghdr *)cbuf); - if (ctype != TLS_RECORD_TYPE_DATA) { - pr_err("queue %d: unhandled TLS record %d\n", - nvme_tcp_queue_id(queue), ctype); - goto free_icresp; - } - } ret = -EINVAL; if (icresp->hdr.type != nvme_tcp_icresp) { pr_err("queue %d: bad type returned %d\n", -- 2.53.0