[PATCH] nvmet-tcp: fix use-after-free when TLS handshake completes before timeout is armed

[PATCH] nvmet-tcp: fix use-after-free when TLS handshake completes before timeout is armed

[Duc Anh Luu]

nvmet_tcp_tls_handshake() takes a queue reference for the handshake
completion callback, submits the handshake with tls_server_hello_psk(),
and only afterwards arms tls_handshake_tmo_work.

Once tls_server_hello_psk() has queued the request and notified the
handshake agent, nvmet_tcp_tls_handshake_done() can run before the
submitter reaches queue_delayed_work(). On failure the callback calls
nvmet_tcp_schedule_release_queue() and then drops its own reference,
which drops the last reference and queues release_work. nvmet_wq is
not ordered, so nvmet_tcp_release_queue_work() can run concurrently:
its cancel_delayed_work_sync() finds no pending timeout work, and it
frees the queue. The submitter then arms the delayed work inside the
freed queue, and the timer fires on freed memory tls_handshake_timeout
seconds later.

Reproduced on mainline during the v7.4 merge window on an NVMe/TCP
port with addr_tsas=tls1.3, with clients whose handshakes tlshd
rejects. It reproduces both with stock tlshd and with a minimal test
agent that fails every request; a non-debug kernel panics in
__run_timers. With KASAN, the unmodified kernel hit:

BUG: KASAN: slab-use-after-free in __run_timers+0x72a/0x8e0
Write of size 8 at addr ffff88800317a1d0 by task swapper/3/0
__run_timers
run_timer_softirq
Allocated by task 12: nvmet_tcp_accept_work
Freed by task 132: kfree <- process_one_work (nvmet_tcp_release_queue_work)
Last potentially related work creation:
queue_work_on <- handshake_complete <- handshake_nl_done_doit

The queue was freed 4ms after allocation and the timer fired 10s later.
With an msleep() inserted between tls_server_hello_psk() and
queue_delayed_work() the report is deterministic and points at the
write in queue_delayed_work_on() from nvmet_tcp_accept_work().

Hold an extra queue reference across the submission and the arming of
the timeout work. If the completion callback has already released the
queue, the final put queues release_work, which cancels the timeout
work that was just armed.

Fixes: 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
Cc: stable@vger.kernel.org
XXX-ADD-YOUR-OWN-Signed-off-by-LINE-HERE-AND-DELETE-THIS-LINE-XXX
---
drivers/nvme/target/tcp.c | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index e59810175262..e55a3b86f3f6 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1910,6 +1910,14 @@ static int nvmet_tcp_tls_handshake(struct 
nvmet_tcp_queue *queue)
return -EINVAL;
}
+ /* Reference owned by the handshake completion callback */
+ kref_get(&queue->kref);
+ /*
+ * The completion callback may run, and drop its reference and the
+ * queue reference, before tls_server_hello_psk() returns. Hold an
+ * extra reference until the timeout work is armed so the queue
+ * cannot be released underneath us.
+ */
kref_get(&queue->kref);
pr_debug("queue %d: TLS ServerHello\n", queue->idx);
memset(&args, 0, sizeof(args));
@@ -1927,6 +1935,7 @@ static int nvmet_tcp_tls_handshake(struct 
nvmet_tcp_queue *queue)
queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work,
tls_handshake_timeout * HZ);
}
+ kref_put(&queue->kref, nvmet_tcp_release_queue);
return ret;
}
#else

Re: [PATCH] nvmet-tcp: fix use-after-free when TLS handshake completes before timeout is armed

[Keith Busch]

On Tue, Nov 03, 2026 at 11:05:18PM +0700, Duc Anh Luu wrote:
> nvmet_tcp_tls_handshake() takes a queue reference for the handshake
> completion callback, submits the handshake with tls_server_hello_psk(),
> and only afterwards arms tls_handshake_tmo_work.
> 
> Once tls_server_hello_psk() has queued the request and notified the
> handshake agent, nvmet_tcp_tls_handshake_done() can run before the
> submitter reaches queue_delayed_work(). On failure the callback calls
> nvmet_tcp_schedule_release_queue() and then drops its own reference,
> which drops the last reference and queues release_work. nvmet_wq is
> not ordered, so nvmet_tcp_release_queue_work() can run concurrently:
> its cancel_delayed_work_sync() finds no pending timeout work, and it
> frees the queue. The submitter then arms the delayed work inside the
> freed queue, and the timer fires on freed memory tls_handshake_timeout
> seconds later.
> 
> Reproduced on mainline during the v7.4 merge window on an NVMe/TCP
> port with addr_tsas=tls1.3, with clients whose handshakes tlshd
> rejects. It reproduces both with stock tlshd and with a minimal test
> agent that fails every request; a non-debug kernel panics in
> __run_timers. With KASAN, the unmodified kernel hit:
> 
> BUG: KASAN: slab-use-after-free in __run_timers+0x72a/0x8e0
> Write of size 8 at addr ffff88800317a1d0 by task swapper/3/0
> __run_timers
> run_timer_softirq
> Allocated by task 12: nvmet_tcp_accept_work
> Freed by task 132: kfree <- process_one_work (nvmet_tcp_release_queue_work)
> Last potentially related work creation:
> queue_work_on <- handshake_complete <- handshake_nl_done_doit
> 
> The queue was freed 4ms after allocation and the timer fired 10s later.
> With an msleep() inserted between tls_server_hello_psk() and
> queue_delayed_work() the report is deterministic and points at the
> write in queue_delayed_work_on() from nvmet_tcp_accept_work().
> 
> Hold an extra queue reference across the submission and the arming of
> the timeout work. If the completion callback has already released the
> queue, the final put queues release_work, which cancels the timeout
> work that was just armed.

This is a rather long commit message for a simple problem. Just the
first two paragraphs and the KASAN splat are sufficient for this
changelog.
 
> Fixes: 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
> Cc: stable@vger.kernel.org
> XXX-ADD-YOUR-OWN-Signed-off-by-LINE-HERE-AND-DELETE-THIS-LINE-XXX

Umm...

> @@ -1910,6 +1910,14 @@ static int nvmet_tcp_tls_handshake(struct
> nvmet_tcp_queue *queue)
> return -EINVAL;
> }
> + /* Reference owned by the handshake completion callback */
> + kref_get(&queue->kref);
> + /*
> + * The completion callback may run, and drop its reference and the
> + * queue reference, before tls_server_hello_psk() returns. Hold an
> + * extra reference until the timeout work is armed so the queue
> + * cannot be released underneath us.
> + */
> kref_get(&queue->kref);
> pr_debug("queue %d: TLS ServerHello\n", queue->idx);
> memset(&args, 0, sizeof(args));
> @@ -1927,6 +1935,7 @@ static int nvmet_tcp_tls_handshake(struct
> nvmet_tcp_queue *queue)
> queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work,
> tls_handshake_timeout * HZ);
> }
> + kref_put(&queue->kref, nvmet_tcp_release_queue);
> return ret;
> }
> #else

Your patch is white space damaged.