Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup

[Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>]

On Oct 20, 2026 / 13:48, Surabhi Gogte (she/her) wrote:
> On Sun, Oct 18, 2026 at 5:26 AM Shin'ichiro Kawasaki
> <shinichiro.kawasaki@wdc.com> wrote:
> >
> > On Sep 22, 2026 / 14:07, Surabhi Gogte wrote:
> >
> > Recently, blktests-CI runs for the kernel at block/for-next branch started
> > reporting failures of the test case nvme/004 (with tcp transport) and md/001.
> > The kernel reported a WARN related to lock->magic, and it caused the failures.
> > FYI, here I share the failure run log at md/001 [1] and dmesg at the failure.
> >
> >   The md/001 confirms a bug fix in md driver. The bug was triggered when a
> >   nvme device with tcp transport is used as its backend. Then the test case
> >   exercises the nvme-tcp driver, which this patch touches.
> >
> > I bisected and found that this commit is the trigger of the failures. When I
> > revert this commit from the block/for-next branch tip with git hash
> > 34fbfb9d8d1f, the failure goes away.
> >
> > Surabhi, could you take a look in the failure?
> >
> 
> Yes, there is a bug in commit 61b0ed3be81b ("nvme-tcp: parallelize I/O
> queue allocation and startup").  To know the queue range before the
> queues are set up, it moved the I/O tag set allocation ahead of I/O
> queue allocation.  blk_mq_alloc_tag_set() calls
> nvme_tcp_init_request() for every request, which takes pf_cache_lock
> of the request's queue, but nvme_tcp_alloc_queue() only initializes
> that mutex later:
>   nvme_tcp_configure_io_queues()
>     nvme_alloc_io_tag_set()
>       blk_mq_alloc_tag_set()
>         blk_mq_alloc_rqs()
>           nvme_tcp_init_request()  mutex_lock(pf_cache_lock) <- WARN
>     nvme_tcp_setup_io_queues()
>       nvme_tcp_alloc_queue()   mutex_init(pf_cache_lock)
> 
> So with CONFIG_DEBUG_MUTEXES the first connect trips
> DEBUG_LOCKS_WARN_ON(lock->magic != lock) in __mutex_lock().
> That matches the call trace in your dmesg.
> 
> Daniel Wagner posted a fix for this as patch 1 of [PATCH 0/5]
> nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs:
> lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-0-355156be27b2@kernel.org
> 
> It initializes pf_cache_lock for all queues in nvme_tcp_alloc_ctrl(),
> right after the queues array is allocated, and drops the
> mutex_destroy() calls so the lock stays valid for the lifetime of the
> controller.
> 
> I ran blktests with his fix on top of block/for-next (34fbfb9d8d1f, the
> same tip you bisected on) and md/001 passes with no WARN related to
> lock->magic.  Without the fix, the same tree and config fails md/001
> with the WARN at kernel/locking/mutex.c:625.
> 
> With the fix : nvme-tcp: initialize pf_cache_lock when allocating the controller
> lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-1-355156be27b2@kernel.org/
> 
> ~/blktests$ sudo ./check md/001
> md/001 (Raid with bitmap on tcp nvmet with opt-io-size over bitmap
> size) [passed]
>     runtime  1.779s  ...  1.814s

Surabhi, thanks for the confirmation. It's great taht The md/001 failure is
gone. Said that, now I face failrues of other test cases: nvme/062 and 063.

While I was running blktests on v7.4-rc1 and v7.4-rc2 kernel, I observed the two
test cases nvme/062 and 063 fail for tcp transport [*]. I bisected and found
that this patch as the commit 61b0ed3be81b is the trigger. I observed the
failure with nvme-cli versions 3.0, 3.1 and 3.2. ktls-utils version is 1.4.0.

  When I reported blktests failures for v7.4-rc1 kernel [#], the failure was
  not observed by some reason. But now the failure is observed in stable manner.

  [#] https://lore.kernel.org/linux-block/avBz7uOGYo2y426_@shinmob/

It looks like nvme connect can fail for tls connection with multiple I/O
queues. I created a blktests patch to limit number of I/O quees to 1 [**],
and observed the failure goes away. But I'm not sure if this is just hiding
the failure.

Are the failures observed in your environment?


[*] nvme/062 and nvme/063 failure

nvme/062 (tr=tcp) (Create TLS-encrypted connections)         [failed]
    runtime  4.678s  ...  13.619s
    --- tests/nvme/062.out	2026-11-09 20:12:01.056000000 +0900
    +++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad	2026-11-15 20:21:09.213000000 +0900
    @@ -1,6 +1,10 @@
     Running nvme/062
     Test unencrypted connection w/ tls not required
     Test encrypted connection w/ tls not required
    +FAIL: nvme connect return error code
    +WARNING: connection is not encrypted
     Test unencrypted connection w/ tls required (should fail)
     Test encrypted connection w/ tls required
    ...
    (Run 'diff -u tests/nvme/062.out /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad' to see the entire diff)
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation)
    runtime  3.123s  ...
WARNING: Test did not clean up port: 0
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation) [failed]
    runtime  3.123s  ...  7.313sost: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
    --- tests/nvme/063.out	2026-11-09 20:12:01.060000000 +0900
    +++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/063.out.bad	2026-11-15 20:21:30.207000000 +0900
    @@ -1,5 +1,4 @@
     Running nvme/063
     Test secure concatenation with SHA256
    -Reset controller
    -Test secure concatenation with SHA384
    -Test complete
    +FAIL: nvme connect return error code
    +WARNING: connection failed


[**] trial blktests patch

diff --git a/tests/nvme/062 b/tests/nvme/062
index eabe2d9..69a05b6 100755
--- a/tests/nvme/062
+++ b/tests/nvme/062
@@ -55,7 +55,7 @@ test() {
 
 	# Test encrypted connection
 	echo "Test encrypted connection w/ tls not required"
-	_nvme_connect_subsys --tls
+	_nvme_connect_subsys --tls --nr-io-queues 1
 
 	ctrl=$(_find_nvme_dev "${def_subsysnqn}")
 	if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then
@@ -77,7 +77,7 @@ test() {
 
 	# Test encrypted connection
 	echo "Test encrypted connection w/ tls required"
-	_nvme_connect_subsys --tls
+	_nvme_connect_subsys --tls --nr-io-queues 1
 
 	ctrl=$(_find_nvme_dev "${def_subsysnqn}")
 	if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null; then
diff --git a/tests/nvme/063 b/tests/nvme/063
index feb07bf..900bb9f 100755
--- a/tests/nvme/063
+++ b/tests/nvme/063
@@ -46,7 +46,7 @@ test() {
 	_set_nvmet_dhgroup "${def_hostnqn}" "ffdhe2048"
 
 	echo "Test secure concatenation with SHA256"
-	_nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
+	_nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
 
 	ctrl=$(_find_nvme_dev "${def_subsysnqn}")
 	if [[ -z "$ctrl" ]]; then
@@ -88,7 +88,7 @@ test() {
 	_set_nvmet_dhgroup "${def_hostnqn}" "ffdhe3072"
 
 	echo "Test secure concatenation with SHA384"
-	_nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
+	_nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
 
 	ctrl=$(_find_nvme_dev "${def_subsysnqn}")
 	if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then