[PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[Daniel Wagner]

Connecting a fabrics controller fails when all CPUs of a hardware
context are isolated with isolcpus=domain or an isolated cpuset
partition. Since commit a46c27026da1 ("blk-mq: don't schedule block
kworker on isolated CPUs") the CPU mask of such a hardware context is
empty and blk_mq_alloc_request_hctx() fails the Connect command with
-EXDEV. Patch 2 lets passthrough requests use the software context of a
housekeeping CPU instead.

Patches 3-5 fix the fabrics queue mapping:

  - Stale io_queues[] entries after a reconnect with fewer queues crash
    blk_mq_map_swqueue().
  - A queue type can get more queues than online CPUs, so connecting
    fails with -EXDEV.
  - Without queues left for polling, the queue claims poll support and
    polled I/O ends up on a non-poll queue.

Patch 1 fixes a regression in nvme-7.4 found while testing: since commit
61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
nvme_tcp_init_request takes a mutex before it is initialized, which
disables lock debugging on the first connect.

I found the fabrics queue setup problems while working on the isolcpus
series for the block layer, which Aaron Tomlin has since taken over and
which has not been merged yet. Maurizio Lombardi reported customer bugs
with isolcpus and nvme fabrics. This series doesn't depend on the block
layer work.

The bugs fixed by patches 3-5 predate commit a249d3066de6
("nvme-fabrics: add queue setup helpers"), which moved the tcp and rdma
code into fabrics.c, so kernels before v6.5 need manual backports.

Testing, QEMU with 8 CPUs, nvme-tcp and nvme-loop against nvmet:

  - isolcpus=managed_irq,domain,2-3: without the series, connecting
    with nr_io_queues=8, with and without nr_write_queues=8
    nr_poll_queues=8, fails on I/O queue 2, whose hardware context only
    has CPU 3. With the series connect and I/O work.
  - New blktests, posted separately:
      nvme/070  connect with an isolated CPU             -> patch 2
      nvme/071  more write and poll queues than CPUs     -> patch 4
      nvme/072  no queues left for polling               -> patch 5
      nvme/073  reconnect with fewer queues              -> patches 3, 5
    Without the series all fail, nvme/073 crashes the kernel. With the
    series all pass.
  - blktests block and nvme (loop, tcp): no regressions.

Reconnecting after all CPUs of a hardware context went offline still
fails, with and without this series: blk-mq marks the hardware context
inactive, but tcp, rdma and loop connect the I/O queues before updating
the CPU mapping. I'll fix this separately, together with moving the
fabrics reconnect sequence into common code.

Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
Daniel Wagner (5):
      nvme-tcp: initialize pf_cache_lock when allocating the controller
      blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs
      nvme-fabrics: reset io_queues before computing the queue mapping
      nvme-fabrics: limit each queue type to the number of online CPUs
      nvme-fabrics: clear the poll map if no poll queues are mapped

 block/blk-mq.c              | 37 +++++++++++++++++++++++++++++++++++--
 drivers/nvme/host/fabrics.c | 30 ++++++++++++++++++++++++------
 drivers/nvme/host/tcp.c     | 13 +++++++++----
 3 files changed, 68 insertions(+), 12 deletions(-)
---
base-commit: 31b8a6af19b73e6167c4b655cd5e86681bac1c41
change-id: 20261013-nvme-fabrics-isolcpu-5b4948362f88

Best regards,
--  
Daniel Wagner <wagi@kernel.org>

[PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
and startup") the I/O tag set is allocated before the I/O queues.
blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
__mutex_lock and lock debugging is disabled.

Initialize pf_cache_lock for all queues when allocating the controller
and keep it for the lifetime of the controller.

Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/tcp.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ec678ed07af8..f876375dffd2 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1531,7 +1531,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
 	mutex_destroy(&queue->queue_lock);
-	mutex_destroy(&queue->pf_cache_lock);
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 	lockdep_unregister_key(&queue->nvme_tcp_sk_key);
@@ -1861,7 +1860,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	INIT_LIST_HEAD(&queue->send_list);
 	mutex_init(&queue->send_mutex);
 	INIT_WORK(&queue->io_work, nvme_tcp_io_work);
-	mutex_init(&queue->pf_cache_lock);
 
 	if (qid > 0)
 		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -2002,7 +2000,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
 	mutex_destroy(&queue->queue_lock);
-	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
 
@@ -3028,7 +3025,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
 	struct nvme_tcp_ctrl *ctrl;
-	int ret;
+	int ret, i;
 
 	ctrl = kzalloc_obj(*ctrl);
 	if (!ctrl)
@@ -3098,6 +3095,14 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		goto out_free_ctrl;
 	}
 
+	/*
+	 * The I/O tag set is allocated before the I/O queues, and initializing
+	 * its requests already allocates the PDUs from the page frag caches of
+	 * the queues, so initialize their locks here.
+	 */
+	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+		mutex_init(&ctrl->queues[i].pf_cache_lock);
+
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
 	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops, 0);

-- 
2.56.0

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Sebastian Andrzej Siewior]

On 2026-10-14 09:46:39 [+0200], Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
> and startup") the I/O tag set is allocated before the I/O queues.
> blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
> which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
> initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
> connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
> __mutex_lock and lock debugging is disabled.
> 
> Initialize pf_cache_lock for all queues when allocating the controller
> and keep it for the lifetime of the controller.
> 
> Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")

I only see 2a8513091d2f0 ("nvme-rdma: parallelize I/O queue allocation
and startup"). That tcp commit is not in next-20261013.

Other than that, it looks reasonable ;)
> Signed-off-by: Daniel Wagner <wagi@kernel.org>

Sebastian

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Daniel Wagner]

On Wed, Oct 14, 2026 at 01:08:04PM +0200, Sebastian Andrzej Siewior wrote:
> > Initialize pf_cache_lock for all queues when allocating the controller
> > and keep it for the lifetime of the controller.
> > 
> > Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
> 
> I only see 2a8513091d2f0 ("nvme-rdma: parallelize I/O queue allocation
> and startup"). That tcp commit is not in next-20261013.

The commit is in

https://git.infradead.org/?p=nvme.git;a=shortlog;h=refs/heads/nvme-7.4

but I missed the nvme-rdma version of it. It's about time to merge this
into common code.

Thanks,
Daniel

[PATCH 2/5] blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

Since commit a46c27026da1 ("blk-mq: don't schedule block kworker on
isolated CPUs") isolated CPUs are removed from hctx->cpumask.  That
commit assumed the mask is only used for kworker placement and CPU
offline handling, but blk_mq_alloc_request_hctx uses it to pick the
software context of the request.  If only isolated CPUs are mapped to a
hardware context, the mask is empty and blk_mq_alloc_request_hctx fails
with -EXDEV, although the isolated CPUs still submit I/O to it.

The nvme fabrics transports allocate the Connect command for every I/O
queue this way, so connecting a controller fails with isolcpus=domain
or an isolated cpuset partition as soon as a hardware context only has
isolated CPUs.

Passthrough requests are inserted directly into hctx->dispatch, so
their software context only selects the CPU the completion runs on and
the hint for freeing the tag.  Use the software context of an online CPU
which is neither isolated from the scheduler domains nor from managed
interrupts for them.  Other requests and inactive hardware contexts
still fail.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 block/blk-mq.c | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a26a11c73ee3..79fb08400c48 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -697,6 +697,25 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
 }
 EXPORT_SYMBOL(blk_mq_alloc_request);
 
+/*
+ * Return an online CPU which is neither isolated from the scheduler domains
+ * nor from managed interrupts, or nr_cpu_ids if there is none.
+ */
+static unsigned int blk_mq_housekeeping_cpu(void)
+{
+	unsigned int cpu;
+
+	rcu_read_lock();
+	for_each_online_cpu(cpu) {
+		if (!cpu_is_isolated(cpu) &&
+		    housekeeping_test_cpu(cpu, HK_TYPE_MANAGED_IRQ))
+			break;
+	}
+	rcu_read_unlock();
+
+	return cpu;
+}
+
 struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	blk_opf_t opf, blk_mq_req_flags_t flags, unsigned int hctx_idx)
 {
@@ -747,8 +766,22 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	if (!blk_mq_hw_queue_mapped(data.hctx))
 		goto out_queue_exit;
 	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
-	if (cpu >= nr_cpu_ids)
-		goto out_queue_exit;
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * Isolated CPUs are not part of hctx->cpumask, so it can be
+		 * empty although online CPUs are mapped to this hardware
+		 * context.  Passthrough requests are inserted directly into
+		 * hctx->dispatch, so use the software context of a
+		 * housekeeping CPU for them.  Inactive hardware contexts have
+		 * no online CPU left.
+		 */
+		if (!blk_op_is_passthrough(opf) ||
+		    test_bit(BLK_MQ_S_INACTIVE, &data.hctx->state))
+			goto out_queue_exit;
+		cpu = blk_mq_housekeeping_cpu();
+		if (cpu >= nr_cpu_ids)
+			goto out_queue_exit;
+	}
 	data.ctx = __blk_mq_get_ctx(q, cpu);
 
 	if (q->elevator)

-- 
2.56.0

[PATCH 3/5] nvme-fabrics: reset io_queues before computing the queue mapping

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

nvmf_set_io_queues only sets the io_queues entries of the queue types it
hands out, but the array is reused on every reconnect.  When the
controller grants fewer queues on a reconnect, stale entries survive and
nvmf_map_queues maps CPUs to hardware contexts that don't exist anymore,
which crashes blk_mq_map_swqueue with a NULL pointer dereference.  E.g.
with nr_io_queues=8, nr_write_queues=8 and nr_poll_queues=8, 24 queues
are mapped 8/8/8 (default/read/poll).  If the controller grants only 16
queues on the reconnect, the poll map still starts at hardware context
16.

Clear the array before computing the new mapping.

Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 59f823dfbbcc..571de549fbaf 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1125,6 +1125,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 			u32 io_queues[HCTX_MAX_TYPES])
 {
+	memset(io_queues, 0, sizeof(*io_queues) * HCTX_MAX_TYPES);
+
 	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
 		/*
 		 * separate read/write queues

-- 
2.56.0

[PATCH 4/5] nvme-fabrics: limit each queue type to the number of online CPUs

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

nvmf_nr_io_queues limits each queue type to the number of online CPUs
when asking for I/O queues, but nvmf_set_io_queues hands out the
granted queues without these limits, and only nr_io_queues is clamped
when parsing the options.

E.g. with 2 CPUs, nr_io_queues=2, nr_write_queues=10 and
nr_poll_queues=10, 2 + 2 + 2 = 6 queues are requested, but handed out
as 4 default, 2 read and 0 poll queues.  blk_mq_map_queues can't create
more CPU masks than there are CPUs, so default queues 2 and 3 take over
the CPUs of default queues 0 and 1, and connecting the I/O queue of
default queue 1 fails with -EXDEV.

Apply the same limits in nvmf_set_io_queues.  A queue type with more
queues than online CPUs always has a hardware context without an online
CPU, so no working setup changes.  The options are not modified, so the
limits follow CPU hotplug on reconnect.

Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
Fixes: a249d3066de6 ("nvme-fabrics: add queue setup helpers")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 571de549fbaf..27a8ce51d2b3 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1125,18 +1125,27 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 			u32 io_queues[HCTX_MAX_TYPES])
 {
+	/*
+	 * Each queue type is mapped to the CPUs separately, so a type with
+	 * more queues than online CPUs has hardware contexts without an online
+	 * CPU.  Use the same limits as nvmf_nr_io_queues.
+	 */
+	u32 nr_io = min(opts->nr_io_queues, num_online_cpus());
+	u32 nr_write = min(opts->nr_write_queues, num_online_cpus());
+	u32 nr_poll = min(opts->nr_poll_queues, num_online_cpus());
+
 	memset(io_queues, 0, sizeof(*io_queues) * HCTX_MAX_TYPES);
 
-	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+	if (nr_write && nr_io < nr_io_queues) {
 		/*
 		 * separate read/write queues
 		 * hand out dedicated default queues only after we have
 		 * sufficient read queues.
 		 */
-		io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		io_queues[HCTX_TYPE_READ] = nr_io;
 		nr_io_queues -= io_queues[HCTX_TYPE_READ];
 		io_queues[HCTX_TYPE_DEFAULT] =
-			min(opts->nr_write_queues, nr_io_queues);
+			min(nr_write, nr_io_queues);
 		nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT];
 	} else {
 		/*
@@ -1145,14 +1154,14 @@ void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 		 * sufficient queue count to have dedicated default queues.
 		 */
 		io_queues[HCTX_TYPE_DEFAULT] =
-			min(opts->nr_io_queues, nr_io_queues);
+			min(nr_io, nr_io_queues);
 		nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT];
 	}
 
-	if (opts->nr_poll_queues && nr_io_queues) {
+	if (nr_poll && nr_io_queues) {
 		/* map dedicated poll queues only if we have queues left */
 		io_queues[HCTX_TYPE_POLL] =
-			min(opts->nr_poll_queues, nr_io_queues);
+			min(nr_poll, nr_io_queues);
 	}
 }
 EXPORT_SYMBOL_GPL(nvmf_set_io_queues);

-- 
2.56.0

[PATCH 5/5] nvme-fabrics: clear the poll map if no poll queues are mapped

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

blk_mq_alloc_tag_set sets the nr_queues of every map to nr_hw_queues.
When nr_poll_queues is set but no queues are left for polling,
nvmf_map_queues skips the poll map and leaves it this way, with all CPUs
mapped to hctx 0.  The queue then claims poll support, polled I/O ends
up on hctx 0, which isn't a poll queue, and hctx 0 is marked as a poll
hctx, so even the Connect command of the first I/O queue is marked as
polled.

Clear the poll map when no poll queues are mapped, so polled I/O is
rejected with -EOPNOTSUPP.

Fixes: 1a9460cef571 ("nvme-tcp: support simple polling")
Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 27a8ce51d2b3..3d7ad86c1955 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1199,6 +1199,13 @@ void nvmf_map_queues(struct blk_mq_tag_set *set, struct nvme_ctrl *ctrl,
 			io_queues[HCTX_TYPE_DEFAULT] +
 			io_queues[HCTX_TYPE_READ];
 		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	} else {
+		/*
+		 * blk_mq_alloc_tag_set sets nr_queues of all maps to
+		 * nr_hw_queues.  Clear the poll map, otherwise the queue claims
+		 * poll support and polled I/O ends up on hctx 0.
+		 */
+		set->map[HCTX_TYPE_POLL].nr_queues = 0;
 	}
 
 	dev_info(ctrl->device,

-- 
2.56.0