[PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[Daniel Wagner]

Connecting a fabrics controller fails when all CPUs of a hardware
context are isolated with isolcpus=domain or an isolated cpuset
partition. Since commit a46c27026da1 ("blk-mq: don't schedule block
kworker on isolated CPUs") the CPU mask of such a hardware context is
empty and blk_mq_alloc_request_hctx() fails the Connect command with
-EXDEV. Patch 2 lets passthrough requests use the software context of a
housekeeping CPU instead.

Patches 3-5 fix the fabrics queue mapping:

  - Stale io_queues[] entries after a reconnect with fewer queues crash
    blk_mq_map_swqueue().
  - A queue type can get more queues than online CPUs, so connecting
    fails with -EXDEV.
  - Without queues left for polling, the queue claims poll support and
    polled I/O ends up on a non-poll queue.

Patch 1 fixes a regression in nvme-7.4 found while testing: since commit
61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
nvme_tcp_init_request takes a mutex before it is initialized, which
disables lock debugging on the first connect.

I found the fabrics queue setup problems while working on the isolcpus
series for the block layer, which Aaron Tomlin has since taken over and
which has not been merged yet. Maurizio Lombardi reported customer bugs
with isolcpus and nvme fabrics. This series doesn't depend on the block
layer work.

The bugs fixed by patches 3-5 predate commit a249d3066de6
("nvme-fabrics: add queue setup helpers"), which moved the tcp and rdma
code into fabrics.c, so kernels before v6.5 need manual backports.

Testing, QEMU with 8 CPUs, nvme-tcp and nvme-loop against nvmet:

  - isolcpus=managed_irq,domain,2-3: without the series, connecting
    with nr_io_queues=8, with and without nr_write_queues=8
    nr_poll_queues=8, fails on I/O queue 2, whose hardware context only
    has CPU 3. With the series connect and I/O work.
  - New blktests, posted separately:
      nvme/070  connect with an isolated CPU             -> patch 2
      nvme/071  more write and poll queues than CPUs     -> patch 4
      nvme/072  no queues left for polling               -> patch 5
      nvme/073  reconnect with fewer queues              -> patches 3, 5
    Without the series all fail, nvme/073 crashes the kernel. With the
    series all pass.
  - blktests block and nvme (loop, tcp): no regressions.

Reconnecting after all CPUs of a hardware context went offline still
fails, with and without this series: blk-mq marks the hardware context
inactive, but tcp, rdma and loop connect the I/O queues before updating
the CPU mapping. I'll fix this separately, together with moving the
fabrics reconnect sequence into common code.

Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
Daniel Wagner (5):
      nvme-tcp: initialize pf_cache_lock when allocating the controller
      blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs
      nvme-fabrics: reset io_queues before computing the queue mapping
      nvme-fabrics: limit each queue type to the number of online CPUs
      nvme-fabrics: clear the poll map if no poll queues are mapped

 block/blk-mq.c              | 37 +++++++++++++++++++++++++++++++++++--
 drivers/nvme/host/fabrics.c | 30 ++++++++++++++++++++++++------
 drivers/nvme/host/tcp.c     | 13 +++++++++----
 3 files changed, 68 insertions(+), 12 deletions(-)
---
base-commit: 31b8a6af19b73e6167c4b655cd5e86681bac1c41
change-id: 20261013-nvme-fabrics-isolcpu-5b4948362f88

Best regards,
--  
Daniel Wagner <wagi@kernel.org>

[PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
and startup") the I/O tag set is allocated before the I/O queues.
blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
__mutex_lock and lock debugging is disabled.

Initialize pf_cache_lock for all queues when allocating the controller
and keep it for the lifetime of the controller.

Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/tcp.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ec678ed07af8..f876375dffd2 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1531,7 +1531,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
 	mutex_destroy(&queue->queue_lock);
-	mutex_destroy(&queue->pf_cache_lock);
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 	lockdep_unregister_key(&queue->nvme_tcp_sk_key);
@@ -1861,7 +1860,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	INIT_LIST_HEAD(&queue->send_list);
 	mutex_init(&queue->send_mutex);
 	INIT_WORK(&queue->io_work, nvme_tcp_io_work);
-	mutex_init(&queue->pf_cache_lock);
 
 	if (qid > 0)
 		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -2002,7 +2000,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
 	mutex_destroy(&queue->queue_lock);
-	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
 
@@ -3028,7 +3025,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
 	struct nvme_tcp_ctrl *ctrl;
-	int ret;
+	int ret, i;
 
 	ctrl = kzalloc_obj(*ctrl);
 	if (!ctrl)
@@ -3098,6 +3095,14 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		goto out_free_ctrl;
 	}
 
+	/*
+	 * The I/O tag set is allocated before the I/O queues, and initializing
+	 * its requests already allocates the PDUs from the page frag caches of
+	 * the queues, so initialize their locks here.
+	 */
+	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+		mutex_init(&ctrl->queues[i].pf_cache_lock);
+
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
 	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops, 0);

-- 
2.56.0

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Sebastian Andrzej Siewior]

On 2026-10-14 09:46:39 [+0200], Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
> and startup") the I/O tag set is allocated before the I/O queues.
> blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
> which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
> initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
> connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
> __mutex_lock and lock debugging is disabled.
> 
> Initialize pf_cache_lock for all queues when allocating the controller
> and keep it for the lifetime of the controller.
> 
> Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")

I only see 2a8513091d2f0 ("nvme-rdma: parallelize I/O queue allocation
and startup"). That tcp commit is not in next-20261013.

Other than that, it looks reasonable ;)
> Signed-off-by: Daniel Wagner <wagi@kernel.org>

Sebastian

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Daniel Wagner]

On Wed, Oct 14, 2026 at 01:08:04PM +0200, Sebastian Andrzej Siewior wrote:
> > Initialize pf_cache_lock for all queues when allocating the controller
> > and keep it for the lifetime of the controller.
> > 
> > Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
> 
> I only see 2a8513091d2f0 ("nvme-rdma: parallelize I/O queue allocation
> and startup"). That tcp commit is not in next-20261013.

The commit is in

https://git.infradead.org/?p=nvme.git;a=shortlog;h=refs/heads/nvme-7.4

but I missed the nvme-rdma version of it. It's about time to merge this
into common code.

Thanks,
Daniel

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Saravanan D]

On Wed, Oct 14, 2026 at 09:46:39AM +0200, Daniel Wagner wrote:
> Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
> and startup") the I/O tag set is allocated before the I/O queues.
> blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
> which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
> initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
> connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
> __mutex_lock and lock debugging is disabled.

I hit the same warning on nvme-7.4, and also a second path to it, a
reconnect that grows the queue count. Every teardown frees the I/O
queues, and nvme_tcp_free_queue() destroys their pf_cache_lock. When
a reconnect raises the queue count, blk_mq_update_nr_hw_queues()
initializes the requests of the added hardware contexts, and with them
takes pf_cache_lock of their queues, before nvme_tcp_setup_io_queues()
allocates those queues.

  shrink, 4 -> 3 queues
    nvme_tcp_free_queue(4)        mutex_destroy(q4 pf_cache_lock)
  grow, 3 -> 4 queues
    blk_mq_update_nr_hw_queues()
      nvme_tcp_init_request()     mutex_lock(q4 pf_cache_lock)  <- WARN
    nvme_tcp_setup_io_queues()
      nvme_tcp_alloc_queue(4)     mutex_init(q4 pf_cache_lock)

  WARNING: kernel/locking/mutex.c:625 at __mutex_lock+0x434/0x117c
  Call trace:
   __mutex_lock+0x434/0x117c (P)
   mutex_lock_nested+0x24/0x30
   nvme_tcp_init_request+0xb0/0x260
   blk_mq_alloc_map_and_rqs+0x388/0x670
   blk_mq_update_nr_hw_queues+0xb30/0x1058
   nvme_tcp_setup_ctrl+0xa9c/0xda4
   nvme_tcp_reconnect_ctrl_work+0x70/0x2d8
   process_one_work+0x7a0/0x1b5c

Dropping the mutex_destroy() from nvme_tcp_free_queue(), as this patch
does, fixes that path too. It may be worth mentioning this fact in the
changelog.

Tested against nvmet-tcp over loopback with lockdep, KASAN and
DEBUG_KOBJECT_RELEASE, covering connect, reset loops under I/O,
shrinking and growing the queue count through attr_qid_max, and
reconnects while the port is removed.

Tested-by: Saravanan D <saravanand@crusoe.ai>

Thanks,
Saravanan D.

Re: [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller

[Hannes Reinecke]

On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> Since commit 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation
> and startup") the I/O tag set is allocated before the I/O queues.
> blk_mq_alloc_tag_set calls nvme_tcp_init_request for every request,
> which takes pf_cache_lock of the request's queue, but nvme_tcp_alloc_queue
> initializes that mutex only later.  With CONFIG_DEBUG_MUTEXES the first
> connect triggers DEBUG_LOCKS_WARN_ON(lock->magic != lock) in
> __mutex_lock and lock debugging is disabled.
> 
> Initialize pf_cache_lock for all queues when allocating the controller
> and keep it for the lifetime of the controller.
> 
> Fixes: 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
>   drivers/nvme/host/tcp.c | 13 +++++++++----
>   1 file changed, 9 insertions(+), 4 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 2/5] blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

Since commit a46c27026da1 ("blk-mq: don't schedule block kworker on
isolated CPUs") isolated CPUs are removed from hctx->cpumask.  That
commit assumed the mask is only used for kworker placement and CPU
offline handling, but blk_mq_alloc_request_hctx uses it to pick the
software context of the request.  If only isolated CPUs are mapped to a
hardware context, the mask is empty and blk_mq_alloc_request_hctx fails
with -EXDEV, although the isolated CPUs still submit I/O to it.

The nvme fabrics transports allocate the Connect command for every I/O
queue this way, so connecting a controller fails with isolcpus=domain
or an isolated cpuset partition as soon as a hardware context only has
isolated CPUs.

Passthrough requests are inserted directly into hctx->dispatch, so
their software context only selects the CPU the completion runs on and
the hint for freeing the tag.  Use the software context of an online CPU
which is neither isolated from the scheduler domains nor from managed
interrupts for them.  Other requests and inactive hardware contexts
still fail.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 block/blk-mq.c | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a26a11c73ee3..79fb08400c48 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -697,6 +697,25 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
 }
 EXPORT_SYMBOL(blk_mq_alloc_request);
 
+/*
+ * Return an online CPU which is neither isolated from the scheduler domains
+ * nor from managed interrupts, or nr_cpu_ids if there is none.
+ */
+static unsigned int blk_mq_housekeeping_cpu(void)
+{
+	unsigned int cpu;
+
+	rcu_read_lock();
+	for_each_online_cpu(cpu) {
+		if (!cpu_is_isolated(cpu) &&
+		    housekeeping_test_cpu(cpu, HK_TYPE_MANAGED_IRQ))
+			break;
+	}
+	rcu_read_unlock();
+
+	return cpu;
+}
+
 struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	blk_opf_t opf, blk_mq_req_flags_t flags, unsigned int hctx_idx)
 {
@@ -747,8 +766,22 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	if (!blk_mq_hw_queue_mapped(data.hctx))
 		goto out_queue_exit;
 	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
-	if (cpu >= nr_cpu_ids)
-		goto out_queue_exit;
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * Isolated CPUs are not part of hctx->cpumask, so it can be
+		 * empty although online CPUs are mapped to this hardware
+		 * context.  Passthrough requests are inserted directly into
+		 * hctx->dispatch, so use the software context of a
+		 * housekeeping CPU for them.  Inactive hardware contexts have
+		 * no online CPU left.
+		 */
+		if (!blk_op_is_passthrough(opf) ||
+		    test_bit(BLK_MQ_S_INACTIVE, &data.hctx->state))
+			goto out_queue_exit;
+		cpu = blk_mq_housekeeping_cpu();
+		if (cpu >= nr_cpu_ids)
+			goto out_queue_exit;
+	}
 	data.ctx = __blk_mq_get_ctx(q, cpu);
 
 	if (q->elevator)

-- 
2.56.0

Re: [PATCH 2/5] blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs

[Hannes Reinecke]

On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> Since commit a46c27026da1 ("blk-mq: don't schedule block kworker on
> isolated CPUs") isolated CPUs are removed from hctx->cpumask.  That
> commit assumed the mask is only used for kworker placement and CPU
> offline handling, but blk_mq_alloc_request_hctx uses it to pick the
> software context of the request.  If only isolated CPUs are mapped to a
> hardware context, the mask is empty and blk_mq_alloc_request_hctx fails
> with -EXDEV, although the isolated CPUs still submit I/O to it.
> 
> The nvme fabrics transports allocate the Connect command for every I/O
> queue this way, so connecting a controller fails with isolcpus=domain
> or an isolated cpuset partition as soon as a hardware context only has
> isolated CPUs.
> 
> Passthrough requests are inserted directly into hctx->dispatch, so
> their software context only selects the CPU the completion runs on and
> the hint for freeing the tag.  Use the software context of an online CPU
> which is neither isolated from the scheduler domains nor from managed
> interrupts for them.  Other requests and inactive hardware contexts
> still fail.
> 
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
>   block/blk-mq.c | 37 +++++++++++++++++++++++++++++++++++--
>   1 file changed, 35 insertions(+), 2 deletions(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index a26a11c73ee3..79fb08400c48 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -697,6 +697,25 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
>   }
>   EXPORT_SYMBOL(blk_mq_alloc_request);
>   
> +/*
> + * Return an online CPU which is neither isolated from the scheduler domains
> + * nor from managed interrupts, or nr_cpu_ids if there is none.
> + */
> +static unsigned int blk_mq_housekeeping_cpu(void)
> +{
> +	unsigned int cpu;
> +
> +	rcu_read_lock();
> +	for_each_online_cpu(cpu) {
> +		if (!cpu_is_isolated(cpu) &&
> +		    housekeeping_test_cpu(cpu, HK_TYPE_MANAGED_IRQ))
> +			break;
> +	}
> +	rcu_read_unlock();
> +
> +	return cpu;
> +}
> +
>   struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
>   	blk_opf_t opf, blk_mq_req_flags_t flags, unsigned int hctx_idx)
>   {
> @@ -747,8 +766,22 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
>   	if (!blk_mq_hw_queue_mapped(data.hctx))
>   		goto out_queue_exit;
>   	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> -	if (cpu >= nr_cpu_ids)
> -		goto out_queue_exit;
> +	if (cpu >= nr_cpu_ids) {
> +		/*
> +		 * Isolated CPUs are not part of hctx->cpumask, so it can be
> +		 * empty although online CPUs are mapped to this hardware
> +		 * context.  Passthrough requests are inserted directly into
> +		 * hctx->dispatch, so use the software context of a
> +		 * housekeeping CPU for them.  Inactive hardware contexts have
> +		 * no online CPU left.
> +		 */
> +		if (!blk_op_is_passthrough(opf) ||
> +		    test_bit(BLK_MQ_S_INACTIVE, &data.hctx->state))
> +			goto out_queue_exit;
> +		cpu = blk_mq_housekeeping_cpu();
> +		if (cpu >= nr_cpu_ids)
> +			goto out_queue_exit;
> +	}
>   	data.ctx = __blk_mq_get_ctx(q, cpu);
>   
>   	if (q->elevator)
> 
Hmm. I wonder.
While the patch looks correct, the reasoning for this patch is slightly 
confusing: 'nvme connect' surely tries to send a 'connect' command on
every queue, and every queue maps to a cpu (-set).
But why do we execute a 'connect' sqe on an isolated CPU? I thought
we would exclude them when connecting the queues?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 3/5] nvme-fabrics: reset io_queues before computing the queue mapping

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

nvmf_set_io_queues only sets the io_queues entries of the queue types it
hands out, but the array is reused on every reconnect.  When the
controller grants fewer queues on a reconnect, stale entries survive and
nvmf_map_queues maps CPUs to hardware contexts that don't exist anymore,
which crashes blk_mq_map_swqueue with a NULL pointer dereference.  E.g.
with nr_io_queues=8, nr_write_queues=8 and nr_poll_queues=8, 24 queues
are mapped 8/8/8 (default/read/poll).  If the controller grants only 16
queues on the reconnect, the poll map still starts at hardware context
16.

Clear the array before computing the new mapping.

Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 59f823dfbbcc..571de549fbaf 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1125,6 +1125,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 			u32 io_queues[HCTX_MAX_TYPES])
 {
+	memset(io_queues, 0, sizeof(*io_queues) * HCTX_MAX_TYPES);
+
 	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
 		/*
 		 * separate read/write queues

-- 
2.56.0

Re: [PATCH 3/5] nvme-fabrics: reset io_queues before computing the queue mapping

[Hannes Reinecke]

On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> nvmf_set_io_queues only sets the io_queues entries of the queue types it
> hands out, but the array is reused on every reconnect.  When the
> controller grants fewer queues on a reconnect, stale entries survive and
> nvmf_map_queues maps CPUs to hardware contexts that don't exist anymore,
> which crashes blk_mq_map_swqueue with a NULL pointer dereference.  E.g.
> with nr_io_queues=8, nr_write_queues=8 and nr_poll_queues=8, 24 queues
> are mapped 8/8/8 (default/read/poll).  If the controller grants only 16
> queues on the reconnect, the poll map still starts at hardware context
> 16.
> 
> Clear the array before computing the new mapping.
> 
> Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
> Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
>   drivers/nvme/host/fabrics.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index 59f823dfbbcc..571de549fbaf 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -1125,6 +1125,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
>   			u32 io_queues[HCTX_MAX_TYPES])
>   {
> +	memset(io_queues, 0, sizeof(*io_queues) * HCTX_MAX_TYPES);
> +
>   	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
>   		/*
>   		 * separate read/write queues
> 

Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 4/5] nvme-fabrics: limit each queue type to the number of online CPUs

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

nvmf_nr_io_queues limits each queue type to the number of online CPUs
when asking for I/O queues, but nvmf_set_io_queues hands out the
granted queues without these limits, and only nr_io_queues is clamped
when parsing the options.

E.g. with 2 CPUs, nr_io_queues=2, nr_write_queues=10 and
nr_poll_queues=10, 2 + 2 + 2 = 6 queues are requested, but handed out
as 4 default, 2 read and 0 poll queues.  blk_mq_map_queues can't create
more CPU masks than there are CPUs, so default queues 2 and 3 take over
the CPUs of default queues 0 and 1, and connecting the I/O queue of
default queue 1 fails with -EXDEV.

Apply the same limits in nvmf_set_io_queues.  A queue type with more
queues than online CPUs always has a hardware context without an online
CPU, so no working setup changes.  The options are not modified, so the
limits follow CPU hotplug on reconnect.

Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
Fixes: a249d3066de6 ("nvme-fabrics: add queue setup helpers")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 571de549fbaf..27a8ce51d2b3 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1125,18 +1125,27 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 			u32 io_queues[HCTX_MAX_TYPES])
 {
+	/*
+	 * Each queue type is mapped to the CPUs separately, so a type with
+	 * more queues than online CPUs has hardware contexts without an online
+	 * CPU.  Use the same limits as nvmf_nr_io_queues.
+	 */
+	u32 nr_io = min(opts->nr_io_queues, num_online_cpus());
+	u32 nr_write = min(opts->nr_write_queues, num_online_cpus());
+	u32 nr_poll = min(opts->nr_poll_queues, num_online_cpus());
+
 	memset(io_queues, 0, sizeof(*io_queues) * HCTX_MAX_TYPES);
 
-	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+	if (nr_write && nr_io < nr_io_queues) {
 		/*
 		 * separate read/write queues
 		 * hand out dedicated default queues only after we have
 		 * sufficient read queues.
 		 */
-		io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		io_queues[HCTX_TYPE_READ] = nr_io;
 		nr_io_queues -= io_queues[HCTX_TYPE_READ];
 		io_queues[HCTX_TYPE_DEFAULT] =
-			min(opts->nr_write_queues, nr_io_queues);
+			min(nr_write, nr_io_queues);
 		nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT];
 	} else {
 		/*
@@ -1145,14 +1154,14 @@ void nvmf_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues,
 		 * sufficient queue count to have dedicated default queues.
 		 */
 		io_queues[HCTX_TYPE_DEFAULT] =
-			min(opts->nr_io_queues, nr_io_queues);
+			min(nr_io, nr_io_queues);
 		nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT];
 	}
 
-	if (opts->nr_poll_queues && nr_io_queues) {
+	if (nr_poll && nr_io_queues) {
 		/* map dedicated poll queues only if we have queues left */
 		io_queues[HCTX_TYPE_POLL] =
-			min(opts->nr_poll_queues, nr_io_queues);
+			min(nr_poll, nr_io_queues);
 	}
 }
 EXPORT_SYMBOL_GPL(nvmf_set_io_queues);

-- 
2.56.0

Re: [PATCH 4/5] nvme-fabrics: limit each queue type to the number of online CPUs

[Hannes Reinecke]

On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> nvmf_nr_io_queues limits each queue type to the number of online CPUs
> when asking for I/O queues, but nvmf_set_io_queues hands out the
> granted queues without these limits, and only nr_io_queues is clamped
> when parsing the options.
> 
> E.g. with 2 CPUs, nr_io_queues=2, nr_write_queues=10 and
> nr_poll_queues=10, 2 + 2 + 2 = 6 queues are requested, but handed out
> as 4 default, 2 read and 0 poll queues.  blk_mq_map_queues can't create
> more CPU masks than there are CPUs, so default queues 2 and 3 take over
> the CPUs of default queues 0 and 1, and connecting the I/O queue of
> default queue 1 fails with -EXDEV.
> 
> Apply the same limits in nvmf_set_io_queues.  A queue type with more
> queues than online CPUs always has a hardware context without an online
> CPU, so no working setup changes.  The options are not modified, so the
> limits follow CPU hotplug on reconnect.
> 
> Fixes: 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited")
> Fixes: a249d3066de6 ("nvme-fabrics: add queue setup helpers")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
>   drivers/nvme/host/fabrics.c | 21 +++++++++++++++------
>   1 file changed, 15 insertions(+), 6 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 5/5] nvme-fabrics: clear the poll map if no poll queues are mapped

[Daniel Wagner]

From: Daniel Wagner <wagi@kernel.org>

blk_mq_alloc_tag_set sets the nr_queues of every map to nr_hw_queues.
When nr_poll_queues is set but no queues are left for polling,
nvmf_map_queues skips the poll map and leaves it this way, with all CPUs
mapped to hctx 0.  The queue then claims poll support, polled I/O ends
up on hctx 0, which isn't a poll queue, and hctx 0 is marked as a poll
hctx, so even the Connect command of the first I/O queue is marked as
polled.

Clear the poll map when no poll queues are mapped, so polled I/O is
rejected with -EOPNOTSUPP.

Fixes: 1a9460cef571 ("nvme-tcp: support simple polling")
Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
 drivers/nvme/host/fabrics.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 27a8ce51d2b3..3d7ad86c1955 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -1199,6 +1199,13 @@ void nvmf_map_queues(struct blk_mq_tag_set *set, struct nvme_ctrl *ctrl,
 			io_queues[HCTX_TYPE_DEFAULT] +
 			io_queues[HCTX_TYPE_READ];
 		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	} else {
+		/*
+		 * blk_mq_alloc_tag_set sets nr_queues of all maps to
+		 * nr_hw_queues.  Clear the poll map, otherwise the queue claims
+		 * poll support and polled I/O ends up on hctx 0.
+		 */
+		set->map[HCTX_TYPE_POLL].nr_queues = 0;
 	}
 
 	dev_info(ctrl->device,

-- 
2.56.0

Re: [PATCH 5/5] nvme-fabrics: clear the poll map if no poll queues are mapped

[Hannes Reinecke]

On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
> 
> blk_mq_alloc_tag_set sets the nr_queues of every map to nr_hw_queues.
> When nr_poll_queues is set but no queues are left for polling,
> nvmf_map_queues skips the poll map and leaves it this way, with all CPUs
> mapped to hctx 0.  The queue then claims poll support, polled I/O ends
> up on hctx 0, which isn't a poll queue, and hctx 0 is marked as a poll
> hctx, so even the Connect command of the first I/O queue is marked as
> polled.
> 
> Clear the poll map when no poll queues are mapped, so polled I/O is
> rejected with -EOPNOTSUPP.
> 
> Fixes: 1a9460cef571 ("nvme-tcp: support simple polling")
> Fixes: 5651cd3c4336 ("nvme-rdma: fix queue mapping when queue count is limited")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
>   drivers/nvme/host/fabrics.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index 27a8ce51d2b3..3d7ad86c1955 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -1199,6 +1199,13 @@ void nvmf_map_queues(struct blk_mq_tag_set *set, struct nvme_ctrl *ctrl,
>   			io_queues[HCTX_TYPE_DEFAULT] +
>   			io_queues[HCTX_TYPE_READ];
>   		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
> +	} else {
> +		/*
> +		 * blk_mq_alloc_tag_set sets nr_queues of all maps to
> +		 * nr_hw_queues.  Clear the poll map, otherwise the queue claims
> +		 * poll support and polled I/O ends up on hctx 0.
> +		 */
> +		set->map[HCTX_TYPE_POLL].nr_queues = 0;
>   	}
>   
>   	dev_info(ctrl->device,
> 
Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[Maurizio Lombardi]

On Wed Oct 14, 2026 at 9:46 AM CEST, Daniel Wagner wrote:
> Connecting a fabrics controller fails when all CPUs of a hardware
> context are isolated with isolcpus=domain or an isolated cpuset
> partition. Since commit a46c27026da1 ("blk-mq: don't schedule block
> kworker on isolated CPUs") the CPU mask of such a hardware context is
> empty and blk_mq_alloc_request_hctx() fails the Connect command with
> -EXDEV. Patch 2 lets passthrough requests use the software context of a
> housekeeping CPU instead.
>
> Patches 3-5 fix the fabrics queue mapping:
>
>   - Stale io_queues[] entries after a reconnect with fewer queues crash
>     blk_mq_map_swqueue().
>   - A queue type can get more queues than online CPUs, so connecting
>     fails with -EXDEV.
>   - Without queues left for polling, the queue claims poll support and
>     polled I/O ends up on a non-poll queue.
>
> Patch 1 fixes a regression in nvme-7.4 found while testing: since commit
> 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
> nvme_tcp_init_request takes a mutex before it is initialized, which
> disables lock debugging on the first connect.
>
> I found the fabrics queue setup problems while working on the isolcpus
> series for the block layer, which Aaron Tomlin has since taken over and
> which has not been merged yet. Maurizio Lombardi reported customer bugs
> with isolcpus and nvme fabrics. This series doesn't depend on the block
> layer work.
>
> The bugs fixed by patches 3-5 predate commit a249d3066de6
> ("nvme-fabrics: add queue setup helpers"), which moved the tcp and rdma
> code into fabrics.c, so kernels before v6.5 need manual backports.
>
> Testing, QEMU with 8 CPUs, nvme-tcp and nvme-loop against nvmet:
>
>   - isolcpus=managed_irq,domain,2-3: without the series, connecting
>     with nr_io_queues=8, with and without nr_write_queues=8
>     nr_poll_queues=8, fails on I/O queue 2, whose hardware context only
>     has CPU 3. With the series connect and I/O work.
>   - New blktests, posted separately:
>       nvme/070  connect with an isolated CPU             -> patch 2
>       nvme/071  more write and poll queues than CPUs     -> patch 4
>       nvme/072  no queues left for polling               -> patch 5
>       nvme/073  reconnect with fewer queues              -> patches 3, 5
>     Without the series all fail, nvme/073 crashes the kernel. With the
>     series all pass.
>   - blktests block and nvme (loop, tcp): no regressions.
>
> Reconnecting after all CPUs of a hardware context went offline still
> fails, with and without this series: blk-mq marks the hardware context
> inactive, but tcp, rdma and loop connect the I/O queues before updating
> the CPU mapping. I'll fix this separately, together with moving the
> fabrics reconnect sequence into common code.
>
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
> Daniel Wagner (5):
>       nvme-tcp: initialize pf_cache_lock when allocating the controller
>       blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs
>       nvme-fabrics: reset io_queues before computing the queue mapping
>       nvme-fabrics: limit each queue type to the number of online CPUs
>       nvme-fabrics: clear the poll map if no poll queues are mapped
>
>  block/blk-mq.c              | 37 +++++++++++++++++++++++++++++++++++--
>  drivers/nvme/host/fabrics.c | 30 ++++++++++++++++++++++++------
>  drivers/nvme/host/tcp.c     | 13 +++++++++----
>  3 files changed, 68 insertions(+), 12 deletions(-)
> ---
> base-commit: 31b8a6af19b73e6167c4b655cd5e86681bac1c41
> change-id: 20261013-nvme-fabrics-isolcpu-5b4948362f88
>
> Best regards,
> --  
> Daniel Wagner <wagi@kernel.org>

Thank you very much for this

I tested nvme/tcp with and without isolcpus and it works for me

Tested-by: Maurizio Lombardi <mlombard@redhat.com>

Re: [PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs

[Keith Busch]

On Wed, Oct 14, 2026 at 09:46:38AM +0200, Daniel Wagner wrote:
> Connecting a fabrics controller fails when all CPUs of a hardware
> context are isolated with isolcpus=domain or an isolated cpuset
> partition. Since commit a46c27026da1 ("blk-mq: don't schedule block
> kworker on isolated CPUs") the CPU mask of such a hardware context is
> empty and blk_mq_alloc_request_hctx() fails the Connect command with
> -EXDEV. Patch 2 lets passthrough requests use the software context of a
> housekeeping CPU instead.
> 
> Patches 3-5 fix the fabrics queue mapping:
> 
>   - Stale io_queues[] entries after a reconnect with fewer queues crash
>     blk_mq_map_swqueue().
>   - A queue type can get more queues than online CPUs, so connecting
>     fails with -EXDEV.
>   - Without queues left for polling, the queue claims poll support and
>     polled I/O ends up on a non-poll queue.
> 
> Patch 1 fixes a regression in nvme-7.4 found while testing: since commit
> 61b0ed3be81b ("nvme-tcp: parallelize I/O queue allocation and startup")
> nvme_tcp_init_request takes a mutex before it is initialized, which
> disables lock debugging on the first connect.

It seems the nvme patches (1, 3-5) can be applied independently of the
blk-mq patch, so I staged those in nvme-7.4.