From: Hannes Reinecke <hare@suse.de>
To: Daniel Wagner <dwagner@suse.de>, Keith Busch <kbusch@kernel.org>,
Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
Sagi Grimberg <sagi@grimberg.me>
Cc: Ming Lei <ming.lei@redhat.com>,
Surabhi Gogte <sgogte@purestorage.com>,
Hannes Reinecke <hare@kernel.org>,
Nilay Shroff <nilay@linux.ibm.com>,
Frederic Weisbecker <frederic@kernel.org>,
Aaron Tomlin <atomlin@atomlin.com>,
Maurizio Lombardi <mlombard@redhat.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
linux-block@vger.kernel.org, linux-nvme@lists.infradead.org,
linux-kernel@vger.kernel.org, Daniel Wagner <wagi@kernel.org>
Subject: Re: [PATCH 2/5] blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs
Date: Wed, 21 Oct 2026 13:10:40 +0200 [thread overview]
Message-ID: <62c3831f-f4e5-484e-b32a-f83ed1316d68@suse.de> (raw)
In-Reply-To: <20261014-nvme-fabrics-isolcpu-v1-2-355156be27b2@kernel.org>
On 10/14/26 9:46 AM, Daniel Wagner wrote:
> From: Daniel Wagner <wagi@kernel.org>
>
> Since commit a46c27026da1 ("blk-mq: don't schedule block kworker on
> isolated CPUs") isolated CPUs are removed from hctx->cpumask. That
> commit assumed the mask is only used for kworker placement and CPU
> offline handling, but blk_mq_alloc_request_hctx uses it to pick the
> software context of the request. If only isolated CPUs are mapped to a
> hardware context, the mask is empty and blk_mq_alloc_request_hctx fails
> with -EXDEV, although the isolated CPUs still submit I/O to it.
>
> The nvme fabrics transports allocate the Connect command for every I/O
> queue this way, so connecting a controller fails with isolcpus=domain
> or an isolated cpuset partition as soon as a hardware context only has
> isolated CPUs.
>
> Passthrough requests are inserted directly into hctx->dispatch, so
> their software context only selects the CPU the completion runs on and
> the hint for freeing the tag. Use the software context of an online CPU
> which is neither isolated from the scheduler domains nor from managed
> interrupts for them. Other requests and inactive hardware contexts
> still fail.
>
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Signed-off-by: Daniel Wagner <wagi@kernel.org>
> ---
> block/blk-mq.c | 37 +++++++++++++++++++++++++++++++++++--
> 1 file changed, 35 insertions(+), 2 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index a26a11c73ee3..79fb08400c48 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -697,6 +697,25 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
> }
> EXPORT_SYMBOL(blk_mq_alloc_request);
>
> +/*
> + * Return an online CPU which is neither isolated from the scheduler domains
> + * nor from managed interrupts, or nr_cpu_ids if there is none.
> + */
> +static unsigned int blk_mq_housekeeping_cpu(void)
> +{
> + unsigned int cpu;
> +
> + rcu_read_lock();
> + for_each_online_cpu(cpu) {
> + if (!cpu_is_isolated(cpu) &&
> + housekeeping_test_cpu(cpu, HK_TYPE_MANAGED_IRQ))
> + break;
> + }
> + rcu_read_unlock();
> +
> + return cpu;
> +}
> +
> struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
> blk_opf_t opf, blk_mq_req_flags_t flags, unsigned int hctx_idx)
> {
> @@ -747,8 +766,22 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
> if (!blk_mq_hw_queue_mapped(data.hctx))
> goto out_queue_exit;
> cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> - if (cpu >= nr_cpu_ids)
> - goto out_queue_exit;
> + if (cpu >= nr_cpu_ids) {
> + /*
> + * Isolated CPUs are not part of hctx->cpumask, so it can be
> + * empty although online CPUs are mapped to this hardware
> + * context. Passthrough requests are inserted directly into
> + * hctx->dispatch, so use the software context of a
> + * housekeeping CPU for them. Inactive hardware contexts have
> + * no online CPU left.
> + */
> + if (!blk_op_is_passthrough(opf) ||
> + test_bit(BLK_MQ_S_INACTIVE, &data.hctx->state))
> + goto out_queue_exit;
> + cpu = blk_mq_housekeeping_cpu();
> + if (cpu >= nr_cpu_ids)
> + goto out_queue_exit;
> + }
> data.ctx = __blk_mq_get_ctx(q, cpu);
>
> if (q->elevator)
>
Hmm. I wonder.
While the patch looks correct, the reasoning for this patch is slightly
confusing: 'nvme connect' surely tries to send a 'connect' command on
every queue, and every queue maps to a cpu (-set).
But why do we execute a 'connect' sqe on an isolated CPU? I thought
we would exclude them when connecting the queues?
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
next prev parent reply other threads:[~2026-10-21 11:10 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-14 7:46 [PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs Daniel Wagner
2026-10-14 7:46 ` [PATCH 1/5] nvme-tcp: initialize pf_cache_lock when allocating the controller Daniel Wagner
2026-10-14 11:08 ` Sebastian Andrzej Siewior
2026-10-14 12:46 ` Daniel Wagner
2026-10-17 21:37 ` Saravanan D
2026-10-21 11:05 ` Hannes Reinecke
2026-10-14 7:46 ` [PATCH 2/5] blk-mq: use a housekeeping CPU for passthrough requests on isolated hctxs Daniel Wagner
2026-10-21 11:10 ` Hannes Reinecke [this message]
2026-10-14 7:46 ` [PATCH 3/5] nvme-fabrics: reset io_queues before computing the queue mapping Daniel Wagner
2026-10-21 11:11 ` Hannes Reinecke
2026-10-14 7:46 ` [PATCH 4/5] nvme-fabrics: limit each queue type to the number of online CPUs Daniel Wagner
2026-10-21 11:12 ` Hannes Reinecke
2026-10-14 7:46 ` [PATCH 5/5] nvme-fabrics: clear the poll map if no poll queues are mapped Daniel Wagner
2026-10-21 11:13 ` Hannes Reinecke
2026-10-20 14:06 ` [PATCH 0/5] nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs Maurizio Lombardi
2026-10-21 21:01 ` Keith Busch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=62c3831f-f4e5-484e-b32a-f83ed1316d68@suse.de \
--to=hare@suse.de \
--cc=atomlin@atomlin.com \
--cc=axboe@kernel.dk \
--cc=bigeasy@linutronix.de \
--cc=dwagner@suse.de \
--cc=frederic@kernel.org \
--cc=hare@kernel.org \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=ming.lei@redhat.com \
--cc=mlombard@redhat.com \
--cc=nilay@linux.ibm.com \
--cc=sagi@grimberg.me \
--cc=sgogte@purestorage.com \
--cc=wagi@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®