[PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay (e.g. upon
hardware queue congestion, tag/resource exhaustion returning
BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
the local isolated CPU's worker pool. This causes the local per-CPU
high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
the isolated CPU. In real-time environments, an active SCHED_FIFO workload
will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
in addition to introducing unwanted latency jitter on nohz_full cores,
thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
from housekeeping_cpumask(HK_TYPE_DOMAIN) via
cpumask_any_and_distribute() when executing on an isolated CPU, and
otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
unbound fallback paths in blk_mq_hctx_next_cpu()

This ensures that:
    1.  When executing on an isolated CPU, deferred queue work is
        cleanly offloaded to an online housekeeping CPU across single-queue
        devices, hardware queues with empty cpumasks, and offline CPU
        fallback paths.

    2.  The per-CPU high-priority kworker on the isolated CPU is not
        scheduled, avoiding priority inversion and latency jitter.

    3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
        preserved, maintaining optimal local execution with zero CPU
        bouncing.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Changes since v2:

 - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
   to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
   isolated partitions

 - Query the housekeeping cpumask under guard(rcu)() to safely handle
   dynamic cpuset updates without lockdep warnings

 - Validate that cpumask_any_and_distribute() returns a valid cpu ID
   before returning, falling back cleanly to WORK_CPU_UNBOUND

 - Link to v2: https://lore.kernel.org/all/20261015020225.422107-1-atomlin@atomlin.com/

Changes since v1:

 - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
   to an online housekeeping CPU only when executing on an isolated
   core, preserving WORK_CPU_UNBOUND and local cache locality for
   housekeeping cores

 - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
   blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
   (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
   CPU fallback path.

 - Elaborate in the commit log on the triggers for asynchronous queue
   dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
   plugging) and the SCHED_FIFO worker starvation hazard on isolated
   cores

 - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/
---
 block/blk-mq.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 025a799f35b9..1a113679f15b 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;
+
+		guard(rcu)();
+
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+	return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2254,7 +2269,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 
 	/* Switch to unbound if no allowable CPUs in this hctx */
 	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 
 	if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2296,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 		 */
 		hctx->next_cpu = next_cpu;
 		hctx->next_cpu_batch = 1;
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 	}
 
 	hctx->next_cpu = next_cpu;
-- 
2.55.0

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Daniel Wagner]

On Fri, Oct 16, 2026 at 01:22:03AM -0400, Aaron Tomlin wrote:
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
> 
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().
> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay (e.g. upon
> hardware queue congestion, tag/resource exhaustion returning
> BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
> the local isolated CPU's worker pool. This causes the local per-CPU
> high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
> the isolated CPU. In real-time environments, an active SCHED_FIFO workload
> will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
> in addition to introducing unwanted latency jitter on nohz_full cores,
> thereby defeating the original objective of commit a46c27026da1.
> 
> Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
> from housekeeping_cpumask(HK_TYPE_DOMAIN) via
> cpumask_any_and_distribute() when executing on an isolated CPU, and
> otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
> unbound fallback paths in blk_mq_hctx_next_cpu()
> 
> This ensures that:
>     1.  When executing on an isolated CPU, deferred queue work is
>         cleanly offloaded to an online housekeeping CPU across single-queue
>         devices, hardware queues with empty cpumasks, and offline CPU
>         fallback paths.
> 
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled, avoiding priority inversion and latency jitter.
> 
>     3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
>         preserved, maintaining optimal local execution with zero CPU
>         bouncing.
> 
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Cc: Ming Lei <ming.lei@redhat.com>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: stable@vger.kernel.org
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
> Changes since v2:
> 
>  - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
>    to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
>    isolated partitions
> 
>  - Query the housekeeping cpumask under guard(rcu)() to safely handle
>    dynamic cpuset updates without lockdep warnings
> 
>  - Validate that cpumask_any_and_distribute() returns a valid cpu ID
>    before returning, falling back cleanly to WORK_CPU_UNBOUND
> 
>  - Link to v2: https://lore.kernel.org/all/20261015020225.422107-1-atomlin@atomlin.com/
> 
> Changes since v1:
> 
>  - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
>    to an online housekeeping CPU only when executing on an isolated
>    core, preserving WORK_CPU_UNBOUND and local cache locality for
>    housekeeping cores
> 
>  - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
>    blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
>    (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
>    CPU fallback path.
> 
>  - Elaborate in the commit log on the triggers for asynchronous queue
>    dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
>    plugging) and the SCHED_FIFO worker starvation hazard on isolated
>    cores
> 
>  - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/

Looks reasonable and related to my patch as well

  https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-4-355156be27b2@kernel.org

and I was able to create a blktests for it:

  https://lore.kernel.org/all/20261014-nvme-isolcpu-v1-0-2fd175a8ef00@kernel.org

Would it possible to extend those tests or add a new one so we get this
also tested? And these two patches then should/could merge.

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Jens Axboe]

On 10/15/26 11:22 PM, Aaron Tomlin wrote:
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 025a799f35b9..1a113679f15b 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
>          return hctx->next_cpu >= nr_cpu_ids;
>  }
>  
> +static inline int blk_mq_hctx_unbound_cpu(void)
> +{
> +	if (cpu_is_isolated(raw_smp_processor_id())) {
> +		int cpu;
> +
> +		guard(rcu)();
> +
> +		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
> +						 cpu_online_mask);
> +		if (cpu < nr_cpu_ids)
> +			return cpu;
> +	}
> +	return WORK_CPU_UNBOUND;
> +}

Not opposed to this, but let's please think about making the
common case the fast path. CPU isolation is not the common
path. That might mean annotating the hctx itself.

-- 
Jens Axboe

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Tue, Oct 20, 2026 at 05:09:25PM +0200, Daniel Wagner wrote:
> On Fri, Oct 16, 2026 at 01:22:03AM -0400, Aaron Tomlin wrote:
> > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > threads from executing block work on isolated CPUs. When all CPUs mapped
> > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > the intention of offloading execution.
> > 
> > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > workqueue core resolves the target CPU via raw_smp_processor_id().
> > Consequently, if a task executing on an isolated CPU submits I/O and
> > triggers an asynchronous queue dispatch with zero delay (e.g. upon
> > hardware queue congestion, tag/resource exhaustion returning
> > BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
> > the local isolated CPU's worker pool. This causes the local per-CPU
> > high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
> > the isolated CPU. In real-time environments, an active SCHED_FIFO workload
> > will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
> > in addition to introducing unwanted latency jitter on nohz_full cores,
> > thereby defeating the original objective of commit a46c27026da1.
> > 
> > Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
> > from housekeeping_cpumask(HK_TYPE_DOMAIN) via
> > cpumask_any_and_distribute() when executing on an isolated CPU, and
> > otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
> > unbound fallback paths in blk_mq_hctx_next_cpu()
> > 
> > This ensures that:
> >     1.  When executing on an isolated CPU, deferred queue work is
> >         cleanly offloaded to an online housekeeping CPU across single-queue
> >         devices, hardware queues with empty cpumasks, and offline CPU
> >         fallback paths.
> > 
> >     2.  The per-CPU high-priority kworker on the isolated CPU is not
> >         scheduled, avoiding priority inversion and latency jitter.
> > 
> >     3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
> >         preserved, maintaining optimal local execution with zero CPU
> >         bouncing.
> > 
> > Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> > Cc: Ming Lei <ming.lei@redhat.com>
> > Cc: Jens Axboe <axboe@kernel.dk>
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> > ---
> > Changes since v2:
> > 
> >  - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
> >    to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
> >    isolated partitions
> > 
> >  - Query the housekeeping cpumask under guard(rcu)() to safely handle
> >    dynamic cpuset updates without lockdep warnings
> > 
> >  - Validate that cpumask_any_and_distribute() returns a valid cpu ID
> >    before returning, falling back cleanly to WORK_CPU_UNBOUND
> > 
> >  - Link to v2: https://lore.kernel.org/all/20261015020225.422107-1-atomlin@atomlin.com/
> > 
> > Changes since v1:
> > 
> >  - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
> >    to an online housekeeping CPU only when executing on an isolated
> >    core, preserving WORK_CPU_UNBOUND and local cache locality for
> >    housekeeping cores
> > 
> >  - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
> >    blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
> >    (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
> >    CPU fallback path.
> > 
> >  - Elaborate in the commit log on the triggers for asynchronous queue
> >    dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
> >    plugging) and the SCHED_FIFO worker starvation hazard on isolated
> >    cores
> > 
> >  - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/
> 
> Looks reasonable and related to my patch as well
> 
>   https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-4-355156be27b2@kernel.org
> 
> and I was able to create a blktests for it:
> 
>   https://lore.kernel.org/all/20261014-nvme-isolcpu-v1-0-2fd175a8ef00@kernel.org
> 
> Would it possible to extend those tests or add a new one so we get this
> also tested? And these two patches then should/could merge.

Hi Daniel,

Thank you for the feedback.

Having test coverage in blktests makes complete sense.
Between extending your NVMe test and adding a new one, I think creating a
new, generic test under tests/block/ is the cleaner path.

While your nvme/070 test targets NVMe fabrics connect and passthrough
request allocation, this fix is in core blk-mq (blk_mq_hctx_next_cpu) and
affects deferred workqueue execution across all block devices. Using a new
test based on null_blk would allow us to:

    1.  Target infrastructure for generic blk-mq workqueue placement testing

    2.  Easily configure a shallow queue depth or inject tag
        exhaustion/busy conditions to reliably trigger asynchronous queue
        dispatch (&hctx->run_work)

    3.  Trace workqueue:workqueue_execute_start to assert that no kblockd
        worker executes on the isolated core

I'll put together a test case for tests/block/.


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Tue, Oct 20, 2026 at 09:23:10AM -0600, Jens Axboe wrote:
> On 10/15/26 11:22 PM, Aaron Tomlin wrote:
> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index 025a799f35b9..1a113679f15b 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
> >          return hctx->next_cpu >= nr_cpu_ids;
> >  }
> >  
> > +static inline int blk_mq_hctx_unbound_cpu(void)
> > +{
> > +	if (cpu_is_isolated(raw_smp_processor_id())) {
> > +		int cpu;
> > +
> > +		guard(rcu)();
> > +
> > +		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
> > +						 cpu_online_mask);
> > +		if (cpu < nr_cpu_ids)
> > +			return cpu;
> > +	}
> > +	return WORK_CPU_UNBOUND;
> > +}
> 
> Not opposed to this, but let's please think about making the
> common case the fast path. CPU isolation is not the common
> path. That might mean annotating the hctx itself.
> 

Hi Jens,

Agreed on keeping the fast path penalty-free.

For multi-queue devices, blk_mq_hctx_empty_cpumask(hctx) already serves as
that annotation (via hctx->next_cpu >= nr_cpu_ids) since isolated CPUs are
stripped during mapping, so normal multi-queue devices never hit that path.

The main concern is single-queue devices (nr_hw_queues == 1), where the
hctx is shared between housekeeping and isolated cores. To avoid calling
cpu_is_isolated() on every deferred dispatch in the common case, we can
split the check and gate it behind a static key:
static_branch_unlikely(&housekeeping_overridden)

I will prepare a v4.

Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Daniel Wagner]

> While your nvme/070 test targets NVMe fabrics connect and passthrough
> request allocation, this fix is in core blk-mq (blk_mq_hctx_next_cpu) and
> affects deferred workqueue execution across all block devices. Using a new
> test based on null_blk would allow us to:
> 
>     1.  Target infrastructure for generic blk-mq workqueue placement testing
> 
>     2.  Easily configure a shallow queue depth or inject tag
>         exhaustion/busy conditions to reliably trigger asynchronous queue
>         dispatch (&hctx->run_work)
> 
>     3.  Trace workqueue:workqueue_execute_start to assert that no kblockd
>         worker executes on the isolated core
> 
> I'll put together a test case for tests/block/.

Sounds good to me! Thanks.

Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Fri, Oct 16, 2026 at 01:22:03AM -0400, Aaron Tomlin wrote:
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
> 
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().
> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay (e.g. upon
> hardware queue congestion, tag/resource exhaustion returning
> BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
> the local isolated CPU's worker pool. This causes the local per-CPU
> high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
> the isolated CPU. In real-time environments, an active SCHED_FIFO workload
> will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
> in addition to introducing unwanted latency jitter on nohz_full cores,
> thereby defeating the original objective of commit a46c27026da1.
> 
> Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
> from housekeeping_cpumask(HK_TYPE_DOMAIN) via
> cpumask_any_and_distribute() when executing on an isolated CPU, and
> otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
> unbound fallback paths in blk_mq_hctx_next_cpu()
> 
> This ensures that:
>     1.  When executing on an isolated CPU, deferred queue work is
>         cleanly offloaded to an online housekeeping CPU across single-queue
>         devices, hardware queues with empty cpumasks, and offline CPU
>         fallback paths.
> 
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled, avoiding priority inversion and latency jitter.
> 
>     3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
>         preserved, maintaining optimal local execution with zero CPU
>         bouncing.
> 
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")

The current code follows behavior of schedule_work() and other related
wq APIs. If you think it is wrong, care to fix schedule_work() too,
otherwise it confuses people, IMHO.


Thanks,
Ming