[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
is queued directly onto the local isolated CPU's worker pool. This
causes the local per-CPU high-priority kworker
(e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
introducing unwanted latency jitter and execution delays, thereby
defeating the original objective of commit a46c27026da1.

Rectify this by selecting an online housekeeping CPU via
housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
is true. This ensures that:
    1.  The delayed work is cleanly offloaded to a housekeeping CPU.

    2.  The per-CPU high-priority kworker on the isolated CPU is not
        scheduled.

    3.  Hardware queues with valid mapped CPUs preserve their per-CPU
        locality and batch round-robin behaviour.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 block/blk-mq.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 025a799f35b9..9480c3aac898 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 	bool tried = false;
 	int next_cpu = hctx->next_cpu;
 
-	/* Switch to unbound if no allowable CPUs in this hctx */
-	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
+	/* Switch to housekeeping CPU if no allowable CPUs in this hctx */
+	if (blk_mq_hctx_empty_cpumask(hctx))
+		return housekeeping_any_cpu(HK_TYPE_TIMER);
+
+	if (hctx->queue->nr_hw_queues == 1)
 		return WORK_CPU_UNBOUND;
 
 	if (--hctx->next_cpu_batch <= 0) {
-- 
2.55.0

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
>
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().

Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
if you think it is one problem.

> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> is queued directly onto the local isolated CPU's worker pool. This

If your theory is correct, why do botter to change to
housekeeping_any_cpu(HK_TYPE_TIMER)
if the user is submitting IO from an isolated CPU?

> causes the local per-CPU high-priority kworker
> (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
> introducing unwanted latency jitter and execution delays, thereby
> defeating the original objective of commit a46c27026da1.
>
> Rectify this by selecting an online housekeeping CPU via
> housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
> is true. This ensures that:
>     1.  The delayed work is cleanly offloaded to a housekeeping CPU.
>
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled.
>
>     3.  Hardware queues with valid mapped CPUs preserve their per-CPU
>         locality and batch round-robin behaviour.
>
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Cc: Ming Lei <ming.lei@redhat.com>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: stable@vger.kernel.org
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
>  block/blk-mq.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 025a799f35b9..9480c3aac898 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
>         bool tried = false;
>         int next_cpu = hctx->next_cpu;
>
> -       /* Switch to unbound if no allowable CPUs in this hctx */
> -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> +       if (blk_mq_hctx_empty_cpumask(hctx))
> +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> +
> +       if (hctx->queue->nr_hw_queues == 1)
>                 return WORK_CPU_UNBOUND;

Why is SQ so special here to keep WORK_CPU_UNBOUND?


Thanks,
Ming Lei

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> >
> > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > threads from executing block work on isolated CPUs. When all CPUs mapped
> > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > the intention of offloading execution.
> >
> > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > workqueue core resolves the target CPU via raw_smp_processor_id().
> 
> Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> if you think it is one problem.

Hi Ming,

Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
to provide a non-isolated CPU when dispatching &hctx->run_work, no?

> > Consequently, if a task executing on an isolated CPU submits I/O and
> > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > is queued directly onto the local isolated CPU's worker pool. This
> 
> If your theory is correct, why do botter to change to
> housekeeping_any_cpu(HK_TYPE_TIMER)
> if the user is submitting IO from an isolated CPU?

I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
architecturally necessary and fully aligned with the rest of the kernel,
no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
defers execution to a background workqueue. That work runs asynchronously
at some indeterminate time in the future, long after the thread has
returned to its time-critical user loop. Having a kworker suddenly preempt
the thread later is likely not desired.


> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index 025a799f35b9..9480c3aac898 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
> >         bool tried = false;
> >         int next_cpu = hctx->next_cpu;
> >
> > -       /* Switch to unbound if no allowable CPUs in this hctx */
> > -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> > +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> > +       if (blk_mq_hctx_empty_cpumask(hctx))
> > +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> > +
> > +       if (hctx->queue->nr_hw_queues == 1)
> >                 return WORK_CPU_UNBOUND;
> 
> Why is SQ so special here to keep WORK_CPU_UNBOUND?

Indeed, it is not special here, and neither is the offline CPU fallback
path (!cpu_online(next_cpu)). Both currently return WORK_CPU_UNBOUND and
suffer from the exact same issue if triggered from an isolated CPU.

To address this consistently without penalising non-isolated CPUs with
unnecessary cross-CPU bounces, we can define a small helper. Furthermore,
the following will address the valid issue raised by Sashiko [1]
comprehensively, using the new helper:
    1.  When executing on an isolated CPU
        cpu_is_isolated(raw_smp_processor_id()) evaluates to true.
        It redirects to an online housekeeping CPU
        (housekeeping_any_cpu(HK_TYPE_TIMER)), preventing kworkers from
        running on isolated cores across all three cases:
            - Single-queue devices (nr_hw_queues == 1)
            - Hardware queues with empty cpumasks
              (blk_mq_hctx_empty_cpumask(hctx))
            - Offline CPU fallback paths (!cpu_online(next_cpu))

    2.  When executing on a housekeeping CPU cpu_is_isolated() is false.
        It returns WORK_CPU_UNBOUND, which kblockd_workqueue executes on
        the local CPU (raw_smp_processor_id()), preserving optimal cache
        locality with zero CPU bouncing.

    3.  When CPU isolation is not configured cpu_is_isolated() uses a
        static branch (static_branch_unlikely(&housekeeping_overridden)),
        compiling to a NOP that returns WORK_CPU_UNBOUND with zero
        overhead.

[1]: https://sashiko.dev/#/patchset/20261012012905.267920-1-atomlin%40atomlin.com

diff --git a/block/blk-mq.c b/block/blk-mq.c
index e28c467a18bb..9fb1a93b2bb1 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2244,6 +2244,13 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }

+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+    if (cpu_is_isolated(raw_smp_processor_id()))
+        return housekeeping_any_cpu(HK_TYPE_TIMER);
+    return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2256,7 +2263,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)

     /* Switch to unbound if no allowable CPUs in this hctx */
     if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();

     if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2288,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
          */
         hctx->next_cpu = next_cpu;
         hctx->next_cpu_batch = 1;
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();
     }

         hctx->next_cpu = next_cpu;



Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 10:21:01PM -0400, Aaron Tomlin wrote:
> On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> > On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> > >
> > > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > > threads from executing block work on isolated CPUs. When all CPUs mapped
> > > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > > the intention of offloading execution.
> > >
> > > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > > workqueue core resolves the target CPU via raw_smp_processor_id().
> > 
> > Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> > if you think it is one problem.
> 
> Hi Ming,
> 
> Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> to provide a non-isolated CPU when dispatching &hctx->run_work, no?

Does every WORK_CPU_UNBOUND user have to handle it?

> 
> > > Consequently, if a task executing on an isolated CPU submits I/O and
> > > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > > is queued directly onto the local isolated CPU's worker pool. This
> > 
> > If your theory is correct, why do botter to change to
> > housekeeping_any_cpu(HK_TYPE_TIMER)
> > if the user is submitting IO from an isolated CPU?
> 
> I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> architecturally necessary and fully aligned with the rest of the kernel,
> no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> defers execution to a background workqueue. That work runs asynchronously
> at some indeterminate time in the future, long after the thread has
> returned to its time-critical user loop. Having a kworker suddenly preempt
> the thread later is likely not desired.

We are talking about `if a task executing on an isolated CPU submits I/O `.

If the task has been submitting IO on isolated CPU, why can't we keep
scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
makes any difference from user viewpoint.


Thanks,
Ming