[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
is queued directly onto the local isolated CPU's worker pool. This
causes the local per-CPU high-priority kworker
(e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
introducing unwanted latency jitter and execution delays, thereby
defeating the original objective of commit a46c27026da1.

Rectify this by selecting an online housekeeping CPU via
housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
is true. This ensures that:
    1.  The delayed work is cleanly offloaded to a housekeeping CPU.

    2.  The per-CPU high-priority kworker on the isolated CPU is not
        scheduled.

    3.  Hardware queues with valid mapped CPUs preserve their per-CPU
        locality and batch round-robin behaviour.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 block/blk-mq.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 025a799f35b9..9480c3aac898 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 	bool tried = false;
 	int next_cpu = hctx->next_cpu;
 
-	/* Switch to unbound if no allowable CPUs in this hctx */
-	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
+	/* Switch to housekeeping CPU if no allowable CPUs in this hctx */
+	if (blk_mq_hctx_empty_cpumask(hctx))
+		return housekeeping_any_cpu(HK_TYPE_TIMER);
+
+	if (hctx->queue->nr_hw_queues == 1)
 		return WORK_CPU_UNBOUND;
 
 	if (--hctx->next_cpu_batch <= 0) {
-- 
2.55.0

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
>
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().

Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
if you think it is one problem.

> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> is queued directly onto the local isolated CPU's worker pool. This

If your theory is correct, why do botter to change to
housekeeping_any_cpu(HK_TYPE_TIMER)
if the user is submitting IO from an isolated CPU?

> causes the local per-CPU high-priority kworker
> (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
> introducing unwanted latency jitter and execution delays, thereby
> defeating the original objective of commit a46c27026da1.
>
> Rectify this by selecting an online housekeeping CPU via
> housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
> is true. This ensures that:
>     1.  The delayed work is cleanly offloaded to a housekeeping CPU.
>
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled.
>
>     3.  Hardware queues with valid mapped CPUs preserve their per-CPU
>         locality and batch round-robin behaviour.
>
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Cc: Ming Lei <ming.lei@redhat.com>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: stable@vger.kernel.org
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
>  block/blk-mq.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 025a799f35b9..9480c3aac898 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
>         bool tried = false;
>         int next_cpu = hctx->next_cpu;
>
> -       /* Switch to unbound if no allowable CPUs in this hctx */
> -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> +       if (blk_mq_hctx_empty_cpumask(hctx))
> +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> +
> +       if (hctx->queue->nr_hw_queues == 1)
>                 return WORK_CPU_UNBOUND;

Why is SQ so special here to keep WORK_CPU_UNBOUND?


Thanks,
Ming Lei

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> >
> > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > threads from executing block work on isolated CPUs. When all CPUs mapped
> > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > the intention of offloading execution.
> >
> > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > workqueue core resolves the target CPU via raw_smp_processor_id().
> 
> Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> if you think it is one problem.

Hi Ming,

Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
to provide a non-isolated CPU when dispatching &hctx->run_work, no?

> > Consequently, if a task executing on an isolated CPU submits I/O and
> > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > is queued directly onto the local isolated CPU's worker pool. This
> 
> If your theory is correct, why do botter to change to
> housekeeping_any_cpu(HK_TYPE_TIMER)
> if the user is submitting IO from an isolated CPU?

I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
architecturally necessary and fully aligned with the rest of the kernel,
no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
defers execution to a background workqueue. That work runs asynchronously
at some indeterminate time in the future, long after the thread has
returned to its time-critical user loop. Having a kworker suddenly preempt
the thread later is likely not desired.


> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index 025a799f35b9..9480c3aac898 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
> >         bool tried = false;
> >         int next_cpu = hctx->next_cpu;
> >
> > -       /* Switch to unbound if no allowable CPUs in this hctx */
> > -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> > +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> > +       if (blk_mq_hctx_empty_cpumask(hctx))
> > +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> > +
> > +       if (hctx->queue->nr_hw_queues == 1)
> >                 return WORK_CPU_UNBOUND;
> 
> Why is SQ so special here to keep WORK_CPU_UNBOUND?

Indeed, it is not special here, and neither is the offline CPU fallback
path (!cpu_online(next_cpu)). Both currently return WORK_CPU_UNBOUND and
suffer from the exact same issue if triggered from an isolated CPU.

To address this consistently without penalising non-isolated CPUs with
unnecessary cross-CPU bounces, we can define a small helper. Furthermore,
the following will address the valid issue raised by Sashiko [1]
comprehensively, using the new helper:
    1.  When executing on an isolated CPU
        cpu_is_isolated(raw_smp_processor_id()) evaluates to true.
        It redirects to an online housekeeping CPU
        (housekeeping_any_cpu(HK_TYPE_TIMER)), preventing kworkers from
        running on isolated cores across all three cases:
            - Single-queue devices (nr_hw_queues == 1)
            - Hardware queues with empty cpumasks
              (blk_mq_hctx_empty_cpumask(hctx))
            - Offline CPU fallback paths (!cpu_online(next_cpu))

    2.  When executing on a housekeeping CPU cpu_is_isolated() is false.
        It returns WORK_CPU_UNBOUND, which kblockd_workqueue executes on
        the local CPU (raw_smp_processor_id()), preserving optimal cache
        locality with zero CPU bouncing.

    3.  When CPU isolation is not configured cpu_is_isolated() uses a
        static branch (static_branch_unlikely(&housekeeping_overridden)),
        compiling to a NOP that returns WORK_CPU_UNBOUND with zero
        overhead.

[1]: https://sashiko.dev/#/patchset/20261012012905.267920-1-atomlin%40atomlin.com

diff --git a/block/blk-mq.c b/block/blk-mq.c
index e28c467a18bb..9fb1a93b2bb1 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2244,6 +2244,13 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }

+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+    if (cpu_is_isolated(raw_smp_processor_id()))
+        return housekeeping_any_cpu(HK_TYPE_TIMER);
+    return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2256,7 +2263,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)

     /* Switch to unbound if no allowable CPUs in this hctx */
     if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();

     if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2288,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
          */
         hctx->next_cpu = next_cpu;
         hctx->next_cpu_batch = 1;
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();
     }

         hctx->next_cpu = next_cpu;



Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 10:21:01PM -0400, Aaron Tomlin wrote:
> On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> > On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> > >
> > > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > > threads from executing block work on isolated CPUs. When all CPUs mapped
> > > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > > the intention of offloading execution.
> > >
> > > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > > workqueue core resolves the target CPU via raw_smp_processor_id().
> > 
> > Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> > if you think it is one problem.
> 
> Hi Ming,
> 
> Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> to provide a non-isolated CPU when dispatching &hctx->run_work, no?

Does every WORK_CPU_UNBOUND user have to handle it?

> 
> > > Consequently, if a task executing on an isolated CPU submits I/O and
> > > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > > is queued directly onto the local isolated CPU's worker pool. This
> > 
> > If your theory is correct, why do botter to change to
> > housekeeping_any_cpu(HK_TYPE_TIMER)
> > if the user is submitting IO from an isolated CPU?
> 
> I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> architecturally necessary and fully aligned with the rest of the kernel,
> no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> defers execution to a background workqueue. That work runs asynchronously
> at some indeterminate time in the future, long after the thread has
> returned to its time-critical user loop. Having a kworker suddenly preempt
> the thread later is likely not desired.

We are talking about `if a task executing on an isolated CPU submits I/O `.

If the task has been submitting IO on isolated CPU, why can't we keep
scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
makes any difference from user viewpoint.


Thanks,
Ming

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Sun, Oct 11, 2026 at 09:51:25PM -0500, Ming Lei wrote:
> > Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> > to provide a non-isolated CPU when dispatching &hctx->run_work, no?
> 
> Does every WORK_CPU_UNBOUND user have to handle it?

[Adding Tejun Heo]

No, because almost all other subsystems passing WORK_CPU_UNBOUND do so on
unbound workqueues (e.g. system_unbound_wq).

In kernel/workqueue.c, when WORK_CPU_UNBOUND is queued on a WQ_UNBOUND
workqueue, workqueue core already handles CPU isolation automatically via
wq_select_unbound_cpu(), which respects wq_unbound_cpumask and steers the
work item to a housekeeping CPU.

kblockd_workqueue is unique in that it is allocated as WQ_PERCPU to
preserve per-CPU locality for multi-queue devices. For WQ_PERCPU
workqueues, workqueue core cannot silently migrate WORK_CPU_UNBOUND to a
remote CPU because callers of bound per-CPU workqueues expect local per-CPU
execution. Thus, blk-mq is in the unusual position of passing
WORK_CPU_UNBOUND to a bound per-CPU workqueue with the expectation that it
avoids isolated cores.

> > I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> > architecturally necessary and fully aligned with the rest of the kernel,
> > no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> > defers execution to a background workqueue. That work runs asynchronously
> > at some indeterminate time in the future, long after the thread has
> > returned to its time-critical user loop. Having a kworker suddenly preempt
> > the thread later is likely not desired.
> 
> We are talking about `if a task executing on an isolated CPU submits I/O `.
> 
> If the task has been submitting IO on isolated CPU, why can't we keep
> scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
> makes any difference from user viewpoint.

When a task submits I/O, synchronous submission occurs directly in syscall
context. Note, in the following example, IOCB_DIRECT is applied since the
file was opened with file->f_flags & O_DIRECT:

  __x64_sys_write
    ksys_write
      vfs_write
        else if (file->f_op->write_iter)
          new_sync_write(file, buf, count, pos)
            // filp->f_op->write_iter(&kiocb, &iter)
            ext4_file_write_iter
              if (iocb->ki_flags & IOCB_DIRECT)
               ext4_dio_write_iter(iocb, from) 
                 iomap_dio_rw(iocb, from, &ext4_iomap_ops,
                              &ext4_dio_write_ops, dio_flags, NULL, 0)
                   __iomap_dio_rw
                     ...
                     iomap_dio_submit_bio
                       blk_crypto_submit_bio(bio)
                         if (!bio_has_crypt_ctx(bio) ...)
                           submit_bio(bio)
                            ...
                             __submit_bio
                               blk_mq_submit_bio(bio)
                                 // blk_mq_run_dispatch_ops(q, blk_mq_try_issue_directly(hctx, rq))
                                 __blk_mq_run_dispatch_ops(q, true blk_mq_try_issue_directly(hctx, rq))
                                   blk_mq_try_issue_directly(hctx, rq)
                                     __blk_mq_issue_directly(hctx, rq, true)
                                       // q->mq_ops->queue_rq(hctx, &bd)


However, if dispatch must be deferred (e.g. BLK_STS_RESOURCE),
blk-mq queues &hctx->run_work. Now, if &hctx->run_work is queued on the
isolated CPU, it must be executed by the local kworker. Because that
kworker runs under SCHED_NORMAL, the user's SCHED_FIFO task completely
starves the kworker. As a result, blk_mq_run_work_fn() never executes,
hardware dispatch stalls, and the I/O pipeline hangs indefinitely.
Albeit, I believe an enabled Fair server, under the EEVDF scheduler, will
prevent this stall.

Anyhow, offloading deferred dispatch to an online housekeeping CPU allows
the kworker to run immediately and complete dispatch without priority
inversion. As noted in commit a46c27026da1:
    "it isn't expected to let block IO disturb these CPUs. blk-mq kworker
     shouldn't be scheduled on isolated CPUs."


Kind regards,
-- 
Aaron Tomlin