[PATCH v4] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[PATCH v4] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay (e.g. upon
hardware queue congestion, tag/resource exhaustion returning
BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
the local isolated CPU's worker pool. This causes the local per-CPU
high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
the isolated CPU. In real-time environments, an active SCHED_FIFO workload
will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
in addition to introducing unwanted latency jitter on nohz_full cores,
thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
from housekeeping_cpumask(HK_TYPE_DOMAIN) via
cpumask_any_and_distribute() when executing on an isolated CPU, and
otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
unbound fallback paths in blk_mq_hctx_next_cpu().

Additionally, for single-queue devices, gate the check behind the static
key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that
common configurations without CPU isolation incur zero overhead on the
fast path.

This ensures that:
    1.  When executing on an isolated CPU, deferred queue work is
        cleanly offloaded to an online housekeeping CPU across single-queue
        devices, hardware queues with empty cpumasks, and offline CPU
        fallback paths.

    2.  The per-CPU high-priority kworker on the isolated CPU is not
        scheduled, avoiding priority inversion and latency jitter.

    3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
        preserved, maintaining optimal local execution with zero CPU
        bouncing while avoiding timer pinning hazards.

    4.  On systems without CPU isolation, single-queue devices maintain
        the zero-overhead fast path via a static key.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Changes since v3:

 - Create companion blktests test case (block/050) to reproduce this issue
   and verify the fix (Daniel Wagner)

 - Gate the single-queue (nr_hw_queues == 1) check behind the static key
   housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the
   common case without CPU isolation incurs zero overhead on the fast
   path (Jens Axboe)

 - Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in
   blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure
   safe dereferencing of the housekeeping cpumask while concurrent cpuset
   updates occur

 - Disable preemption via guard(preempt)() around cpumask_any_and_distribute()
   to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid
   data races when accessing per-CPU distribution state

 - Link to v3: https://lore.kernel.org/all/20261016052203.489760-1-atomlin@atomlin.com/

Changes since v2:

 - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
   to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
   isolated partitions

 - Query the housekeeping cpumask under guard(rcu)() to safely handle
   dynamic cpuset updates without lockdep warnings

 - Validate that cpumask_any_and_distribute() returns a valid cpu ID
   before returning, falling back cleanly to WORK_CPU_UNBOUND

 - Link to v2: https://lore.kernel.org/lkml/20261015020225.422107-1-atomlin@atomlin.com/

Changes since v1:

 - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
   to an online housekeeping CPU only when executing on an isolated
   core, preserving WORK_CPU_UNBOUND and local cache locality for
   housekeeping cores

 - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
   blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
   (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
   CPU fallback path.

 - Elaborate in the commit log on the triggers for asynchronous queue
   dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
   plugging) and the SCHED_FIFO worker starvation hazard on isolated
   cores

 - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/
---
 block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 025a799f35b9..f8f862a0e525 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	guard(rcu)();
+
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;
+
+		guard(preempt)();
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+
+	return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2252,9 +2269,17 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 	bool tried = false;
 	int next_cpu = hctx->next_cpu;
 
-	/* Switch to unbound if no allowable CPUs in this hctx */
-	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
+	if (hctx->queue->nr_hw_queues == 1) {
+#ifdef CONFIG_CPU_ISOLATION
+		if (static_branch_unlikely(&housekeeping_overridden))
+			return blk_mq_hctx_unbound_cpu();
+#endif
 		return WORK_CPU_UNBOUND;
+	}
+
+	/* Switch to unbound if no allowable CPUs in this hctx */
+	if (blk_mq_hctx_empty_cpumask(hctx))
+		return blk_mq_hctx_unbound_cpu();
 
 	if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2306,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 		 */
 		hctx->next_cpu = next_cpu;
 		hctx->next_cpu_batch = 1;
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 	}
 
 	hctx->next_cpu = next_cpu;
-- 
2.55.0