From: Aaron Tomlin <atomlin@atomlin.com>
To: axboe@kernel.dk
Cc: tom.leiming@gmail.com, hare@suse.de, dwagner@suse.de,
emilne@redhat.com, frederic@kernel.org,
marco.crivellari@suse.com, tj@kernel.org, atomlin@atomlin.com,
sean@ashe.io, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
Subject: [PATCH v5] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty
Date: Sat, 21 Nov 2026 18:52:03 -0500 [thread overview]
Message-ID: <20261121235203.811792-1-atomlin@atomlin.com> (raw)
Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.
However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay (for instance,
upon hardware queue congestion, tag/resource exhaustion returning
BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
the local isolated CPU's worker pool. This schedules the local per-CPU
high-priority kworker (such as kworker/6:1H-kblockd) on the isolated
CPU. In real-time environments, an active SCHED_FIFO workload will
starve this SCHED_NORMAL worker and stall hardware dispatch, whilst also
introducing unwanted latency jitter on nohz_full cores, defeating that
commit's original objective.
Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
from housekeeping_cpumask(HK_TYPE_DOMAIN) via
cpumask_any_and_distribute() when executing on an isolated CPU, and
otherwise return WORK_CPU_UNBOUND. Gate the helper behind the static key
housekeeping_overridden under CONFIG_CPU_ISOLATION so that common
configurations without CPU isolation incur zero overhead across
single-queue devices, hardware queues with empty cpumasks, and offline
CPU fallback paths, whilst non-isolated CPUs preserve local execution
via WORK_CPU_UNBOUND. Guard the slow-path check and cpumask selection
with guard(preempt)() to pin raw_smp_processor_id(), satisfy RCU
dereferencing of housekeeping_cpumask(HK_TYPE_DOMAIN), and protect the
per-CPU distribution state.
Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <tom.leiming@gmail.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Marco Crivellari <marco.crivellari@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Note: As Frederic Weisbecker pointed out on v4, workqueue-level support
(WQ_PREFER_PERCPU) is currently in development to handle isolated CPUs
automatically, at which point kblockd_workqueue will be a candidate for
simplification; in the meantime, this provides a minimal fix suitable
for backporting.
Changes since v4:
- Rebased against v7.4-rc1
- Hoisted #ifdef CONFIG_CPU_ISOLATION and
static_branch_unlikely(&housekeeping_overridden) into
blk_mq_hctx_unbound_cpu() so blk_mq_hctx_next_cpu() retains a single
fast-path check and non-isolated configurations avoid any RCU or
preemption guards across all unbound fallback paths
- Moved guard(preempt)() ahead of cpu_is_isolated(raw_smp_processor_id())
to prevent migration between reading the local CPU ID and selecting a
housekeeping CPU under CONFIG_PREEMPT_RCU, whilst satisfying
rcu_dereference_all_check() without a separate guard(rcu)()
- Consolidated the commit message summary into prose
- Link to v4: https://lore.kernel.org/lkml/20261022034737.47820-1-atomlin@atomlin.com/
Changes since v3:
- Create companion blktests test case (block/050) to reproduce this issue
and verify the fix (Daniel Wagner)
- Gate the single-queue (nr_hw_queues == 1) check behind the static key
housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the
common case without CPU isolation incurs zero overhead on the fast
path (Jens Axboe)
- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in
blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure
safe dereferencing of the housekeeping cpumask while concurrent cpuset
updates occur
- Disable preemption via guard(preempt)() around cpumask_any_and_distribute()
to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid
data races when accessing per-CPU distribution state
- Link to v3: https://lore.kernel.org/all/20261016052203.489760-1-atomlin@atomlin.com/
Changes since v2:
- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
isolated partitions
- Query the housekeeping cpumask under guard(rcu)() to safely handle
dynamic cpuset updates without lockdep warnings
- Validate that cpumask_any_and_distribute() returns a valid cpu ID
before returning, falling back cleanly to WORK_CPU_UNBOUND
- Link to v2: https://lore.kernel.org/lkml/20261015020225.422107-1-atomlin@atomlin.com/
Changes since v1:
- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
to an online housekeeping CPU only when executing on an isolated
core, preserving WORK_CPU_UNBOUND and local cache locality for
housekeeping cores
- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
(nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
CPU fallback path.
- Elaborate in the commit log on the triggers for asynchronous queue
dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
plugging) and the SCHED_FIFO worker starvation hazard on isolated
cores
- Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/
---
block/blk-mq.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index d9ab295be4d9..4657a3287f3a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2252,6 +2252,25 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
return hctx->next_cpu >= nr_cpu_ids;
}
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+#ifdef CONFIG_CPU_ISOLATION
+ if (static_branch_unlikely(&housekeeping_overridden)) {
+ int cpu;
+
+ guard(preempt)();
+ if (!cpu_is_isolated(raw_smp_processor_id()))
+ return WORK_CPU_UNBOUND;
+
+ cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+ cpu_online_mask);
+ if (cpu < nr_cpu_ids)
+ return cpu;
+ }
+#endif
+ return WORK_CPU_UNBOUND;
+}
+
/*
* It'd be great if the workqueue API had a way to pass
* in a mask and had some smarts for more clever placement.
@@ -2265,7 +2284,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
/* Switch to unbound if no allowable CPUs in this hctx */
if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
- return WORK_CPU_UNBOUND;
+ return blk_mq_hctx_unbound_cpu();
if (--hctx->next_cpu_batch <= 0) {
select_cpu:
@@ -2292,7 +2311,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
*/
hctx->next_cpu = next_cpu;
hctx->next_cpu_batch = 1;
- return WORK_CPU_UNBOUND;
+ return blk_mq_hctx_unbound_cpu();
}
hctx->next_cpu = next_cpu;
--
2.55.0
reply other threads:[~2026-11-21 23:52 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121235203.811792-1-atomlin@atomlin.com \
--to=atomlin@atomlin.com \
--cc=axboe@kernel.dk \
--cc=dwagner@suse.de \
--cc=emilne@redhat.com \
--cc=frederic@kernel.org \
--cc=hare@suse.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=marco.crivellari@suse.com \
--cc=sean@ashe.io \
--cc=stable@vger.kernel.org \
--cc=tj@kernel.org \
--cc=tom.leiming@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®