[PATCH v16 6/9] blk-mq: use hk cpus only when isolcpus=managed_irq_strict is enabled

[Aaron Tomlin <atomlin@atomlin.com>]

From: Daniel Wagner <wagi@kernel.org>

Extend the capabilities of the generic CPU to hardware queue (hctx)
mapping code, so it maps housekeeping CPUs and isolated CPUs to the
hardware queues evenly.

Example mapping result:

    16 online CPUs

    isolcpus=managed_irq_strict,2-3,6-7,12-13

        Queue mapping:
                hctx0: default 0 2
                hctx1: default 1 3
                hctx2: default 4 6
                hctx3: default 5 7
                hctx4: default 8 12
                hctx5: default 9 13
                hctx6: default 10
                hctx7: default 11
                hctx8: default 14
                hctx9: default 15

        IRQ mapping:
                irq 42 affinity 0 effective 0  nvme0q0
                irq 43 affinity 0 effective 0  nvme0q1
                irq 44 affinity 1 effective 1  nvme0q2
                irq 45 affinity 4 effective 4  nvme0q3
                irq 46 affinity 5 effective 5  nvme0q4
                irq 47 affinity 8 effective 8  nvme0q5
                irq 48 affinity 9 effective 9  nvme0q6
                irq 49 affinity 10 effective 10  nvme0q7
                irq 50 affinity 11 effective 11  nvme0q8
                irq 51 affinity 14 effective 14  nvme0q9
                irq 52 affinity 15 effective 15  nvme0q10

In this scenario, the system has 16 online CPUs with 6 isolated cores
(2, 3, 6, 7, 12, and 13) and 10 housekeeping cores (0, 1, 4, 5, 8, 9,
10, 11, 14, and 15):

    1.  Queue allocation and ownership

        Rather than allocating 16 hardware queues, the block layer
        allocates only 10 hardware contexts (hctx0 to hctx9),
        corresponding strictly to the 10 housekeeping CPUs. The 6
        isolated CPUs do not own dedicated hardware queues; instead,
        they are mapped across the existing active housekeeping queues
        (e.g. isolated CPU 2 shares hctx0 with CPU 0). This ensures
        tasks running on isolated CPUs can still issue I/O without
        restriction.

    2.  Interrupt routing

        All device interrupts, including the NVMe admin queue (irq 42)
        and the 10 I/O completion queues (irq 43 to 52), target
        housekeeping CPUs exclusively. When a task on isolated CPU 2
        issues I/O via hctx0, the resulting completion interrupt (irq
        43) fires on housekeeping CPU 0. Consequently, isolated CPUs are
        never interrupted by device hardware, guaranteeing zero latency
        disturbance for isolated workloads.

A corner case is when the number of online CPUs and present CPUs
differ and the driver asks for less queues than online CPUs, e.g.

    8 online CPUs, 16 possible CPUs

    isolcpus=managed_irq_strict,2-3,6-7,12-13
    virtio_blk.num_request_queues=2

        Queue mapping:
                hctx0: default 0 1 2 3 4 5 6 7 8 12 13
                hctx1: default 9 10 11 14 15

        IRQ mapping
                irq 27 affinity 0 effective 0 virtio0-config
                irq 28 affinity 0-1,4-5,8 effective 5 virtio0-req.0
                irq 29 affinity 9-11,14-15 effective 0 virtio0-req.1

This corner case demonstrates behaviour when hardware queue counts are
constrained (only 2 request queues) on a system with CPU hotplug (8
CPUs online out of 16 possible):

    1.  Coarse queue grouping

        Because the driver requests only 2 queues, the 10 possible
        housekeeping CPUs are partitioned into two groups: hctx0
        receives CPUs 0-1, 4-5, and 8, while hctx1 receives CPUs 9-11
        and 14-15. All isolated CPUs (both online cores 2-3, 6-7 and
        offline cores 12-13) are mapped to hctx0 to share submission
        capacity without allocating excess queues.

    2.  Isolation and hotplug protection in interrupt affinity

        Although hctx0 serves both housekeeping and isolated CPUs, the
        resulting interrupt affinity mask for irq 28 (affinity
        0-1,4-5,8) strictly includes only the housekeeping cores,
        completely excluding isolated cores 2-3, 6-7, and 12-13.
        Furthermore, for hctx1 (irq 29), whose assigned housekeeping
        CPUs (9-11, 14-15) are currently offline, the kernel routes the
        effective interrupt to an available online housekeeping core
        (CPU 0), guaranteeing that interrupts never spill onto isolated
        cores under any hotplug state.

Noteworthy is that for the normal/default configuration (without
isolcpus=) the mapping will change for systems which have non
hyperthreading CPUs. The main assignment loop will completely rely that
group_mask_cpus_evenly to do the right thing. The old code would
distribute the CPUs linearly over the hardware context:

    queue mapping for /dev/nvme0n1
            hctx0: default 0 8
            hctx1: default 1 9
            hctx2: default 2 10
            hctx3: default 3 11
            hctx4: default 4 12
            hctx5: default 5 13
            hctx6: default 6 14
            hctx7: default 7 15

The assign each hardware context the map generated by the
group_mask_cpus_evenly function:

    queue mapping for /dev/nvme0n1
            hctx0: default 0 1
            hctx1: default 2 3
            hctx2: default 4 5
            hctx3: default 6 7
            hctx4: default 8 9
            hctx5: default 10 11
            hctx6: default 12 13
            hctx7: default 14 15

In case of hyperthreading CPUs, the resulting map stays the same.

Signed-off-by: Daniel Wagner <wagi@kernel.org>
Co-developed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 block/blk-mq-cpumap.c | 163 +++++++++++++++++++++++++++++++++++++-----
 1 file changed, 145 insertions(+), 18 deletions(-)

diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c
index 705da074ad6c..cec5b26bd57c 100644
--- a/block/blk-mq-cpumap.c
+++ b/block/blk-mq-cpumap.c
@@ -22,8 +22,15 @@ static unsigned int blk_mq_num_queues(const struct cpumask *mask,
 {
 	unsigned int num;
 
-	num = cpumask_weight(mask);
-	return min_not_zero(num, max_queues);
+	if (housekeeping_enabled(HK_TYPE_MANAGED_IRQ_STRICT))
+		num = cpumask_weight_and(mask, housekeeping_cpumask(HK_TYPE_MANAGED_IRQ_STRICT));
+	else
+		num = cpumask_weight(mask);
+	/*
+	 * Ensure that a count of zero does not inadvertently result in
+	 * allocating the maximum number of queues.
+	 */
+	return min_not_zero(num ?: 1U, max_queues);
 }
 
 /**
@@ -33,7 +40,8 @@ static unsigned int blk_mq_num_queues(const struct cpumask *mask,
  *		ignored.
  *
  * Calculates the number of queues to be used for a multiqueue
- * device based on the number of possible CPUs.
+ * device based on the number of possible CPUs. This helper
+ * takes isolcpus settings into account.
  */
 unsigned int blk_mq_num_possible_queues(unsigned int max_queues)
 {
@@ -48,7 +56,8 @@ EXPORT_SYMBOL_GPL(blk_mq_num_possible_queues);
  *		ignored.
  *
  * Calculates the number of queues to be used for a multiqueue
- * device based on the number of online CPUs.
+ * device based on the number of online CPUs. This helper
+ * takes isolcpus settings into account.
  */
 unsigned int blk_mq_num_online_queues(unsigned int max_queues)
 {
@@ -56,23 +65,81 @@ unsigned int blk_mq_num_online_queues(unsigned int max_queues)
 }
 EXPORT_SYMBOL_GPL(blk_mq_num_online_queues);
 
+static void blk_mq_map_fallback(struct blk_mq_queue_map *qmap)
+{
+	unsigned int cpu;
+
+	/*
+	 * Map all CPUs to the first hctx of this specific map, respecting
+	 * the map's boundaries so secondary maps do not route into the default map.
+	 */
+	for_each_possible_cpu(cpu)
+		qmap->mq_map[cpu] = qmap->queue_offset;
+}
+
 void blk_mq_map_queues(struct blk_mq_queue_map *qmap)
 {
-	const struct cpumask *masks;
+	struct cpumask *masks;
+	const struct cpumask *constraint;
 	unsigned int queue, cpu, nr_masks;
+	unsigned long *active_hctx;
 
-	masks = group_cpus_evenly(qmap->nr_queues, &nr_masks);
-	if (!masks) {
-		for_each_possible_cpu(cpu)
-			qmap->mq_map[cpu] = qmap->queue_offset;
-		return;
-	}
+	active_hctx = bitmap_zalloc(qmap->nr_queues, GFP_KERNEL);
+	if (!active_hctx)
+		goto fallback;
 
-	for (queue = 0; queue < qmap->nr_queues; queue++) {
-		for_each_cpu(cpu, &masks[queue % nr_masks])
+	if (housekeeping_enabled(HK_TYPE_MANAGED_IRQ_STRICT))
+		constraint = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ_STRICT);
+	else
+		constraint = cpu_possible_mask;
+
+	/* Map CPUs to the hardware contexts (hctx) */
+	masks = group_mask_cpus_evenly(qmap->nr_queues, constraint, &nr_masks);
+	if (!masks)
+		goto free_fallback_hctx;
+
+	/*
+	 * Iterate directly over the generated CPU masks.
+	 * Calculate the final, highest hardware queue index that maps to this
+	 * mask. This skips all intermediate overwrites and safely evaluates
+	 * active_hctx only for queues that survive the mapping.
+	 */
+	for (unsigned int idx = 0; idx < nr_masks; idx++) {
+		queue = qmap->nr_queues - 1 -
+			((qmap->nr_queues - 1 - idx) % nr_masks);
+
+		for_each_cpu(cpu, &masks[idx])
 			qmap->mq_map[cpu] = qmap->queue_offset + queue;
+
+		__set_bit(queue, active_hctx);
+	}
+
+	/*
+	 * If the active_hctx bitmap is empty, attempting to route unassigned
+	 * CPUs will map them out-of-bounds. Fall back instead.
+	 */
+	if (bitmap_empty(active_hctx, qmap->nr_queues))
+		goto free_fallback;
+
+	/* Map any unassigned CPU evenly to the hardware contexts (hctx) */
+	queue = find_first_bit(active_hctx, qmap->nr_queues);
+	for_each_cpu_andnot(cpu, cpu_possible_mask, constraint) {
+		qmap->mq_map[cpu] = qmap->queue_offset + queue;
+		queue = find_next_bit_wrap(active_hctx, qmap->nr_queues, queue + 1);
 	}
+
+	kfree(masks);
+	bitmap_free(active_hctx);
+
+	return;
+
+free_fallback:
 	kfree(masks);
+free_fallback_hctx:
+	bitmap_free(active_hctx);
+
+fallback:
+	blk_mq_map_fallback(qmap);
 }
 EXPORT_SYMBOL_GPL(blk_mq_map_queues);
 
@@ -109,24 +176,84 @@ void blk_mq_map_hw_queues(struct blk_mq_queue_map *qmap,
 			  struct device *dev, unsigned int offset)
 
 {
-	const struct cpumask *mask;
+	cpumask_var_t mask;
+	const struct cpumask *constraint;
+	unsigned long *active_hctx;
 	unsigned int queue, cpu;
 
 	if (!dev->bus->irq_get_affinity)
+		goto map_software;
+
+	active_hctx = bitmap_zalloc(qmap->nr_queues, GFP_KERNEL);
+	if (!active_hctx)
 		goto fallback;
 
+	if (!zalloc_cpumask_var(&mask, GFP_KERNEL)) {
+		bitmap_free(active_hctx);
+		goto fallback;
+	}
+
+	if (housekeeping_enabled(HK_TYPE_MANAGED_IRQ_STRICT))
+		constraint = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ_STRICT);
+	else
+		constraint = cpu_possible_mask;
+
+	/* Map CPUs to the hardware contexts (hctx) */
 	for (queue = 0; queue < qmap->nr_queues; queue++) {
-		mask = dev->bus->irq_get_affinity(dev, queue + offset);
-		if (!mask)
-			goto fallback;
+		const struct cpumask *affinity_mask;
+
+		affinity_mask = dev->bus->irq_get_affinity(dev, offset + queue);
+		if (!affinity_mask)
+			goto free_map_software;
 
-		for_each_cpu(cpu, mask)
+		for_each_cpu(cpu, affinity_mask) {
 			qmap->mq_map[cpu] = qmap->queue_offset + queue;
+			cpumask_set_cpu(cpu, mask);
+		}
 	}
 
+	/*
+	 * Evaluate active_hctx after mapping to handle overlapping masks.
+	 * This ensures queues that were overwritten do not falsely pass validation.
+	 */
+	for_each_cpu(cpu, mask) {
+		if (cpumask_test_cpu(cpu, constraint)) {
+			queue = qmap->mq_map[cpu] - qmap->queue_offset;
+			__set_bit(queue, active_hctx);
+		}
+	}
+
+	/*
+	 * If no assigned CPU matches the constraint, the active_hctx
+	 * bitmap will be empty. Fall back instead of routing out of bounds.
+	 */
+	if (bitmap_empty(active_hctx, qmap->nr_queues))
+		goto free_fallback;
+
+	/* Map any unassigned CPU evenly to the hardware contexts (hctx) */
+	queue = find_first_bit(active_hctx, qmap->nr_queues);
+	for_each_cpu_andnot(cpu, cpu_possible_mask, mask) {
+		qmap->mq_map[cpu] = qmap->queue_offset + queue;
+		queue = find_next_bit_wrap(active_hctx, qmap->nr_queues, queue + 1);
+	}
+
+	bitmap_free(active_hctx);
+	free_cpumask_var(mask);
+
 	return;
 
+free_fallback:
+	bitmap_free(active_hctx);
+	free_cpumask_var(mask);
+
 fallback:
+	blk_mq_map_fallback(qmap);
+	return;
+
+free_map_software:
+	free_cpumask_var(mask);
+	bitmap_free(active_hctx);
+map_software:
 	blk_mq_map_queues(qmap);
 }
 EXPORT_SYMBOL_GPL(blk_mq_map_hw_queues);
-- 
2.55.0