[RFC PATCH v2 00/23] sched: Scale cache-aware aggregation at LLC granularity

[RFC PATCH v2 00/23] sched: Scale cache-aware aggregation at LLC granularity

[Jianyong Wu]

[Problem Statement]
The current cache-aware scheduling implementation adopts an
LLC-centric task aggregation model. While effective for workloads
that fit within a single LLC domain, this design is fundamentally
limited by its fixed aggregation scope and cannot scale across LLCs.

[Proposal]
Peter Zijlstra suggested scaling resources at LLC granularity instead,
and this patch set implements that idea.

To achieve this goal, we need to answer two questions: in which order
should LLCs be considered, and how far along that order does a thread
group need to spread? The scheduler aggregates threads near the
preferred LLC first, then expands through the ordered LLCs as the
thread group's aggregate demand requires.

For each preferred LLC, the topology-based affinity model defines a
fixed order in which LLCs should be used. The scheduler estimates the
thread group's total utilization and selects the smallest leading
portion of that order whose allowed capacity can accommodate the
group. A task may migrate to an LLC in that portion only if the
destination can also accommodate the task.

For example, in a simplified system, there are 4 NUMA nodes, each
node with 4 LLCs. The node distance matrix is:
        NODE0    NODE1    NODE2    NODE3
NODE0   10       20       30       40
NODE1   20       10       25       30
NODE2   30       25       10       20
NODE3   40       30       20       10

Given a preferred LLC, its node's distance row ranks the other NUMA
nodes. For example, if the preferred LLC is in NODE0, NODE1 is ranked
before NODE2, so LLCs in NODE1 precede LLCs in NODE2 in the resulting
affinity sequence.

To rank LLCs inside the node containing the preferred LLC, there is no
existing LLC distance matrix to use. So, we construct one which has no
real physical meaning and is only a hint for constructing the affinity
sequence. For example, consider this LLC distance matrix:

        LLC0    LLC1    LLC2    LLC3
LLC0    0       2       3       4
LLC1    2       0       4       1
LLC2    3       4       0       2
LLC3    4       1       2       0

Each row ranks the other LLCs relative to the LLC represented by that
row. With LLC0 as the preferred LLC, its row ranks LLC1 before LLC2 and
LLC2 before LLC3. A different preferred LLC selects a different row and
may therefore produce a different order.

Combining the node ranking, the selected preferred-LLC row, and the
ascending LLC-ID order used inside other nodes produces one affinity
sequence for a given preferred LLC. It does not define a global order
between arbitrary LLC pairs.

Note that there may be identical numbers inside a row of a real
node distance. So, we should build a new node distance matrix in
which the distance values within each row are unique. This node
distance matrix is only used for migration decisions in the
cache-aware scheduling (CAS) extension.

Once a preferred LLC affinity sequence is constructed, the scheduler
uses it as an input to migration decisions. For example, assume LLC0
is the preferred LLC and its affinity sequence is as follows:
LLC0->LLC1->LLC2->LLC3...->LLC15. (Note that this is a simple example.
Generally, for the node containing the preferred LLC, the sequence is
decided by the LLC distance matrix inside the node. For other nodes,
LLCs are ordered by ascending LLC ID within each node,
while the node sequence is decided by the node distance matrix.)

The affinity sequence gives an ordered expansion preference, rather
than an unconditional per-LLC saturation gate. The scheduler estimates
the aggregate utilization of a thread group and derives the furthest
LLC-distance tier required for that demand. A destination LLC within
the resulting range is eligible when it has capacity for the task,
even if an earlier LLC in the sequence is not yet saturated. For
example, LLC3 may be selected when the estimated range reaches LLC3
and LLC3 has capacity. A destination outside that range still uses
the ordered saturation check to preserve aggregation.

Thus, for a given preferred LLC, the affinity sequence is fixed, while
the scheduler dynamically determines how many LLCs from the beginning
of that sequence the thread group may use, based on its estimated total
utilization.

Before this implementation, we tried to maintain an LLC mask for each
process's thread group to record its current resource scope. However,
this approach proved to be difficult to maintain. The mask is updated in
task_cache_work() but consulted during load balancing, so rapid
workload changes can leave the mask temporarily out of date. As a
result, it may no longer accurately represent the thread group's
actual resource usage, making migration decisions unreliable.

The LLC mask is also insufficient for guiding source group
selection during load balancing. The mask is associated with a
specific thread group, but when load balancing searches for a
source sched_group or runqueue, there is no available task context
to retrieve the corresponding LLC mask. We need a task-independent
mechanism that can provide such information during load balancing.

This patch set also tries to address NUMA balancing-related impacts.
With NUMA balancing enabled, the preferred LLC fails to stay stable
since the scan range is limited to the current task's preferred node.
We fix this by adding all preferred nodes of active tasks in the thread
group to the scan range.

We also seek to suppress task-migration conflicts between CAS and
NUMA balancing, including splitting NUMA balancing into task and page
migration paths (suggested by Chen Yu <yu.c.chen@intel.com>) and removing
unnecessary preferred node checks in load balancing code. These parts
are unfinished and open for further discussion.

[Patch organization]
The series is organized as follows:

  Patches 1-6 build the topology infrastructure: LLC-to-node mapping,
  unique node-distance values, node/LLC traversal helpers, and an sd_node
  scheduling domain.

  Patches 7-10 collect preferred-NUMA-node information for tasks and CPUs,
  and add the per-sched-domain state needed by load balancing.

  Patches 11-16 implement LLC-granular migration decisions. They calculate
  affinity gain, select source runqueues and groups, decide migration
  eligibility, and allow active balancing to expand to another LLC.

  Patches 17-19 handle the interaction with NUMA balancing: they make its
  decisions LLC-granular, scan all preferred nodes of a thread group, and
  remove no-longer-needed preferred LLC/node restrictions.

  Patches 20-22 estimate whole-thread-group utilization, derive the LLC
  range needed for that utilization, allow migration within that range, and
  walk the preferred node starting from the preferred LLC.

  Patch 23 exposes a task's preferred LLC for scheduler debugging.

This patch set is far from perfect and still contains some unresolved
issues. Before proceeding further, I would like to confirm whether I am
heading in the right direction. Therefore, I am sending these patches
out to gather early feedback.

v1: https://lore.kernel.org/all/20260625030759.25928-1-wujianyong@hygon.cn/

Changes since v1:
(1) No longer scale resources based on sched domain boundaries.
(2) Add a new node distance matrix with no duplicate values within a row.
(3) Add an intra-node LLC distance matrix to rank candidate LLCs
    relative to a given preferred LLC within a node.
(4) Change the way affinity gain is calculated.
(5) Change the way migration permission is decided.
(6) Fix saturation issues in some tests like schbench.
(7) Include all preferred nodes related to the thread group into the scan
    range.
(8) Add a tunable to enable/disable task/page migration for NUMA balancing
    independently. (Suggested by Chen Yu)
(9) Estimate whole thread-group utilization to derive its LLC capacity
    range.
(10) Permit migration to an LLC within that estimated range, even when an
     earlier LLC in the affinity sequence is not saturated.
(11) Walk LLCs in the preferred node from the preferred LLC.
(12) Fix a bug in the debug print patch. (Suggested by XIAO WU)

Tested on a Hygon machine with the following topology:
* 2 sockets
* 4 NUMA nodes per socket
* 4 LLCs per NUMA node
* 4 cores per LLC domain
* 2 SMT threads per core

The scheduler changes apply cleanly on Linus' tree at 0f23d56f17fd
("Merge tag 'linux_kselftest-next-7.3-rc1' of ...kselftest") and build
there.

Functional test:
Ran a busy loop test program with 4, 8, 12, 16, 20, 24, 28, 32, and
64 threads. The scheduler is able to roughly scale resources at LLC
granularity.

Performance test:
* llc_gran refers to this patch set.
* The baseline is the same tree without this series, that is Linus' tree at
  0f23d56f17fd. Note that the baseline already carries CONFIG_SCHED_CACHE,
  so these numbers compare the existing single-LLC aggregation against the
  LLC-granular expansion this series adds, not against a kernel without
  cache-aware scheduling.
* Both kernels run with NUMA balancing disabled and with 'aggr_tolerance'
  set to 90, so the two aggregate under the same threshold and the
  difference reflects the code rather than the tunable.
* Each test is repeated at least 20 times; results take the average value.

To reproduce:

  # baseline
  git checkout 0f23d56f17fd
  make olddefconfig && make -j$(nproc) && install and boot it

  # llc_gran
  git am v2-00*.patch     # the 23 patches of this series
  make olddefconfig && make -j$(nproc) && install and boot it

  # on each kernel, before measuring
  echo 0  > /proc/sys/kernel/numa_balancing
  echo 90 > /sys/kernel/debug/sched/llc_balancing/aggr_tolerance

  # then
  for f in 2 4 8 12 16 20 24 32 48; do
      for g in 1 2; do
          hackbench -T -p -f $f -g $g -l 100000
      done
  done
  for t in 2 4 6 8 12 16 32 48 64 96 128; do
      schbench -m 1 -t $t -r 30
  done

The reported schbench figure is the 99.0th percentile of the Wakeup
Latencies block from the final cumulative report.

[hackbench]
(lower is better, normalized to baseline; the figure in parentheses is the
standard deviation, as a percentage of the mean)
test cmd: hackbench -T -p -f $f -g $g -l 100000

pipe   groups  baseline          llc_gran          lg improve
============================================================
2      1       1.000 (47.90%)    0.919 (3.73%)     8.094%
2      2       1.000 (58.79%)    0.799 (11.97%)    20.126%
4      1       1.000 (42.69%)    0.930 (26.53%)    6.966%
4      2       1.000 (29.86%)    0.717 (8.96%)     28.312%
8      1       1.000 (20.44%)    0.654 (7.32%)     34.613%
8      2       1.000 (11.39%)    0.711 (12.57%)    28.928%
12     1       1.000 (13.72%)    0.646 (9.45%)     35.399%
12     2       1.000 (10.97%)    0.701 (6.73%)     29.907%
16     1       1.000 (12.48%)    0.703 (7.51%)     29.711%
16     2       1.000 (5.28%)     0.728 (6.41%)     27.171%
20     1       1.000 (5.89%)     0.663 (3.76%)     33.651%
20     2       1.000 (5.47%)     0.861 (4.25%)     13.894%
24     1       1.000 (6.18%)     0.678 (2.88%)     32.167%
24     2       1.000 (6.23%)     0.942 (6.78%)     5.776%
32     1       1.000 (4.64%)     0.690 (3.70%)     31.023%
32     2       1.000 (1.51%)     1.002 (4.86%)     -0.151%
48     1       1.000 (2.08%)     0.942 (2.45%)     5.847%
48     2       1.000 (1.51%)     1.041 (2.49%)     -4.060%

llc_gran is faster at 16 of the 18 configurations, by a median of 27.7%.
The largest gains occur in the middle of the range, especially for one group
from 8 to 32 pipes; gains taper off at both ends and vary with group count.

Two configurations are slower. Only 48 pipes with 2 groups exceeds
run-to-run noise, at -4.1%; 32 pipes with 2 groups is within it.

Run-to-run variance improves at 12 of the 18 configurations, most clearly
at the small end, where the baseline reaches 30-59% against 4-27% for
llc_gran.

[schbench]
p99 wakeup latency (lower is better, normalized to baseline; the figure in
parentheses is the standard deviation, as a percentage of the mean)
test cmd: schbench -m 1 -t $threads -r 30

threads   baseline          llc_gran          lg improve
=========================================================
2         1.000 (33.11%)    0.801 (6.27%)     19.86%
4         1.000 (31.09%)    0.787 (11.43%)    21.32%
6         1.000 (21.89%)    0.832 (11.11%)    16.79%
8         1.000 (15.20%)    0.863 (14.70%)    13.72%
12        1.000 (14.23%)    0.883 (1.97%)     11.68%
16        1.000 (11.30%)    0.919 (1.21%)     8.14%
32        1.000 (12.81%)    0.784 (2.97%)     21.55%
48        1.000 (6.79%)     0.670 (3.67%)     33.02%
64        1.000 (3.04%)     0.647 (3.28%)     35.33%
96        1.000 (3.18%)     0.863 (18.11%)    13.72%
128       1.000 (4.66%)     0.952 (15.24%)    4.79%

llc_gran improves the p99 wakeup latency at all 11 thread counts, by a
median of 16.8%, and 10 of the 11 exceed run-to-run noise. The gain is
largest at 48 and 64 threads (+33.0% and +35.3%). Only 128 threads, at
+4.8%, stays within noise.

Variance improves at 8 of the 11 thread counts. It remains higher than the
baseline at 96 and 128 threads, where llc_gran runs at 15-18% against
3-5%.

The root cause of the hackbench regression at 48 pipes, and of the wider
spread in wakeup latency at high thread counts, will be investigated in
future work.

Further testing across a wider range of workloads and hardware platforms
is needed.

Jianyong Wu (23):
  sched/topology: Add llc_to_node() to translate LLC id to NUMA node
  sched/topology: Introduce a NUMA distance matrix with unique distance
    values
  sched/topology: Introduce a macro to traverse node
  sched/topology: Introduce a method to calculate the llc distance
  sched/topology: Introduce a macro to traverse LLC inside node
  sched/topology: Add sd_node for the NODE sched domain
  sched/cache: Prioritize preferred NUMA node selection over LLC
    selection
  sched/topology: Introduce a per-CPU tasks NUMA preferred counter
  sched/cache: Account percpu sd task NUMA preference
  sched/topology: Add per-sd scratch for the load balance affinity score
  sched/cache: Introduce helpers for task migration decisions
  sched/cache: Introduce rq affinity gain calculation
  sched/cache: Pick optimal src rq/group using affinity promotion metric
  sched/cache: Drop prefer_sibling restriction for llc_balance
  sched/cache: Judge migration eligibility in LLC granularity
  sched/cache: Allow un-throttled active balance to spread out of a full
    LLC
  sched/fair: Fine-granularity NUMA balancing
  sched/cache: Scan all prefer nodes in thread group
  sched/cache: Remove preferred LLC/node check no longer needed
  sched/cache: Estimate utilization of the whole thread group
  sched/cache: Spread workloads within an estimated LLC range
  sched/cache: Walk the preferred node from the preferred LLC
  sched/debug: Print task preferred LLC for scheduler debugging

 include/linux/mm_types.h       |    6 +-
 include/linux/sched.h          |    5 +
 include/linux/sched/sysctl.h   |   12 +
 include/linux/sched/topology.h |    7 +
 include/linux/topology.h       |    6 +
 kernel/sched/core.c            |   87 +++
 kernel/sched/debug.c           |   28 +-
 kernel/sched/fair.c            | 1112 +++++++++++++++++++++++++++-----
 kernel/sched/sched.h           |   49 ++
 kernel/sched/topology.c        |  873 ++++++++++++++++++++++++-
 mm/memory.c                    |    3 +
 11 files changed, 2007 insertions(+), 181 deletions(-)

-- 
2.34.1

[RFC PATCH v2 01/23] sched/topology: Add llc_to_node() to translate LLC id to NUMA node

[Jianyong Wu]

Building an LLC affinity/distance matrix requires knowing which NUMA
node each LLC belongs to. Add a per-LLC-id -> NUMA-node map
(llc_to_node_map) derived from the possible CPU set, and expose it
via llc_to_node().

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/topology.h |  4 +++
 kernel/sched/topology.c  | 56 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git a/include/linux/topology.h b/include/linux/topology.h
index 709a2dcf4c73..9967739a180c 100644
--- a/include/linux/topology.h
+++ b/include/linux/topology.h
@@ -177,6 +177,10 @@ static inline int cpu_to_mem(int cpu)
 
 #endif	/* [!]CONFIG_HAVE_MEMORYLESS_NODES */
 
+#ifdef CONFIG_SCHED_CACHE
+int llc_to_node(int llc);
+#endif
+
 #if defined(topology_die_id) && defined(topology_die_cpumask)
 #define TOPOLOGY_DIE_SYSFS
 #endif
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 622e2e01974c..c6928c6b17b6 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -685,6 +685,11 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+#ifdef CONFIG_SCHED_CACHE
+static int __rcu *llc_to_node_map;
+static void rebuild_llc_node_map(int size);
+#endif
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -856,6 +861,56 @@ DEFINE_STATIC_KEY_FALSE(sched_cache_active);
 /* user wants cache aware scheduling [0 or 1] */
 int sysctl_sched_cache_user = 1;
 
+int llc_to_node(int llc)
+{
+	int node = -1;
+	int *map = NULL;
+
+	rcu_read_lock();
+	map = rcu_dereference(llc_to_node_map);
+	if (map && llc >= 0 && llc <= max_lid)
+		node = map[llc];
+	rcu_read_unlock();
+
+	return node;
+}
+
+static void rebuild_llc_node_map(int size)
+{
+	int *new_map, *old_map;
+	u8 *seen_llc;
+	int cpu, llc;
+
+	new_map = kcalloc(size, sizeof(int), GFP_KERNEL);
+	if (!new_map)
+		return;
+	seen_llc = kcalloc(size, sizeof(*seen_llc), GFP_KERNEL);
+	if (!seen_llc) {
+		kfree(new_map);
+		return;
+	}
+
+	/*
+	 * for_each_possible_cpu() revisits the same LLC non-consecutively
+	 * under SMT (each node's LLCs are walked once per thread), so
+	 * dedup by llc id via seen_llc[], not by comparing against the
+	 * immediately preceding CPU's llc.
+	 */
+	for_each_possible_cpu(cpu) {
+		llc = per_cpu(sd_llc_id, cpu);
+		if (llc < 0 || llc >= size || seen_llc[llc])
+			continue;
+		seen_llc[llc] = 1;
+		new_map[llc] = cpu_to_node(cpu);
+	}
+	kfree(seen_llc);
+
+	old_map = rcu_dereference_protected(llc_to_node_map, true);
+	rcu_assign_pointer(llc_to_node_map, new_map);
+	synchronize_rcu();
+	kfree(old_map);
+}
+
 /*
  * Get the effective LLC size in bytes that @cpu's bottom sched_domain
  * can use. A CPU within a cpuset partition can only use a proportion
@@ -925,6 +980,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 		}
 	}
 
+	rebuild_llc_node_map(max_lid + 1);
 	return true;
 err:
 	for_each_cpu(i, cpu_map) {
-- 
2.34.1

[RFC PATCH v2 02/23] sched/topology: Introduce a NUMA distance matrix with unique distance values

[Jianyong Wu]

Builds a refined node distance matrix based on the raw NUMA distance matrix
provided by BIOS. The refined matrix preserves the relative ordering of
NUMA distances, while assigning distinct distance values to node pairs that
originally shared identical distances within each matrix row. This matrix
is exclusively used for cache-aware scheduling and has no impact on existing
NUMA topology logic such as sched domain construction.

For example, consider a system with 4 NUMA nodes. The raw BIOS-provided
distance matrix may look like this:

        NODE0  NODE1  NODE2  NODE3
NODE0   10     20     20     30
NODE1   20     10     20     25
NODE2   20     20     10     20
NODE3   30     25     20     10

Multiple duplicate distance values exist within each row. After the
deduplication step, the refined distance matrix becomes:

        NODE0  NODE1  NODE2  NODE3
NODE0   10     15     20     30
NODE1   15     10     12     25
NODE2   20     12     10     15
NODE3   30     25     15     10

All entries in each row are now unique, while adhering to two core principles:
1. The relative distance ordering from the original matrix is preserved.
   For instance, original distance(NODE0, NODE1) < distance(NODE0, NODE3),
   and this relative relationship is retained in the refined matrix as well.
2. The matrix remains symmetric across its main diagonal. Maintaining
   symmetry is critical to guarantee consistent pairwise node distances.

Each row of this refined NUMA distance matrix is sorted in ascending order to
generate a unique per-node affinity sequence. This sequence will guide
thread migration logic introduced in subsequent patches.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/topology.h |   1 +
 kernel/sched/topology.c  | 243 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 240 insertions(+), 4 deletions(-)

diff --git a/include/linux/topology.h b/include/linux/topology.h
index 9967739a180c..e01bc9ae6209 100644
--- a/include/linux/topology.h
+++ b/include/linux/topology.h
@@ -179,6 +179,7 @@ static inline int cpu_to_mem(int cpu)
 
 #ifdef CONFIG_SCHED_CACHE
 int llc_to_node(int llc);
+int sched_cache_node_distance(int node0, int node1);
 #endif
 
 #if defined(topology_die_id) && defined(topology_die_cpumask)
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index c6928c6b17b6..4c68dfe86019 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -685,9 +685,22 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+#ifdef CONFIG_NUMA
+/*
+ * The de-duplicated NUMA distance tiers. Defined here, rather than next to
+ * the rest of the NUMA topology state further down, because the cache aware
+ * code below has to reach them as well.
+ */
+static int			sched_numa_node_levels;
+static int			*sched_numa_node_distance;
+#endif
+
 #ifdef CONFIG_SCHED_CACHE
 static int __rcu *llc_to_node_map;
 static void rebuild_llc_node_map(int size);
+static int __rcu *sched_cache_node_dist_matrix;
+static int sched_cache_node_dist_size;
+static void rebuild_node_distance_matrix(int size);
 #endif
 
 static void update_top_cache_domain(int cpu)
@@ -875,6 +888,38 @@ int llc_to_node(int llc)
 	return node;
 }
 
+/*
+ * De-duplicated NUMA-node distance, looked up from
+ * sched_cache_node_dist_matrix[]. Unlike node_distance(), no two nodes
+ * seen from the same node compare equal, while every relative ordering
+ * of the original node_distance() values (e.g. node_distance(a,b) >
+ * node_distance(a,c)) is preserved. Falls back to the raw
+ * node_distance() if the matrix has not been built yet (e.g. very
+ * early boot, before the first rebuild_node_distance_matrix() call).
+ */
+int sched_cache_node_distance(int node0, int node1)
+{
+	int *matrix;
+	int size;
+	int dist = -1;
+
+	if (node0 == node1)
+		return 0;
+
+	rcu_read_lock();
+	matrix = rcu_dereference(sched_cache_node_dist_matrix);
+	size = READ_ONCE(sched_cache_node_dist_size);
+	if (matrix && node0 >= 0 && node0 < size
+		   && node1 >= 0 && node1 < size)
+		dist = matrix[node0 * size + node1];
+	rcu_read_unlock();
+
+	if (dist >= 0)
+		return dist;
+
+	return node_distance(node0, node1);
+}
+
 static void rebuild_llc_node_map(int size)
 {
 	int *new_map, *old_map;
@@ -911,6 +956,197 @@ static void rebuild_llc_node_map(int size)
 	kfree(old_map);
 }
 
+/*
+ * Build a de-duplicated NUMA-node distance matrix using a tiering +
+ * greedy edge-coloring technique, applied directly to node pairs.
+ * Reuses the already sorted, de-duplicated distance tier list
+ * (sched_numa_node_distance[]/sched_numa_node_levels).
+ *
+ * Guarantees, compared to the raw node_distance() table:
+ *   - still symmetric;
+ *   - every relative ordering between two node_distance() values is
+ *     preserved (if node_distance(a,b) > node_distance(a,c) then
+ *     sched_cache_node_distance(a,b) > sched_cache_node_distance(a,c),
+ *     and likewise for equal/less-than);
+ *   - looked at from any single node, the distances to every other
+ *     node are pairwise distinct (this is the property the raw table
+ *     can violate, e.g. two nodes genuinely equidistant from a third).
+ */
+static void rebuild_node_distance_matrix(int size)
+{
+	int *matrix = NULL, *old_matrix;
+	int *tier_dist = NULL;
+	int *tier_ncolors = NULL;
+	int nr_tiers = 0;
+	unsigned long **used_colors = NULL;
+	int i, j, t;
+	int scale;
+
+	if (size <= 0)
+		return;
+
+#ifdef CONFIG_NUMA
+	nr_tiers = READ_ONCE(sched_numa_node_levels);
+	if (nr_tiers > 0) {
+		tier_dist = kmalloc_array(nr_tiers, sizeof(int), GFP_KERNEL);
+		if (tier_dist) {
+			int *d;
+
+			rcu_read_lock();
+			d = rcu_dereference(sched_numa_node_distance);
+			if (d)
+				memcpy(tier_dist, d, nr_tiers * sizeof(int));
+			else
+				nr_tiers = 0;
+			rcu_read_unlock();
+		} else {
+			nr_tiers = 0;
+		}
+
+		if (!nr_tiers) {
+			kfree(tier_dist);
+			tier_dist = NULL;
+		}
+	}
+#endif
+
+	matrix = kcalloc(size * size, sizeof(int), GFP_KERNEL);
+	if (!matrix)
+		goto out;
+
+	if (!tier_dist) {
+		/*
+		 * NUMA distance tiers not available yet (e.g. very early
+		 * boot or a non-NUMA build): fall back to the raw
+		 * node_distance() table, same as sched_cache_node_distance()
+		 * would have returned anyway.
+		 */
+		for (i = 0; i < size; i++) {
+			for (j = 0; j < size; j++) {
+				if (i == j)
+					continue;
+				matrix[i * size + j] = node_distance(i, j);
+			}
+		}
+		goto commit;
+	}
+
+	tier_ncolors = kcalloc(nr_tiers, sizeof(*tier_ncolors), GFP_KERNEL);
+	if (!tier_ncolors)
+		goto out;
+
+	used_colors = kcalloc(size, sizeof(*used_colors), GFP_KERNEL);
+	if (!used_colors)
+		goto out;
+	for (i = 0; i < size; i++) {
+		used_colors[i] = bitmap_zalloc(size, GFP_KERNEL);
+		if (!used_colors[i])
+			goto out;
+	}
+
+	/* Greedy edge-color each tier; pack (tier, color) into matrix[]. */
+	for (t = 0; t < nr_tiers; t++) {
+		int dist = tier_dist[t];
+		int tier_max_color = 0;
+
+		for (i = 0; i < size; i++)
+			bitmap_zero(used_colors[i], size);
+
+		for (i = 0; i < size; i++) {
+			for (j = i + 1; j < size; j++) {
+				int color;
+
+				if (node_distance(i, j) != dist)
+					continue;
+
+				/* smallest color free at both endpoints */
+				color = 0;
+				for (;;) {
+					color = find_next_zero_bit(used_colors[i], size, color);
+					if (WARN_ONCE(color >= size,
+						      "sched_cache: no free color left in NUMA tier (node=%d)",
+						      i)) {
+						color = size - 1;
+						break;
+					}
+					if (!test_bit(color, used_colors[j]))
+						break;
+					color++;
+				}
+
+				set_bit(color, used_colors[i]);
+				set_bit(color, used_colors[j]);
+				if (color + 1 > tier_max_color)
+					tier_max_color = color + 1;
+
+				matrix[i * size + j] = t * (size + 1) + color;
+				matrix[j * size + i] = matrix[i * size + j];
+			}
+		}
+		tier_ncolors[t] = tier_max_color;
+	}
+
+	/*
+	 * Pick the smallest integer scale such that every tier's
+	 * span of increments (0..tier_ncolors[t]-1) still fits strictly
+	 * inside the gap to the next distinct tier once scaled.
+	 */
+	scale = 1;
+	for (;;) {
+		bool need_more = false;
+
+		for (t = 0; t < nr_tiers - 1; t++) {
+			int gap;
+
+			if (tier_ncolors[t] <= 1)
+				continue;
+			gap = tier_dist[t + 1] - tier_dist[t];
+			if (tier_ncolors[t] - 1 >= gap * scale) {
+				need_more = true;
+				break;
+			}
+		}
+		if (!need_more)
+			break;
+		if (++scale > 1000) {
+			scale = 1000;
+			break;
+		}
+	}
+
+	/* Unpack (tier, color) into the final, close-to-original value. */
+	for (i = 0; i < size; i++) {
+		for (j = 0; j < size; j++) {
+			int packed, t2, color;
+
+			if (i == j)
+				continue;
+
+			packed = matrix[i * size + j];
+			t2 = packed / (size + 1);
+			color = packed % (size + 1);
+			matrix[i * size + j] = tier_dist[t2] * scale + color;
+		}
+	}
+
+commit:
+	old_matrix = rcu_dereference_protected(sched_cache_node_dist_matrix, true);
+	sched_cache_node_dist_size = size;
+	rcu_assign_pointer(sched_cache_node_dist_matrix, matrix);
+	synchronize_rcu();
+	kfree(old_matrix);
+	matrix = NULL;
+out:
+	if (used_colors) {
+		for (i = 0; i < size; i++)
+			bitmap_free(used_colors[i]);
+		kfree(used_colors);
+	}
+	kfree(tier_ncolors);
+	kfree(tier_dist);
+	kfree(matrix);
+}
+
 /*
  * Get the effective LLC size in bytes that @cpu's bottom sched_domain
  * can use. A CPU within a cpuset partition can only use a proportion
@@ -981,6 +1217,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 	}
 
 	rebuild_llc_node_map(max_lid + 1);
+	rebuild_node_distance_matrix(nr_node_ids);
 	return true;
 err:
 	for_each_cpu(i, cpu_map) {
@@ -1949,15 +2186,13 @@ enum numa_topology_type sched_numa_topology_type;
 
 /*
  * sched_domains_numa_distance is derived from sched_numa_node_distance
- * and provides a simplified view of NUMA distances used specifically
- * for building NUMA scheduling domains.
+ * (defined near the top of this file) and provides a simplified view of
+ * NUMA distances used specifically for building NUMA scheduling domains.
  */
 static int			sched_domains_numa_levels;
-static int			sched_numa_node_levels;
 
 int				sched_max_numa_distance;
 static int			*sched_domains_numa_distance;
-static int			*sched_numa_node_distance;
 static struct cpumask		***sched_domains_numa_masks;
 #endif /* CONFIG_NUMA */
 
-- 
2.34.1

[RFC PATCH v2 03/23] sched/topology: Introduce a macro to traverse node

[Jianyong Wu]

Add a node traversal macro that leverages the refined node distance matrix
introduced in the prior patch.

For a given target CPU, the macro walks NUMA nodes sorted by ascending
distance entries taken from the matching row of the refined distance matrix.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/sched.h    |  16 ++++++
 kernel/sched/topology.c | 117 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 133 insertions(+)

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf502ba26..88b71e0f5ef0 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4108,6 +4108,22 @@ static inline bool sched_cache_enabled(void)
 
 extern void sched_cache_active_set(void);
 
+extern int sched_node_order_count(int cpu);
+extern int sched_node_order_at(int cpu, int idx);
+
+/*
+ * Walk every NUMA node in the system starting from @cpu's own node,
+ * nearest-first by the de-duplicated sched_cache_node_distance()
+ * matrix (topology.c), until every node has been yielded. @node
+ * receives the node id at each step (index 0 is always @cpu's own
+ * node).
+ */
+#define for_each_sched_node(cpu, node)					\
+	for (int __sn_idx = 0, __sn_nr = sched_node_order_count((cpu)); \
+	     __sn_idx < __sn_nr &&					\
+	     ((node) = sched_node_order_at((cpu), __sn_idx)) >= 0;	\
+	     __sn_idx++)
+
 #endif
 
 void sched_domains_free_llc_id(int cpu);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 4c68dfe86019..166ff4e65a99 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -701,6 +701,7 @@ static void rebuild_llc_node_map(int size);
 static int __rcu *sched_cache_node_dist_matrix;
 static int sched_cache_node_dist_size;
 static void rebuild_node_distance_matrix(int size);
+static void rebuild_node_order(int size);
 #endif
 
 static void update_top_cache_domain(int cpu)
@@ -1147,6 +1148,121 @@ static void rebuild_node_distance_matrix(int size)
 	kfree(matrix);
 }
 
+struct sched_node_order_topology {
+	int nr_node;
+	int *order;
+};
+
+static struct sched_node_order_topology __rcu *sched_node_order_topo;
+
+static void free_node_order_topology(struct sched_node_order_topology *topo)
+{
+	if (!topo)
+		return;
+	kfree(topo->order);
+	kfree(topo);
+}
+
+/*
+ * Generic {distance, id} sort key: candidates are ranked by @dist, with
+ * @id as a deterministic tie-break when two candidates land on the same
+ * @dist.
+ */
+struct dist_key {
+	int dist;
+	int id;
+};
+
+static int dist_key_cmp(const void *a, const void *b)
+{
+	const struct dist_key *ka = a, *kb = b;
+
+	if (ka->dist != kb->dist)
+		return ka->dist - kb->dist;
+	return ka->id - kb->id;
+}
+
+static void rebuild_node_order(int size)
+{
+	struct sched_node_order_topology *topo, *old;
+	struct dist_key *cand;
+	int root, i, k;
+
+	if (size <= 0)
+		return;
+
+	topo = kzalloc_obj(*topo);
+	if (!topo)
+		return;
+
+	topo->nr_node = size;
+	topo->order = kmalloc_array((size_t)size * size, sizeof(int), GFP_KERNEL);
+	cand = kmalloc_array(size, sizeof(*cand), GFP_KERNEL);
+	if (!topo->order || !cand) {
+		kfree(cand);
+		free_node_order_topology(topo);
+		return;
+	}
+
+	for (root = 0; root < size; root++) {
+		for (i = 0; i < size; i++) {
+			cand[i].id = i;
+			cand[i].dist = sched_cache_node_distance(root, i);
+		}
+
+		/* sched_cache_node_distance() guarantees no ties within a row. */
+		sort(cand, size, sizeof(*cand), dist_key_cmp, NULL);
+
+		for (k = 0; k < size; k++)
+			topo->order[root * size + k] = cand[k].id;
+	}
+	kfree(cand);
+
+	old = rcu_dereference_protected(sched_node_order_topo,
+					lockdep_is_held(&sched_domains_mutex));
+	rcu_assign_pointer(sched_node_order_topo, topo);
+	synchronize_rcu();
+	free_node_order_topology(old);
+}
+
+/*
+ * Return the total number of steps in @cpu's for_each_sched_node() walk,
+ * i.e. the number of nodes in the system. 1 if the topology cache isn't
+ * populated yet (e.g. very early boot).
+ */
+int sched_node_order_count(int cpu)
+{
+	struct sched_node_order_topology *topo = rcu_dereference_all(sched_node_order_topo);
+	int root = cpu_to_node(cpu);
+
+	if (root < 0)
+		return 0;
+	if (!topo || root >= topo->nr_node)
+		return 1;
+
+	return topo->nr_node;
+}
+
+/*
+ * Return the node id at position @idx (0-indexed, 0 == @cpu's own node)
+ * in @cpu's node's ascending sched_cache_node_distance() order. -1 if
+ * @idx is out of range, which ends the for_each_sched_node() walk.
+ */
+int sched_node_order_at(int cpu, int idx)
+{
+	struct sched_node_order_topology *topo = rcu_dereference_all(sched_node_order_topo);
+	int root = cpu_to_node(cpu);
+
+	if (root < 0)
+		return -1;
+	if (!topo || root >= topo->nr_node)
+		return idx == 0 ? root : -1;
+	if (idx < 0 || idx >= topo->nr_node)
+		return -1;
+
+	return topo->order[root * topo->nr_node + idx];
+}
+
 /*
  * Get the effective LLC size in bytes that @cpu's bottom sched_domain
  * can use. A CPU within a cpuset partition can only use a proportion
@@ -1218,6 +1334,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 
 	rebuild_llc_node_map(max_lid + 1);
 	rebuild_node_distance_matrix(nr_node_ids);
+	rebuild_node_order(nr_node_ids);
 	return true;
 err:
 	for_each_cpu(i, cpu_map) {
-- 
2.34.1

[RFC PATCH v2 04/23] sched/topology: Introduce a method to calculate the llc distance

[Jianyong Wu]

The previous patch addresses NUMA inter-node distance handling. This
patch shifts focus to the LLC layer and introduces a scheme to compute
pairwise distances between sibling LLCs inside one NUMA node.
The calculation only guarantees distinct distance values for all peers
when viewed from a single reference LLC; it does not enforce globally
unique distances for every arbitrary LLC pair on the node.

One valid calculation scheme is defined by the formula below:

  dist(LLC1, LLC2) = (rank1 + rank2) % k + 1

Constraints: LLC1 and LLC2 must reside on the identical NUMA node.
rank1 = local index of LLC1 within its node; rank2 follows the same rule.
k = total count of LLCs available on the node.

The formula never yields zero, so diagonal self-distance entries are
explicitly hardcoded to 0.

For demonstration, consider a NUMA node with four LLCs (local ranks 0~3):
        LLC0    LLC1    LLC2    LLC3
LLC0    0       2       3       4
LLC1    2       0       4       1
LLC2    3       4       0       2
LLC3    4       1       2       0

Sorting each matrix row in ascending order generates a unique LLC affinity
sequence, which drives thread migration policies implemented in follow-up
patches.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/topology.h |  1 +
 kernel/sched/topology.c  | 99 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 100 insertions(+)

diff --git a/include/linux/topology.h b/include/linux/topology.h
index e01bc9ae6209..fe20ee9dc153 100644
--- a/include/linux/topology.h
+++ b/include/linux/topology.h
@@ -180,6 +180,7 @@ static inline int cpu_to_mem(int cpu)
 #ifdef CONFIG_SCHED_CACHE
 int llc_to_node(int llc);
 int sched_cache_node_distance(int node0, int node1);
+int llc_intra_node_distance(int llc1, int llc2, int node);
 #endif
 
 #if defined(topology_die_id) && defined(topology_die_cpumask)
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 166ff4e65a99..a47ed44f7b1f 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -702,6 +702,21 @@ static int __rcu *sched_cache_node_dist_matrix;
 static int sched_cache_node_dist_size;
 static void rebuild_node_distance_matrix(int size);
 static void rebuild_node_order(int size);
+
+/*
+ * Per-LLC rank within its own node (0-based) plus each node's LLC
+ * count, used by llc_intra_node_distance() to derive a same-node LLC
+ * distance.
+ */
+struct llc_local_rank_topology {
+	int nr_llc;
+	int nr_node;
+	int *rank;		/* [nr_llc]: LLC's 0-based rank within its node */
+	int *node_count;	/* [nr_node]: number of LLCs in the node */
+};
+
+static struct llc_local_rank_topology __rcu *llc_local_rank_topo;
+static void rebuild_llc_local_rank(int size);
 #endif
 
 static void update_top_cache_domain(int cpu)
@@ -921,6 +936,39 @@ int sched_cache_node_distance(int node0, int node1)
 	return node_distance(node0, node1);
 }
 
+/*
+ * LLC distance restricted to same-node pairs, derived from each LLC's
+ * rank within its node (llc_local_rank_topo)
+ * dist = (rank1 + rank2) % k + 1, where k is the LLC count of
+ * the shared node. Except that @llc1 equals @llc2, the distance is 0.
+ * This formuler ensure that there is no identical distance between a
+ * certain llc with any of its sibling llcs in the same node.
+ *
+ * NOTE: It's the caller's responsibility to check if llc1 and llc2 are
+ * in the same node.
+ */
+int llc_intra_node_distance(int llc1, int llc2, int node)
+{
+	struct llc_local_rank_topology *topo;
+	int rank1, rank2, k;
+
+	if (llc1 == llc2)
+		return 0;
+
+	rcu_read_lock();
+	topo = rcu_dereference(llc_local_rank_topo);
+	if (!topo || llc1 >= topo->nr_llc || llc2 >= topo->nr_llc || node < 0) {
+		rcu_read_unlock();
+		return -1;
+	}
+	rank1 = topo->rank[llc1];
+	rank2 = topo->rank[llc2];
+	k = topo->node_count[node];
+	rcu_read_unlock();
+
+	return (rank1 + rank2) % k + 1;
+}
+
 static void rebuild_llc_node_map(int size)
 {
 	int *new_map, *old_map;
@@ -1263,6 +1311,56 @@ int sched_node_order_at(int cpu, int idx)
 	return topo->order[root * topo->nr_node + idx];
 }
 
+static void free_llc_local_rank_topology(struct llc_local_rank_topology *topo)
+{
+	if (!topo)
+		return;
+
+	kfree(topo->rank);
+	kfree(topo->node_count);
+	kfree(topo);
+}
+
+/*
+ * Compute each LLC's rank within its own node (ascending llc id) and
+ * each node's LLC count, consumed by llc_intra_node_distance() to
+ * derive a same-node LLC distance on the fly.
+ */
+static void rebuild_llc_local_rank(int size)
+{
+	struct llc_local_rank_topology *topo, *old;
+	int llc, node;
+
+	if (size <= 0)
+		return;
+
+	topo = kzalloc_obj(*topo);
+	if (!topo)
+		return;
+
+	topo->nr_llc = size;
+	topo->nr_node = nr_node_ids;
+	topo->rank = kcalloc(size, sizeof(int), GFP_KERNEL);
+	topo->node_count = kcalloc(nr_node_ids, sizeof(int), GFP_KERNEL);
+	if (!topo->rank || !topo->node_count) {
+		free_llc_local_rank_topology(topo);
+		return;
+	}
+
+	for (llc = 0; llc < size; llc++) {
+		node = llc_to_node(llc);
+		if (node < 0 || node >= nr_node_ids)
+			continue;
+		topo->rank[llc] = topo->node_count[node]++;
+	}
+
+	old = rcu_dereference_protected(llc_local_rank_topo,
+					 lockdep_is_held(&sched_domains_mutex));
+	rcu_assign_pointer(llc_local_rank_topo, topo);
+	synchronize_rcu();
+	free_llc_local_rank_topology(old);
+}
+
 /*
  * Get the effective LLC size in bytes that @cpu's bottom sched_domain
  * can use. A CPU within a cpuset partition can only use a proportion
@@ -1335,6 +1433,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 	rebuild_llc_node_map(max_lid + 1);
 	rebuild_node_distance_matrix(nr_node_ids);
 	rebuild_node_order(nr_node_ids);
+	rebuild_llc_local_rank(max_lid + 1);
 	return true;
 err:
 	for_each_cpu(i, cpu_map) {
-- 
2.34.1

[RFC PATCH v2 05/23] sched/topology: Introduce a macro to traverse LLC inside node

[Jianyong Wu]

Introduce a macro to traverse LLC only inside a node. Give a node id, this
macro can return the LLC cpumask in the order based on the LLC distance
construct in the previous patch.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/sched.h    |  14 +++
 kernel/sched/topology.c | 214 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 228 insertions(+)

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 88b71e0f5ef0..38453a78c885 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4124,6 +4124,20 @@ extern int sched_node_order_at(int cpu, int idx);
 	     ((node) = sched_node_order_at((cpu), __sn_idx)) >= 0;	\
 	     __sn_idx++)
 
+extern int llc_node_count(int node);
+extern const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out);
+
+/*
+ * Walk each individual LLC belonging to NUMA node @node, nearest-first
+ * by the same-node-only llc_intra_node_distance() ordering instead of
+ * the plain ascending llc_id order of for_each_llc_in_node().
+ */
+#define for_each_llc_node_span(node, span)					\
+	for (int __lns_idx = 0, __lns_nr = llc_node_count((node));		\
+	     __lns_idx < __lns_nr &&						\
+	     ((span) = llc_node_span_by_dist((node), __lns_idx, NULL)) != NULL; \
+	     __lns_idx++)
+
 #endif
 
 void sched_domains_free_llc_id(int cpu);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index a47ed44f7b1f..eaaca6c9a805 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -6,6 +6,7 @@
 #include <linux/sched/isolation.h>
 #include <linux/sched/clock.h>
 #include <linux/bsearch.h>
+#include <linux/sort.h>
 #include "sched.h"
 
 DEFINE_MUTEX(sched_domains_mutex);
@@ -698,6 +699,7 @@ static int			*sched_numa_node_distance;
 #ifdef CONFIG_SCHED_CACHE
 static int __rcu *llc_to_node_map;
 static void rebuild_llc_node_map(int size);
+static void rebuild_llc_aggr_windows(int size);
 static int __rcu *sched_cache_node_dist_matrix;
 static int sched_cache_node_dist_size;
 static void rebuild_node_distance_matrix(int size);
@@ -969,6 +971,24 @@ int llc_intra_node_distance(int llc1, int llc2, int node)
 	return (rank1 + rank2) % k + 1;
 }
 
+/*
+ * Number of LLCs in @node, sourced from llc_local_rank_topo
+ * (rebuild_llc_local_rank()).
+ */
+static int llc_local_rank_node_count(int node)
+{
+	struct llc_local_rank_topology *topo;
+	int count;
+
+	rcu_read_lock();
+	topo = rcu_dereference(llc_local_rank_topo);
+	count = (topo && node >= 0 && node < topo->nr_node) ?
+		topo->node_count[node] : 0;
+	rcu_read_unlock();
+
+	return count;
+}
+
 static void rebuild_llc_node_map(int size)
 {
 	int *new_map, *old_map;
@@ -1434,6 +1454,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 	rebuild_node_distance_matrix(nr_node_ids);
 	rebuild_node_order(nr_node_ids);
 	rebuild_llc_local_rank(max_lid + 1);
+	rebuild_llc_aggr_windows(max_lid + 1);
 	return true;
 err:
 	for_each_cpu(i, cpu_map) {
@@ -2585,6 +2606,199 @@ const struct cpumask *tl_mc_mask(struct sched_domain_topology_level *tl, int cpu
 
 #define llc_mask(cpu) arch_llc_mask(cpu)
 
+#ifdef CONFIG_SCHED_CACHE
+/*
+ * For each node we remember where its slice of the flattened per-node
+ * LLC-id list (node_llc[]) starts (used by the for_each_llc_in_node()
+ * iterator; per-node LLC counts come from llc_local_rank_topo, see
+ * llc_local_rank_node_count()). We also cache each LLC's own cpumask
+ * (llc_span[]).
+ */
+struct llc_aggr_topology {
+	int nr_llc;
+	int nr_node;
+	int *node_offset;		/* [nr_node]: node's slice start in node_llc[] */
+	int *node_llc;			/* [nr_llc]: flattened per-node LLC ids, ascending */
+	int *node_llc_by_dist;		/* [nr_llc]: same slices as node_llc[], sorted by
+					 * llc_intra_node_distance() from each node's
+					 * lowest-id LLC instead of by id
+					 */
+	struct cpumask **llc_span;	/* [nr_llc]: cpumask of each LLC */
+};
+
+static struct llc_aggr_topology __rcu *llc_aggr_topo;
+
+static void free_llc_aggr_topology(struct llc_aggr_topology *topo)
+{
+	int i;
+
+	if (!topo)
+		return;
+
+	if (topo->llc_span) {
+		for (i = 0; i < topo->nr_llc; i++)
+			kfree(topo->llc_span[i]);
+		kfree(topo->llc_span);
+	}
+	kfree(topo->node_offset);
+	kfree(topo->node_llc);
+	kfree(topo->node_llc_by_dist);
+	kfree(topo);
+}
+
+static void rebuild_llc_aggr_windows(int size)
+{
+	struct llc_aggr_topology *topo, *old;
+	int *seen;
+	u8 *seen_llc;
+	int cpu, llc, node, i;
+
+	if (size <= 0)
+		return;
+
+	topo = kzalloc_obj(*topo);
+	if (!topo)
+		return;
+
+	topo->nr_llc  = size;
+	topo->nr_node = nr_node_ids;
+	topo->node_offset = kcalloc(nr_node_ids, sizeof(int), GFP_KERNEL);
+	topo->node_llc    = kcalloc(size, sizeof(int), GFP_KERNEL);
+	topo->node_llc_by_dist = kcalloc(size, sizeof(int), GFP_KERNEL);
+	topo->llc_span    = kcalloc(size, sizeof(struct cpumask *), GFP_KERNEL);
+	seen = kcalloc(nr_node_ids, sizeof(int), GFP_KERNEL);
+	seen_llc = kcalloc(size, sizeof(*seen_llc), GFP_KERNEL);
+	if (!topo->node_offset ||
+	    !topo->node_llc || !topo->node_llc_by_dist ||
+	    !topo->llc_span || !seen || !seen_llc) {
+		kfree(seen);
+		kfree(seen_llc);
+		free_llc_aggr_topology(topo);
+		return;
+	}
+
+	/* Cache each LLC's cpumask. */
+	for_each_possible_cpu(cpu) {
+		llc = per_cpu(sd_llc_id, cpu);
+		if (llc < 0 || llc >= size || seen_llc[llc])
+			continue;
+		seen_llc[llc] = 1;
+
+		topo->llc_span[llc] = kzalloc(cpumask_size(), GFP_KERNEL);
+		if (!topo->llc_span[llc]) {
+			kfree(seen);
+			kfree(seen_llc);
+			free_llc_aggr_topology(topo);
+			return;
+		}
+		cpumask_copy(topo->llc_span[llc], llc_mask(cpu));
+	}
+
+	for (i = 1; i < nr_node_ids; i++)
+		topo->node_offset[i] = topo->node_offset[i - 1] +
+					llc_local_rank_node_count(i - 1);
+
+	/* Fill node_llc[] (ascending llc_id per node). */
+	memset(seen_llc, 0, size * sizeof(*seen_llc));
+	for_each_possible_cpu(cpu) {
+		llc = per_cpu(sd_llc_id, cpu);
+		if (llc < 0 || llc >= size || seen_llc[llc])
+			continue;
+		seen_llc[llc] = 1;
+
+		node = llc_to_node(llc);
+		if (node < 0 || node >= nr_node_ids)
+			continue;
+
+		topo->node_llc[topo->node_offset[node] + seen[node]] = llc;
+		seen[node]++;
+	}
+	kfree(seen);
+	kfree(seen_llc);
+
+	/*
+	 * fill node_llc_by_dist[], the same per-node slices as
+	 * node_llc[] above but sorted ascending by
+	 * llc_intra_node_distance() from that node's lowest-id LLC (the
+	 * same one node_llc[]'s slice starts with) instead of by id. Used
+	 * by for_each_llc_node_span() to walk a node's own LLCs
+	 * nearest-first, unlike the plain id order of for_each_llc_in_node().
+	 * llc_intra_node_distance() guarantees no two same-node LLCs tie
+	 * from a fixed anchor's point of view, so this sort is unambiguous.
+	 */
+	for (node = 0; node < nr_node_ids; node++) {
+		struct dist_key *cand;
+		int base = topo->node_offset[node];
+		int cnt = llc_local_rank_node_count(node);
+		int anchor, k;
+
+		if (cnt == 0)
+			continue;
+
+		anchor = topo->node_llc[base];
+
+		cand = kmalloc_array(cnt, sizeof(*cand), GFP_KERNEL);
+		if (!cand) {
+			free_llc_aggr_topology(topo);
+			return;
+		}
+
+		for (k = 0; k < cnt; k++) {
+			int llc_id = topo->node_llc[base + k];
+
+			cand[k].id = llc_id;
+			cand[k].dist = llc_intra_node_distance(anchor, llc_id, node);
+		}
+
+		sort(cand, cnt, sizeof(*cand), dist_key_cmp, NULL);
+
+		for (k = 0; k < cnt; k++)
+			topo->node_llc_by_dist[base + k] = cand[k].id;
+		kfree(cand);
+	}
+
+
+	old = rcu_dereference_protected(llc_aggr_topo,
+					lockdep_is_held(&sched_domains_mutex));
+	rcu_assign_pointer(llc_aggr_topo, topo);
+	synchronize_rcu();
+	free_llc_aggr_topology(old);
+}
+
+/* Return the number of LLCs belonging to NUMA node @node. */
+int llc_node_count(int node)
+{
+	struct llc_aggr_topology *topo = rcu_dereference_all(llc_aggr_topo);
+
+	if (!topo || node < 0 || node >= topo->nr_node)
+		return 0;
+
+	return llc_local_rank_node_count(node);
+}
+
+/*
+ * Return a pointer to the cached cpumask of the LLC at position @index
+ * within NUMA node @node, ordered ascending by
+ * llc_intra_node_distance() from that node's lowest-id LLC.
+ */
+const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out)
+{
+	struct llc_aggr_topology *topo = rcu_dereference_all(llc_aggr_topo);
+	int llc;
+
+	if (!topo || node < 0 || node >= topo->nr_node)
+		return NULL;
+	if (index < 0 || index >= llc_local_rank_node_count(node))
+		return NULL;
+
+	llc = topo->node_llc_by_dist[topo->node_offset[node] + index];
+	if (llc_out)
+		*llc_out = llc;
+
+	return topo->llc_span[llc];
+}
+#endif /* CONFIG_SCHED_CACHE */
+
 const struct cpumask *tl_pkg_mask(struct sched_domain_topology_level *tl, int cpu)
 {
 	return cpu_node_mask(cpu);
-- 
2.34.1

[RFC PATCH v2 06/23] sched/topology: Add sd_node for the NODE sched domain

[Jianyong Wu]

There is no straightforward way to retrieve the NODE-level sched domain,
which will be used in subsequent patches. To simplify this, add a per-CPU
variable `sd_node` for convenience.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/sched.h    |  1 +
 kernel/sched/topology.c | 17 +++++++++++++++++
 2 files changed, 18 insertions(+)

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 38453a78c885..3d3f1d40a672 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2197,6 +2197,7 @@ DECLARE_PER_CPU(int, sd_llc_id);
 DECLARE_PER_CPU(int, sd_share_id);
 DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared);
 DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared);
+DECLARE_PER_CPU(struct sched_domain __rcu *, sd_node);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index eaaca6c9a805..46ae1f94e9f5 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -679,6 +679,7 @@ DEFINE_PER_CPU(int, sd_llc_id) = -1;
 DEFINE_PER_CPU(int, sd_share_id);
 DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared);
 DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared);
+DEFINE_PER_CPU(struct sched_domain __rcu *, sd_node);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
@@ -721,6 +722,19 @@ static struct llc_local_rank_topology __rcu *llc_local_rank_topo;
 static void rebuild_llc_local_rank(int size);
 #endif
 
+/* Get sched domain by its name */
+static struct sched_domain *get_sched_domain_by_name(int cpu, char *name)
+{
+	struct sched_domain *sd = NULL;
+
+	for_each_domain(cpu, sd) {
+		if (!strcmp(sd->name, name))
+			break;
+	}
+
+	return sd;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -753,6 +767,9 @@ static void update_top_cache_domain(int cpu)
 	 */
 	per_cpu(sd_share_id, cpu) = id;
 
+	sd = get_sched_domain_by_name(cpu, "NODE");
+	rcu_assign_pointer(per_cpu(sd_node, cpu), sd);
+
 	sd = lowest_flag_domain(cpu, SD_NUMA);
 	rcu_assign_pointer(per_cpu(sd_numa, cpu), sd);
 
-- 
2.34.1

[RFC PATCH v2 07/23] sched/cache: Prioritize preferred NUMA node selection over LLC selection

[Jianyong Wu]

In the current implementation, the preferred LLC is selected based on the
LLC with the largest running time of the thread group. However, the
preferred LLC may be prone to frequent migration when the workload
spreads across the entire system, especially when the number of CPUs
sharing an LLC is small. A better approach is to first select a preferred
NUMA node in the same way, and then select the preferred LLC within that
preferred NUMA node.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 131 +++++++++++++++++++++++++++-----------------
 1 file changed, 82 insertions(+), 49 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f15f5764818e..30de09ebd9f6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1826,12 +1826,13 @@ static inline void update_avg_scale(u64 *avg, u64 sample)
 
 static void task_cache_work(struct callback_head *work)
 {
-	int cpu, m_a_cpu = -1, nr_running = 0, curr_cpu;
 	unsigned long next_scan, now = jiffies;
 	struct task_struct *p = current, *cur;
-	unsigned long curr_m_a_occ = 0;
 	struct mm_struct *mm = p->mm;
-	unsigned long m_a_occ = 0;
+	unsigned long m_a_n_occ = 0, curr_m_a_n_occ = 0, curr_m_a_occ = 0;
+	unsigned long pref_llc_occ = 0;
+	int cpu, m_a_n_cpu = -1, nr_running = 0, curr_cpu;
+	int pref_cpu, pref_llc_cpu = -1, new_cpu = -1;
 	cpumask_var_t cpus;
 
 	WARN_ON_ONCE(work != &p->cache_work);
@@ -1863,6 +1864,13 @@ static void task_cache_work(struct callback_head *work)
 	if (!zalloc_cpumask_var(&cpus, GFP_KERNEL))
 		return;
 
+	/*
+	 * Sample the current preference once, up front. Re-reading it during
+	 * the scan would let a choice made while walking one node feed into
+	 * the comparison made for the next one.
+	 */
+	pref_cpu = READ_ONCE(mm->sc_stat.cpu);
+
 	scoped_guard (cpus_read_lock) {
 		guard(rcu)();
 
@@ -1870,68 +1878,93 @@ static void task_cache_work(struct callback_head *work)
 
 		for_each_cpu(cpu, cpus) {
 			/* XXX sched_cluster_active */
-			struct sched_domain *sd = rcu_dereference_all(per_cpu(sd_llc, cpu));
-			unsigned long occ, m_occ = 0, a_occ = 0;
-			int m_cpu = -1, i;
+			struct sched_domain *nsd = per_cpu(sd_node, cpu);
+			unsigned long occ, m_occ, a_occ, a_n_occ = 0;
+			unsigned long m_llc_occ = 0;
+			int m_llc_cpu = -1, m_cpu, i, k;
 
-			if (!sd)
+			if (!nsd)
 				continue;
 
-			for_each_cpu(i, sched_domain_span(sd)) {
-				occ = fraction_mm_sched(cpu_rq(i),
+			for_each_cpu_and(k, sched_domain_span(nsd), cpus) {
+				a_occ = m_occ = 0;
+				m_cpu = -1;
+				/* XXX sched_cluster_active */
+				struct sched_domain *sd =
+					rcu_dereference_all(per_cpu(sd_llc, k));
+
+				if (!sd)
+					continue;
+
+				for_each_cpu(i, sched_domain_span(sd)) {
+					occ = fraction_mm_sched(cpu_rq(i),
 							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
-				a_occ += occ;
-				if (occ > m_occ) {
-					m_occ = occ;
-					m_cpu = i;
+					a_occ += occ;
+					if (occ > m_occ) {
+						m_occ = occ;
+						m_cpu = i;
+					}
+
+					cur = rcu_dereference_all(cpu_rq(i)->curr);
+					if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
+					    cur->mm == mm)
+						nr_running++;
 				}
 
-				cur = rcu_dereference_all(cpu_rq(i)->curr);
-				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
-				    cur->mm == mm)
-					nr_running++;
+				cpumask_andnot(cpus, cpus, sched_domain_span(sd));
+				if (a_occ > m_llc_occ) {
+					m_llc_occ = a_occ;
+					m_llc_cpu = m_cpu;
+				}
+				if (pref_cpu >= 0 && llc_id(pref_cpu) == llc_id(k))
+					curr_m_a_occ = a_occ;
+				/* record for numa node */
+				a_n_occ += a_occ;
 			}
 
 			/*
-			 * Compare the accumulated occupancy of each LLC. The
-			 * reason for using accumulated occupancy rather than average
-			 * per CPU occupancy is that it works better in asymmetric LLC
-			 * scenarios.
-			 * For example, if there are 2 threads in a 4CPU LLC and 3
-			 * threads in an 8CPU LLC, it might be better to choose the one
-			 * with 3 threads. However, this would not be the case if the
-			 * occupancy is divided by the number of CPUs in an LLC (i.e.,
-			 * if average per CPU occupancy is used).
-			 * Besides, NUMA balancing fault statistics behave similarly:
-			 * the total number of faults per node is compared rather than
-			 * the average number of faults per CPU. This strategy is also
-			 * followed here.
+			 * Remember the busiest LLC of the node the preference
+			 * currently points at, so that the LLC level decision can
+			 * be taken once, after the whole scan.
 			 */
-			if (a_occ > m_a_occ) {
-				m_a_occ = a_occ;
-				m_a_cpu = m_cpu;
+			if (pref_cpu >= 0 && m_llc_cpu >= 0 &&
+			    cpu_to_node(pref_cpu) == cpu_to_node(m_llc_cpu)) {
+				pref_llc_occ = m_llc_occ;
+				pref_llc_cpu = m_llc_cpu;
 			}
 
-			if (llc_id(cpu) == llc_id(READ_ONCE(mm->sc_stat.cpu)))
-				curr_m_a_occ = a_occ;
+			if (a_n_occ > m_a_n_occ) {
+				m_a_n_occ = a_n_occ;
+				m_a_n_cpu = m_llc_cpu;
+			}
 
-			cpumask_andnot(cpus, cpus, sched_domain_span(sd));
+			if (pref_cpu >= 0 &&
+			    cpu_to_node(cpu) == cpu_to_node(pref_cpu))
+				curr_m_a_n_occ = a_n_occ;
 		}
 	}
 
-	if (m_a_occ > (2 * curr_m_a_occ)) {
-		/*
-		 * Avoid switching sc_stat.cpu too fast.
-		 * The reason to choose 2X is because:
-		 * 1. It is better to keep the preferred LLC stable,
-		 *    rather than changing it frequently and cause migrations
-		 * 2. 2X means the new preferred LLC has at least 1 more
-		 *    busy CPU than the old one(200% vs 100%, eg)
-		 * 3. 2X is chosen based on test results, as it delivers
-		 *    the optimal performance gain so far.
-		 */
-		WRITE_ONCE(mm->sc_stat.cpu, m_a_cpu);
-	}
+	/*
+	 * Avoid switching sc_stat.cpu too fast. The reason to choose 2X is
+	 * because:
+	 * 1. It is better to keep the preferred LLC stable, rather than
+	 *    changing it frequently and cause migrations
+	 * 2. 2X means the new preferred LLC has at least 1 more busy CPU than
+	 *    the old one(200% vs 100%, eg)
+	 * 3. 2X is chosen based on test results, as it delivers the optimal
+	 *    performance gain so far.
+	 *
+	 * Moving to another node takes precedence over moving inside the
+	 * current one, as it did when the two updates were applied in that
+	 * order.
+	 */
+	if (m_a_n_occ > 2 * curr_m_a_n_occ)
+		new_cpu = m_a_n_cpu;
+	else if (pref_llc_cpu >= 0 && pref_llc_occ > 2 * curr_m_a_occ)
+		new_cpu = pref_llc_cpu;
+
+	if (new_cpu >= 0)
+		WRITE_ONCE(mm->sc_stat.cpu, new_cpu);
 
 	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
 	free_cpumask_var(cpus);
-- 
2.34.1

[RFC PATCH v2 08/23] sched/topology: Introduce a per-CPU tasks NUMA preferred counter

[Jianyong Wu]

Like the existed task LLC preferred counter, sd->llc_counts, introduce
sd->numa_counts to denotes the task number that prefer each NUMA node in
a certain rq.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/sched/topology.h |  1 +
 kernel/sched/topology.c        | 33 +++++++++++++++++++++++++++------
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h
index b5d9d7c2b8ad..b31d2cf16592 100644
--- a/include/linux/sched/topology.h
+++ b/include/linux/sched/topology.h
@@ -121,6 +121,7 @@ struct sched_domain {
 	unsigned int llc_max;
 	unsigned int *llc_counts __counted_by_ptr(llc_max);
 	unsigned long llc_bytes;
+	unsigned int *numa_counts;
 #endif
 
 #ifdef CONFIG_SCHEDSTATS
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 46ae1f94e9f5..901e593e70d7 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -641,8 +641,9 @@ static void destroy_sched_domain(struct sched_domain *sd)
 	free_sched_domain_shared(sd->shared);
 
 #ifdef CONFIG_SCHED_CACHE
-	/* only the bottom sd has llc_counts array */
+	/* only the bottom sd has llc_counts/numa_counts array */
 	kfree(sd->llc_counts);
+	kfree(sd->numa_counts);
 #endif
 	kfree(sd);
 }
@@ -853,10 +854,12 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
 			sd->llc_counts = tmp->llc_counts;
 			sd->llc_max = tmp->llc_max;
 			sd->llc_bytes = tmp->llc_bytes;
+			sd->numa_counts = tmp->numa_counts;
 			/* make sure destroy_sched_domain() does not free it */
 			tmp->llc_counts = NULL;
 			tmp->llc_max = 0;
 			tmp->llc_bytes = 0;
+			tmp->numa_counts = NULL;
 #endif
 			/*
 			 * sched groups hold the flags of the child sched
@@ -1434,7 +1437,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 			 struct s_data *d)
 {
 	struct sched_domain *sd, *top_llc, *parent;
-	unsigned int *p;
+	unsigned int *p_llc, *p_node;
 	int i;
 
 	for_each_cpu(i, cpu_map) {
@@ -1442,9 +1445,13 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 		if (!sd)
 			goto err;
 
-		p = kcalloc_node(max_lid + 1, sizeof(unsigned int),
+		p_llc = kcalloc_node(max_lid + 1, sizeof(unsigned int),
 				 GFP_KERNEL, cpu_to_node(i));
-		if (!p)
+
+		p_node = kcalloc_node(nr_node_ids, sizeof(unsigned int),
+				 GFP_KERNEL, cpu_to_node(i));
+
+		if (!p_llc || !p_node)
 			goto err;
 
 		top_llc = sd;
@@ -1459,12 +1466,25 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 
 		if (top_llc->flags & SD_SHARE_LLC) {
 			sd->llc_max = max_lid + 1;
-			sd->llc_counts = p;
+			sd->llc_counts = p_llc;
 			sd->llc_bytes = get_effective_llc_bytes(i, top_llc);
 		} else {
 			/* avoid memory leak */
-			kfree(p);
+			kfree(p_llc);
+		}
+
+		parent = top_llc;
+		/* Like above, find the lowest SD_NUMA domain */
+		for (parent = rcu_dereference_protected(top_llc->parent, true);
+		     parent; parent = rcu_dereference_protected(parent->parent, true)) {
+			if (parent->flags & SD_NUMA)
+				break;
 		}
+
+		if (parent)
+			sd->numa_counts = p_node;
+		else
+			kfree(p_node);
 	}
 
 	rebuild_llc_node_map(max_lid + 1);
@@ -1481,6 +1501,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 			sd->llc_counts = NULL;
 			sd->llc_max = 0;
 			sd->llc_bytes = 0;
+			sd->numa_counts = NULL;
 		}
 	}
 
-- 
2.34.1

[RFC PATCH v2 09/23] sched/cache: Account percpu sd task NUMA preference

[Jianyong Wu]

Account for per-NUMA task count preferences within per-CPU sched domains,
mirroring the existing implementation of LLC task preference logic.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 30de09ebd9f6..dc7bbdb1ab98 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1529,8 +1529,13 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p)
 	p->pref_llc_queued = pref_llc_queued;
 
 	sd = rcu_dereference_all(rq->sd);
-	if (sd && (unsigned int)pref_llc < sd->llc_max)
+	if (sd && (unsigned int)pref_llc < sd->llc_max) {
+		int pref_numa = llc_to_node(pref_llc);
 		sd->llc_counts[pref_llc]++;
+		if (sd->numa_counts &&
+		    (unsigned int)pref_numa < nr_node_ids)
+			sd->numa_counts[pref_numa]++;
+	}
 }
 
 static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
@@ -1567,8 +1572,14 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
 		 * This undercount is temporary and accurate accounting
 		 * will resume once the rq has a chance to be idle.
 		 */
-		if (sd->llc_counts[pref_llc])
+		if (sd->llc_counts[pref_llc]) {
+			int pref_numa = llc_to_node(pref_llc);
 			sd->llc_counts[pref_llc]--;
+			if (sd->numa_counts &&
+			    (unsigned int)pref_numa < nr_node_ids &&
+			    sd->numa_counts[pref_numa])
+				sd->numa_counts[pref_numa]--;
+		}
 	}
 }
 
@@ -1727,11 +1738,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	mm_sched_llc = get_pref_llc(p, mm);
 
 	/* task not on rq accounted later in account_entity_enqueue() */
-	if (task_running_on_cpu(rq->cpu, p) &&
-	    READ_ONCE(p->preferred_llc) != mm_sched_llc) {
-		account_llc_dequeue(rq, p);
-		WRITE_ONCE(p->preferred_llc, mm_sched_llc);
-		account_llc_enqueue(rq, p);
+	if (task_running_on_cpu(rq->cpu, p)) {
+		if (READ_ONCE(p->preferred_llc) != mm_sched_llc) {
+			account_llc_dequeue(rq, p);
+			WRITE_ONCE(p->preferred_llc, mm_sched_llc);
+			account_llc_enqueue(rq, p);
+		}
 	}
 }
 
-- 
2.34.1

[RFC PATCH v2 10/23] sched/topology: Add per-sd scratch for the load balance affinity score

[Jianyong Wu]

The affinity score computed during load balancing needs two small arrays,
one entry per LLC: the LLCs that sit closer to the balance destination than
to the source, and the weight of each. Their size is only known once the
sched domains have been built, which is why the load balancer currently has
to allocate them on every call.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/sched/topology.h |  6 ++++++
 kernel/sched/topology.c        | 28 +++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h
index b31d2cf16592..569ff562dbdb 100644
--- a/include/linux/sched/topology.h
+++ b/include/linux/sched/topology.h
@@ -122,6 +122,12 @@ struct sched_domain {
 	unsigned int *llc_counts __counted_by_ptr(llc_max);
 	unsigned long llc_bytes;
 	unsigned int *numa_counts;
+	/*
+	 * Scratch for the load balance affinity score, llc_max entries each.
+	 * Only the bottom sd owns them, like llc_counts above.
+	 */
+	int *affi_ids;
+	int *affi_weights;
 #endif
 
 #ifdef CONFIG_SCHEDSTATS
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 901e593e70d7..73d0c44dea92 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -644,6 +644,8 @@ static void destroy_sched_domain(struct sched_domain *sd)
 	/* only the bottom sd has llc_counts/numa_counts array */
 	kfree(sd->llc_counts);
 	kfree(sd->numa_counts);
+	kfree(sd->affi_ids);
+	kfree(sd->affi_weights);
 #endif
 	kfree(sd);
 }
@@ -855,11 +857,15 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
 			sd->llc_max = tmp->llc_max;
 			sd->llc_bytes = tmp->llc_bytes;
 			sd->numa_counts = tmp->numa_counts;
+			sd->affi_ids = tmp->affi_ids;
+			sd->affi_weights = tmp->affi_weights;
 			/* make sure destroy_sched_domain() does not free it */
 			tmp->llc_counts = NULL;
 			tmp->llc_max = 0;
 			tmp->llc_bytes = 0;
 			tmp->numa_counts = NULL;
+			tmp->affi_ids = NULL;
+			tmp->affi_weights = NULL;
 #endif
 			/*
 			 * sched groups hold the flags of the child sched
@@ -1438,6 +1444,7 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 {
 	struct sched_domain *sd, *top_llc, *parent;
 	unsigned int *p_llc, *p_node;
+	int *p_ids, *p_w;
 	int i;
 
 	for_each_cpu(i, cpu_map) {
@@ -1451,8 +1458,22 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 		p_node = kcalloc_node(nr_node_ids, sizeof(unsigned int),
 				 GFP_KERNEL, cpu_to_node(i));
 
-		if (!p_llc || !p_node)
+		/* Scratch for the affinity score, see cal_affinity_score(). */
+		p_ids = kcalloc_node(max_lid + 1, sizeof(int),
+				 GFP_KERNEL, cpu_to_node(i));
+		p_w = kcalloc_node(max_lid + 1, sizeof(int),
+				 GFP_KERNEL, cpu_to_node(i));
+
+		if (!p_llc || !p_node || !p_ids || !p_w) {
+			kfree(p_llc);
+			kfree(p_node);
+			kfree(p_ids);
+			kfree(p_w);
 			goto err;
+		}
+
+		sd->affi_ids = p_ids;
+		sd->affi_weights = p_w;
 
 		top_llc = sd;
 		/*
@@ -1498,10 +1519,15 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
 		sd = *per_cpu_ptr(d->sd, i);
 		if (sd) {
 			kfree(sd->llc_counts);
+			kfree(sd->numa_counts);
+			kfree(sd->affi_ids);
+			kfree(sd->affi_weights);
 			sd->llc_counts = NULL;
 			sd->llc_max = 0;
 			sd->llc_bytes = 0;
 			sd->numa_counts = NULL;
+			sd->affi_ids = NULL;
+			sd->affi_weights = NULL;
 		}
 	}
 
-- 
2.34.1

[RFC PATCH v2 11/23] sched/cache: Introduce helpers for task migration decisions

[Jianyong Wu]

Cache-aware scheduling makes migration decisions purely based on LLC
affinity, allowing moves only toward a task's preferred LLC. This rigid
policy cannot handle workloads that do not fit within a single LLC.
A better approach is on-demand thread aggregation across LLCs: as the
thread count increases, additional LLCs are recruited to host threads,
enabling workload scaling at LLC granularity.

To realize this behaviour, we need to pick the next eligible LLC once
currently selected LLCs become saturated.

Earlier patches have built node-level and LLC-level distance matrices.
From these matrices we obtain a node-affinity sequence, and within each
node an intra-node LLC-affinity sequence. These sequences provide the
priority order used to pick subsequent LLC candidates.

This patch adds a helper routine to determine whether task migration
is permitted. It checks whether the destination CPU resides within the
first eligible LLC. Migration is permitted if this condition holds,
and vice versa.

The first eligible LLC is resolved via a two-stage policy. The first
stage operates at NUMA-node granularity: we iterate over the
node-affinity sequence to find the first node that can accommodate
the task. Once such a node is found, the second stage selects the
first capacity-available LLC within this node by walking the
intra-node LLC-affinity sequence.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 185 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dc7bbdb1ab98..cfbd596992ab 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10610,6 +10610,191 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
 	return mig_llc;
 }
 
+/*
+ * Like get_llc_stats but for sched domain that above LLC level.
+ * Based on get_llc_stats, we can accumulate utilization and cap for
+ * sched domain in the granularity of LLC.
+ */
+static bool get_span_stats(const struct cpumask *span, unsigned long *util_out,
+					unsigned long *cap_out)
+{
+	cpumask_var_t mask;
+	int cpu;
+	unsigned long util_tmp, cap_tmp, util = 0, cap = 0;
+	struct sched_domain *sd_tmp;
+
+	if (!span || !util_out || !cap_out)
+		return false;
+
+	if (!alloc_cpumask_var(&mask, GFP_ATOMIC))
+		return false;
+
+	cpumask_copy(mask, span);
+	for_each_cpu(cpu, mask) {
+		if (!get_llc_stats(cpu, &util_tmp, &cap_tmp)) {
+			free_cpumask_var(mask);
+			return false;
+		}
+
+		sd_tmp = rcu_dereference(per_cpu(sd_llc, cpu));
+		cpumask_andnot(mask, mask, sched_domain_span(sd_tmp));
+		util += util_tmp;
+		cap += cap_tmp;
+	}
+
+	*util_out = util;
+	*cap_out = cap;
+
+	free_cpumask_var(mask);
+	return true;
+}
+
+/*
+ * Decide if migration should happen on a specific node.
+ * The node here is an LLC or a NUMA.
+ */
+static enum llc_mig __maybe_unused can_migrate_node(int src_cpu, int dst_cpu,
+			struct task_struct *p, bool to_pref)
+{
+	const struct cpumask *span;
+	struct mm_struct *mm;
+	unsigned long dst_util, dst_cap, tsk_util = 0;
+	unsigned long src_util = 0, src_cap = 0;
+	unsigned long acc_util = 0, acc_cap = 0;
+	int node, target_cpu = src_cpu;
+	int get_src = 0;
+
+	if (!get_llc_stats(dst_cpu, &dst_util, &dst_cap))
+		return mig_unrestricted;
+
+	if (!get_llc_stats(src_cpu, &src_util, &src_cap))
+		src_cap = 0;
+
+	if (p) {
+		mm = p->mm;
+		if (mm) {
+			if (mm->sc_stat.cpu >= 0)
+				target_cpu = mm->sc_stat.cpu;
+		}
+		tsk_util = task_util(p);
+	}
+
+	dst_util = dst_util + tsk_util;
+
+	if (to_pref) {
+		unsigned long dst_pre = dst_util - tsk_util;
+
+		if (fits_llc_capacity(dst_util, dst_cap))
+			return mig_llc;
+
+		/*
+		 * The destination is over the margin. That is a reason to
+		 * refuse a task while the margin can still be met, but not
+		 * while every LLC of the node is over it: no placement
+		 * satisfies the margin then, and refusing every migration
+		 * leaves the imbalance in place.
+		 *
+		 * Let the task through when the move still lowers the peak,
+		 * that is when the source is noticeably heavier than the
+		 * destination and carries at least two more tasks worth of
+		 * utilization. The second condition keeps the destination
+		 * from becoming the heavier side, which would bounce the
+		 * task straight back.
+		 */
+		if (src_cap && util_greater(src_util, dst_pre) &&
+		    src_util >= dst_pre + 2 * tsk_util)
+			return mig_llc;
+
+		return mig_forbid;
+	}
+
+	for_each_sched_node(target_cpu, node) {
+		unsigned long u = 0, c = 0, nu, nc;
+
+		/*
+		 * The walk starts at the anchor, so the nodes it crosses before
+		 * reaching the source say nothing about this migration: the task
+		 * does not live there and is not going there. Judging them only
+		 * lets an unrelated node with room refuse the move. Start at the
+		 * node the task actually sits on.
+		 */
+		if (!get_src) {
+			if (!cpumask_test_cpu(src_cpu, cpumask_of_node(node)))
+				continue;
+			else
+				get_src = 1;
+		}
+
+		if (cpumask_test_cpu(dst_cpu, cpumask_of_node(node))) {
+			nu = 0;
+			nc = 0;
+			for_each_llc_node_span(node, span) {
+				get_span_stats(span, &u, &c);
+				nu += u;
+				nc += c;
+				if (cpumask_test_cpu(dst_cpu, span)) {
+					if (fits_llc_capacity(u + tsk_util, c))
+						return mig_llc;
+
+					/*
+					 * The destination is over the margin,
+					 * but so may be the source. Refusing
+					 * then leaves the peak where it is:
+					 * a LLC at seven tasks stays at seven
+					 * while a neighbour in the same node
+					 * sits at four, because taking one
+					 * more would put that neighbour over
+					 * the margin as well.
+					 *
+					 * Let the task through when the move
+					 * still lowers the peak, guarded the
+					 * same way as the aggregation path:
+					 * the source must be noticeably
+					 * heavier and carry at least two more
+					 * tasks worth of utilization, so the
+					 * destination cannot end up the
+					 * heavier side and bounce it back.
+					 */
+					if (src_cap &&
+					    util_greater(src_util, u + tsk_util) &&
+					    src_util >= u + 2 * tsk_util)
+						return mig_llc;
+
+					return mig_forbid;
+				/*
+				 * A nearer LLC only justifies vetoing this
+				 * migration if the task would actually fit
+				 * there, so account for its utilization the
+				 * same way the destination branch above does.
+				 * Without it a LLC already holding one task
+				 * per core still reads as having room and
+				 * vetoes every migration towards a farther,
+				 * genuinely idle LLC.
+				 */
+				} else if (fits_llc_capacity(acc_util + nu + tsk_util,
+						acc_cap + nc)
+						&& fits_llc_capacity(u + tsk_util, c)
+						&& !util_greater(u, dst_pre))
+					return mig_forbid;
+			}
+		}
+
+		/* Don't migrate if this is a good place to live. */
+		for_each_llc_node_span(node, span) {
+			get_span_stats(span, &u, &c);
+			if (cpumask_test_cpu(src_cpu, span)) {
+				if (fits_llc_capacity(u, c))
+					return mig_forbid;
+			} else {
+				if (fits_llc_capacity(u + tsk_util, c))
+					return mig_forbid;
+			}
+		}
+	}
+
+	return mig_unrestricted;
+}
+
 /*
  * Check if task p can migrate from source LLC to
  * destination LLC in terms of cache aware load balance.
-- 
2.34.1

[RFC PATCH v2 12/23] sched/cache: Introduce rq affinity gain calculation

[Jianyong Wu]

The cache-aware scheduler's current source runqueue selection logic
only matches a task's preferred LLC against the destination LLC.
This misses many migration opportunities that deliver improved NUMA
affinity even when preferred LLCs do not align.

As an illustrative example: source CPUs reside on NODE0, destination
CPUs on NODE1. A task on the source runqueue has its preferred LLC
located on NODE2. If the NUMA distance NODE0<->NODE2 is 20, and
NODE1<->NODE2 is 15, migrating this task reduces remote memory latency.
The existing policy cannot capture this beneficial case.

To fix this gap, implement a new scoring algorithm to quantify total
affinity promotion for a source runqueue given source and destination
LLCs. The algorithm operates in two distinct phases:

Iterate all system nodes, for the NUMA-level domain load balance, or
intra-node LLCs, for NODE-level domain load balance, and filter those that
yield improved affinity.
Take NUMA-level domain as an example. If tasks bound to NUMAi migrate from
the source CPU to destination CPU. Compute the distance delta Di for each
NUMAi, via:

  Di = node_distance(src_node, NUMAi) - node_distance(dst_node, NUMAi)

Aggregate total affinity promotion score for the candidate runqueue
by summing weighted contributions from all resident tasks. Per-task
weight and total score are calculated as follows:

  W_i = Rt_i * Di
  p = sum_i(W_i)

Here p is the total affinity gain of the runqueue; Rt_i denotes the
count of tasks on the runqueue with NUMAi as their preferred node,
tracked via rq->sd->node_count.

By this way, we can select the right src sched group or the src rq in load
balance.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 169 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 169 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfbd596992ab..505a1ea9537f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11920,6 +11920,7 @@ sched_reduced_capacity(struct rq *rq, struct sched_domain *sd)
 }
 
 #ifdef CONFIG_SCHED_CACHE
+extern int max_lid;
 /*
  * Record the statistics for this scheduler group for later
  * use. These values guide load balancing on aggregating tasks
@@ -11998,6 +11999,174 @@ static bool update_llc_busiest(struct lb_env *env,
 	 */
 	return sgs->nr_pref_dst_llc > busiest->nr_pref_dst_llc;
 }
+
+/*
+ * Get all LLCs that are closer to the destination LLC than to the
+ * source LLC inside a NUMA.
+ */
+static int get_affi_llcs(struct sched_domain *sd, int src_llc, int dst_llc,
+			 int *affi_llcs, int *affi)
+{
+	int j = 0, src_dist, dst_dist, cur_node;
+
+	if (src_llc == dst_llc || sd->flags & (SD_NUMA | SD_SHARE_LLC))
+		return 0;
+
+	cur_node = llc_to_node(src_llc);
+	if (cur_node != llc_to_node(dst_llc))
+		return 0;
+
+	for (int i = 0; i <= max_lid; i++) {
+		if (cur_node == llc_to_node(i)) {
+			src_dist = llc_intra_node_distance(src_llc, i, cur_node);
+			dst_dist = llc_intra_node_distance(dst_llc, i, cur_node);
+			if (src_dist < 0 || dst_dist < 0)
+				continue;
+			if (src_dist > dst_dist) {
+				affi[j] = clamp(src_dist - dst_dist, 1, 1024);
+				affi_llcs[j++] = i;
+			}
+		} else {
+			/*
+			 * Assume the distance between LLCm on nodeA and LLCn on nodeB follows:
+			 * llc_dist(LLCm, LLCn) = node_dist(nodeA, nodeB) + m + n,
+			 * where m and n represent the intra-node index of LLCm and LLCn
+			 * within their respective nodes. Under this definition, we derive:
+			 * llc_dist(src_llc, i) - llc_dist(dst_llc, i) = src_llc - dst_llc
+			 */
+			if (src_llc > dst_llc) {
+				affi[j] = clamp(src_llc - dst_llc, 1, 1024);
+				affi_llcs[j++] = i;
+			}
+		}
+	}
+
+	return j;
+}
+
+static int get_affi_numas(int src_node, int dst_node, int *affi_nodes, int *affi)
+{
+	int j = 0, src_dist, dst_dist, node;
+
+	for_each_node(node) {
+		src_dist = sched_cache_node_distance(src_node, node);
+		dst_dist = sched_cache_node_distance(dst_node, node);
+		if (src_dist < 0 || dst_dist < 0)
+			continue;
+
+		if (src_dist > dst_dist) {
+			affi[j] = clamp(src_dist - dst_dist, 4, 1024);
+			affi_nodes[j++] = node;
+		}
+	}
+
+	return j;
+}
+
+static int calc_affinity_numa_score(struct sched_domain *sd, int src_cpu, int dst_cpu,
+			int *affi_node, int *affi, int *last_node, int *num)
+{
+	int src_node, dst_node, score = 0;
+
+	src_node = cpu_to_node(src_cpu);
+	dst_node = cpu_to_node(dst_cpu);
+	if (src_node != *last_node) {
+		*last_node = src_node;
+		memset(affi_node, 0, (max_lid + 1) * sizeof(int));
+		memset(affi, 0, (max_lid + 1) * sizeof(int));
+		*num = get_affi_numas(src_node, dst_node, affi_node, affi);
+	}
+
+	for (int i = 0; i < *num; i++) {
+		if ((unsigned int)affi_node[i] < nr_node_ids)
+			score += sd->numa_counts[affi_node[i]] * affi[i];
+	}
+
+	return score;
+}
+
+static int calc_affinity_llc_score(struct sched_domain *sd_cur, struct sched_domain *sd,
+			int src_cpu, int dst_cpu, int *affi_llc,
+			int *affi, int *last_llc, int *num)
+{
+	int src_llc, dst_llc, score = 0;
+
+	src_llc = llc_id(src_cpu);
+	dst_llc = llc_id(dst_cpu);
+
+	if (src_llc != *last_llc) {
+		*last_llc = src_llc;
+		memset(affi_llc, 0, (max_lid + 1) * sizeof(int));
+		memset(affi, 0, (max_lid + 1) * sizeof(int));
+		*num = get_affi_llcs(sd_cur, src_llc, dst_llc, affi_llc, affi);
+	}
+
+	for (int i = 0; i < *num; i++) {
+		if ((unsigned int)affi_llc[i] < sd->llc_max)
+			score += sd->llc_counts[affi_llc[i]] * affi[i];
+	}
+
+	return score;
+}
+
+/*
+ * Scratch arrays used while scoring rqs during load balancing. They live on
+ * the bottom sched_domain of the CPU running the balance, so they share the
+ * sched_domain lifetime and need neither a separate allocation nor any RCU
+ * handling of their own. Load balancing runs with preemption disabled, so
+ * this_rq() stays stable for a whole pass, and two CPUs balancing at the same
+ * time each get their own arrays.
+ */
+static void lb_affi_scratch(int **ids, int **weights)
+{
+	struct sched_domain *sd = rcu_dereference(this_rq()->sd);
+
+	*ids = sd ? sd->affi_ids : NULL;
+	*weights = sd ? sd->affi_weights : NULL;
+}
+
+/*
+ * To locate a source sched group/rq during load balancing, we require
+ * a metric to evaluate the migration benefit of each rq. For cache-aware
+ * scheduling, the primary optimization target is affinity improvement.
+ *
+ * This implementation quantifies affinity gains for candidate rqs by
+ * computing an affinity score for each rq.
+ *
+ * The score is calculated given source LLC/node and destination LLC/node,
+ * under two distinct scenarios: NUMA-domain load balance and NODE-domain
+ * load balance. For NUMA-level balancing, affinity gain is accounted at
+ * NUMA granularity; for node-level balancing, evaluation proceeds at LLC
+ * granularity.
+ *
+ * Taking NUMA-level balancing as an example, the score is computed as:
+ * Di = node_distance(src_node, NUMAi) - node_distance(dst_node, NUMAi)   (1)
+ * Wi = Rt_i * Di                                                       (2)
+ * p  = sum_i Wi                                                        (3)
+ *
+ * Where:
+ * i        Index of a remote NUMA node
+ * Di       Affinity gain, derived from the difference between node
+ *          distance from source to NUMAi and distance from destination
+ *          to NUMAi
+ * Rt_i     Count of tasks on this rq whose preferred NUMA is NUMAi,
+ *          retrieved via rq->sd->node_count
+ */
+static int cal_affinity_score(struct sched_domain *sd, struct rq *rq, int src_cpu, int dst_cpu,
+			int *affi_nodes, int *affi, int *last_node, int *num)
+{
+	struct sched_domain *sd_tmp = rcu_dereference(rq->sd);
+
+	if (!affi_nodes || !affi || !last_node || !num || !sd || !sd_tmp)
+		return 0;
+
+	if (sd->flags & SD_NUMA)
+		return calc_affinity_numa_score(sd_tmp, src_cpu, dst_cpu, affi_nodes,
+						affi, last_node, num);
+
+	return calc_affinity_llc_score(sd, sd_tmp, src_cpu, dst_cpu, affi_nodes,
+				       affi, last_node, num);
+}
 #else
 static inline void record_sg_llc_stats(struct lb_env *env, struct sg_lb_stats *sgs,
 				       struct sched_group *group)
-- 
2.34.1

[RFC PATCH v2 13/23] sched/cache: Pick optimal src rq/group using affinity promotion metric

[Jianyong Wu]

Current source group/rq selection only considers LLC preference and
overlooks potential NUMA affinity improvements. Utilize the affinity
gain calculation from the prior commit to choose optimal source sched
group and runqueue in load balancing.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 505a1ea9537f..584297528a9e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10653,7 +10653,7 @@ static bool get_span_stats(const struct cpumask *span, unsigned long *util_out,
  * Decide if migration should happen on a specific node.
  * The node here is an LLC or a NUMA.
  */
-static enum llc_mig __maybe_unused can_migrate_node(int src_cpu, int dst_cpu,
+static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 			struct task_struct *p, bool to_pref)
 {
 	const struct cpumask *span;
@@ -11983,8 +11983,8 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
 		return false;
 
 	if (sgs->nr_pref_dst_llc &&
-	    can_migrate_llc(cpumask_first(sched_group_span(group)),
-			    env->dst_cpu, 0, true) == mig_llc)
+	    can_migrate_node(cpumask_first(sched_group_span(group)),
+			    env->dst_cpu, NULL, true) == mig_llc)
 		return true;
 
 	return false;
@@ -12203,6 +12203,12 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 {
 	int i, nr_running, local_group, sd_flags = env->sd->flags;
 	bool balancing_at_rd = !env->sd->parent;
+#ifdef CONFIG_SCHED_CACHE
+	int last_node = -1, node_num = 0;
+	int *cache_data, *affi;
+
+	lb_affi_scratch(&cache_data, &affi);
+#endif
 
 	memset(sgs, 0, sizeof(*sgs));
 
@@ -12232,7 +12238,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			if (llc_id(i) != dst_llc) {
 				sd_tmp = rcu_dereference_all(rq->sd);
 				if (sd_tmp && (unsigned int)dst_llc < sd_tmp->llc_max)
-					sgs->nr_pref_dst_llc += sd_tmp->llc_counts[dst_llc];
+					sgs->nr_pref_dst_llc += cal_affinity_score(env->sd, rq, i,
+					    env->dst_cpu, cache_data, affi, &last_node, &node_num);
 			}
 		}
 #endif
@@ -13330,9 +13337,13 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 	unsigned long busiest_util = 0, busiest_load = 0, busiest_capacity = 1;
 	unsigned int __maybe_unused busiest_pref_llc = 0;
 	struct sched_domain __maybe_unused *sd_tmp;
+	int __maybe_unused dst_llc, __maybe_unused *cache_data, __maybe_unused *affi;
+	int __maybe_unused last_node = -1, __maybe_unused node_num = 0, i;
 	unsigned int busiest_nr = 0;
-	int __maybe_unused dst_llc;
-	int i;
+
+#ifdef CONFIG_SCHED_CACHE
+	lb_affi_scratch(&cache_data, &affi);
+#endif
 
 	for_each_cpu_and(i, sched_group_span(group), env->cpus) {
 		unsigned long capacity, load, util;
@@ -13466,8 +13477,8 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 
 			if (sd_tmp && (unsigned)dst_llc < sd_tmp->llc_max) {
 				unsigned int this_pref_llc =
-					sd_tmp->llc_counts[dst_llc];
-
+					cal_affinity_score(env->sd, rq, i, env->dst_cpu,
+					    cache_data, affi, &last_node, &node_num);
 				if (busiest_pref_llc < this_pref_llc) {
 					busiest_pref_llc = this_pref_llc;
 					busiest = rq;
-- 
2.34.1

[RFC PATCH v2 14/23] sched/cache: Drop prefer_sibling restriction for llc_balance

[Jianyong Wu]

group_llc_balance performs balancing across LLC and NUMA domains.
The prefer_sibling constraint unnecessarily limits its scope, so remove
this requirement entirely from the branch condition.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 584297528a9e..5c35a43e5e38 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13274,9 +13274,9 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 	 * Try to move all excess tasks to a sibling domain of the busiest
 	 * group's child domain.
 	 */
-	if (sds.prefer_sibling && local->group_type == group_has_spare &&
-	    (busiest->group_type == group_llc_balance ||
-	    sibling_imbalance(env, &sds, busiest, local) > 1))
+	if (local->group_type == group_has_spare &&
+	    ((busiest->group_type == group_llc_balance) || (sds.prefer_sibling &&
+	    sibling_imbalance(env, &sds, busiest, local) > 1)))
 		goto force_balance;
 
 	if (busiest->group_type != group_overloaded) {
-- 
2.34.1

[RFC PATCH v2 15/23] sched/cache: Judge migration eligibility in LLC granularity

[Jianyong Wu]

Helpers compute affinity gain has been introduced in the
previous patches. Leverage this helper to assess migration eligibility
during load balance, covering paths like can_migrate_llc_task and
alb_break_llc.

In alb_break_llc, change the logic to only handle runqueues with a single
task as we need the preferred LLC as a input to decide if it need migrate.

Runqueues holding multiple tasks are evaluated in can_migrate_task instead.
We move the NUMA affinity check ahead of active load balance logic here.
Without this reordering, multi-task runqueues skip alb_break_llc filtering,
which could trigger need_active_balance = true and set LBF_ACTIVE_LB
even when the task already resides on its optimal node, which is an
undesirable migration scenario we want to avoid.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 126 +++++++++++++++++---------------------------
 1 file changed, 49 insertions(+), 77 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 5c35a43e5e38..0defdd4316b8 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10562,54 +10562,6 @@ enum llc_mig {
 	mig_unrestricted	/* G: Don't restrict generic load balance migration */
 };
 
-/*
- * Check if task can be moved from the source LLC to the
- * destination LLC without breaking cache aware preferrence.
- * src_cpu and dst_cpu are arbitrary CPUs within the source
- * and destination LLCs, respectively.
- */
-static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
-				    unsigned long tsk_util,
-				    bool to_pref)
-{
-	unsigned long src_util, dst_util, src_cap, dst_cap;
-
-	if (!get_llc_stats(src_cpu, &src_util, &src_cap) ||
-	    !get_llc_stats(dst_cpu, &dst_util, &dst_cap))
-		return mig_unrestricted;
-
-	src_util = src_util < tsk_util ? 0 : src_util - tsk_util;
-	dst_util = dst_util + tsk_util;
-
-	if (!fits_llc_capacity(dst_util, dst_cap) &&
-	    !fits_llc_capacity(src_util, src_cap))
-		return mig_unrestricted;
-
-	if (to_pref) {
-		/*
-		 * Don't migrate if we will get preferred LLC too
-		 * heavily loaded and if the dest is much busier
-		 * than the src, in which case migration will
-		 * increase the imbalance too much.
-		 */
-		if (!fits_llc_capacity(dst_util, dst_cap) &&
-		    util_greater(dst_util, src_util))
-			return mig_forbid;
-	} else {
-		/*
-		 * Don't migrate if we will leave preferred LLC
-		 * too idle, or if this migration leads to the
-		 * non-preferred LLC falls within sysctl_aggr_imb percent
-		 * of preferred LLC, leading to migration again
-		 * back to preferred LLC.
-		 */
-		if (fits_llc_capacity(src_util, src_cap) ||
-		    !util_greater(src_util, dst_util))
-			return mig_forbid;
-	}
-	return mig_llc;
-}
-
 /*
  * Like get_llc_stats but for sched domain that above LLC level.
  * Based on get_llc_stats, we can accumulate utilization and cap for
@@ -10795,6 +10747,29 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 	return mig_unrestricted;
 }
 
+/* Decide if the migration improve the affinity */
+static bool if_to_prefer(int src_cpu, int dst_cpu, int pref_llc)
+{
+	int src_dist, dst_dist, src_node, dst_node, pref_node;
+
+	src_node = cpu_to_node(src_cpu);
+	dst_node = cpu_to_node(dst_cpu);
+	pref_node = llc_to_node(pref_llc);
+
+	if (src_node == pref_node && dst_node == pref_node) {
+		src_dist = llc_intra_node_distance(llc_id(src_cpu), pref_llc, pref_node);
+		dst_dist = llc_intra_node_distance(llc_id(dst_cpu), pref_llc, pref_node);
+	} else {
+		src_dist = sched_cache_node_distance(src_node, pref_node);
+		dst_dist = sched_cache_node_distance(dst_node, pref_node);
+	}
+
+	if (src_dist < 0 || dst_dist < 0)
+		return false;
+
+	return src_dist > dst_dist;
+}
+
 /*
  * Check if task p can migrate from source LLC to
  * destination LLC in terms of cache aware load balance.
@@ -10822,15 +10797,10 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,
 		return mig_unrestricted;
 	}
 
-	if (cpus_share_cache(dst_cpu, cpu))
-		to_pref = true;
-	else if (cpus_share_cache(src_cpu, cpu))
-		to_pref = false;
-	else
-		return mig_unrestricted;
+	to_pref = if_to_prefer(src_cpu, dst_cpu, llc_id(cpu));
 
-	return can_migrate_llc(src_cpu, dst_cpu,
-			       task_util(p), to_pref);
+	return can_migrate_node(src_cpu, dst_cpu,
+			       p, to_pref);
 }
 
 /*
@@ -10844,33 +10814,35 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,
 static inline bool
 alb_break_llc(struct lb_env *env)
 {
+	int pref_llc = -1;
+	bool to_pref = false;
+
 	if (!sched_cache_enabled())
 		return false;
 
 	if (cpus_share_cache(env->src_cpu, env->dst_cpu))
 		return false;
 	/*
-	 * All tasks prefer to stay on their current CPU.
-	 * Do not pull a task from its preferred CPU if:
-	 * 1. It is the only task running and does not exceed
-	 *    imbalance allowance; OR
-	 * 2. Migrating it away from its preferred LLC would violate
-	 *    the cache-aware scheduling policy.
+	 * We need the preferred LLC to decide whether we can perform migration.
+	 * Therefore, we need to obtain task_struct, which is only meaningful
+	 * in the case that only one task on the rq.
+	 * For cases with more than one task on the rq, we need to check
+	 * this in can_migrate_task().
 	 */
-	if (env->src_rq->nr_pref_llc_running &&
-	    env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable) {
-		unsigned long util = 0;
+	if (env->src_rq->nr_running == 1) {
 		struct task_struct *cur;
 
-		if (env->src_rq->nr_running <= 1)
-			return true;
-
 		cur = rcu_dereference_all(env->src_rq->curr);
-		if (cur && cur->sched_class == &fair_sched_class)
-			util = task_util(cur);
+		if (!cur || !cur->mm)
+			return false;
 
-		if (can_migrate_llc(env->src_cpu, env->dst_cpu,
-				    util, false) == mig_forbid)
+		if (cur->sched_class == &fair_sched_class)
+			pref_llc = llc_id(cur->mm->sc_stat.cpu);
+
+		to_pref = if_to_prefer(env->src_cpu, env->dst_cpu, pref_llc);
+
+		if (can_migrate_node(env->src_cpu, env->dst_cpu,
+				    cur, to_pref) == mig_forbid)
 			return true;
 	}
 
@@ -11022,14 +10994,11 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 
 	/*
 	 * Aggressive migration if:
-	 * 1) active balance
-	 * 2) destination numa is preferred
+	 * 1) destination numa is preferred
+	 * 2) active balance
 	 * 3) task is cache cold, or
 	 * 4) too many balance attempts have failed.
 	 */
-	if (env->flags & LBF_ACTIVE_LB)
-		return 1;
-
 	degrades = migrate_degrades_locality(p, env);
 	if (!degrades) {
 		/*
@@ -11055,6 +11024,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		hot = degrades > 0;
 	}
 
+	if (env->flags & LBF_ACTIVE_LB)
+		return 1;
+
 	if (!hot || env->sd->nr_balance_failed > env->sd->cache_nice_tries) {
 		if (hot)
 			p->sched_task_hot = 1;
-- 
2.34.1

[RFC PATCH v2 16/23] sched/cache: Allow un-throttled active balance to spread out of a full LLC

[Jianyong Wu]

Cache-aware aggregation reaches active balance through two asymmetric
paths. Pulling a task back to its preferred LLC uses migrate_llc_task,
which need_active_balance() lets through unconditionally. Pushing a task
the other way. Out of an LLC that is already over the aggregation cap
and into one that still has room. It only happens via migrate_task, which
first has to wait for nr_balance_failed to exceed cache_nice_tries + 2.

Let the spread direction reach active balance without the
nr_balance_failed wait as well. The source LLC must already be over the
aggregation cap and the destination must still fit under it with a full
CPU of load added, so this can only undo an overshoot and never fights
the aggregation itself.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 58 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0defdd4316b8..42ab6347e667 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13506,6 +13506,61 @@ imbalanced_active_balance(struct lb_env *env)
 	return 0;
 }
 
+#ifdef CONFIG_SCHED_CACHE
+/*
+ * Cache-aware aggregation pulls tasks back to the preferred LLC via
+ * migrate_llc_task, which reaches active balance without any
+ * nr_balance_failed gating. Spreading tasks the other way. It only
+ * happens through migrate_task, which must first wait for
+ * nr_balance_failed to exceed cache_nice_tries + 2. That asymmetry
+ * lets a few LLCs stay oversubscribed while other LLCs of the same
+ * preferred region stay idle.
+ *
+ * Allow the spread direction to active balance un-throttled as well. The
+ * source LLC must be over the aggregation cap and the destination must
+ * still fit under it, so this can only undo an overshoot and never fights
+ * the aggregation itself. Region containment is still enforced by
+ * alb_break_llc()/can_migrate_node(), which run before this.
+ */
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	unsigned long src_util, src_cap, dst_util, dst_cap;
+
+	if (!sched_cache_enabled())
+		return false;
+
+	if (env->migration_type != migrate_task)
+		return false;
+
+	if (cpus_share_cache(env->src_cpu, env->dst_cpu))
+		return false;
+
+	if (!get_llc_stats(env->src_cpu, &src_util, &src_cap) ||
+	    !get_llc_stats(env->dst_cpu, &dst_util, &dst_cap))
+		return false;
+
+	if (fits_llc_capacity(src_util, src_cap))
+		return false;
+
+	/*
+	 * Only move when the destination still fits with a full CPU of the load
+	 * added, and when the source leads by at least two CPUs worth, so that
+	 * moving one task cannot invert the imbalance and bounce it straight
+	 * back.
+	 */
+	if (!fits_llc_capacity(dst_util + SCHED_CAPACITY_SCALE, dst_cap))
+		return false;
+
+	/* Avoid threads ping-pong migration */
+	return util_greater(src_util, dst_util);
+}
+#else
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	return false;
+}
+#endif
+
 static int need_active_balance(struct lb_env *env)
 {
 	struct sched_domain *sd = env->sd;
@@ -13519,6 +13574,9 @@ static int need_active_balance(struct lb_env *env)
 	if (imbalanced_active_balance(env))
 		return 1;
 
+	if (llc_spread_active_balance(env))
+		return 1;
+
 	/*
 	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
 	 * It's worth migrating the task if the src_cpu's capacity is reduced
-- 
2.34.1

[RFC PATCH v2 17/23] sched/fair: Fine-granularity NUMA balancing

[Jianyong Wu]

NUMA balancing currently consists of two coupled components:
task migration and page migration. These two mechanisms work in tandem.
However, task migration may conflict with other subsystems such as CAS,
which aggregates tasks solely via task migration. Conflicts arise when
both CAS and NUMA balancing are enabled simultaneously.

This patch addresses the problem by introducing a fine-grained NUMA
balancing mechanism, allowing task migration and page migration to
be enabled or disabled independently.

Suggested-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/sched/sysctl.h | 12 +++++
 kernel/sched/core.c          | 87 ++++++++++++++++++++++++++++++++++++
 kernel/sched/fair.c          |  3 ++
 mm/memory.c                  |  3 ++
 4 files changed, 105 insertions(+)

diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h
index 5a64582b086b..3434f044087e 100644
--- a/include/linux/sched/sysctl.h
+++ b/include/linux/sched/sysctl.h
@@ -23,10 +23,22 @@ enum sched_tunable_scaling {
 #define NUMA_BALANCING_NORMAL		0x1
 #define NUMA_BALANCING_MEMORY_TIERING	0x2
 
+/*
+ * Fine-grained control of the NUMA balancing sub-features. These are
+ * independent of the master NUMA_BALANCING_NORMAL switch and let the page
+ * migration and task migration paths be disabled separately.
+ */
+#define NUMA_BALANCING_PAGE_MIGRATION	0x1
+#define NUMA_BALANCING_TASK_MIGRATION	0x2
+#define NUMA_BALANCING_MIGRATE_DEFAULT	(NUMA_BALANCING_PAGE_MIGRATION | \
+					 NUMA_BALANCING_TASK_MIGRATION)
+
 #ifdef CONFIG_NUMA_BALANCING
 extern int sysctl_numa_balancing_mode;
+extern unsigned int numa_balancing_migrate_mode;
 #else
 #define sysctl_numa_balancing_mode	0
+#define numa_balancing_migrate_mode	NUMA_BALANCING_MIGRATE_DEFAULT
 #endif
 
 #endif /* _LINUX_SCHED_SYSCTL_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f6..8b6f2c4373ea 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4621,6 +4621,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);
 
 int sysctl_numa_balancing_mode;
 
+unsigned int numa_balancing_migrate_mode __read_mostly =
+	NUMA_BALANCING_MIGRATE_DEFAULT;
+
 static void __set_numabalancing_state(bool enabled)
 {
 	if (enabled)
@@ -4674,6 +4677,83 @@ static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 	}
 	return err;
 }
+
+static void numa_balancing_mode_show(char *buf, size_t size)
+{
+	unsigned int mode = numa_balancing_migrate_mode;
+
+	switch (mode & NUMA_BALANCING_MIGRATE_DEFAULT) {
+	case NUMA_BALANCING_MIGRATE_DEFAULT:
+		strscpy(buf, "page_mig task_mig", size);
+		break;
+	case NUMA_BALANCING_PAGE_MIGRATION:
+		strscpy(buf, "page_mig", size);
+		break;
+	case NUMA_BALANCING_TASK_MIGRATION:
+		strscpy(buf, "task_mig", size);
+		break;
+	default:
+		strscpy(buf, "none", size);
+	}
+}
+
+static int numa_balancing_mode_parse(char *buf)
+{
+	unsigned int mode = numa_balancing_migrate_mode;
+	char *tok;
+
+	while ((tok = strsep(&buf, " \t\n,")) != NULL) {
+		if (!*tok)
+			continue;
+
+		if (!strcmp(tok, "default") || !strcmp(tok, "enable") ||
+		    !strcmp(tok, "all")) {
+			mode = NUMA_BALANCING_MIGRATE_DEFAULT;
+		} else if (!strcmp(tok, "disable") || !strcmp(tok, "none") ||
+			   !strcmp(tok, "off")) {
+			mode = 0;
+		} else if (!strcmp(tok, "no_page_mig")) {
+			mode &= ~NUMA_BALANCING_PAGE_MIGRATION;
+		} else if (!strcmp(tok, "no_task_mig")) {
+			mode &= ~NUMA_BALANCING_TASK_MIGRATION;
+		} else if (!strcmp(tok, "page_mig")) {
+			mode |= NUMA_BALANCING_PAGE_MIGRATION;
+		} else if (!strcmp(tok, "task_mig")) {
+			mode |= NUMA_BALANCING_TASK_MIGRATION;
+		} else {
+			return -EINVAL;
+		}
+	}
+
+	numa_balancing_migrate_mode = mode;
+	return 0;
+}
+
+static int proc_numa_balancing_mode(const struct ctl_table *table, int write,
+				      void *buffer, size_t *lenp, loff_t *ppos)
+{
+	struct ctl_table t;
+	char buf[32];
+	int err;
+
+	if (write && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	t = *table;
+	t.data = buf;
+	t.maxlen = sizeof(buf);
+
+	if (write) {
+		buf[0] = '\0';
+		err = proc_dostring(&t, write, buffer, lenp, ppos);
+		if (err)
+			return err;
+		return numa_balancing_mode_parse(buf);
+	}
+
+	numa_balancing_mode_show(buf, sizeof(buf));
+	return proc_dostring(&t, write, buffer, lenp, ppos);
+}
 #endif /* CONFIG_PROC_SYSCTL */
 #endif /* CONFIG_NUMA_BALANCING */
 
@@ -4787,6 +4867,13 @@ static const struct ctl_table sched_core_sysctls[] = {
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= SYSCTL_FOUR,
 	},
+	{
+		.procname	= "numa_balancing_mode",
+		.data		= NULL, /* filled in by handler */
+		.maxlen		= 32,
+		.mode		= 0644,
+		.proc_handler	= proc_numa_balancing_mode,
+	},
 #endif /* CONFIG_NUMA_BALANCING */
 };
 static int __init sched_core_sysctl_init(void)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 42ab6347e667..00f49e767583 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3312,6 +3312,9 @@ static int task_numa_migrate(struct task_struct *p)
 	struct rq *best_rq;
 	int nid, ret, dist;
 
+	if (!(numa_balancing_migrate_mode & NUMA_BALANCING_TASK_MIGRATION))
+		return -EAGAIN;
+
 	/*
 	 * Pick the lowest SD_NUMA domain, as that would have the smallest
 	 * imbalance and would be the first to start moving tasks about.
diff --git a/mm/memory.c b/mm/memory.c
index 6b8280cfc1db..47c9614dd54c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6049,6 +6049,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 		*flags |= TNF_FAULT_LOCAL;
 	}
 
+	if (!(numa_balancing_migrate_mode & NUMA_BALANCING_PAGE_MIGRATION))
+		return NUMA_NO_NODE;
+
 	return mpol_misplaced(folio, vmf, addr);
 }
 
-- 
2.34.1

[RFC PATCH v2 18/23] sched/cache: Scan all prefer nodes in thread group

[Jianyong Wu]

get_scan_cpumasks only scan the current task's preferred node.However,
each of task inside a thread group may have its own preferred node. Thus,
each scan may have different scan scope which may make it unstable to find
preferred LLC.

To address this issue, record all of the active preferred nodes from all
of the whole thread group. Then, take them all into the scan scope.
Additinally, remove the preferred node from the scan scope if it has not
been touched for a while like preferred llc invalidation does.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/mm_types.h |  6 +-----
 include/linux/sched.h    |  1 +
 kernel/sched/fair.c      | 42 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+), 5 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index b18c2b2e7d2c..7fef34c6f33a 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1626,11 +1626,7 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 
 #define mm_alloc_sched(...)	alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__))
 
-static inline void mm_destroy_sched(struct mm_struct *mm)
-{
-	free_percpu(mm->sc_stat.pcpu_sched);
-	mm->sc_stat.pcpu_sched = NULL;
-}
+void mm_destroy_sched(struct mm_struct *mm);
 #else /* !CONFIG_SCHED_CACHE */
 
 static inline int mm_alloc_sched(struct mm_struct *mm) { return 0; }
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1e4136c2b2a3..7fd4ea8c9037 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2393,6 +2393,7 @@ struct sched_cache_time {
 
 struct sched_cache_stat {
 	struct sched_cache_time __percpu *pcpu_sched;
+	unsigned long *node_epoch;
 	raw_spinlock_t lock;
 	unsigned long epoch;
 	u64 nr_running_avg;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 00f49e767583..27cb05373a1d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1605,6 +1605,9 @@ void mm_init_sched(struct mm_struct *mm,
 	mm->sc_stat.next_scan = jiffies;
 	mm->sc_stat.nr_running_avg = 0;
 	mm->sc_stat.footprint = 0;
+	mm->sc_stat.node_epoch = kcalloc(num_possible_nodes(),
+					 sizeof(*mm->sc_stat.node_epoch),
+					 GFP_KERNEL);
 	/*
 	 * The update to mm->sc_stat should not be reordered
 	 * before initialization to mm's other fields, in case
@@ -1613,6 +1616,14 @@ void mm_init_sched(struct mm_struct *mm,
 	smp_store_release(&mm->sc_stat.pcpu_sched, _pcpu_sched);
 }
 
+void mm_destroy_sched(struct mm_struct *mm)
+{
+	free_percpu(mm->sc_stat.pcpu_sched);
+	mm->sc_stat.pcpu_sched = NULL;
+	kfree(mm->sc_stat.node_epoch);
+	mm->sc_stat.node_epoch = NULL;
+}
+
 /* because why would C be fully specified */
 static __always_inline void __shr_u64(u64 *val, unsigned int n)
 {
@@ -1700,7 +1711,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 {
 	struct sched_cache_time *pcpu_sched;
 	struct mm_struct *mm = p->mm;
+	unsigned long *node_epoch;
 	int mm_sched_llc = -1;
+	int node, pref_nid;
 	unsigned long epoch;
 
 	if (!sched_cache_enabled())
@@ -1724,6 +1737,11 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 		epoch = rq->cpu_epoch;
 	}
 
+	node_epoch = mm->sc_stat.node_epoch;
+	pref_nid = READ_ONCE(p->numa_preferred_nid);
+	if (pref_nid != NUMA_NO_NODE && mm->sc_stat.node_epoch)
+		mm->sc_stat.node_epoch[pref_nid] = epoch;
+
 	/*
 	 * If this process hasn't hit task_cache_work() for a while invalidate
 	 * its preferred state.
@@ -1733,6 +1751,16 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	    exceed_llc_capacity(mm, cpu_of(rq))) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
+
+		if (node_epoch)
+			memset(mm->sc_stat.node_epoch, 0,
+				sizeof(*node_epoch) * num_possible_nodes());
+	} else if (node_epoch) {
+		for_each_node(node) {
+			if ((long)(epoch - READ_ONCE(node_epoch[node])) >
+			    llc_epoch_affinity_timeout)
+				WRITE_ONCE(node_epoch[node], 0);
+		}
 	}
 
 	mm_sched_llc = get_pref_llc(p, mm);
@@ -1777,10 +1805,12 @@ static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p)
 {
 #ifdef CONFIG_NUMA_BALANCING
 	int cpu, curr_cpu, nid, pref_nid;
+	unsigned long *node_epoch;
 
 	if (!static_branch_likely(&sched_numa_balancing))
 		goto out;
 
+	node_epoch = p->mm->sc_stat.node_epoch;
 	cpu = READ_ONCE(p->mm->sc_stat.cpu);
 	if (cpu != -1)
 		nid = cpu_to_node(cpu);
@@ -1801,6 +1831,18 @@ static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p)
 	if (pref_nid == NUMA_NO_NODE)
 		goto out;
 
+	if (node_epoch) {
+		int node;
+
+		for_each_node(node) {
+			if (node == pref_nid)
+				continue;
+
+			if (node_epoch[node] > 0)
+				cpumask_or(cpus, cpus, cpumask_of_node(node));
+		}
+	}
+
 	cpumask_or(cpus, cpus, cpumask_of_node(pref_nid));
 
 	/* honor the task's preferred LLC CPU */
-- 
2.34.1

[RFC PATCH v2 19/23] sched/cache: Remove preferred LLC/node check no longer needed

[Jianyong Wu]

When CAS extends to the whole system, preferred LLC may not locate inside
preferred node of a certain thread and migration can happen to move to a
destination cpu that is not thread group's preferred LLC. So, all of these
check should be removed.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c | 29 +----------------------------
 1 file changed, 1 insertion(+), 28 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 27cb05373a1d..45dd3e5e0a86 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1680,27 +1680,9 @@ static int get_pref_llc(struct task_struct *p, struct mm_struct *mm)
 		return -1;
 
 	mm_sched_cpu = READ_ONCE(mm->sc_stat.cpu);
-	if (mm_sched_cpu != -1) {
+	if (mm_sched_cpu != -1)
 		mm_sched_llc = llc_id(mm_sched_cpu);
 
-#ifdef CONFIG_NUMA_BALANCING
-		/*
-		 * Don't assign preferred LLC if it
-		 * conflicts with NUMA balancing.
-		 * This can happen when sched_setnuma() gets
-		 * called, however it is not much of an issue
-		 * because we expect account_mm_sched() to get
-		 * called fairly regularly -- at a higher rate
-		 * than sched_setnuma() at least -- and thus the
-		 * conflict only exists for a short period of time.
-		 */
-		if (static_branch_likely(&sched_numa_balancing) &&
-		    p->numa_preferred_nid >= 0 &&
-		    cpu_to_node(mm_sched_cpu) != p->numa_preferred_nid)
-			mm_sched_llc = -1;
-#endif
-	}
-
 	return mm_sched_llc;
 }
 
@@ -10917,15 +10899,6 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env)
 	if (env->sd->nr_balance_failed >= env->sd->cache_nice_tries + 1)
 		return false;
 
-	/*
-	 * We know the env->src_cpu has some tasks prefer to
-	 * run on env->dst_cpu, skip the tasks do not prefer
-	 * env->dst_cpu, and find the one that prefers.
-	 */
-	if (env->migration_type == migrate_llc_task &&
-	    READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu))
-		return true;
-
 	if (can_migrate_llc_task(env->src_cpu,
 				 env->dst_cpu, p) != mig_forbid)
 		return false;
-- 
2.34.1

[RFC PATCH v2 20/23] sched/cache: Estimate utilization of the whole thread group

[Jianyong Wu]

Since we want to spread threads across multiple LLCs, it is helpful
to know the overall utilization of the thread-group when deciding
whether a task may be migrated to a destination LLC.

We track this estimate using an asymmetric EWMA: rising load is
weighted by 1/2, while falling load uses a weight of 1/8.
This allows fast expansion of placement, while preventing premature
range contraction caused by short-term idle periods.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/sched.h |  1 +
 kernel/sched/fair.c   | 44 +++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 7fd4ea8c9037..96263bcb9e84 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2397,6 +2397,7 @@ struct sched_cache_stat {
 	raw_spinlock_t lock;
 	unsigned long epoch;
 	u64 nr_running_avg;
+	u64 util_avg;
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 45dd3e5e0a86..848abcf93c47 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1604,6 +1604,7 @@ void mm_init_sched(struct mm_struct *mm,
 	mm->sc_stat.cpu = -1;
 	mm->sc_stat.next_scan = jiffies;
 	mm->sc_stat.nr_running_avg = 0;
+	mm->sc_stat.util_avg = 0;
 	mm->sc_stat.footprint = 0;
 	mm->sc_stat.node_epoch = kcalloc(num_possible_nodes(),
 					 sizeof(*mm->sc_stat.node_epoch),
@@ -1860,11 +1861,35 @@ static inline void update_avg_scale(u64 *avg, u64 sample)
 	*avg += div64_s64(diff, divisor);
 }
 
+/*
+ * Keep a conservative estimate of the total CFS utilization generated by an
+ * mm.  Rise quickly so that spreading is not delayed when demand grows, but
+ * decay slowly so that a short idle interval does not immediately pull the
+ * workload back into fewer LLCs.
+ *
+ * The value is expressed in scheduler capacity units: SCHED_CAPACITY_SCALE is
+ * one fully utilized CPU.
+ */
+static inline void update_mm_util_avg(u64 *avg, u64 sample)
+{
+	s64 diff;
+	u32 divisor;
+
+	if (sample >= *avg)
+		diff = sample - *avg;
+	else
+		diff = -(*avg - sample);
+
+	divisor = diff > 0 ? 2 : 8;
+	*avg += div64_s64(diff, divisor);
+}
+
 static void task_cache_work(struct callback_head *work)
 {
 	unsigned long next_scan, now = jiffies;
 	struct task_struct *p = current, *cur;
 	struct mm_struct *mm = p->mm;
+	u64 group_util = 0;
 	unsigned long m_a_n_occ = 0, curr_m_a_n_occ = 0, curr_m_a_occ = 0;
 	unsigned long pref_llc_occ = 0;
 	int cpu, m_a_n_cpu = -1, nr_running = 0, curr_cpu;
@@ -1933,9 +1958,23 @@ static void task_cache_work(struct callback_head *work)
 					continue;
 
 				for_each_cpu(i, sched_domain_span(sd)) {
+					unsigned long cpu_util, mm_util;
+
 					occ = fraction_mm_sched(cpu_rq(i),
 							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
 					a_occ += occ;
+
+					/*
+					 * fraction_mm_sched() is a share of executed CFS
+					 * time, not an absolute utilization.  Scale the
+					 * CPU's PELT utilization by that share to estimate
+					 * this mm's utilization on the CPU.
+					 */
+					cpu_util = cpu_util_cfs(i);
+					mm_util = mul_u64_u32_div(cpu_util,
+							min_t(unsigned long, occ, NICE_0_LOAD),
+							NICE_0_LOAD);
+					group_util += mm_util;
 					if (occ > m_occ) {
 						m_occ = occ;
 						m_cpu = i;
@@ -2003,6 +2042,7 @@ static void task_cache_work(struct callback_head *work)
 		WRITE_ONCE(mm->sc_stat.cpu, new_cpu);
 
 	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
+	update_mm_util_avg(&mm->sc_stat.util_avg, group_util);
 	free_cpumask_var(cpus);
 }
 
@@ -10639,7 +10679,7 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 	struct mm_struct *mm;
 	unsigned long dst_util, dst_cap, tsk_util = 0;
 	unsigned long src_util = 0, src_cap = 0;
-	unsigned long acc_util = 0, acc_cap = 0;
+	unsigned long acc_util = 0, acc_cap = 0, dst_pre;
 	int node, target_cpu = src_cpu;
 	int get_src = 0;
 
@@ -10659,9 +10699,9 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 	}
 
 	dst_util = dst_util + tsk_util;
+	dst_pre = dst_util - tsk_util;
 
 	if (to_pref) {
-		unsigned long dst_pre = dst_util - tsk_util;
 
 		if (fits_llc_capacity(dst_util, dst_cap))
 			return mig_llc;
-- 
2.34.1

[RFC PATCH v2 21/23] sched/cache: Spread workloads within an estimated LLC range

[Jianyong Wu]

Before this patch, migration of a task is forbidden if the previous
LLC in the preferred-LLC affinity sequence is not saturated, even
when the destination LLC falls within the thread-group's load range.
This hinders load spreading and causes LLC-level load aggregation.

To fix this issue, we first estimate the total load of the thread
group. Migration to the destination LLC is then permitted as long
as the target LLC lies within the acceptable load range. This change
accelerates load spreading and mitigates LLC load-aggregation.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 include/linux/sched.h   |   3 +
 kernel/sched/fair.c     | 332 +++++++++++++++++++++++++++++++++++++---
 kernel/sched/sched.h    |   1 +
 kernel/sched/topology.c |  14 +-
 4 files changed, 325 insertions(+), 25 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 96263bcb9e84..808a114c533d 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1425,6 +1425,8 @@ struct task_struct {
 	int				preferred_llc;
 	/* 1: task was enqueued to its preferred LLC, 0 otherwise */
 	int				pref_llc_queued;
+	/* 1: task contributed to sched_domain::llc_counts */
+	int				pref_llc_counted;
 #endif
 
 	struct rseq_data		rseq;
@@ -2398,6 +2400,7 @@ struct sched_cache_stat {
 	unsigned long epoch;
 	u64 nr_running_avg;
 	u64 util_avg;
+	u64 llc_range;
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 848abcf93c47..def42490fa2a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1410,6 +1410,41 @@ __read_mostly unsigned int llc_epoch_affinity_timeout = EPOCH_LLC_AFFINITY_TIMEO
 __read_mostly unsigned int llc_imb_pct		= 20;
 __read_mostly unsigned int llc_overaggr_pct	= 50;
 
+#define MM_LLC_RANGE_INVALID	U64_MAX
+
+static u64 mm_llc_range_pack(int cpu, int dist)
+{
+	return (u64)(u32)cpu << 32 | (u32)dist;
+}
+
+static int mm_llc_range_cpu(u64 range)
+{
+	return upper_32_bits(range);
+}
+
+static int mm_llc_range_dist(u64 range)
+{
+	return lower_32_bits(range);
+}
+
+static int mm_estimated_range_dist(int pref_cpu, u64 group_util,
+				   u64 nr_running);
+static int mm_cache_distance(int pref_cpu, int cpu);
+
+static bool mm_llc_in_estimated_range(struct mm_struct *mm, int pref_cpu,
+				      int cpu)
+{
+	u64 range = READ_ONCE(mm->sc_stat.llc_range);
+	int dist;
+
+	if (range == MM_LLC_RANGE_INVALID ||
+	    mm_llc_range_cpu(range) != pref_cpu)
+		return false;
+
+	dist = mm_cache_distance(pref_cpu, cpu);
+	return dist >= 0 && dist <= mm_llc_range_dist(range);
+}
+
 static int llc_id(int cpu)
 {
 	if (cpu < 0)
@@ -1499,12 +1534,42 @@ static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p,
 			(scale * per_cpu(sd_llc_size, cpu)));
 }
 
+static bool task_llc_affinity_counted(struct rq *rq, struct task_struct *p,
+				      int pref_llc)
+{
+	u64 range;
+	int pref_cpu;
+
+	if (pref_llc < 0)
+		return false;
+
+	/*
+	 * LLCs inside the estimated range are equivalent homes. Do not add
+	 * an anchor-directed affinity score while a task is already inside
+	 * that range; retain the anchor preference outside it so load balance
+	 * can still pull an escaped task back.
+	 */
+	pref_cpu = p->mm ? READ_ONCE(p->mm->sc_stat.cpu) : -1;
+	if (pref_cpu >= 0 && llc_id(pref_cpu) == pref_llc) {
+		range = READ_ONCE(p->mm->sc_stat.llc_range);
+		/* A single-LLC range still needs the original anchor affinity. */
+		if (range != MM_LLC_RANGE_INVALID &&
+		    mm_llc_range_cpu(range) == pref_cpu &&
+		    mm_llc_range_dist(range) > 0 &&
+		    mm_llc_in_estimated_range(p->mm, pref_cpu, cpu_of(rq)))
+			return false;
+	}
+
+	return true;
+}
+
 static void account_llc_enqueue(struct rq *rq, struct task_struct *p)
 {
 	int pref_llc, pref_llc_queued;
 	struct sched_domain *sd;
 
 	pref_llc = p->preferred_llc;
+	p->pref_llc_counted = 0;
 	if (pref_llc < 0)
 		return;
 
@@ -1529,12 +1594,14 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p)
 	p->pref_llc_queued = pref_llc_queued;
 
 	sd = rcu_dereference_all(rq->sd);
-	if (sd && (unsigned int)pref_llc < sd->llc_max) {
+	if (sd && task_llc_affinity_counted(rq, p, pref_llc) &&
+	    (unsigned int)pref_llc < sd->llc_max) {
 		int pref_numa = llc_to_node(pref_llc);
 		sd->llc_counts[pref_llc]++;
 		if (sd->numa_counts &&
 		    (unsigned int)pref_numa < nr_node_ids)
 			sd->numa_counts[pref_numa]++;
+		p->pref_llc_counted = 1;
 	}
 }
 
@@ -1542,8 +1609,11 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
 {
 	struct sched_domain *sd;
 	int pref_llc;
+	int pref_llc_counted;
 
 	pref_llc = p->preferred_llc;
+	pref_llc_counted = p->pref_llc_counted;
+	p->pref_llc_counted = 0;
 	if (pref_llc < 0)
 		return;
 
@@ -1559,7 +1629,8 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
 	}
 
 	sd = rcu_dereference_all(rq->sd);
-	if (sd && (unsigned int)pref_llc < sd->llc_max) {
+	if (sd && pref_llc_counted &&
+	    (unsigned int)pref_llc < sd->llc_max) {
 		/*
 		 * There is a race condition between dequeue
 		 * and CPU hotplug. After a task has been enqueued
@@ -1605,6 +1676,7 @@ void mm_init_sched(struct mm_struct *mm,
 	mm->sc_stat.next_scan = jiffies;
 	mm->sc_stat.nr_running_avg = 0;
 	mm->sc_stat.util_avg = 0;
+	mm->sc_stat.llc_range = MM_LLC_RANGE_INVALID;
 	mm->sc_stat.footprint = 0;
 	mm->sc_stat.node_epoch = kcalloc(num_possible_nodes(),
 					 sizeof(*mm->sc_stat.node_epoch),
@@ -1695,6 +1767,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	struct sched_cache_time *pcpu_sched;
 	struct mm_struct *mm = p->mm;
 	unsigned long *node_epoch;
+	bool pref_llc_counted;
 	int mm_sched_llc = -1;
 	int node, pref_nid;
 	unsigned long epoch;
@@ -1734,6 +1807,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	    exceed_llc_capacity(mm, cpu_of(rq))) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
+		WRITE_ONCE(mm->sc_stat.llc_range, MM_LLC_RANGE_INVALID);
 
 		if (node_epoch)
 			memset(mm->sc_stat.node_epoch, 0,
@@ -1747,10 +1821,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	}
 
 	mm_sched_llc = get_pref_llc(p, mm);
+	pref_llc_counted = task_llc_affinity_counted(rq, p, mm_sched_llc);
 
 	/* task not on rq accounted later in account_entity_enqueue() */
 	if (task_running_on_cpu(rq->cpu, p)) {
-		if (READ_ONCE(p->preferred_llc) != mm_sched_llc) {
+		if (READ_ONCE(p->preferred_llc) != mm_sched_llc ||
+		    READ_ONCE(p->pref_llc_counted) != pref_llc_counted) {
 			account_llc_dequeue(rq, p);
 			WRITE_ONCE(p->preferred_llc, mm_sched_llc);
 			account_llc_enqueue(rq, p);
@@ -1918,6 +1994,7 @@ static void task_cache_work(struct callback_head *work)
 	    exceed_llc_capacity(mm, curr_cpu)) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
+		WRITE_ONCE(mm->sc_stat.llc_range, MM_LLC_RANGE_INVALID);
 
 		return;
 	}
@@ -2038,11 +2115,34 @@ static void task_cache_work(struct callback_head *work)
 	else if (pref_llc_cpu >= 0 && pref_llc_occ > 2 * curr_m_a_occ)
 		new_cpu = pref_llc_cpu;
 
+	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
+	update_mm_util_avg(&mm->sc_stat.util_avg, group_util);
+
+	/*
+	 * The estimated LLC range is consumed from the load-balance hot path.
+	 * Compute its distance frontier here, where a topology walk is cheap,
+	 * and publish the anchor and frontier in one 64-bit value so readers
+	 * never combine values from different preference updates.
+	 */
+	pref_cpu = new_cpu >= 0 ? new_cpu : READ_ONCE(mm->sc_stat.cpu);
+	if (pref_cpu >= 0) {
+		int range_dist = mm_estimated_range_dist(pref_cpu,
+					READ_ONCE(mm->sc_stat.util_avg),
+					READ_ONCE(mm->sc_stat.nr_running_avg));
+
+		if (range_dist >= 0)
+			WRITE_ONCE(mm->sc_stat.llc_range,
+				   mm_llc_range_pack(pref_cpu, range_dist));
+		else
+			WRITE_ONCE(mm->sc_stat.llc_range,
+				   MM_LLC_RANGE_INVALID);
+	} else {
+		WRITE_ONCE(mm->sc_stat.llc_range, MM_LLC_RANGE_INVALID);
+	}
+
 	if (new_cpu >= 0)
 		WRITE_ONCE(mm->sc_stat.cpu, new_cpu);
 
-	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
-	update_mm_util_avg(&mm->sc_stat.util_avg, group_util);
 	free_cpumask_var(cpus);
 }
 
@@ -2057,6 +2157,7 @@ void init_sched_mm(struct task_struct *p)
 	 * polluting account_llc_enqueue().
 	 */
 	p->preferred_llc = -1;
+	p->pref_llc_counted = 0;
 }
 
 #else /* CONFIG_SCHED_CACHE */
@@ -2068,8 +2169,7 @@ void init_sched_mm(struct task_struct *p) { }
 
 static void task_tick_cache(struct rq *rq, struct task_struct *p) { }
 
-static inline int get_pref_llc(struct task_struct *p,
-			       struct mm_struct *mm)
+static inline int get_pref_llc(struct task_struct *p, struct mm_struct *mm)
 {
 	return -1;
 }
@@ -10526,6 +10626,16 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_
 }
 
 #ifdef CONFIG_SCHED_CACHE
+static u32 llc_aggr_capacity_pct(void)
+{
+	u32 aggr_pct = READ_ONCE(llc_overaggr_pct);
+
+	if (cpu_smt_num_threads == 1)
+		aggr_pct = aggr_pct * 3 / 2;
+
+	return aggr_pct;
+}
+
 /*
  * The margin used when comparing LLC utilization with CPU capacity.
  * It determines the LLC load level where active LLC aggregation is
@@ -10536,16 +10646,18 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_
  */
 static bool fits_llc_capacity(unsigned long util, unsigned long max)
 {
-	u32 aggr_pct = llc_overaggr_pct;
+	return util * 100 < max * llc_aggr_capacity_pct();
+}
 
-	/*
-	 * For single core systems, raise the aggregation
-	 * threshold to accommodate more tasks.
-	 */
-	if (cpu_smt_num_threads == 1)
-		aggr_pct = (aggr_pct * 3 / 2);
+/* The estimated range may be filled exactly to the aggregation margin. */
+static bool fits_llc_range_capacity(unsigned long util, unsigned long max)
+{
+	return util * 100 <= max * llc_aggr_capacity_pct();
+}
 
-	return util * 100 < max * aggr_pct;
+static bool fits_llc_full_capacity(unsigned long util, unsigned long max)
+{
+	return util <= max;
 }
 
 /*
@@ -10668,6 +10780,143 @@ static bool get_span_stats(const struct cpumask *span, unsigned long *util_out,
 	return true;
 }
 
+/*
+ * Return the cache distance from the mm's preferred CPU to @cpu. LLCs in
+ * the preferred NUMA node are ordered relative to the preferred LLC. For a
+ * remote node, start at the node distance and add the target LLC's position
+ * in that node's cached LLC order. This makes the first LLC in a remote node
+ * the nearest one and opens the remaining LLCs one at a time.
+ */
+static int mm_cache_distance(int pref_cpu, int cpu)
+{
+	int pref_node = cpu_to_node(pref_cpu);
+	int node = cpu_to_node(cpu);
+	int anchor_llc, node_dist, offset;
+
+	if (pref_node == node)
+		return llc_intra_node_distance(llc_id(pref_cpu), llc_id(cpu),
+					       pref_node);
+
+	/*
+	 * for_each_llc_node_span() orders a remote node from its first LLC.
+	 * Reuse that anchor here. From the first LLC, the intra-node distance
+	 * is 0 for itself and local_rank + 1 for every other LLC, so removing
+	 * the non-zero bias gives the required zero-based offset.
+	 */
+	if (!llc_node_span_by_dist(node, 0, &anchor_llc))
+		return -1;
+
+	offset = llc_intra_node_distance(anchor_llc, llc_id(cpu), node);
+	if (offset < 0)
+		return -1;
+	if (offset)
+		offset--;
+
+	node_dist = sched_cache_node_distance(pref_node, node);
+	if (node_dist < 0)
+		return -1;
+
+	/*
+	 * De-duplicated node distances are only guaranteed to differ by one,
+	 * while @offset spans the node's LLC count. Adding them directly lets
+	 * two nodes share distance values, so a single tier would open LLCs
+	 * from several nodes at once instead of filling the nearest node
+	 * first. Scale the node distance past the widest node to keep the
+	 * per-node ranges disjoint.
+	 */
+	return node_dist * READ_ONCE(llc_node_stride) + offset;
+}
+
+/*
+ * Convert the mm utilization estimate into the furthest cache-distance tier
+ * it needs. This runs from task_cache_work(), rather than can_migrate_node(),
+ * so load balance does not walk every LLC for every migration candidate.
+ *
+ * Use an inclusive capacity boundary: an mm whose utilization is exactly 50%
+ * of an LLC fits that LLC and does not require the next tier. If the estimate
+ * exceeds the configured budget of the whole machine, return the last tier so
+ * every LLC remains eligible.
+ */
+static int mm_estimated_range_dist(int pref_cpu, u64 group_util,
+				   u64 nr_running)
+{
+	const struct cpumask *span;
+	u64 budget = 0;
+	unsigned long util, cap;
+	int frontier = -1, last_dist = -1, node;
+
+	if (!group_util)
+		return -1;
+
+	/*
+	 * Build the range one cache-distance tier at a time.  The topology
+	 * iterators order nodes relative to @pref_cpu, but LLCs within each
+	 * node are cached relative to that node's first LLC.  Consequently,
+	 * their nested iteration order is not necessarily monotonic from the
+	 * mm's actual preferred LLC.  Returning the distance of the first LLC
+	 * visited could therefore make a one-LLC estimate cover several LLCs.
+	 *
+	 * A distance is also a range boundary, so all LLCs at an equal distance
+	 * become eligible together.  Account their capacity together as well;
+	 * otherwise one remote LLC's capacity could open its whole distance
+	 * tier.
+	 */
+	for (;;) {
+		u64 tier_budget = 0, tier_capacity = 0;
+		int next_dist = INT_MAX;
+
+		for_each_sched_node(pref_cpu, node) {
+			for_each_llc_node_span(node, span) {
+				int cpu = cpumask_first(span);
+				int dist = mm_cache_distance(pref_cpu, cpu);
+
+				if (dist < 0)
+					return -1;
+				if (dist > frontier && dist < next_dist)
+					next_dist = dist;
+			}
+		}
+
+		if (next_dist == INT_MAX)
+			break;
+
+		for_each_sched_node(pref_cpu, node) {
+			for_each_llc_node_span(node, span) {
+				int cpu = cpumask_first(span);
+
+				if (mm_cache_distance(pref_cpu, cpu) != next_dist)
+					continue;
+				if (!get_llc_stats(cpu, &util, &cap))
+					return -1;
+
+				tier_capacity += cap;
+				tier_budget += mul_u64_u32_div(cap,
+						llc_aggr_capacity_pct(), 100);
+			}
+		}
+
+		/*
+		 * A workload whose active parallelism and utilization both fit a
+		 * single LLC does not oversubscribe it.  Keep the original single
+		 * anchor in that case: spreading communicating threads merely because
+		 * they cross the aggregation margin loses locality without creating
+		 * CPU capacity.  Larger workloads still grow at the configured margin.
+		 */
+		if (next_dist == 0 && group_util <= tier_capacity &&
+		    nr_running * SCHED_CAPACITY_SCALE <= tier_capacity)
+			return 0;
+
+		budget += tier_budget;
+		last_dist = next_dist;
+		if (group_util <= budget)
+			return next_dist;
+
+		frontier = next_dist;
+	}
+
+	return last_dist;
+}
+
 /*
  * Decide if migration should happen on a specific node.
  * The node here is an LLC or a NUMA.
@@ -10676,11 +10925,12 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 			struct task_struct *p, bool to_pref)
 {
 	const struct cpumask *span;
-	struct mm_struct *mm;
+	struct mm_struct *mm = NULL;
+	bool single_llc_range = false;
 	unsigned long dst_util, dst_cap, tsk_util = 0;
 	unsigned long src_util = 0, src_cap = 0;
 	unsigned long acc_util = 0, acc_cap = 0, dst_pre;
-	int node, target_cpu = src_cpu;
+	int node, target_cpu = src_cpu, mm_cpu = -1;
 	int get_src = 0;
 
 	if (!get_llc_stats(dst_cpu, &dst_util, &dst_cap))
@@ -10692,8 +10942,16 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 	if (p) {
 		mm = p->mm;
 		if (mm) {
-			if (mm->sc_stat.cpu >= 0)
-				target_cpu = mm->sc_stat.cpu;
+			mm_cpu = READ_ONCE(mm->sc_stat.cpu);
+			if (mm_cpu >= 0) {
+				u64 range = READ_ONCE(mm->sc_stat.llc_range);
+
+				target_cpu = mm_cpu;
+				single_llc_range =
+					range != MM_LLC_RANGE_INVALID &&
+					mm_llc_range_cpu(range) == mm_cpu &&
+					mm_llc_range_dist(range) == 0;
+			}
 		}
 		tsk_util = task_util(p);
 	}
@@ -10701,7 +10959,27 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 	dst_util = dst_util + tsk_util;
 	dst_pre = dst_util - tsk_util;
 
+	/*
+	 * Do not let a nearer LLC that can hold one task veto a destination
+	 * which the mm's aggregate demand is expected to need. The range check
+	 * says the destination belongs to the area needed by the whole mm. Let
+	 * it fill exactly to the aggregation margin; the ordinary strict check
+	 * would reject that boundary and strand runnable tasks on busier LLCs.
+	 */
+	if (!to_pref && mm_cpu >= 0 &&
+	    mm_llc_in_estimated_range(mm, target_cpu, dst_cpu) &&
+	    fits_llc_range_capacity(dst_util, dst_cap))
+		return mig_llc;
+
 	if (to_pref) {
+		/*
+		 * A single-LLC range means the mm fits without oversubscribing
+		 * the anchor.  Preserve that locality up to the LLC's full
+		 * capacity instead of applying the aggregation margin.
+		 */
+		if (single_llc_range && cpus_share_cache(dst_cpu, target_cpu) &&
+		    fits_llc_full_capacity(dst_util, dst_cap))
+			return mig_llc;
 
 		if (fits_llc_capacity(dst_util, dst_cap))
 			return mig_llc;
@@ -10727,6 +11005,11 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 		return mig_forbid;
 	}
 
+	/* Do not split a fitting single-LLC mm merely at the aggr margin. */
+	if (single_llc_range && cpus_share_cache(src_cpu, target_cpu) &&
+	    src_cap && fits_llc_full_capacity(src_util, src_cap))
+		return mig_forbid;
+
 	for_each_sched_node(target_cpu, node) {
 		unsigned long u = 0, c = 0, nu, nc;
 
@@ -10861,6 +11144,7 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,
 	    exceed_llc_capacity(mm, dst_cpu)) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
+		WRITE_ONCE(mm->sc_stat.llc_range, MM_LLC_RANGE_INVALID);
 		return mig_unrestricted;
 	}
 
@@ -13601,12 +13885,12 @@ static inline bool llc_spread_active_balance(struct lb_env *env)
 		return false;
 
 	/*
-	 * Only move when the destination still fits with a full CPU of the load
-	 * added, and when the source leads by at least two CPUs worth, so that
-	 * moving one task cannot invert the imbalance and bounce it straight
-	 * back.
+	 * Only move when the destination still fits with a full CPU of load
+	 * added. The estimated range deliberately permits the exact aggregation
+	 * boundary, otherwise a three-task LLC cannot pull the fourth task needed
+	 * to balance a four-task-per-LLC workload.
 	 */
-	if (!fits_llc_capacity(dst_util + SCHED_CAPACITY_SCALE, dst_cap))
+	if (!fits_llc_range_capacity(dst_util + SCHED_CAPACITY_SCALE, dst_cap))
 		return false;
 
 	/* Avoid threads ping-pong migration */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3d3f1d40a672..e13775e9b177 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4125,6 +4125,7 @@ extern int sched_node_order_at(int cpu, int idx);
 	     ((node) = sched_node_order_at((cpu), __sn_idx)) >= 0;	\
 	     __sn_idx++)
 
+extern int llc_node_stride;
 extern int llc_node_count(int node);
 extern const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out);
 
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 73d0c44dea92..13d360070679 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -722,6 +722,13 @@ struct llc_local_rank_topology {
 };
 
 static struct llc_local_rank_topology __rcu *llc_local_rank_topo;
+
+/*
+ * Largest number of LLCs held by any single NUMA node. mm_cache_distance()
+ * multiplies the node distance by this so that one node's per-LLC offsets
+ * cannot reach into the next node's range.
+ */
+int llc_node_stride __read_mostly = 1;
 static void rebuild_llc_local_rank(int size);
 #endif
 
@@ -1375,7 +1382,7 @@ static void free_llc_local_rank_topology(struct llc_local_rank_topology *topo)
 static void rebuild_llc_local_rank(int size)
 {
 	struct llc_local_rank_topology *topo, *old;
-	int llc, node;
+	int llc, node, stride;
 
 	if (size <= 0)
 		return;
@@ -1400,6 +1407,11 @@ static void rebuild_llc_local_rank(int size)
 		topo->rank[llc] = topo->node_count[node]++;
 	}
 
+	stride = 1;
+	for (node = 0; node < nr_node_ids; node++)
+		stride = max(stride, topo->node_count[node]);
+	WRITE_ONCE(llc_node_stride, stride);
+
 	old = rcu_dereference_protected(llc_local_rank_topo,
 					 lockdep_is_held(&sched_domains_mutex));
 	rcu_assign_pointer(llc_local_rank_topo, topo);
-- 
2.34.1

[RFC PATCH v2 22/23] sched/cache: Walk the preferred node from the preferred LLC

[Jianyong Wu]

for_each_llc_node_span() hands out a node's LLCs ordered by
llc_intra_node_distance() from that node's first LLC. That anchor is the
right one for every LLC outside the node, which all see the same order,
but not for the node the mm's preferred LLC lives in.

llc_intra_node_distance(a, b) is (rank_a + rank_b) % k + 1, so the order
seen from a rank-r anchor is a rotation of the order seen from rank 0.
With four LLCs per node and the anchor at rank 2, the canonical walk
yields distances 3, 4, 0, 2 instead of 0, 2, 3, 4. Callers that accumulate
while walking therefore treat farther LLCs as nearer ones.

Add llc_node_span_from() and for_each_llc_node_span_from(), which take the
anchor as an argument and undo the rotation, and use them wherever a
preferred LLC is already known. Nodes the anchor does not belong to keep
the canonical order, so only the preferred node's walk changes.

Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/fair.c     |  8 ++---
 kernel/sched/sched.h    | 17 +++++++++++
 kernel/sched/topology.c | 68 +++++++++++++++++++++++++++++++++++++----
 3 files changed, 83 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index def42490fa2a..21513e3dd65a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10866,7 +10866,7 @@ static int mm_estimated_range_dist(int pref_cpu, u64 group_util,
 		int next_dist = INT_MAX;
 
 		for_each_sched_node(pref_cpu, node) {
-			for_each_llc_node_span(node, span) {
+			for_each_llc_node_span_from(node, llc_id(pref_cpu), span) {
 				int cpu = cpumask_first(span);
 				int dist = mm_cache_distance(pref_cpu, cpu);
 
@@ -10881,7 +10881,7 @@ static int mm_estimated_range_dist(int pref_cpu, u64 group_util,
 			break;
 
 		for_each_sched_node(pref_cpu, node) {
-			for_each_llc_node_span(node, span) {
+			for_each_llc_node_span_from(node, llc_id(pref_cpu), span) {
 				int cpu = cpumask_first(span);
 
 				if (mm_cache_distance(pref_cpu, cpu) != next_dist)
@@ -11030,7 +11030,7 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 		if (cpumask_test_cpu(dst_cpu, cpumask_of_node(node))) {
 			nu = 0;
 			nc = 0;
-			for_each_llc_node_span(node, span) {
+			for_each_llc_node_span_from(node, llc_id(target_cpu), span) {
 				get_span_stats(span, &u, &c);
 				nu += u;
 				nc += c;
@@ -11082,7 +11082,7 @@ static enum llc_mig can_migrate_node(int src_cpu, int dst_cpu,
 		}
 
 		/* Don't migrate if this is a good place to live. */
-		for_each_llc_node_span(node, span) {
+		for_each_llc_node_span_from(node, llc_id(target_cpu), span) {
 			get_span_stats(span, &u, &c);
 			if (cpumask_test_cpu(src_cpu, span)) {
 				if (fits_llc_capacity(u, c))
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e13775e9b177..409e889d660f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4128,6 +4128,8 @@ extern int sched_node_order_at(int cpu, int idx);
 extern int llc_node_stride;
 extern int llc_node_count(int node);
 extern const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out);
+extern const struct cpumask *llc_node_span_from(int node, int anchor_llc,
+						int index, int *llc_out);
 
 /*
  * Walk each individual LLC belonging to NUMA node @node, nearest-first
@@ -4140,6 +4142,21 @@ extern const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc
 	     ((span) = llc_node_span_by_dist((node), __lns_idx, NULL)) != NULL; \
 	     __lns_idx++)
 
+/*
+ * Same walk as for_each_llc_node_span(), but ordered from @anchor_llc when
+ * that LLC lives in @node. Callers that already have a preferred LLC should
+ * use this so the walk starts at the anchor instead of the node's first LLC;
+ * the two orders differ because llc_intra_node_distance() wraps modulo the
+ * node's LLC count. For any node the anchor does not belong to, this falls
+ * back to the canonical order.
+ */
+#define for_each_llc_node_span_from(node, anchor_llc, span)			\
+	for (int __lns_idx = 0, __lns_nr = llc_node_count((node));		\
+	     __lns_idx < __lns_nr &&						\
+	     ((span) = llc_node_span_from((node), (anchor_llc), __lns_idx,	\
+					  NULL)) != NULL;			\
+	     __lns_idx++)
+
 #endif
 
 void sched_domains_free_llc_id(int cpu);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 13d360070679..a0c3ec1278a6 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1022,6 +1022,25 @@ static int llc_local_rank_node_count(int node)
 	return count;
 }
 
+/*
+ * 0-based rank of @llc within its own NUMA node, -1 when unknown.
+ * Ranks are handed out in ascending LLC id order, so the node's first
+ * LLC always has rank 0.
+ */
+static int llc_local_rank(int llc)
+{
+	struct llc_local_rank_topology *topo;
+	int rank = -1;
+
+	rcu_read_lock();
+	topo = rcu_dereference(llc_local_rank_topo);
+	if (topo && llc >= 0 && llc < topo->nr_llc)
+		rank = topo->rank[llc];
+	rcu_read_unlock();
+
+	return rank;
+}
+
 static void rebuild_llc_node_map(int size)
 {
 	int *new_map, *old_map;
@@ -2854,25 +2873,62 @@ int llc_node_count(int node)
 
 /*
  * Return a pointer to the cached cpumask of the LLC at position @index
- * within NUMA node @node, ordered ascending by
- * llc_intra_node_distance() from that node's lowest-id LLC.
+ * within NUMA node @node, ordered ascending by llc_intra_node_distance()
+ * from @anchor_llc.
+ *
+ * @anchor_llc is only honoured when it belongs to @node. Pass -1, or an LLC
+ * from any other node, to get the canonical order anchored at the node's own
+ * first LLC - the order every LLC outside the node agrees on.
  */
-const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out)
+const struct cpumask *llc_node_span_from(int node, int anchor_llc, int index,
+					 int *llc_out)
 {
 	struct llc_aggr_topology *topo = rcu_dereference_all(llc_aggr_topo);
-	int llc;
+	int base, cnt, rank, self_slot, slot, llc;
 
 	if (!topo || node < 0 || node >= topo->nr_node)
 		return NULL;
-	if (index < 0 || index >= llc_local_rank_node_count(node))
+
+	cnt = llc_local_rank_node_count(node);
+	if (index < 0 || index >= cnt)
 		return NULL;
 
-	llc = topo->node_llc_by_dist[topo->node_offset[node] + index];
+	base = topo->node_offset[node];
+
+	if (anchor_llc < 0 || llc_to_node(anchor_llc) != node) {
+		llc = topo->node_llc_by_dist[base + index];
+	} else if (!index) {
+		llc = anchor_llc;		/* distance 0 */
+	} else {
+		rank = llc_local_rank(anchor_llc);
+		if (rank < 0)
+			return NULL;
+
+		/*
+		 * llc_intra_node_distance(anchor, x) is
+		 * (rank + rank_x) % cnt + 1, so walking that modular sum
+		 * ascending walks the distances ascending. The anchor itself
+		 * owns slot (2 * rank) % cnt and was already returned at
+		 * index 0, so step over it.
+		 */
+		self_slot = (2 * rank) % cnt;
+		slot = index - 1;
+		if (slot >= self_slot)
+			slot++;
+
+		llc = topo->node_llc[base + (slot - rank + cnt) % cnt];
+	}
+
 	if (llc_out)
 		*llc_out = llc;
 
 	return topo->llc_span[llc];
 }
+
+const struct cpumask *llc_node_span_by_dist(int node, int index, int *llc_out)
+{
+	return llc_node_span_from(node, -1, index, llc_out);
+}
 #endif /* CONFIG_SCHED_CACHE */
 
 const struct cpumask *tl_pkg_mask(struct sched_domain_topology_level *tl, int cpu)
-- 
2.34.1

[RFC PATCH v2 23/23] sched/debug: Print task preferred LLC for scheduler debugging

[Jianyong Wu]

Expose each task's preferred LLC to aid diagnosis of cache-aware
scheduling decisions.

Suggested-by: XIAO WU <xiaowu.417@qq.com>
Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
 kernel/sched/debug.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 40584b27ea0c..d18dbc1da67c 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -9,7 +9,9 @@
 #include <linux/debugfs.h>
 #include <linux/nmi.h>
 #include <linux/log2.h>
+#include <linux/sched/clock.h>
 #include "sched.h"
+#include <linux/sched/debug.h>
 
 /*
  * This allows printing both to /sys/kernel/debug/sched/debug and
@@ -1304,7 +1306,6 @@ void print_numa_stats(struct seq_file *m, int node, unsigned long tsf,
 }
 #endif
 
-
 static void sched_show_numa(struct task_struct *p, struct seq_file *m)
 {
 #ifdef CONFIG_NUMA_BALANCING
@@ -1320,6 +1321,30 @@ static void sched_show_numa(struct task_struct *p, struct seq_file *m)
 #endif /* CONFIG_NUMA_BALANCING */
 }
 
+static void sched_show_cache(struct task_struct *p, struct seq_file *m)
+{
+#ifdef CONFIG_SCHED_CACHE
+	struct mm_struct *mm = NULL;
+	int sc_cpu, sc_llc, sc_node, pref_llc, pref_node;
+
+	mm = get_task_mm(p);
+
+	if (mm) {
+		sc_cpu = READ_ONCE(mm->sc_stat.cpu);
+		sc_llc  = (sc_cpu >= 0) ? per_cpu(sd_llc_id, sc_cpu) : -1;
+		sc_node = (sc_cpu >= 0) ? cpu_to_node(sc_cpu) : -1;
+		pref_llc  = READ_ONCE(p->preferred_llc);
+		pref_node = (pref_llc >= 0) ? llc_to_node(pref_llc) : -1;
+
+		SEQ_printf(m, "sc_stat_cpu=%d, sc_llc=%d, sc_node=%d\n",
+			   sc_cpu, sc_llc, sc_node);
+		SEQ_printf(m, "preferred_llc=%d, preferred_llc_node=%d\n",
+			   pref_llc, pref_node);
+		mmput(mm);
+	}
+#endif /* CONFIG_SCHED_CACHE */
+}
+
 void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 						  struct seq_file *m)
 {
@@ -1439,6 +1464,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 	}
 
 	sched_show_numa(p, m);
+	sched_show_cache(p, m);
 }
 
 void proc_sched_set_task(struct task_struct *p)
-- 
2.34.1

Re: [RFC PATCH v2 01/23] sched/topology: Add llc_to_node() to translate LLC id to NUMA node

[Peter Zijlstra]

On Thu, Aug 27, 2026 at 08:27:54PM +0800, Jianyong Wu wrote:
> +static void rebuild_llc_node_map(int size)
> +{
> +	int *new_map, *old_map;
> +	u8 *seen_llc;
> +	int cpu, llc;
> +
> +	new_map = kcalloc(size, sizeof(int), GFP_KERNEL);
> +	if (!new_map)
> +		return;
> +	seen_llc = kcalloc(size, sizeof(*seen_llc), GFP_KERNEL);
> +	if (!seen_llc) {
> +		kfree(new_map);
> +		return;
> +	}
> +
> +	/*
> +	 * for_each_possible_cpu() revisits the same LLC non-consecutively
> +	 * under SMT (each node's LLCs are walked once per thread), so
> +	 * dedup by llc id via seen_llc[], not by comparing against the
> +	 * immediately preceding CPU's llc.
> +	 */
> +	for_each_possible_cpu(cpu) {
> +		llc = per_cpu(sd_llc_id, cpu);
> +		if (llc < 0 || llc >= size || seen_llc[llc])
> +			continue;
> +		seen_llc[llc] = 1;
> +		new_map[llc] = cpu_to_node(cpu);
> +	}
> +	kfree(seen_llc);
> +
> +	old_map = rcu_dereference_protected(llc_to_node_map, true);
> +	rcu_assign_pointer(llc_to_node_map, new_map);
> +	synchronize_rcu();
> +	kfree(old_map);

Could that not be: kfree_rcu_mightsleep(old_map); ?