[PATCH v4 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work

[PATCH v4 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work

[Luo Gengkun]

Hi everyone,
The cache-aware scheduling patches have now been merged into the mainline.
The goal of this patch is to reduce the overhead of task_cache_work() by
minimizing the number of scanned CPUs, which yields significant
performance gains in multi-instance scenarios like Redis. To facilitate
testing, this debug patch is introduced. Below are the benchmark results
based on hackbench:

Test steps:
echo NO_SC_VISIT > /sys/kernel/debug/sched/features
echo NO_SC_NODE > /sys/kernel/debug/sched/features
echo 0 > /sys/kernel/debug/sched/llc_balancing/enabled
./launch.sh hackbench baseline

echo NO_SC_VISIT > /sys/kernel/debug/sched/features
echo SC_NODE > /sys/kernel/debug/sched/features
echo 1 > /sys/kernel/debug/sched/llc_balancing/enabled
./launch.sh hackbench schedcache
  
echo SC_VISIT > /sys/kernel/debug/sched/features
echo NO_SC_NODE > /sys/kernel/debug/sched/features
echo 1 > /sys/kernel/debug/sched/llc_balancing/enabled
./launch.sh hackbench schedcache_visit

Test results:
./launch.sh compare hackbench baseline schedcache
=========================================
Hackbench Comparison: baseline vs schedcache
=========================================
MODE       GROUPS  FDS |     baseline(std)  |  schedcache(std)   |    DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
threads         1   10 |    113.200 (4.22%) |     67.300 (1.32%) |     40.55% | IMPROVED
threads         1    2 |     16.555 (4.11%) |     11.020 (1.66%) |     33.43% | IMPROVED
threads         1   20 |    250.774 (1.26%) |    265.026 (5.44%) |     -5.68% | REGRESSED
threads         1    4 |     42.117 (1.44%) |     27.758 (1.64%) |     34.09% | IMPROVED
threads         1    6 |     65.140 (4.31%) |     39.182 (1.38%) |     39.85% | IMPROVED
threads         1    8 |     84.286 (1.29%) |     53.721 (1.58%) |     36.26% | IMPROVED
threads         2   10 |    122.592 (0.44%) |    113.365 (4.93%) |      7.53% | IMPROVED
threads         2    2 |     17.702 (4.09%) |     10.473 (0.42%) |     40.84% | IMPROVED
threads         2   20 |    336.457 (1.77%) |    314.108 (1.51%) |      6.64% | IMPROVED
threads         2    4 |     43.989 (0.88%) |     27.067 (3.38%) |     38.47% | IMPROVED
threads         2    6 |     69.322 (0.85%) |     41.707 (4.19%) |     39.84% | IMPROVED
threads         2    8 |    103.767 (1.81%) |     58.518 (3.00%) |     43.61% | IMPROVED
threads         4   10 |    148.882 (3.56%) |    149.449 (1.06%) |     -0.38% | REGRESSED
threads         4    2 |     18.909 (2.96%) |     11.063 (2.08%) |     41.49% | IMPROVED
threads         4   20 |    724.943 (2.14%) |    631.222 (3.92%) |     12.93% | IMPROVED
threads         4    4 |     48.191 (1.91%) |     27.352 (5.35%) |     43.24% | IMPROVED
threads         4    6 |     79.725 (3.84%) |     78.732 (4.10%) |      1.25% | IMPROVED
threads         4    8 |    108.768 (1.36%) |    105.928 (1.65%) |      2.61% | IMPROVED
     
Hackbench Comparison: schedcache vs schedcache_visit
=========================================
MODE       GROUPS  FDS |    schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+----------------------+---------+-----------
threads         1   10 |     67.300 (1.32%) |     67.014 (0.96%)   |   0.42% | IMPROVED
threads         1    2 |     11.020 (1.66%) |     10.557 (1.46%)   |   4.20% | IMPROVED
threads         1   20 |    265.026 (5.44%) |   212.366 (16.32%)   |  19.87% | IMPROVED
threads         1    4 |     27.758 (1.64%) |     25.711 (1.32%)   |   7.37% | IMPROVED
threads         1    6 |     39.182 (1.38%) |     38.914 (0.34%)   |   0.68% | IMPROVED
threads         1    8 |     53.721 (1.58%) |     52.889 (0.27%)   |   1.55% | IMPROVED
threads         2   10 |    121.203 (6.99%) |    124.254 (1.38%)   |  -2.52% | REGRESSED
threads         2    2 |     10.473 (0.42%) |     11.206 (5.91%)   |  -7.00% | REGRESSED
threads         2   20 |    314.108 (1.51%) |    301.754 (1.95%)   |   3.93% | IMPROVED
threads         2    4 |     27.067 (3.38%) |     28.028 (2.01%)   |  -3.55% | REGRESSED
threads         2    6 |     41.707 (4.19%) |     42.149 (3.35%)   |  -1.06% | REGRESSED
threads         2    8 |     58.518 (3.00%) |     57.133 (4.39%)   |   2.37% | IMPROVED
threads         4   10 |    149.449 (1.06%) |    141.407 (0.08%)   |   5.38% | IMPROVED
threads         4    2 |     11.063 (2.08%) |     11.360 (5.85%)   |  -2.68% | REGRESSED
threads         4   20 |    631.222 (3.92%) |    622.780 (2.49%)   |   1.34% | IMPROVED
threads         4    4 |     27.352 (5.35%) |     27.947 (5.37%)   |  -2.18% | REGRESSED
threads         4    6 |     78.732 (4.10%) |     73.911 (0.70%)   |   6.12% | IMPROVED
threads         4    8 |    105.928 (1.65%) |    107.535 (3.29%)   |  -1.52% | REGRESSED 

---
Changes history
**v4 Changes:**
1. Rebase to the master.
2. epoch_timeout is introduced to evict expired CPUs instead of relying on
epoch, because epoch is refreshed periodically due to invocations of
fraction_mm_sched().
3. Move the increasement of nr_running before fraction_mm_sched().
4. Remove the redundant 'work->next' reset at the end of task_cache_work().
4. Add a debug patch to show the number of CPUs scanned to show the
benefit of this optimization.

Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/

**v3 Changes:**
1. Remove the static key and enable this feature by default.
2. Reuse llc_epoch_affinity_timeout instead of introducing
llc_epoch_visited_timeout.
3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into
fraction_mm_sched() to avoid race between task_cache_work() and
__update_mm_sched(). 
4. Reset work->next at the end of task_cache_work() to prevent concurrent
executions by multiple threads within the same process.

Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/

**v2 Changes:**
1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead.
2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead.
---

Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/
---

Luo Gengkun (2):
  sched/cache: Reduce the overhead of task_cache_work by only scan the
    visisted cpus
  -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched
    feature to track scan cost

 include/linux/sched.h        |  2 ++
 include/trace/events/sched.h | 21 ++++++++++++++++
 kernel/sched/fair.c          | 46 +++++++++++++++++++++++++++++-------
 kernel/sched/features.h      |  2 ++
 4 files changed, 62 insertions(+), 9 deletions(-)

-- 
2.34.1

[PATCH v4 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Luo Gengkun]

The overhead of task_cache_work() is high, especially in multi-NUMA
systems. Currently, task_cache_work() tries to find the pref_llc by
scanning all CPUs in the system. However, most of these scans are
meaningless, such as those for CPUs that have never been visited or were
accessed a long time ago.

To address this problem, introduce visited_cpus to track the visited CPUs
and evict them once they have not been accessed for a duration exceeding
llc_epoch_affinity_timeout. With this patch, get_scan_cpumasks() is no
longer need and is therefore removed.

Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
---
 include/linux/sched.h |  2 ++
 kernel/sched/fair.c   | 81 +++++++++++++++----------------------------
 2 files changed, 29 insertions(+), 54 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index b3204a15d512..9f3946c6bd5d 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2385,6 +2385,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)
 struct sched_cache_time {
 	u64 runtime;
 	unsigned long epoch;
+	unsigned long epoch_timeout;
 };
 
 struct sched_cache_stat {
@@ -2395,6 +2396,7 @@ struct sched_cache_stat {
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
+	struct cpumask visited_cpus;
 } ____cacheline_aligned_in_smp;
 
 #else
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..61f71857e9b1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm,
 		pcpu_sched->runtime = 0;
 		/* a slightly stale cpu epoch is acceptible */
 		pcpu_sched->epoch = rq->cpu_epoch;
+		pcpu_sched->epoch_timeout = rq->cpu_epoch;
 		epoch = rq->cpu_epoch;
 	}
 
@@ -1594,6 +1595,7 @@ void mm_init_sched(struct mm_struct *mm,
 	mm->sc_stat.next_scan = jiffies;
 	mm->sc_stat.nr_running_avg = 0;
 	mm->sc_stat.footprint = 0;
+	cpumask_clear(&mm->sc_stat.visited_cpus);
 	/*
 	 * The update to mm->sc_stat should not be reordered
 	 * before initialization to mm's other fields, in case
@@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq,
 	}
 }
 
-static unsigned long fraction_mm_sched(struct rq *rq,
-				       struct sched_cache_time *pcpu_sched)
+static unsigned long fraction_mm_sched(int cpu,
+				       struct mm_struct *mm)
 {
+	struct sched_cache_time *pcpu_sched =
+		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
+	struct rq *rq = cpu_rq(cpu);
+
 	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
 
+	/* Skip the rq that has not been hit for a long time */
+	if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
+		cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus);
+		return 0;
+	}
+
 	__update_mm_sched(rq, pcpu_sched);
 
 	/*
@@ -1711,6 +1723,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 		pcpu_sched->runtime += delta_exec;
 		rq->cpu_runtime += delta_exec;
 		epoch = rq->cpu_epoch;
+		pcpu_sched->epoch_timeout = epoch;
+		if (!cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus))
+			cpumask_set_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus);
 	}
 
 	/*
@@ -1761,51 +1776,6 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
 	}
 }
 
-static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p)
-{
-#ifdef CONFIG_NUMA_BALANCING
-	int cpu, curr_cpu, nid, pref_nid;
-
-	if (!static_branch_likely(&sched_numa_balancing))
-		goto out;
-
-	cpu = READ_ONCE(p->mm->sc_stat.cpu);
-	if (cpu != -1)
-		nid = cpu_to_node(cpu);
-	curr_cpu = task_cpu(p);
-
-	/*
-	 * Scanning in the preferred NUMA node is ideal. However, the NUMA
-	 * preferred node is per-task rather than per-process. It is possible
-	 * for different threads of the process to have distinct preferred
-	 * nodes; consequently, the process-wide preferred LLC may bounce
-	 * between different nodes. As a workaround, maintain the scan
-	 * CPU mask to also cover the process's current preferred LLC and the
-	 * current running node to mitigate the bouncing risk.
-	 * TBD: numa_group should be considered during task aggregation.
-	 */
-	pref_nid = p->numa_preferred_nid;
-	/* honor the task's preferred node */
-	if (pref_nid == NUMA_NO_NODE)
-		goto out;
-
-	cpumask_or(cpus, cpus, cpumask_of_node(pref_nid));
-
-	/* honor the task's preferred LLC CPU */
-	if (cpu != -1 && !cpumask_test_cpu(cpu, cpus) && nid != NUMA_NO_NODE)
-		cpumask_or(cpus, cpus, cpumask_of_node(nid));
-
-	/* make sure the task's current running node is included */
-	if (!cpumask_test_cpu(curr_cpu, cpus))
-		cpumask_or(cpus, cpus, cpumask_of_node(cpu_to_node(curr_cpu)));
-
-	return;
-
-out:
-#endif
-	cpumask_copy(cpus, cpu_online_mask);
-}
-
 static inline void update_avg_scale(u64 *avg, u64 sample)
 {
 	int factor = per_cpu(sd_llc_size, raw_smp_processor_id());
@@ -1866,7 +1836,7 @@ static void task_cache_work(struct callback_head *work)
 	scoped_guard (cpus_read_lock) {
 		guard(rcu)();
 
-		get_scan_cpumasks(cpus, p);
+		cpumask_and(cpus, cpu_online_mask, &mm->sc_stat.visited_cpus);
 
 		for_each_cpu(cpu, cpus) {
 			/* XXX sched_cluster_active */
@@ -1878,18 +1848,21 @@ static void task_cache_work(struct callback_head *work)
 				continue;
 
 			for_each_cpu(i, sched_domain_span(sd)) {
-				occ = fraction_mm_sched(cpu_rq(i),
-							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
+				cur = rcu_dereference_all(cpu_rq(i)->curr);
+				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
+				    cur->mm == mm)
+					nr_running++;
+
+				occ = fraction_mm_sched(i, mm);
+				if (occ == 0)
+					continue;
+
 				a_occ += occ;
 				if (occ > m_occ) {
 					m_occ = occ;
 					m_cpu = i;
 				}
 
-				cur = rcu_dereference_all(cpu_rq(i)->curr);
-				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
-				    cur->mm == mm)
-					nr_running++;
 			}
 
 			/*
-- 
2.34.1

[PATCH v4 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost

[Luo Gengkun]

To evaluate the effectiveness of the previous patch, this debug patch re-adds
get_scan_cpumasks() and introduces two sched features as well as trace
events to facilitate testing.

Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
---
 include/trace/events/sched.h | 21 +++++++++++++
 kernel/sched/fair.c          | 61 ++++++++++++++++++++++++++++++++++--
 kernel/sched/features.h      |  2 ++
 3 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index 535860581f15..aced624f198d 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -10,6 +10,27 @@
 #include <linux/tracepoint.h>
 #include <linux/binfmts.h>
 
+TRACE_EVENT(sched_cache_scan,
+
+	TP_PROTO(struct task_struct *t, int scan),
+
+	TP_ARGS(t, scan),
+
+	TP_STRUCT__entry(
+		__string(	comm,	t->comm		)
+		__field(	pid_t,	pid		)
+		__field(	int,	scan		)
+	),
+
+	TP_fast_assign(
+		__assign_str(comm);
+		__entry->pid	= t->pid;
+		__entry->scan	= scan;
+	),
+
+	TP_printk("comm=%s pid=%d scan=%d", __get_str(comm), __entry->pid,
+					__entry->scan)
+);
 /*
  * Tracepoint for calling kthread_stop, performed to end a kthread:
  */
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 61f71857e9b1..4016b3291db4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1647,7 +1647,8 @@ static unsigned long fraction_mm_sched(int cpu,
 	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
 
 	/* Skip the rq that has not been hit for a long time */
-	if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
+	if (sched_feat(SC_VISIT) &&
+	    (rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
 		cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus);
 		return 0;
 	}
@@ -1724,7 +1725,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 		rq->cpu_runtime += delta_exec;
 		epoch = rq->cpu_epoch;
 		pcpu_sched->epoch_timeout = epoch;
-		if (!cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus))
+		if (sched_feat(SC_VISIT) &&
+		    !cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus))
 			cpumask_set_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus);
 	}
 
@@ -1776,6 +1778,51 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
 	}
 }
 
+static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p)
+{
+#ifdef CONFIG_NUMA_BALANCING
+	int cpu, curr_cpu, nid, pref_nid;
+
+	if (!static_branch_likely(&sched_numa_balancing))
+		goto out;
+
+	cpu = READ_ONCE(p->mm->sc_stat.cpu);
+	if (cpu != -1)
+		nid = cpu_to_node(cpu);
+	curr_cpu = task_cpu(p);
+
+	/*
+	 * Scanning in the preferred NUMA node is ideal. However, the NUMA
+	 * preferred node is per-task rather than per-process. It is possible
+	 * for different threads of the process to have distinct preferred
+	 * nodes; consequently, the process-wide preferred LLC may bounce
+	 * between different nodes. As a workaround, maintain the scan
+	 * CPU mask to also cover the process's current preferred LLC and the
+	 * current running node to mitigate the bouncing risk.
+	 * TBD: numa_group should be considered during task aggregation.
+	 */
+	pref_nid = p->numa_preferred_nid;
+	/* honor the task's preferred node */
+	if (pref_nid == NUMA_NO_NODE)
+		goto out;
+
+	cpumask_or(cpus, cpus, cpumask_of_node(pref_nid));
+
+	/* honor the task's preferred LLC CPU */
+	if (cpu != -1 && !cpumask_test_cpu(cpu, cpus) && nid != NUMA_NO_NODE)
+		cpumask_or(cpus, cpus, cpumask_of_node(nid));
+
+	/* make sure the task's current running node is included */
+	if (!cpumask_test_cpu(curr_cpu, cpus))
+		cpumask_or(cpus, cpus, cpumask_of_node(cpu_to_node(curr_cpu)));
+
+	return;
+
+out:
+#endif
+	cpumask_copy(cpus, cpu_online_mask);
+}
+
 static inline void update_avg_scale(u64 *avg, u64 sample)
 {
 	int factor = per_cpu(sd_llc_size, raw_smp_processor_id());
@@ -1803,6 +1850,7 @@ static void task_cache_work(struct callback_head *work)
 	struct mm_struct *mm = p->mm;
 	unsigned long m_a_occ = 0;
 	cpumask_var_t cpus;
+	int scanned = 0;
 
 	WARN_ON_ONCE(work != &p->cache_work);
 
@@ -1836,7 +1884,12 @@ static void task_cache_work(struct callback_head *work)
 	scoped_guard (cpus_read_lock) {
 		guard(rcu)();
 
-		cpumask_and(cpus, cpu_online_mask, &mm->sc_stat.visited_cpus);
+		if (sched_feat(SC_NODE))
+			get_scan_cpumasks(cpus, p);
+		if (sched_feat(SC_VISIT))
+			cpumask_and(cpus, cpu_online_mask, &mm->sc_stat.visited_cpus);
+		else
+			cpumask_copy(cpus, cpu_online_mask);
 
 		for_each_cpu(cpu, cpus) {
 			/* XXX sched_cluster_active */
@@ -1853,6 +1906,7 @@ static void task_cache_work(struct callback_head *work)
 				    cur->mm == mm)
 					nr_running++;
 
+				scanned++;
 				occ = fraction_mm_sched(i, mm);
 				if (occ == 0)
 					continue;
@@ -1908,6 +1962,7 @@ static void task_cache_work(struct callback_head *work)
 
 	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
 	free_cpumask_var(cpus);
+	trace_sched_cache_scan(p, scanned);
 }
 
 void init_sched_mm(struct task_struct *p)
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..219173454320 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -142,3 +142,5 @@ SCHED_FEAT(LATENCY_WARN, false)
  */
 SCHED_FEAT(NI_RANDOM, true)
 SCHED_FEAT(NI_RATE, true)
+SCHED_FEAT(SC_NODE, true)
+SCHED_FEAT(SC_VISIT, true)
-- 
2.34.1

Re: [PATCH v4 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[K Prateek Nayak]

Hello Luo,

On 6/18/2026 12:12 PM, Luo Gengkun wrote:
> The overhead of task_cache_work() is high, especially in multi-NUMA
> systems. Currently, task_cache_work() tries to find the pref_llc by
> scanning all CPUs in the system. However, most of these scans are
> meaningless, such as those for CPUs that have never been visited or were
> accessed a long time ago.
> 
> To address this problem, introduce visited_cpus to track the visited CPUs
> and evict them once they have not been accessed for a duration exceeding
> llc_epoch_affinity_timeout. With this patch, get_scan_cpumasks() is no
> longer need and is therefore removed.

Please include performance numbers here.

I'm not convinced by the amount of benefits this brings. For hackbench,
the only stable improvement I can see from the cover letter is:

threads         1    4 |     27.758 (1.64%) |     25.711 (1.32%)   |   7.37% | IMPROVED

Since you mention Redis may benefit from this, do you actually have
numbers for Redis, or any other real world workload on your system?

[..snip..]

> @@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq,
>  	}
>  }
>  
> -static unsigned long fraction_mm_sched(struct rq *rq,
> -				       struct sched_cache_time *pcpu_sched)
> +static unsigned long fraction_mm_sched(int cpu,
> +				       struct mm_struct *mm)
>  {
> +	struct sched_cache_time *pcpu_sched =
> +		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
> +	struct rq *rq = cpu_rq(cpu);
> +
>  	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>  
> +	/* Skip the rq that has not been hit for a long time */
> +	if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
> +		cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus);

This makes me think your issue is more with pcpu_sched->runtime
not decaying fast enough?

If you haven't run in 50us, the pcpu_sched->runtime would already be
decayed by >> 5 but that is not enough? 

> +		return 0;
> +	}
> +
>  	__update_mm_sched(rq, pcpu_sched);
>  
>  	/*


>  static inline void update_avg_scale(u64 *avg, u64 sample)
>  {
>  	int factor = per_cpu(sd_llc_size, raw_smp_processor_id());
> @@ -1866,7 +1836,7 @@ static void task_cache_work(struct callback_head *work)
>  	scoped_guard (cpus_read_lock) {
>  		guard(rcu)();
>  
> -		get_scan_cpumasks(cpus, p);
> +		cpumask_and(cpus, cpu_online_mask, &mm->sc_stat.visited_cpus);

Doesn't this violate NUMA_BALANCING constraints? That was the whole point
of get_scan_cpumasks()

>  
>  		for_each_cpu(cpu, cpus) {
>  			/* XXX sched_cluster_active */
> @@ -1878,18 +1848,21 @@ static void task_cache_work(struct callback_head *work)
>  				continue;
>  
>  			for_each_cpu(i, sched_domain_span(sd)) {

What does your system topology look like? We visit all CPUs of LLC here
so even if a single CPU of LLC is set in visited, you'll still visit all
CPUs of LLC here ...

> -				occ = fraction_mm_sched(cpu_rq(i),
> -							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
> +				cur = rcu_dereference_all(cpu_rq(i)->curr);
> +				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
> +				    cur->mm == mm)
> +					nr_running++;
> +
> +				occ = fraction_mm_sched(i, mm);

... and all this does is return 0 if the mm hasn't run on that CPU for
over 50us. All this is pointing to more aggressive decay maybe somehow
helping you.

> +				if (occ == 0)
> +					continue;
> +
-- 
Thanks and Regards,
Prateek

Re: [PATCH v4 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Chen, Yu C]

On 6/19/2026 2:54 PM, K Prateek Nayak wrote:
> Hello Luo,
> 
> On 6/18/2026 12:12 PM, Luo Gengkun wrote:
>> The overhead of task_cache_work() is high, especially in multi-NUMA
>> systems. Currently, task_cache_work() tries to find the pref_llc by
>> scanning all CPUs in the system. However, most of these scans are
>> meaningless, such as those for CPUs that have never been visited or were
>> accessed a long time ago.
>>
>> To address this problem, introduce visited_cpus to track the visited CPUs
>> and evict them once they have not been accessed for a duration exceeding
>> llc_epoch_affinity_timeout. With this patch, get_scan_cpumasks() is no
>> longer need and is therefore removed.
> 
> Please include performance numbers here.
> 
> I'm not convinced by the amount of benefits this brings. For hackbench,
> the only stable improvement I can see from the cover letter is:
> 
> threads         1    4 |     27.758 (1.64%) |     25.711 (1.32%)   |   7.37% | IMPROVED
> 
> Since you mention Redis may benefit from this, do you actually have
> numbers for Redis, or any other real world workload on your system?
> 

I agree that metrics for Redis and other workloads would be quite helpful.
Gengkun seems to share the test data I sent over earlier, yet I haven’t 
observed
a noticeable difference in scores between current node-based scans and 
visit_cpu scans.

Hi Gengkun,
I wonder if you conducted Redis tests on an AMD machine, if the Redis 
performance
figures were captured from that environment, it would be helpful if you 
could
share the corresponding dataset. Should there be no noticeable score 
variance,
could you verify whether the total scan count has been reduced?

> [..snip..]
> 
>> @@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq,
>>   	}
>>   }
>>   
>> -static unsigned long fraction_mm_sched(struct rq *rq,
>> -				       struct sched_cache_time *pcpu_sched)
>> +static unsigned long fraction_mm_sched(int cpu,
>> +				       struct mm_struct *mm)
>>   {
>> +	struct sched_cache_time *pcpu_sched =
>> +		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
>> +	struct rq *rq = cpu_rq(cpu);
>> +
>>   	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>>   
>> +	/* Skip the rq that has not been hit for a long time */
>> +	if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
>> +		cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus);
> 
> This makes me think your issue is more with pcpu_sched->runtime
> not decaying fast enough?
> 
> If you haven't run in 50us, the pcpu_sched->runtime would already be
> decayed by >> 5 but that is not enough?
>

If I understanding correctly, Gengkun intended to reduce the number of
CPUs being scanned. If pcpu_sched->runtime has decayed by 32, the 
existing code may still check that CPU. And in the visited_cpu proposed
here, I suppose those CPUs will be skipped?

>>   
>>   		for_each_cpu(cpu, cpus) {
>>   			/* XXX sched_cluster_active */
>> @@ -1878,18 +1848,21 @@ static void task_cache_work(struct callback_head *work)
>>   				continue;
>>   
>>   			for_each_cpu(i, sched_domain_span(sd)) {
> 
> What does your system topology look like? We visit all CPUs of LLC here
> so even if a single CPU of LLC is set in visited, you'll still visit all
> CPUs of LLC here ...
> 

Yeah, maybe for_each_cpu_and(i, sched_domain_span(sd), cpus)

thanks,
Chenyu

Re: [PATCH v4 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Luo Gengkun]

On 2026/6/20 14:27, Chen, Yu C wrote:
> 
> On 6/19/2026 2:54 PM, K Prateek Nayak wrote:
>> Hello Luo,
>>
>> On 6/18/2026 12:12 PM, Luo Gengkun wrote:
>>> The overhead of task_cache_work() is high, especially in multi-NUMA
>>> systems. Currently, task_cache_work() tries to find the pref_llc by
>>> scanning all CPUs in the system. However, most of these scans are
>>> meaningless, such as those for CPUs that have never been visited or were
>>> accessed a long time ago.
>>>
>>> To address this problem, introduce visited_cpus to track the visited CPUs
>>> and evict them once they have not been accessed for a duration exceeding
>>> llc_epoch_affinity_timeout. With this patch, get_scan_cpumasks() is no
>>> longer need and is therefore removed.
>>
>> Please include performance numbers here.The hackbench data is shown below:

echo NO_SC_VISIT > /sys/kernel/debug/sched/features
echo SC_NODE > /sys/kernel/debug/sched/features
        hackbench-186777  [257] ..... 73726.150430: sched_cache_scan: comm=hackbench pid=186777 scan=384
        hackbench-186776  [067] ..... 73726.160434: sched_cache_scan: comm=hackbench pid=186776 scan=384
        hackbench-186771  [064] ..... 73726.170435: sched_cache_scan: comm=hackbench pid=186771 scan=384
        hackbench-186771  [064] ..... 73726.180429: sched_cache_scan: comm=hackbench pid=186771 scan=384

echo SC_VISIT > /sys/kernel/debug/sched/features
echo NO_SC_NODE > /sys/kernel/debug/sched/features
        hackbench-149213  [254] ..... 69412.500073: sched_cache_scan: comm=hackbench pid=149213 scan=8
        hackbench-149213  [254] ..... 69412.510073: sched_cache_scan: comm=hackbench pid=149213 scan=8
        hackbench-149213  [254] ..... 69412.520076: sched_cache_scan: comm=hackbench pid=149213 scan=8
        hackbench-149212  [063] ..... 69412.530077: sched_cache_scan: comm=hackbench pid=149212 scan=8
        hackbench-149213  [254] ..... 69412.540074: sched_cache_scan: comm=hackbench pid=149213 scan=8
        hackbench-149217  [250] ..... 69412.550073: sched_cache_scan: comm=hackbench pid=149217 scan=8

[root@localhost tracing]# taskset -pc 149217
pid 149217's current affinity list: 0-383

The data above demonstrates that this patch can effectively reduce the number of CPUs
that need to be scanned.
>>
>> I'm not convinced by the amount of benefits this brings. For hackbench,
>> the only stable improvement I can see from the cover letter is:
>>
>> threads         1    4 |     27.758 (1.64%) |     25.711 (1.32%)   |   7.37% | IMPROVED

For the thread-mode testing of hackbench, the data provided in the cover letter
aims to demonstrate that this patch does not introduce any performance
degradation. Since there are only a few processes involved in thread-mode, the
overhead incurred by task_cache_work remains negligible. This patch will show an
improvement when multiple processes exist on the system.
>>
>> Since you mention Redis may benefit from this, do you actually have
>> numbers for Redis, or any other real world workload on your system?
>
My previous evaluations were conducted on a Kunpeng platform. The results
are shown below (A total of 384 Redis instances are deployed here):

valkey-benchmark rps  |      baseline       |           schedcache            |     schedcache_visit
----------------------+---------------------+---------------------+-----------+-------------------+--------
                       |   avg latency(ms)   |    avg latency(ms)  |   DIFF(%) |   avg latency(ms) |   DIFF(%)
400000                |      0.37           |        0.496        |   -34.1%  |         0.299     |   +19.18%
  
However, over the past few days, I have also carried out the testing on an AMD.
The test machine is equipped with dual AMD EPYC 9654 processors, and with SMT
enabled, the system provies a total of 384 CPUs.

The lscpu output is as follows:

Caches (sum of all):
   L1d:                       6 MiB (192 instances)
   L1i:                       6 MiB (192 instances)
   L2:                        192 MiB (192 instances)
   L3:                        768 MiB (24 instances)
NUMA:
   NUMA node(s):              4
   NUMA node0 CPU(s):         0-47,192-239
   NUMA node1 CPU(s):         48-95,240-287
   NUMA node2 CPU(s):         96-143,288-335
   NUMA node3 CPU(s):         144-191,336-383

The Redis version used is 7.0.15, and the evaluation was performed via valkey_benchmark.
The benchmarks were executed on the same machine with a strict NUMA topology binding:
redis-server instances were bound to NUMA nodes 0 and 1, while the valkey_benchmark
processes were bound to NUMA nodes 2 and 3. A total of 192 Redis instances were deployed.

[root@localhost redis]# ps -ef | grep redis-server | wc -l
193

The tracing data is pasted as follow:

     redis-server-55831   [262] ..... 67888.162975: sched_cache_scan: comm=redis-server pid=55831 scan=9
     redis-server-56691   [012] ..... 67888.162975: sched_cache_scan: comm=redis-server pid=56691 scan=14
     redis-server-56651   [022] ..... 67888.162975: sched_cache_scan: comm=redis-server pid=56651 scan=8
  valkey-benchmar-145367  [200] ..... 67888.162976: sched_cache_scan: comm=valkey-benchmar pid=145367 scan=16
  valkey-benchmar-144397  [006] ..... 67888.162977: sched_cache_scan: comm=valkey-benchmar pid=144397 scan=16
  valkey-benchmar-145534  [284] ..... 67888.162977: sched_cache_scan: comm=valkey-benchmar pid=145534 scan=16
  valkey-benchmar-145253  [032] ..... 67888.162978: sched_cache_scan: comm=valkey-benchmar pid=145253 scan=16

[root@localhost tracing]# taskset -pc 55831
pid 55831's current affinity list: 0-95,192-287

I conducted benchmarks under various RPS (Requests Per Second) loads, and the P99
latency results are listed below.

valkey-benchmark rps  |      baseline       |           schedcache            |     schedcache_visit
----------------------+---------------------+---------------------+-----------+-------------------+--------
                       |   p99 latency(ms)   |    p99 latency(ms)  |   DIFF(%) |   p99 latency(ms) |  DIFF(%)
200000                |      0.26           |        0.383        |   -47.3%  |         0.264     |   -1.5%
300000                |      0.343          |        0.475        |   -38.4%  |         0.35      |   -2.0%
400000                |      0.445          |        0.567        |   -27.4%  |         0.453     |   -1.7%

Additionally, the output of perf top -e cycles:k highlights the overhead incurred
by task_cache_work:

valkey-benchmark rps  |         schedcache                        |     schedcache_visit
----------------------+-------------------------------------------+---------------------------------------
200000                |  1.12%  [kernel]  [k] task_cache_work     |  0.02%  [kernel]  [k] task_cache_work
300000                |  0.92%  [kernel]  [k] task_cache_work     |  0.02%  [kernel]  [k] task_cache_work
400000                |  0.82%  [kernel]  [k] task_cache_work     |  0.02%  [kernel]  [k] task_cache_work

Although the performance data is not as good as that on Kunpeng, the schedcache_visit
version is still better than the original one.
> 
> I agree that metrics for Redis and other workloads would be quite helpful.
> Gengkun seems to share the test data I sent over earlier, yet I haven’t observed
> a noticeable difference in scores between current node-based scans and visit_cpu scans.
> 
> Hi Gengkun,
> I wonder if you conducted Redis tests on an AMD machine, if the Redis performance
> figures were captured from that environment, it would be helpful if you could
> share the corresponding dataset. Should there be no noticeable score variance,
> could you verify whether the total scan count has been reduced?
> 
>> [..snip..]
>>
>>> @@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq,
>>>       }
>>>   }
>>> -static unsigned long fraction_mm_sched(struct rq *rq,
>>> -                       struct sched_cache_time *pcpu_sched)
>>> +static unsigned long fraction_mm_sched(int cpu,
>>> +                       struct mm_struct *mm)
>>>   {
>>> +    struct sched_cache_time *pcpu_sched =
>>> +        per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
>>> +    struct rq *rq = cpu_rq(cpu);
>>> +
>>>       guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>>> +    /* Skip the rq that has not been hit for a long time */
>>> +    if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) {
>>> +        cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus);
>>
>> This makes me think your issue is more with pcpu_sched->runtime
>> not decaying fast enough?
>>
>> If you haven't run in 50us, the pcpu_sched->runtime would already be
>> decayed by >> 5 but that is not enough?
>>
> 
> If I understanding correctly, Gengkun intended to reduce the number of
> CPUs being scanned. If pcpu_sched->runtime has decayed by 32, the existing code may still check that CPU. And in the visited_cpu proposed
> here, I suppose those CPUs will be skipped?
Yes, this patch is designed to reduce the overhead of task_cache_work by
decreasing the number of CPUs that need to be scanned.
> 
>>>           for_each_cpu(cpu, cpus) {
>>>               /* XXX sched_cluster_active */
>>> @@ -1878,18 +1848,21 @@ static void task_cache_work(struct callback_head *work)
>>>                   continue;
>>>               for_each_cpu(i, sched_domain_span(sd)) {
>>
>> What does your system topology look like? We visit all CPUs of LLC here
>> so even if a single CPU of LLC is set in visited, you'll still visit all
>> CPUs of LLC here ...
>>
> 
> Yeah, maybe for_each_cpu_and(i, sched_domain_span(sd), cpus)
Agreed. The aforementioned testing were conducted based on the for_each_cpu_and version.
The evaluation code is shown below:
-                       for_each_cpu(i, sched_domain_span(sd)) {
+                       if (sched_feat(SC_VISIT)) {
+                               cpumask_and(llc_cpus, sched_domain_span(sd), &mm->sc_stat.visited_cpus);
+                               target_cpus = llc_cpus;
+                       }
+                       else
+                               target_cpus = sched_domain_span(sd);
+
+                       for_each_cpu(i, target_cpus) {

thanks,
Gengkun
> 
> thanks,
> Chenyu
>