* [PATCH v11 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work
@ 2026-11-11 11:04 Luo Gengkun
2026-11-11 11:04 ` [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
2026-11-11 11:04 ` [PATCH v11 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun
0 siblings, 2 replies; 4+ messages in thread
From: Luo Gengkun @ 2026-11-11 11:04 UTC (permalink / raw)
To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen,
yu.c.chen, kprateek.nayak
Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, linux-kernel
Hi everyone,
The goal of this patch is to reduce the overhead of task_cache_work() by
minimizing the number of scanned CPUs, which yields significant
performance gains in multi-instance scenarios like Redis. To facilitate
testing, a debug patch is appended.
Below are the Redis test results conducted on AMD server.
When NUMA balancing is disabled:
valkey-benchmark rps | baseline | schedcache | schedcache_visit
| (no cache aware) | (without this series) | (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
| p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
400000 | 0.436 | 0.554 | -25.68% | 0.441 | -1.14%
Additionally, the output of perf top -e cycles:k highlights the overhead
incurred by task_cache_work():
valkey-benchmark rps | schedcache | schedcache_visit
| (without this series) | (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000 | 0.81% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
The trace data is shown below:
echo NO_SC_VISIT > /sys/kernel/debug/sched/features
redis-server-24660 [195] ..... 2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384
valkey-benchmar-48836 [317] ..... 2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384
valkey-benchmar-48487 [310] ..... 2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384
echo SC_VISIT > /sys/kernel/debug/sched/features
redis-server-29080 [225] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14
redis-server-26694 [018] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13
valkey-benchmar-48198 [103] ..... 2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16
The above results show that this series significantly reduces the overhead
of task_cache_work() by decreasing the number of CPUs that need to be
scanned, from 384 to 16, when NUMA balancing is disabled.
Even with NUMA balancing enabled, this patch provides better performance
benefits. The following tests were conducted with NUMA balancing enabled:
valkey-benchmark rps | baseline | schedcache | schedcache_visit
| (no cache aware) | (without this series) | (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
| p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
400000 | 0.437 | 0.454 | -3.89% | 0.442 | -1.14%
The overhead of task_cache_work() is as follows:
valkey-benchmark rps | schedcache | schedcache_visit
| (without this series) | (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000 | 0.13% [kernel] [k] task_cache_work | 0.03% [kernel] [k] task_cache_work
Hackbench testing was also conducted. The results below show that this
series does not affect the accuracy of cache-aware scheduling.
=========================================
Hackbench Comparison: baseline vs schedcache
=========================================
MODE GROUPS FDS | baseline(std) | schedcache(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
threads 1 10 | 113.200 (4.22%) | 67.300 (1.32%) | 40.55% | IMPROVED
threads 1 2 | 16.555 (4.11%) | 11.020 (1.66%) | 33.43% | IMPROVED
threads 1 20 | 250.774 (1.26%) | 265.026 (5.44%) | -5.68% | REGRESSED
threads 1 4 | 42.117 (1.44%) | 27.758 (1.64%) | 34.09% | IMPROVED
threads 1 6 | 65.140 (4.31%) | 39.182 (1.38%) | 39.85% | IMPROVED
threads 1 8 | 84.286 (1.29%) | 53.721 (1.58%) | 36.26% | IMPROVED
threads 2 10 | 122.592 (0.44%) | 113.365 (4.93%) | 7.53% | IMPROVED
threads 2 2 | 17.702 (4.09%) | 10.473 (0.42%) | 40.84% | IMPROVED
threads 2 20 | 336.457 (1.77%) | 314.108 (1.51%) | 6.64% | IMPROVED
threads 2 4 | 43.989 (0.88%) | 27.067 (3.38%) | 38.47% | IMPROVED
threads 2 6 | 69.322 (0.85%) | 41.707 (4.19%) | 39.84% | IMPROVED
threads 2 8 | 103.767 (1.81%) | 58.518 (3.00%) | 43.61% | IMPROVED
threads 4 10 | 148.882 (3.56%) | 149.449 (1.06%) | -0.38% | REGRESSED
threads 4 2 | 18.909 (2.96%) | 11.063 (2.08%) | 41.49% | IMPROVED
threads 4 20 | 724.943 (2.14%) | 631.222 (3.92%) | 12.93% | IMPROVED
threads 4 4 | 48.191 (1.91%) | 27.352 (5.35%) | 43.24% | IMPROVED
threads 4 6 | 79.725 (3.84%) | 78.732 (4.10%) | 1.25% | IMPROVED
threads 4 8 | 108.768 (1.36%) | 105.928 (1.65%) | 2.61% | IMPROVED
Hackbench Comparison: schedcache vs schedcache_visit
=========================================
MODE GROUPS FDS | schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+----------------------+---------+-----------
threads 1 10 | 67.300 (1.32%) | 67.014 (0.96%) | 0.42% | IMPROVED
threads 1 2 | 11.020 (1.66%) | 10.557 (1.46%) | 4.20% | IMPROVED
threads 1 20 | 265.026 (5.44%) | 212.366 (16.32%) | 19.87% | IMPROVED
threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED
threads 1 6 | 39.182 (1.38%) | 38.914 (0.34%) | 0.68% | IMPROVED
threads 1 8 | 53.721 (1.58%) | 52.889 (0.27%) | 1.55% | IMPROVED
threads 2 10 | 121.203 (6.99%) | 124.254 (1.38%) | -2.52% | REGRESSED
threads 2 2 | 10.473 (0.42%) | 11.206 (5.91%) | -7.00% | REGRESSED
threads 2 20 | 314.108 (1.51%) | 301.754 (1.95%) | 3.93% | IMPROVED
threads 2 4 | 27.067 (3.38%) | 28.028 (2.01%) | -3.55% | REGRESSED
threads 2 6 | 41.707 (4.19%) | 42.149 (3.35%) | -1.06% | REGRESSED
threads 2 8 | 58.518 (3.00%) | 57.133 (4.39%) | 2.37% | IMPROVED
threads 4 10 | 149.449 (1.06%) | 141.407 (0.08%) | 5.38% | IMPROVED
threads 4 2 | 11.063 (2.08%) | 11.360 (5.85%) | -2.68% | REGRESSED
threads 4 20 | 631.222 (3.92%) | 622.780 (2.49%) | 1.34% | IMPROVED
threads 4 4 | 27.352 (5.35%) | 27.947 (5.37%) | -2.18% | REGRESSED
threads 4 6 | 78.732 (4.10%) | 73.911 (0.70%) | 6.12% | IMPROVED
threads 4 8 | 105.928 (1.65%) | 107.535 (3.29%) | -1.52% | REGRESSED
---
Changes history
**v11 Changes:**
1. Addressed code review feedback from Tim Chen:
- Removed a redundant 'continue' statement to minimize code changes.
- Shortened and clarified the lockless comment for visited_cpus.
- Updated the eviction comment and fixed a long-line coding style issue.
2. Collected Tested-by tag from K Prateek Nayak <kprateek.nayak@amd.com>
3. Collected Reviewed-by and Tested-by tags from Chen Yu <yu.c.chen@intel.com>
Link to v10: https://lore.kernel.org/all/20261009032943.95091-1-luogengkun@huaweicloud.com
**v10 Changes:**
1. Rebase onto the latest mainline.
2. Dropped Reviewed-by and Tested-by tags due to the rebase.
Link to v9: https://lore.kernel.org/all/20260731024417.1106503-1-luogengkun2@huawei.com/
**v9 Changes:**
1. Using for_each_cpu_and(i, sched_domain_span(sd), cpus) instead of
for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) in
the inner loop to reduce the race window to read mm->sc_stat.pcpu_sched.
2. Update the comment to reflect that a single scanner is elected per epoch
in task_cache_work().
Link to v8: https://lore.kernel.org/all/20260723040429.630176-1-luogengkun2@huawei.com/
**v8 Changes:**
1. Remove get_scan_cpumasks() since visited_cpus provides the exact CPUs to
scan.
Link to v7: https://lore.kernel.org/all/20260720122214.3977092-1-luogengkun2@huawei.com/
**v7 Changes:**
1. Add code comments in task_cache_work() to clarify the race condition
between setting and reading visited_cpus.
Link to v6: https://lore.kernel.org/all/20260717041231.3324851-1-luogengkun2@huawei.com/
**v6 Changes:**
1. Switched to cpumask_var_t to dynamically allocate visited_cpus inside
mm_struct, preventing per-process memory bloat when NR_CPUS is configured
large.
2. Intersected the LLC scheduling domain span directly with visited_cpus
to prevent skipping CPU is cross-node LLC topolgies.
3. Renamed epoch_timeout to epoch_last_visit to better reflect its purpose.
4. Moved __update_mm_sched to execute before the timeout check in
fraction_mm_sched()
Link to v5: https://lore.kernel.org/all/20260709130053.2749834-1-luogengkun2@huawei.com/
**v5 Changes:**
1. Restore get_scan_cpumasks() to avoid violating NUMA_BALANCING constraints.
2. Use for_each_cpu_and() to filter CPUs within the LLC domain.
Link to v4: https://lore.kernel.org/all/20260618064300.80072-1-luogengkun2@huawei.com/
**v4 Changes:**
1. Rebase to the master.
2. epoch_timeout is introduced to evict expired CPUs instead of relying on
epoch, because epoch is refreshed periodically due to invocations of
fraction_mm_sched().
3. Move the increasement of nr_running before fraction_mm_sched().
4. Remove the redundant 'work->next' reset at the end of task_cache_work().
5. Add a debug patch to show the number of CPUs scanned to show the
benefit of this optimization.
Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/
**v3 Changes:**
1. Remove the static key and enable this feature by default.
2. Reuse llc_epoch_affinity_timeout instead of introducing
llc_epoch_visited_timeout.
3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into
fraction_mm_sched() to avoid race between task_cache_work() and
__update_mm_sched().
4. Reset work->next at the end of task_cache_work() to prevent concurrent
executions by multiple threads within the same process.
Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/
**v2 Changes:**
1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead.
2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead.
Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/
Luo Gengkun (2):
sched/cache: Reduce the overhead of task_cache_work by only scan the
visisted cpus
-- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched
feature to track scan cost
include/linux/sched.h | 2 ++
include/trace/events/sched.h | 21 +++++++++++
kernel/sched/fair.c | 68 +++++++++++++++++++++++++++++-------
kernel/sched/features.h | 1 +
4 files changed, 80 insertions(+), 12 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus
2026-11-11 11:04 [PATCH v11 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun
@ 2026-11-11 11:04 ` Luo Gengkun
2026-11-16 7:24 ` Chen Yu
2026-11-11 11:04 ` [PATCH v11 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun
1 sibling, 1 reply; 4+ messages in thread
From: Luo Gengkun @ 2026-11-11 11:04 UTC (permalink / raw)
To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen,
yu.c.chen, kprateek.nayak
Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, linux-kernel
The overhead of task_cache_work() is high, especially in multi-NUMA systems.
Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs
in the system. However, most of these scans are meaningless, such as those
for CPUs that have never been visited or were accessed a long time ago.
To address this problem, introduce visited_cpus to track the visited CPUs
and evict them once they have not been accessed for a duration exceeding
llc_epoch_affinity_timeout.
Now that we know exactly which CPUs to scan from visited_cpus, we can remove
get_scan_cpumasks().
Below are the Redis test results conducted on AMD server.
When NUMA balancing is disabled:
valkey-benchmark rps | baseline | schedcache | schedcache_visit
| (no cache aware) | (without this series) | (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
| p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
400000 | 0.436 | 0.554 | -25.68% | 0.441 | -1.14%
Additionally, the output of perf top -e cycles:k highlights the overhead
incurred by task_cache_work():
valkey-benchmark rps | schedcache | schedcache_visit
| (without this series) | (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000 | 0.81% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
The trace data is shown below:
echo NO_SC_VISIT > /sys/kernel/debug/sched/features
redis-server-24660 [195] ..... 2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384
valkey-benchmar-48836 [317] ..... 2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384
valkey-benchmar-48487 [310] ..... 2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384
echo SC_VISIT > /sys/kernel/debug/sched/features
redis-server-29080 [225] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14
redis-server-26694 [018] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13
valkey-benchmar-48198 [103] ..... 2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16
The above results show that this series significantly reduces the overhead
of task_cache_work() by decreasing the number of CPUs that need to be
scanned, from 384 to 16, when NUMA balancing is disabled.
Even with NUMA balancing enabled, this patch provides better performance
benefits. The following tests were conducted with NUMA balancing enabled:
valkey-benchmark rps | baseline | schedcache | schedcache_visit
| (no cache aware) | (without this series) | (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
| p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
400000 | 0.437 | 0.454 | -3.89% | 0.442 | -1.14%
The overhead of task_cache_work() is as follows:
valkey-benchmark rps | schedcache | schedcache_visit
| (without this series) | (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000 | 0.13% [kernel] [k] task_cache_work | 0.03% [kernel] [k] task_cache_work
Hackbench testing was also conducted. The results below show that this
series does not affect the accuracy of cache-aware scheduling.
=========================================
Hackbench Comparison: baseline vs schedcache
=========================================
MODE GROUPS FDS | baseline(std) | schedcache(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
threads 1 10 | 113.200 (4.22%) | 67.300 (1.32%) | 40.55% | IMPROVED
threads 1 2 | 16.555 (4.11%) | 11.020 (1.66%) | 33.43% | IMPROVED
threads 1 20 | 250.774 (1.26%) | 265.026 (5.44%) | -5.68% | REGRESSED
threads 1 4 | 42.117 (1.44%) | 27.758 (1.64%) | 34.09% | IMPROVED
threads 1 6 | 65.140 (4.31%) | 39.182 (1.38%) | 39.85% | IMPROVED
threads 1 8 | 84.286 (1.29%) | 53.721 (1.58%) | 36.26% | IMPROVED
threads 2 10 | 122.592 (0.44%) | 113.365 (4.93%) | 7.53% | IMPROVED
threads 2 2 | 17.702 (4.09%) | 10.473 (0.42%) | 40.84% | IMPROVED
threads 2 20 | 336.457 (1.77%) | 314.108 (1.51%) | 6.64% | IMPROVED
threads 2 4 | 43.989 (0.88%) | 27.067 (3.38%) | 38.47% | IMPROVED
threads 2 6 | 69.322 (0.85%) | 41.707 (4.19%) | 39.84% | IMPROVED
threads 2 8 | 103.767 (1.81%) | 58.518 (3.00%) | 43.61% | IMPROVED
threads 4 10 | 148.882 (3.56%) | 149.449 (1.06%) | -0.38% | REGRESSED
threads 4 2 | 18.909 (2.96%) | 11.063 (2.08%) | 41.49% | IMPROVED
threads 4 20 | 724.943 (2.14%) | 631.222 (3.92%) | 12.93% | IMPROVED
threads 4 4 | 48.191 (1.91%) | 27.352 (5.35%) | 43.24% | IMPROVED
threads 4 6 | 79.725 (3.84%) | 78.732 (4.10%) | 1.25% | IMPROVED
threads 4 8 | 108.768 (1.36%) | 105.928 (1.65%) | 2.61% | IMPROVED
Hackbench Comparison: schedcache vs schedcache_visit
=========================================
MODE GROUPS FDS | schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+----------------------+---------+-----------
threads 1 10 | 67.300 (1.32%) | 67.014 (0.96%) | 0.42% | IMPROVED
threads 1 2 | 11.020 (1.66%) | 10.557 (1.46%) | 4.20% | IMPROVED
threads 1 20 | 265.026 (5.44%) | 212.366 (16.32%) | 19.87% | IMPROVED
threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED
threads 1 6 | 39.182 (1.38%) | 38.914 (0.34%) | 0.68% | IMPROVED
threads 1 8 | 53.721 (1.58%) | 52.889 (0.27%) | 1.55% | IMPROVED
threads 2 10 | 121.203 (6.99%) | 124.254 (1.38%) | -2.52% | REGRESSED
threads 2 2 | 10.473 (0.42%) | 11.206 (5.91%) | -7.00% | REGRESSED
threads 2 20 | 314.108 (1.51%) | 301.754 (1.95%) | 3.93% | IMPROVED
threads 2 4 | 27.067 (3.38%) | 28.028 (2.01%) | -3.55% | REGRESSED
threads 2 6 | 41.707 (4.19%) | 42.149 (3.35%) | -1.06% | REGRESSED
threads 2 8 | 58.518 (3.00%) | 57.133 (4.39%) | 2.37% | IMPROVED
threads 4 10 | 149.449 (1.06%) | 141.407 (0.08%) | 5.38% | IMPROVED
threads 4 2 | 11.063 (2.08%) | 11.360 (5.85%) | -2.68% | REGRESSED
threads 4 20 | 631.222 (3.92%) | 622.780 (2.49%) | 1.34% | IMPROVED
threads 4 4 | 27.352 (5.35%) | 27.947 (5.37%) | -2.18% | REGRESSED
threads 4 6 | 78.732 (4.10%) | 73.911 (0.70%) | 6.12% | IMPROVED
threads 4 8 | 105.928 (1.65%) | 107.535 (3.29%) | -1.52% | REGRESSED
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Luo Gengkun <luogengkun@huaweicloud.com>
---
include/linux/sched.h | 2 +
kernel/sched/fair.c | 99 ++++++++++++++++++-------------------------
2 files changed, 43 insertions(+), 58 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..1b08925c6358 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2385,6 +2385,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)
struct sched_cache_time {
u64 runtime;
unsigned long epoch;
+ unsigned long epoch_last_visit;
};
struct sched_cache_group {
@@ -2397,6 +2398,7 @@ struct sched_cache_group {
int cpu;
refcount_t refcnt;
struct rcu_head rcu;
+ cpumask_var_t visited_cpus;
} ____cacheline_aligned_in_smp;
struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *grp);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79f6b3a4fd7f..c97f08eaa9e5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1599,11 +1599,11 @@ int mm_init_sched(struct mm_struct *mm,
int i;
grp = kzalloc_obj(*grp);
- if (!grp) {
- free_percpu(_pcpu_sched);
- mm->sched_cache_grp = NULL;
- return -ENOMEM;
- }
+ if (!grp)
+ goto err_free_pcpu;
+
+ if (!zalloc_cpumask_var(&grp->visited_cpus, GFP_KERNEL))
+ goto err_free_grp;
for_each_possible_cpu(i) {
struct sched_cache_time *pcpu_sched = per_cpu_ptr(_pcpu_sched, i);
@@ -1612,6 +1612,7 @@ int mm_init_sched(struct mm_struct *mm,
pcpu_sched->runtime = 0;
/* a slightly stale cpu epoch is acceptible */
pcpu_sched->epoch = rq->cpu_epoch;
+ pcpu_sched->epoch_last_visit = rq->cpu_epoch;
epoch = rq->cpu_epoch;
}
@@ -1637,6 +1638,14 @@ int mm_init_sched(struct mm_struct *mm,
*/
smp_store_release(&mm->sched_cache_grp, grp);
return 0;
+
+err_free_grp:
+ kfree(grp);
+
+err_free_pcpu:
+ free_percpu(_pcpu_sched);
+ mm->sched_cache_grp = NULL;
+ return -ENOMEM;
}
static void sched_cache_group_free_rcu(struct rcu_head *rcu)
@@ -1645,6 +1654,7 @@ static void sched_cache_group_free_rcu(struct rcu_head *rcu)
container_of(rcu, struct sched_cache_group, rcu);
free_percpu(grp->pcpu_sched);
+ free_cpumask_var(grp->visited_cpus);
kfree(grp);
}
@@ -1785,13 +1795,24 @@ static inline void __update_mm_sched(struct rq *rq,
}
}
-static unsigned long fraction_mm_sched(struct rq *rq,
- struct sched_cache_time *pcpu_sched)
+static unsigned long fraction_mm_sched(int cpu,
+ struct sched_cache_group *grp)
{
+ struct sched_cache_time *pcpu_sched =
+ per_cpu_ptr(grp->pcpu_sched, cpu);
+ struct rq *rq = cpu_rq(cpu);
+
guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
__update_mm_sched(rq, pcpu_sched);
+ /* Evict the CPU if this sched group has not run on it for a while */
+ if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) >
+ llc_epoch_affinity_timeout) {
+ cpumask_clear_cpu(cpu, grp->visited_cpus);
+ return 0;
+ }
+
/*
* Runtime is a geometric series (r=0.5) and as such will sum to twice
* the accumulation period, this means the multiplcation here should
@@ -1867,6 +1888,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
pcpu_sched->runtime += delta_exec;
rq->cpu_runtime += delta_exec;
epoch = rq->cpu_epoch;
+ pcpu_sched->epoch_last_visit = epoch;
+ if (!cpumask_test_cpu(cpu_of(rq), grp->visited_cpus))
+ cpumask_set_cpu(cpu_of(rq), grp->visited_cpus);
}
/*
@@ -1917,52 +1941,6 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
}
}
-static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p,
- struct sched_cache_group *grp)
-{
-#ifdef CONFIG_NUMA_BALANCING
- int cpu, curr_cpu, nid, pref_nid;
-
- if (!static_branch_likely(&sched_numa_balancing))
- goto out;
-
- cpu = READ_ONCE(grp->cpu);
- if (cpu != -1)
- nid = cpu_to_node(cpu);
- curr_cpu = task_cpu(p);
-
- /*
- * Scanning in the preferred NUMA node is ideal. However, the NUMA
- * preferred node is per-task rather than per-process. It is possible
- * for different threads of the process to have distinct preferred
- * nodes; consequently, the process-wide preferred LLC may bounce
- * between different nodes. As a workaround, maintain the scan
- * CPU mask to also cover the process's current preferred LLC and the
- * current running node to mitigate the bouncing risk.
- * TBD: numa_group should be considered during task aggregation.
- */
- pref_nid = p->numa_preferred_nid;
- /* honor the task's preferred node */
- if (pref_nid == NUMA_NO_NODE)
- goto out;
-
- cpumask_or(cpus, cpus, cpumask_of_node(pref_nid));
-
- /* honor the task's preferred LLC CPU */
- if (cpu != -1 && !cpumask_test_cpu(cpu, cpus) && nid != NUMA_NO_NODE)
- cpumask_or(cpus, cpus, cpumask_of_node(nid));
-
- /* make sure the task's current running node is included */
- if (!cpumask_test_cpu(curr_cpu, cpus))
- cpumask_or(cpus, cpus, cpumask_of_node(cpu_to_node(curr_cpu)));
-
- return;
-
-out:
-#endif
- cpumask_copy(cpus, cpu_online_mask);
-}
-
static inline void update_avg_scale(u64 *avg, u64 sample)
{
int factor = per_cpu(sd_llc_size, raw_smp_processor_id());
@@ -2011,7 +1989,7 @@ static void task_cache_work(struct callback_head *work)
if (time_before(now, next_scan))
return;
- /* only 1 thread is allowed to scan */
+ /* elect a single scanner per epoch */
if (!try_cmpxchg(&grp->next_scan, &next_scan,
now + max_t(unsigned long,
READ_ONCE(llc_epoch_period), 1)))
@@ -2032,7 +2010,13 @@ static void task_cache_work(struct callback_head *work)
scoped_guard (cpus_read_lock) {
guard(rcu)();
- get_scan_cpumasks(cpus, p, grp);
+ /*
+ * account_mm_sched() can set a bit in visited_cpus while
+ * it is read here locklessly for efficiency. A CPU missed
+ * this way has only just been used by this sched group,
+ * so its share small, and the next scan will include it.
+ */
+ cpumask_and(cpus, cpu_online_mask, grp->visited_cpus);
for_each_cpu(cpu, cpus) {
/* XXX sched_cluster_active */
@@ -2043,9 +2027,8 @@ static void task_cache_work(struct callback_head *work)
if (!sd)
continue;
- for_each_cpu(i, sched_domain_span(sd)) {
- occ = fraction_mm_sched(cpu_rq(i),
- per_cpu_ptr(grp->pcpu_sched, i));
+ for_each_cpu_and(i, sched_domain_span(sd), cpus) {
+ occ = fraction_mm_sched(i, grp);
a_occ += occ;
if (occ > m_occ) {
m_occ = occ;
--
2.34.1
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v11 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost
2026-11-11 11:04 [PATCH v11 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun
2026-11-11 11:04 ` [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
@ 2026-11-11 11:04 ` Luo Gengkun
1 sibling, 0 replies; 4+ messages in thread
From: Luo Gengkun @ 2026-11-11 11:04 UTC (permalink / raw)
To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen,
yu.c.chen, kprateek.nayak
Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, linux-kernel
This commit re-adds get_scan_cpumasks() and introduces two sched features
as well as trace events to track the scanning cost for testing purposes.
Signed-off-by: Luo Gengkun <luogengkun@huaweicloud.com>
---
include/trace/events/sched.h | 21 +++++++++++
kernel/sched/fair.c | 69 +++++++++++++++++++++++++++++++++---
kernel/sched/features.h | 1 +
3 files changed, 87 insertions(+), 4 deletions(-)
diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index 535860581f15..aced624f198d 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -10,6 +10,27 @@
#include <linux/tracepoint.h>
#include <linux/binfmts.h>
+TRACE_EVENT(sched_cache_scan,
+
+ TP_PROTO(struct task_struct *t, int scan),
+
+ TP_ARGS(t, scan),
+
+ TP_STRUCT__entry(
+ __string( comm, t->comm )
+ __field( pid_t, pid )
+ __field( int, scan )
+ ),
+
+ TP_fast_assign(
+ __assign_str(comm);
+ __entry->pid = t->pid;
+ __entry->scan = scan;
+ ),
+
+ TP_printk("comm=%s pid=%d scan=%d", __get_str(comm), __entry->pid,
+ __entry->scan)
+);
/*
* Tracepoint for calling kthread_stop, performed to end a kthread:
*/
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c97f08eaa9e5..7a363da2cf63 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1807,7 +1807,7 @@ static unsigned long fraction_mm_sched(int cpu,
__update_mm_sched(rq, pcpu_sched);
/* Evict the CPU if this sched group has not run on it for a while */
- if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) >
+ if (sched_feat(SC_VISIT) && (rq->cpu_epoch - pcpu_sched->epoch_last_visit) >
llc_epoch_affinity_timeout) {
cpumask_clear_cpu(cpu, grp->visited_cpus);
return 0;
@@ -1889,7 +1889,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
rq->cpu_runtime += delta_exec;
epoch = rq->cpu_epoch;
pcpu_sched->epoch_last_visit = epoch;
- if (!cpumask_test_cpu(cpu_of(rq), grp->visited_cpus))
+ if (sched_feat(SC_VISIT) &&
+ !cpumask_test_cpu(cpu_of(rq), grp->visited_cpus))
cpumask_set_cpu(cpu_of(rq), grp->visited_cpus);
}
@@ -1941,6 +1942,52 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
}
}
+static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p,
+ struct sched_cache_group *grp)
+{
+#ifdef CONFIG_NUMA_BALANCING
+ int cpu, curr_cpu, nid, pref_nid;
+
+ if (!static_branch_likely(&sched_numa_balancing))
+ goto out;
+
+ cpu = READ_ONCE(grp->cpu);
+ if (cpu != -1)
+ nid = cpu_to_node(cpu);
+ curr_cpu = task_cpu(p);
+
+ /*
+ * Scanning in the preferred NUMA node is ideal. However, the NUMA
+ * preferred node is per-task rather than per-process. It is possible
+ * for different threads of the process to have distinct preferred
+ * nodes; consequently, the process-wide preferred LLC may bounce
+ * between different nodes. As a workaround, maintain the scan
+ * CPU mask to also cover the process's current preferred LLC and the
+ * current running node to mitigate the bouncing risk.
+ * TBD: numa_group should be considered during task aggregation.
+ */
+ pref_nid = p->numa_preferred_nid;
+ /* honor the task's preferred node */
+ if (pref_nid == NUMA_NO_NODE)
+ goto out;
+
+ cpumask_or(cpus, cpus, cpumask_of_node(pref_nid));
+
+ /* honor the task's preferred LLC CPU */
+ if (cpu != -1 && !cpumask_test_cpu(cpu, cpus) && nid != NUMA_NO_NODE)
+ cpumask_or(cpus, cpus, cpumask_of_node(nid));
+
+ /* make sure the task's current running node is included */
+ if (!cpumask_test_cpu(curr_cpu, cpus))
+ cpumask_or(cpus, cpus, cpumask_of_node(cpu_to_node(curr_cpu)));
+
+ return;
+
+out:
+#endif
+ cpumask_copy(cpus, cpu_online_mask);
+}
+
static inline void update_avg_scale(u64 *avg, u64 sample)
{
int factor = per_cpu(sd_llc_size, raw_smp_processor_id());
@@ -1963,11 +2010,13 @@ static void task_cache_work(struct callback_head *work)
{
struct sched_cache_group *grp __free(sched_cache_group_put) = NULL;
cpumask_var_t cpus __free(free_cpumask_var) = CPUMASK_VAR_NULL;
+ cpumask_var_t scan_cpus __free(free_cpumask_var) = CPUMASK_VAR_NULL;
int cpu, m_a_cpu = -1, nr_running = 0, curr_cpu;
unsigned long next_scan, now = jiffies;
struct task_struct *p = current, *cur;
unsigned long curr_m_a_occ = 0;
unsigned long m_a_occ = 0;
+ int scanned = 0;
WARN_ON_ONCE(work != &p->cache_work);
@@ -2007,6 +2056,9 @@ static void task_cache_work(struct callback_head *work)
if (!zalloc_cpumask_var(&cpus, GFP_KERNEL))
return;
+ if (!zalloc_cpumask_var(&scan_cpus, GFP_KERNEL))
+ return;
+
scoped_guard (cpus_read_lock) {
guard(rcu)();
@@ -2016,7 +2068,10 @@ static void task_cache_work(struct callback_head *work)
* this way has only just been used by this sched group,
* so its share small, and the next scan will include it.
*/
- cpumask_and(cpus, cpu_online_mask, grp->visited_cpus);
+ if (sched_feat(SC_VISIT))
+ cpumask_and(cpus, cpu_online_mask, grp->visited_cpus);
+ else
+ get_scan_cpumasks(cpus, p, grp);
for_each_cpu(cpu, cpus) {
/* XXX sched_cluster_active */
@@ -2027,7 +2082,11 @@ static void task_cache_work(struct callback_head *work)
if (!sd)
continue;
- for_each_cpu_and(i, sched_domain_span(sd), cpus) {
+ cpumask_copy(scan_cpus, sched_domain_span(sd));
+ if (sched_feat(SC_VISIT))
+ cpumask_and(scan_cpus, scan_cpus, cpus);
+
+ for_each_cpu(i, scan_cpus) {
occ = fraction_mm_sched(i, grp);
a_occ += occ;
if (occ > m_occ) {
@@ -2043,6 +2102,7 @@ static void task_cache_work(struct callback_head *work)
if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
rcu_access_pointer(cur->sched_cache_grp) == grp)
nr_running++;
+ scanned++;
}
/*
@@ -2087,6 +2147,7 @@ static void task_cache_work(struct callback_head *work)
}
update_avg_scale(&grp->nr_running_avg, nr_running);
+ trace_sched_cache_scan(p, scanned);
}
void init_sched_mm(struct task_struct *p)
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..990714ebabfd 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -142,3 +142,4 @@ SCHED_FEAT(LATENCY_WARN, false)
*/
SCHED_FEAT(NI_RANDOM, true)
SCHED_FEAT(NI_RATE, true)
+SCHED_FEAT(SC_VISIT, false)
--
2.34.1
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus
2026-11-11 11:04 ` [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
@ 2026-11-16 7:24 ` Chen Yu
0 siblings, 0 replies; 4+ messages in thread
From: Chen Yu @ 2026-11-16 7:24 UTC (permalink / raw)
To: Luo Gengkun, tim.c.chen
Cc: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen,
yu.c.chen, kprateek.nayak, dietmar.eggemann, rostedt, bsegall,
mgorman, vschneid, linux-kernel
On Wed, Nov 11, 2026 at 11:04:08AM +0000, Luo Gengkun wrote:
> Date: Wed, 11 Nov 2026 11:04:08 +0000
> From: Luo Gengkun <luogengkun@huaweicloud.com>
> To: peterz@infradead.org, mingo@redhat.com, juri.lelli@redhat.com,
> vincent.guittot@linaro.org, tim.c.chen@linux.intel.com,
> yu.c.chen@intel.com, kprateek.nayak@amd.com
> Cc: dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
> mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org
> Subject: [PATCH v11 1/2] sched/cache: Reduce the overhead of
> task_cache_work by only scan the visisted cpus
> X-Mailer: git-send-email 2.34.1
>
> The overhead of task_cache_work() is high, especially in multi-NUMA systems.
> Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs
> in the system. However, most of these scans are meaningless, such as those
> for CPUs that have never been visited or were accessed a long time ago.
>
> To address this problem, introduce visited_cpus to track the visited CPUs
> and evict them once they have not been accessed for a duration exceeding
> llc_epoch_affinity_timeout.
>
> Now that we know exactly which CPUs to scan from visited_cpus, we can remove
> get_scan_cpumasks().
>
> Below are the Redis test results conducted on AMD server.
>
> When NUMA balancing is disabled:
>
> valkey-benchmark rps | baseline | schedcache | schedcache_visit
> | (no cache aware) | (without this series) | (with this series)
> ----------------------+---------------------+---------------------+-----------+-------------------+--------
> | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
> 400000 | 0.436 | 0.554 | -25.68% | 0.441 | -1.14%
>
> Additionally, the output of perf top -e cycles:k highlights the overhead
> incurred by task_cache_work():
>
> valkey-benchmark rps | schedcache | schedcache_visit
> | (without this series) | (with this series)
> ----------------------+-------------------------------------------+---------------------------------------
> 400000 | 0.81% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
>
> The trace data is shown below:
>
> echo NO_SC_VISIT > /sys/kernel/debug/sched/features
> redis-server-24660 [195] ..... 2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384
> valkey-benchmar-48836 [317] ..... 2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384
> valkey-benchmar-48487 [310] ..... 2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384
>
> echo SC_VISIT > /sys/kernel/debug/sched/features
> redis-server-29080 [225] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14
> redis-server-26694 [018] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13
> valkey-benchmar-48198 [103] ..... 2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16
>
SC_VISIT is not in the formal patch, so maybe adjust the comment above to say that this SC_VISIT is a trial
test feature and provide the link. But it is still OK to keep this as it is now.
Tim, may I know if this version looks good to you?
thanks,
Chenyu
>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Tested-by: Chen Yu <yu.c.chen@intel.com>
> Reviewed-by: Chen Yu <yu.c.chen@intel.com>
> Signed-off-by: Luo Gengkun <luogengkun@huaweicloud.com>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-16 7:23 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-11 11:04 [PATCH v11 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun
2026-11-11 11:04 ` [PATCH v11 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
2026-11-16 7:24 ` Chen Yu
2026-11-11 11:04 ` [PATCH v11 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®