Re: [PATCH v8 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Luo Gengkun <luogengkun2@huawei.com>]

On 2026/7/27 9:09, Chen, Yu C wrote:
> On 7/23/2026 12:04 PM, Luo Gengkun wrote:
> 
> [ ... ]
> 
>> +static unsigned long fraction_mm_sched(int cpu,
>> +                       struct mm_struct *mm)
>>   {
>> +    struct sched_cache_time *pcpu_sched =
>> +        per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
>> +    struct rq *rq = cpu_rq(cpu);
>> +
>>       guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>>       __update_mm_sched(rq, pcpu_sched);
>> +    /* Skip the rq that has not been hit for a long time */
>> +    if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
> 
> In v2 there is a check if the cpu has been set before writing:
> cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus)
> https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/
> do we need to bring that back?
> 
I don't think we need it back. Here is why:

In v2, for_each_cpu was used instead of for_each_cpu_and in the inner loop,
meaning some CPUs being checked might not have been set. Therefore,
cpumask_test_cpu was necessary to filter out those cases.

Now, with for_each_cpu_and(i, sched_domain_span(sd), &mm->sc_stat.visited_cpus),
we can ensure each scanned CPU is set, so the issue no longer exists.
Furthermore, the only place where the visited_cpus bits are cleared is
task_cache_work(), which is only called once per scan period, there is no
risk of the bit being cleared concurrently mid-loop.

However, is there a possibility that the current task_cache_work() execution
hasn't finished yet when the next scan window arrives? For instance, if the
current task work is heavily delayed or preempted by unexpected interrupt,
jiffies could advance past next_scan before the loop completes.
If we move the `work->next = work;` to the very end of task_cache_work(),
would that resolve this issue? By doing so, the existing `work->next == work`
check in task_tick_cache() should fail and no new task work will be submitted.

Please let me know if I'm missing something.


>> +        cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
>> +        return 0;
>> +    }
>> +
> 
> [ ... ]
> 
>> -            for_each_cpu(i, sched_domain_span(sd)) {
>> -                occ = fraction_mm_sched(cpu_rq(i),
>> -                            per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
>> +            for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
> 
> Does using
> for_each_cpu_and(i, sched_domain_span(sd), cpus)
> reduce the race window to read mm->sc_stat.pcpu_sched?
Agree. Since we now solely use visited_cpus to filter CPUs and get_scan_cpumasks()
has been removed, this inner loop will no longer incorrectly skip those CPUs
in cross-node LLC domains. Thus, the issue previously raised by Sashiko
no longer exists.


> 
> [ ... ]
> 
> I ran hackbench[1] with above changes on Intel Sapphire Rapids with memory interleave enabled,
> and on AMD Milan. There is no much difference on Sapphire Rapids, while a slight regress on
> Milan AMD EPYC 9554P 64-Core when NUMA balancing is enabled(but it looks to be within
> run-to-run variance) Please double check if this is the case on your AMD server.
> And you can add my tag in next version:
> 
> Tested-by: Chen Yu <yu.c.chen@intel.com>
> 
> 
> Data on Milan
> ./launch.sh compare hackbench base_numab visit_numab
> =========================================
> Hackbench Comparison: base_numab vs visit_numab
> =========================================
> MODE       GROUPS  FDS |      base_numab(s) |     visit_numab(s) | DIFF(%) | VERDICT
> ---------- ------ -----+--------------------+--------------------+------------+-----------
> process         1   10 |     34.564 ±0.53% |     34.103 ±1.48% | 1.33% | IMPROVED
> process         1   16 |     60.556 ±0.52% |     62.296 ±2.11% | -2.87% | REGRESSED
> process         1    2 |      2.981 ±0.03% |     3.814 ±21.34% | -27.94% | REGRESSED <--run-to-run variance
> process         1   20 |     80.683 ±1.51% |     81.392 ±0.28% | -0.88% | REGRESSED
> process         1    4 |     8.446 ±12.50% |      7.185 ±6.86% | 14.93% | IMPROVED
> process         1    6 |     11.355 ±1.09% |     11.775 ±0.98% | -3.70% | REGRESSED
> process         1    8 |     19.282 ±6.61% |     19.976 ±7.10% | -3.60% | REGRESSED
> process         2   10 |     36.729 ±2.08% |     36.899 ±1.25% | -0.46% | REGRESSED
> process         2   16 |     63.758 ±0.47% |     63.373 ±0.74% | 0.60% | IMPROVED
> process         2    2 |      3.240 ±4.01% |      3.289 ±6.17% | -1.51% | REGRESSED
> process         2   20 |     85.519 ±1.97% |     83.396 ±0.26% | 2.48% | IMPROVED
> process         2    4 |      8.551 ±3.08% |      9.132 ±9.56% | -6.79% | REGRESSED
> process         2    6 |    15.256 ±10.90% |     11.869 ±1.21% | 22.20% | IMPROVED
> process         2    8 |     26.729 ±2.24% |    25.681 ±11.10% | 3.92% | IMPROVED
> process         4   10 |     39.921 ±1.01% |     39.274 ±0.37% | 1.62% | IMPROVED
> process         4   16 |     74.032 ±2.08% |     83.334 ±5.79% | -12.56% | REGRESSED
> process         4    2 |     4.339 ±11.59% |     4.367 ±10.33% | -0.65% | REGRESSED
> process         4   20 |    120.967 ±0.67% |    124.640 ±3.31% | -3.04% | REGRESSED
> process         4    4 |     11.088 ±4.61% |     11.250 ±6.90% | -1.46% | REGRESSED
> process         4    6 |     21.023 ±0.77% |     20.639 ±1.40% | 1.83% | IMPROVED
> process         4    8 |     31.368 ±0.36% |     30.306 ±0.77% | 3.39% | IMPROVED
> process         8   10 |     56.356 ±1.47% |     55.669 ±0.98% | 1.22% | IMPROVED
> process         8   16 |    144.752 ±0.84% |    139.409 ±0.61% | 3.69% | IMPROVED
> process         8    2 |      5.697 ±4.14% |      5.947 ±5.28% | -4.39% | REGRESSED
> process         8   20 |    220.395 ±0.32% |    215.589 ±0.66% | 2.18% | IMPROVED
> process         8    4 |     14.589 ±7.09% |     14.148 ±2.56% | 3.02% | IMPROVED
> process         8    6 |     23.872 ±1.39% |     24.558 ±4.92% | -2.87% | REGRESSED
> process         8    8 |     34.434 ±0.60% |     36.422 ±4.85% | -5.77% | REGRESSED
> threads         1   10 |     35.582 ±0.62% |     35.697 ±0.94% | -0.32% | REGRESSED
> threads         1   16 |     62.996 ±1.36% |     63.255 ±0.65% | -0.41% | REGRESSED
> threads         1    2 |      3.216 ±0.16% |      3.253 ±1.20% | -1.15% | REGRESSED
> threads         1   20 |     83.611 ±1.53% |     82.771 ±0.23% | 1.00% | IMPROVED
> threads         1    4 |      6.952 ±0.50% |      6.957 ±0.06% | -0.07% | REGRESSED
> threads         1    6 |     14.028 ±3.49% |     12.997 ±0.41% | 7.35% | IMPROVED
> threads         1    8 |     21.511 ±1.71% |     21.395 ±2.73% | 0.54% | IMPROVED
> threads         2   10 |     37.655 ±1.42% |     38.516 ±0.54% | -2.29% | REGRESSED
> threads         2   16 |     67.366 ±2.88% |     68.230 ±3.80% | -1.28% | REGRESSED
> threads         2    2 |      3.301 ±1.21% |      3.344 ±1.50% | -1.30% | REGRESSED
> threads         2   20 |     94.511 ±1.12% |     87.065 ±0.73% | 7.88% | IMPROVED
> threads         2    4 |      8.793 ±7.80% |     10.709 ±3.75% | -21.79% | REGRESSED
> threads         2    6 |    16.934 ±11.50% |    16.483 ±15.58% | 2.66% | IMPROVED
> threads         2    8 |     26.700 ±7.11% |     25.430 ±3.37% | 4.76% | IMPROVED
> threads         4   10 |     40.121 ±1.36% |     40.582 ±2.06% | -1.15% | REGRESSED
> threads         4   16 |     83.463 ±2.77% |     79.598 ±0.41% | 4.63% | IMPROVED
> threads         4    2 |      4.855 ±3.95% |     4.753 ±15.72% | 2.10% | IMPROVED
> threads         4   20 |    134.076 ±0.44% |    131.447 ±1.24% | 1.96% | IMPROVED
> threads         4    4 |     13.193 ±6.68% |    11.804 ±13.90% | 10.53% | IMPROVED
> threads         4    6 |     21.001 ±3.50% |     18.046 ±0.96% | 14.07% | IMPROVED
> threads         4    8 |     30.247 ±2.34% |     28.422 ±2.57% | 6.03% | IMPROVED
> threads         8   10 |     58.323 ±0.46% |     60.638 ±1.94% | -3.97% | REGRESSED
> threads         8   16 |    149.357 ±0.83% |    153.554 ±0.57% | -2.81% | REGRESSED
> threads         8    2 |      5.157 ±3.26% |      5.542 ±6.69% | -7.47% | REGRESSED
> threads         8   20 |    227.619 ±0.40% |    229.307 ±1.23% | -0.74% | REGRESSED
> threads         8    4 |     15.586 ±3.65% |     14.631 ±0.11% | 6.13% | IMPROVED
> threads         8    6 |     25.401 ±2.47% |     24.989 ±2.39% | 1.62% | IMPROVED
> threads         8    8 |     36.215 ±3.22% |     37.484 ±3.35% | -3.50% | REGRESSED
> 
> Note: Values shown as mean ±%stddev (seconds). DIFF based on means.
>        DIFF(%) = (base_numab - visit_numab) / base_numab * 100
>        Positive = improvement (visit_numab faster), Negative = regression.
> 
> 
> [1] https://github.com/chen-yu-surf/bench_tool.git
> 

Thanks for the testing and the tag!

I believe that the results are likely due to run-to-run variance. Below are my test
results, and I noticed that some test cases regressed on my side while they improved
on yours. For example, the threads 2 2 case showed completely opposite trends between
our environments.

Data AMD EPYC 9654
=========================================
Hackbench comparison
BASE: base_numab_0728
TEST: visit_numab_0728
=========================================
MODE     G  FD |         BASE(s) |         TEST(s) |   DIFF% | RESULT
------- -- ----+-----------------+-----------------+---------+----------
process  1  10 |     63.856/4.7% |     65.950/0.8% |  -3.28% | REGRESSED
process  1  16 |    107.365/1.5% |    106.472/1.0% |   0.83% | IMPROVED
process  1   2 |      9.813/0.7% |      9.215/3.8% |   6.09% | IMPROVED
process  1  20 |    130.314/0.9% |    132.356/0.9% |  -1.57% | REGRESSED
process  1   4 |     18.624/2.9% |     19.663/6.3% |  -5.58% | REGRESSED
process  1   6 |     33.364/3.5% |     32.145/1.4% |   3.65% | IMPROVED
process  1   8 |     45.797/4.5% |    45.128/10.7% |   1.46% | IMPROVED
process  2  10 |     63.760/1.1% |     64.099/0.4% |  -0.53% | REGRESSED
process  2  16 |    107.555/1.3% |    105.975/0.6% |   1.47% | IMPROVED
process  2   2 |      9.995/1.5% |      9.794/3.1% |   2.01% | IMPROVED
process  2  20 |    132.837/0.3% |    133.416/1.5% |  -0.44% | REGRESSED
process  2   4 |     19.617/3.5% |     21.275/5.1% |  -8.45% | REGRESSED
process  2   6 |     33.962/3.2% |     34.376/2.6% |  -1.22% | REGRESSED
process  2   8 |     47.400/4.2% |     49.125/1.2% |  -3.64% | REGRESSED
process  4  10 |     63.588/1.6% |     64.254/0.9% |  -1.05% | REGRESSED
process  4  16 |    106.697/0.6% |    105.565/0.8% |   1.06% | IMPROVED
process  4   2 |      8.720/4.4% |      9.915/4.9% | -13.70% | REGRESSED
process  4  20 |    132.137/0.9% |    133.264/0.3% |  -0.85% | REGRESSED
process  4   4 |     22.398/3.8% |     19.267/4.1% |  13.98% | IMPROVED
process  4   6 |     33.693/2.8% |     35.592/0.7% |  -5.64% | REGRESSED
process  4   8 |     50.416/2.5% |     49.572/2.9% |   1.67% | IMPROVED
process  8  10 |     63.858/0.9% |     65.193/0.9% |  -2.09% | REGRESSED
process  8  16 |    106.599/1.0% |    106.779/0.6% |  -0.17% | REGRESSED
process  8   2 |      9.581/4.7% |      8.549/4.1% |  10.77% | IMPROVED
process  8  20 |    134.607/0.4% |    133.946/1.2% |   0.49% | IMPROVED
process  8   4 |     20.298/2.6% |     22.213/3.6% |  -9.43% | REGRESSED
process  8   6 |     36.864/1.3% |     36.373/0.9% |   1.33% | IMPROVED
process  8   8 |     51.013/1.6% |     51.373/0.9% |  -0.71% | REGRESSED
threads  1  10 |     64.432/1.4% |     65.835/2.6% |  -2.18% | REGRESSED
threads  1  16 |    109.816/3.2% |     99.865/3.9% |   9.06% | IMPROVED
threads  1   2 |     7.246/30.6% |      5.060/0.7% |  30.17% | IMPROVED
threads  1  20 |    116.748/0.6% |    121.446/4.4% |  -4.02% | REGRESSED
threads  1   4 |    18.362/20.5% |     11.685/1.5% |  36.36% | IMPROVED
threads  1   6 |     31.788/1.7% |     35.312/4.7% | -11.09% | REGRESSED
threads  1   8 |     44.421/3.6% |     51.299/6.4% | -15.48% | REGRESSED
threads  2  10 |     65.545/4.0% |     70.301/2.3% |  -7.26% | REGRESSED
threads  2  16 |     96.461/5.7% |    107.936/1.9% | -11.90% | REGRESSED
threads  2   2 |      5.634/5.5% |      5.472/2.3% |   2.88% | IMPROVED
threads  2  20 |    115.252/2.2% |    134.723/0.5% | -16.89% | REGRESSED
threads  2   4 |    23.177/10.5% |     20.270/9.2% |  12.54% | IMPROVED
threads  2   6 |     40.291/4.0% |     38.460/3.2% |   4.54% | IMPROVED
threads  2   8 |     50.297/6.5% |     49.422/2.8% |   1.74% | IMPROVED
threads  4  10 |     60.081/6.4% |     65.470/2.2% |  -8.97% | REGRESSED
threads  4  16 |    104.858/5.6% |    109.031/1.7% |  -3.98% | REGRESSED
threads  4   2 |     10.508/4.1% |     10.232/2.7% |   2.63% | IMPROVED
threads  4  20 |    136.411/1.2% |    134.034/1.6% |   1.74% | IMPROVED
threads  4   4 |     23.156/4.8% |     24.758/3.9% |  -6.92% | REGRESSED
threads  4   6 |     36.906/3.9% |     40.785/0.6% | -10.51% | REGRESSED
threads  4   8 |     48.687/6.0% |     53.476/4.8% |  -9.84% | REGRESSED
threads  8  10 |     64.297/4.5% |     65.892/0.4% |  -2.48% | REGRESSED
threads  8  16 |    110.432/0.4% |    109.087/1.6% |   1.22% | IMPROVED
threads  8   2 |      9.147/5.5% |      9.948/7.2% |  -8.76% | REGRESSED
threads  8  20 |    138.656/1.1% |    138.635/0.8% |   0.02% | IMPROVED
threads  8   4 |     21.609/4.1% |     24.328/5.5% | -12.58% | REGRESSED
threads  8   6 |     42.429/4.2% |     38.484/2.4% |   9.30% | IMPROVED
threads  8   8 |     52.500/3.0% |     52.328/0.9% |   0.33% | IMPROVED

thanks,
Gengkun
>