From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout06.his.huawei.com (canpmsgout06.his.huawei.com [113.46.200.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0975D418A39 for ; Tue, 28 Jul 2026 08:54:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.221 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785228854; cv=none; b=f1rA0V+Fbj+Uzq4AT8zXIQtE2YmZeMsJKYfub+ZSwWEM4HsoNeSY8rbyWH3JWAE4Va1DDYUSfRzMJ3o48vSmpl/oF3BbhVBg0szZ+gIYmkfG5rxXx3TaeTZ9fYEhJ8Z/PrfB9LXRHmkarq6+BDzusVgZcRj1LszMb15KoPR0Qms= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785228854; c=relaxed/simple; bh=OsHwWVEaubIYY/y6qv3DKQSBLjUzMre0qQC1MwE4Njk=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=DBIEKJ7izq3FX00hzFuZq/hTdF8W0TCJ9WaJOHX6sse7/NdL/i4HQUzlgChMgv6Wdvpssi8kQSGv0hBE8ZCtoRTFYTLcMTedR0KSDYGkAwAg0PkJV4k/qPHg9aU80Vmr7vr8AAV+4/QFfJD5Xngu0QNfFGZ9P8ylrJXgtm1Lumw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=24KY1FGM; arc=none smtp.client-ip=113.46.200.221 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="24KY1FGM" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=GS96A1bYlH4QcQAuqFWLoUFTNET1yZD4VlFENQVT+a8=; b=24KY1FGMuGwufF3ute1jISgagoXcbubNhqp+nM/Er8fBqqZvYuffxJ0SrgzNNIuDmGLmgm+LX SsXp56tB8TjQGCNhz/fXMVkvPzDoZImoCYKgScE/i7LP1MB02o+i8BFyvx0nLep8JB7hbIzWjEU YIwpaXBTArahT3xwVOq8tvM= Received: from mail.maildlp.com (unknown [172.19.162.223]) by canpmsgout06.his.huawei.com (SkyGuard) with ESMTPS id 4h8TY007kTzRhR0; Tue, 28 Jul 2026 16:44:36 +0800 (CST) Received: from kwepemj100017.china.huawei.com (unknown [7.202.194.11]) by mail.maildlp.com (Postfix) with ESMTPS id 265B240561; Tue, 28 Jul 2026 16:54:01 +0800 (CST) Received: from [10.67.108.244] (10.67.108.244) by kwepemj100017.china.huawei.com (7.202.194.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Tue, 28 Jul 2026 16:54:00 +0800 Message-ID: <1dc03c84-9bc2-4db1-bab4-3f603fba54cd@huawei.com> Date: Tue, 28 Jul 2026 16:53:59 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v8 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Content-Language: en-US To: "Chen, Yu C" CC: , , , , , , , , , , , , "chen.yu@linux.dev" References: <20260723040429.630176-1-luogengkun2@huawei.com> <20260723040429.630176-2-luogengkun2@huawei.com> From: Luo Gengkun In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: kwepems500002.china.huawei.com (7.221.188.17) To kwepemj100017.china.huawei.com (7.202.194.11) On 2026/7/27 9:09, Chen, Yu C wrote: > On 7/23/2026 12:04 PM, Luo Gengkun wrote: > > [ ... ] > >> +static unsigned long fraction_mm_sched(int cpu, >> +                       struct mm_struct *mm) >>   { >> +    struct sched_cache_time *pcpu_sched = >> +        per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); >> +    struct rq *rq = cpu_rq(cpu); >> + >>       guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); >>       __update_mm_sched(rq, pcpu_sched); >> +    /* Skip the rq that has not been hit for a long time */ >> +    if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) { > > In v2 there is a check if the cpu has been set before writing: > cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus) > https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/ > do we need to bring that back? > I don't think we need it back. Here is why: In v2, for_each_cpu was used instead of for_each_cpu_and in the inner loop, meaning some CPUs being checked might not have been set. Therefore, cpumask_test_cpu was necessary to filter out those cases. Now, with for_each_cpu_and(i, sched_domain_span(sd), &mm->sc_stat.visited_cpus), we can ensure each scanned CPU is set, so the issue no longer exists. Furthermore, the only place where the visited_cpus bits are cleared is task_cache_work(), which is only called once per scan period, there is no risk of the bit being cleared concurrently mid-loop. However, is there a possibility that the current task_cache_work() execution hasn't finished yet when the next scan window arrives? For instance, if the current task work is heavily delayed or preempted by unexpected interrupt, jiffies could advance past next_scan before the loop completes. If we move the `work->next = work;` to the very end of task_cache_work(), would that resolve this issue? By doing so, the existing `work->next == work` check in task_tick_cache() should fail and no new task work will be submitted. Please let me know if I'm missing something. >> +        cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus); >> +        return 0; >> +    } >> + > > [ ... ] > >> -            for_each_cpu(i, sched_domain_span(sd)) { >> -                occ = fraction_mm_sched(cpu_rq(i), >> -                            per_cpu_ptr(mm->sc_stat.pcpu_sched, i)); >> +            for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) { > > Does using > for_each_cpu_and(i, sched_domain_span(sd), cpus) > reduce the race window to read mm->sc_stat.pcpu_sched? Agree. Since we now solely use visited_cpus to filter CPUs and get_scan_cpumasks() has been removed, this inner loop will no longer incorrectly skip those CPUs in cross-node LLC domains. Thus, the issue previously raised by Sashiko no longer exists. > > [ ... ] > > I ran hackbench[1] with above changes on Intel Sapphire Rapids with memory interleave enabled, > and on AMD Milan. There is no much difference on Sapphire Rapids, while a slight regress on > Milan AMD EPYC 9554P 64-Core when NUMA balancing is enabled(but it looks to be within > run-to-run variance) Please double check if this is the case on your AMD server. > And you can add my tag in next version: > > Tested-by: Chen Yu > > > Data on Milan > ./launch.sh compare hackbench base_numab visit_numab > ========================================= > Hackbench Comparison: base_numab vs visit_numab > ========================================= > MODE       GROUPS  FDS |      base_numab(s) |     visit_numab(s) | DIFF(%) | VERDICT > ---------- ------ -----+--------------------+--------------------+------------+----------- > process         1   10 |     34.564 ±0.53% |     34.103 ±1.48% | 1.33% | IMPROVED > process         1   16 |     60.556 ±0.52% |     62.296 ±2.11% | -2.87% | REGRESSED > process         1    2 |      2.981 ±0.03% |     3.814 ±21.34% | -27.94% | REGRESSED <--run-to-run variance > process         1   20 |     80.683 ±1.51% |     81.392 ±0.28% | -0.88% | REGRESSED > process         1    4 |     8.446 ±12.50% |      7.185 ±6.86% | 14.93% | IMPROVED > process         1    6 |     11.355 ±1.09% |     11.775 ±0.98% | -3.70% | REGRESSED > process         1    8 |     19.282 ±6.61% |     19.976 ±7.10% | -3.60% | REGRESSED > process         2   10 |     36.729 ±2.08% |     36.899 ±1.25% | -0.46% | REGRESSED > process         2   16 |     63.758 ±0.47% |     63.373 ±0.74% | 0.60% | IMPROVED > process         2    2 |      3.240 ±4.01% |      3.289 ±6.17% | -1.51% | REGRESSED > process         2   20 |     85.519 ±1.97% |     83.396 ±0.26% | 2.48% | IMPROVED > process         2    4 |      8.551 ±3.08% |      9.132 ±9.56% | -6.79% | REGRESSED > process         2    6 |    15.256 ±10.90% |     11.869 ±1.21% | 22.20% | IMPROVED > process         2    8 |     26.729 ±2.24% |    25.681 ±11.10% | 3.92% | IMPROVED > process         4   10 |     39.921 ±1.01% |     39.274 ±0.37% | 1.62% | IMPROVED > process         4   16 |     74.032 ±2.08% |     83.334 ±5.79% | -12.56% | REGRESSED > process         4    2 |     4.339 ±11.59% |     4.367 ±10.33% | -0.65% | REGRESSED > process         4   20 |    120.967 ±0.67% |    124.640 ±3.31% | -3.04% | REGRESSED > process         4    4 |     11.088 ±4.61% |     11.250 ±6.90% | -1.46% | REGRESSED > process         4    6 |     21.023 ±0.77% |     20.639 ±1.40% | 1.83% | IMPROVED > process         4    8 |     31.368 ±0.36% |     30.306 ±0.77% | 3.39% | IMPROVED > process         8   10 |     56.356 ±1.47% |     55.669 ±0.98% | 1.22% | IMPROVED > process         8   16 |    144.752 ±0.84% |    139.409 ±0.61% | 3.69% | IMPROVED > process         8    2 |      5.697 ±4.14% |      5.947 ±5.28% | -4.39% | REGRESSED > process         8   20 |    220.395 ±0.32% |    215.589 ±0.66% | 2.18% | IMPROVED > process         8    4 |     14.589 ±7.09% |     14.148 ±2.56% | 3.02% | IMPROVED > process         8    6 |     23.872 ±1.39% |     24.558 ±4.92% | -2.87% | REGRESSED > process         8    8 |     34.434 ±0.60% |     36.422 ±4.85% | -5.77% | REGRESSED > threads         1   10 |     35.582 ±0.62% |     35.697 ±0.94% | -0.32% | REGRESSED > threads         1   16 |     62.996 ±1.36% |     63.255 ±0.65% | -0.41% | REGRESSED > threads         1    2 |      3.216 ±0.16% |      3.253 ±1.20% | -1.15% | REGRESSED > threads         1   20 |     83.611 ±1.53% |     82.771 ±0.23% | 1.00% | IMPROVED > threads         1    4 |      6.952 ±0.50% |      6.957 ±0.06% | -0.07% | REGRESSED > threads         1    6 |     14.028 ±3.49% |     12.997 ±0.41% | 7.35% | IMPROVED > threads         1    8 |     21.511 ±1.71% |     21.395 ±2.73% | 0.54% | IMPROVED > threads         2   10 |     37.655 ±1.42% |     38.516 ±0.54% | -2.29% | REGRESSED > threads         2   16 |     67.366 ±2.88% |     68.230 ±3.80% | -1.28% | REGRESSED > threads         2    2 |      3.301 ±1.21% |      3.344 ±1.50% | -1.30% | REGRESSED > threads         2   20 |     94.511 ±1.12% |     87.065 ±0.73% | 7.88% | IMPROVED > threads         2    4 |      8.793 ±7.80% |     10.709 ±3.75% | -21.79% | REGRESSED > threads         2    6 |    16.934 ±11.50% |    16.483 ±15.58% | 2.66% | IMPROVED > threads         2    8 |     26.700 ±7.11% |     25.430 ±3.37% | 4.76% | IMPROVED > threads         4   10 |     40.121 ±1.36% |     40.582 ±2.06% | -1.15% | REGRESSED > threads         4   16 |     83.463 ±2.77% |     79.598 ±0.41% | 4.63% | IMPROVED > threads         4    2 |      4.855 ±3.95% |     4.753 ±15.72% | 2.10% | IMPROVED > threads         4   20 |    134.076 ±0.44% |    131.447 ±1.24% | 1.96% | IMPROVED > threads         4    4 |     13.193 ±6.68% |    11.804 ±13.90% | 10.53% | IMPROVED > threads         4    6 |     21.001 ±3.50% |     18.046 ±0.96% | 14.07% | IMPROVED > threads         4    8 |     30.247 ±2.34% |     28.422 ±2.57% | 6.03% | IMPROVED > threads         8   10 |     58.323 ±0.46% |     60.638 ±1.94% | -3.97% | REGRESSED > threads         8   16 |    149.357 ±0.83% |    153.554 ±0.57% | -2.81% | REGRESSED > threads         8    2 |      5.157 ±3.26% |      5.542 ±6.69% | -7.47% | REGRESSED > threads         8   20 |    227.619 ±0.40% |    229.307 ±1.23% | -0.74% | REGRESSED > threads         8    4 |     15.586 ±3.65% |     14.631 ±0.11% | 6.13% | IMPROVED > threads         8    6 |     25.401 ±2.47% |     24.989 ±2.39% | 1.62% | IMPROVED > threads         8    8 |     36.215 ±3.22% |     37.484 ±3.35% | -3.50% | REGRESSED > > Note: Values shown as mean ±%stddev (seconds). DIFF based on means. >       DIFF(%) = (base_numab - visit_numab) / base_numab * 100 >       Positive = improvement (visit_numab faster), Negative = regression. > > > [1] https://github.com/chen-yu-surf/bench_tool.git > Thanks for the testing and the tag! I believe that the results are likely due to run-to-run variance. Below are my test results, and I noticed that some test cases regressed on my side while they improved on yours. For example, the threads 2 2 case showed completely opposite trends between our environments. Data AMD EPYC 9654 ========================================= Hackbench comparison BASE: base_numab_0728 TEST: visit_numab_0728 ========================================= MODE G FD | BASE(s) | TEST(s) | DIFF% | RESULT ------- -- ----+-----------------+-----------------+---------+---------- process 1 10 | 63.856/4.7% | 65.950/0.8% | -3.28% | REGRESSED process 1 16 | 107.365/1.5% | 106.472/1.0% | 0.83% | IMPROVED process 1 2 | 9.813/0.7% | 9.215/3.8% | 6.09% | IMPROVED process 1 20 | 130.314/0.9% | 132.356/0.9% | -1.57% | REGRESSED process 1 4 | 18.624/2.9% | 19.663/6.3% | -5.58% | REGRESSED process 1 6 | 33.364/3.5% | 32.145/1.4% | 3.65% | IMPROVED process 1 8 | 45.797/4.5% | 45.128/10.7% | 1.46% | IMPROVED process 2 10 | 63.760/1.1% | 64.099/0.4% | -0.53% | REGRESSED process 2 16 | 107.555/1.3% | 105.975/0.6% | 1.47% | IMPROVED process 2 2 | 9.995/1.5% | 9.794/3.1% | 2.01% | IMPROVED process 2 20 | 132.837/0.3% | 133.416/1.5% | -0.44% | REGRESSED process 2 4 | 19.617/3.5% | 21.275/5.1% | -8.45% | REGRESSED process 2 6 | 33.962/3.2% | 34.376/2.6% | -1.22% | REGRESSED process 2 8 | 47.400/4.2% | 49.125/1.2% | -3.64% | REGRESSED process 4 10 | 63.588/1.6% | 64.254/0.9% | -1.05% | REGRESSED process 4 16 | 106.697/0.6% | 105.565/0.8% | 1.06% | IMPROVED process 4 2 | 8.720/4.4% | 9.915/4.9% | -13.70% | REGRESSED process 4 20 | 132.137/0.9% | 133.264/0.3% | -0.85% | REGRESSED process 4 4 | 22.398/3.8% | 19.267/4.1% | 13.98% | IMPROVED process 4 6 | 33.693/2.8% | 35.592/0.7% | -5.64% | REGRESSED process 4 8 | 50.416/2.5% | 49.572/2.9% | 1.67% | IMPROVED process 8 10 | 63.858/0.9% | 65.193/0.9% | -2.09% | REGRESSED process 8 16 | 106.599/1.0% | 106.779/0.6% | -0.17% | REGRESSED process 8 2 | 9.581/4.7% | 8.549/4.1% | 10.77% | IMPROVED process 8 20 | 134.607/0.4% | 133.946/1.2% | 0.49% | IMPROVED process 8 4 | 20.298/2.6% | 22.213/3.6% | -9.43% | REGRESSED process 8 6 | 36.864/1.3% | 36.373/0.9% | 1.33% | IMPROVED process 8 8 | 51.013/1.6% | 51.373/0.9% | -0.71% | REGRESSED threads 1 10 | 64.432/1.4% | 65.835/2.6% | -2.18% | REGRESSED threads 1 16 | 109.816/3.2% | 99.865/3.9% | 9.06% | IMPROVED threads 1 2 | 7.246/30.6% | 5.060/0.7% | 30.17% | IMPROVED threads 1 20 | 116.748/0.6% | 121.446/4.4% | -4.02% | REGRESSED threads 1 4 | 18.362/20.5% | 11.685/1.5% | 36.36% | IMPROVED threads 1 6 | 31.788/1.7% | 35.312/4.7% | -11.09% | REGRESSED threads 1 8 | 44.421/3.6% | 51.299/6.4% | -15.48% | REGRESSED threads 2 10 | 65.545/4.0% | 70.301/2.3% | -7.26% | REGRESSED threads 2 16 | 96.461/5.7% | 107.936/1.9% | -11.90% | REGRESSED threads 2 2 | 5.634/5.5% | 5.472/2.3% | 2.88% | IMPROVED threads 2 20 | 115.252/2.2% | 134.723/0.5% | -16.89% | REGRESSED threads 2 4 | 23.177/10.5% | 20.270/9.2% | 12.54% | IMPROVED threads 2 6 | 40.291/4.0% | 38.460/3.2% | 4.54% | IMPROVED threads 2 8 | 50.297/6.5% | 49.422/2.8% | 1.74% | IMPROVED threads 4 10 | 60.081/6.4% | 65.470/2.2% | -8.97% | REGRESSED threads 4 16 | 104.858/5.6% | 109.031/1.7% | -3.98% | REGRESSED threads 4 2 | 10.508/4.1% | 10.232/2.7% | 2.63% | IMPROVED threads 4 20 | 136.411/1.2% | 134.034/1.6% | 1.74% | IMPROVED threads 4 4 | 23.156/4.8% | 24.758/3.9% | -6.92% | REGRESSED threads 4 6 | 36.906/3.9% | 40.785/0.6% | -10.51% | REGRESSED threads 4 8 | 48.687/6.0% | 53.476/4.8% | -9.84% | REGRESSED threads 8 10 | 64.297/4.5% | 65.892/0.4% | -2.48% | REGRESSED threads 8 16 | 110.432/0.4% | 109.087/1.6% | 1.22% | IMPROVED threads 8 2 | 9.147/5.5% | 9.948/7.2% | -8.76% | REGRESSED threads 8 20 | 138.656/1.1% | 138.635/0.8% | 0.02% | IMPROVED threads 8 4 | 21.609/4.1% | 24.328/5.5% | -12.58% | REGRESSED threads 8 6 | 42.429/4.2% | 38.484/2.4% | 9.30% | IMPROVED threads 8 8 | 52.500/3.0% | 52.328/0.9% | 0.33% | IMPROVED thanks, Gengkun >