From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout11.his.huawei.com (canpmsgout11.his.huawei.com [113.46.200.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E8C3423765 for ; Thu, 9 Jul 2026 12:38:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.226 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783600685; cv=none; b=NQ32GnsjOGPqTJDUxQYnxfipwyWEHObHPs91jxEihGy/VoK76ytC/ye0TK95qONmh4aRmkoQrX0aJu/usylnrjBTgIDR4yk5QGc37C12QJdtW8H6l53gDgWsmvIqR6tToR2vP8J7d8khnqUbhaXdf3roYBsiSV7TKbzKpl+MpXk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783600685; c=relaxed/simple; bh=4YZNhs2TGsTXRr0tK8UaXxQvyTeOQqFIx4ooiE/G+0w=; h=From:To:CC:Subject:Date:Message-ID:MIME-Version:Content-Type; b=hRSQ7m0MYjuIRCYgzPiUL+TuNgjH5xha8ONq4LqjhvK0geamyt3rCTYWaxqH83MgK3yAiST/CsU4ewSM5MySKT5fsAN1t9NdaX55g1HZ1kRmoF4S8ac1GiUCt/WcC023m917bpRKe+qRLbP5EfqL2CSb7KzbCmdbJ0jIOkK/zp8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=zMInuzLp; arc=none smtp.client-ip=113.46.200.226 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="zMInuzLp" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=X8qi+FGnIxRY0r20y5UUuDDEOdTNzEcKoL3nWBHR71g=; b=zMInuzLpoEneB8hyUpWVSPsSdfCMgRKA8ugt7wIRsU6FoyXDuyE8hZ31KQ/KciocVwBSIZH8k ol0FbtMiCulNAcHIcOAwjVzsEZI+0OzKaY1ItbCiUshH0sDYDnHn8usCPX61K/Ecorh37DLlTT6 WHy93cyE+/cfvzm37wK6Xk4= Received: from mail.maildlp.com (unknown [172.19.163.163]) by canpmsgout11.his.huawei.com (SkyGuard) with ESMTPS id 4gwvQL3gfKzKm73; Thu, 9 Jul 2026 20:28:42 +0800 (CST) Received: from kwepemj100017.china.huawei.com (unknown [7.202.194.11]) by mail.maildlp.com (Postfix) with ESMTPS id B845D4057A; Thu, 9 Jul 2026 20:37:59 +0800 (CST) Received: from huawei.com (10.67.174.193) by kwepemj100017.china.huawei.com (7.202.194.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Thu, 9 Jul 2026 20:37:58 +0800 From: Luo Gengkun To: , , , , , CC: , , , , , , Subject: [PATCH v5 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Date: Thu, 9 Jul 2026 13:00:51 +0000 Message-ID: <20260709130053.2749834-1-luogengkun2@huawei.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: kwepems200002.china.huawei.com (7.221.188.68) To kwepemj100017.china.huawei.com (7.202.194.11) Hi everyone, The cache-aware scheduling patches have now been merged into the mainline. The goal of this series is to reduce the overhead of task_cache_work() by minimizing the number of scanned CPUs, which yields significant performance gains in multi-instance scenarios like Redis. To facilitate testing, a debug patch is appended. Below are the Redis test results conducted on AMD server. valkey-benchmark rps | baseline | schedcache | schedcache_visit | (no cache aware) | (without this series) | (with this series) ----------------------+---------------------+---------------------+-----------+-------------------+-------- | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%) 200000 | 0.26 | 0.383 | -47.3% | 0.264 | -1.5% 300000 | 0.343 | 0.475 | -38.4% | 0.35 | -2.0% 400000 | 0.445 | 0.567 | -27.4% | 0.453 | -1.7% Additionally, the output of perf top -e cycles:k highlights the overhead incurred by task_cache_work: valkey-benchmark rps | schedcache | schedcache_visit ----------------------+-------------------------------------------+--------------------------------------- 200000 | 1.12% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work 300000 | 0.92% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work 400000 | 0.82% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work The trace data is shown below: echo NO_SC_VISIT > /sys/kernel/debug/sched/features redis-server-24660 [195] ..... 2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384 valkey-benchmar-48836 [317] ..... 2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384 valkey-benchmar-48487 [310] ..... 2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384 redis-server-24004 [005] ..... 2916.006016: sched_cache_scan: comm=redis-server pid=24004 scan=384 valkey-benchmar-48810 [322] ..... 2916.006017: sched_cache_scan: comm=valkey-benchmar pid=48810 scan=384 redis-server-29634 [287] ..... 2916.006024: sched_cache_scan: comm=redis-server pid=29634 scan=384 redis-server-26480 [081] ..... 2916.006027: sched_cache_scan: comm=redis-server pid=26480 scan=384 redis-server-29582 [086] ..... 2916.006027: sched_cache_scan: comm=redis-server pid=29582 scan=384 echo SC_VISIT > /sys/kernel/debug/sched/features redis-server-29080 [225] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14 redis-server-26694 [018] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13 valkey-benchmar-48198 [103] ..... 2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16 redis-server-26825 [269] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26825 scan=1 valkey-benchmar-48878 [295] ..... 2876.257910: sched_cache_scan: comm=valkey-benchmar pid=48878 scan=16 valkey-benchmar-47522 [290] ..... 2876.257910: sched_cache_scan: comm=valkey-benchmar pid=47522 scan=16 redis-server-25282 [048] ..... 2876.257910: sched_cache_scan: comm=redis-server pid=25282 scan=1 redis-server-29634 [071] ..... 2876.257910: sched_cache_scan: comm=redis-server pid=29634 scan=2 The above results show that this series significantly reduces the overhead of task_cache_work() by decreasing the number of CPUs that need to be scanned, from 384 to 16. Hackbench testing was also conducted. The results below show that this series does not affect the accuracy of cache-aware scheduling. ========================================= Hackbench Comparison: baseline vs schedcache ========================================= MODE GROUPS FDS | baseline(std) | schedcache(std) | DIFF(%) | VERDICT ---------- ------ -----+--------------------+--------------------+------------+----------- threads 1 10 | 113.200 (4.22%) | 67.300 (1.32%) | 40.55% | IMPROVED threads 1 2 | 16.555 (4.11%) | 11.020 (1.66%) | 33.43% | IMPROVED threads 1 20 | 250.774 (1.26%) | 265.026 (5.44%) | -5.68% | REGRESSED threads 1 4 | 42.117 (1.44%) | 27.758 (1.64%) | 34.09% | IMPROVED threads 1 6 | 65.140 (4.31%) | 39.182 (1.38%) | 39.85% | IMPROVED threads 1 8 | 84.286 (1.29%) | 53.721 (1.58%) | 36.26% | IMPROVED threads 2 10 | 122.592 (0.44%) | 113.365 (4.93%) | 7.53% | IMPROVED threads 2 2 | 17.702 (4.09%) | 10.473 (0.42%) | 40.84% | IMPROVED threads 2 20 | 336.457 (1.77%) | 314.108 (1.51%) | 6.64% | IMPROVED threads 2 4 | 43.989 (0.88%) | 27.067 (3.38%) | 38.47% | IMPROVED threads 2 6 | 69.322 (0.85%) | 41.707 (4.19%) | 39.84% | IMPROVED threads 2 8 | 103.767 (1.81%) | 58.518 (3.00%) | 43.61% | IMPROVED threads 4 10 | 148.882 (3.56%) | 149.449 (1.06%) | -0.38% | REGRESSED threads 4 2 | 18.909 (2.96%) | 11.063 (2.08%) | 41.49% | IMPROVED threads 4 20 | 724.943 (2.14%) | 631.222 (3.92%) | 12.93% | IMPROVED threads 4 4 | 48.191 (1.91%) | 27.352 (5.35%) | 43.24% | IMPROVED threads 4 6 | 79.725 (3.84%) | 78.732 (4.10%) | 1.25% | IMPROVED threads 4 8 | 108.768 (1.36%) | 105.928 (1.65%) | 2.61% | IMPROVED Hackbench Comparison: schedcache vs schedcache_visit ========================================= MODE GROUPS FDS | schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT ---------- ------ -----+--------------------+----------------------+---------+----------- threads 1 10 | 67.300 (1.32%) | 67.014 (0.96%) | 0.42% | IMPROVED threads 1 2 | 11.020 (1.66%) | 10.557 (1.46%) | 4.20% | IMPROVED threads 1 20 | 265.026 (5.44%) | 212.366 (16.32%) | 19.87% | IMPROVED threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED threads 1 6 | 39.182 (1.38%) | 38.914 (0.34%) | 0.68% | IMPROVED threads 1 8 | 53.721 (1.58%) | 52.889 (0.27%) | 1.55% | IMPROVED threads 2 10 | 121.203 (6.99%) | 124.254 (1.38%) | -2.52% | REGRESSED threads 2 2 | 10.473 (0.42%) | 11.206 (5.91%) | -7.00% | REGRESSED threads 2 20 | 314.108 (1.51%) | 301.754 (1.95%) | 3.93% | IMPROVED threads 2 4 | 27.067 (3.38%) | 28.028 (2.01%) | -3.55% | REGRESSED threads 2 6 | 41.707 (4.19%) | 42.149 (3.35%) | -1.06% | REGRESSED threads 2 8 | 58.518 (3.00%) | 57.133 (4.39%) | 2.37% | IMPROVED threads 4 10 | 149.449 (1.06%) | 141.407 (0.08%) | 5.38% | IMPROVED threads 4 2 | 11.063 (2.08%) | 11.360 (5.85%) | -2.68% | REGRESSED threads 4 20 | 631.222 (3.92%) | 622.780 (2.49%) | 1.34% | IMPROVED threads 4 4 | 27.352 (5.35%) | 27.947 (5.37%) | -2.18% | REGRESSED threads 4 6 | 78.732 (4.10%) | 73.911 (0.70%) | 6.12% | IMPROVED threads 4 8 | 105.928 (1.65%) | 107.535 (3.29%) | -1.52% | REGRESSED --- Changes history **v5 Changes:** 1. Restore get_scan_cpumasks() to avoid violating NUMA_BALANCING constraints. 2. Use for_each_cpu_and() to filter CPUs within the LLC domain. Link to v4: https://lore.kernel.org/all/20260618064300.80072-1-luogengkun2@huawei.com/ **v4 Changes:** 1. Rebase to the master. 2. epoch_timeout is introduced to evict expired CPUs instead of relying on epoch, because epoch is refreshed periodically due to invocations of fraction_mm_sched(). 3. Move the increasement of nr_running before fraction_mm_sched(). 4. Remove the redundant 'work->next' reset at the end of task_cache_work(). 5. Add a debug patch to show the number of CPUs scanned to show the benefit of this optimization. Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/ **v3 Changes:** 1. Remove the static key and enable this feature by default. 2. Reuse llc_epoch_affinity_timeout instead of introducing llc_epoch_visited_timeout. 3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into fraction_mm_sched() to avoid race between task_cache_work() and __update_mm_sched(). 4. Reset work->next at the end of task_cache_work() to prevent concurrent executions by multiple threads within the same process. Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/ **v2 Changes:** 1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead. 2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead. Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/ --- Luo Gengkun (2): sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost include/linux/sched.h | 2 ++ include/trace/events/sched.h | 21 +++++++++++++++ kernel/sched/fair.c | 52 +++++++++++++++++++++++++++++------- kernel/sched/features.h | 1 + 4 files changed, 66 insertions(+), 10 deletions(-) -- 2.34.1