From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout07.his.huawei.com (canpmsgout07.his.huawei.com [113.46.200.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA7262D5436 for ; Thu, 18 Jun 2026 06:16:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.222 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781763393; cv=none; b=pcqhirE2HkAKj1VQGjVcP6h40/hexN3bOCChXYpLB4ez0RESXDc/QJdg6JoZ2x69ewprFzC+1x7pFCLsd+diPwkg6R8yzhb8sakzwz6YHgq9g+bWJRBHvuNHaoJ/kzCiTLKsdkN9Q6J+Egj1M1qMbdSPHLLWWADDEOygMSNwabo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781763393; c=relaxed/simple; bh=pAWYEEQhidfV5/P66n+f+4uYfz7w7uXF2Ai+1LYIHJs=; h=From:To:CC:Subject:Date:Message-ID:MIME-Version:Content-Type; b=itGRr9CQVx2tu7NKrpeUloU0+FGzPjwFyHX9Ku+Km+8P/x7GGZFRp/2Quj+yljVKaeATos4V1cePPEpi0DbSC8d4/3N0IUvCGTsOz+BJbSXiYVGssZzs4ADRVa1nzVL/aIa4Nf3eMAsSfVXdQALn9U+sNkKI128zWyl72l9/lyE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=l+mdNZ/p; arc=none smtp.client-ip=113.46.200.222 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="l+mdNZ/p" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=I8XSvU9l+SRETG7aiU1ZfiMgOOsUrENJ6/sNkvKyVBI=; b=l+mdNZ/p6MCnytrmgjir1Mlnu33OB2NtWyqBaV7n0mC2LifAhn4FLZUvXLEpfnSVqCTzJGpEj Lowt7bcup2c1wfjeOUMV7OKFg3zMF3eRJSsvunEmE48D4+vFad1OODWxFQtNoZorp+B2hJfCLXr cjbs4FSRjCN3ZD+UCbWVvbA= Received: from mail.maildlp.com (unknown [172.19.163.214]) by canpmsgout07.his.huawei.com (SkyGuard) with ESMTPS id 4ggqz96mF7zLlsR; Thu, 18 Jun 2026 14:08:21 +0800 (CST) Received: from kwepemj100017.china.huawei.com (unknown [7.202.194.11]) by mail.maildlp.com (Postfix) with ESMTPS id 95D2D40561; Thu, 18 Jun 2026 14:16:20 +0800 (CST) Received: from huawei.com (10.67.174.193) by kwepemj100017.china.huawei.com (7.202.194.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Thu, 18 Jun 2026 14:16:19 +0800 From: Luo Gengkun To: , , , , , CC: , , , , , , Subject: [PATCH v4 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Date: Thu, 18 Jun 2026 06:42:58 +0000 Message-ID: <20260618064300.80072-1-luogengkun2@huawei.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: kwepems100001.china.huawei.com (7.221.188.238) To kwepemj100017.china.huawei.com (7.202.194.11) Hi everyone, The cache-aware scheduling patches have now been merged into the mainline. The goal of this patch is to reduce the overhead of task_cache_work() by minimizing the number of scanned CPUs, which yields significant performance gains in multi-instance scenarios like Redis. To facilitate testing, this debug patch is introduced. Below are the benchmark results based on hackbench: Test steps: echo NO_SC_VISIT > /sys/kernel/debug/sched/features echo NO_SC_NODE > /sys/kernel/debug/sched/features echo 0 > /sys/kernel/debug/sched/llc_balancing/enabled ./launch.sh hackbench baseline echo NO_SC_VISIT > /sys/kernel/debug/sched/features echo SC_NODE > /sys/kernel/debug/sched/features echo 1 > /sys/kernel/debug/sched/llc_balancing/enabled ./launch.sh hackbench schedcache echo SC_VISIT > /sys/kernel/debug/sched/features echo NO_SC_NODE > /sys/kernel/debug/sched/features echo 1 > /sys/kernel/debug/sched/llc_balancing/enabled ./launch.sh hackbench schedcache_visit Test results: ./launch.sh compare hackbench baseline schedcache ========================================= Hackbench Comparison: baseline vs schedcache ========================================= MODE GROUPS FDS | baseline(std) | schedcache(std) | DIFF(%) | VERDICT ---------- ------ -----+--------------------+--------------------+------------+----------- threads 1 10 | 113.200 (4.22%) | 67.300 (1.32%) | 40.55% | IMPROVED threads 1 2 | 16.555 (4.11%) | 11.020 (1.66%) | 33.43% | IMPROVED threads 1 20 | 250.774 (1.26%) | 265.026 (5.44%) | -5.68% | REGRESSED threads 1 4 | 42.117 (1.44%) | 27.758 (1.64%) | 34.09% | IMPROVED threads 1 6 | 65.140 (4.31%) | 39.182 (1.38%) | 39.85% | IMPROVED threads 1 8 | 84.286 (1.29%) | 53.721 (1.58%) | 36.26% | IMPROVED threads 2 10 | 122.592 (0.44%) | 113.365 (4.93%) | 7.53% | IMPROVED threads 2 2 | 17.702 (4.09%) | 10.473 (0.42%) | 40.84% | IMPROVED threads 2 20 | 336.457 (1.77%) | 314.108 (1.51%) | 6.64% | IMPROVED threads 2 4 | 43.989 (0.88%) | 27.067 (3.38%) | 38.47% | IMPROVED threads 2 6 | 69.322 (0.85%) | 41.707 (4.19%) | 39.84% | IMPROVED threads 2 8 | 103.767 (1.81%) | 58.518 (3.00%) | 43.61% | IMPROVED threads 4 10 | 148.882 (3.56%) | 149.449 (1.06%) | -0.38% | REGRESSED threads 4 2 | 18.909 (2.96%) | 11.063 (2.08%) | 41.49% | IMPROVED threads 4 20 | 724.943 (2.14%) | 631.222 (3.92%) | 12.93% | IMPROVED threads 4 4 | 48.191 (1.91%) | 27.352 (5.35%) | 43.24% | IMPROVED threads 4 6 | 79.725 (3.84%) | 78.732 (4.10%) | 1.25% | IMPROVED threads 4 8 | 108.768 (1.36%) | 105.928 (1.65%) | 2.61% | IMPROVED Hackbench Comparison: schedcache vs schedcache_visit ========================================= MODE GROUPS FDS | schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT ---------- ------ -----+--------------------+----------------------+---------+----------- threads 1 10 | 67.300 (1.32%) | 67.014 (0.96%) | 0.42% | IMPROVED threads 1 2 | 11.020 (1.66%) | 10.557 (1.46%) | 4.20% | IMPROVED threads 1 20 | 265.026 (5.44%) | 212.366 (16.32%) | 19.87% | IMPROVED threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED threads 1 6 | 39.182 (1.38%) | 38.914 (0.34%) | 0.68% | IMPROVED threads 1 8 | 53.721 (1.58%) | 52.889 (0.27%) | 1.55% | IMPROVED threads 2 10 | 121.203 (6.99%) | 124.254 (1.38%) | -2.52% | REGRESSED threads 2 2 | 10.473 (0.42%) | 11.206 (5.91%) | -7.00% | REGRESSED threads 2 20 | 314.108 (1.51%) | 301.754 (1.95%) | 3.93% | IMPROVED threads 2 4 | 27.067 (3.38%) | 28.028 (2.01%) | -3.55% | REGRESSED threads 2 6 | 41.707 (4.19%) | 42.149 (3.35%) | -1.06% | REGRESSED threads 2 8 | 58.518 (3.00%) | 57.133 (4.39%) | 2.37% | IMPROVED threads 4 10 | 149.449 (1.06%) | 141.407 (0.08%) | 5.38% | IMPROVED threads 4 2 | 11.063 (2.08%) | 11.360 (5.85%) | -2.68% | REGRESSED threads 4 20 | 631.222 (3.92%) | 622.780 (2.49%) | 1.34% | IMPROVED threads 4 4 | 27.352 (5.35%) | 27.947 (5.37%) | -2.18% | REGRESSED threads 4 6 | 78.732 (4.10%) | 73.911 (0.70%) | 6.12% | IMPROVED threads 4 8 | 105.928 (1.65%) | 107.535 (3.29%) | -1.52% | REGRESSED --- Changes history **v4 Changes:** 1. Rebase to the master. 2. epoch_timeout is introduced to evict expired CPUs instead of relying on epoch, because epoch is refreshed periodically due to invocations of fraction_mm_sched(). 3. Move the increasement of nr_running before fraction_mm_sched(). 4. Remove the redundant 'work->next' reset at the end of task_cache_work(). 4. Add a debug patch to show the number of CPUs scanned to show the benefit of this optimization. Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/ **v3 Changes:** 1. Remove the static key and enable this feature by default. 2. Reuse llc_epoch_affinity_timeout instead of introducing llc_epoch_visited_timeout. 3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into fraction_mm_sched() to avoid race between task_cache_work() and __update_mm_sched(). 4. Reset work->next at the end of task_cache_work() to prevent concurrent executions by multiple threads within the same process. Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/ **v2 Changes:** 1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead. 2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead. --- Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/ --- Luo Gengkun (2): sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost include/linux/sched.h | 2 ++ include/trace/events/sched.h | 21 ++++++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++------- kernel/sched/features.h | 2 ++ 4 files changed, 62 insertions(+), 9 deletions(-) -- 2.34.1