From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout06.his.huawei.com (canpmsgout06.his.huawei.com [113.46.200.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 621C637D13B for ; Fri, 10 Apr 2026 10:12:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.221 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775815970; cv=none; b=T1UjwO5A9vKsBDwzK/6VyfezNfOboxiW+pJ0hCU1s6xg/jKEzl0Qmc26LHlK/b24eqeljm7InmLMiu5eQa3k5iG/yvU2dH+mZeqPd++0meQ04fqn7xblcVkDza1mf1JGmF1qeiIjoa/RVwl+IOBXNTXm02jZcxWawScGpHESqeI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775815970; c=relaxed/simple; bh=0QCrCy95bNqxW/kltYRd091mBJQPyz+FRjrL+zYGvKc=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=njs3208NgAMEG+L4RDcH5CXFgdqHq1jgXmF0HgyMyFDgX/gQW19KHHO0KHCA4USnUf+jZcKLsxy85GAxCkweFRPLaeNPeqbNzKas6SKPEXG+dHjc+VhBFf3CpCxCeQeXiB/rf4TMtNrNZcTaDZWQmTEphzH5fWS7wxA2bVtVeKM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=g6jO0cFn; arc=none smtp.client-ip=113.46.200.221 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="g6jO0cFn" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=k/zmVWiW+SogI+q1C/OMWSVvBxm/7lyiaAZ/J8PQf3w=; b=g6jO0cFn35cCFlpizg9zQ4qvfTb5MChc/tmv/UGHoL+sxuOzUKOi2lEWA4r4pj8fCrwLpfEoE mle7J/PvrL8jHxt/RjkIEaOWwVIgeZv4kEZIwRnzw99DdrPrWCVbCdaO2xkTvp/2O2K6a/wFjVo RKYR3shTYxrhJEcT/UzcOmc= Received: from mail.maildlp.com (unknown [172.19.162.140]) by canpmsgout06.his.huawei.com (SkyGuard) with ESMTPS id 4fsXWk2BGczRhQx; Fri, 10 Apr 2026 18:06:26 +0800 (CST) Received: from kwepemj100017.china.huawei.com (unknown [7.202.194.11]) by mail.maildlp.com (Postfix) with ESMTPS id 95605203C0; Fri, 10 Apr 2026 18:12:41 +0800 (CST) Received: from [10.67.108.244] (10.67.108.244) by kwepemj100017.china.huawei.com (7.202.194.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Fri, 10 Apr 2026 18:12:40 +0800 Message-ID: Date: Fri, 10 Apr 2026 18:12:39 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [Patch v4 02/22] sched/cache: Limit the scan number of CPUs when calculating task occupancy Content-Language: en-US To: Peter Zijlstra CC: Tim Chen , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Chen Yu , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , References: <57ed5fcec9b242803fe4ea2ce6e7f3de6a6efc6b.1775065312.git.tim.c.chen@linux.intel.com> <20260409134108.GA3102624@noisy.programming.kicks-ass.net> From: Luo Gengkun In-Reply-To: <20260409134108.GA3102624@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: kwepems100001.china.huawei.com (7.221.188.238) To kwepemj100017.china.huawei.com (7.202.194.11) On 2026/4/9 21:41, Peter Zijlstra wrote: > On Thu, Apr 09, 2026 at 09:17:10PM +0800, Luo Gengkun wrote: >> >> >> On 2026/4/2 5:52, Tim Chen wrote: >>> From: Chen Yu >>> >>> When NUMA balancing is enabled, the kernel currently iterates over all >>> online CPUs to aggregate process-wide occupancy data. On large systems, >>> this global scan introduces significant overhead. >>> >>> To reduce scan latency, limit the search to a subset of relevant CPUs: >>> 1. The task's preferred NUMA node. >>> 2. The node where the task is currently running. >>> 3. The node that contains the task's current preferred LLC.. >>> >>> While focusing solely on the preferred NUMA node is ideal, a >>> process-wide scan must remain flexible because the "preferred node" >>> is a per-task attribute. Different threads within the same process may >>> have different preferred nodes, causing the process-wide preference to >>> migrate. Maintaining a mask that covers both the preferred and active >>> running nodes ensures accuracy while significantly reducing the number of >>> CPUs inspected. >> >> To address the issue of scanning overhead, there is a more targeted >> approach: only scanning the CPUs actually accessed by the process, and >> evicting these CPUs when they remain unaccessed for a specific period of >> time. >> >> This significantly reduces unnecessary scanning in most scenario. I have >> attached the patch below for review. Please feel free to integrate or modify >> these changes. >> >> Thansk! >> Luo Gengkun > > Please fix your MUA, whatever you tried to send is horribly white space > mangled. > Apologies for the formatting mess. The following should display properly. From de33b29f31a882ac8c0038cac4ec659c7fdfe60d Mon Sep 17 00:00:00 2001 From: Luo Gengkun Date: Thu, 9 Apr 2026 07:22:16 +0000 Subject: [PATCH] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus. The overhead of task_cache_work is high, espeically in multi-NUMA system. Currently, task_cache_work try to find the pref_llc by scan all cpus in the system. However, most of these scans are meaningless, such as those for cpus that have never been visited or were accessed a long time ago. To address this problem, this patch introduces visited_cpus to track the visited cpus and uses llc_epoch_visited_timeout to evict cpus that have timed out. Signed-off-by: Luo Gengkun --- include/linux/sched.h | 1 + kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 19 ++++++++++++++++--- kernel/sched/sched.h | 1 + 4 files changed, 20 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index dfa4bfd099c6..f2327a13fda8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2390,6 +2390,7 @@ struct sched_cache_time { struct sched_cache_stat { struct sched_cache_time __percpu *pcpu_sched; + struct cpumask visited_cpus; raw_spinlock_t lock; unsigned long epoch; u64 nr_running_avg; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 4469e1c152c8..da66acabff4f 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -675,6 +675,8 @@ static __init int sched_init_debug(void) &llc_epoch_period); debugfs_create_u32("epoch_affinity_timeout", 0644, llc, &llc_epoch_affinity_timeout); + debugfs_create_u32("epoch_visited_timeout", 0644, llc, + &llc_epoch_visited_timeout); debugfs_create_u32("overaggr_pct", 0644, llc, &llc_overaggr_pct); debugfs_create_u32("imb_pct", 0644, llc, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e4e22696a0b1..4ccabc412aa1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1285,6 +1285,7 @@ static void set_next_buddy(struct sched_entity *se); __read_mostly unsigned int llc_aggr_tolerance = 1; __read_mostly unsigned int llc_epoch_period = EPOCH_PERIOD; __read_mostly unsigned int llc_epoch_affinity_timeout = EPOCH_LLC_AFFINITY_TIMEOUT; +__read_mostly unsigned int llc_epoch_visited_timeout = EPOCH_LLC_AFFINITY_TIMEOUT; __read_mostly unsigned int llc_imb_pct = 20; __read_mostly unsigned int llc_overaggr_pct = 50; @@ -1466,6 +1467,7 @@ void mm_init_sched(struct mm_struct *mm, raw_spin_lock_init(&mm->sc_stat.lock); mm->sc_stat.epoch = epoch; mm->sc_stat.cpu = -1; + cpumask_clear(&mm->sc_stat.visited_cpus); /* * The update to mm->sc_stat should not be reordered @@ -1582,6 +1584,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) pcpu_sched->runtime += delta_exec; rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; + cpumask_set_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus); } /* @@ -1724,7 +1727,10 @@ static void task_cache_work(struct callback_head *work) return; scoped_guard (cpus_read_lock) { - get_scan_cpumasks(cpus, p); + if (unlikely(llc_epoch_visited_timeout == 0)) + get_scan_cpumasks(cpus, p); + else + cpumask_and(cpus, cpu_online_mask, &mm->sc_stat.visited_cpus); for_each_cpu(cpu, cpus) { /* XXX sched_cluster_active */ @@ -1736,8 +1742,15 @@ static void task_cache_work(struct callback_head *work) continue; for_each_cpu(i, sched_domain_span(sd)) { - occ = fraction_mm_sched(cpu_rq(i), - per_cpu_ptr(mm->sc_stat.pcpu_sched, i)); + struct rq *rq = cpu_rq(i); + struct sched_cache_time *pcpu_sched = per_cpu_ptr(mm->sc_stat.pcpu_sched, i); + /* Skip the rq that has not been hit for a long time */ + if (llc_epoch_visited_timeout && (rq->cpu_epoch - pcpu_sched->epoch) > + llc_epoch_visited_timeout) { + cpumask_clear_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus); + continue; + } + occ = fraction_mm_sched(rq, pcpu_sched); a_occ += occ; if (occ > m_occ) { m_occ = occ; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b757812725f7..93908bb17cc5 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -4041,6 +4041,7 @@ extern int sysctl_sched_cache_user; extern unsigned int llc_aggr_tolerance; extern unsigned int llc_epoch_period; extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_epoch_visited_timeout; extern unsigned int llc_imb_pct; extern unsigned int llc_overaggr_pct; -- 2.34.1