From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D13A732ABF9 for ; Tue, 23 Dec 2025 12:13:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=162.62.58.211 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766492045; cv=none; b=A+Gy5fEHjN8PnbPWF6uN4/r8uNdXAASD2MzYcLj7N658N1ina0eye+x3P2misLIzqgOqPkKIGpM83B1RlLqkfAgVWDOxGpzwId7LTgtfk9CfYdsVmEMCeoHV7NvSXpfv1sndTYuvnISoR/b67f8kenpznRODPPFKjnvsfY/CJIA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766492045; c=relaxed/simple; bh=jg5vmiPdhOvxsQEGuW0VYbO9a9teKQbb8J0j50+dv8Q=; h=Message-ID:Content-Type:Mime-Version:Subject:From:In-Reply-To: Date:Cc:References:To; b=UPpswTHd0BgdjCpjQY/y4g101WpjywkA74Wd2wp/DHYG/+sR4TYwGslDlAh7sYFv2B5Y4KkpJOL+xS5iOXJYnrL/9YKx27/KN6cMA3I4OPUTESKJ0F4RDOEiZwN/ZbSMuH2YM0UK+kt/bvGM8PS4xnmL8L3y2pboBPlA5RRx70k= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name; spf=pass smtp.mailfrom=cyyself.name; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b=VaBaCsUV; arc=none smtp.client-ip=162.62.58.211 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cyyself.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b="VaBaCsUV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1766492028; bh=4kyUzbMBpwn6eFgoXfhWW0C83DrfFKonYrD07871GbM=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=VaBaCsUVrspVG7duywrT8IANj68squgdv6EusctUy+p7VTT0B1diIeGUsu+1O6idb 4pwzjuWLsBZpT8myFm6CShIp6ftnMNSfchjInhM/ZG9bC434fJr91w9wK3teWQrpCF KanLk/eRXyx9SzVFzR23FG/Tq/D1GlAMD9a5aJxQ= Received: from smtpclient.apple ([2408:8207:18a2:b28f:715c:3fb:9720:6a0a]) by newxmesmtplogicsvrszc41-0.qq.com (NewEsmtp) with SMTP id 31AA02A2; Tue, 23 Dec 2025 20:12:26 +0800 X-QQ-mid: xmsmtpt1766491946t4hg6wmum Message-ID: X-QQ-XMAILINFO: Nr4sKL92GIu+5jUEh6nupSNz/CKrfV1skse6eZ+HJbHb10XDsQFy3g+6AY63Kg VTHHQroA63OS2EjNTse7J9ZoVMFhys6cXNgF0HCKCGeG1+aB880rytRRPhorB0Ts24vNK3AyKcvS KglMKBCPaj8laAmKxAGq2iug+Vt9+7pR5O1alwG34FC5PC9oBBOAKnEndF4VxeDJ2iH9NN0TRHd4 cYFE4waMp/2VFtmjD9ibjQ9gsxUPuCgw/+/EyBeOonT296uLMByaa7KQPYnNqwGDtVu7yUgrFdds fTx/hZa0nyKd+n/PndFBlB/qh+iWa3Kcj/tNIshADWS+QtqREk5s3dTno5Tfjw0iyUowfAMKAmnh JXWi3GfYV6Sw2CjQ5wmTtf2GNRCQuaeKt0ptq0Kk/bqPi2UjSEK9tqPiqgmAeUx+zO4o0IAi4Uek A8+O7bWLWd6TUN0AMiJKdM5Qr8hFJ/PJ/m0tN+vgHlOLVU+jg0WYaFdhM7QkE+oAsPNGXJUra63d 4W2iJGosqmxDd6wABGTwuuh/iV+3Wrip2Y+RoVdV/yHy5kVOJc79bK+4dIzqfc3homdrKnRpW/Xi QMhxLp4W+xLz+M682C2ny599+f11+rbQfeVlgPW57Ijocz6yqJWXbRAvxvxEVcf+gjpbnzVfT/uu fUPyUPro2KVI6AMpfoEb9PfrX57mG6o6sdxt6yC76p93fG9CpwhmXuh2wdRHMyCFrvRGO7YTCpjJ qPMFQ5eCcbbK89wWFOcfoyMMuooJnHL6JRFOfHDrr/klzd5TlmRnKme9UNOsZ2JZBpVMLJfdbbxV PES4gfAwyEjZ3lyByDlSIDWVSrS3hLAt0fqgg3DXyKAx77gsRa/Y9liGd66kBt6C9DcBGf0sA3VK cpP73ksNhNEnWRr1R/IklOeTuG/rCiPobp7xgmiSYFnQ1hnmoufAEn04IK+9QFd31k5q2E7N0Yf3 NIbaP/PxUWfm/pBSPCd4jHv5+LfI8UJvXBu57WjLIrFyc42ZUWIX2kEir6aQKuJCfAoNU6dK07fQ RvYvvcS+H9SalQsI2ebeHh5/D170bIkaTDASK1aDTl+SQ4VVPR X-QQ-XMRINFO: M/715EihBoGS47X28/vv4NpnfpeBLnr4Qg== Content-Type: text/plain; charset=us-ascii Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.200.81.1.6\)) Subject: Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling From: Yangyu Chen In-Reply-To: Date: Tue, 23 Dec 2025 20:12:16 +0800 Cc: Peter Zijlstra , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Chen Yu , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable X-OQ-MSGID: <403D50A1-67C6-4C58-AC4C-33FF276522C6@cyyself.name> References: To: Tim Chen X-Mailer: Apple Mail (2.3864.200.81.1.6) > On 4 Dec 2025, at 07:07, Tim Chen wrote: >=20 > From: Chen Yu >=20 > Introduce a set of debugfs knobs to control the enabling of > and parameters for cache-aware load balancing. >=20 > (1) llc_enabled > llc_enabled acts as the primary switch - users can toggle it to > enable or disable cache aware load balancing. >=20 > (2) llc_aggr_tolerance > With sched_cache enabled, the scheduler uses a process's RSS as a > proxy for its LLC footprint to determine if aggregating tasks on the > preferred LLC could cause cache contention. If RSS exceeds the LLC > size, aggregation is skipped. Some workloads with large RSS but small > actual memory footprints may still benefit from aggregation. Since > the kernel cannot efficiently track per-task cache usage (resctrl is > user-space only), userspace can provide a more accurate hint. >=20 > Introduce /sys/kernel/debug/sched/llc_aggr_tolerance to let > users control how strictly RSS limits aggregation. Values range from > 0 to 100: >=20 > - 0: Cache-aware scheduling is disabled. > - 1: Strict; tasks with RSS larger than LLC size are skipped. > - 100: Aggressive; tasks are aggregated regardless of RSS. >=20 Hi Chen Yu and Tim Chen, Maybe we should have something like prctl(PR_LLC_AGGR_TOLERANCE, 100). I have tested this version of the patch on my EPYC Milan 7V13 (7763 = variant) physical machine, with 32M LLC for each 8-core CCX. I found = that I need to tune "llc_aggr_tolerance" to 100, else I can't get = cache-aware scheduling to work on Verilated [1] XiangShan [2] running = the chacha20 [3] as I mentioned before [4]. But if I set it to 100, I will lose some performance on stream copy = benchmarks since the bandwidth is limited per CCX. Thus, I think we = should have a new prctl to let userspace software hint the kernel that = this task can be bound by latency between cores, and should use this = feature no matter the RSS exceed the LLC size. I finally have an EPYC Milan at home today. I'm glad to test this patch = further. And I'm very willing to have this thus we can have a fast = verilator[1] without numactl manually. [1] https://github.com/verilator/verilator [2] https://github.com/OpenXiangShan/Xiangshan [3] https://github.com/cyyself/chacha20-xiangshan [4] = https://lore.kernel.org/lkml/tencent_6E51A3175F8AE0A7F684A319EE63CC56C806@= qq.com/ Thanks, Yangyu Chen > For exaImple, with a 32MB L3 cache: >=20 > - llc_aggr_tolerance=3D1 -> tasks with RSS > 32MB are skipped. > - llc_aggr_tolerance=3D99 -> tasks with RSS > 784GB are skipped > (784GB =3D (1 + (99 - 1) * 256) * 32MB). >=20 > Similarly, /sys/kernel/debug/sched/llc_aggr_tolerance also controls > how strictly the number of active threads is considered when doing > cache aware load balance. The number of SMTs is also considered. > High SMT counts reduce the aggregation capacity, preventing excessive > task aggregation on SMT-heavy systems like Power10/Power11. >=20 > For example, with 8 Cores/16 CPUs in a L3: >=20 > - llc_aggr_tolerance=3D1 -> tasks with nr_running > 8 are skipped. > - llc_aggr_tolerance=3D99 -> tasks with nr_running > 785 are skipped > 785 =3D (1 + (99 - 1) * 8). >=20 > (3) llc_epoch_period/llc_epoch_affinity_timeout > Besides, llc_epoch_period and llc_epoch_affinity_timeout are also = turned > into tunable. >=20 > Suggested-by: K Prateek Nayak > Suggested-by: Madadi Vineeth Reddy > Suggested-by: Shrikanth Hegde > Suggested-by: Tingyin Duan > Co-developed-by: Tim Chen > Signed-off-by: Tim Chen > Signed-off-by: Chen Yu > --- >=20 > Notes: > v1->v2: Remove the smt_nr check in fits_llc_capacity(). > (Aaron Lu) >=20 > include/linux/sched.h | 4 ++- > kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ > kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- > kernel/sched/sched.h | 5 ++++ > kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- > 5 files changed, 178 insertions(+), 10 deletions(-) >=20 > diff --git a/include/linux/sched.h b/include/linux/sched.h > index 466ba8b7398c..95bf080bbbf0 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -2436,9 +2436,11 @@ extern void migrate_enable(void); > DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) >=20 > #ifdef CONFIG_SCHED_CACHE > +DECLARE_STATIC_KEY_FALSE(sched_cache_on); > + > static inline bool sched_cache_enabled(void) > { > - return false; > + return static_branch_unlikely(&sched_cache_on); > } > #endif >=20 > diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c > index 02e16b70a790..cde324672103 100644 > --- a/kernel/sched/debug.c > +++ b/kernel/sched/debug.c > @@ -169,6 +169,53 @@ static const struct file_operations = sched_feat_fops =3D { > .release =3D single_release, > }; >=20 > +#ifdef CONFIG_SCHED_CACHE > +#define SCHED_CACHE_CREATE_CONTROL(name, max) \ > +static ssize_t sched_cache_write_##name(struct file *filp, \ > + const char __user *ubuf, \ > + size_t cnt, loff_t *ppos) \ > +{ \ > + char buf[16]; \ > + unsigned int val; \ > + if (cnt > 15) \ > + cnt =3D 15; \ > + if (copy_from_user(&buf, ubuf, cnt)) \ > + return -EFAULT; \ > + buf[cnt] =3D '\0'; \ > + if (kstrtouint(buf, 10, &val)) \ > + return -EINVAL; \ > + if (val > (max)) \ > + return -EINVAL; \ > + llc_##name =3D val; \ > + if (!strcmp(#name, "enabled")) \ > + sched_cache_set(false); \ > + *ppos +=3D cnt; \ > + return cnt; \ > +} \ > +static int sched_cache_show_##name(struct seq_file *m, void *v) \ > +{ \ > + seq_printf(m, "%d\n", llc_##name); \ > + return 0; \ > +} \ > +static int sched_cache_open_##name(struct inode *inode, \ > + struct file *filp) \ > +{ \ > + return single_open(filp, sched_cache_show_##name, NULL); \ > +} \ > +static const struct file_operations sched_cache_fops_##name =3D { \ > + .open =3D sched_cache_open_##name, \ > + .write =3D sched_cache_write_##name, \ > + .read =3D seq_read, \ > + .llseek =3D seq_lseek, \ > + .release =3D single_release, \ > +} > + > +SCHED_CACHE_CREATE_CONTROL(overload_pct, 100); > +SCHED_CACHE_CREATE_CONTROL(imb_pct, 100); > +SCHED_CACHE_CREATE_CONTROL(aggr_tolerance, 100); > +SCHED_CACHE_CREATE_CONTROL(enabled, 1); > +#endif /* SCHED_CACHE */ > + > static ssize_t sched_scaling_write(struct file *filp, const char = __user *ubuf, > size_t cnt, loff_t *ppos) > { > @@ -523,6 +570,21 @@ static __init int sched_init_debug(void) > debugfs_create_u32("hot_threshold_ms", 0644, numa, = &sysctl_numa_balancing_hot_threshold); > #endif /* CONFIG_NUMA_BALANCING */ >=20 > +#ifdef CONFIG_SCHED_CACHE > + debugfs_create_file("llc_overload_pct", 0644, debugfs_sched, NULL, > + &sched_cache_fops_overload_pct); > + debugfs_create_file("llc_imb_pct", 0644, debugfs_sched, NULL, > + &sched_cache_fops_imb_pct); > + debugfs_create_file("llc_aggr_tolerance", 0644, debugfs_sched, NULL, > + &sched_cache_fops_aggr_tolerance); > + debugfs_create_file("llc_enabled", 0644, debugfs_sched, NULL, > + &sched_cache_fops_enabled); > + debugfs_create_u32("llc_epoch_period", 0644, debugfs_sched, > + &llc_epoch_period); > + debugfs_create_u32("llc_epoch_affinity_timeout", 0644, = debugfs_sched, > + &llc_epoch_affinity_timeout); > +#endif > + > debugfs_create_file("debug", 0444, debugfs_sched, NULL, = &sched_debug_fops); >=20 > debugfs_fair_server_init(); > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 424ec601cfdf..a2e2d6742481 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct = sched_entity *se) >=20 > __read_mostly unsigned int llc_overload_pct =3D 50; > __read_mostly unsigned int llc_imb_pct =3D 20; > +__read_mostly unsigned int llc_aggr_tolerance =3D 1; > +__read_mostly unsigned int llc_epoch_period =3D EPOCH_PERIOD; > +__read_mostly unsigned int llc_epoch_affinity_timeout =3D = EPOCH_LLC_AFFINITY_TIMEOUT; >=20 > static int llc_id(int cpu) > { > @@ -1223,11 +1226,22 @@ static int llc_id(int cpu) > return llc; > } >=20 > +static inline int get_sched_cache_scale(int mul) > +{ > + if (!llc_aggr_tolerance) > + return 0; > + > + if (llc_aggr_tolerance =3D=3D 100) > + return INT_MAX; > + > + return (1 + (llc_aggr_tolerance - 1) * mul); > +} > + > static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) > { > + unsigned int llc, scale; > struct cacheinfo *ci; > unsigned long rss; > - unsigned int llc; >=20 > /* > * get_cpu_cacheinfo_level() can not be used > @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct = mm_struct *mm, int cpu) > rss =3D get_mm_counter(mm, MM_ANONPAGES) + > get_mm_counter(mm, MM_SHMEMPAGES); >=20 > - return (llc <=3D (rss * PAGE_SIZE)); > + /* > + * Scale the LLC size by 256*llc_aggr_tolerance > + * and compare it to the task's RSS size. > + * > + * Suppose the L3 size is 32MB. If the > + * llc_aggr_tolerance is 1: > + * When the RSS is larger than 32MB, the process > + * is regarded as exceeding the LLC capacity. If > + * the llc_aggr_tolerance is 99: > + * When the RSS is larger than 784GB, the process > + * is regarded as exceeding the LLC capacity because: > + * 784GB =3D (1 + (99 - 1) * 256) * 32MB > + */ > + scale =3D get_sched_cache_scale(256); > + if (scale =3D=3D INT_MAX) > + return false; > + > + return ((llc * scale) <=3D (rss * PAGE_SIZE)); > } >=20 > static bool exceed_llc_nr(struct mm_struct *mm, int cpu) > { > - int smt_nr =3D 1; > + int smt_nr =3D 1, scale; >=20 > #ifdef CONFIG_SCHED_SMT > if (sched_smt_active()) > smt_nr =3D cpumask_weight(cpu_smt_mask(cpu)); > #endif > + /* > + * Scale the Core number in a LLC by llc_aggr_tolerance > + * and compare it to the task's active threads. > + * > + * Suppose the number of Cores in LLC is 8. > + * Every core has 2 SMTs. > + * If the llc_aggr_tolerance is 1: When the > + * nr_running is larger than 8, the process > + * is regarded as exceeding the LLC capacity. > + * If the llc_aggr_tolerance is 99: > + * When the nr_running is larger than 785, > + * the process is regarded as exceeding > + * the LLC capacity: > + * 785 =3D 1 + (99 - 1) * 8 > + */ > + scale =3D get_sched_cache_scale(1); > + if (scale =3D=3D INT_MAX) > + return false; >=20 > - return ((mm->nr_running_avg * smt_nr) > per_cpu(sd_llc_size, cpu)); > + return ((mm->nr_running_avg * smt_nr) > (scale * = per_cpu(sd_llc_size, cpu))); > } >=20 > static void account_llc_enqueue(struct rq *rq, struct task_struct *p) > @@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq = *rq, struct mm_sched *pcpu_sched) > long delta =3D now - rq->cpu_epoch_next; >=20 > if (delta > 0) { > - n =3D (delta + EPOCH_PERIOD - 1) / EPOCH_PERIOD; > + n =3D (delta + llc_epoch_period - 1) / llc_epoch_period; > rq->cpu_epoch +=3D n; > - rq->cpu_epoch_next +=3D n * EPOCH_PERIOD; > + rq->cpu_epoch_next +=3D n * llc_epoch_period; > __shr_u64(&rq->cpu_runtime, n); > } >=20 > @@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct = task_struct *p, s64 delta_exec) > * has only 1 thread, or has too many active threads, invalidate > * its preferred state. > */ > - if (epoch - READ_ONCE(mm->mm_sched_epoch) > = EPOCH_LLC_AFFINITY_TIMEOUT || > + if (epoch - READ_ONCE(mm->mm_sched_epoch) > = llc_epoch_affinity_timeout || > get_nr_threads(p) <=3D 1 || > exceed_llc_nr(mm, cpu_of(rq)) || > exceed_llc_capacity(mm, cpu_of(rq))) { > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index 40798a06e058..15d126bd3728 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -2852,6 +2852,11 @@ extern unsigned int = sysctl_numa_balancing_hot_threshold; > #ifdef CONFIG_SCHED_CACHE > extern unsigned int llc_overload_pct; > extern unsigned int llc_imb_pct; > +extern unsigned int llc_aggr_tolerance; > +extern unsigned int llc_epoch_period; > +extern unsigned int llc_epoch_affinity_timeout; > +extern unsigned int llc_enabled; > +void sched_cache_set(bool locked); > #endif >=20 > #ifdef CONFIG_SCHED_HRTICK > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c > index 9799e3a9a609..818599ddaaef 100644 > --- a/kernel/sched/topology.c > +++ b/kernel/sched/topology.c > @@ -26,6 +26,49 @@ int max_llcs; >=20 > static bool sched_cache_present; >=20 > +unsigned int llc_enabled =3D 1; > +DEFINE_STATIC_KEY_FALSE(sched_cache_on); > + > +/* > + * Enable/disable cache aware scheduling according to > + * user input and the presence of hardware support. > + */ > +static void _sched_cache_set(bool enable, bool locked) > +{ > + if (enable) { > + if (locked) > + static_branch_enable_cpuslocked(&sched_cache_on); > + else > + static_branch_enable(&sched_cache_on); > + } else { > + if (locked) > + static_branch_disable_cpuslocked(&sched_cache_on); > + else > + static_branch_disable(&sched_cache_on); > + } > +} > + > +void sched_cache_set(bool locked) > +{ > + /* hardware does not support */ > + if (!sched_cache_present) { > + if (static_branch_likely(&sched_cache_on)) > + _sched_cache_set(false, locked); > + > + return; > + } > + > + /* user wants it or not ?*/ > + if (llc_enabled) { > + if (!static_branch_likely(&sched_cache_on)) > + _sched_cache_set(true, locked); > + > + } else { > + if (static_branch_likely(&sched_cache_on)) > + _sched_cache_set(false, locked); > + } > +} > + > static unsigned int *alloc_new_pref_llcs(unsigned int *old, unsigned = int **gc) > { > unsigned int *new =3D NULL; > @@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs) > * new buffer. > */ > tmp_llc_pref =3D alloc_percpu_noprof(unsigned int *); > - if (!tmp_llc_pref) > - return -ENOMEM; > + if (!tmp_llc_pref) { > + sched_cache_present =3D false; > + ret =3D -ENOMEM; > + > + goto out; > + } >=20 > for_each_present_cpu(i) > *per_cpu_ptr(tmp_llc_pref, i) =3D NULL; > @@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) > new =3D alloc_new_pref_llcs(rq->nr_pref_llc, per_cpu_ptr(tmp_llc_pref, = i)); > if (!new) { > ret =3D -ENOMEM; > + sched_cache_present =3D false; >=20 > goto release_old; > } > @@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs) > if (!ret) > max_llcs =3D new_max_llcs; >=20 > +out: > + sched_cache_set(true); > return ret; > } >=20 > --=20 > 2.32.0