[PATCH 0/4] sched/cache: per-process control of cache aware scheduling

[PATCH 0/4] sched/cache: per-process control of cache aware scheduling

[Yangyu Chen]

Cache aware scheduling is currently controlled only through global
debugfs knobs, but the right aggressiveness is workload and platform
specific. A multi-threaded Verilator run is one example: its RSS is
large while only a small part of it is hot, so an RSS-based footprint
estimate should not decide whether it is aggregated; and packing its
threads onto the SMT siblings of one LLC beats spreading them across
LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
EPYC Milan). Such choices cannot be made globally for the whole
machine.

This series adds per-process (per mm_struct) control. The first
patch splits the llc_aggr_tolerance debugfs knob into independent
thread-count and footprint tolerances and hardens the unbounded
percentage knobs against multiplication overflow. The second adds
prctl(PR_SCHED_CACHE), a single prctl command whose sub-command
selects GET or SET, covering a per-process enable, both tolerances
and the overaggr percentage, plus an inherit mask choosing which
attributes survive execve() (fork() always inherits). The remaining
two patches add selftests and documentation.

A numactl-like launcher built on this interface, llcctl, is
available at:

  https://github.com/cyyself/llcctl

It sets the attributes and the inherit mask on itself and executes
the target command, so the whole process tree runs with the
requested settings, e.g.:

  llcctl -d -- ./membw-hungry        # opt out, spread across LLCs
  llcctl -n 2 -p 100 -- make -j16    # pack up to 2 threads/core

Tested on x86-64 in QEMU with an EPYC-like topology (128 CPUs, 8 LLC
domains of 8 cores / 16 threads each): the selftests pass as both
root and an unprivileged user, and aggregation was verified end to
end by observing thread placement of a 24-thread spinner with and
without llcctl (evenly spread by default; packed into a single LLC
with raised tolerances). No new dmesg warnings.

Yangyu Chen (4):
  sched/cache: Split llc_aggr_tolerance into nr and size tolerances
  sched/cache: Add PR_SCHED_CACHE prctl for per-mm control
  selftests/prctl: Add PR_SCHED_CACHE tests
  docs/scheduler: Document cache aware scheduling controls

 Documentation/scheduler/index.rst             |   1 +
 Documentation/scheduler/sched-cache.rst       | 125 ++++++
 include/linux/mm_types.h                      |  10 +-
 include/linux/sched.h                         |  16 +
 include/uapi/linux/prctl.h                    |  38 ++
 kernel/fork.c                                 |   2 +-
 kernel/sched/debug.c                          |   6 +-
 kernel/sched/fair.c                           | 205 +++++++--
 kernel/sched/sched.h                          |   3 +-
 kernel/sched/syscalls.c                       | 120 ++++++
 kernel/sys.c                                  |   5 +
 .../trace/beauty/include/uapi/linux/prctl.h   |  37 ++
 tools/testing/selftests/prctl/.gitignore      |   1 +
 tools/testing/selftests/prctl/Makefile        |  15 +-
 tools/testing/selftests/prctl/config          |   1 +
 tools/testing/selftests/prctl/sched-cache.c   | 394 ++++++++++++++++++
 16 files changed, 931 insertions(+), 48 deletions(-)
 create mode 100644 Documentation/scheduler/sched-cache.rst
 create mode 100644 tools/testing/selftests/prctl/sched-cache.c


base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
-- 
2.47.3

[PATCH 1/4] sched/cache: Split llc_aggr_tolerance into nr and size tolerances

[Yangyu Chen]

llc_aggr_tolerance scales two independent gates with a single value:
the number of cores in an LLC compared against the process's active
thread count in invalid_llc_nr(), and the LLC size compared against
the process's memory footprint in exceed_llc_capacity().

The right settings are workload and platform specific. A Verilator
run is one example: its RSS is large while only a small part of it
is hot, so an RSS-based footprint estimate overstates its cache
usage and should not decide whether it is aggregated. And running
its threads on the SMT siblings of a single LLC can beat spreading
them across LLCs on some platforms (e.g. AMD EPYC Turin) but not on
others (e.g. EPYC Milan). Tuning for such a workload means a larger
thread-count tolerance while the footprint tolerance keeps its own
setting - which a single combined knob cannot express.

Split the knob into llc_aggr_tolerance_nr for the thread-count gate
and llc_aggr_tolerance_size for the footprint gate, and let
get_sched_cache_scale() take the tolerance value from the caller.
Both new debugfs files keep the semantics of the old knob (0 disables
aggregation for that gate, values >= 100 mean unlimited).

While making the tunables independently settable, also make the
percentage knobs safe for any value: llc_overaggr_pct and llc_imb_pct
are exposed via debugfs as unbounded u32 values and feed percentage
multiplications:

	util * 100 < max * aggr_pct               (fits_llc_capacity())
	util1 * 100 > util2 * (100 + llc_imb_pct) (util_greater())

CONFIG_SCHED_CACHE only depends on SMP, so this also builds on 32-bit
where unsigned long is 32 bits: a large enough percentage makes
max * aggr_pct (or util2 * (100 + pct)) wrap and produce a garbage
comparison, and the SMT-1 bump aggr_pct * 3 / 2 can wrap the u32
itself. Rather than capping the tunables, detect the overflow at the
point of use with check_mul_overflow()/check_add_overflow() and fall
back to the saturated meaning of the comparison: an overflowing
threshold is effectively unlimited (the LLC always has room), and an
overflowing bias is effectively infinite (the destination utilization
is never considered noticeably greater).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
---
 kernel/sched/debug.c |  6 +++--
 kernel/sched/fair.c  | 62 +++++++++++++++++++++++++++++++-------------
 kernel/sched/sched.h |  3 ++-
 3 files changed, 50 insertions(+), 21 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 40584b27ea0c..b7be245b9fc2 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -672,8 +672,10 @@ static __init int sched_init_debug(void)
 	llc = debugfs_create_dir("llc_balancing", debugfs_sched);
 	debugfs_create_file("enabled", 0644, llc, NULL,
 			    &sched_cache_enable_fops);
-	debugfs_create_u32("aggr_tolerance", 0644, llc,
-			   &llc_aggr_tolerance);
+	debugfs_create_u32("aggr_tolerance_nr", 0644, llc,
+			   &llc_aggr_tolerance_nr);
+	debugfs_create_u32("aggr_tolerance_size", 0644, llc,
+			   &llc_aggr_tolerance_size);
 	debugfs_create_u32("epoch_period", 0644, llc,
 			   &llc_epoch_period);
 	debugfs_create_u32("epoch_affinity_timeout", 0644, llc,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c31acbfa3247 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1404,7 +1404,8 @@ static void set_next_buddy(struct sched_entity *se);
  */
 #define EPOCH_PERIOD	(HZ / 100)	/* 10 ms */
 #define EPOCH_LLC_AFFINITY_TIMEOUT	5	/* 50 ms */
-__read_mostly unsigned int llc_aggr_tolerance	= 1;
+__read_mostly unsigned int llc_aggr_tolerance_nr	= 1;
+__read_mostly unsigned int llc_aggr_tolerance_size	= 1;
 __read_mostly unsigned int llc_epoch_period	= EPOCH_PERIOD;
 __read_mostly unsigned int llc_epoch_affinity_timeout = EPOCH_LLC_AFFINITY_TIMEOUT;
 __read_mostly unsigned int llc_imb_pct		= 20;
@@ -1418,10 +1419,8 @@ static int llc_id(int cpu)
 	return per_cpu(sd_llc_id, cpu);
 }
 
-static inline int get_sched_cache_scale(int mul)
+static inline int get_sched_cache_scale(unsigned int tol, int mul)
 {
-	unsigned int tol = READ_ONCE(llc_aggr_tolerance);
-
 	if (!tol)
 		return 0;
 
@@ -1453,23 +1452,23 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 		footprint = READ_ONCE(mm->sc_stat.footprint);
 
 		/*
-		 * Scale the LLC size by 256*llc_aggr_tolerance
+		 * Scale the LLC size by 256*llc_aggr_tolerance_size
 		 * and compare it to the task's footprint.
 		 *
 		 * Suppose the L3 size is 32MB. If the
-		 * llc_aggr_tolerance is 1:
+		 * llc_aggr_tolerance_size is 1:
 		 * When the footprint is larger than 32MB, the
 		 * process is regarded as exceeding the LLC
-		 * capacity. If the llc_aggr_tolerance is 99:
+		 * capacity. If the llc_aggr_tolerance_size is 99:
 		 * When the footprint is larger than 784GB, the
 		 * process is regarded as exceeding the LLC
 		 * capacity:
 		 * 784GB = (1 + (99 - 1) * 256) * 32MB
-		 * If the llc_aggr_tolerance is 100:
+		 * If the llc_aggr_tolerance_size is 100:
 		 * ignore the footprint and do the aggregation
 		 * anyway.
 		 */
-		scale = get_sched_cache_scale(256);
+		scale = get_sched_cache_scale(READ_ONCE(llc_aggr_tolerance_size), 256);
 		if (scale == INT_MAX)
 			return false;
 
@@ -1488,10 +1487,10 @@ static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p,
 		return true;
 
 	/*
-	 * Scale the number of 'cores' in a LLC by llc_aggr_tolerance
+	 * Scale the number of 'cores' in a LLC by llc_aggr_tolerance_nr
 	 * and compare it to the task's active threads.
 	 */
-	scale = get_sched_cache_scale(1);
+	scale = get_sched_cache_scale(READ_ONCE(llc_aggr_tolerance_nr), 1);
 	if (scale == INT_MAX)
 		return false;
 
@@ -10418,20 +10417,34 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_
  * done.
  * Derived from fits_capacity().
  *
+ * llc_overaggr_pct is an unbounded debugfs u32 and CONFIG_SCHED_CACHE
+ * only depends on SMP, so max * aggr_pct can wrap on 32-bit. A
+ * threshold large enough to overflow is effectively unlimited: the
+ * LLC always has room, so treat that as "fits" rather than letting
+ * the multiplication wrap.
+ *
  * (default: ~50%, tunable via debugfs)
  */
 static bool fits_llc_capacity(unsigned long util, unsigned long max)
 {
-	u32 aggr_pct = llc_overaggr_pct;
+	u32 aggr_pct = READ_ONCE(llc_overaggr_pct);
+	unsigned long thresh;
+	u32 bumped;
 
 	/*
 	 * For single core systems, raise the aggregation
 	 * threshold to accommodate more tasks.
 	 */
-	if (cpu_smt_num_threads == 1)
-		aggr_pct = (aggr_pct * 3 / 2);
+	if (cpu_smt_num_threads == 1) {
+		if (check_mul_overflow(aggr_pct, 3U, &bumped))
+			return true;
+		aggr_pct = bumped / 2;
+	}
+
+	if (check_mul_overflow(max, (unsigned long)aggr_pct, &thresh))
+		return true;
 
-	return util * 100 < max * aggr_pct;
+	return util * 100 < thresh;
 }
 
 /*
@@ -10439,10 +10452,23 @@ static bool fits_llc_capacity(unsigned long util, unsigned long max)
  * is 'util1' noticeably greater than 'util2'
  * Derived from capacity_greater().
  * Bias is in perentage.
+ *
+ * Allows dst util to be bigger than src util by up to bias percent.
+ * llc_imb_pct is an unbounded debugfs u32; a bias large enough to
+ * overflow util2 * (100 + llc_imb_pct) is effectively infinite, so
+ * util1 is never noticeably greater - treat that as "not greater"
+ * rather than letting the multiplication wrap.
  */
-/* Allows dst util to be bigger than src util by up to bias percent */
-#define util_greater(util1, util2) \
-	((util1) * 100 > (util2) * (100 + llc_imb_pct))
+static bool util_greater(unsigned long util1, unsigned long util2)
+{
+	unsigned long bias, rhs;
+
+	if (check_add_overflow(100UL, (unsigned long)READ_ONCE(llc_imb_pct), &bias) ||
+	    check_mul_overflow(util2, bias, &rhs))
+		return false;
+
+	return util1 * 100 > rhs;
+}
 
 static __maybe_unused bool get_llc_stats(int cpu, unsigned long *util,
 					 unsigned long *cap)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf502ba26..0ebafdcf2c7f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4095,7 +4095,8 @@ static inline void mm_cid_switch_to(struct task_struct *prev, struct task_struct
 DECLARE_STATIC_KEY_FALSE(sched_cache_present);
 DECLARE_STATIC_KEY_FALSE(sched_cache_active);
 extern int sysctl_sched_cache_user;
-extern unsigned int llc_aggr_tolerance;
+extern unsigned int llc_aggr_tolerance_nr;
+extern unsigned int llc_aggr_tolerance_size;
 extern unsigned int llc_epoch_period;
 extern unsigned int llc_epoch_affinity_timeout;
 extern unsigned int llc_imb_pct;
-- 
2.47.3

[PATCH 2/4] sched/cache: Add PR_SCHED_CACHE prctl for per-mm control

[Yangyu Chen]

Cache aware scheduling is currently controlled only through global
debugfs knobs, but the right aggressiveness is workload and platform
specific. A multi-threaded Verilator run is one example: its RSS is
large while only a small part of it is hot, so an RSS-based footprint
estimate should not decide whether it is aggregated; and packing its
threads onto the SMT siblings of one LLC beats spreading them across
LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
EPYC Milan). Such choices cannot be made globally for the whole
machine. Add a prctl interface to override the knobs per process
(per mm_struct):

  prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
  prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);

A single prctl command implements both directions, like
PR_RSEQ_SLICE_EXTENSION. The attributes are a per-process enable
(effective only while the feature is globally active), the two
aggregation tolerances, the overaggr percentage (applied where a
task's own migration is admitted; group level statistics span many
processes and keep using the global value), and an inherit mask
selecting which attributes an mm created by execve() keeps. fork()
always inherits everything, and the mask itself lives on the
task_struct so it survives both, which lets a numactl-like launcher
configure a workload and exec it.

The overrides live in mm->sc_stat with -1 meaning "follow the global
default"; GET stores the raw value through an int pointer so this
sentinel round-trips without being mistaken for an errno.
mm_init_sched() gains the creating task to tell fork (p != current)
from exec (p == current) apart. A disabled mm has its preferred LLC
invalidated at the existing invalidation points, so all group-level
statistics self-neutralize.

Also sync the tools/perf/trace/beauty copy of prctl.h.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
---
 include/linux/mm_types.h                      |  10 +-
 include/linux/sched.h                         |  16 ++
 include/uapi/linux/prctl.h                    |  38 +++++
 kernel/fork.c                                 |   2 +-
 kernel/sched/fair.c                           | 149 +++++++++++++++---
 kernel/sched/syscalls.c                       | 120 ++++++++++++++
 kernel/sys.c                                  |   5 +
 .../trace/beauty/include/uapi/linux/prctl.h   |  37 +++++
 8 files changed, 352 insertions(+), 25 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index b18c2b2e7d2c..eb8e77d6e476 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1609,10 +1609,11 @@ static inline unsigned int mm_cid_size(void)
 #endif /* CONFIG_SCHED_MM_CID */
 
 #ifdef CONFIG_SCHED_CACHE
-void mm_init_sched(struct mm_struct *mm,
+void mm_init_sched(struct mm_struct *mm, struct task_struct *p,
 		   struct sched_cache_time __percpu *pcpu_sched);
 
-static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
+static inline int mm_alloc_sched_noprof(struct mm_struct *mm,
+					struct task_struct *p)
 {
 	struct sched_cache_time __percpu *pcpu_sched =
 		alloc_percpu_noprof(struct sched_cache_time);
@@ -1620,7 +1621,7 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 	if (!pcpu_sched)
 		return -ENOMEM;
 
-	mm_init_sched(mm, pcpu_sched);
+	mm_init_sched(mm, p, pcpu_sched);
 	return 0;
 }
 
@@ -1633,7 +1634,8 @@ static inline void mm_destroy_sched(struct mm_struct *mm)
 }
 #else /* !CONFIG_SCHED_CACHE */
 
-static inline int mm_alloc_sched(struct mm_struct *mm) { return 0; }
+static inline int mm_alloc_sched(struct mm_struct *mm,
+				 struct task_struct *p) { return 0; }
 static inline void mm_destroy_sched(struct mm_struct *mm) { }
 
 #endif /* CONFIG_SCHED_CACHE */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 373bcc0598d1..5a3fd080676a 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1424,6 +1424,8 @@ struct task_struct {
 	int				preferred_llc;
 	/* 1: task was enqueued to its preferred LLC, 0 otherwise */
 	int				pref_llc_queued;
+	/* PR_SCHED_CACHE_INHERIT flags kept across execve() */
+	unsigned int			sched_cache_inherit;
 #endif
 
 	struct rseq_data		rseq;
@@ -2338,6 +2340,11 @@ static inline void sched_core_fork(struct task_struct *p) { }
 static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }
 #endif
 
+#ifdef CONFIG_SCHED_CACHE
+extern int sched_cache_prctl(unsigned long opt, unsigned long attr,
+			     unsigned long val, unsigned long arg5);
+#endif
+
 extern void sched_set_stop_task(int cpu, struct task_struct *stop);
 
 #ifdef CONFIG_MEM_ALLOC_PROFILING
@@ -2398,6 +2405,15 @@ struct sched_cache_stat {
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
+	/*
+	 * Per-process overrides of the cache aware scheduling
+	 * knobs, set via prctl(PR_SCHED_CACHE). -1 makes an
+	 * attribute follow the system-wide default.
+	 */
+	int user_enabled;
+	int aggr_tolerance_nr;
+	int aggr_tolerance_size;
+	int overaggr_pct;
 } ____cacheline_aligned_in_smp;
 
 #else
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index b6ec6f693719..dbc1c511b284 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -416,4 +416,42 @@ struct prctl_mm_map {
 # define PR_CFI_DISABLE		_BITUL(1)
 # define PR_CFI_LOCK		_BITUL(2)
 
+/*
+ * Get or set the per-process (per address space) cache aware
+ * scheduling attributes.
+ *
+ * PR_SCHED_CACHE_GET stores the attribute selected by arg3 into the
+ * int pointed to by arg4. PR_SCHED_CACHE_SET sets the attribute
+ * selected by arg3 to the value in arg4.
+ */
+#define PR_SCHED_CACHE				82
+# define PR_SCHED_CACHE_GET			1
+# define PR_SCHED_CACHE_SET			2
+/* Attributes for PR_SCHED_CACHE_GET/PR_SCHED_CACHE_SET */
+# define PR_SCHED_CACHE_ENABLE			1
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_NR	2
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE	3
+# define PR_SCHED_CACHE_OVERAGGR_PCT		4
+# define PR_SCHED_CACHE_INHERIT			5
+/*
+ * Attribute value that resets an attribute to the system default;
+ * an unset value attribute also reads back as this via
+ * PR_SCHED_CACHE_GET.
+ */
+# define PR_SCHED_CACHE_DEFAULT			(-1)
+/*
+ * Flags for PR_SCHED_CACHE_INHERIT: which attributes the new address
+ * space keeps across execve(). New address spaces created by fork()
+ * always inherit all attributes.
+ */
+# define PR_SCHED_CACHE_INHERIT_ENABLE			(1UL << 0)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR	(1UL << 1)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE	(1UL << 2)
+# define PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT		(1UL << 3)
+# define PR_SCHED_CACHE_INHERIT_MASK			\
+	(PR_SCHED_CACHE_INHERIT_ENABLE |		\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR |	\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE |	\
+	 PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT)
+
 #endif /* _LINUX_PRCTL_H */
diff --git a/kernel/fork.c b/kernel/fork.c
index f0e2e131a9a5..3c7c979e1d3d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1135,7 +1135,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
 	if (mm_alloc_cid(mm, p))
 		goto fail_cid;
 
-	if (mm_alloc_sched(mm))
+	if (mm_alloc_sched(mm, p))
 		goto fail_sched;
 
 	if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c31acbfa3247..d085a8438d3d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -44,6 +44,7 @@
 #include <linux/memory-tiers.h>
 #include <linux/mempolicy.h>
 #include <linux/mutex_api.h>
+#include <linux/prctl.h>
 #include <linux/profile.h>
 #include <linux/psi.h>
 #include <linux/ratelimit.h>
@@ -1430,6 +1431,61 @@ static inline int get_sched_cache_scale(unsigned int tol, int mul)
 	return (1 + (tol - 1) * mul);
 }
 
+/*
+ * Effective cache aware scheduling state of @mm. The per-process
+ * prctl(PR_SCHED_CACHE_ENABLE) attribute overrides the global
+ * default. Only meaningful when sched_cache_enabled().
+ */
+static bool sched_cache_mm_enabled(struct mm_struct *mm)
+{
+	int enabled;
+
+	/*
+	 * Statically allocated mms (init_mm, efi_mm) never go through
+	 * mm_init_sched(): their sc_stat is zero-initialized rather
+	 * than set up, recognizable by the NULL pcpu_sched. Treat them
+	 * as disabled instead of interpreting the zeroes (e.g.
+	 * sc_stat.cpu == 0 would read as a valid preferred CPU).
+	 */
+	if (!mm || !mm->sc_stat.pcpu_sched)
+		return false;
+
+	/*
+	 * -1 means no per-process override: follow the global enable,
+	 * which is on in every path that reaches this - they are all
+	 * behind sched_cache_enabled().
+	 */
+	enabled = READ_ONCE(mm->sc_stat.user_enabled);
+
+	return enabled != 0;
+}
+
+/*
+ * The following helpers return the effective value of a cache aware
+ * scheduling knob for @mm: the per-process attribute if one was set
+ * via prctl(PR_SCHED_CACHE), the global tunable otherwise.
+ */
+static inline unsigned int mm_aggr_tolerance_nr(struct mm_struct *mm)
+{
+	int tol = mm ? READ_ONCE(mm->sc_stat.aggr_tolerance_nr) : -1;
+
+	return tol >= 0 ? tol : READ_ONCE(llc_aggr_tolerance_nr);
+}
+
+static inline unsigned int mm_aggr_tolerance_size(struct mm_struct *mm)
+{
+	int tol = mm ? READ_ONCE(mm->sc_stat.aggr_tolerance_size) : -1;
+
+	return tol >= 0 ? tol : READ_ONCE(llc_aggr_tolerance_size);
+}
+
+static inline unsigned int mm_overaggr_pct(struct mm_struct *mm)
+{
+	int pct = mm ? READ_ONCE(mm->sc_stat.overaggr_pct) : -1;
+
+	return pct >= 0 ? pct : READ_ONCE(llc_overaggr_pct);
+}
+
 static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 {
 #ifdef CONFIG_NUMA_BALANCING
@@ -1468,7 +1524,7 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 		 * ignore the footprint and do the aggregation
 		 * anyway.
 		 */
-		scale = get_sched_cache_scale(READ_ONCE(llc_aggr_tolerance_size), 256);
+		scale = get_sched_cache_scale(mm_aggr_tolerance_size(mm), 256);
 		if (scale == INT_MAX)
 			return false;
 
@@ -1490,7 +1546,7 @@ static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p,
 	 * Scale the number of 'cores' in a LLC by llc_aggr_tolerance_nr
 	 * and compare it to the task's active threads.
 	 */
-	scale = get_sched_cache_scale(READ_ONCE(llc_aggr_tolerance_nr), 1);
+	scale = get_sched_cache_scale(mm_aggr_tolerance_nr(mm), 1);
 	if (scale == INT_MAX)
 		return false;
 
@@ -1571,7 +1627,7 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
 	}
 }
 
-void mm_init_sched(struct mm_struct *mm,
+void mm_init_sched(struct mm_struct *mm, struct task_struct *p,
 		   struct sched_cache_time __percpu *_pcpu_sched)
 {
 	unsigned long epoch = 0;
@@ -1593,6 +1649,34 @@ void mm_init_sched(struct mm_struct *mm,
 	mm->sc_stat.next_scan = jiffies;
 	mm->sc_stat.nr_running_avg = 0;
 	mm->sc_stat.footprint = 0;
+	mm->sc_stat.user_enabled = -1;
+	mm->sc_stat.aggr_tolerance_nr = -1;
+	mm->sc_stat.aggr_tolerance_size = -1;
+	mm->sc_stat.overaggr_pct = -1;
+
+	/*
+	 * A new mm created by fork() (@p is the new child) inherits all
+	 * of the parent's prctl(PR_SCHED_CACHE) attributes. Across
+	 * execve() (@p is current) only the attributes marked in the
+	 * calling thread's PR_SCHED_CACHE_INHERIT mask survive.
+	 */
+	if (current->mm) {
+		struct sched_cache_stat *src = &current->mm->sc_stat;
+		unsigned int inherit = ~0U;
+
+		if (p == current)
+			inherit = READ_ONCE(current->sched_cache_inherit);
+
+		if (inherit & PR_SCHED_CACHE_INHERIT_ENABLE)
+			mm->sc_stat.user_enabled = READ_ONCE(src->user_enabled);
+		if (inherit & PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR)
+			mm->sc_stat.aggr_tolerance_nr = READ_ONCE(src->aggr_tolerance_nr);
+		if (inherit & PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE)
+			mm->sc_stat.aggr_tolerance_size = READ_ONCE(src->aggr_tolerance_size);
+		if (inherit & PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT)
+			mm->sc_stat.overaggr_pct = READ_ONCE(src->overaggr_pct);
+	}
+
 	/*
 	 * The update to mm->sc_stat should not be reordered
 	 * before initialization to mm's other fields, in case
@@ -1713,10 +1797,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	}
 
 	/*
-	 * If this process hasn't hit task_cache_work() for a while invalidate
-	 * its preferred state.
+	 * If this process hasn't hit task_cache_work() for a while, or
+	 * has cache aware scheduling disabled via prctl(PR_SCHED_CACHE),
+	 * invalidate its preferred state.
 	 */
 	if ((long)(epoch - READ_ONCE(mm->sc_stat.epoch)) > llc_epoch_affinity_timeout ||
+	    !sched_cache_mm_enabled(mm) ||
 	    invalid_llc_nr(mm, p, cpu_of(rq)) ||
 	    exceed_llc_capacity(mm, cpu_of(rq))) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
@@ -1747,6 +1833,9 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)
 	    !mm->sc_stat.pcpu_sched)
 		return;
 
+	if (!sched_cache_mm_enabled(mm))
+		return;
+
 	epoch = rq->cpu_epoch;
 	/* avoid moving backwards */
 	if (time_after_eq(mm->sc_stat.epoch, epoch))
@@ -1851,7 +1940,8 @@ static void task_cache_work(struct callback_head *work)
 		return;
 
 	curr_cpu = task_cpu(p);
-	if (invalid_llc_nr(mm, p, curr_cpu) ||
+	if (!sched_cache_mm_enabled(mm) ||
+	    invalid_llc_nr(mm, p, curr_cpu) ||
 	    exceed_llc_capacity(mm, curr_cpu)) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
@@ -1918,7 +2008,14 @@ static void task_cache_work(struct callback_head *work)
 		}
 	}
 
-	if (m_a_occ > (2 * curr_m_a_occ)) {
+	/*
+	 * Re-check the per-mm enable after the scan: a concurrent
+	 * prctl() may have disabled cache aware scheduling for this
+	 * mm and reset sc_stat.cpu while we were scanning - do not
+	 * undo that reset. The check is best effort; a lost race is
+	 * corrected at the next tick.
+	 */
+	if (sched_cache_mm_enabled(mm) && m_a_occ > (2 * curr_m_a_occ)) {
 		/*
 		 * Avoid switching sc_stat.cpu too fast.
 		 * The reason to choose 2X is because:
@@ -10417,17 +10514,19 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_
  * done.
  * Derived from fits_capacity().
  *
+ * The per-mm percentage is bounded by the prctl, but the global
  * llc_overaggr_pct is an unbounded debugfs u32 and CONFIG_SCHED_CACHE
  * only depends on SMP, so max * aggr_pct can wrap on 32-bit. A
  * threshold large enough to overflow is effectively unlimited: the
  * LLC always has room, so treat that as "fits" rather than letting
  * the multiplication wrap.
  *
- * (default: ~50%, tunable via debugfs)
+ * (default: ~50%, tunable via debugfs and prctl(PR_SCHED_CACHE))
  */
-static bool fits_llc_capacity(unsigned long util, unsigned long max)
+static bool fits_llc_capacity(unsigned long util, unsigned long max,
+			      struct mm_struct *mm)
 {
-	u32 aggr_pct = READ_ONCE(llc_overaggr_pct);
+	u32 aggr_pct = mm_overaggr_pct(mm);
 	unsigned long thresh;
 	u32 bumped;
 
@@ -10549,7 +10648,8 @@ enum llc_mig {
  */
 static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
 				    unsigned long tsk_util,
-				    bool to_pref)
+				    bool to_pref,
+				    struct mm_struct *mm)
 {
 	unsigned long src_util, dst_util, src_cap, dst_cap;
 
@@ -10560,8 +10660,8 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
 	src_util = src_util < tsk_util ? 0 : src_util - tsk_util;
 	dst_util = dst_util + tsk_util;
 
-	if (!fits_llc_capacity(dst_util, dst_cap) &&
-	    !fits_llc_capacity(src_util, src_cap))
+	if (!fits_llc_capacity(dst_util, dst_cap, mm) &&
+	    !fits_llc_capacity(src_util, src_cap, mm))
 		return mig_unrestricted;
 
 	if (to_pref) {
@@ -10571,7 +10671,7 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
 		 * than the src, in which case migration will
 		 * increase the imbalance too much.
 		 */
-		if (!fits_llc_capacity(dst_util, dst_cap) &&
+		if (!fits_llc_capacity(dst_util, dst_cap, mm) &&
 		    util_greater(dst_util, src_util))
 			return mig_forbid;
 	} else {
@@ -10582,7 +10682,7 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,
 		 * of preferred LLC, leading to migration again
 		 * back to preferred LLC.
 		 */
-		if (fits_llc_capacity(src_util, src_cap) ||
+		if (fits_llc_capacity(src_util, src_cap, mm) ||
 		    !util_greater(src_util, dst_util))
 			return mig_forbid;
 	}
@@ -10608,8 +10708,9 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,
 	if (cpu < 0 || cpus_share_cache(src_cpu, dst_cpu))
 		return mig_unrestricted;
 
-	/* skip cache aware load balance for too many threads */
-	if (invalid_llc_nr(mm, p, dst_cpu) ||
+	/* skip cache aware load balance for disabled mm or too many threads */
+	if (!sched_cache_mm_enabled(mm) ||
+	    invalid_llc_nr(mm, p, dst_cpu) ||
 	    exceed_llc_capacity(mm, dst_cpu)) {
 		if (READ_ONCE(mm->sc_stat.cpu) != -1)
 			WRITE_ONCE(mm->sc_stat.cpu, -1);
@@ -10624,7 +10725,7 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,
 		return mig_unrestricted;
 
 	return can_migrate_llc(src_cpu, dst_cpu,
-			       task_util(p), to_pref);
+			       task_util(p), to_pref, mm);
 }
 
 /*
@@ -10663,8 +10764,12 @@ alb_break_llc(struct lb_env *env)
 		if (cur && cur->sched_class == &fair_sched_class)
 			util = task_util(cur);
 
+		/*
+		 * No stable mm context here: rq->curr's mm may be
+		 * dropped at any time, use the global threshold.
+		 */
 		if (can_migrate_llc(env->src_cpu, env->dst_cpu,
-				    util, false) == mig_forbid)
+				    util, false, NULL) == mig_forbid)
 			return true;
 	}
 
@@ -11775,9 +11880,13 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
 	if (env->sd->nr_balance_failed >= env->sd->cache_nice_tries + 1)
 		return false;
 
+	/*
+	 * Group level statistics aggregate tasks of many processes,
+	 * there is no single owning mm: use the global threshold.
+	 */
 	if (sgs->nr_pref_dst_llc &&
 	    can_migrate_llc(cpumask_first(sched_group_span(group)),
-			    env->dst_cpu, 0, true) == mig_llc)
+			    env->dst_cpu, 0, true, NULL) == mig_llc)
 		return true;
 
 	return false;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b215b0ead9a6..dec114eff269 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -7,6 +7,7 @@
  *  Copyright (C) 1991-2002  Linus Torvalds
  *  Copyright (C) 1998-2024  Ingo Molnar, Red Hat
  */
+#include <linux/prctl.h>
 #include <linux/sched.h>
 #include <linux/cpuset.h>
 #include <linux/sched/debug.h>
@@ -1576,3 +1577,122 @@ SYSCALL_DEFINE2(sched_rr_get_interval_time32, pid_t, pid,
 	return retval;
 }
 #endif
+
+#ifdef CONFIG_SCHED_CACHE
+/*
+ * PR_SCHED_CACHE_DEFAULT is the int -1. Depending on how userspace
+ * passed it (int through prctl()'s varargs, long, or from a 32-bit
+ * task) it arrives either sign-extended (the first comparison, -1
+ * converts to ULONG_MAX) or zero-extended to 0xffffffff (the second);
+ * accept both.
+ */
+static bool sched_cache_val_default(unsigned long val)
+{
+	return val == (unsigned long)PR_SCHED_CACHE_DEFAULT ||
+	       val == (unsigned int)PR_SCHED_CACHE_DEFAULT;
+}
+
+static int sched_cache_set_attr(unsigned long attr, unsigned long val)
+{
+	struct mm_struct *mm = current->mm;
+	bool def = sched_cache_val_default(val);
+	int ival = def ? -1 : (int)val;
+
+	switch (attr) {
+	case PR_SCHED_CACHE_ENABLE:
+		if (!def && val > 1)
+			return -EINVAL;
+		WRITE_ONCE(mm->sc_stat.user_enabled, ival);
+		/*
+		 * Drop the preferred LLC hint on any change: a process
+		 * that became disabled must stop being honored right
+		 * away, and one that became enabled re-establishes the
+		 * hint within an epoch anyway. This is best effort: an
+		 * in-flight task_cache_work() scan re-checks the enable
+		 * before publishing a new preference, and a lost race
+		 * is corrected at the next tick.
+		 */
+		WRITE_ONCE(mm->sc_stat.cpu, -1);
+		break;
+	case PR_SCHED_CACHE_AGGR_TOLERANCE_NR:
+		if (!def && val > 100)
+			return -EINVAL;
+		WRITE_ONCE(mm->sc_stat.aggr_tolerance_nr, ival);
+		break;
+	case PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE:
+		if (!def && val > 100)
+			return -EINVAL;
+		WRITE_ONCE(mm->sc_stat.aggr_tolerance_size, ival);
+		break;
+	case PR_SCHED_CACHE_OVERAGGR_PCT:
+		/*
+		 * Bound the percentage so that scaling an LLC capacity
+		 * by it cannot overflow, even on 32-bit. Anything in the
+		 * hundreds already means "never treat the LLC as busy".
+		 */
+		if (!def && val > 1000)
+			return -EINVAL;
+		WRITE_ONCE(mm->sc_stat.overaggr_pct, ival);
+		break;
+	case PR_SCHED_CACHE_INHERIT:
+		/* the default inherit mask is empty */
+		if (def)
+			val = 0;
+		if (val & ~PR_SCHED_CACHE_INHERIT_MASK)
+			return -EINVAL;
+		WRITE_ONCE(current->sched_cache_inherit, val);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int sched_cache_get_attr(unsigned long attr, unsigned long uptr)
+{
+	struct mm_struct *mm = current->mm;
+	int val;
+
+	switch (attr) {
+	case PR_SCHED_CACHE_ENABLE:
+		val = READ_ONCE(mm->sc_stat.user_enabled);
+		break;
+	case PR_SCHED_CACHE_AGGR_TOLERANCE_NR:
+		val = READ_ONCE(mm->sc_stat.aggr_tolerance_nr);
+		break;
+	case PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE:
+		val = READ_ONCE(mm->sc_stat.aggr_tolerance_size);
+		break;
+	case PR_SCHED_CACHE_OVERAGGR_PCT:
+		val = READ_ONCE(mm->sc_stat.overaggr_pct);
+		break;
+	case PR_SCHED_CACHE_INHERIT:
+		val = READ_ONCE(current->sched_cache_inherit);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return put_user(val, (int __user *)uptr);
+}
+
+int sched_cache_prctl(unsigned long opt, unsigned long attr,
+		      unsigned long val, unsigned long arg5)
+{
+	if (arg5)
+		return -EINVAL;
+
+	if (!current->mm)
+		return -EINVAL;
+
+	switch (opt) {
+	case PR_SCHED_CACHE_GET:
+		return sched_cache_get_attr(attr, val);
+	case PR_SCHED_CACHE_SET:
+		return sched_cache_set_attr(attr, val);
+	default:
+		return -EINVAL;
+	}
+}
+#endif /* CONFIG_SCHED_CACHE */
diff --git a/kernel/sys.c b/kernel/sys.c
index df69bd71de03..e2b105d72c7e 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -2807,6 +2807,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 	case PR_SCHED_CORE:
 		error = sched_core_share_pid(arg2, arg3, arg4, arg5);
 		break;
+#endif
+#ifdef CONFIG_SCHED_CACHE
+	case PR_SCHED_CACHE:
+		error = sched_cache_prctl(arg2, arg3, arg4, arg5);
+		break;
 #endif
 	case PR_SET_MDWE:
 		error = prctl_set_mdwe(arg2, arg3, arg4, arg5);
diff --git a/tools/perf/trace/beauty/include/uapi/linux/prctl.h b/tools/perf/trace/beauty/include/uapi/linux/prctl.h
index 560f99bc4782..dbc1c511b284 100644
--- a/tools/perf/trace/beauty/include/uapi/linux/prctl.h
+++ b/tools/perf/trace/beauty/include/uapi/linux/prctl.h
@@ -416,5 +416,42 @@ struct prctl_mm_map {
 # define PR_CFI_DISABLE		_BITUL(1)
 # define PR_CFI_LOCK		_BITUL(2)
 
+/*
+ * Get or set the per-process (per address space) cache aware
+ * scheduling attributes.
+ *
+ * PR_SCHED_CACHE_GET stores the attribute selected by arg3 into the
+ * int pointed to by arg4. PR_SCHED_CACHE_SET sets the attribute
+ * selected by arg3 to the value in arg4.
+ */
+#define PR_SCHED_CACHE				82
+# define PR_SCHED_CACHE_GET			1
+# define PR_SCHED_CACHE_SET			2
+/* Attributes for PR_SCHED_CACHE_GET/PR_SCHED_CACHE_SET */
+# define PR_SCHED_CACHE_ENABLE			1
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_NR	2
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE	3
+# define PR_SCHED_CACHE_OVERAGGR_PCT		4
+# define PR_SCHED_CACHE_INHERIT			5
+/*
+ * Attribute value that resets an attribute to the system default;
+ * an unset value attribute also reads back as this via
+ * PR_SCHED_CACHE_GET.
+ */
+# define PR_SCHED_CACHE_DEFAULT			(-1)
+/*
+ * Flags for PR_SCHED_CACHE_INHERIT: which attributes the new address
+ * space keeps across execve(). New address spaces created by fork()
+ * always inherit all attributes.
+ */
+# define PR_SCHED_CACHE_INHERIT_ENABLE			(1UL << 0)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR	(1UL << 1)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE	(1UL << 2)
+# define PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT		(1UL << 3)
+# define PR_SCHED_CACHE_INHERIT_MASK			\
+	(PR_SCHED_CACHE_INHERIT_ENABLE |		\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR |	\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE |	\
+	 PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT)
 
 #endif /* _LINUX_PRCTL_H */
-- 
2.47.3

[PATCH 3/4] selftests/prctl: Add PR_SCHED_CACHE tests

[Yangyu Chen]

Test the per-mm cache aware scheduling prctl: get/set round trips
for every attribute including resets via PR_SCHED_CACHE_DEFAULT in
both its sign-extended and 32-bit truncated form, argument
validation (out of range values, unknown attributes and sub-commands,
nonzero unused arguments, a faulting GET pointer, and that rejected
values are not stored), that attribute values live on the shared mm
while the inherit mask stays per thread, that fork() always inherits
without changes leaking back to the parent, and that execve() resets
attributes unless selected in the inherit mask, verified by
re-executing the test binary in a checker mode.

All tests SKIP on kernels without PR_SCHED_CACHE.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
---
 tools/testing/selftests/prctl/.gitignore    |   1 +
 tools/testing/selftests/prctl/Makefile      |  15 +-
 tools/testing/selftests/prctl/config        |   1 +
 tools/testing/selftests/prctl/sched-cache.c | 394 ++++++++++++++++++++
 4 files changed, 406 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/prctl/sched-cache.c

diff --git a/tools/testing/selftests/prctl/.gitignore b/tools/testing/selftests/prctl/.gitignore
index 05d5e31661df..1b22c9f45c4b 100644
--- a/tools/testing/selftests/prctl/.gitignore
+++ b/tools/testing/selftests/prctl/.gitignore
@@ -2,5 +2,6 @@
 disable-tsc-ctxt-sw-stress-test
 disable-tsc-on-off-stress-test
 disable-tsc-test
+sched-cache
 set-anon-vma-name-test
 set-process-name
diff --git a/tools/testing/selftests/prctl/Makefile b/tools/testing/selftests/prctl/Makefile
index e770e86fad9a..f889721a0749 100644
--- a/tools/testing/selftests/prctl/Makefile
+++ b/tools/testing/selftests/prctl/Makefile
@@ -1,14 +1,19 @@
 # SPDX-License-Identifier: GPL-2.0
-ifndef CROSS_COMPILE
 ARCH ?= $(shell uname -m 2>/dev/null || echo not)
 override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/)
 
+# sched-cache tests an arch-independent prctl
+TEST_PROGS := sched-cache
+
+ifndef CROSS_COMPILE
 ifeq ($(ARCH),x86)
-TEST_PROGS := disable-tsc-ctxt-sw-stress-test disable-tsc-on-off-stress-test \
+TEST_PROGS += disable-tsc-ctxt-sw-stress-test disable-tsc-on-off-stress-test \
 		disable-tsc-test set-anon-vma-name-test set-process-name
+endif
+endif
+
+LDLIBS += -pthread
+
 all: $(TEST_PROGS)
 
 include ../lib.mk
-
-endif
-endif
diff --git a/tools/testing/selftests/prctl/config b/tools/testing/selftests/prctl/config
index c6ed03c544e5..f44bf7e888e6 100644
--- a/tools/testing/selftests/prctl/config
+++ b/tools/testing/selftests/prctl/config
@@ -1 +1,2 @@
 CONFIG_ANON_VMA_NAME=y
+CONFIG_SCHED_CACHE=y
diff --git a/tools/testing/selftests/prctl/sched-cache.c b/tools/testing/selftests/prctl/sched-cache.c
new file mode 100644
index 000000000000..6babbf205a0f
--- /dev/null
+++ b/tools/testing/selftests/prctl/sched-cache.c
@@ -0,0 +1,394 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Tests for prctl(PR_SCHED_CACHE): per-process (per-mm) control of
+ * cache aware scheduling.
+ *
+ * The prctl stores attributes on the calling process's mm, so values
+ * are shared by all threads, always inherited over fork(), and
+ * inherited over execve() according to the per-thread
+ * PR_SCHED_CACHE_INHERIT mask.
+ */
+#define _GNU_SOURCE
+
+#include <errno.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "kselftest_harness.h"
+
+#ifndef PR_SCHED_CACHE
+#define PR_SCHED_CACHE				82
+# define PR_SCHED_CACHE_GET			1
+# define PR_SCHED_CACHE_SET			2
+# define PR_SCHED_CACHE_ENABLE			1
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_NR	2
+# define PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE	3
+# define PR_SCHED_CACHE_OVERAGGR_PCT		4
+# define PR_SCHED_CACHE_INHERIT			5
+# define PR_SCHED_CACHE_DEFAULT			(-1)
+# define PR_SCHED_CACHE_INHERIT_ENABLE			(1UL << 0)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR	(1UL << 1)
+# define PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE	(1UL << 2)
+# define PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT		(1UL << 3)
+# define PR_SCHED_CACHE_INHERIT_MASK			\
+	(PR_SCHED_CACHE_INHERIT_ENABLE |		\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR |	\
+	 PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_SIZE |	\
+	 PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT)
+#endif
+
+static int sc_get(unsigned long attr, int *val)
+{
+	if (prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, val, 0))
+		return -errno;
+	return 0;
+}
+
+static int sc_set(unsigned long attr, unsigned long val)
+{
+	if (prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, val, 0))
+		return -errno;
+	return 0;
+}
+
+static int sc_supported(void)
+{
+	int val;
+
+	return sc_get(PR_SCHED_CACHE_ENABLE, &val) == 0;
+}
+
+static void sc_reset_all(void)
+{
+	sc_set(PR_SCHED_CACHE_ENABLE, PR_SCHED_CACHE_DEFAULT);
+	sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, PR_SCHED_CACHE_DEFAULT);
+	sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, PR_SCHED_CACHE_DEFAULT);
+	sc_set(PR_SCHED_CACHE_OVERAGGR_PCT, PR_SCHED_CACHE_DEFAULT);
+	sc_set(PR_SCHED_CACHE_INHERIT, 0);
+}
+
+TEST(get_set_roundtrip)
+{
+	int val;
+
+	if (!sc_supported())
+		SKIP(return, "PR_SCHED_CACHE not supported");
+
+	sc_reset_all();
+
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_ENABLE, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_OVERAGGR_PCT, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_INHERIT, &val));
+	ASSERT_EQ(0, val);
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_ENABLE, 0));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_ENABLE, &val));
+	ASSERT_EQ(0, val);
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_ENABLE, 1));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_ENABLE, &val));
+	ASSERT_EQ(1, val);
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 7));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val));
+	ASSERT_EQ(7, val);
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, 13));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, &val));
+	ASSERT_EQ(13, val);
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_OVERAGGR_PCT, 155));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_OVERAGGR_PCT, &val));
+	ASSERT_EQ(155, val);
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT,
+			    PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_INHERIT, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR, (unsigned long)val);
+
+	/* DEFAULT resets the inherit mask to empty */
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT, PR_SCHED_CACHE_DEFAULT));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_INHERIT, &val));
+	ASSERT_EQ(0, val);
+
+	/* both the sign-extended and the 32-bit truncated DEFAULT reset */
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR,
+			    PR_SCHED_CACHE_DEFAULT));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, 0xffffffffUL));
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+
+	sc_reset_all();
+}
+
+TEST(invalid_arguments)
+{
+	int val;
+
+	if (!sc_supported())
+		SKIP(return, "PR_SCHED_CACHE not supported");
+
+	sc_reset_all();
+
+	/* out of range values */
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_ENABLE, 2));
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 101));
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, 101));
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_OVERAGGR_PCT, 1001));
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_INHERIT,
+				  PR_SCHED_CACHE_INHERIT_MASK + 1));
+
+	/* unknown attribute */
+	ASSERT_EQ(-EINVAL, sc_set(0, 1));
+	ASSERT_EQ(-EINVAL, sc_set(PR_SCHED_CACHE_INHERIT + 1, 1));
+	ASSERT_EQ(-EINVAL, sc_get(0, &val));
+	ASSERT_EQ(-EINVAL, sc_get(PR_SCHED_CACHE_INHERIT + 1, &val));
+
+	/* unknown op */
+	errno = 0;
+	ASSERT_EQ(-1, prctl(PR_SCHED_CACHE, 0, PR_SCHED_CACHE_ENABLE, 0, 0));
+	ASSERT_EQ(EINVAL, errno);
+	errno = 0;
+	ASSERT_EQ(-1, prctl(PR_SCHED_CACHE, 3, PR_SCHED_CACHE_ENABLE, 0, 0));
+	ASSERT_EQ(EINVAL, errno);
+
+	/* nonzero unused argument */
+	errno = 0;
+	ASSERT_EQ(-1, prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET,
+			    PR_SCHED_CACHE_ENABLE, 1, 1));
+	ASSERT_EQ(EINVAL, errno);
+
+	/* bad GET pointer */
+	errno = 0;
+	ASSERT_EQ(-1, prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET,
+			    PR_SCHED_CACHE_ENABLE, NULL, 0));
+	ASSERT_EQ(EFAULT, errno);
+
+	/* values rejected with -EINVAL must not be stored */
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_ENABLE, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_DEFAULT, val);
+}
+
+struct thread_ctx {
+	int nr_seen;
+	int inherit_seen;
+	int ret;
+};
+
+static void *thread_fn(void *arg)
+{
+	struct thread_ctx *ctx = arg;
+
+	ctx->ret = sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &ctx->nr_seen);
+	if (!ctx->ret)
+		ctx->ret = sc_get(PR_SCHED_CACHE_INHERIT, &ctx->inherit_seen);
+	if (!ctx->ret)
+		ctx->ret = sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 42);
+	/* clearing this thread's mask must not affect the creator's */
+	if (!ctx->ret)
+		ctx->ret = sc_set(PR_SCHED_CACHE_INHERIT, 0);
+
+	return NULL;
+}
+
+TEST(values_shared_by_threads_inherit_mask_is_not)
+{
+	struct thread_ctx ctx = { .nr_seen = -2, .inherit_seen = -2 };
+	pthread_t thread;
+	int val;
+
+	if (!sc_supported())
+		SKIP(return, "PR_SCHED_CACHE not supported");
+
+	sc_reset_all();
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 31));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT,
+			    PR_SCHED_CACHE_INHERIT_ENABLE));
+
+	ASSERT_EQ(0, pthread_create(&thread, NULL, thread_fn, &ctx));
+	ASSERT_EQ(0, pthread_join(thread, NULL));
+	ASSERT_EQ(0, ctx.ret);
+
+	/* attribute values live on the shared mm */
+	ASSERT_EQ(31, ctx.nr_seen);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val));
+	ASSERT_EQ(42, val);
+
+	/*
+	 * The inherit mask is per thread: the new thread starts with a
+	 * copy of its creator's mask, and clearing it in the thread
+	 * does not touch the creator's.
+	 */
+	ASSERT_EQ(PR_SCHED_CACHE_INHERIT_ENABLE,
+		  (unsigned long)ctx.inherit_seen);
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_INHERIT, &val));
+	ASSERT_EQ(PR_SCHED_CACHE_INHERIT_ENABLE, (unsigned long)val);
+
+	sc_reset_all();
+}
+
+TEST(fork_always_inherits)
+{
+	pid_t pid;
+	int status;
+	int val;
+
+	if (!sc_supported())
+		SKIP(return, "PR_SCHED_CACHE not supported");
+
+	sc_reset_all();
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_ENABLE, 1));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 9));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, 11));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_OVERAGGR_PCT, 77));
+
+	pid = fork();
+	ASSERT_LE(0, pid);
+	if (pid == 0) {
+		int val;
+
+		if (sc_get(PR_SCHED_CACHE_ENABLE, &val) || val != 1)
+			_exit(1);
+		if (sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val) || val != 9)
+			_exit(2);
+		if (sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, &val) || val != 11)
+			_exit(3);
+		if (sc_get(PR_SCHED_CACHE_OVERAGGR_PCT, &val) || val != 77)
+			_exit(4);
+
+		/* changes in the child must not leak back to the parent */
+		if (sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 50))
+			_exit(5);
+
+		_exit(0);
+	}
+
+	ASSERT_EQ(pid, waitpid(pid, &status, 0));
+	ASSERT_TRUE(WIFEXITED(status));
+	ASSERT_EQ(0, WEXITSTATUS(status));
+
+	ASSERT_EQ(0, sc_get(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, &val));
+	ASSERT_EQ(9, val);
+
+	sc_reset_all();
+}
+
+static int exec_check(int expect_enable, int expect_nr,
+		      int expect_size, int expect_pct)
+{
+	char enable[16], nr[16], size[16], pct[16];
+	pid_t pid;
+	int status;
+
+	snprintf(enable, sizeof(enable), "%d", expect_enable);
+	snprintf(nr, sizeof(nr), "%d", expect_nr);
+	snprintf(size, sizeof(size), "%d", expect_size);
+	snprintf(pct, sizeof(pct), "%d", expect_pct);
+
+	pid = fork();
+	if (pid < 0)
+		return -1;
+
+	if (pid == 0) {
+		setenv("SCHED_CACHE_EXEC_MODE", "1", 1);
+		setenv("SCHED_CACHE_EXPECT_ENABLE", enable, 1);
+		setenv("SCHED_CACHE_EXPECT_NR", nr, 1);
+		setenv("SCHED_CACHE_EXPECT_SIZE", size, 1);
+		setenv("SCHED_CACHE_EXPECT_PCT", pct, 1);
+		execl("/proc/self/exe", "sched-cache", NULL);
+		_exit(126);
+	}
+
+	if (waitpid(pid, &status, 0) != pid)
+		return -1;
+	if (!WIFEXITED(status))
+		return -1;
+
+	return WEXITSTATUS(status);
+}
+
+TEST(execve_inheritance)
+{
+	if (!sc_supported())
+		SKIP(return, "PR_SCHED_CACHE not supported");
+
+	sc_reset_all();
+
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_ENABLE, 1));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 21));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE, 22));
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_OVERAGGR_PCT, 23));
+
+	/* without inherit flags execve() resets everything */
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT, 0));
+	ASSERT_EQ(0, exec_check(PR_SCHED_CACHE_DEFAULT,
+				PR_SCHED_CACHE_DEFAULT,
+				PR_SCHED_CACHE_DEFAULT,
+				PR_SCHED_CACHE_DEFAULT));
+
+	/* selected attributes survive execve() */
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT,
+			    PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR |
+			    PR_SCHED_CACHE_INHERIT_OVERAGGR_PCT));
+	ASSERT_EQ(0, exec_check(PR_SCHED_CACHE_DEFAULT, 21,
+				PR_SCHED_CACHE_DEFAULT, 23));
+
+	/* all of them */
+	ASSERT_EQ(0, sc_set(PR_SCHED_CACHE_INHERIT,
+			    PR_SCHED_CACHE_INHERIT_MASK));
+	ASSERT_EQ(0, exec_check(1, 21, 22, 23));
+
+	sc_reset_all();
+}
+
+static int exec_expect(const char *env, unsigned long attr)
+{
+	const char *str = getenv(env);
+	int val;
+
+	if (!str)
+		return 1;
+	if (sc_get(attr, &val))
+		return 1;
+	if (val != atoi(str))
+		return 1;
+
+	return 0;
+}
+
+static int exec_check_main(void)
+{
+	int bad = 0;
+
+	bad |= exec_expect("SCHED_CACHE_EXPECT_ENABLE", PR_SCHED_CACHE_ENABLE);
+	bad |= exec_expect("SCHED_CACHE_EXPECT_NR",
+			   PR_SCHED_CACHE_AGGR_TOLERANCE_NR);
+	bad |= exec_expect("SCHED_CACHE_EXPECT_SIZE",
+			   PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE);
+	bad |= exec_expect("SCHED_CACHE_EXPECT_PCT",
+			   PR_SCHED_CACHE_OVERAGGR_PCT);
+
+	return bad;
+}
+
+int main(int argc, char **argv)
+{
+	if (getenv("SCHED_CACHE_EXEC_MODE"))
+		return exec_check_main();
+
+	return test_harness_run(argc, argv);
+}
-- 
2.47.3

[PATCH 4/4] docs/scheduler: Document cache aware scheduling controls

[Yangyu Chen]

Cache aware scheduling gained a debugfs control surface when it was
merged and now also a per-process prctl (PR_SCHED_CACHE), but neither
is documented. Add Documentation/scheduler/sched-cache.rst describing
what the feature does and when aggregation is skipped, the
/sys/kernel/debug/sched/llc_balancing/ knobs including the split
aggr_tolerance_nr/aggr_tolerance_size tolerances, the prctl
attributes with their ranges and the GET/SET calling convention, and
the inheritance semantics across fork() and execve() with a
numactl-like launcher example.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Yangyu Chen <cyy@cyyself.name>
---
 Documentation/scheduler/index.rst       |   1 +
 Documentation/scheduler/sched-cache.rst | 125 ++++++++++++++++++++++++
 2 files changed, 126 insertions(+)
 create mode 100644 Documentation/scheduler/sched-cache.rst

diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
index 17ce8d76befc..8c6e633d4d79 100644
--- a/Documentation/scheduler/index.rst
+++ b/Documentation/scheduler/index.rst
@@ -10,6 +10,7 @@ Scheduler
     membarrier
     sched-arch
     sched-bwc
+    sched-cache
     sched-deadline
     sched-design-CFS
     sched-eevdf
diff --git a/Documentation/scheduler/sched-cache.rst b/Documentation/scheduler/sched-cache.rst
new file mode 100644
index 000000000000..8bd92c7c7d3d
--- /dev/null
+++ b/Documentation/scheduler/sched-cache.rst
@@ -0,0 +1,125 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================
+Cache Aware Scheduling
+======================
+
+Cache aware scheduling (``CONFIG_SCHED_CACHE``) aggregates the threads
+of a process on a preferred last level cache (LLC) domain when that is
+expected to improve cache locality: the scheduler samples the per-LLC
+CPU occupancy of every multi-threaded process and biases load balance
+towards the LLC the process already uses the most.
+
+Aggregation is skipped for processes that would not benefit from it:
+single-threaded processes, processes with more active threads than the
+LLC has cores, and processes whose memory footprint exceeds the LLC
+size (footprint tracking requires ``CONFIG_NUMA_BALANCING``).
+
+Global control (debugfs)
+========================
+
+The feature is controlled globally through
+``/sys/kernel/debug/sched/llc_balancing/``:
+
+``enabled``
+	Boolean. When on (the default), every process participates
+	unless it opted out via ``prctl(PR_SCHED_CACHE)``; when off,
+	the feature is inactive for every process.
+
+``aggr_tolerance_nr``
+	Scales the number of cores of an LLC before it is compared to
+	the process's number of active threads. 0 disables aggregation
+	through this gate, 1 is the strict default, values of 100 or
+	more disable the thread count check entirely.
+
+``aggr_tolerance_size``
+	Scales the LLC size (by 256 per step) before it is compared to
+	the process's memory footprint. Same 0/1/100 semantics as
+	``aggr_tolerance_nr``. Only effective with
+	``CONFIG_NUMA_BALANCING``.
+
+``overaggr_pct``
+	The LLC utilization threshold, in percent of the LLC capacity,
+	below which the LLC is considered idle enough to aggregate more
+	tasks into it (default 50).
+
+``imb_pct``
+	Utilization imbalance hysteresis, in percent, used when
+	comparing two LLCs (default 20).
+
+``epoch_period``, ``epoch_affinity_timeout``
+	Occupancy sampling period (jiffies) and the number of epochs
+	after which a process's LLC preference expires.
+
+Per-process control (prctl)
+===========================
+
+``prctl(PR_SCHED_CACHE)`` overrides the global knobs for one process.
+The attributes live on the process's address space (``mm_struct``):
+they are shared by all threads of the process, and a single prctl
+command implements both directions::
+
+	prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
+	prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);
+
+``PR_SCHED_CACHE_GET`` stores the raw attribute value into the ``int``
+pointed to by the fourth argument and returns 0. A value attribute
+that was never set reads back as -1 (== ``PR_SCHED_CACHE_DEFAULT``),
+meaning "follow the global default"; the ``PR_SCHED_CACHE_INHERIT``
+mask reads back as the mask itself (default 0). Setting an attribute
+to ``PR_SCHED_CACHE_DEFAULT`` resets it (for the inherit mask this
+clears all bits).
+
+Attributes:
+
+``PR_SCHED_CACHE_ENABLE``
+	0 or 1. A process can opt out of the globally enabled feature
+	with 0; 1 (like the default) participates. It cannot activate
+	the feature when the global knob is off or the hardware has a
+	single LLC.
+
+``PR_SCHED_CACHE_AGGR_TOLERANCE_NR``
+	0..100. Per-process version of ``aggr_tolerance_nr``.
+
+``PR_SCHED_CACHE_AGGR_TOLERANCE_SIZE``
+	0..100. Per-process version of ``aggr_tolerance_size``.
+
+``PR_SCHED_CACHE_OVERAGGR_PCT``
+	0..1000. Per-process version of ``overaggr_pct``. It applies
+	where a task's own migration is admitted; group-level load
+	balance statistics keep using the global value, because they
+	aggregate tasks of many processes.
+
+``PR_SCHED_CACHE_INHERIT``
+	A bitmask selecting which attributes a new address space
+	created by execve() keeps: ``PR_SCHED_CACHE_INHERIT_ENABLE``,
+	``..._AGGR_TOLERANCE_NR``, ``..._AGGR_TOLERANCE_SIZE`` and
+	``..._OVERAGGR_PCT``. The mask is per thread and is itself kept
+	across fork() and execve().
+
+Inheritance
+===========
+
+fork() always copies all attributes to the child, like other process
+properties. execve() resets attributes to the default unless the
+corresponding ``PR_SCHED_CACHE_INHERIT`` bit is set in the calling
+thread.
+
+This allows a numactl-like launcher to configure a workload::
+
+	prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET,
+	      PR_SCHED_CACHE_AGGR_TOLERANCE_NR, 100, 0);
+	prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET,
+	      PR_SCHED_CACHE_INHERIT,
+	      PR_SCHED_CACHE_INHERIT_AGGR_TOLERANCE_NR, 0);
+	execve(workload, ...);
+
+Errors
+======
+
+``EINVAL``
+	Unknown sub-command or attribute, value out of range, nonzero
+	unused argument, calling task has no mm, or the kernel was
+	built without ``CONFIG_SCHED_CACHE``.
+``EFAULT``
+	Invalid pointer passed to ``PR_SCHED_CACHE_GET``.
-- 
2.47.3

Re: [PATCH 2/4] sched/cache: Add PR_SCHED_CACHE prctl for per-mm control

[Chen, Yu C]

Hi Yangyu,

thanks very much for enhancing cache-aware scheduling,

On 7/22/2026 5:10 PM, Yangyu Chen wrote:
> Cache aware scheduling is currently controlled only through global
> debugfs knobs, but the right aggressiveness is workload and platform
> specific. A multi-threaded Verilator run is one example: its RSS is
> large while only a small part of it is hot, so an RSS-based footprint
> estimate should not decide whether it is aggregated; and packing its
> threads onto the SMT siblings of one LLC beats spreading them across
> LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
> EPYC Milan). Such choices cannot be made globally for the whole
> machine. Add a prctl interface to override the knobs per process
> (per mm_struct):
> 
>    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
>    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);
> 
> A single prctl command implements both directions, like
> PR_RSEQ_SLICE_EXTENSION. The attributes are a per-process enable
> (effective only while the feature is globally active), the two
> aggregation tolerances, the overaggr percentage (applied where a
> task's own migration is admitted; group level statistics span many
> processes and keep using the global value), and an inherit mask
> selecting which attributes an mm created by execve() keeps. fork()
> always inherits everything, and the mask itself lives on the
> task_struct so it survives both, which lets a numactl-like launcher
> configure a workload and exec it.
> 
> The overrides live in mm->sc_stat with -1 meaning "follow the global
> default"; GET stores the raw value through an int pointer so this
> sentinel round-trips without being mistaken for an errno.
> mm_init_sched() gains the creating task to tell fork (p != current)
> from exec (p == current) apart. A disabled mm has its preferred LLC
> invalidated at the existing invalidation points, so all group-level
> statistics self-neutralize.
> 

We are working on a version that uses prctl to turn on/off/share
among tasks, which performs similar operations to core-scheduling
based on cookies[1]. That version decouples the mm from
cache-aware scheduling so that processes, tasks, or cgroups can
tag tasks with different "cookies". We are also exploring how to
leverage schedqos (from Qais) to take advantage of these
interfaces.

Your enhancement for tuning the parameters could be applied on top
of that, I suppose.

[1] https://github.com/chen-yu-surf/linux/commits/cache_aware_prctl_v1.4/

Re: [PATCH 2/4] sched/cache: Add PR_SCHED_CACHE prctl for per-mm control

[Chen, Yu C]

Hi Yangyu,

On 7/22/2026 5:10 PM, Yangyu Chen wrote:
> Cache aware scheduling is currently controlled only through global
> debugfs knobs, but the right aggressiveness is workload and platform
> specific. A multi-threaded Verilator run is one example: its RSS is
> large while only a small part of it is hot, so an RSS-based footprint
> estimate should not decide whether it is aggregated; and packing its
> threads onto the SMT siblings of one LLC beats spreading them across
> LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
> EPYC Milan). Such choices cannot be made globally for the whole
> machine. Add a prctl interface to override the knobs per process
> (per mm_struct):
> 
>    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
>    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);
> 
> A single prctl command implements both directions, like
> PR_RSEQ_SLICE_EXTENSION. The attributes are a per-process enable
> (effective only while the feature is globally active), the two
> aggregation tolerances, the overaggr percentage (applied where a
> task's own migration is admitted; group level statistics span many
> processes and keep using the global value), and an inherit mask
> selecting which attributes an mm created by execve() keeps. fork()
> always inherits everything, and the mask itself lives on the
> task_struct so it survives both, which lets a numactl-like launcher
> configure a workload and exec it.
> 
> The overrides live in mm->sc_stat with -1 meaning "follow the global
> default"; GET stores the raw value through an int pointer so this
> sentinel round-trips without being mistaken for an errno.
> mm_init_sched() gains the creating task to tell fork (p != current)
> from exec (p == current) apart. A disabled mm has its preferred LLC
> invalidated at the existing invalidation points, so all group-level
> statistics self-neutralize.
> 
> Also sync the tools/perf/trace/beauty copy of prctl.h.
> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Yangyu Chen <cyy@cyyself.name>

[ ... ]

> +static int sched_cache_set_attr(unsigned long attr, unsigned long val)
> +{
> +	struct mm_struct *mm = current->mm;

As preparation work, should we first decouple sc_stat from
mm_struct and tie this stat to per-task task_struct? In this
way, we could have per-task cache preference control and extend
it to tasks/threads/process/cgroup if needed, which looks more
flexible IMO. We have a proposal here:
https://github.com/chen-yu-surf/linux/commit/bd43a0b6dd189d5091fb88630208cb7bf67b3165.patch

which introduces a pointer in task_struct:
struct sched_cache_group __rcu  *sched_cache_grp;

> +	bool def = sched_cache_val_default(val);
> +	int ival = def ? -1 : (int)val;
> +
> +	switch (attr) {
> +	case PR_SCHED_CACHE_ENABLE:
> +		if (!def && val > 1)
> +			return -EINVAL;
> +		WRITE_ONCE(mm->sc_stat.user_enabled, ival);
> +		/*
> +		 * Drop the preferred LLC hint on any change: a process
> +		 * that became disabled must stop being honored right
> +		 * away, and one that became enabled re-establishes the
> +		 * hint within an epoch anyway. This is best effort: an
> +		 * in-flight task_cache_work() scan re-checks the enable
> +		 * before publishing a new preference, and a lost race
> +		 * is corrected at the next tick.
> +		 */
> +		WRITE_ONCE(mm->sc_stat.cpu, -1);
> +		break;

After we switching from per mm_struct to per task control, we could provide
fine-gain control at task/process/process group granularity(similar to 
core-scheduling)

int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid,
           unsigned long cookie, unsigned long type);
pid argument: the PID of the target task. 0 means "the calling task."
pid_type : PIDTYPE_PID targets the single thread,
            PIDTYPE_TGID the whole thread group and PIDTYPE_PGID the process
            group of the target task.

And the proposal is here:
https://github.com/chen-yu-surf/linux/commit/17718b7cef1d03948e9fd3bcd0b5a49aba7aae2d.patch

thanks,
Chenyu

Re: [PATCH 2/4] sched/cache: Add PR_SCHED_CACHE prctl for per-mm control

[Tim Chen]

On Wed, 2026-07-22 at 18:13 +0800, Chen, Yu C wrote:
> Hi Yangyu,
> 
> On 7/22/2026 5:10 PM, Yangyu Chen wrote:
> > Cache aware scheduling is currently controlled only through global
> > debugfs knobs, but the right aggressiveness is workload and platform
> > specific. A multi-threaded Verilator run is one example: its RSS is
> > large while only a small part of it is hot, so an RSS-based footprint
> > estimate should not decide whether it is aggregated; and packing its
> > threads onto the SMT siblings of one LLC beats spreading them across
> > LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
> > EPYC Milan). Such choices cannot be made globally for the whole
> > machine. Add a prctl interface to override the knobs per process
> > (per mm_struct):
> > 
> >    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
> >    prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);
> > 
> > A single prctl command implements both directions, like
> > PR_RSEQ_SLICE_EXTENSION. The attributes are a per-process enable
> > (effective only while the feature is globally active), the two
> > aggregation tolerances, the overaggr percentage (applied where a
> > task's own migration is admitted; group level statistics span many
> > processes and keep using the global value), and an inherit mask
> > selecting which attributes an mm created by execve() keeps. fork()
> > always inherits everything, and the mask itself lives on the
> > task_struct so it survives both, which lets a numactl-like launcher
> > configure a workload and exec it.
> > 
> > The overrides live in mm->sc_stat with -1 meaning "follow the global
> > default"; GET stores the raw value through an int pointer so this
> > sentinel round-trips without being mistaken for an errno.
> > mm_init_sched() gains the creating task to tell fork (p != current)
> > from exec (p == current) apart. A disabled mm has its preferred LLC
> > invalidated at the existing invalidation points, so all group-level
> > statistics self-neutralize.
> > 
> > Also sync the tools/perf/trace/beauty copy of prctl.h.
> > 
> > Assisted-by: Claude:claude-fable-5
> > Signed-off-by: Yangyu Chen <cyy@cyyself.name>
> 
> [ ... ]
> 
> > +static int sched_cache_set_attr(unsigned long attr, unsigned long val)
> > +{
> > +	struct mm_struct *mm = current->mm;
> 
> As preparation work, should we first decouple sc_stat from
> mm_struct and tie this stat to per-task task_struct? In this
> way, we could have per-task cache preference control and extend
> it to tasks/threads/process/cgroup if needed, which looks more
> flexible IMO. We have a proposal here:
> https://github.com/chen-yu-surf/linux/commit/bd43a0b6dd189d5091fb88630208cb7bf67b3165.patch
> 
> which introduces a pointer in task_struct:
> struct sched_cache_group __rcu  *sched_cache_grp;
> 
> > +	bool def = sched_cache_val_default(val);
> > +	int ival = def ? -1 : (int)val;
> > +
> > +	switch (attr) {
> > +	case PR_SCHED_CACHE_ENABLE:
> > +		if (!def && val > 1)
> > +			return -EINVAL;
> > +		WRITE_ONCE(mm->sc_stat.user_enabled, ival);
> > +		/*
> > +		 * Drop the preferred LLC hint on any change: a process
> > +		 * that became disabled must stop being honored right
> > +		 * away, and one that became enabled re-establishes the
> > +		 * hint within an epoch anyway. This is best effort: an
> > +		 * in-flight task_cache_work() scan re-checks the enable
> > +		 * before publishing a new preference, and a lost race
> > +		 * is corrected at the next tick.
> > +		 */
> > +		WRITE_ONCE(mm->sc_stat.cpu, -1);
> > +		break;
> 
> After we switching from per mm_struct to per task control, we could provide
> fine-gain control at task/process/process group granularity(similar to 
> core-scheduling)

We are planning to introduce the concept of a sched_group.  And tasks in a sched
group can be grouped by mm, or using prctl to explicitly group them together.

We could enhance prctl to introduce per sched_group parameters like aggr_tolerance*
if it makes sense.

Tim

> 
> int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid,
>            unsigned long cookie, unsigned long type);
> pid argument: the PID of the target task. 0 means "the calling task."
> pid_type : PIDTYPE_PID targets the single thread,
>             PIDTYPE_TGID the whole thread group and PIDTYPE_PGID the process
>             group of the target task.
> 
> And the proposal is here:
> https://github.com/chen-yu-surf/linux/commit/17718b7cef1d03948e9fd3bcd0b5a49aba7aae2d.patch
> 
> thanks,
> Chenyu