Re: [PATCH 0/4] powerpc/topology: Introduce llc_override debugfs

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Tim, Thanks for taking a look and findings.

On 10/13/26 2:45 AM, Tim Chen wrote:
> On Mon, 2026-10-12 at 15:50 +0530, Shrikanth Hegde wrote:
>> There is a new feature in scheduler called as Cache Aware Scheduling [1].
>> It tries to keep related tasks within the same LLC domain while
>> keeping the system reasonably balanced.
>>
>> Cache Aware Scheduling assumes that the scheduler LLC domain is large enough
>> to be a useful placement target. This is true on many systems where an LLC spans
>> several cores, but not on PowerPC systems where the scheduler-visible LLC is at
>> the SMT level. In that case, sd_llc_size is effectively the SMT width.
>>
>> This becomes visible in the capacity check:
>>          return !fits_capacity((mm->sc_stat.nr_running_avg * cpu_smt_num_threads),
>>                                scale * per_cpu(sd_llc_size, cpu));
>>
>> As a result, Cache aware scheduling may be enabled but has no room to operate.
>>
>> Hence provide a debugfs option to set LLC at specific domain so that
>> one can try experiment their workload with cache aware scheduling.
>> This also provides ways to explore it on system with SMT=4 to see if
>> there are any opportunities to improve or uncover bugs if there is any.
>>
>> Note it is a debug option and it is recommended to keep it disabled.
>> There is no performance guarantees across different values.
>>
>> [1]: https://lore.kernel.org/all/cover.1778703694.git.tim.c.chen@linux.intel.com/
> 
> Shrikanth,
> 
> This is an interesting idea. One question from a quick look at the code.
> 
> The override moves sd_llc, sd_llc_size and sd_llc_shared, but not
> sd_llc_id. The id comes from arch_llc_mask(), which is
> cpu_l2_cache_mask() on powerpc, so after your patches it is still
> unique per core. It is also sticky: build_sched_domains() assigns it
> only when it is -1, and it is reset only on CPU offline. So a rebuild
> from the knob keeps the old ids.
> 
> Cache aware scheduling uses this id as the identity of an LLC:
> p->preferred_llc, nr_pref_dst_llc/llc_counts[], and the
> llc_id(cpu) == llc_id(grp->cpu) check in task_cache_work(). In that
> loop the occupancy is now summed over the MC span, but the id check
> rarely matches and is broken.
> 
> To make this work, arch_llc_mask() needs to follow the override, and
> the llc ids need to be reset when the topology is forced to rebuild.
> 

Thanks for catching this. I missed to see that.

One way to fix it is with a change like below.
But more thoughts after the patch.

---
  arch/powerpc/include/asm/topology.h |  7 +------
  arch/powerpc/kernel/smp.c           | 23 +++++++++++++++++++++++
  include/linux/sched.h               |  2 +-
  kernel/sched/topology.c             |  7 +++++++
  4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h
index 44ec416fa489..f4555632e692 100644
--- a/arch/powerpc/include/asm/topology.h
+++ b/arch/powerpc/include/asm/topology.h
@@ -135,12 +135,7 @@ struct cpumask *cpu_coregroup_mask(int cpu);
  const struct cpumask *cpu_die_mask(int cpu);
  int cpu_die_id(int cpu);
  
-/*
- * Points to where the LLC is. On power9 this will point at CACHE
- * domain, On others it will point to SMT domain. In all cases
- * cpu_l2_cache_mask points to where LLC is
- */
-#define arch_llc_mask(cpu)     cpu_l2_cache_mask(cpu)
+const struct cpumask *arch_llc_mask(int cpu);
  
  #ifdef CONFIG_PPC64
  #include <asm/smp.h>
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 6a4ee7d3b8e5..139c54fc3e2d 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1864,6 +1864,28 @@ void __noreturn arch_cpu_idle_dead(void)
  
  #endif
  
+/*
+ * Points to where the LLC is. On power9 this will point at CACHE
+ * domain, On others it will point to SMT domain. In all cases
+ * cpu_l2_cache_mask points to where LLC is.
+ *
+ * If there is llc_override debug knob, return appropriate mask
+ */
+const struct cpumask *arch_llc_mask(int cpu)
+{
+	switch (READ_ONCE(llc_override)) {
+	case LLC_OVERRIDE_SMT:
+		return cpu_smt_mask(cpu);
+	case LLC_OVERRIDE_MC:
+		return cpu_coregroup_mask(cpu);
+	case LLC_OVERRIDE_PKG:
+		return cpu_die_mask(cpu);
+	case LLC_OVERRIDE_DISABLED:
+	default:
+		return cpu_l2_cache_mask(cpu);
+	}
+}
+
  int arch_update_cpu_topology(void)
  {
  	if (READ_ONCE(llc_override_changed)) {
@@ -1923,6 +1945,7 @@ static ssize_t llc_override_set(struct file *file, const char __user *ubuf,
  		pr_info("sched LLC override set to %s. Rebuilding sched domains\n",
  			llc_override_str[override]);
  
+		sched_reset_llc_id();
  		rebuild_sched_domains();
  	}
  
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 7b91cae4a59c..4d7e2ee841a9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2505,5 +2505,5 @@ extern void migrate_enable(void);
  #endif /* MODULE */
  
  DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
-
+void sched_reset_llc_id(void);
  #endif
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index dd2b1d70ec9d..63c9bca4c5eb 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3087,6 +3087,13 @@ void sched_domains_free_llc_id(int cpu)
  	sched_domains_mutex_unlock();
  }
  
+void sched_reset_llc_id(void)
+{
+	int cpu;
+
+	for_each_online_cpu(cpu)
+		sched_domains_free_llc_id(cpu);
+}
  /*
   * Build sched domains for a given set of CPUs and attach the sched domains
   * to the individual CPUs
-- 
2.52.0



Another way to avoid all this is with kernel parameter i think.
That would fix the LLC at specific domain and this reset/rebuild of llc_id
won't be necessary. If one is happy to change the debugfs, i guess they
would okay with rebooting the kernel with specific parameter too.
And at this point, I think that would mean the code will be arch/powerpc/
contained too.
(Yes, that would need updating arch_llc_mask, but no need to do the rebuilding
the sd_llc_id stuff)


So two ways.

- debugfs way with above changes.
- kernel parameters way.


Tim, Peter, Maddy, What do you guys suggest?
>>
>> Tim, Chenyu, Peter,
>> Are there any selftest or patches which is used for
>> functional validation of cache aware scheduling?
> 
> We haven't created one. We validate mostly with workloads such as
> hackbench, schbench and yogini (which can vary the number of threads
> over time and track CPU assignment). I agree they are not ideal for
> functional checks of the load balance path.
> 
>> per-LLC occupancy in /proc/{PID}/schedstat may not tell the full
>> story when there are multiple processes and multiple LLC's.
>>
>> Atleast for functional verification, we might need one if there is
>> not one already. The reason being,
>> - At present, we may need a custom workload. hackbench/schbench are
>>    wakeup intensive and typically bail out even before load balance.
>> - stress-ng is per process. So doesn't get cache aware scheduling.
>> - A selftest could spwan specific number of threads till llc size etc
>>    and ensure packing happens till core_count(llc_size) by default.
>>    This could also be used for tunable verfifications.
>> - Provide an ability to observe how much of the workload was contained
>>    within the LLC due to cache aware scheduling or number of load
>>    balances stopped due to preferred llc.
> 
> A selftest would be great and much welcomed.
> 

Cool. Will give it a try. but only after start implementing
will come to know the challenges. LLM's are there to help anyway!.

> Tim
>