[PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Andrea Righi]

NVIDIA Olympus implements SMT with two symmetric processing elements (PEs).
When only one PE is active, the core operates in single-thread mode and
that PE can use the full core resources. When both PEs are active, the core
operates in two-thread mode and the PEs share those resources. This
behavior is common to SMT implementations, but Olympus is particularly
sensitive to brief sibling activations because returning from two-thread
mode to single-thread mode after a sibling becomes idle is not immediate.
As described by commit 293f9611ae735 ("sched/fair: Prefer fully idle cores
for NOHZ balancing"):

  Briefly activating an otherwise idle sibling can reduce the performance
  available to the other sibling and this effect does not necessarily end
  once the activated sibling becomes idle: after the ILB finishes and its
  CPU enters WFI, full single-thread performance is restored only after the
  sibling has remained idle for a qualification interval (10 Ki cycles on
  the tested Vera system).

That change prevents the NOHZ idle load balancer from unnecessarily waking
a sibling of a busy PE. However, ordinary task placement can still select
either sibling of an idle core and repeated changes of the active PE can
keep Olympus cores in two-thread mode despite little or no useful overlap
between the siblings.

This series makes PE0 the preferred sibling of an Olympus core using
SD_ASYM_PACKING and teaches the fair scheduler's idle-selection paths to
honor asymmetric SMT priority. The scheduler first selects an idle core
according to its existing placement and capacity rules, then chooses the
highest-priority available sibling within that core. The generic scheduler
behavior is enabled only when an architecture supplies an SD_ASYM_PACKING
SMT domain.

PE0 and PE1 have equal steady-state capacity, the preference does not
identify a faster PE. PE0 is used only as a canonical choice when both
siblings are available. Consistently selecting the same sibling avoids
alternating the active PE across wakeups, lets PE1 remain idle for longer,
and allows more cores to remain in, or return to, full-resource
single-thread mode.

The series was tested on a two-node Vera system using an 88-thread
single-precision GEMM on the 88 physical cores of NUMA node 0.

With the workload allowed to choose either sibling of every core, observed
throughput improved from approximately 9.4 TFLOP/s on the baseline kernel
to approximately 10.1 TFLOP/s with this series applied. Repeated runs also
became more predictable because the workload consistently settled on PE0
while PE1 remained quiet.

Changes in v4:
 - Honor the SMT sibling priority in the slow path (Srikar Dronamraju)
 - Rename the consolidated helper to select_idle_smt_cpu() (Srikar Dronamraju)
 - Link to v3: https://lore.kernel.org/r/20260907163513.4172411-1-arighi@nvidia.com

Changes in v3:
 - Consolidate the SMT-priority adjustment in select_idle_sibling() after an
   idle candidate has been selected (K Prateek Nayak)
 - Fold the asym SMT checks into select_idle_smt_priority() and scan the
   scheduling-domain span directly (K Prateek Nayak)
 - Link to v2: https://lore.kernel.org/r/20260904091838.3617894-1-arighi@nvidia.com

Changes in v2:
 - Clarify that the generic scheduler change also covers POWER7 (Dietmar
   Eggemann)
 - Simplify sched_smt_asym_prefer() by inspecting the lowest scheduling
   domain directly (Dietmar Eggemann)
 - Link to v1: https://lore.kernel.org/r/20260831181800.1668646-1-arighi@nvidia.com

Andrea Righi (2):
      arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores
      sched/fair: Honor asymmetric SMT priority in idle selection

 arch/arm64/include/asm/topology.h |  1 +
 arch/arm64/kernel/smp.c           |  1 +
 arch/arm64/kernel/topology.c      | 62 ++++++++++++++++++++++++++++
 kernel/sched/fair.c               | 85 +++++++++++++++++++++++++++++++--------
 kernel/sched/sched.h              |  6 +++
 kernel/sched/topology.c           | 36 +++++++++++++++++
 6 files changed, 174 insertions(+), 17 deletions(-)

[PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores

[Andrea Righi]

NVIDIA Olympus implements spatial SMT with symmetric steady-state PE
capacity but two different resource modes. One-Thread Active mode gives
one PE the full core, while waking the other PE restores Two-Thread
Active mode and partitions decode, issue, cache, TLB, and vector
resources. Returning to full-resource mode requires the sibling to
remain in WFI for 10 Ki cycles.

Measurements show that pinned workloads perform equally on either PE,
but freely migratable workloads lose substantial throughput when they
alternate between PE identities. Consistently selecting PE0 keeps PE1
idle, avoids repeated SMT repartitioning, and restores
one-thread-per-core performance.

Describe this scheduling preference with SD_ASYM_PACKING and give PE0,
identified by MPIDR_EL1.Aff0, the higher arch_asym_cpu_priority(). This
is independent of SD_ASYM_CPUCAPACITY: SMT siblings retain equal
capacity, while physical cores with different maximum frequencies are
handled by a higher scheduling domain.

Firmware currently provides no interface for describing the preferred
SMT sibling. Detect Olympus by MIDR until such an interface is
available.

Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 arch/arm64/include/asm/topology.h |  1 +
 arch/arm64/kernel/smp.c           |  1 +
 arch/arm64/kernel/topology.c      | 62 +++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+)

diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h
index b9eaf4ad70850..edc1c59b3448d 100644
--- a/arch/arm64/include/asm/topology.h
+++ b/arch/arm64/include/asm/topology.h
@@ -18,6 +18,7 @@ int pcibus_to_node(struct pci_bus *bus);
 #include <linux/arch_topology.h>
 
 void update_freq_counters_refs(void);
+void arm64_init_sched_topology(void);
 
 /* Replace task scheduler's default frequency-invariant accounting */
 #define arch_scale_freq_tick topology_scale_freq_tick
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a117..0135ac4eea8bd 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -443,6 +443,7 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	hyp_mode_check();
 	setup_system_features();
 	setup_user_features();
+	arm64_init_sched_topology();
 	mark_linear_text_alias_ro();
 }
 
diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
index d28438f8b83f1..0dd9eec1c4946 100644
--- a/arch/arm64/kernel/topology.c
+++ b/arch/arm64/kernel/topology.c
@@ -19,6 +19,8 @@
 #include <linux/init.h>
 #include <linux/percpu.h>
 #include <linux/sched/isolation.h>
+#include <linux/sched/topology.h>
+#include <linux/smp.h>
 #include <linux/xarray.h>
 
 #include <asm/cpu.h>
@@ -44,6 +46,66 @@
 static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale) =  1UL << (2 * SCHED_CAPACITY_SHIFT);
 static cpumask_var_t amu_fie_cpus;
 
+/*
+ * Switching the active PE on an NVIDIA Olympus SMT core can keep the core in
+ * two-thread active mode, with resources partitioned between the PEs.
+ *
+ * Prefer PE0 so PE1 can remain idle and the core can stay in full-resource
+ * mode. Firmware does not currently describe this preference, so detect
+ * Olympus by MIDR until a firmware interface is available.
+ */
+static bool olympus_prefer_pe0 __ro_after_init;
+
+#ifdef CONFIG_SCHED_SMT
+static int arm64_smt_flags(void)
+{
+	int flags = cpu_smt_flags();
+
+	if (olympus_prefer_pe0)
+		flags |= SD_ASYM_PACKING;
+
+	return flags;
+}
+#endif
+
+static struct sched_domain_topology_level arm64_asym_smt_topology[] = {
+#ifdef CONFIG_SCHED_SMT
+	SDTL_INIT(tl_smt_mask, arm64_smt_flags, SMT),
+#endif
+#ifdef CONFIG_SCHED_CLUSTER
+	SDTL_INIT(tl_cls_mask, cpu_cluster_flags, CLS),
+#endif
+#ifdef CONFIG_SCHED_MC
+	SDTL_INIT(tl_mc_mask, cpu_core_flags, MC),
+#endif
+	SDTL_INIT(tl_pkg_mask, NULL, PKG),
+	{ NULL, },
+};
+
+void __init arm64_init_sched_topology(void)
+{
+	if (!IS_ENABLED(CONFIG_SCHED_SMT))
+		return;
+
+	if ((read_cpuid_id() & MIDR_CPU_MODEL_MASK) != MIDR_NVIDIA_OLYMPUS)
+		return;
+
+	if (!topology_core_has_smt(smp_processor_id()))
+		return;
+
+	olympus_prefer_pe0 = true;
+	set_sched_topology(arm64_asym_smt_topology);
+	pr_info("Enabling PE0 SMT preference for NVIDIA Olympus\n");
+}
+
+int arch_asym_cpu_priority(int cpu)
+{
+	if (!olympus_prefer_pe0)
+		return 0;
+
+	return MPIDR_AFFINITY_LEVEL(cpu_logical_map(cpu), 0) == 0;
+}
+
 struct amu_cntr_sample {
 	u64		arch_const_cycles_prev;
 	u64		arch_core_cycles_prev;
-- 
2.55.0

[PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
level to order hardware threads. Idle CPU selection does not consult
that order, so a task can wake on an arbitrary sibling and remain there
until load balancing corrects the placement. On these systems, that
initial choice can prevent the core from entering its preferred
lower-thread resource mode and cause a large and persistent performance
loss.

When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.

Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, target, previous, recently-used CPU fast
paths and the slow path. Inspect the lowest scheduling domain directly,
but require both CPUs to share its span because isolcpus can split
hardware siblings across scheduling domains.

Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.

Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
 kernel/sched/sched.h    |  6 +++
 kernel/sched/topology.c | 36 +++++++++++++++++
 3 files changed, 110 insertions(+), 17 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8bd308c2d5b1..37837c36288a0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
 	return false;
 }
 
+/*
+ * Redirect a CPU to a higher-priority available sibling in its SMT domain,
+ * subject to task affinity.
+ */
+static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
+{
+	struct sched_domain *sd;
+	int best = cpu;
+	int sibling;
+
+	if (!sched_smt_asym_active())
+		return cpu;
+
+	sd = rcu_dereference_all(cpu_rq(cpu)->sd);
+	if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
+	    !(sd->flags & SD_ASYM_PACKING))
+		return cpu;
+
+	for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
+		if (sibling == best || !choose_idle_cpu(sibling, p))
+			continue;
+
+		if (sched_asym_prefer(sibling, best))
+			best = sibling;
+	}
+
+	return best;
+}
+
 /*
  * Scans the local SMT mask to see if the entire core is idle, and records this
  * information in sd_balance_shared->has_idle_cores.
@@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		goto select_smt_priority;
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(prev, target))
-			return prev;
+		    cpus_share_resources(prev, target)) {
+			target = prev;
+			goto select_smt_priority;
+		}
 
 		prev_aff = prev;
 	}
@@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    prev == smp_processor_id() &&
 	    this_rq()->nr_running <= 1 &&
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
-		return prev;
+		target = prev;
+		goto select_smt_priority;
 	}
 
 	/* Check a recently used CPU as a potential idle candidate: */
@@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+		    cpus_share_resources(recent_used_cpu, target)) {
+			target = recent_used_cpu;
+			goto select_smt_priority;
+		}
 
 	} else {
 		recent_used_cpu = -1;
@@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		 */
 		if (sd) {
 			i = select_idle_capacity(p, sd, target);
-			return ((unsigned)i < nr_cpumask_bits) ? i : target;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
+			return target;
 		}
 	}
 
@@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!has_idle_core && cpus_share_cache(prev, target)) {
 			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
 		}
 	}
 
 	i = select_idle_cpu(p, sd, has_idle_core, target);
-	if ((unsigned)i < nr_cpumask_bits)
-		return i;
+	if ((unsigned int)i < nr_cpumask_bits) {
+		target = i;
+		goto select_smt_priority;
+	}
 
 	/*
 	 * For cluster machines which have lower sharing cache like L2 or
@@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
 	 * use them if possible when no idle CPU found in select_idle_cpu().
 	 */
-	if ((unsigned int)prev_aff < nr_cpumask_bits)
-		return prev_aff;
-	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
-		return recent_used_cpu;
+	if ((unsigned int)prev_aff < nr_cpumask_bits) {
+		target = prev_aff;
+		goto select_smt_priority;
+	}
+	if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
+		target = recent_used_cpu;
+		goto select_smt_priority;
+	}
 
 	return target;
+
+select_smt_priority:
+	return select_idle_smt_cpu(p, target);
 }
 
 /**
@@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	}
 
 	/* Slow path */
-	if (unlikely(sd))
-		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
+	if (unlikely(sd)) {
+		new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
+		return select_idle_smt_cpu(p, new_cpu);
+	}
 
 	/* Fast path */
 	if (wake_flags & WF_TTWU)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c3ad70e58b8e..568cb1ed2dd6b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 extern struct static_key_false sched_asym_cpucapacity;
+extern struct static_key_false sched_smt_asym_packing;
 extern struct static_key_false sched_cluster_active;
 
 static __always_inline bool sched_asym_cpucap_active(void)
@@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
 	return static_branch_unlikely(&sched_asym_cpucapacity);
 }
 
+static __always_inline bool sched_smt_asym_active(void)
+{
+	return static_branch_unlikely(&sched_smt_asym_packing);
+}
+
 struct sched_group_capacity {
 	atomic_t		ref;
 	/*
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 0248227d983a7..06c40eb5932af 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
+DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+static bool has_asym_smt_domain(int cpu)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if (sd->flags & SD_ASYM_PACKING)
+			return true;
+	}
+
+	return false;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	struct rq *rq = NULL;
 	int i, ret = -ENOMEM;
 	bool has_asym = false;
+	bool has_asym_smt = false;
 	bool has_cluster = false;
 
 	if (WARN_ON(cpumask_empty(cpu_map)))
@@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 
 		cpu_attach_domain(sd, d.rd, i);
 
+		if (has_asym_smt_domain(i))
+			has_asym_smt = true;
+
 		if (lowest_flag_domain(i, SD_CLUSTER))
 			has_cluster = true;
 	}
@@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	if (has_asym)
 		static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_inc_cpuslocked(&sched_smt_asym_packing);
+
 	if (has_cluster)
 		static_branch_inc_cpuslocked(&sched_cluster_active);
 
@@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
 static void detach_destroy_domains(const struct cpumask *cpu_map)
 {
 	unsigned int cpu = cpumask_any(cpu_map);
+	bool has_asym_smt = false;
 	int i;
 
+	rcu_read_lock();
+	for_each_cpu(i, cpu_map) {
+		if (has_asym_smt_domain(i)) {
+			has_asym_smt = true;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
 	if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
 		static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_dec_cpuslocked(&sched_smt_asym_packing);
+
 	if (static_branch_unlikely(&sched_cluster_active))
 		static_branch_dec_cpuslocked(&sched_cluster_active);
 
-- 
2.55.0

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[K Prateek Nayak]

Hello Andrea,

On 9/8/2026 1:53 PM, Andrea Righi wrote:
> @@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  	}
>  
>  	/* Slow path */
> -	if (unlikely(sd))
> -		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> +	if (unlikely(sd)) {
> +		new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> +		return select_idle_smt_cpu(p, new_cpu);

nit. I personally feel this can be better integrated into the
sched_balance_find_dst_cpu(). Something like the following:

  (Only build tested)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8bd308c2d5b..1012dfb33f08 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12353,6 +12353,17 @@ static inline void update_sg_wakeup_stats(struct sched_domain *sd,
 
 	}
 
+	/*
+	 * If we are on a SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING
+	 * domain, use the group_asym_packing classification to
+	 * decide placement based on rankings of idle siblings.
+	 */
+	if (unlikely(sched_smt_asym_active() &&
+		     (sd->flags & SD_SHARE_CPUCAPACITY) &&
+		     (sd->flags & SD_ASYM_PACKING) &&
+		     sgs->idle_cpus))
+		sgs->group_asym_packing = 1;
+
 	sgs->group_capacity = group->sgc->capacity;
 
 	sgs->group_weight = group->group_weight;
@@ -12393,9 +12404,15 @@ static bool update_pick_idlest(struct sched_group *idlest,
 			return false;
 		break;
 
+	case group_asym_packing:
+		/*
+		 * Only possible for sched_smt_asym_active().
+		 * Select the idle SMT that is more preferred.
+		 */
+		return sched_asym_prefer(idlest->asym_prefer_cpu,
+					 group->asym_prefer_cpu);
 	case group_llc_balance:
 	case group_imbalanced:
-	case group_asym_packing:
 	case group_smt_balance:
 		/* Those types are not used in the slow wakeup path */
 		return false;
---

It leads to slightly more branches but they are super predictable when
iterating at a particular sched_domain level so the overhead should be
negligible.

I don't have any strong feelings either ways. Thoughts?

-- 
Thanks and Regards,
Prateek

Re: [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores

[K Prateek Nayak]

Hello Andrea,

On 9/8/2026 1:53 PM, Andrea Righi wrote:
> NVIDIA Olympus implements spatial SMT with symmetric steady-state PE
> capacity but two different resource modes. One-Thread Active mode gives
> one PE the full core, while waking the other PE restores Two-Thread
> Active mode and partitions decode, issue, cache, TLB, and vector
> resources. Returning to full-resource mode requires the sibling to
> remain in WFI for 10 Ki cycles.
> 
> Measurements show that pinned workloads perform equally on either PE,
> but freely migratable workloads lose substantial throughput when they
> alternate between PE identities. Consistently selecting PE0 keeps PE1
> idle, avoids repeated SMT repartitioning, and restores
> one-thread-per-core performance.
> 
> Describe this scheduling preference with SD_ASYM_PACKING and give PE0,
> identified by MPIDR_EL1.Aff0, the higher arch_asym_cpu_priority(). This
> is independent of SD_ASYM_CPUCAPACITY: SMT siblings retain equal
> capacity, while physical cores with different maximum frequencies are
> handled by a higher scheduling domain.
> 
> Firmware currently provides no interface for describing the preferred
> SMT sibling. Detect Olympus by MIDR until such an interface is
> available.
> 
> Signed-off-by: Andrea Righi <arighi@nvidia.com>

Feel free to include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

Just one nit below ...
> +/*
> + * Switching the active PE on an NVIDIA Olympus SMT core can keep the core in
> + * two-thread active mode, with resources partitioned between the PEs.
> + *
> + * Prefer PE0 so PE1 can remain idle and the core can stay in full-resource
> + * mode. Firmware does not currently describe this preference, so detect
> + * Olympus by MIDR until a firmware interface is available.
> + */
> +static bool olympus_prefer_pe0 __ro_after_init;

nit. What is the purpose of "olympus_prefer_pe0"?

> +
> +#ifdef CONFIG_SCHED_SMT
> +static int arm64_smt_flags(void)
> +{
> +	int flags = cpu_smt_flags();
> +
> +	if (olympus_prefer_pe0)
> +		flags |= SD_ASYM_PACKING;

If this flag function is being used, olympus_prefer_pe0 is always true,
and the scheduler will never call into arch_asym_cpu_priority() without
SD_ASYM_PACKING being set so olympus_prefer_pe0 is always true there
too.

Seems redundant unless you were planning to do set_sched_topology()
unconditionally for whole of arm64.

> +
> +	return flags;
> +}

-- 
Thanks and Regards,
Prateek

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Prateek,

On Wed, Sep 09, 2026 at 01:10:48AM +0530, K Prateek Nayak wrote:
> Hello Andrea,
> 
> On 9/8/2026 1:53 PM, Andrea Righi wrote:
> > @@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  	}
> >  
> >  	/* Slow path */
> > -	if (unlikely(sd))
> > -		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > +	if (unlikely(sd)) {
> > +		new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > +		return select_idle_smt_cpu(p, new_cpu);
> 
> nit. I personally feel this can be better integrated into the
> sched_balance_find_dst_cpu(). Something like the following:
> 
>   (Only build tested)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index b8bd308c2d5b..1012dfb33f08 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12353,6 +12353,17 @@ static inline void update_sg_wakeup_stats(struct sched_domain *sd,
>  
>  	}
>  
> +	/*
> +	 * If we are on a SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING
> +	 * domain, use the group_asym_packing classification to
> +	 * decide placement based on rankings of idle siblings.
> +	 */
> +	if (unlikely(sched_smt_asym_active() &&
> +		     (sd->flags & SD_SHARE_CPUCAPACITY) &&
> +		     (sd->flags & SD_ASYM_PACKING) &&
> +		     sgs->idle_cpus))
> +		sgs->group_asym_packing = 1;

Integrating the preference in the slow-path selection sounds appealing, but I
don't think group_asym_packing can be used as a destination classificaiton here.

The intended policy is to prefer PE0 over PE1 when both siblings of the selected
SMT core are idle. And if PE0 is busy, PE1 should remain a valid destination. It
shouldn't make a busy PE0 preferable to an idle PE1.

IIUC group_type is ordered for busiest-group selection, group_asym_packing
describes a source group whole load should be moved to a "more preferred" CPU.
Marking an idle SMT group as group_asym_packing could make it rank worse than a
fully busy group.

Example: a fork on SMT2 can have the busy local PE0 classified as
group_has_spare or group_fully_busy, while the idle PE1 is forced to
group_asym_packing, sched_balance_find_dst_group() can then consider the busy
local group the better destination and stack the new task on PE0. That may
preserve one-thread mode for a short task, but it can also reduce throughput for
sustained work.

> +
>  	sgs->group_capacity = group->sgc->capacity;
>  
>  	sgs->group_weight = group->group_weight;
> @@ -12393,9 +12404,15 @@ static bool update_pick_idlest(struct sched_group *idlest,
>  			return false;
>  		break;
>  
> +	case group_asym_packing:
> +		/*
> +		 * Only possible for sched_smt_asym_active().
> +		 * Select the idle SMT that is more preferred.
> +		 */
> +		return sched_asym_prefer(idlest->asym_prefer_cpu,
> +					 group->asym_prefer_cpu);

update_pick_idlest() returns true when @group should replace @idlest, so I think
the operands would need to be reversed.

But even with that, the local-versus-idlest comparison still returns NULL when
both sides are group_asym_packing. Therefore, if the slow path initially lands
on an idle PE1 while PE0 is also idle, it would not switch to PE0.

>  	case group_llc_balance:
>  	case group_imbalanced:
> -	case group_asym_packing:
>  	case group_smt_balance:
>  		/* Those types are not used in the slow wakeup path */
>  		return false;
> ---
> 
> It leads to slightly more branches but they are super predictable when
> iterating at a particular sched_domain level so the overhead should be
> negligible.
> 
> I don't have any strong feelings either ways. Thoughts?

A deeper integration could preserve the normal group_has_spare classification
and use SMT priority only as a tie-breaker between otherwise equivalent
available siblings. It'd also need to handle the local-versus-idlest comparison
and preserve choose_idle_cpu() semantics I think.

For now, applying select_idle_smt_cpu() after the existing slow-path selection
seems simpler. It lets the existing load and capacity logic choose the core
first, then applies the preference only among available siblings within that
core.

Thanks,
-Andrea

Re: [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores

[Andrea Righi]

Hi Prateek,

On Wed, Sep 09, 2026 at 01:39:07AM +0530, K Prateek Nayak wrote:
> Hello Andrea,
> 
> On 9/8/2026 1:53 PM, Andrea Righi wrote:
> > NVIDIA Olympus implements spatial SMT with symmetric steady-state PE
> > capacity but two different resource modes. One-Thread Active mode gives
> > one PE the full core, while waking the other PE restores Two-Thread
> > Active mode and partitions decode, issue, cache, TLB, and vector
> > resources. Returning to full-resource mode requires the sibling to
> > remain in WFI for 10 Ki cycles.
> > 
> > Measurements show that pinned workloads perform equally on either PE,
> > but freely migratable workloads lose substantial throughput when they
> > alternate between PE identities. Consistently selecting PE0 keeps PE1
> > idle, avoids repeated SMT repartitioning, and restores
> > one-thread-per-core performance.
> > 
> > Describe this scheduling preference with SD_ASYM_PACKING and give PE0,
> > identified by MPIDR_EL1.Aff0, the higher arch_asym_cpu_priority(). This
> > is independent of SD_ASYM_CPUCAPACITY: SMT siblings retain equal
> > capacity, while physical cores with different maximum frequencies are
> > handled by a higher scheduling domain.
> > 
> > Firmware currently provides no interface for describing the preferred
> > SMT sibling. Detect Olympus by MIDR until such an interface is
> > available.
> > 
> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> 
> Feel free to include:
> 
> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> 
> Just one nit below ...
> > +/*
> > + * Switching the active PE on an NVIDIA Olympus SMT core can keep the core in
> > + * two-thread active mode, with resources partitioned between the PEs.
> > + *
> > + * Prefer PE0 so PE1 can remain idle and the core can stay in full-resource
> > + * mode. Firmware does not currently describe this preference, so detect
> > + * Olympus by MIDR until a firmware interface is available.
> > + */
> > +static bool olympus_prefer_pe0 __ro_after_init;
> 
> nit. What is the purpose of "olympus_prefer_pe0"?
> 
> > +
> > +#ifdef CONFIG_SCHED_SMT
> > +static int arm64_smt_flags(void)
> > +{
> > +	int flags = cpu_smt_flags();
> > +
> > +	if (olympus_prefer_pe0)
> > +		flags |= SD_ASYM_PACKING;
> 
> If this flag function is being used, olympus_prefer_pe0 is always true,
> and the scheduler will never call into arch_asym_cpu_priority() without
> SD_ASYM_PACKING being set so olympus_prefer_pe0 is always true there
> too.
> 
> Seems redundant unless you were planning to do set_sched_topology()
> unconditionally for whole of arm64.

Right, there's no plan to install this topology unconditionally.

Since arm64_asym_smt_topology is installed only after detecting an Olympus
system with SMT, the boolean doesn't provide any additional state and can be
removed. I'll do that in the next version.

Thanks!
-Andrea

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[K Prateek Nayak]

Hello Andrea,

On 9/9/2026 2:19 AM, Andrea Righi wrote:
>> nit. I personally feel this can be better integrated into the
>> sched_balance_find_dst_cpu(). Something like the following:
>>
>>   (Only build tested)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index b8bd308c2d5b..1012dfb33f08 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -12353,6 +12353,17 @@ static inline void update_sg_wakeup_stats(struct sched_domain *sd,
>>  
>>  	}
>>  
>> +	/*
>> +	 * If we are on a SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING
>> +	 * domain, use the group_asym_packing classification to
>> +	 * decide placement based on rankings of idle siblings.
>> +	 */
>> +	if (unlikely(sched_smt_asym_active() &&
>> +		     (sd->flags & SD_SHARE_CPUCAPACITY) &&
>> +		     (sd->flags & SD_ASYM_PACKING) &&
>> +		     sgs->idle_cpus))
>> +		sgs->group_asym_packing = 1;
> 
> Integrating the preference in the slow-path selection sounds appealing, but I
> don't think group_asym_packing can be used as a destination classificaiton here.
> 
> The intended policy is to prefer PE0 over PE1 when both siblings of the selected
> SMT core are idle. And if PE0 is busy, PE1 should remain a valid destination. It
> shouldn't make a busy PE0 preferable to an idle PE1.
> 
> IIUC group_type is ordered for busiest-group selection, group_asym_packing
> describes a source group whole load should be moved to a "more preferred" CPU.
> Marking an idle SMT group as group_asym_packing could make it rank worse than a
> fully busy group.
> 
> Example: a fork on SMT2 can have the busy local PE0 classified as
> group_has_spare or group_fully_busy, while the idle PE1 is forced to
> group_asym_packing, sched_balance_find_dst_group() can then consider the busy
> local group the better destination and stack the new task on PE0. That may
> preserve one-thread mode for a short task, but it can also reduce throughput for
> sustained work.

Ah! Sorry for not realizing that earlier. Probably needs a special case in
"group_has_spare" instead of using the "group_asym_packing" which is always
considered busier than some other classifications but we can always work on
it later.

For now, I can confirm that this shows no performance impact on systems
I've tested this on (4th gen EPYC, and a 128C Ampere ARM server) and the
fast-paths are inlined correctly into select_task_rq_fair() so feel free to
include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

-- 
Thanks and Regards,
Prateek

Re: [PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Dietmar Eggemann]

On 08.09.26 10:23, Andrea Righi wrote:

[...]

> The series was tested on a two-node Vera system using an 88-thread
> single-precision GEMM on the 88 physical cores of NUMA node 0.

Can we use 'OpenBLAS benchmark/sgemm.goto' as an open alternative for
your NVIDIA internal single-precision GEMM benchmark?

IIUC, you used it for the 'Prefer fully idle cores for NOHZ balancing'
work: https://lore.kernel.org/r/anIq6pU5KXTTFCDN@gpd4

If yes, I assume you would run something like:

export OMP_NUM_THREADS=88
numactl -C XXX --membind=0 ./benchmark/sgemm.goto 16384 16384 16384

Essentially you want to show that those 88 compute intensive tasks each
runs on his own core alone and so you get a higher TFLOPS value.

[...]

Re: [PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Andrea Righi]

Hi Dietmar,

On Wed, Sep 09, 2026 at 09:20:35AM +0200, Dietmar Eggemann wrote:
> On 08.09.26 10:23, Andrea Righi wrote:
> 
> [...]
> 
> > The series was tested on a two-node Vera system using an 88-thread
> > single-precision GEMM on the 88 physical cores of NUMA node 0.
> 
> Can we use 'OpenBLAS benchmark/sgemm.goto' as an open alternative for
> your NVIDIA internal single-precision GEMM benchmark?
> 
> IIUC, you used it for the 'Prefer fully idle cores for NOHZ balancing'
> work: https://lore.kernel.org/r/anIq6pU5KXTTFCDN@gpd4
> 
> If yes, I assume you would run something like:
> 
> export OMP_NUM_THREADS=88
> numactl -C XXX --membind=0 ./benchmark/sgemm.goto 16384 16384 16384
> 
> Essentially you want to show that those 88 compute intensive tasks each
> runs on his own core alone and so you get a higher TFLOPS value.
> 
> [...]

Yes, sure! I'll re-run some tests with that and share the results in a bit.

Thanks,
-Andrea

Re: [PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Andrea Righi]

Hello,

On Wed, Sep 09, 2026 at 09:26:09AM +0200, Andrea Righi wrote:
> Hi Dietmar,
> 
> On Wed, Sep 09, 2026 at 09:20:35AM +0200, Dietmar Eggemann wrote:
> > On 08.09.26 10:23, Andrea Righi wrote:
> > 
> > [...]
> > 
> > > The series was tested on a two-node Vera system using an 88-thread
> > > single-precision GEMM on the 88 physical cores of NUMA node 0.
> > 
> > Can we use 'OpenBLAS benchmark/sgemm.goto' as an open alternative for
> > your NVIDIA internal single-precision GEMM benchmark?
> > 
> > IIUC, you used it for the 'Prefer fully idle cores for NOHZ balancing'
> > work: https://lore.kernel.org/r/anIq6pU5KXTTFCDN@gpd4
> > 
> > If yes, I assume you would run something like:
> > 
> > export OMP_NUM_THREADS=88
> > numactl -C XXX --membind=0 ./benchmark/sgemm.goto 16384 16384 16384
> > 
> > Essentially you want to show that those 88 compute intensive tasks each
> > runs on his own core alone and so you get a higher TFLOPS value.
> > 
> > [...]
> 
> Yes, sure! I'll re-run some tests with that and share the results in a bit.
> 
> Thanks,
> -Andrea

I repeated the tests using the latest patch series [1] both with OpenBLAS and
NVPL (internal GEMM benchmark).

Kernels and test configuration
------------------------------

mainline: Linux 7.3.0-rc2
smt-pe0-prio: Linux 7.3.0-rc2 + patch series [1] applied

Both tests used:
 - 88 threads on NUMA node 0 (CPU list 0-87,176-263)
 - performance governor with cppc_cpufreq
 - same OpenBLAS binary and NVPL container image
 - metrics over 5 repetitions

Results
-------

Delta is (smt-pe0-prio / mainline - 1): higher is better.

 +---------------------+-------+---------------------+-----------------------+--------+
 | Throughput          | Runs  | mainline TFLOP/s    | smt-pe0-prio TFLOP/s  | Delta  |
 +---------------------+-------+---------------------+-----------------------+--------+
 | OpenBLAS            | 5 / 5 | 7.11876 +/- 0.06734 |  7.34669 +/- 0.01936  | +3.20% |
 | NVPL                | 5 / 5 | 9.64742 +/- 0.17311 | 10.29695 +/- 0.01786  | +6.73% |
 +---------------------+-------+----------------------+----------------------+--------+

Hardware statistics
-------------------

ST = single-thread mode
SMT = two-thread mode

Delta is (smt-pe0-prio / mainline - 1): lower is better.

OpenBLAS:
 +------------------------------+----------------------+----------------------+----------+
 | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
 +------------------------------+----------------------+----------------------+----------+
 | ST-to-SMT completed/run      | 10145.6 +/- 1835.2   | 1981.6 +/- 94.3      |  -80.47% |
 | SMT-to-ST completed/run      | 10342.6 +/- 1853.6   | 1946.2 +/- 93.1      |  -81.18% |
 | ST-to-SMT transitions/s      |   845.5 +/- 152.9    |  176.9 +/- 3.4       |  -79.08% |
 | SMT-to-ST transitions/s      |   861.9 +/- 154.5    |  173.7 +/- 1.5       |  -79.84% |
 | ST-to-SMT latency cycles/run | 15.785M +/- 3.315M   |  2.477M +/- 0.090M   |  -84.30% |
 | SMT-to-ST latency cycles/run |  9.545M +/- 1.762M   |  1.815M +/- 0.047M   |  -80.98% |
 +------------------------------+----------------------+----------------------+----------+

NVPL:
 +------------------------------+----------------------+----------------------+----------+
 | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
 +------------------------------+----------------------+----------------------+----------+
 | ST-to-SMT completed/run      | 7771.0 +/- 1312.2    | 2162.6 +/- 137.8     |  -72.17% |
 | SMT-to-ST completed/run      | 7759.8 +/- 1352.2    | 2135.2 +/- 108.0     |  -72.48% |
 | SMT-to-ST aborted/run        |    0.6 +/- 0.5       |    0.2 +/- 0.4       |  -66.67% |
 | ST-to-SMT transitions/s      |  777.1 +/- 131.2     |  251.5 +/- 4.8       |  -67.64% |
 | SMT-to-ST transitions/s      |  776.0 +/- 135.2     |  248.5 +/- 5.8       |  -67.98% |
 | ST-to-SMT latency cycles/run | 13.285M +/- 3.742M   |  2.971M +/- 0.296M   |  -77.64% |
 | SMT-to-ST latency cycles/run |  8.528M +/- 2.223M   |  2.287M +/- 0.126M   |  -73.18% |
 +------------------------------+----------------------+----------------------+----------+

Conclusion
----------

The patch leaves both workloads almost entirely in ST mode and substantially
reduces ST/SMT mode-transition churn.

Relative to mainline, completed ST-to-SMT transitions fall by 80.5% for OpenBLAS
and 72.2% for NVPL. This agrees with the throughput result: the scheduling
preference avoids repeatedly switching the active PE identity and allows cores
to remain in full-resource ST mode for longer intervals.

[1] https://lore.kernel.org/r/20260909062649.469633-1-arighi@nvidia.com

-Andrea

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Vincent Guittot]

On Tue, 8 Sept 2026 at 10:24, Andrea Righi <arighi@nvidia.com> wrote:
>
> POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
> level to order hardware threads. Idle CPU selection does not consult
> that order, so a task can wake on an arbitrary sibling and remain there
> until load balancing corrects the placement. On these systems, that
> initial choice can prevent the core from entering its preferred
> lower-thread resource mode and cause a large and persistent performance
> loss.
>
> When idle selection finds an available CPU in an SMT core, choose the
> highest-priority available sibling. On SMT2 Olympus this only changes
> selection on fully idle cores. A partially idle core has only one
> available CPU. On wider SMT systems such as POWER7, it also fills
> available siblings in priority order while the core is partially busy.
>
> Apply the preference to idle-core and idle-CPU scans,
> asymmetric-capacity scans, target, previous, recently-used CPU fast
> paths and the slow path. Inspect the lowest scheduling domain directly,
> but require both CPUs to share its span because isolcpus can split
> hardware siblings across scheduling domains.
>
> Keep physical-core capacity selection independent from SMT sibling
> ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> maximum capacities, then SD_ASYM_PACKING selects the preferred available
> sibling inside the chosen core, whose siblings continue to share equal
> capacity.
>
> Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> ---
>  kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
>  kernel/sched/sched.h    |  6 +++
>  kernel/sched/topology.c | 36 +++++++++++++++++
>  3 files changed, 110 insertions(+), 17 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index b8bd308c2d5b1..37837c36288a0 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
>         return false;
>  }
>
> +/*
> + * Redirect a CPU to a higher-priority available sibling in its SMT domain,
> + * subject to task affinity.
> + */
> +static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
> +{
> +       struct sched_domain *sd;
> +       int best = cpu;
> +       int sibling;
> +
> +       if (!sched_smt_asym_active())

I wonder if it's worth creating a new static key. All other pieces
related to asym packing use sched_smt_active() to opt out the related
code

Other than that looks good to me

> +               return cpu;
> +
> +       sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> +       if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
> +           !(sd->flags & SD_ASYM_PACKING))
> +               return cpu;
> +
> +       for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
> +               if (sibling == best || !choose_idle_cpu(sibling, p))
> +                       continue;
> +
> +               if (sched_asym_prefer(sibling, best))
> +                       best = sibling;
> +       }
> +
> +       return best;
> +}
> +
>  /*
>   * Scans the local SMT mask to see if the entire core is idle, and records this
>   * information in sd_balance_shared->has_idle_cores.
> @@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>
>         if (choose_idle_cpu(target, p) &&
>             asym_fits_cpu(task_util, util_min, util_max, target))
> -               return target;
> +               goto select_smt_priority;
>
>         /*
>          * If the previous CPU is cache affine and idle, don't be stupid:
> @@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>             asym_fits_cpu(task_util, util_min, util_max, prev)) {
>
>                 if (!static_branch_unlikely(&sched_cluster_active) ||
> -                   cpus_share_resources(prev, target))
> -                       return prev;
> +                   cpus_share_resources(prev, target)) {
> +                       target = prev;
> +                       goto select_smt_priority;
> +               }
>
>                 prev_aff = prev;
>         }
> @@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>             prev == smp_processor_id() &&
>             this_rq()->nr_running <= 1 &&
>             asym_fits_cpu(task_util, util_min, util_max, prev)) {
> -               return prev;
> +               target = prev;
> +               goto select_smt_priority;
>         }
>
>         /* Check a recently used CPU as a potential idle candidate: */
> @@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>             asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
>
>                 if (!static_branch_unlikely(&sched_cluster_active) ||
> -                   cpus_share_resources(recent_used_cpu, target))
> -                       return recent_used_cpu;
> +                   cpus_share_resources(recent_used_cpu, target)) {
> +                       target = recent_used_cpu;
> +                       goto select_smt_priority;
> +               }
>
>         } else {
>                 recent_used_cpu = -1;
> @@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>                  */
>                 if (sd) {
>                         i = select_idle_capacity(p, sd, target);
> -                       return ((unsigned)i < nr_cpumask_bits) ? i : target;
> +                       if ((unsigned int)i < nr_cpumask_bits) {
> +                               target = i;
> +                               goto select_smt_priority;
> +                       }
> +                       return target;
>                 }
>         }
>
> @@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>
>                 if (!has_idle_core && cpus_share_cache(prev, target)) {
>                         i = select_idle_smt(p, sd, prev);
> -                       if ((unsigned int)i < nr_cpumask_bits)
> -                               return i;
> +                       if ((unsigned int)i < nr_cpumask_bits) {
> +                               target = i;
> +                               goto select_smt_priority;
> +                       }
>                 }
>         }
>
>         i = select_idle_cpu(p, sd, has_idle_core, target);
> -       if ((unsigned)i < nr_cpumask_bits)
> -               return i;
> +       if ((unsigned int)i < nr_cpumask_bits) {
> +               target = i;
> +               goto select_smt_priority;
> +       }
>
>         /*
>          * For cluster machines which have lower sharing cache like L2 or
> @@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>          * first. But prev_cpu or recent_used_cpu may also be a good candidate,
>          * use them if possible when no idle CPU found in select_idle_cpu().
>          */
> -       if ((unsigned int)prev_aff < nr_cpumask_bits)
> -               return prev_aff;
> -       if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> -               return recent_used_cpu;
> +       if ((unsigned int)prev_aff < nr_cpumask_bits) {
> +               target = prev_aff;
> +               goto select_smt_priority;
> +       }
> +       if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
> +               target = recent_used_cpu;
> +               goto select_smt_priority;
> +       }
>
>         return target;
> +
> +select_smt_priority:
> +       return select_idle_smt_cpu(p, target);
>  }
>
>  /**
> @@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>         }
>
>         /* Slow path */
> -       if (unlikely(sd))
> -               return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> +       if (unlikely(sd)) {
> +               new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> +               return select_idle_smt_cpu(p, new_cpu);
> +       }
>
>         /* Fast path */
>         if (wake_flags & WF_TTWU)
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 6c3ad70e58b8e..568cb1ed2dd6b 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
>  DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
>
>  extern struct static_key_false sched_asym_cpucapacity;
> +extern struct static_key_false sched_smt_asym_packing;
>  extern struct static_key_false sched_cluster_active;
>
>  static __always_inline bool sched_asym_cpucap_active(void)
> @@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
>         return static_branch_unlikely(&sched_asym_cpucapacity);
>  }
>
> +static __always_inline bool sched_smt_asym_active(void)
> +{
> +       return static_branch_unlikely(&sched_smt_asym_packing);
> +}
> +
>  struct sched_group_capacity {
>         atomic_t                ref;
>         /*
> diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> index 0248227d983a7..06c40eb5932af 100644
> --- a/kernel/sched/topology.c
> +++ b/kernel/sched/topology.c
> @@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
>  DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
>
>  DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
> +DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
>  DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
>
> +static bool has_asym_smt_domain(int cpu)
> +{
> +       struct sched_domain *sd;
> +
> +       for_each_domain(cpu, sd) {
> +               if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> +                       break;
> +
> +               if (sd->flags & SD_ASYM_PACKING)
> +                       return true;
> +       }
> +
> +       return false;
> +}
> +
>  static void update_top_cache_domain(int cpu)
>  {
>         struct sched_domain_shared *sds = NULL;
> @@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
>         struct rq *rq = NULL;
>         int i, ret = -ENOMEM;
>         bool has_asym = false;
> +       bool has_asym_smt = false;
>         bool has_cluster = false;
>
>         if (WARN_ON(cpumask_empty(cpu_map)))
> @@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
>
>                 cpu_attach_domain(sd, d.rd, i);
>
> +               if (has_asym_smt_domain(i))
> +                       has_asym_smt = true;
> +
>                 if (lowest_flag_domain(i, SD_CLUSTER))
>                         has_cluster = true;
>         }
> @@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
>         if (has_asym)
>                 static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
>
> +       if (has_asym_smt)
> +               static_branch_inc_cpuslocked(&sched_smt_asym_packing);
> +
>         if (has_cluster)
>                 static_branch_inc_cpuslocked(&sched_cluster_active);
>
> @@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
>  static void detach_destroy_domains(const struct cpumask *cpu_map)
>  {
>         unsigned int cpu = cpumask_any(cpu_map);
> +       bool has_asym_smt = false;
>         int i;
>
> +       rcu_read_lock();
> +       for_each_cpu(i, cpu_map) {
> +               if (has_asym_smt_domain(i)) {
> +                       has_asym_smt = true;
> +                       break;
> +               }
> +       }
> +       rcu_read_unlock();
> +
>         if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
>                 static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
>
> +       if (has_asym_smt)
> +               static_branch_dec_cpuslocked(&sched_smt_asym_packing);
> +
>         if (static_branch_unlikely(&sched_cluster_active))
>                 static_branch_dec_cpuslocked(&sched_cluster_active);
>
> --
> 2.55.0
>

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Vincent,

On Wed, Sep 09, 2026 at 04:42:44PM +0200, Vincent Guittot wrote:
> On Tue, 8 Sept 2026 at 10:24, Andrea Righi <arighi@nvidia.com> wrote:
> >
> > POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
> > level to order hardware threads. Idle CPU selection does not consult
> > that order, so a task can wake on an arbitrary sibling and remain there
> > until load balancing corrects the placement. On these systems, that
> > initial choice can prevent the core from entering its preferred
> > lower-thread resource mode and cause a large and persistent performance
> > loss.
> >
> > When idle selection finds an available CPU in an SMT core, choose the
> > highest-priority available sibling. On SMT2 Olympus this only changes
> > selection on fully idle cores. A partially idle core has only one
> > available CPU. On wider SMT systems such as POWER7, it also fills
> > available siblings in priority order while the core is partially busy.
> >
> > Apply the preference to idle-core and idle-CPU scans,
> > asymmetric-capacity scans, target, previous, recently-used CPU fast
> > paths and the slow path. Inspect the lowest scheduling domain directly,
> > but require both CPUs to share its span because isolcpus can split
> > hardware siblings across scheduling domains.
> >
> > Keep physical-core capacity selection independent from SMT sibling
> > ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> > maximum capacities, then SD_ASYM_PACKING selects the preferred available
> > sibling inside the chosen core, whose siblings continue to share equal
> > capacity.
> >
> > Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > ---
> >  kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
> >  kernel/sched/sched.h    |  6 +++
> >  kernel/sched/topology.c | 36 +++++++++++++++++
> >  3 files changed, 110 insertions(+), 17 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index b8bd308c2d5b1..37837c36288a0 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
> >         return false;
> >  }
> >
> > +/*
> > + * Redirect a CPU to a higher-priority available sibling in its SMT domain,
> > + * subject to task affinity.
> > + */
> > +static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
> > +{
> > +       struct sched_domain *sd;
> > +       int best = cpu;
> > +       int sibling;
> > +
> > +       if (!sched_smt_asym_active())
> 
> I wonder if it's worth creating a new static key. All other pieces
> related to asym packing use sched_smt_active() to opt out the related
> code

The intent was to keep the additional sd dereference and flag checks out of the
wakeup path for the more common symmetric SMT systems; sched_smt_active()
remains enabled on those systems, the new key lets them return immediately.

Without it, the additional cost should be small when everything is cache-hot
(roughly a couple of dependent loads, flag tests and branches), but this is a
hot path and a cache miss could make it more noticeable. I haven't measured
whether the saving is significant, though. If the extra key and its topology
accounting are not considered worth the potential saving, we can remove it and
use sched_smt_active() instead.

Thanks for looking at this!
-Andrea

> 
> Other than that looks good to me
> 
> > +               return cpu;
> > +
> > +       sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> > +       if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
> > +           !(sd->flags & SD_ASYM_PACKING))
> > +               return cpu;
> > +
> > +       for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
> > +               if (sibling == best || !choose_idle_cpu(sibling, p))
> > +                       continue;
> > +
> > +               if (sched_asym_prefer(sibling, best))
> > +                       best = sibling;
> > +       }
> > +
> > +       return best;
> > +}
> > +
> >  /*
> >   * Scans the local SMT mask to see if the entire core is idle, and records this
> >   * information in sd_balance_shared->has_idle_cores.
> > @@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >
> >         if (choose_idle_cpu(target, p) &&
> >             asym_fits_cpu(task_util, util_min, util_max, target))
> > -               return target;
> > +               goto select_smt_priority;
> >
> >         /*
> >          * If the previous CPU is cache affine and idle, don't be stupid:
> > @@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >             asym_fits_cpu(task_util, util_min, util_max, prev)) {
> >
> >                 if (!static_branch_unlikely(&sched_cluster_active) ||
> > -                   cpus_share_resources(prev, target))
> > -                       return prev;
> > +                   cpus_share_resources(prev, target)) {
> > +                       target = prev;
> > +                       goto select_smt_priority;
> > +               }
> >
> >                 prev_aff = prev;
> >         }
> > @@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >             prev == smp_processor_id() &&
> >             this_rq()->nr_running <= 1 &&
> >             asym_fits_cpu(task_util, util_min, util_max, prev)) {
> > -               return prev;
> > +               target = prev;
> > +               goto select_smt_priority;
> >         }
> >
> >         /* Check a recently used CPU as a potential idle candidate: */
> > @@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >             asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
> >
> >                 if (!static_branch_unlikely(&sched_cluster_active) ||
> > -                   cpus_share_resources(recent_used_cpu, target))
> > -                       return recent_used_cpu;
> > +                   cpus_share_resources(recent_used_cpu, target)) {
> > +                       target = recent_used_cpu;
> > +                       goto select_smt_priority;
> > +               }
> >
> >         } else {
> >                 recent_used_cpu = -1;
> > @@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >                  */
> >                 if (sd) {
> >                         i = select_idle_capacity(p, sd, target);
> > -                       return ((unsigned)i < nr_cpumask_bits) ? i : target;
> > +                       if ((unsigned int)i < nr_cpumask_bits) {
> > +                               target = i;
> > +                               goto select_smt_priority;
> > +                       }
> > +                       return target;
> >                 }
> >         }
> >
> > @@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >
> >                 if (!has_idle_core && cpus_share_cache(prev, target)) {
> >                         i = select_idle_smt(p, sd, prev);
> > -                       if ((unsigned int)i < nr_cpumask_bits)
> > -                               return i;
> > +                       if ((unsigned int)i < nr_cpumask_bits) {
> > +                               target = i;
> > +                               goto select_smt_priority;
> > +                       }
> >                 }
> >         }
> >
> >         i = select_idle_cpu(p, sd, has_idle_core, target);
> > -       if ((unsigned)i < nr_cpumask_bits)
> > -               return i;
> > +       if ((unsigned int)i < nr_cpumask_bits) {
> > +               target = i;
> > +               goto select_smt_priority;
> > +       }
> >
> >         /*
> >          * For cluster machines which have lower sharing cache like L2 or
> > @@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >          * first. But prev_cpu or recent_used_cpu may also be a good candidate,
> >          * use them if possible when no idle CPU found in select_idle_cpu().
> >          */
> > -       if ((unsigned int)prev_aff < nr_cpumask_bits)
> > -               return prev_aff;
> > -       if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> > -               return recent_used_cpu;
> > +       if ((unsigned int)prev_aff < nr_cpumask_bits) {
> > +               target = prev_aff;
> > +               goto select_smt_priority;
> > +       }
> > +       if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
> > +               target = recent_used_cpu;
> > +               goto select_smt_priority;
> > +       }
> >
> >         return target;
> > +
> > +select_smt_priority:
> > +       return select_idle_smt_cpu(p, target);
> >  }
> >
> >  /**
> > @@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >         }
> >
> >         /* Slow path */
> > -       if (unlikely(sd))
> > -               return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > +       if (unlikely(sd)) {
> > +               new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > +               return select_idle_smt_cpu(p, new_cpu);
> > +       }
> >
> >         /* Fast path */
> >         if (wake_flags & WF_TTWU)
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 6c3ad70e58b8e..568cb1ed2dd6b 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
> >  DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
> >
> >  extern struct static_key_false sched_asym_cpucapacity;
> > +extern struct static_key_false sched_smt_asym_packing;
> >  extern struct static_key_false sched_cluster_active;
> >
> >  static __always_inline bool sched_asym_cpucap_active(void)
> > @@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
> >         return static_branch_unlikely(&sched_asym_cpucapacity);
> >  }
> >
> > +static __always_inline bool sched_smt_asym_active(void)
> > +{
> > +       return static_branch_unlikely(&sched_smt_asym_packing);
> > +}
> > +
> >  struct sched_group_capacity {
> >         atomic_t                ref;
> >         /*
> > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> > index 0248227d983a7..06c40eb5932af 100644
> > --- a/kernel/sched/topology.c
> > +++ b/kernel/sched/topology.c
> > @@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
> >  DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
> >
> >  DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
> > +DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
> >  DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
> >
> > +static bool has_asym_smt_domain(int cpu)
> > +{
> > +       struct sched_domain *sd;
> > +
> > +       for_each_domain(cpu, sd) {
> > +               if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> > +                       break;
> > +
> > +               if (sd->flags & SD_ASYM_PACKING)
> > +                       return true;
> > +       }
> > +
> > +       return false;
> > +}
> > +
> >  static void update_top_cache_domain(int cpu)
> >  {
> >         struct sched_domain_shared *sds = NULL;
> > @@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> >         struct rq *rq = NULL;
> >         int i, ret = -ENOMEM;
> >         bool has_asym = false;
> > +       bool has_asym_smt = false;
> >         bool has_cluster = false;
> >
> >         if (WARN_ON(cpumask_empty(cpu_map)))
> > @@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> >
> >                 cpu_attach_domain(sd, d.rd, i);
> >
> > +               if (has_asym_smt_domain(i))
> > +                       has_asym_smt = true;
> > +
> >                 if (lowest_flag_domain(i, SD_CLUSTER))
> >                         has_cluster = true;
> >         }
> > @@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> >         if (has_asym)
> >                 static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
> >
> > +       if (has_asym_smt)
> > +               static_branch_inc_cpuslocked(&sched_smt_asym_packing);
> > +
> >         if (has_cluster)
> >                 static_branch_inc_cpuslocked(&sched_cluster_active);
> >
> > @@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
> >  static void detach_destroy_domains(const struct cpumask *cpu_map)
> >  {
> >         unsigned int cpu = cpumask_any(cpu_map);
> > +       bool has_asym_smt = false;
> >         int i;
> >
> > +       rcu_read_lock();
> > +       for_each_cpu(i, cpu_map) {
> > +               if (has_asym_smt_domain(i)) {
> > +                       has_asym_smt = true;
> > +                       break;
> > +               }
> > +       }
> > +       rcu_read_unlock();
> > +
> >         if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
> >                 static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
> >
> > +       if (has_asym_smt)
> > +               static_branch_dec_cpuslocked(&sched_smt_asym_packing);
> > +
> >         if (static_branch_unlikely(&sched_cluster_active))
> >                 static_branch_dec_cpuslocked(&sched_cluster_active);
> >
> > --
> > 2.55.0
> >

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Vincent Guittot]

On Wed, 9 Sept 2026 at 17:18, Andrea Righi <arighi@nvidia.com> wrote:
>
> Hi Vincent,
>
> On Wed, Sep 09, 2026 at 04:42:44PM +0200, Vincent Guittot wrote:
> > On Tue, 8 Sept 2026 at 10:24, Andrea Righi <arighi@nvidia.com> wrote:
> > >
> > > POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
> > > level to order hardware threads. Idle CPU selection does not consult
> > > that order, so a task can wake on an arbitrary sibling and remain there
> > > until load balancing corrects the placement. On these systems, that
> > > initial choice can prevent the core from entering its preferred
> > > lower-thread resource mode and cause a large and persistent performance
> > > loss.
> > >
> > > When idle selection finds an available CPU in an SMT core, choose the
> > > highest-priority available sibling. On SMT2 Olympus this only changes
> > > selection on fully idle cores. A partially idle core has only one
> > > available CPU. On wider SMT systems such as POWER7, it also fills
> > > available siblings in priority order while the core is partially busy.
> > >
> > > Apply the preference to idle-core and idle-CPU scans,
> > > asymmetric-capacity scans, target, previous, recently-used CPU fast
> > > paths and the slow path. Inspect the lowest scheduling domain directly,
> > > but require both CPUs to share its span because isolcpus can split
> > > hardware siblings across scheduling domains.
> > >
> > > Keep physical-core capacity selection independent from SMT sibling
> > > ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> > > maximum capacities, then SD_ASYM_PACKING selects the preferred available
> > > sibling inside the chosen core, whose siblings continue to share equal
> > > capacity.
> > >
> > > Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> > > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > > ---
> > >  kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
> > >  kernel/sched/sched.h    |  6 +++
> > >  kernel/sched/topology.c | 36 +++++++++++++++++
> > >  3 files changed, 110 insertions(+), 17 deletions(-)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index b8bd308c2d5b1..37837c36288a0 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
> > >         return false;
> > >  }
> > >
> > > +/*
> > > + * Redirect a CPU to a higher-priority available sibling in its SMT domain,
> > > + * subject to task affinity.
> > > + */
> > > +static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
> > > +{
> > > +       struct sched_domain *sd;
> > > +       int best = cpu;
> > > +       int sibling;
> > > +
> > > +       if (!sched_smt_asym_active())
> >
> > I wonder if it's worth creating a new static key. All other pieces
> > related to asym packing use sched_smt_active() to opt out the related
> > code
>
> The intent was to keep the additional sd dereference and flag checks out of the
> wakeup path for the more common symmetric SMT systems; sched_smt_active()
> remains enabled on those systems, the new key lets them return immediately.
>
> Without it, the additional cost should be small when everything is cache-hot
> (roughly a couple of dependent loads, flag tests and branches), but this is a
> hot path and a cache miss could make it more noticeable. I haven't measured
> whether the saving is significant, though. If the extra key and its topology
> accounting are not considered worth the potential saving, we can remove it and
> use sched_smt_active() instead.

If we start having a static key per sub part of a feature like the
asym packing, that can quickly become unmanageable. In this case we
should better have a a static key for whole asym_packing feature
instead

Vincent

>
> Thanks for looking at this!
> -Andrea
>
> >
> > Other than that looks good to me
> >
> > > +               return cpu;
> > > +
> > > +       sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> > > +       if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
> > > +           !(sd->flags & SD_ASYM_PACKING))
> > > +               return cpu;
> > > +
> > > +       for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
> > > +               if (sibling == best || !choose_idle_cpu(sibling, p))
> > > +                       continue;
> > > +
> > > +               if (sched_asym_prefer(sibling, best))
> > > +                       best = sibling;
> > > +       }
> > > +
> > > +       return best;
> > > +}
> > > +
> > >  /*
> > >   * Scans the local SMT mask to see if the entire core is idle, and records this
> > >   * information in sd_balance_shared->has_idle_cores.
> > > @@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >
> > >         if (choose_idle_cpu(target, p) &&
> > >             asym_fits_cpu(task_util, util_min, util_max, target))
> > > -               return target;
> > > +               goto select_smt_priority;
> > >
> > >         /*
> > >          * If the previous CPU is cache affine and idle, don't be stupid:
> > > @@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >             asym_fits_cpu(task_util, util_min, util_max, prev)) {
> > >
> > >                 if (!static_branch_unlikely(&sched_cluster_active) ||
> > > -                   cpus_share_resources(prev, target))
> > > -                       return prev;
> > > +                   cpus_share_resources(prev, target)) {
> > > +                       target = prev;
> > > +                       goto select_smt_priority;
> > > +               }
> > >
> > >                 prev_aff = prev;
> > >         }
> > > @@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >             prev == smp_processor_id() &&
> > >             this_rq()->nr_running <= 1 &&
> > >             asym_fits_cpu(task_util, util_min, util_max, prev)) {
> > > -               return prev;
> > > +               target = prev;
> > > +               goto select_smt_priority;
> > >         }
> > >
> > >         /* Check a recently used CPU as a potential idle candidate: */
> > > @@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >             asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
> > >
> > >                 if (!static_branch_unlikely(&sched_cluster_active) ||
> > > -                   cpus_share_resources(recent_used_cpu, target))
> > > -                       return recent_used_cpu;
> > > +                   cpus_share_resources(recent_used_cpu, target)) {
> > > +                       target = recent_used_cpu;
> > > +                       goto select_smt_priority;
> > > +               }
> > >
> > >         } else {
> > >                 recent_used_cpu = -1;
> > > @@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >                  */
> > >                 if (sd) {
> > >                         i = select_idle_capacity(p, sd, target);
> > > -                       return ((unsigned)i < nr_cpumask_bits) ? i : target;
> > > +                       if ((unsigned int)i < nr_cpumask_bits) {
> > > +                               target = i;
> > > +                               goto select_smt_priority;
> > > +                       }
> > > +                       return target;
> > >                 }
> > >         }
> > >
> > > @@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >
> > >                 if (!has_idle_core && cpus_share_cache(prev, target)) {
> > >                         i = select_idle_smt(p, sd, prev);
> > > -                       if ((unsigned int)i < nr_cpumask_bits)
> > > -                               return i;
> > > +                       if ((unsigned int)i < nr_cpumask_bits) {
> > > +                               target = i;
> > > +                               goto select_smt_priority;
> > > +                       }
> > >                 }
> > >         }
> > >
> > >         i = select_idle_cpu(p, sd, has_idle_core, target);
> > > -       if ((unsigned)i < nr_cpumask_bits)
> > > -               return i;
> > > +       if ((unsigned int)i < nr_cpumask_bits) {
> > > +               target = i;
> > > +               goto select_smt_priority;
> > > +       }
> > >
> > >         /*
> > >          * For cluster machines which have lower sharing cache like L2 or
> > > @@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> > >          * first. But prev_cpu or recent_used_cpu may also be a good candidate,
> > >          * use them if possible when no idle CPU found in select_idle_cpu().
> > >          */
> > > -       if ((unsigned int)prev_aff < nr_cpumask_bits)
> > > -               return prev_aff;
> > > -       if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> > > -               return recent_used_cpu;
> > > +       if ((unsigned int)prev_aff < nr_cpumask_bits) {
> > > +               target = prev_aff;
> > > +               goto select_smt_priority;
> > > +       }
> > > +       if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
> > > +               target = recent_used_cpu;
> > > +               goto select_smt_priority;
> > > +       }
> > >
> > >         return target;
> > > +
> > > +select_smt_priority:
> > > +       return select_idle_smt_cpu(p, target);
> > >  }
> > >
> > >  /**
> > > @@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >         }
> > >
> > >         /* Slow path */
> > > -       if (unlikely(sd))
> > > -               return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > > +       if (unlikely(sd)) {
> > > +               new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> > > +               return select_idle_smt_cpu(p, new_cpu);
> > > +       }
> > >
> > >         /* Fast path */
> > >         if (wake_flags & WF_TTWU)
> > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > index 6c3ad70e58b8e..568cb1ed2dd6b 100644
> > > --- a/kernel/sched/sched.h
> > > +++ b/kernel/sched/sched.h
> > > @@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
> > >  DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
> > >
> > >  extern struct static_key_false sched_asym_cpucapacity;
> > > +extern struct static_key_false sched_smt_asym_packing;
> > >  extern struct static_key_false sched_cluster_active;
> > >
> > >  static __always_inline bool sched_asym_cpucap_active(void)
> > > @@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
> > >         return static_branch_unlikely(&sched_asym_cpucapacity);
> > >  }
> > >
> > > +static __always_inline bool sched_smt_asym_active(void)
> > > +{
> > > +       return static_branch_unlikely(&sched_smt_asym_packing);
> > > +}
> > > +
> > >  struct sched_group_capacity {
> > >         atomic_t                ref;
> > >         /*
> > > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> > > index 0248227d983a7..06c40eb5932af 100644
> > > --- a/kernel/sched/topology.c
> > > +++ b/kernel/sched/topology.c
> > > @@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
> > >  DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
> > >
> > >  DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
> > > +DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
> > >  DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
> > >
> > > +static bool has_asym_smt_domain(int cpu)
> > > +{
> > > +       struct sched_domain *sd;
> > > +
> > > +       for_each_domain(cpu, sd) {
> > > +               if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> > > +                       break;
> > > +
> > > +               if (sd->flags & SD_ASYM_PACKING)
> > > +                       return true;
> > > +       }
> > > +
> > > +       return false;
> > > +}
> > > +
> > >  static void update_top_cache_domain(int cpu)
> > >  {
> > >         struct sched_domain_shared *sds = NULL;
> > > @@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> > >         struct rq *rq = NULL;
> > >         int i, ret = -ENOMEM;
> > >         bool has_asym = false;
> > > +       bool has_asym_smt = false;
> > >         bool has_cluster = false;
> > >
> > >         if (WARN_ON(cpumask_empty(cpu_map)))
> > > @@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> > >
> > >                 cpu_attach_domain(sd, d.rd, i);
> > >
> > > +               if (has_asym_smt_domain(i))
> > > +                       has_asym_smt = true;
> > > +
> > >                 if (lowest_flag_domain(i, SD_CLUSTER))
> > >                         has_cluster = true;
> > >         }
> > > @@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> > >         if (has_asym)
> > >                 static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
> > >
> > > +       if (has_asym_smt)
> > > +               static_branch_inc_cpuslocked(&sched_smt_asym_packing);
> > > +
> > >         if (has_cluster)
> > >                 static_branch_inc_cpuslocked(&sched_cluster_active);
> > >
> > > @@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
> > >  static void detach_destroy_domains(const struct cpumask *cpu_map)
> > >  {
> > >         unsigned int cpu = cpumask_any(cpu_map);
> > > +       bool has_asym_smt = false;
> > >         int i;
> > >
> > > +       rcu_read_lock();
> > > +       for_each_cpu(i, cpu_map) {
> > > +               if (has_asym_smt_domain(i)) {
> > > +                       has_asym_smt = true;
> > > +                       break;
> > > +               }
> > > +       }
> > > +       rcu_read_unlock();
> > > +
> > >         if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
> > >                 static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
> > >
> > > +       if (has_asym_smt)
> > > +               static_branch_dec_cpuslocked(&sched_smt_asym_packing);
> > > +
> > >         if (static_branch_unlikely(&sched_cluster_active))
> > >                 static_branch_dec_cpuslocked(&sched_cluster_active);
> > >
> > > --
> > > 2.55.0
> > >

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Vincent,

On Wed, Sep 09, 2026 at 05:42:43PM +0200, Vincent Guittot wrote:
> On Wed, 9 Sept 2026 at 17:18, Andrea Righi <arighi@nvidia.com> wrote:
> >
> > Hi Vincent,
> >
> > On Wed, Sep 09, 2026 at 04:42:44PM +0200, Vincent Guittot wrote:
> > > On Tue, 8 Sept 2026 at 10:24, Andrea Righi <arighi@nvidia.com> wrote:
> > > >
> > > > POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
> > > > level to order hardware threads. Idle CPU selection does not consult
> > > > that order, so a task can wake on an arbitrary sibling and remain there
> > > > until load balancing corrects the placement. On these systems, that
> > > > initial choice can prevent the core from entering its preferred
> > > > lower-thread resource mode and cause a large and persistent performance
> > > > loss.
> > > >
> > > > When idle selection finds an available CPU in an SMT core, choose the
> > > > highest-priority available sibling. On SMT2 Olympus this only changes
> > > > selection on fully idle cores. A partially idle core has only one
> > > > available CPU. On wider SMT systems such as POWER7, it also fills
> > > > available siblings in priority order while the core is partially busy.
> > > >
> > > > Apply the preference to idle-core and idle-CPU scans,
> > > > asymmetric-capacity scans, target, previous, recently-used CPU fast
> > > > paths and the slow path. Inspect the lowest scheduling domain directly,
> > > > but require both CPUs to share its span because isolcpus can split
> > > > hardware siblings across scheduling domains.
> > > >
> > > > Keep physical-core capacity selection independent from SMT sibling
> > > > ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> > > > maximum capacities, then SD_ASYM_PACKING selects the preferred available
> > > > sibling inside the chosen core, whose siblings continue to share equal
> > > > capacity.
> > > >
> > > > Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> > > > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > > > ---
> > > >  kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
> > > >  kernel/sched/sched.h    |  6 +++
> > > >  kernel/sched/topology.c | 36 +++++++++++++++++
> > > >  3 files changed, 110 insertions(+), 17 deletions(-)
> > > >
> > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > index b8bd308c2d5b1..37837c36288a0 100644
> > > > --- a/kernel/sched/fair.c
> > > > +++ b/kernel/sched/fair.c
> > > > @@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
> > > >         return false;
> > > >  }
> > > >
> > > > +/*
> > > > + * Redirect a CPU to a higher-priority available sibling in its SMT domain,
> > > > + * subject to task affinity.
> > > > + */
> > > > +static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
> > > > +{
> > > > +       struct sched_domain *sd;
> > > > +       int best = cpu;
> > > > +       int sibling;
> > > > +
> > > > +       if (!sched_smt_asym_active())
> > >
> > > I wonder if it's worth creating a new static key. All other pieces
> > > related to asym packing use sched_smt_active() to opt out the related
> > > code
> >
> > The intent was to keep the additional sd dereference and flag checks out of the
> > wakeup path for the more common symmetric SMT systems; sched_smt_active()
> > remains enabled on those systems, the new key lets them return immediately.
> >
> > Without it, the additional cost should be small when everything is cache-hot
> > (roughly a couple of dependent loads, flag tests and branches), but this is a
> > hot path and a cache miss could make it more noticeable. I haven't measured
> > whether the saving is significant, though. If the extra key and its topology
> > accounting are not considered worth the potential saving, we can remove it and
> > use sched_smt_active() instead.
> 
> If we start having a static key per sub part of a feature like the
> asym packing, that can quickly become unmanageable. In this case we
> should better have a a static key for whole asym_packing feature
> instead

Makes sense, I'll remove the static key and use sched_smt_active().

Thanks,
-Andrea

Re: [PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Dietmar Eggemann]

On 09.09.26 14:39, Andrea Righi wrote:
> Hello,
> 
> On Wed, Sep 09, 2026 at 09:26:09AM +0200, Andrea Righi wrote:
>> Hi Dietmar,
>>
>> On Wed, Sep 09, 2026 at 09:20:35AM +0200, Dietmar Eggemann wrote:
>>> On 08.09.26 10:23, Andrea Righi wrote:
>>>
>>> [...]
>>>
>>>> The series was tested on a two-node Vera system using an 88-thread
>>>> single-precision GEMM on the 88 physical cores of NUMA node 0.
>>>
>>> Can we use 'OpenBLAS benchmark/sgemm.goto' as an open alternative for
>>> your NVIDIA internal single-precision GEMM benchmark?
>>>
>>> IIUC, you used it for the 'Prefer fully idle cores for NOHZ balancing'
>>> work: https://lore.kernel.org/r/anIq6pU5KXTTFCDN@gpd4
>>>
>>> If yes, I assume you would run something like:
>>>
>>> export OMP_NUM_THREADS=88
>>> numactl -C XXX --membind=0 ./benchmark/sgemm.goto 16384 16384 16384
>>>
>>> Essentially you want to show that those 88 compute intensive tasks each
>>> runs on his own core alone and so you get a higher TFLOPS value.
>>>
>>> [...]
>>
>> Yes, sure! I'll re-run some tests with that and share the results in a bit.
>>
>> Thanks,
>> -Andrea
> 
> I repeated the tests using the latest patch series [1] both with OpenBLAS and
> NVPL (internal GEMM benchmark).
> 
> Kernels and test configuration
> ------------------------------
> 
> mainline: Linux 7.3.0-rc2
> smt-pe0-prio: Linux 7.3.0-rc2 + patch series [1] applied
> 
> Both tests used:
>  - 88 threads on NUMA node 0 (CPU list 0-87,176-263)
>  - performance governor with cppc_cpufreq
>  - same OpenBLAS binary and NVPL container image
>  - metrics over 5 repetitions
> 
> Results
> -------
> 
> Delta is (smt-pe0-prio / mainline - 1): higher is better.
> 
>  +---------------------+-------+---------------------+-----------------------+--------+
>  | Throughput          | Runs  | mainline TFLOP/s    | smt-pe0-prio TFLOP/s  | Delta  |
>  +---------------------+-------+---------------------+-----------------------+--------+
>  | OpenBLAS            | 5 / 5 | 7.11876 +/- 0.06734 |  7.34669 +/- 0.01936  | +3.20% |
>  | NVPL                | 5 / 5 | 9.64742 +/- 0.17311 | 10.29695 +/- 0.01786  | +6.73% |
>  +---------------------+-------+----------------------+----------------------+--------+
> 
> Hardware statistics
> -------------------
> 
> ST = single-thread mode
> SMT = two-thread mode
> 
> Delta is (smt-pe0-prio / mainline - 1): lower is better.

Thanks for the test results. Good to see that we have an openly
available benchmark for this.

> OpenBLAS:
>  +------------------------------+----------------------+----------------------+----------+
>  | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
>  +------------------------------+----------------------+----------------------+----------+
>  | ST-to-SMT completed/run      | 10145.6 +/- 1835.2   | 1981.6 +/- 94.3      |  -80.47% |
>  | SMT-to-ST completed/run      | 10342.6 +/- 1853.6   | 1946.2 +/- 93.1      |  -81.18% |
>  | ST-to-SMT transitions/s      |   845.5 +/- 152.9    |  176.9 +/- 3.4       |  -79.08% |
>  | SMT-to-ST transitions/s      |   861.9 +/- 154.5    |  173.7 +/- 1.5       |  -79.84% |
>  | ST-to-SMT latency cycles/run | 15.785M +/- 3.315M   |  2.477M +/- 0.090M   |  -84.30% |
>  | SMT-to-ST latency cycles/run |  9.545M +/- 1.762M   |  1.815M +/- 0.047M   |  -80.98% |
>  +------------------------------+----------------------+----------------------+----------+
> 
> NVPL:
>  +------------------------------+----------------------+----------------------+----------+
>  | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
>  +------------------------------+----------------------+----------------------+----------+
>  | ST-to-SMT completed/run      | 7771.0 +/- 1312.2    | 2162.6 +/- 137.8     |  -72.17% |
>  | SMT-to-ST completed/run      | 7759.8 +/- 1352.2    | 2135.2 +/- 108.0     |  -72.48% |
>  | SMT-to-ST aborted/run        |    0.6 +/- 0.5       |    0.2 +/- 0.4       |  -66.67% |
>  | ST-to-SMT transitions/s      |  777.1 +/- 131.2     |  251.5 +/- 4.8       |  -67.64% |
>  | SMT-to-ST transitions/s      |  776.0 +/- 135.2     |  248.5 +/- 5.8       |  -67.98% |
>  | ST-to-SMT latency cycles/run | 13.285M +/- 3.742M   |  2.971M +/- 0.296M   |  -77.64% |
>  | SMT-to-ST latency cycles/run |  8.528M +/- 2.223M   |  2.287M +/- 0.126M   |  -73.18% |
>  +------------------------------+----------------------+----------------------+----------+
> 
> Conclusion
> ----------
> 
> The patch leaves both workloads almost entirely in ST mode and substantially
> reduces ST/SMT mode-transition churn.
> 
> Relative to mainline, completed ST-to-SMT transitions fall by 80.5% for OpenBLAS
> and 72.2% for NVPL. This agrees with the throughput result: the scheduling
> preference avoids repeatedly switching the active PE identity and allows cores
> to remain in full-resource ST mode for longer intervals.
> 
> [1] https://lore.kernel.org/r/20260909062649.469633-1-arighi@nvidia.com
I was able to run 'BLAS SGEMM' on ThunderX2 (ARM64) (SMT-4) on
'tip/sched/core' (base) and v1 and v5 (w/ small changes to get it
running on THX2).

$ awk '/^cpu0[[:space:]]/{print
$1;show=1;next}/^cpu[0-9]+[[:space:]]/&&show{exit}show&&/^domain/{print
$1,$2,$3}' /proc/schedstat

cpu0
domain0 SMT
00000000,00000000,00000000,00000000,00000001,00000001,00000001,00000001
domain1 MC
00000000,00000000,00000000,00000000,ffffffff,ffffffff,ffffffff,ffffffff
domain2 NUMA
ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff

$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 ... 127
node 0 size: 64270 MB
node 0 free: 62366 MB
node 1 cpus: 128 ... 255
node 1 size: 128599 MB
node 1 free: 126960 MB
node distances:
node     0    1
   0:   10   20
   1:   20   10

---

export OMP_NUM_THREADS=32
export BM="./OpenBLAS/benchmark/sgemm.goto 16384 16384 16384"

(a) 8 cores/32 CPUs (hw threads)
    $ numactl -C 0-7,32-39,64-71,96-103 -m 0 $BM

(b) 16 cores/32 CPUs (hw threads)
    $ numactl -C 0-15,32-47 -m 0 $BM

(c) 32 cores/32 CPUs (hw threads):
    $ numactl -C 0-31 -m 0 $BM

(d) Entire NUMA node 0 (unconstrained)	<-- !!!
    $ numactl -C 0-127 -m 0 $BM

(e) 32 cores/32 CPUs (hw threads):
    $ numactl -C 31-63 -m 0 $BM

(f) 32 cores/32 CPUs (hw threads):
    $ numactl -C 64-95 -m 0 $BM

(g) 32 cores/32 CPUs (hw threads):
    $ numactl -C 96-127 -m 0 $BM

---

MFLOPS values:

       v5        v1        base

(a) 225349.00 227305.68 227246.94

(b) 376297.15 373908.56 380849.55

(c) 877144.86 867642.35 861639.57

(d) 868285.22 865953.05 861662.81	<-- !!!

(e) 868182.05

(f) 866835.49

(g) 867389.74

---

So it doesn't seem to change much (v5 vs. base (d)).

When I look into the trace file then I can see that I have 32 benchmark
tasks running for 10s constantly (no sleep/wakeup) so with 32 cores and
32 task, the SMT aware select_idle_sibling() (symmetric CPU capacity)
should already place 1 task per core and then the tasks run there for
10s w/o migration. So I can't see how you're improvement can happen
since the benchmark has tasks <= cores (32 in my case, 88 in yours)?

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Dietmar Eggemann]

On 09.09.26 08:26, Andrea Righi wrote:

[...]

> +static bool has_asym_smt_domain(int cpu)
> +{
> +	struct sched_domain *sd;
> +
> +	for_each_domain(cpu, sd) {
> +		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> +			break;
> +
> +		if (sd->flags & SD_ASYM_PACKING)
> +			return true;
> +	}
> +
> +	return false;
> +}

Replace for_each_domain() w/ sd = rcu_dereference_all(cpu_rq(cpu)->sd)
here as well?

[...]

[PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
level to order hardware threads. Idle CPU selection does not consult
that order, so a task can wake on an arbitrary sibling and remain there
until load balancing corrects the placement. On these systems, that
initial choice can prevent the core from entering its preferred
lower-thread resource mode and cause a large and persistent performance
loss.

When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.

Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, target, previous, recently-used CPU fast
paths and the slow path. Inspect the lowest scheduling domain directly,
but require both CPUs to share its span because isolcpus can split
hardware siblings across scheduling domains.

Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.

Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 kernel/sched/fair.c     | 85 ++++++++++++++++++++++++++++++++---------
 kernel/sched/sched.h    |  6 +++
 kernel/sched/topology.c | 36 +++++++++++++++++
 3 files changed, 110 insertions(+), 17 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8bd308c2d5b1..37837c36288a0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
 	return false;
 }
 
+/*
+ * Redirect a CPU to a higher-priority available sibling in its SMT domain,
+ * subject to task affinity.
+ */
+static inline int select_idle_smt_cpu(struct task_struct *p, int cpu)
+{
+	struct sched_domain *sd;
+	int best = cpu;
+	int sibling;
+
+	if (!sched_smt_asym_active())
+		return cpu;
+
+	sd = rcu_dereference_all(cpu_rq(cpu)->sd);
+	if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
+	    !(sd->flags & SD_ASYM_PACKING))
+		return cpu;
+
+	for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
+		if (sibling == best || !choose_idle_cpu(sibling, p))
+			continue;
+
+		if (sched_asym_prefer(sibling, best))
+			best = sibling;
+	}
+
+	return best;
+}
+
 /*
  * Scans the local SMT mask to see if the entire core is idle, and records this
  * information in sd_balance_shared->has_idle_cores.
@@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		goto select_smt_priority;
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(prev, target))
-			return prev;
+		    cpus_share_resources(prev, target)) {
+			target = prev;
+			goto select_smt_priority;
+		}
 
 		prev_aff = prev;
 	}
@@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    prev == smp_processor_id() &&
 	    this_rq()->nr_running <= 1 &&
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
-		return prev;
+		target = prev;
+		goto select_smt_priority;
 	}
 
 	/* Check a recently used CPU as a potential idle candidate: */
@@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+		    cpus_share_resources(recent_used_cpu, target)) {
+			target = recent_used_cpu;
+			goto select_smt_priority;
+		}
 
 	} else {
 		recent_used_cpu = -1;
@@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		 */
 		if (sd) {
 			i = select_idle_capacity(p, sd, target);
-			return ((unsigned)i < nr_cpumask_bits) ? i : target;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
+			return target;
 		}
 	}
 
@@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!has_idle_core && cpus_share_cache(prev, target)) {
 			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
 		}
 	}
 
 	i = select_idle_cpu(p, sd, has_idle_core, target);
-	if ((unsigned)i < nr_cpumask_bits)
-		return i;
+	if ((unsigned int)i < nr_cpumask_bits) {
+		target = i;
+		goto select_smt_priority;
+	}
 
 	/*
 	 * For cluster machines which have lower sharing cache like L2 or
@@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
 	 * use them if possible when no idle CPU found in select_idle_cpu().
 	 */
-	if ((unsigned int)prev_aff < nr_cpumask_bits)
-		return prev_aff;
-	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
-		return recent_used_cpu;
+	if ((unsigned int)prev_aff < nr_cpumask_bits) {
+		target = prev_aff;
+		goto select_smt_priority;
+	}
+	if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
+		target = recent_used_cpu;
+		goto select_smt_priority;
+	}
 
 	return target;
+
+select_smt_priority:
+	return select_idle_smt_cpu(p, target);
 }
 
 /**
@@ -9747,8 +9796,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	}
 
 	/* Slow path */
-	if (unlikely(sd))
-		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
+	if (unlikely(sd)) {
+		new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
+		return select_idle_smt_cpu(p, new_cpu);
+	}
 
 	/* Fast path */
 	if (wake_flags & WF_TTWU)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c3ad70e58b8e..568cb1ed2dd6b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 extern struct static_key_false sched_asym_cpucapacity;
+extern struct static_key_false sched_smt_asym_packing;
 extern struct static_key_false sched_cluster_active;
 
 static __always_inline bool sched_asym_cpucap_active(void)
@@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
 	return static_branch_unlikely(&sched_asym_cpucapacity);
 }
 
+static __always_inline bool sched_smt_asym_active(void)
+{
+	return static_branch_unlikely(&sched_smt_asym_packing);
+}
+
 struct sched_group_capacity {
 	atomic_t		ref;
 	/*
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 0248227d983a7..06c40eb5932af 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
+DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+static bool has_asym_smt_domain(int cpu)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if (sd->flags & SD_ASYM_PACKING)
+			return true;
+	}
+
+	return false;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	struct rq *rq = NULL;
 	int i, ret = -ENOMEM;
 	bool has_asym = false;
+	bool has_asym_smt = false;
 	bool has_cluster = false;
 
 	if (WARN_ON(cpumask_empty(cpu_map)))
@@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 
 		cpu_attach_domain(sd, d.rd, i);
 
+		if (has_asym_smt_domain(i))
+			has_asym_smt = true;
+
 		if (lowest_flag_domain(i, SD_CLUSTER))
 			has_cluster = true;
 	}
@@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	if (has_asym)
 		static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_inc_cpuslocked(&sched_smt_asym_packing);
+
 	if (has_cluster)
 		static_branch_inc_cpuslocked(&sched_cluster_active);
 
@@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
 static void detach_destroy_domains(const struct cpumask *cpu_map)
 {
 	unsigned int cpu = cpumask_any(cpu_map);
+	bool has_asym_smt = false;
 	int i;
 
+	rcu_read_lock();
+	for_each_cpu(i, cpu_map) {
+		if (has_asym_smt_domain(i)) {
+			has_asym_smt = true;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
 	if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
 		static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_dec_cpuslocked(&sched_smt_asym_packing);
+
 	if (static_branch_unlikely(&sched_cluster_active))
 		static_branch_dec_cpuslocked(&sched_cluster_active);
 
-- 
2.55.0

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Srikar,

On Tue, Sep 08, 2026 at 11:07:20AM +0530, Srikar Dronamraju wrote:
> * Andrea Righi <arighi@nvidia.com> [2026-09-04 11:18:05]:
> 
> Hi Andrea,
> 
> > POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity
> > SMT level to order hardware threads. Idle CPU selection does not consult
> > that order, so a task can wake on an arbitrary sibling and remain there
> > until load balancing corrects the placement. On these systems, that
> > initial choice can prevent the core from entering its preferred
> > lower-thread resource mode and cause a large and persistent performance
> > loss.
> > 
> 
> > When idle selection finds an available CPU in an SMT core, choose the
> > highest-priority available sibling. On SMT2 Olympus this only changes
> > selection on fully idle cores. A partially idle core has only one
> > available CPU. On wider SMT systems such as POWER7, it also fills
> > available siblings in priority order while the core is partially busy.
> > 
> 
> Don't we need changes in the slow path too?
> Something like this?
> https://lore.kernel.org/all/20251204175405.1511340-2-srikar@linux.ibm.com/T/#u

Ah yes, good catch! I'll include the WF_FORK / WF_EXEC slow-path as well.

> 
> > Apply the preference to idle-core and idle-CPU scans,
> > asymmetric-capacity scans, and the target, previous, and recently-used
> > CPU fast paths. Inspect the lowest scheduling domain directly, but
> > require both CPUs to share its span because isolcpus can split hardware
> > siblings across scheduling domains.
> > 
> > Keep physical-core capacity selection independent from SMT sibling
> > ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> > maximum capacities, then SD_ASYM_PACKING selects the preferred available
> > sibling inside the chosen core, whose siblings continue to share equal
> > capacity.
> > 
> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > ---
> >  kernel/sched/fair.c     | 79 ++++++++++++++++++++++++++++++++++++-----
> >  kernel/sched/sched.h    |  6 ++++
> >  kernel/sched/topology.c | 36 +++++++++++++++++++
> >  3 files changed, 112 insertions(+), 9 deletions(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index b8bd308c2d5b1..ff9a7b1fcbe7f 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8587,6 +8587,63 @@ static inline bool test_idle_cores(int cpu)
> >  	return false;
> >  }
> >  
> > +/*
> > + * Return true when @cpu has a higher asymmetric-packing priority than
> > + * @other in their shared SMT scheduling domain.
> > + */
> > +static bool sched_smt_asym_prefer(int cpu, int other)
> > +{
> > +	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> > +
> > +	if (!sd)
> > +		return false;
> > +
> > +	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
> > +	    !(sd->flags & SD_ASYM_PACKING))
> > +		return false;
> > +
> > +	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
> > +		return false;
> > +
> > +	return sched_asym_prefer(cpu, other);
> > +}
> > +
> > +/*
> > + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
> > + */
> > +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> > +{
> > +	int best = cpu;
> > +	int sibling;
> > +
> > +	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
> > +		if (sibling == best || !choose_idle_cpu(sibling, p))
> > +			continue;
> > +
> > +		if (sched_smt_asym_prefer(sibling, best))
> > +			best = sibling;
> > +	}
> > +
> > +	return best;
> > +}
> > +
> > +static inline int
> > +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> > +{
> > +	if (!sched_smt_asym_active())
> > +		return cpu;
> > +
> > +	return __select_idle_smt_cpu(p, cpu, cpus);
> 
> Nit: I see __select_idle_smt_cpu called only here.
> Cant we fold __select_idle_smt_cpu() here itself.

Agreed, will fold in v4.

> 
> > +}
> > +
> > +/*
> > + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
> > + */
> > +static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
> > +{
> > +	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
> > +}
> > +
> 
> nit: Can we also replace select_idle_smt_priority with select_idle_smt_cpu()
> itself.

Ack, we can use select_idle_smt_cpu() directly.

> 
> Otherwise looks good to me
> 
> Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>

Thanks for taking a look at this!
-Andrea

> 
> -- 
> Thanks and Regards
> Srikar Dronamraju

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Srikar Dronamraju]

* Andrea Righi <arighi@nvidia.com> [2026-09-04 11:18:05]:

Hi Andrea,

> POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity
> SMT level to order hardware threads. Idle CPU selection does not consult
> that order, so a task can wake on an arbitrary sibling and remain there
> until load balancing corrects the placement. On these systems, that
> initial choice can prevent the core from entering its preferred
> lower-thread resource mode and cause a large and persistent performance
> loss.
> 

> When idle selection finds an available CPU in an SMT core, choose the
> highest-priority available sibling. On SMT2 Olympus this only changes
> selection on fully idle cores. A partially idle core has only one
> available CPU. On wider SMT systems such as POWER7, it also fills
> available siblings in priority order while the core is partially busy.
> 

Don't we need changes in the slow path too?
Something like this?
https://lore.kernel.org/all/20251204175405.1511340-2-srikar@linux.ibm.com/T/#u

> Apply the preference to idle-core and idle-CPU scans,
> asymmetric-capacity scans, and the target, previous, and recently-used
> CPU fast paths. Inspect the lowest scheduling domain directly, but
> require both CPUs to share its span because isolcpus can split hardware
> siblings across scheduling domains.
> 
> Keep physical-core capacity selection independent from SMT sibling
> ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
> maximum capacities, then SD_ASYM_PACKING selects the preferred available
> sibling inside the chosen core, whose siblings continue to share equal
> capacity.
> 
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> ---
>  kernel/sched/fair.c     | 79 ++++++++++++++++++++++++++++++++++++-----
>  kernel/sched/sched.h    |  6 ++++
>  kernel/sched/topology.c | 36 +++++++++++++++++++
>  3 files changed, 112 insertions(+), 9 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index b8bd308c2d5b1..ff9a7b1fcbe7f 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8587,6 +8587,63 @@ static inline bool test_idle_cores(int cpu)
>  	return false;
>  }
>  
> +/*
> + * Return true when @cpu has a higher asymmetric-packing priority than
> + * @other in their shared SMT scheduling domain.
> + */
> +static bool sched_smt_asym_prefer(int cpu, int other)
> +{
> +	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> +
> +	if (!sd)
> +		return false;
> +
> +	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
> +	    !(sd->flags & SD_ASYM_PACKING))
> +		return false;
> +
> +	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
> +		return false;
> +
> +	return sched_asym_prefer(cpu, other);
> +}
> +
> +/*
> + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
> + */
> +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> +{
> +	int best = cpu;
> +	int sibling;
> +
> +	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
> +		if (sibling == best || !choose_idle_cpu(sibling, p))
> +			continue;
> +
> +		if (sched_smt_asym_prefer(sibling, best))
> +			best = sibling;
> +	}
> +
> +	return best;
> +}
> +
> +static inline int
> +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> +{
> +	if (!sched_smt_asym_active())
> +		return cpu;
> +
> +	return __select_idle_smt_cpu(p, cpu, cpus);

Nit: I see __select_idle_smt_cpu called only here.
Cant we fold __select_idle_smt_cpu() here itself.

> +}
> +
> +/*
> + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
> + */
> +static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
> +{
> +	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
> +}
> +

nit: Can we also replace select_idle_smt_priority with select_idle_smt_cpu()
itself.

Otherwise looks good to me

Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>

-- 
Thanks and Regards
Srikar Dronamraju

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Shrikanth Hegde]

On 9/7/26 9:27 AM, K Prateek Nayak wrote:
> Hello Andrea,
> 
> On 9/4/2026 2:48 PM, Andrea Righi wrote:
>> +/*
>> + * Return true when @cpu has a higher asymmetric-packing priority than
>> + * @other in their shared SMT scheduling domain.
>> + */
>> +static bool sched_smt_asym_prefer(int cpu, int other)
>> +{
>> +	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
>> +
>> +	if (!sd)
>> +		return false;
>> +
>> +	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
>> +	    !(sd->flags & SD_ASYM_PACKING))
>> +		return false;
>> +
>> +	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
>> +		return false;
>> +
>> +	return sched_asym_prefer(cpu, other);
>> +}
>> +
>> +/*
>> + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
>> + */
>> +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
>> +{
>> +	int best = cpu;
>> +	int sibling;
>> +
>> +	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
>> +		if (sibling == best || !choose_idle_cpu(sibling, p))
>> +			continue;
>> +
>> +		if (sched_smt_asym_prefer(sibling, best))
>> +			best = sibling;
> 
> nit. Since sched_smt_asym_prefer() is only used here, and we know rq->sd
> is the one that can have SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING, perhaps
> you can inline the check here do a:
> 
>      sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> 
>      if (!sd)
>           return cpu;
> 
>     if (!(sd->flags & SD_SHARE_CPUCAPACITY) || !(sd->flags & SD_ASYM_PACKING))
>         return cpu;
> 
>     for_each_cpu_and (sibling, sched_domain_span(sd), cpus) {
>         ...
>     }
> 
>    ...
> 
> 
> That way, you don't need to dereference cpu_rq(cpu)->sd every time in
> sched_smt_asym_prefer() and check cpumask_test_cpu(). Both, domain
> span and task affinity will be covered at once.
> 
> Thoughts?
> 
>> +	}
>> +
>> +	return best;
>> +}
>> +
>> +static inline int
>> +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
>> +{
>> +	if (!sched_smt_asym_active())
>> +		return cpu;
>> +
>> +	return __select_idle_smt_cpu(p, cpu, cpus);
>> +}
>> +
>> +/*
>> + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
>> + */
>> +static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
>> +{
>> +	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
>> +}
>> +
>>   /*
>>    * Scans the local SMT mask to see if the entire core is idle, and records this
>>    * information in sd_balance_shared->has_idle_cores.
>> @@ -8645,7 +8702,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
>>   	}
>>   
>>   	if (idle)
>> -		return core;
>> +		return select_idle_smt_cpu(p, core, cpus);
>>   
>>   	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
>>   	return -1;
>> @@ -8668,7 +8725,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>>   		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
>>   			continue;
>>   		if (choose_idle_cpu(cpu, p))
>> -			return cpu;
>> +			return select_idle_smt_priority(p, cpu);
>>   	}
>>   
>>   	return -1;
>> @@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>>   						return -1;
>>   					idle_cpu = __select_idle_cpu(cpu, p);
>>   					if ((unsigned int)idle_cpu < nr_cpumask_bits)
>> -						return idle_cpu;
>> +						return select_idle_smt_priority(p, idle_cpu);
> 
> Question for Shrikanth: On larger SMT (SMT-4, SMT-8), does the ranking
> make that big of a difference if the core is already busy?
> 

Only on Power7 we had AYSM PACKING.
There IPC of CPU0 > CPU1 > CPU2 > CPU4 for the four siblings IIRC irrespective of busy
or idle.

PS: I haven't seen the patches in detail yet.

> Does the overehead of additional search get offset by the benefit of
> being placed on a better ranked thread? If not, maybe the paths for
> !has_idle_core can stay as is?
> 
>>   				}
>>   			}
>>   			cpumask_andnot(cpus, cpus, sched_group_span(sg));
>> @@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>>   	if (has_idle_core)
>>   		set_idle_cores(target, false);
>>   
>> -	return idle_cpu;
>> +	return (unsigned int)idle_cpu < nr_cpumask_bits ?
>> +		select_idle_smt_priority(p, idle_cpu) : idle_cpu;
> 
> Since every path does a select_idle_smt_priority() - be it coming from
> select_idle_core(), the early-return from the cluster scan, or just an
> idle CPU from the LLc scan, can't we simply just do it once in
> select_idle_sibling()?
> 
> Something like:
> 
>    (Only build tested)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index f79fcba4afec..7c97585141dd 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8964,7 +8964,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   
>   	if (choose_idle_cpu(target, p) &&
>   	    asym_fits_cpu(task_util, util_min, util_max, target))
> -		return target;
> +		goto out;
>   
>   	/*
>   	 * If the previous CPU is cache affine and idle, don't be stupid:
> @@ -8974,8 +8974,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
>   
>   		if (!static_branch_unlikely(&sched_cluster_active) ||
> -		    cpus_share_resources(prev, target))
> -			return prev;
> +		    cpus_share_resources(prev, target)) {
> +			target = prev;
> +			goto out;
> +		}
>   
>   		prev_aff = prev;
>   	}
> @@ -8993,7 +8995,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   	    prev == smp_processor_id() &&
>   	    this_rq()->nr_running <= 1 &&
>   	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
> -		return prev;
> +		target = prev;
> +		goto out;
>   	}
>   
>   	/* Check a recently used CPU as a potential idle candidate: */
> @@ -9007,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
>   
>   		if (!static_branch_unlikely(&sched_cluster_active) ||
> -		    cpus_share_resources(recent_used_cpu, target))
> -			return recent_used_cpu;
> +		    cpus_share_resources(recent_used_cpu, target)) {
> +			target = recent_used_cpu;
> +			goto out;
> +		}
>   
>   	} else {
>   		recent_used_cpu = -1;
> @@ -9030,7 +9035,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   		 */
>   		if (sd) {
>   			i = select_idle_capacity(p, sd, target);
> -			return ((unsigned)i < nr_cpumask_bits) ? i : target;
> +			target = ((unsigned)i < nr_cpumask_bits) ? i : target;
> +			goto out;
>   		}
>   	}
>   
> @@ -9043,27 +9049,31 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>   
>   		if (!has_idle_core && cpus_share_cache(prev, target)) {
>   			i = select_idle_smt(p, sd, prev);
> -			if ((unsigned int)i < nr_cpumask_bits)
> -				return i;
> +			if ((unsigned int)i < nr_cpumask_bits) {
> +				target = i;
> +				goto out;
> +			}
>   		}
>   	}
>   
>   	i = select_idle_cpu(p, sd, has_idle_core, target);
>   	if ((unsigned)i < nr_cpumask_bits)
> -		return i;
> -
> +		target = i;
>   	/*
>   	 * For cluster machines which have lower sharing cache like L2 or
>   	 * LLC Tag, we tend to find an idle CPU in the target's cluster
>   	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
>   	 * use them if possible when no idle CPU found in select_idle_cpu().
>   	 */
> -	if ((unsigned int)prev_aff < nr_cpumask_bits)
> -		return prev_aff;
> -	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> -		return recent_used_cpu;
> +	else if ((unsigned int)prev_aff < nr_cpumask_bits)
> +		target = prev_aff;
> +	else if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> +		target = recent_used_cpu;
> +out:
> +	if (!sched_smt_asym_active())
> +		return target;
>   
> -	return target;
> +	return select_idle_smt_priority(p, target);
>   }
>   
>   /**
> ---
> 
> That way, it lives in a single place, and we don't have to pepper
> select_idle_smt_priority() everywhere. Thoughts?
> 

[PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT
level to order hardware threads. Idle CPU selection does not consult
that order, so a task can wake on an arbitrary sibling and remain there
until load balancing corrects the placement. On these systems, that
initial choice can prevent the core from entering its preferred
lower-thread resource mode and cause a large and persistent performance
loss.

When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.

Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, and the target, previous, and recently-used
CPU fast paths. Inspect the lowest scheduling domain directly, but
require both CPUs to share its span because isolcpus can split hardware
siblings across scheduling domains.

Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.

Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 kernel/sched/fair.c     | 79 +++++++++++++++++++++++++++++++++--------
 kernel/sched/sched.h    |  6 ++++
 kernel/sched/topology.c | 36 +++++++++++++++++++
 3 files changed, 106 insertions(+), 15 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8bd308c2d5b1..4fa3b47a407e9 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8587,6 +8587,35 @@ static inline bool test_idle_cores(int cpu)
 	return false;
 }
 
+/*
+ * Redirect a CPU to a higher-priority available sibling in its SMT domain,
+ * subject to task affinity.
+ */
+static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
+{
+	struct sched_domain *sd;
+	int best = cpu;
+	int sibling;
+
+	if (!sched_smt_asym_active())
+		return cpu;
+
+	sd = rcu_dereference_all(cpu_rq(cpu)->sd);
+	if (!sd || !(sd->flags & SD_SHARE_CPUCAPACITY) ||
+	    !(sd->flags & SD_ASYM_PACKING))
+		return cpu;
+
+	for_each_cpu_and(sibling, sched_domain_span(sd), p->cpus_ptr) {
+		if (sibling == best || !choose_idle_cpu(sibling, p))
+			continue;
+
+		if (sched_asym_prefer(sibling, best))
+			best = sibling;
+	}
+
+	return best;
+}
+
 /*
  * Scans the local SMT mask to see if the entire core is idle, and records this
  * information in sd_balance_shared->has_idle_cores.
@@ -8971,7 +9000,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		goto select_smt_priority;
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8981,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(prev, target))
-			return prev;
+		    cpus_share_resources(prev, target)) {
+			target = prev;
+			goto select_smt_priority;
+		}
 
 		prev_aff = prev;
 	}
@@ -9000,7 +9031,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    prev == smp_processor_id() &&
 	    this_rq()->nr_running <= 1 &&
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
-		return prev;
+		target = prev;
+		goto select_smt_priority;
 	}
 
 	/* Check a recently used CPU as a potential idle candidate: */
@@ -9014,8 +9046,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+		    cpus_share_resources(recent_used_cpu, target)) {
+			target = recent_used_cpu;
+			goto select_smt_priority;
+		}
 
 	} else {
 		recent_used_cpu = -1;
@@ -9037,7 +9071,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		 */
 		if (sd) {
 			i = select_idle_capacity(p, sd, target);
-			return ((unsigned)i < nr_cpumask_bits) ? i : target;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
+			return target;
 		}
 	}
 
@@ -9050,14 +9088,18 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!has_idle_core && cpus_share_cache(prev, target)) {
 			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto select_smt_priority;
+			}
 		}
 	}
 
 	i = select_idle_cpu(p, sd, has_idle_core, target);
-	if ((unsigned)i < nr_cpumask_bits)
-		return i;
+	if ((unsigned int)i < nr_cpumask_bits) {
+		target = i;
+		goto select_smt_priority;
+	}
 
 	/*
 	 * For cluster machines which have lower sharing cache like L2 or
@@ -9065,12 +9107,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
 	 * use them if possible when no idle CPU found in select_idle_cpu().
 	 */
-	if ((unsigned int)prev_aff < nr_cpumask_bits)
-		return prev_aff;
-	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
-		return recent_used_cpu;
+	if ((unsigned int)prev_aff < nr_cpumask_bits) {
+		target = prev_aff;
+		goto select_smt_priority;
+	}
+	if ((unsigned int)recent_used_cpu < nr_cpumask_bits) {
+		target = recent_used_cpu;
+		goto select_smt_priority;
+	}
 
 	return target;
+
+select_smt_priority:
+	return select_idle_smt_priority(p, target);
 }
 
 /**
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c3ad70e58b8e..568cb1ed2dd6b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 extern struct static_key_false sched_asym_cpucapacity;
+extern struct static_key_false sched_smt_asym_packing;
 extern struct static_key_false sched_cluster_active;
 
 static __always_inline bool sched_asym_cpucap_active(void)
@@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
 	return static_branch_unlikely(&sched_asym_cpucapacity);
 }
 
+static __always_inline bool sched_smt_asym_active(void)
+{
+	return static_branch_unlikely(&sched_smt_asym_packing);
+}
+
 struct sched_group_capacity {
 	atomic_t		ref;
 	/*
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 0248227d983a7..06c40eb5932af 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
+DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+static bool has_asym_smt_domain(int cpu)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if (sd->flags & SD_ASYM_PACKING)
+			return true;
+	}
+
+	return false;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	struct rq *rq = NULL;
 	int i, ret = -ENOMEM;
 	bool has_asym = false;
+	bool has_asym_smt = false;
 	bool has_cluster = false;
 
 	if (WARN_ON(cpumask_empty(cpu_map)))
@@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 
 		cpu_attach_domain(sd, d.rd, i);
 
+		if (has_asym_smt_domain(i))
+			has_asym_smt = true;
+
 		if (lowest_flag_domain(i, SD_CLUSTER))
 			has_cluster = true;
 	}
@@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	if (has_asym)
 		static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_inc_cpuslocked(&sched_smt_asym_packing);
+
 	if (has_cluster)
 		static_branch_inc_cpuslocked(&sched_cluster_active);
 
@@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
 static void detach_destroy_domains(const struct cpumask *cpu_map)
 {
 	unsigned int cpu = cpumask_any(cpu_map);
+	bool has_asym_smt = false;
 	int i;
 
+	rcu_read_lock();
+	for_each_cpu(i, cpu_map) {
+		if (has_asym_smt_domain(i)) {
+			has_asym_smt = true;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
 	if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
 		static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_dec_cpuslocked(&sched_smt_asym_packing);
+
 	if (static_branch_unlikely(&sched_cluster_active))
 		static_branch_dec_cpuslocked(&sched_cluster_active);
 
-- 
2.55.0

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

On Mon, Sep 07, 2026 at 03:10:00PM +0530, K Prateek Nayak wrote:
...
> >> @@ -9043,27 +9049,31 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >>  
> >>  		if (!has_idle_core && cpus_share_cache(prev, target)) {
> >>  			i = select_idle_smt(p, sd, prev);
> >> -			if ((unsigned int)i < nr_cpumask_bits)
> >> -				return i;
> >> +			if ((unsigned int)i < nr_cpumask_bits) {
> >> +				target = i;
> >> +				goto out;
> >> +			}
> >>  		}
> >>  	}
> >>  
> >>  	i = select_idle_cpu(p, sd, has_idle_core, target);
> >>  	if ((unsigned)i < nr_cpumask_bits)
> >> -		return i;
> >> -
> >> +		target = i;
> > 
> > Not sure about this final fallback. Is it worth doing an additional
> > select_idle_smt_priority() after idle scan failed or stopped because the
> > SIS_UTIL scan budget was exhausted?
> 
> I see what you mean! We'll end up doing a:
> 
>   select_idle_smt_priority(p, target)
> 
> at the end which might indeed be wasteful.

Exactly.

> > 
> > It seems better to jump to out only when one of these paths has actually
> > selected a candidate:
> > 
> > 	i = select_idle_cpu(p, sd, has_idle_core, target);
> > 	if ((unsigned int)i < nr_cpumask_bits) {
> > 		target = i;
> > 		goto out;
> > 	}
> > 
> > The prev_aff and recent_used_cpu fallbacks can jump to "out" as well, since they
> > were already verified as suitable candidates. If none of those paths succeeds, I
> > think the existing final "return target" should remain unchanged.
> > 
> > Does that make sense?
> 
> Correct me if I'm wrong but you are suggesting to keep the current
> return intact and put out label after it like:
> 
>     /* If no suitable target was found */
>     return target;
> 
> out:
>    if (!sched_smt_asym_active())
>        return target;
> 
>    return select_idle_smt_priority(p, target);
> ---
> 
> That makes sense to me!

Correct, I'm going to run some tests with this and will send a v3 later.

Thanks!
-Andrea

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[K Prateek Nayak]

Hello Andrea,

On 9/7/2026 2:41 PM, Andrea Righi wrote:
>>> @@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>>>  						return -1;
>>>  					idle_cpu = __select_idle_cpu(cpu, p);
>>>  					if ((unsigned int)idle_cpu < nr_cpumask_bits)
>>> -						return idle_cpu;
>>> +						return select_idle_smt_priority(p, idle_cpu);
>>
>> Question for Shrikanth: On larger SMT (SMT-4, SMT-8), does the ranking
>> make that big of a difference if the core is already busy?
>>
>> Does the overehead of additional search get offset by the benefit of
>> being placed on a better ranked thread? If not, maybe the paths for
>> !has_idle_core can stay as is?
> 
> On Olympus it'd be fine either way, since it's an SMT2. For wider SMT systems I
> also defer the question to Shrikanth, I don't have any of them to test. :)

Same! Best I can do is a VM with -cpus ...,threads=8 but performance on
those are super flaky to make any meaningful deductions.

> 
>>
>>>  				}
>>>  			}
>>>  			cpumask_andnot(cpus, cpus, sched_group_span(sg));
>>> @@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>>>  	if (has_idle_core)
>>>  		set_idle_cores(target, false);
>>>  
>>> -	return idle_cpu;
>>> +	return (unsigned int)idle_cpu < nr_cpumask_bits ?
>>> +		select_idle_smt_priority(p, idle_cpu) : idle_cpu;
>>
>> Since every path does a select_idle_smt_priority() - be it coming from
>> select_idle_core(), the early-return from the cluster scan, or just an
>> idle CPU from the LLc scan, can't we simply just do it once in
>> select_idle_sibling()?
>>
>> Something like:
> 
> Yes, consolidating it in select_idle_sibling() looks cleaner. One comment below.
> 
>>
>>   (Only build tested)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index f79fcba4afec..7c97585141dd 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -8964,7 +8964,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  
>>  	if (choose_idle_cpu(target, p) &&
>>  	    asym_fits_cpu(task_util, util_min, util_max, target))
>> -		return target;
>> +		goto out;
>>  
>>  	/*
>>  	 * If the previous CPU is cache affine and idle, don't be stupid:
>> @@ -8974,8 +8974,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
>>  
>>  		if (!static_branch_unlikely(&sched_cluster_active) ||
>> -		    cpus_share_resources(prev, target))
>> -			return prev;
>> +		    cpus_share_resources(prev, target)) {
>> +			target = prev;
>> +			goto out;
>> +		}
>>  
>>  		prev_aff = prev;
>>  	}
>> @@ -8993,7 +8995,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  	    prev == smp_processor_id() &&
>>  	    this_rq()->nr_running <= 1 &&
>>  	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
>> -		return prev;
>> +		target = prev;
>> +		goto out;
>>  	}
>>  
>>  	/* Check a recently used CPU as a potential idle candidate: */
>> @@ -9007,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
>>  
>>  		if (!static_branch_unlikely(&sched_cluster_active) ||
>> -		    cpus_share_resources(recent_used_cpu, target))
>> -			return recent_used_cpu;
>> +		    cpus_share_resources(recent_used_cpu, target)) {
>> +			target = recent_used_cpu;
>> +			goto out;
>> +		}
>>  
>>  	} else {
>>  		recent_used_cpu = -1;
>> @@ -9030,7 +9035,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  		 */
>>  		if (sd) {
>>  			i = select_idle_capacity(p, sd, target);
>> -			return ((unsigned)i < nr_cpumask_bits) ? i : target;
>> +			target = ((unsigned)i < nr_cpumask_bits) ? i : target;
>> +			goto out;
>>  		}
>>  	}
>>  
>> @@ -9043,27 +9049,31 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  
>>  		if (!has_idle_core && cpus_share_cache(prev, target)) {
>>  			i = select_idle_smt(p, sd, prev);
>> -			if ((unsigned int)i < nr_cpumask_bits)
>> -				return i;
>> +			if ((unsigned int)i < nr_cpumask_bits) {
>> +				target = i;
>> +				goto out;
>> +			}
>>  		}
>>  	}
>>  
>>  	i = select_idle_cpu(p, sd, has_idle_core, target);
>>  	if ((unsigned)i < nr_cpumask_bits)
>> -		return i;
>> -
>> +		target = i;
> 
> Not sure about this final fallback. Is it worth doing an additional
> select_idle_smt_priority() after idle scan failed or stopped because the
> SIS_UTIL scan budget was exhausted?

I see what you mean! We'll end up doing a:

  select_idle_smt_priority(p, target)

at the end which might indeed be wasteful.
> 
> It seems better to jump to out only when one of these paths has actually
> selected a candidate:
> 
> 	i = select_idle_cpu(p, sd, has_idle_core, target);
> 	if ((unsigned int)i < nr_cpumask_bits) {
> 		target = i;
> 		goto out;
> 	}
> 
> The prev_aff and recent_used_cpu fallbacks can jump to "out" as well, since they
> were already verified as suitable candidates. If none of those paths succeeds, I
> think the existing final "return target" should remain unchanged.
> 
> Does that make sense?

Correct me if I'm wrong but you are suggesting to keep the current
return intact and put out label after it like:

    /* If no suitable target was found */
    return target;

out:
   if (!sched_smt_asym_active())
       return target;

   return select_idle_smt_priority(p, target);
---

That makes sense to me!

-- 
Thanks and Regards,
Prateek

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Prateek,

On Mon, Sep 07, 2026 at 09:27:22AM +0530, K Prateek Nayak wrote:
> Hello Andrea,
> 
> On 9/4/2026 2:48 PM, Andrea Righi wrote:
> > +/*
> > + * Return true when @cpu has a higher asymmetric-packing priority than
> > + * @other in their shared SMT scheduling domain.
> > + */
> > +static bool sched_smt_asym_prefer(int cpu, int other)
> > +{
> > +	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> > +
> > +	if (!sd)
> > +		return false;
> > +
> > +	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
> > +	    !(sd->flags & SD_ASYM_PACKING))
> > +		return false;
> > +
> > +	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
> > +		return false;
> > +
> > +	return sched_asym_prefer(cpu, other);
> > +}
> > +
> > +/*
> > + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
> > + */
> > +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> > +{
> > +	int best = cpu;
> > +	int sibling;
> > +
> > +	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
> > +		if (sibling == best || !choose_idle_cpu(sibling, p))
> > +			continue;
> > +
> > +		if (sched_smt_asym_prefer(sibling, best))
> > +			best = sibling;
> 
> nit. Since sched_smt_asym_prefer() is only used here, and we know rq->sd
> is the one that can have SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING, perhaps
> you can inline the check here do a:
> 
>     sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> 
>     if (!sd)
>          return cpu;
> 
>    if (!(sd->flags & SD_SHARE_CPUCAPACITY) || !(sd->flags & SD_ASYM_PACKING))
>        return cpu;
> 
>    for_each_cpu_and (sibling, sched_domain_span(sd), cpus) {
>        ...
>    }
> 
>   ...
> 
> 
> That way, you don't need to dereference cpu_rq(cpu)->sd every time in
> sched_smt_asym_prefer() and check cpumask_test_cpu(). Both, domain
> span and task affinity will be covered at once.
> 
> Thoughts?

Yes, agreed. I like this way more.

> 
> > +	}
> > +
> > +	return best;
> > +}
> > +
> > +static inline int
> > +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> > +{
> > +	if (!sched_smt_asym_active())
> > +		return cpu;
> > +
> > +	return __select_idle_smt_cpu(p, cpu, cpus);
> > +}
> > +
> > +/*
> > + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
> > + */
> > +static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
> > +{
> > +	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
> > +}
> > +
> >  /*
> >   * Scans the local SMT mask to see if the entire core is idle, and records this
> >   * information in sd_balance_shared->has_idle_cores.
> > @@ -8645,7 +8702,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
> >  	}
> >  
> >  	if (idle)
> > -		return core;
> > +		return select_idle_smt_cpu(p, core, cpus);
> >  
> >  	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
> >  	return -1;
> > @@ -8668,7 +8725,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
> >  		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
> >  			continue;
> >  		if (choose_idle_cpu(cpu, p))
> > -			return cpu;
> > +			return select_idle_smt_priority(p, cpu);
> >  	}
> >  
> >  	return -1;
> > @@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
> >  						return -1;
> >  					idle_cpu = __select_idle_cpu(cpu, p);
> >  					if ((unsigned int)idle_cpu < nr_cpumask_bits)
> > -						return idle_cpu;
> > +						return select_idle_smt_priority(p, idle_cpu);
> 
> Question for Shrikanth: On larger SMT (SMT-4, SMT-8), does the ranking
> make that big of a difference if the core is already busy?
> 
> Does the overehead of additional search get offset by the benefit of
> being placed on a better ranked thread? If not, maybe the paths for
> !has_idle_core can stay as is?

On Olympus it'd be fine either way, since it's an SMT2. For wider SMT systems I
also defer the question to Shrikanth, I don't have any of them to test. :)

> 
> >  				}
> >  			}
> >  			cpumask_andnot(cpus, cpus, sched_group_span(sg));
> > @@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
> >  	if (has_idle_core)
> >  		set_idle_cores(target, false);
> >  
> > -	return idle_cpu;
> > +	return (unsigned int)idle_cpu < nr_cpumask_bits ?
> > +		select_idle_smt_priority(p, idle_cpu) : idle_cpu;
> 
> Since every path does a select_idle_smt_priority() - be it coming from
> select_idle_core(), the early-return from the cluster scan, or just an
> idle CPU from the LLc scan, can't we simply just do it once in
> select_idle_sibling()?
> 
> Something like:

Yes, consolidating it in select_idle_sibling() looks cleaner. One comment below.

> 
>   (Only build tested)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index f79fcba4afec..7c97585141dd 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8964,7 +8964,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  
>  	if (choose_idle_cpu(target, p) &&
>  	    asym_fits_cpu(task_util, util_min, util_max, target))
> -		return target;
> +		goto out;
>  
>  	/*
>  	 * If the previous CPU is cache affine and idle, don't be stupid:
> @@ -8974,8 +8974,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
>  
>  		if (!static_branch_unlikely(&sched_cluster_active) ||
> -		    cpus_share_resources(prev, target))
> -			return prev;
> +		    cpus_share_resources(prev, target)) {
> +			target = prev;
> +			goto out;
> +		}
>  
>  		prev_aff = prev;
>  	}
> @@ -8993,7 +8995,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  	    prev == smp_processor_id() &&
>  	    this_rq()->nr_running <= 1 &&
>  	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
> -		return prev;
> +		target = prev;
> +		goto out;
>  	}
>  
>  	/* Check a recently used CPU as a potential idle candidate: */
> @@ -9007,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
>  
>  		if (!static_branch_unlikely(&sched_cluster_active) ||
> -		    cpus_share_resources(recent_used_cpu, target))
> -			return recent_used_cpu;
> +		    cpus_share_resources(recent_used_cpu, target)) {
> +			target = recent_used_cpu;
> +			goto out;
> +		}
>  
>  	} else {
>  		recent_used_cpu = -1;
> @@ -9030,7 +9035,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  		 */
>  		if (sd) {
>  			i = select_idle_capacity(p, sd, target);
> -			return ((unsigned)i < nr_cpumask_bits) ? i : target;
> +			target = ((unsigned)i < nr_cpumask_bits) ? i : target;
> +			goto out;
>  		}
>  	}
>  
> @@ -9043,27 +9049,31 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  
>  		if (!has_idle_core && cpus_share_cache(prev, target)) {
>  			i = select_idle_smt(p, sd, prev);
> -			if ((unsigned int)i < nr_cpumask_bits)
> -				return i;
> +			if ((unsigned int)i < nr_cpumask_bits) {
> +				target = i;
> +				goto out;
> +			}
>  		}
>  	}
>  
>  	i = select_idle_cpu(p, sd, has_idle_core, target);
>  	if ((unsigned)i < nr_cpumask_bits)
> -		return i;
> -
> +		target = i;

Not sure about this final fallback. Is it worth doing an additional
select_idle_smt_priority() after idle scan failed or stopped because the
SIS_UTIL scan budget was exhausted?

It seems better to jump to out only when one of these paths has actually
selected a candidate:

	i = select_idle_cpu(p, sd, has_idle_core, target);
	if ((unsigned int)i < nr_cpumask_bits) {
		target = i;
		goto out;
	}

The prev_aff and recent_used_cpu fallbacks can jump to "out" as well, since they
were already verified as suitable candidates. If none of those paths succeeds, I
think the existing final "return target" should remain unchanged.

Does that make sense?

Thanks for looking at this!
-Andrea

>  	/*
>  	 * For cluster machines which have lower sharing cache like L2 or
>  	 * LLC Tag, we tend to find an idle CPU in the target's cluster
>  	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
>  	 * use them if possible when no idle CPU found in select_idle_cpu().
>  	 */
> -	if ((unsigned int)prev_aff < nr_cpumask_bits)
> -		return prev_aff;
> -	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> -		return recent_used_cpu;
> +	else if ((unsigned int)prev_aff < nr_cpumask_bits)
> +		target = prev_aff;
> +	else if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
> +		target = recent_used_cpu;
> +out:
> +	if (!sched_smt_asym_active())
> +		return target;
>  
> -	return target;
> +	return select_idle_smt_priority(p, target);
>  }
>  
>  /**
> ---
> 
> That way, it lives in a single place, and we don't have to pepper
> select_idle_smt_priority() everywhere. Thoughts?
> 
> -- 
> Thanks and Regards,
> Prateek
> 

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[K Prateek Nayak]

Hello Andrea,

On 9/4/2026 2:48 PM, Andrea Righi wrote:
> +/*
> + * Return true when @cpu has a higher asymmetric-packing priority than
> + * @other in their shared SMT scheduling domain.
> + */
> +static bool sched_smt_asym_prefer(int cpu, int other)
> +{
> +	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> +
> +	if (!sd)
> +		return false;
> +
> +	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
> +	    !(sd->flags & SD_ASYM_PACKING))
> +		return false;
> +
> +	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
> +		return false;
> +
> +	return sched_asym_prefer(cpu, other);
> +}
> +
> +/*
> + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
> + */
> +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> +{
> +	int best = cpu;
> +	int sibling;
> +
> +	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
> +		if (sibling == best || !choose_idle_cpu(sibling, p))
> +			continue;
> +
> +		if (sched_smt_asym_prefer(sibling, best))
> +			best = sibling;

nit. Since sched_smt_asym_prefer() is only used here, and we know rq->sd
is the one that can have SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING, perhaps
you can inline the check here do a:

    sd = rcu_dereference_all(cpu_rq(cpu)->sd);

    if (!sd)
         return cpu;

   if (!(sd->flags & SD_SHARE_CPUCAPACITY) || !(sd->flags & SD_ASYM_PACKING))
       return cpu;

   for_each_cpu_and (sibling, sched_domain_span(sd), cpus) {
       ...
   }

  ...


That way, you don't need to dereference cpu_rq(cpu)->sd every time in
sched_smt_asym_prefer() and check cpumask_test_cpu(). Both, domain
span and task affinity will be covered at once.

Thoughts?

> +	}
> +
> +	return best;
> +}
> +
> +static inline int
> +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
> +{
> +	if (!sched_smt_asym_active())
> +		return cpu;
> +
> +	return __select_idle_smt_cpu(p, cpu, cpus);
> +}
> +
> +/*
> + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
> + */
> +static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
> +{
> +	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
> +}
> +
>  /*
>   * Scans the local SMT mask to see if the entire core is idle, and records this
>   * information in sd_balance_shared->has_idle_cores.
> @@ -8645,7 +8702,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
>  	}
>  
>  	if (idle)
> -		return core;
> +		return select_idle_smt_cpu(p, core, cpus);
>  
>  	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
>  	return -1;
> @@ -8668,7 +8725,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>  		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
>  			continue;
>  		if (choose_idle_cpu(cpu, p))
> -			return cpu;
> +			return select_idle_smt_priority(p, cpu);
>  	}
>  
>  	return -1;
> @@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>  						return -1;
>  					idle_cpu = __select_idle_cpu(cpu, p);
>  					if ((unsigned int)idle_cpu < nr_cpumask_bits)
> -						return idle_cpu;
> +						return select_idle_smt_priority(p, idle_cpu);

Question for Shrikanth: On larger SMT (SMT-4, SMT-8), does the ranking
make that big of a difference if the core is already busy?

Does the overehead of additional search get offset by the benefit of
being placed on a better ranked thread? If not, maybe the paths for
!has_idle_core can stay as is?

>  				}
>  			}
>  			cpumask_andnot(cpus, cpus, sched_group_span(sg));
> @@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>  	if (has_idle_core)
>  		set_idle_cores(target, false);
>  
> -	return idle_cpu;
> +	return (unsigned int)idle_cpu < nr_cpumask_bits ?
> +		select_idle_smt_priority(p, idle_cpu) : idle_cpu;

Since every path does a select_idle_smt_priority() - be it coming from
select_idle_core(), the early-return from the cluster scan, or just an
idle CPU from the LLc scan, can't we simply just do it once in
select_idle_sibling()?

Something like:

  (Only build tested)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f79fcba4afec..7c97585141dd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8964,7 +8964,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		goto out;
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8974,8 +8974,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(prev, target))
-			return prev;
+		    cpus_share_resources(prev, target)) {
+			target = prev;
+			goto out;
+		}
 
 		prev_aff = prev;
 	}
@@ -8993,7 +8995,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    prev == smp_processor_id() &&
 	    this_rq()->nr_running <= 1 &&
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {
-		return prev;
+		target = prev;
+		goto out;
 	}
 
 	/* Check a recently used CPU as a potential idle candidate: */
@@ -9007,8 +9010,10 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
-		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+		    cpus_share_resources(recent_used_cpu, target)) {
+			target = recent_used_cpu;
+			goto out;
+		}
 
 	} else {
 		recent_used_cpu = -1;
@@ -9030,7 +9035,8 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		 */
 		if (sd) {
 			i = select_idle_capacity(p, sd, target);
-			return ((unsigned)i < nr_cpumask_bits) ? i : target;
+			target = ((unsigned)i < nr_cpumask_bits) ? i : target;
+			goto out;
 		}
 	}
 
@@ -9043,27 +9049,31 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!has_idle_core && cpus_share_cache(prev, target)) {
 			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
+			if ((unsigned int)i < nr_cpumask_bits) {
+				target = i;
+				goto out;
+			}
 		}
 	}
 
 	i = select_idle_cpu(p, sd, has_idle_core, target);
 	if ((unsigned)i < nr_cpumask_bits)
-		return i;
-
+		target = i;
 	/*
 	 * For cluster machines which have lower sharing cache like L2 or
 	 * LLC Tag, we tend to find an idle CPU in the target's cluster
 	 * first. But prev_cpu or recent_used_cpu may also be a good candidate,
 	 * use them if possible when no idle CPU found in select_idle_cpu().
 	 */
-	if ((unsigned int)prev_aff < nr_cpumask_bits)
-		return prev_aff;
-	if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
-		return recent_used_cpu;
+	else if ((unsigned int)prev_aff < nr_cpumask_bits)
+		target = prev_aff;
+	else if ((unsigned int)recent_used_cpu < nr_cpumask_bits)
+		target = recent_used_cpu;
+out:
+	if (!sched_smt_asym_active())
+		return target;
 
-	return target;
+	return select_idle_smt_priority(p, target);
 }
 
 /**
---

That way, it lives in a single place, and we don't have to pepper
select_idle_smt_priority() everywhere. Thoughts?

-- 
Thanks and Regards,
Prateek

[PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity
SMT level to order hardware threads. Idle CPU selection does not consult
that order, so a task can wake on an arbitrary sibling and remain there
until load balancing corrects the placement. On these systems, that
initial choice can prevent the core from entering its preferred
lower-thread resource mode and cause a large and persistent performance
loss.

When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.

Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, and the target, previous, and recently-used
CPU fast paths. Inspect the lowest scheduling domain directly, but
require both CPUs to share its span because isolcpus can split hardware
siblings across scheduling domains.

Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.

Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 kernel/sched/fair.c     | 79 ++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h    |  6 ++++
 kernel/sched/topology.c | 36 +++++++++++++++++++
 3 files changed, 112 insertions(+), 9 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b8bd308c2d5b1..ff9a7b1fcbe7f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8587,6 +8587,63 @@ static inline bool test_idle_cores(int cpu)
 	return false;
 }
 
+/*
+ * Return true when @cpu has a higher asymmetric-packing priority than
+ * @other in their shared SMT scheduling domain.
+ */
+static bool sched_smt_asym_prefer(int cpu, int other)
+{
+	struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
+
+	if (!sd)
+		return false;
+
+	if (!(sd->flags & SD_SHARE_CPUCAPACITY) ||
+	    !(sd->flags & SD_ASYM_PACKING))
+		return false;
+
+	if (!cpumask_test_cpu(other, sched_domain_span(sd)))
+		return false;
+
+	return sched_asym_prefer(cpu, other);
+}
+
+/*
+ * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
+ */
+static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
+{
+	int best = cpu;
+	int sibling;
+
+	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
+		if (sibling == best || !choose_idle_cpu(sibling, p))
+			continue;
+
+		if (sched_smt_asym_prefer(sibling, best))
+			best = sibling;
+	}
+
+	return best;
+}
+
+static inline int
+select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
+{
+	if (!sched_smt_asym_active())
+		return cpu;
+
+	return __select_idle_smt_cpu(p, cpu, cpus);
+}
+
+/*
+ * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
+ */
+static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
+{
+	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
+}
+
 /*
  * Scans the local SMT mask to see if the entire core is idle, and records this
  * information in sd_balance_shared->has_idle_cores.
@@ -8645,7 +8702,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 	}
 
 	if (idle)
-		return core;
+		return select_idle_smt_cpu(p, core, cpus);
 
 	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
 	return -1;
@@ -8668,7 +8725,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
 		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
 			continue;
 		if (choose_idle_cpu(cpu, p))
-			return cpu;
+			return select_idle_smt_priority(p, cpu);
 	}
 
 	return -1;
@@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 						return -1;
 					idle_cpu = __select_idle_cpu(cpu, p);
 					if ((unsigned int)idle_cpu < nr_cpumask_bits)
-						return idle_cpu;
+						return select_idle_smt_priority(p, idle_cpu);
 				}
 			}
 			cpumask_andnot(cpus, cpus, sched_group_span(sg));
@@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 	if (has_idle_core)
 		set_idle_cores(target, false);
 
-	return idle_cpu;
+	return (unsigned int)idle_cpu < nr_cpumask_bits ?
+		select_idle_smt_priority(p, idle_cpu) : idle_cpu;
 }
 
 /*
@@ -8858,7 +8916,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		 * immediately.
 		 */
 		if (fits > 0 && preferred_core)
-			return cpu;
+			return select_idle_smt_cpu(p, cpu, cpus);
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
 		 * Look for the CPU with best capacity.
@@ -8915,6 +8973,9 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT)
 		set_idle_cores(target, false);
 
+	if (best_cpu >= 0)
+		best_cpu = select_idle_smt_cpu(p, best_cpu, cpus);
+
 	return best_cpu;
 }
 
@@ -8971,7 +9032,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		return select_idle_smt_priority(p, target);
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8982,9 +9043,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
 		    cpus_share_resources(prev, target))
-			return prev;
+			return select_idle_smt_priority(p, prev);
 
-		prev_aff = prev;
+		prev_aff = select_idle_smt_priority(p, prev);
 	}
 
 	/*
@@ -9015,7 +9076,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
 		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+			return select_idle_smt_priority(p, recent_used_cpu);
 
 	} else {
 		recent_used_cpu = -1;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c3ad70e58b8e..568cb1ed2dd6b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 extern struct static_key_false sched_asym_cpucapacity;
+extern struct static_key_false sched_smt_asym_packing;
 extern struct static_key_false sched_cluster_active;
 
 static __always_inline bool sched_asym_cpucap_active(void)
@@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
 	return static_branch_unlikely(&sched_asym_cpucapacity);
 }
 
+static __always_inline bool sched_smt_asym_active(void)
+{
+	return static_branch_unlikely(&sched_smt_asym_packing);
+}
+
 struct sched_group_capacity {
 	atomic_t		ref;
 	/*
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 0248227d983a7..06c40eb5932af 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
+DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+static bool has_asym_smt_domain(int cpu)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if (sd->flags & SD_ASYM_PACKING)
+			return true;
+	}
+
+	return false;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	struct rq *rq = NULL;
 	int i, ret = -ENOMEM;
 	bool has_asym = false;
+	bool has_asym_smt = false;
 	bool has_cluster = false;
 
 	if (WARN_ON(cpumask_empty(cpu_map)))
@@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 
 		cpu_attach_domain(sd, d.rd, i);
 
+		if (has_asym_smt_domain(i))
+			has_asym_smt = true;
+
 		if (lowest_flag_domain(i, SD_CLUSTER))
 			has_cluster = true;
 	}
@@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	if (has_asym)
 		static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_inc_cpuslocked(&sched_smt_asym_packing);
+
 	if (has_cluster)
 		static_branch_inc_cpuslocked(&sched_cluster_active);
 
@@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
 static void detach_destroy_domains(const struct cpumask *cpu_map)
 {
 	unsigned int cpu = cpumask_any(cpu_map);
+	bool has_asym_smt = false;
 	int i;
 
+	rcu_read_lock();
+	for_each_cpu(i, cpu_map) {
+		if (has_asym_smt_domain(i)) {
+			has_asym_smt = true;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
 	if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
 		static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_dec_cpuslocked(&sched_smt_asym_packing);
+
 	if (static_branch_unlikely(&sched_cluster_active))
 		static_branch_dec_cpuslocked(&sched_cluster_active);
 
-- 
2.55.0

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

Hi Dietmar,

On Thu, Sep 03, 2026 at 12:59:59PM +0200, Dietmar Eggemann wrote:
> On 31.08.26 20:10, Andrea Righi wrote:
> > SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU
> > selection does not consult that order. A task can therefore wake on an
> > arbitrary sibling and remain there until load balancing corrects the
> > placement. On SMT implementations where changing the active sibling
> > repartitions core resources, that initial choice can cause a large and
> > persistent performance loss.
> 
> I assume this sentence refers to Olympus/Vera and Power7?

Yes. Olympus/Vera is the platform motivating this series, but this is affecting
POWER7 as well, since it's also using SD_ASYM_PACKING at the SMT level. I'll
change the description to make that scope explicit.

> 
> [...]
> 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 8dff37059faf7..3c49aa63742cb 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu)
> >  	return false;
> >  }
> >  
> > +/*
> > + * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT
> > + * scheduling domain.
> > + */
> > +static bool sched_smt_asym_prefer(int cpu, int other)
> > +{
> > +	struct sched_domain *sd;
> > +
> > +	for_each_domain(cpu, sd) {
> > +		/*
> > +		 * Only honor priorities declared at shared-capacity SMT levels.
> > +		 * SD_ASYM_PACKING at higher levels may describe core ordering.
> > +		 */
> > +		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> > +			break;
> > +
> > +		if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd)))
> 
> Looks like 'other' is always part of the mask?

It's always in the same hardware SMT sibling mask, but it's not necessarily in
the same scheduling-domain span, isolcpus could split siblings across scheduling
domain, select_idle_smt() has the same sched_domain_span() check for the same
resason IIUC.

> 
> > +			return sched_asym_prefer(cpu, other);
> > +	}
> > +
> > +	return false;
> > +}
> 
> SMT will always the lowest SD, so for_each_domain() is not necessary:

Agreed, walking the domain hierarchy is unnecessary. I'll use the lowest domain
directly.

> 
> static bool sched_smt_asym_prefer(int cpu, int other)
> {
>         struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);
> 
>         if (sd && ((sd->flags & (SD_SHARE_CPUCAPACITY |
> SD_ASYM_PACKING)) == (SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING)))
>                 return sched_asym_prefer(cpu, other);
> 
>         return false;
> }
> 
> [...]
> 
> > @@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
> >  		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
> >  			continue;
> >  		if (choose_idle_cpu(cpu, p))
> > -			return cpu;
> > +			return select_idle_smt_priority(p, cpu);
> 
> This one is weird for SMT2. AFAICS, select_idle_smt() is called when
> there are no idle cores. So if you find an idle CPU this is what you
> will return anyway.

Correcdt for SMT2: once the core is partially idle, there's only one idle
sibling and the priority lookup returns that same CPU.

The hook is useful for wider asym SMT though. POWER7 uses SMT4 with
SD_ASYM_PACKING at the SMT level, so a partially idle core can still have
multiple idle threads. In that case select_idle_smt() would otherwise return the
first idle thread rather than the highest-priority one.

> 
> I guess your tests on Olympus/Vera do wakeups via select_idle_capacity()
> so you haven't touched this one.

Correct, Vera also has SD_ASYM_CPUCAPACITY, so the scan path used by these tests
is select_idle_capacity().

Thanks for taking a look at this!
-Andrea

Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Dietmar Eggemann]

On 31.08.26 20:10, Andrea Righi wrote:
> SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU
> selection does not consult that order. A task can therefore wake on an
> arbitrary sibling and remain there until load balancing corrects the
> placement. On SMT implementations where changing the active sibling
> repartitions core resources, that initial choice can cause a large and
> persistent performance loss.

I assume this sentence refers to Olympus/Vera and Power7?

[...]

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 8dff37059faf7..3c49aa63742cb 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu)
>  	return false;
>  }
>  
> +/*
> + * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT
> + * scheduling domain.
> + */
> +static bool sched_smt_asym_prefer(int cpu, int other)
> +{
> +	struct sched_domain *sd;
> +
> +	for_each_domain(cpu, sd) {
> +		/*
> +		 * Only honor priorities declared at shared-capacity SMT levels.
> +		 * SD_ASYM_PACKING at higher levels may describe core ordering.
> +		 */
> +		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
> +			break;
> +
> +		if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd)))

Looks like 'other' is always part of the mask?

> +			return sched_asym_prefer(cpu, other);
> +	}
> +
> +	return false;
> +}

SMT will always the lowest SD, so for_each_domain() is not necessary:

static bool sched_smt_asym_prefer(int cpu, int other)
{
        struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd);

        if (sd && ((sd->flags & (SD_SHARE_CPUCAPACITY |
SD_ASYM_PACKING)) == (SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING)))
                return sched_asym_prefer(cpu, other);

        return false;
}

[...]

> @@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>  		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
>  			continue;
>  		if (choose_idle_cpu(cpu, p))
> -			return cpu;
> +			return select_idle_smt_priority(p, cpu);

This one is weird for SMT2. AFAICS, select_idle_smt() is called when
there are no idle cores. So if you find an idle CPU this is what you
will return anyway.

I guess your tests on Olympus/Vera do wakeups via select_idle_capacity()
so you haven't touched this one.

[...]

[PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection

[Andrea Righi]

SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU
selection does not consult that order. A task can therefore wake on an
arbitrary sibling and remain there until load balancing corrects the
placement. On SMT implementations where changing the active sibling
repartitions core resources, that initial choice can cause a large and
persistent performance loss.

When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 this only changes selection
on fully idle cores, because a partially idle core has only one
available CPU. On wider SMT cores it also fills available siblings in
priority order while the core is partially busy.

Apply the preference to idle-core and idle-CPU scans, asymmetric
capacity scans, and the target, previous, and recently-used CPU fast
paths.

Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY can first select among cores with
different maximum capacities, SD_ASYM_PACKING then selects the preferred
available sibling inside the chosen core, whose siblings continue to
share equal capacity.

Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
 kernel/sched/fair.c     | 81 ++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h    |  6 +++
 kernel/sched/topology.c | 36 ++++++++++++++++++
 3 files changed, 114 insertions(+), 9 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8dff37059faf7..3c49aa63742cb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu)
 	return false;
 }
 
+/*
+ * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT
+ * scheduling domain.
+ */
+static bool sched_smt_asym_prefer(int cpu, int other)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		/*
+		 * Only honor priorities declared at shared-capacity SMT levels.
+		 * SD_ASYM_PACKING at higher levels may describe core ordering.
+		 */
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd)))
+			return sched_asym_prefer(cpu, other);
+	}
+
+	return false;
+}
+
+/*
+ * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus.
+ */
+static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
+{
+	int best = cpu;
+	int sibling;
+
+	for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) {
+		if (sibling == best || !choose_idle_cpu(sibling, p))
+			continue;
+
+		if (sched_smt_asym_prefer(sibling, best))
+			best = sibling;
+	}
+
+	return best;
+}
+
+static inline int
+select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus)
+{
+	if (!sched_smt_asym_active())
+		return cpu;
+
+	return __select_idle_smt_cpu(p, cpu, cpus);
+}
+
+/*
+ * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity.
+ */
+static inline int select_idle_smt_priority(struct task_struct *p, int cpu)
+{
+	return select_idle_smt_cpu(p, cpu, p->cpus_ptr);
+}
+
 /*
  * Scans the local SMT mask to see if the entire core is idle, and records this
  * information in sd_balance_shared->has_idle_cores.
@@ -8645,7 +8704,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 	}
 
 	if (idle)
-		return core;
+		return select_idle_smt_cpu(p, core, cpus);
 
 	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
 	return -1;
@@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
 		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
 			continue;
 		if (choose_idle_cpu(cpu, p))
-			return cpu;
+			return select_idle_smt_priority(p, cpu);
 	}
 
 	return -1;
@@ -8720,7 +8779,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 						return -1;
 					idle_cpu = __select_idle_cpu(cpu, p);
 					if ((unsigned int)idle_cpu < nr_cpumask_bits)
-						return idle_cpu;
+						return select_idle_smt_priority(p, idle_cpu);
 				}
 			}
 			cpumask_andnot(cpus, cpus, sched_group_span(sg));
@@ -8745,7 +8804,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 	if (has_idle_core)
 		set_idle_cores(target, false);
 
-	return idle_cpu;
+	return (unsigned int)idle_cpu < nr_cpumask_bits ?
+		select_idle_smt_priority(p, idle_cpu) : idle_cpu;
 }
 
 /*
@@ -8858,7 +8918,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		 * immediately.
 		 */
 		if (fits > 0 && preferred_core)
-			return cpu;
+			return select_idle_smt_cpu(p, cpu, cpus);
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
 		 * Look for the CPU with best capacity.
@@ -8915,6 +8975,9 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT)
 		set_idle_cores(target, false);
 
+	if (best_cpu >= 0)
+		best_cpu = select_idle_smt_cpu(p, best_cpu, cpus);
+
 	return best_cpu;
 }
 
@@ -8971,7 +9034,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	if (choose_idle_cpu(target, p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
-		return target;
+		return select_idle_smt_priority(p, target);
 
 	/*
 	 * If the previous CPU is cache affine and idle, don't be stupid:
@@ -8982,9 +9045,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
 		    cpus_share_resources(prev, target))
-			return prev;
+			return select_idle_smt_priority(p, prev);
 
-		prev_aff = prev;
+		prev_aff = select_idle_smt_priority(p, prev);
 	}
 
 	/*
@@ -9015,7 +9078,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 		if (!static_branch_unlikely(&sched_cluster_active) ||
 		    cpus_share_resources(recent_used_cpu, target))
-			return recent_used_cpu;
+			return select_idle_smt_priority(p, recent_used_cpu);
 
 	} else {
 		recent_used_cpu = -1;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e656c7059bf86..73731e9439b97 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 extern struct static_key_false sched_asym_cpucapacity;
+extern struct static_key_false sched_smt_asym_packing;
 extern struct static_key_false sched_cluster_active;
 
 static __always_inline bool sched_asym_cpucap_active(void)
@@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void)
 	return static_branch_unlikely(&sched_asym_cpucapacity);
 }
 
+static __always_inline bool sched_smt_asym_active(void)
+{
+	return static_branch_unlikely(&sched_smt_asym_packing);
+}
+
 struct sched_group_capacity {
 	atomic_t		ref;
 	/*
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 0248227d983a7..06c40eb5932af 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
 
 DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
+DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing);
 DEFINE_STATIC_KEY_FALSE(sched_cluster_active);
 
+static bool has_asym_smt_domain(int cpu)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (!(sd->flags & SD_SHARE_CPUCAPACITY))
+			break;
+
+		if (sd->flags & SD_ASYM_PACKING)
+			return true;
+	}
+
+	return false;
+}
+
 static void update_top_cache_domain(int cpu)
 {
 	struct sched_domain_shared *sds = NULL;
@@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	struct rq *rq = NULL;
 	int i, ret = -ENOMEM;
 	bool has_asym = false;
+	bool has_asym_smt = false;
 	bool has_cluster = false;
 
 	if (WARN_ON(cpumask_empty(cpu_map)))
@@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 
 		cpu_attach_domain(sd, d.rd, i);
 
+		if (has_asym_smt_domain(i))
+			has_asym_smt = true;
+
 		if (lowest_flag_domain(i, SD_CLUSTER))
 			has_cluster = true;
 	}
@@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	if (has_asym)
 		static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_inc_cpuslocked(&sched_smt_asym_packing);
+
 	if (has_cluster)
 		static_branch_inc_cpuslocked(&sched_cluster_active);
 
@@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map)
 static void detach_destroy_domains(const struct cpumask *cpu_map)
 {
 	unsigned int cpu = cpumask_any(cpu_map);
+	bool has_asym_smt = false;
 	int i;
 
+	rcu_read_lock();
+	for_each_cpu(i, cpu_map) {
+		if (has_asym_smt_domain(i)) {
+			has_asym_smt = true;
+			break;
+		}
+	}
+	rcu_read_unlock();
+
 	if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
 		static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
 
+	if (has_asym_smt)
+		static_branch_dec_cpuslocked(&sched_smt_asym_packing);
+
 	if (static_branch_unlikely(&sched_cluster_active))
 		static_branch_dec_cpuslocked(&sched_cluster_active);
 
-- 
2.55.0