[RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Kenan.Liu]

From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>

Multithreading workloads in VM with Qemu may encounter an unexpected
phenomenon: one hyperthread of a physical core is busy while its sibling
is idle. Such as:

%Cpu0  : 19.8 us,  3.8 sy,  0.0 ni, 76.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  6.4 us,  1.0 sy,  0.0 ni, 92.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  : 15.8 us,  4.5 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  4.7 us,  0.7 sy,  0.0 ni, 94.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  : 15.5 us,  4.5 sy,  0.0 ni, 80.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  4.7 us,  0.7 sy,  0.0 ni, 94.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  : 13.4 us,  3.4 sy,  0.0 ni, 83.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  2.7 us,  0.3 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  : 16.1 us,  4.8 sy,  0.0 ni, 79.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  :  2.0 us,  0.3 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 : 17.5 us,  5.2 sy,  0.0 ni, 77.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu12 : 17.6 us,  4.5 sy,  0.0 ni, 77.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu13 :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu14 : 16.1 us,  4.1 sy,  0.0 ni, 79.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

The main reason is that hyperthread index is consecutive in qemu native x86 CPU
model which is different from the physical topology. As the current kernel scheduler
implementation, hyperthread with an even ID number will be picked up in a much
higher probability during load-balancing and load-deploying.

This RFC targets to solve the problem by adjusting CFS loabalance policy:
1. Explore CPU topology and adjust CFS loadbalance policy when we found machine
with qemu native CPU topology.
2. Export a procfs to control the traverse length when select idle cpu.

Kenan.Liu (2):
  sched/fair: Adjust CFS loadbalance for machine with qemu native CPU
    topology.
  sched/fair: Export a param to control the traverse len when select
    idle cpu.

 kernel/sched/fair.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 99 insertions(+), 4 deletions(-)

-- 
1.8.3.1

[RFC PATCH 1/2] sched/fair: Adjust CFS loadbalance for machine with qemu native CPU topology.

[Kenan.Liu]

From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>

Multithreading workloads in VM with Qemu may encounter an unexpected
phenomenon: one hyperthread of a physical core is busy while its sibling
is idle. The main reason is that hyperthread index is consecutive in qemu
native x86 CPU model which is different from the physical topology. As the
current kernel scheduler implementation, hyperthread with an even ID
number will be picked up in a much higher probability during load-balancing
and load-deploying. To solve the imbalance, when on a machine with multi
core and hyperthread index is consecutive per core, change the result of
select_idle_core() according to the hyperthread on which the task ran
before.

Signed-off-by: Kenan.Liu <Kenan.Liu@linux.alibaba.com>
Signed-off-by: Ben Luo <luoben@linux.alibaba.com>
---
 kernel/sched/fair.c | 38 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 36 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a80a739..ad7c93f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -125,6 +125,9 @@
 static unsigned int normalized_sysctl_sched_wakeup_granularity	= 1000000UL;
 
 const_debug unsigned int sysctl_sched_migration_cost	= 500000UL;
+static bool smt_neighbour_topo;
+static bool core_smt_topo_detect;
+static unsigned int smt_nr_cpu = 2;
 
 int sched_thermal_decay_shift;
 static int __init setup_sched_thermal_decay_shift(char *str)
@@ -140,6 +143,26 @@ static int __init setup_sched_thermal_decay_shift(char *str)
 __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift);
 
 #ifdef CONFIG_SMP
+static void explore_core_smp_topology(void)
+{
+	int cpu = smp_processor_id(), sibling;
+	const struct cpumask *smt_mask = cpu_smt_mask(cpu);
+
+	if (nr_cpu_ids <= 2)
+		return;
+
+	smt_nr_cpu = cpumask_weight(smt_mask);
+	if (smt_nr_cpu < 2)
+		return;
+
+	for_each_cpu(sibling, cpu_smt_mask(cpu)) {
+		if (cpu == sibling)
+			continue;
+		if (abs(cpu - sibling) == 1)
+			smt_neighbour_topo = true;
+	}
+}
+
 /*
  * For asym packing, by default the lower numbered CPU has higher priority.
  */
@@ -6887,9 +6910,16 @@ void __update_idle_core(struct rq *rq)
 static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
 {
 	bool idle = true;
-	int cpu;
+	int cpu, sibling = core;
+
+	if (!core_smt_topo_detect) {
+		explore_core_smp_topology();
+		core_smt_topo_detect = true;
+	}
 
 	for_each_cpu(cpu, cpu_smt_mask(core)) {
+		if (cpu != core)
+			sibling = cpu;
 		if (!available_idle_cpu(cpu)) {
 			idle = false;
 			if (*idle_cpu == -1) {
@@ -6905,8 +6935,12 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 			*idle_cpu = cpu;
 	}
 
-	if (idle)
+	if (idle) {
+		if (!smt_neighbour_topo || unlikely(core % smt_nr_cpu))
+			return core;
+		core = task_cpu(p) % smt_nr_cpu ? core : sibling;
 		return core;
+	}
 
 	cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
 	return -1;
-- 
1.8.3.1

[RFC PATCH 2/2] sched/fair: Export a param to control the traverse len when select idle cpu.

[Kenan.Liu]

From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>

The variable 'nr' decides the length of traverse when we try to find an
idle cpu in function select_idle_cpu(). A fixed value such as 4 may not
perform well in all scenes and may lead to un-acceptable overhead. Export
two sysctl parameters to enable adjustments.

Signed-off-by: Kenan.Liu <Kenan.Liu@linux.alibaba.com>
Signed-off-by: Ben Luo <luoben@linux.alibaba.com>
---
 kernel/sched/fair.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 63 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ad7c93f..e10de3b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -125,6 +125,9 @@
 static unsigned int normalized_sysctl_sched_wakeup_granularity	= 1000000UL;
 
 const_debug unsigned int sysctl_sched_migration_cost	= 500000UL;
+unsigned int __read_mostly sysctl_sched_idle_search_nr_default = 4;
+unsigned int __read_mostly sysctl_sched_idle_search_nr_threshold = 4;
+
 static bool smt_neighbour_topo;
 static bool core_smt_topo_detect;
 static unsigned int smt_nr_cpu = 2;
@@ -207,6 +210,50 @@ int __weak arch_asym_cpu_priority(int cpu)
 #endif
 
 #ifdef CONFIG_SYSCTL
+static int sched_set_idle_search_nr_default(struct ctl_table *table, int write, void *buffer,
+		size_t *lenp, loff_t *ppos)
+{
+	static DEFINE_MUTEX(mutex);
+	unsigned int old_nr;
+	int ret;
+
+	mutex_lock(&mutex);
+	old_nr = sysctl_sched_idle_search_nr_default;
+	ret = proc_douintvec(table, write, buffer, lenp, ppos);
+	if (!ret && write) {
+		if (sysctl_sched_idle_search_nr_default == 0) {
+			sysctl_sched_idle_search_nr_default = old_nr;
+			mutex_unlock(&mutex);
+			return -EINVAL;
+		}
+	}
+
+	mutex_unlock(&mutex);
+	return ret;
+}
+
+static int sched_set_idle_search_nr_threshold(struct ctl_table *table, int write, void *buffer,
+		size_t *lenp, loff_t *ppos)
+{
+	static DEFINE_MUTEX(mutex);
+	unsigned int old_threshold;
+	int ret;
+
+	mutex_lock(&mutex);
+	old_threshold = sysctl_sched_idle_search_nr_threshold;
+	ret = proc_douintvec(table, write, buffer, lenp, ppos);
+	if (!ret && write) {
+		if (sysctl_sched_idle_search_nr_threshold == 0) {
+			sysctl_sched_idle_search_nr_threshold = old_threshold;
+			mutex_unlock(&mutex);
+			return -EINVAL;
+		}
+	}
+
+	mutex_unlock(&mutex);
+	return ret;
+}
+
 static struct ctl_table sched_fair_sysctls[] = {
 	{
 		.procname       = "sched_child_runs_first",
@@ -235,6 +282,20 @@ int __weak arch_asym_cpu_priority(int cpu)
 		.extra1		= SYSCTL_ZERO,
 	},
 #endif /* CONFIG_NUMA_BALANCING */
+	{
+		.procname	= "sched_cfs_idle_search_nr_default",
+		.data		= &sysctl_sched_idle_search_nr_default,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_set_idle_search_nr_default,
+	},
+	{
+		.procname	= "sched_cfs_idle_search_nr_threshold",
+		.data		= &sysctl_sched_idle_search_nr_threshold,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_set_idle_search_nr_threshold,
+	},
 	{}
 };
 
@@ -7027,10 +7088,10 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 		avg_cost = this_sd->avg_scan_cost + 1;
 
 		span_avg = sd->span_weight * avg_idle;
-		if (span_avg > 4*avg_cost)
+		if (span_avg > sysctl_sched_idle_search_nr_threshold * avg_cost)
 			nr = div_u64(span_avg, avg_cost);
 		else
-			nr = 4;
+			nr = sysctl_sched_idle_search_nr_default;
 
 		time = cpu_clock(this);
 	}
-- 
1.8.3.1

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Peter Zijlstra]

On Thu, Jul 20, 2023 at 04:34:11PM +0800, Kenan.Liu wrote:
> From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>
> 
> Multithreading workloads in VM with Qemu may encounter an unexpected
> phenomenon: one hyperthread of a physical core is busy while its sibling
> is idle. Such as:

Is this with vCPU pinning? Without that, guest topology makes no sense
what so ever.

> The main reason is that hyperthread index is consecutive in qemu native x86 CPU
> model which is different from the physical topology. 

I'm sorry, what? That doesn't make sense. SMT enumeration is all over
the place for Intel, but some actually do have (n,n+1) SMT. On AMD it's
always (n,n+1) IIRC.

> As the current kernel scheduler
> implementation, hyperthread with an even ID number will be picked up in a much
> higher probability during load-balancing and load-deploying.

How so?

> This RFC targets to solve the problem by adjusting CFS loabalance policy:
> 1. Explore CPU topology and adjust CFS loadbalance policy when we found machine
> with qemu native CPU topology.
> 2. Export a procfs to control the traverse length when select idle cpu.
> 
> Kenan.Liu (2):
>   sched/fair: Adjust CFS loadbalance for machine with qemu native CPU
>     topology.
>   sched/fair: Export a param to control the traverse len when select
>     idle cpu.

NAK, qemu can either provide a fake topology to the guest using normal
x86 means (MADT/CPUID) or do some paravirt topology setup, but this is
quite insane.

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Kenan.Liu]

Hi Peter, thanks for your attention,

please refer to my answer to your question inline:


在 2023/7/20 下午4:50, Peter Zijlstra 写道:
> On Thu, Jul 20, 2023 at 04:34:11PM +0800, Kenan.Liu wrote:
>> From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>
>>
>> Multithreading workloads in VM with Qemu may encounter an unexpected
>> phenomenon: one hyperthread of a physical core is busy while its sibling
>> is idle. Such as:
> Is this with vCPU pinning? Without that, guest topology makes no sense
> what so ever.


vCPU is pinned on host and the imbalance phenomenon we observed is inside
VM, not for the vCPU threads on host.


>> The main reason is that hyperthread index is consecutive in qemu native x86 CPU
>> model which is different from the physical topology.
> I'm sorry, what? That doesn't make sense. SMT enumeration is all over
> the place for Intel, but some actually do have (n,n+1) SMT. On AMD it's
> always (n,n+1) IIRC.
>
>> As the current kernel scheduler
>> implementation, hyperthread with an even ID number will be picked up in a much
>> higher probability during load-balancing and load-deploying.
> How so?


The SMT topology in qemu native x86 CPU model is (0,1),…,(n,n+1),…,
but nomarlly seen SMT topo in physical machine is like (0,n),(1,n+1),…,
n means the total core number of the machine.

The imbalance happens when the number of runnable threads is less
than the number of hyperthreads, select_idle_core() would be called
to decide which cpu be selected to run the waken-up task.

select_idle_core() will return the checked cpu number if the whole
core is idle. On the contrary, if any one HT of the core is busy,
select_idle_core() would clear the whole core out from cpumask and
check the next core.

select_idle_core():
     …
     if (idle)
         return core;

     cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
     return -1;

In this manner, except the very beginning of for_each_cpu_wrap() loop,
HT with even ID number is always be checked at first, and be returned
to the caller if the whole core is idle, so the odd indexed HT almost
has no chance to be selected.

select_idle_cpu():
     …
     for_each_cpu_wrap(cpu, cpus, target + 1) {
         if (has_idle_core) {
             i = select_idle_core(p, cpu, cpus, &idle_cpu);

And this will NOT happen when the SMT topo is (0,n),(1,n+1),…, because
when the loop starts from the bottom half of SMT number, HTs with larger
number will be checked first, when it starts from the top half, their
siblings with smaller number take the first place of inner core searching.


>
>> This RFC targets to solve the problem by adjusting CFS loabalance policy:
>> 1. Explore CPU topology and adjust CFS loadbalance policy when we found machine
>> with qemu native CPU topology.
>> 2. Export a procfs to control the traverse length when select idle cpu.
>>
>> Kenan.Liu (2):
>>    sched/fair: Adjust CFS loadbalance for machine with qemu native CPU
>>      topology.
>>    sched/fair: Export a param to control the traverse len when select
>>      idle cpu.
> NAK, qemu can either provide a fake topology to the guest using normal
> x86 means (MADT/CPUID) or do some paravirt topology setup, but this is
> quite insane.
Thanks,

Kenan.Liu

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Vincent Guittot]

On Fri, 21 Jul 2023 at 04:59, Kenan.Liu <Kenan.Liu@linux.alibaba.com> wrote:
>
> Hi Peter, thanks for your attention,
>
> please refer to my answer to your question inline:
>
>
> 在 2023/7/20 下午4:50, Peter Zijlstra 写道:
> > On Thu, Jul 20, 2023 at 04:34:11PM +0800, Kenan.Liu wrote:
> >> From: "Kenan.Liu" <Kenan.Liu@linux.alibaba.com>
> >>
> >> Multithreading workloads in VM with Qemu may encounter an unexpected
> >> phenomenon: one hyperthread of a physical core is busy while its sibling
> >> is idle. Such as:
> > Is this with vCPU pinning? Without that, guest topology makes no sense
> > what so ever.
>
>
> vCPU is pinned on host and the imbalance phenomenon we observed is inside
> VM, not for the vCPU threads on host.
>
>
> >> The main reason is that hyperthread index is consecutive in qemu native x86 CPU
> >> model which is different from the physical topology.
> > I'm sorry, what? That doesn't make sense. SMT enumeration is all over
> > the place for Intel, but some actually do have (n,n+1) SMT. On AMD it's
> > always (n,n+1) IIRC.
> >
> >> As the current kernel scheduler
> >> implementation, hyperthread with an even ID number will be picked up in a much
> >> higher probability during load-balancing and load-deploying.
> > How so?
>
>
> The SMT topology in qemu native x86 CPU model is (0,1),…,(n,n+1),…,
> but nomarlly seen SMT topo in physical machine is like (0,n),(1,n+1),…,
> n means the total core number of the machine.
>
> The imbalance happens when the number of runnable threads is less
> than the number of hyperthreads, select_idle_core() would be called
> to decide which cpu be selected to run the waken-up task.
>
> select_idle_core() will return the checked cpu number if the whole
> core is idle. On the contrary, if any one HT of the core is busy,
> select_idle_core() would clear the whole core out from cpumask and
> check the next core.
>
> select_idle_core():
>      …
>      if (idle)
>          return core;
>
>      cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
>      return -1;
>
> In this manner, except the very beginning of for_each_cpu_wrap() loop,
> HT with even ID number is always be checked at first, and be returned
> to the caller if the whole core is idle, so the odd indexed HT almost
> has no chance to be selected.
>
> select_idle_cpu():
>      …
>      for_each_cpu_wrap(cpu, cpus, target + 1) {
>          if (has_idle_core) {
>              i = select_idle_core(p, cpu, cpus, &idle_cpu);
>
> And this will NOT happen when the SMT topo is (0,n),(1,n+1),…, because
> when the loop starts from the bottom half of SMT number, HTs with larger
> number will be checked first, when it starts from the top half, their
> siblings with smaller number take the first place of inner core searching.

But why is it a problem ? Your system is almost idle and 1 HT per core
is used. Who cares to select evenly one HT or the other as long as we
select an idle core in priority ?

This seems related to
https://lore.kernel.org/lkml/BYAPR21MB1688FE804787663C425C2202D753A@BYAPR21MB1688.namprd21.prod.outlook.com/
we concluded that it was not a problem

>
>
> >
> >> This RFC targets to solve the problem by adjusting CFS loabalance policy:
> >> 1. Explore CPU topology and adjust CFS loadbalance policy when we found machine
> >> with qemu native CPU topology.
> >> 2. Export a procfs to control the traverse length when select idle cpu.
> >>
> >> Kenan.Liu (2):
> >>    sched/fair: Adjust CFS loadbalance for machine with qemu native CPU
> >>      topology.
> >>    sched/fair: Export a param to control the traverse len when select
> >>      idle cpu.
> > NAK, qemu can either provide a fake topology to the guest using normal
> > x86 means (MADT/CPUID) or do some paravirt topology setup, but this is
> > quite insane.
> Thanks,
>
> Kenan.Liu

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Peter Zijlstra]

On Fri, Jul 21, 2023 at 10:33:44AM +0200, Vincent Guittot wrote:
> On Fri, 21 Jul 2023 at 04:59, Kenan.Liu <Kenan.Liu@linux.alibaba.com> wrote:

> > The SMT topology in qemu native x86 CPU model is (0,1),…,(n,n+1),…,
> > but nomarlly seen SMT topo in physical machine is like (0,n),(1,n+1),…,
> > n means the total core number of the machine.
> >
> > The imbalance happens when the number of runnable threads is less
> > than the number of hyperthreads, select_idle_core() would be called
> > to decide which cpu be selected to run the waken-up task.
> >
> > select_idle_core() will return the checked cpu number if the whole
> > core is idle. On the contrary, if any one HT of the core is busy,
> > select_idle_core() would clear the whole core out from cpumask and
> > check the next core.
> >
> > select_idle_core():
> >      …
> >      if (idle)
> >          return core;
> >
> >      cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
> >      return -1;
> >
> > In this manner, except the very beginning of for_each_cpu_wrap() loop,
> > HT with even ID number is always be checked at first, and be returned
> > to the caller if the whole core is idle, so the odd indexed HT almost
> > has no chance to be selected.
> >
> > select_idle_cpu():
> >      …
> >      for_each_cpu_wrap(cpu, cpus, target + 1) {
> >          if (has_idle_core) {
> >              i = select_idle_core(p, cpu, cpus, &idle_cpu);
> >
> > And this will NOT happen when the SMT topo is (0,n),(1,n+1),…, because
> > when the loop starts from the bottom half of SMT number, HTs with larger
> > number will be checked first, when it starts from the top half, their
> > siblings with smaller number take the first place of inner core searching.
> 
> But why is it a problem ? Your system is almost idle and 1 HT per core
> is used. Who cares to select evenly one HT or the other as long as we
> select an idle core in priority ?

Right, why is this a problem? Hyperthreads are supposed to be symmetric,
it doesn't matter which of the two are active, the important thing is to
only have one active if we can.

(Unlike Power7, they have asymmetric SMT)

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[luoben]

On 2023/7/21 17:13, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Jul 21, 2023 at 10:33:44AM +0200, Vincent Guittot wrote:
> > On Fri, 21 Jul 2023 at 04:59, Kenan.Liu <Kenan.Liu@linux.alibaba.com> wrote:
> 
> >> The SMT topology in qemu native x86 CPU model is (0,1),…,(n,n+1),…,
> >> but nomarlly seen SMT topo in physical machine is like (0,n),(1,n+1),…,
> >> n means the total core number of the machine.
> >>
> >> The imbalance happens when the number of runnable threads is less
> >> than the number of hyperthreads, select_idle_core() would be called
> >> to decide which cpu be selected to run the waken-up task.
> >>
> >> select_idle_core() will return the checked cpu number if the whole
> >> core is idle. On the contrary, if any one HT of the core is busy,
> >> select_idle_core() would clear the whole core out from cpumask and
> >> check the next core.
> >>
> >> select_idle_core():
> >>       …
> >>       if (idle)
> >>           return core;
> >>
> >>       cpumask_andnot(cpus, cpus, cpu_smt_mask(core));
> >>       return -1;
> >>
> >> In this manner, except the very beginning of for_each_cpu_wrap() loop,
> >> HT with even ID number is always be checked at first, and be returned
> >> to the caller if the whole core is idle, so the odd indexed HT almost
> >> has no chance to be selected.
> >>
> >> select_idle_cpu():
> >>       …
> >>       for_each_cpu_wrap(cpu, cpus, target + 1) {
> >>           if (has_idle_core) {
> >>               i = select_idle_core(p, cpu, cpus, &idle_cpu);
> >>
> >> And this will NOT happen when the SMT topo is (0,n),(1,n+1),…, because
> >> when the loop starts from the bottom half of SMT number, HTs with larger
> >> number will be checked first, when it starts from the top half, their
> >> siblings with smaller number take the first place of inner core searching.
> >
> > But why is it a problem ? Your system is almost idle and 1 HT per core
> > is used. Who cares to select evenly one HT or the other as long as we
> > select an idle core in priority ?
> 
> Right, why is this a problem? Hyperthreads are supposed to be symmetric,
> it doesn't matter which of the two are active, the important thing is to
> only have one active if we can.
> 
> (Unlike Power7, they have asymmetric SMT)
> 

hi Peter and Vincent,

Some upper-level monitoring logic may take the CPU usage as a metric for
computing resource scaling. Imbalanced scheduling can create the illusion
of CPU resource scarcity, leading to more frequent triggering of resource
expansion by the upper-level scheduling system. However, this is actually
a waste of resources. So, we think this may be a problem.

Could you please take a further look at PATCH#2? We found that the default
'nr' value did not perform well under our scenario, and we believe that
adjustable variables would be more appropriate.

Our scenario is as follows:
16 processes are running in a 32 CPU VM, with 8 threads per process,
they are all running the same job. 

The expected result is that the CPU usage is evenly distributed, but
we found that even-numbered cores were used for scheduling decisions
and consumed more CPU resources (5%~20%), mainly because of the default
value of nr=4. In this scenario, we found that nr=2 is more suitable.

Thanks,
Ben

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Peter Zijlstra]

On Mon, Jul 24, 2023 at 02:57:57PM +0800, luoben@linux.alibaba.com wrote:

> > Right, why is this a problem? Hyperthreads are supposed to be symmetric,
> > it doesn't matter which of the two are active, the important thing is to
> > only have one active if we can.
> > 
> > (Unlike Power7, they have asymmetric SMT)
> > 
> 
> hi Peter and Vincent,
> 
> Some upper-level monitoring logic may take the CPU usage as a metric for
> computing resource scaling. Imbalanced scheduling can create the illusion
> of CPU resource scarcity, leading to more frequent triggering of resource
> expansion by the upper-level scheduling system. However, this is actually
> a waste of resources. So, we think this may be a problem.

This is a problem of your monitoring logic -- there is absolutely no
functional problem with the kernel AFAICT.

Teach the thing about SMT instead.

> Could you please take a further look at PATCH#2? We found that the default
> 'nr' value did not perform well under our scenario, and we believe that
> adjustable variables would be more appropriate.

That patch is tweaking default disabled code -- which we should be
removing sometime soon I suppose. So no.

Re: [RFC PATCH 0/2] Adjust CFS loadbalance to adapt QEMU CPU topology.

[Peter Zijlstra]

On Fri, Jul 21, 2023 at 10:58:50AM +0800, Kenan.Liu wrote:

> > > As the current kernel scheduler
> > > implementation, hyperthread with an even ID number will be picked up in a much
> > > higher probability during load-balancing and load-deploying.
> > How so?
> 
> 
> The SMT topology in qemu native x86 CPU model is (0,1),…,(n,n+1),…,
> but nomarlly seen SMT topo in physical machine is like (0,n),(1,n+1),…,
> n means the total core number of the machine.

That is only common on Intel hardware, AMD (and some Intel) will in fact
enumerate SMT like Qemu does.