[PATCH 0/2] x86/smpboot: tidy sched-topology and drop useless SMT level

[PATCH 0/2] x86/smpboot: tidy sched-topology and drop useless SMT level

[Li Chen]

From: Li Chen <chenl311@chinatelecom.cn>

This two–patch series cleans up sched-domain topology handling and
eliminates hundreds of pointless attach/destroy cycles when SMT is
not available.

Patch 1  (from Thomas, unchanged) gets rid of the #ifdef maze in
build_sched_topology() by statically initialising the topology array.

Patch 2  (mine) is a follow-up that simply memmoves the array when
cpu_smt_num_threads <= 1, so the SMT level never gets created and
immediately torn down again.

Tested on Qemu.

Li Chen (1):
  x86/smpboot: avoid SMT domain attach/destroy if SMT is not enabled

Thomas Gleixner (1):
  x86/smpboot: Decrapify build_sched_topology()

 arch/x86/kernel/smpboot.c | 59 +++++++++++++++++++++------------------
 1 file changed, 32 insertions(+), 27 deletions(-)

-- 
2.49.0

[PATCH 1/2] x86/smpboot: Decrapify build_sched_topology()

[Li Chen]

From: Thomas Gleixner <tglx@linutronix.de>

The #ifdeffery and the initializers in build_sched_topology() are just
disgusting. The SCHED_SMT #ifdef is also pointless because SCHED_SMT is
unconditionally enabled when SMP is enabled.

Statically initialize the domain levels in the topology array and let
build_sched_topology() invalidate the package domain level when NUMA in
package is available.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/smpboot.c | 45 +++++++++++++++------------------------
 1 file changed, 17 insertions(+), 28 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index fc78c2325fd29..7d202f9785362 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -478,43 +478,32 @@ static int x86_cluster_flags(void)
  */
 static bool x86_has_numa_in_package;
 
-static struct sched_domain_topology_level x86_topology[6];
+#define DOMAIN(maskfn, flagsfn, dname) { .mask = maskfn, .sd_flags = flagsfn, .name = #dname }
 
-static void __init build_sched_topology(void)
-{
-	int i = 0;
-
-#ifdef CONFIG_SCHED_SMT
-	x86_topology[i++] = (struct sched_domain_topology_level){
-		cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT)
-	};
-#endif
+static struct sched_domain_topology_level x86_topology[] = {
+	DOMAIN(cpu_smt_mask, cpu_smt_flags, SMT),
 #ifdef CONFIG_SCHED_CLUSTER
-	x86_topology[i++] = (struct sched_domain_topology_level){
-		cpu_clustergroup_mask, x86_cluster_flags, SD_INIT_NAME(CLS)
-	};
+	DOMAIN(cpu_clustergroup_mask, x86_cluster_flags, CLS),
 #endif
 #ifdef CONFIG_SCHED_MC
-	x86_topology[i++] = (struct sched_domain_topology_level){
-		cpu_coregroup_mask, x86_core_flags, SD_INIT_NAME(MC)
-	};
+	DOMAIN(cpu_coregroup_mask, x86_core_flags, MC),
 #endif
-	/*
-	 * When there is NUMA topology inside the package skip the PKG domain
-	 * since the NUMA domains will auto-magically create the right spanning
-	 * domains based on the SLIT.
-	 */
-	if (!x86_has_numa_in_package) {
-		x86_topology[i++] = (struct sched_domain_topology_level){
-			cpu_cpu_mask, x86_sched_itmt_flags, SD_INIT_NAME(PKG)
-		};
-	}
+	DOMAIN(cpu_cpu_mask, x86_sched_itmt_flags, PKG),
+	{ NULL },
+};
 
+static void __init build_sched_topology(void)
+{
 	/*
-	 * There must be one trailing NULL entry left.
+	 * When there is NUMA topology inside the package invalidate the
+	 * PKG domain since the NUMA domains will auto-magically create the
+	 * right spanning domains based on the SLIT.
 	 */
-	BUG_ON(i >= ARRAY_SIZE(x86_topology)-1);
+	if (x86_has_numa_in_package) {
+		unsigned int pkgdom = ARRAY_SIZE(x86_topology) - 2;
 
+		memset(&x86_topology[pkgdom], 0, sizeof(x86_topology[pkgdom]));
+	}
 	set_sched_topology(x86_topology);
 }
 
-- 
2.49.0

[PATCH 2/2] x86/smpboot: avoid SMT domain attach/destroy if SMT is not enabled

[Li Chen]

From: Li Chen <chenl311@chinatelecom.cn>

Currently, the SMT domain is added into sched_domain_topology
by default if CONFIG_SCHED_SMT is enabled.

If cpu_attach_domain finds that the CPU SMT domain’s cpumask_weight
is just 1, it will destroy_sched_domain it.

On a large machine, such as one with 512 cores, this results in
512 redundant domain attach/destroy operations.

We can avoid these unnecessary operations by simply checking
cpu_smt_num_threads and not inserting SMT domain into x86_topology if SMT
is not enabled.

Signed-off-by: Li Chen <chenl311@chinatelecom.cn>
---
 arch/x86/kernel/smpboot.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 7d202f9785362..9ff8b10715cc1 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -492,8 +492,24 @@ static struct sched_domain_topology_level x86_topology[] = {
 	{ NULL },
 };
 
+static void __init maybe_remove_smt_level(void)
+{
+	if (cpu_smt_num_threads <= 1) {
+		/*
+		 * SMT level is x86_topology[0].  Shift the array left by one,
+		 * keep the sentinel { NULL } at the end.
+		 */
+		memmove(&x86_topology[0], &x86_topology[1],
+			sizeof(x86_topology) - sizeof(x86_topology[0]));
+		memset(&x86_topology[ARRAY_SIZE(x86_topology) - 1], 0,
+		       sizeof(x86_topology[0]));
+	}
+}
+
 static void __init build_sched_topology(void)
 {
+	maybe_remove_smt_level();
+
 	/*
 	 * When there is NUMA topology inside the package invalidate the
 	 * PKG domain since the NUMA domains will auto-magically create the
-- 
2.49.0

Re: [PATCH 2/2] x86/smpboot: avoid SMT domain attach/destroy if SMT is not enabled

[Thomas Gleixner]

On Tue, Jun 24 2025 at 16:08, Li Chen wrote:
> From: Li Chen <chenl311@chinatelecom.cn>
>
> Currently, the SMT domain is added into sched_domain_topology
> by default if CONFIG_SCHED_SMT is enabled.
>
> If cpu_attach_domain finds that the CPU SMT domain’s cpumask_weight

If cpu_attach_domain()

IIRC, I told you that before.

> is just 1, it will destroy_sched_domain it.
>
> On a large machine, such as one with 512 cores, this results in
> 512 redundant domain attach/destroy operations.
>
> We can avoid these unnecessary operations by simply checking

s/We can avoid/Avoid/

Care to read my reviews? If you disagree, then discuss it with me, but
silently ignoring it them is not an option.

> cpu_smt_num_threads and not inserting SMT domain into x86_topology if SMT

not inserting? That's not what this new version does.

> +static void __init maybe_remove_smt_level(void)
> +{
> +	if (cpu_smt_num_threads <= 1) {
> +		/*
> +		 * SMT level is x86_topology[0].  Shift the array left by one,
> +		 * keep the sentinel { NULL } at the end.
> +		 */
> +		memmove(&x86_topology[0], &x86_topology[1],
> +			sizeof(x86_topology) - sizeof(x86_topology[0]));
> +		memset(&x86_topology[ARRAY_SIZE(x86_topology) - 1], 0,
> +		       sizeof(x86_topology[0]));

So this sets the last entry in the array, aka the original sentinel in
the last array entry, to zero...

This is completely pointless. The above memmove() copies

	topo[1 .. (N - 1)] 
to
	topo[0 .. (N - 2)]

Where N = ARRAY_SIZE(topo).

Therefore
        topo[N - 1] == NULL
and
        topo[N - 2] == NULL

No?

But then what's worse is that you fail to take that removal into account
for the x86_has_numa_in_package case, which still unconditionally sets
topo[N - 2] to zero even if the SMT level had been removed...

Please take your time and do not rush out half baked stuff.

Thanks,

        tglx

Re: [PATCH 2/2] x86/smpboot: avoid SMT domain attach/destroy if SMT is not enabled

[Li Chen]

Hi Thomas. 

 ---- On Tue, 24 Jun 2025 21:36:10 +0800  Thomas Gleixner <tglx@linutronix.de> wrote --- 
 > On Tue, Jun 24 2025 at 16:08, Li Chen wrote:
 > > From: Li Chen <chenl311@chinatelecom.cn>
 > >
 > > Currently, the SMT domain is added into sched_domain_topology
 > > by default if CONFIG_SCHED_SMT is enabled.
 > >
 > > If cpu_attach_domain finds that the CPU SMT domain’s cpumask_weight
 > 
 > If cpu_attach_domain()
 > 
 > IIRC, I told you that before.
 > 
 > > is just 1, it will destroy_sched_domain it.
 > >
 > > On a large machine, such as one with 512 cores, this results in
 > > 512 redundant domain attach/destroy operations.
 > >
 > > We can avoid these unnecessary operations by simply checking
 > 
 > s/We can avoid/Avoid/
 > 
 > Care to read my reviews? If you disagree, then discuss it with me, but

I'm sorry that I forget to say that your previous wording review have already been fixed in v2 https://lore.kernel.org/all/20250624085559.69436-3-me@linux.beauty/

And I would replace cpu_attach_domain with cpu_attach_domain().

Sorry for wasting your time. 

 > silently ignoring it them is not an option.
 > 
 > > cpu_smt_num_threads and not inserting SMT domain into x86_topology if SMT
 > 
 > not inserting? That's not what this new version does.
 > 
 > > +static void __init maybe_remove_smt_level(void)
 > > +{
 > > +    if (cpu_smt_num_threads <= 1) {
 > > +        /*
 > > +         * SMT level is x86_topology[0].  Shift the array left by one,
 > > +         * keep the sentinel { NULL } at the end.
 > > +         */
 > > +        memmove(&x86_topology[0], &x86_topology[1],
 > > +            sizeof(x86_topology) - sizeof(x86_topology[0]));
 > > +        memset(&x86_topology[ARRAY_SIZE(x86_topology) - 1], 0,
 > > +               sizeof(x86_topology[0]));
 > 
 > So this sets the last entry in the array, aka the original sentinel in
 > the last array entry, to zero...
 > 
 > This is completely pointless. The above memmove() copies
 > 
 >     topo[1 .. (N - 1)] 
 > to
 >     topo[0 .. (N - 2)]
 > 
 > Where N = ARRAY_SIZE(topo).
 > 
 > Therefore
 >         topo[N - 1] == NULL
 > and
 >         topo[N - 2] == NULL
 > 
 > No?
 > 
 > But then what's worse is that you fail to take that removal into account
 > for the x86_has_numa_in_package case, which still unconditionally sets
 > topo[N - 2] to zero even if the SMT level had been removed...
 > 
 > Please take your time and do not rush out half baked stuff.
 
Sorry again for my mistake. I will fix it in v3.

Regards,
Li