Re: [PATCH v6 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Dietmar Eggemann <dietmar.eggemann@arm.com>]

On 17.09.26 16:05, Andrea Righi wrote:

[...]

> The first patch teaches the fair scheduler's idle-selection paths to honor
> SD_ASYM_PACKING at the shared-capacity SMT level. The scheduler first selects a
> candidate CPU and core according to its existing placement and capacity rules,
> then chooses the highest-priority available sibling within that core. This also
> completes the existing POWER7 SD_ASYM_PACKING behavior by applying its
> hardware-thread ordering during idle selection.
> 
> Olympus firmware does not currently provide an interface to describe the
> preferred SMT sibling. Adding such a firmware or ACPI interface will take time
> and will not help systems with existing firmware. At the same time, inferring
> this policy from MIDR would encode a platform-specific decision in the kernel
> and make it harder to replace with a proper firmware ABI.
> 
> The second patch therefore adds the sched_smt_asym_packing= boot option. Using
> sched_smt_asym_packing=on explicitly opts the SMT scheduling domain into
> SD_ASYM_PACKING without requiring architecture-specific detection. Priority
> remains defined by arch_asym_cpu_priority(). The weak default orders siblings by
> -cpu, consistently selecting the lowest-numbered available logical CPU.
> Architecture overrides remain authoritative, so siblings assigned equal
> priorities remain unordered. The default auto mode preserves
> architecture-provided topology policy, including the existing powerpc behavior,
> while off provides an explicit override to disable SMT asymmetric packing.
> 
> On Olympus, PE0 and PE1 have equal steady-state capacity; this preference does
> not identify a faster PE. The lower-numbered logical CPU is used only as a
> canonical choice when both siblings are available. Consistently selecting the
> same sibling avoids alternating the active PE across wakeups, lets the other
> sibling remain idle for longer, and allows more cores to remain in, or return
> to, full-resource single-thread mode.

So I think that Power7 and other SMT machines won't ever turn
'sched_smt_asym_packing' on then.
As we have seen in https://lore.kernel.org/r/aqSEB2N_NQbBVab6@gpd4 this
code will only benefit NVIDIAs Spatial SMT, i.e. dynamically
partitioning a physical core, rather than conventional SMT where two
threads opportunistically compete for most of the same machinery.

IIUC on Spatial SMT, placing the workload on PE0 is beneficial because
even if interrupts and other per-CPU housekeeping activities have to be
handled by PE0 next to the benchmark tasks, the main thing is that PE0
stays in full-resource single-thread mode (PE1 stays idle).
> 
> The v6 series was tested on a two-node Vera system using 88-thread
> single-precision GEMM workloads on the 88 physical cores of NUMA node 0, with
> sched_smt_asym_packing=on and the workloads allowed to choose either sibling of
> every core. Each result covers five runs.
> 
> Two BLAS implementations were tested: OpenBLAS, an open-source BLAS library that
> provides a publicly reproducible benchmark, and NVIDIA Performance Libraries
> (NVPL), NVIDIA's optimized BLAS implementation.
> 
> OpenBLAS was evaluated using benchmark/sgemm.goto with an M=N=K=16384
> single-precision GEMM. NVPL was evaluated using benchblas with the same matrix
> dimensions, non-transposed inputs, alpha=1 and beta=0. The numbers below are the
> mean and standard deviation from five runs.

So I assume that you have 88 consistently running benchmark threads and
in case your new code let them run more likely on PE0 instead PE1 you
see this throughput increase.

> OpenBLAS throughput increased from 7.11876 +/- 0.06734 TFLOP/s on the baseline
> kernel to 7.34669 +/- 0.01936 TFLOP/s with this series (+3.20%). NVPL throughput
> increased from 9.64742 +/- 0.17311 TFLOP/s to 10.29695 +/- 0.01786 TFLOP/s
> (+6.73%).

I assume further that NVPL has the same benchmark task model, maybe with
a couple of sleep/wakeups in between?

[...]