Re: [PATCH v6 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Andrea Righi <arighi@nvidia.com>]

Hi Dietmar,

On Mon, Sep 21, 2026 at 11:04:42AM +0200, Dietmar Eggemann wrote:
> On 17.09.26 16:05, Andrea Righi wrote:
> 
> [...]
> 
> > The first patch teaches the fair scheduler's idle-selection paths to honor
> > SD_ASYM_PACKING at the shared-capacity SMT level. The scheduler first selects a
> > candidate CPU and core according to its existing placement and capacity rules,
> > then chooses the highest-priority available sibling within that core. This also
> > completes the existing POWER7 SD_ASYM_PACKING behavior by applying its
> > hardware-thread ordering during idle selection.
> > 
> > Olympus firmware does not currently provide an interface to describe the
> > preferred SMT sibling. Adding such a firmware or ACPI interface will take time
> > and will not help systems with existing firmware. At the same time, inferring
> > this policy from MIDR would encode a platform-specific decision in the kernel
> > and make it harder to replace with a proper firmware ABI.
> > 
> > The second patch therefore adds the sched_smt_asym_packing= boot option. Using
> > sched_smt_asym_packing=on explicitly opts the SMT scheduling domain into
> > SD_ASYM_PACKING without requiring architecture-specific detection. Priority
> > remains defined by arch_asym_cpu_priority(). The weak default orders siblings by
> > -cpu, consistently selecting the lowest-numbered available logical CPU.
> > Architecture overrides remain authoritative, so siblings assigned equal
> > priorities remain unordered. The default auto mode preserves
> > architecture-provided topology policy, including the existing powerpc behavior,
> > while off provides an explicit override to disable SMT asymmetric packing.
> > 
> > On Olympus, PE0 and PE1 have equal steady-state capacity; this preference does
> > not identify a faster PE. The lower-numbered logical CPU is used only as a
> > canonical choice when both siblings are available. Consistently selecting the
> > same sibling avoids alternating the active PE across wakeups, lets the other
> > sibling remain idle for longer, and allows more cores to remain in, or return
> > to, full-resource single-thread mode.
> 
> So I think that Power7 and other SMT machines won't ever turn
> 'sched_smt_asym_packing' on then.

Right, POWER7 already sets SD_ASYM_PACKING on its SMT domain (recognized through
processor version register), so it doesn't need the boot option.

The boot option is only for systems such as Olympus whose firmware doesn't
currently describe a sibling preference.

> As we have seen in https://lore.kernel.org/r/aqSEB2N_NQbBVab6@gpd4 this
> code will only benefit NVIDIAs Spatial SMT, i.e. dynamically
> partitioning a physical core, rather than conventional SMT where two
> threads opportunistically compete for most of the same machinery.

Right, on conventional SMT systems with symmetric siblings and no
architecture-defined priority, I don't expect any performance benefit enabling
this option.

> 
> IIUC on Spatial SMT, placing the workload on PE0 is beneficial because
> even if interrupts and other per-CPU housekeeping activities have to be
> handled by PE0 next to the benchmark tasks, the main thing is that PE0
> stays in full-resource single-thread mode (PE1 stays idle).

Correct. On Olympus the lower-numbered logical sibling is PE0, prioritizing it
consistently reduces the activations of both PEs, so PE0 stays in full-resource
mode.

> > 
> > The v6 series was tested on a two-node Vera system using 88-thread
> > single-precision GEMM workloads on the 88 physical cores of NUMA node 0, with
> > sched_smt_asym_packing=on and the workloads allowed to choose either sibling of
> > every core. Each result covers five runs.
> > 
> > Two BLAS implementations were tested: OpenBLAS, an open-source BLAS library that
> > provides a publicly reproducible benchmark, and NVIDIA Performance Libraries
> > (NVPL), NVIDIA's optimized BLAS implementation.
> > 
> > OpenBLAS was evaluated using benchmark/sgemm.goto with an M=N=K=16384
> > single-precision GEMM. NVPL was evaluated using benchblas with the same matrix
> > dimensions, non-transposed inputs, alpha=1 and beta=0. The numbers below are the
> > mean and standard deviation from five runs.
> 
> So I assume that you have 88 consistently running benchmark threads and
> in case your new code let them run more likely on PE0 instead PE1 you
> see this throughput increase.

Yes, both benchmarks used 88 threads across 88 physical cores, with both
siblings eligible. I captured per-thread scheduler traces and perf counters,
with PE prioritization, placement favored PE0 and single-thread/two-thread mode
transitions fell consistently. In the earlier five-run comparison the ST/SMT
transitions fell by about ~80%.

> 
> > OpenBLAS throughput increased from 7.11876 +/- 0.06734 TFLOP/s on the baseline
> > kernel to 7.34669 +/- 0.01936 TFLOP/s with this series (+3.20%). NVPL throughput
> > increased from 9.64742 +/- 0.17311 TFLOP/s to 10.29695 +/- 0.01786 TFLOP/s
> > (+6.73%).
> 
> I assume further that NVPL has the same benchmark task model, maybe with
> a couple of sleep/wakeups in between?

Correct, I just checked the sleep/wakeup pattern: across 88 threads, the NVPL
run shows ~2.6K interruptible sleeps/wakeups and ~3K futex waits. The GEMM
OpenBLAS run shows only ~250 sleeps/wakeups and futex waits. All those repeated
wakeups give the scheduler more opportunities to choose a sibling and are likely
the reason of NVPL's bigger improvement.

Thanks,
-Andrea