Re: [PATCH v4 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus

[Andrea Righi <arighi@nvidia.com>]

Hello,

On Wed, Sep 09, 2026 at 09:26:09AM +0200, Andrea Righi wrote:
> Hi Dietmar,
> 
> On Wed, Sep 09, 2026 at 09:20:35AM +0200, Dietmar Eggemann wrote:
> > On 08.09.26 10:23, Andrea Righi wrote:
> > 
> > [...]
> > 
> > > The series was tested on a two-node Vera system using an 88-thread
> > > single-precision GEMM on the 88 physical cores of NUMA node 0.
> > 
> > Can we use 'OpenBLAS benchmark/sgemm.goto' as an open alternative for
> > your NVIDIA internal single-precision GEMM benchmark?
> > 
> > IIUC, you used it for the 'Prefer fully idle cores for NOHZ balancing'
> > work: https://lore.kernel.org/r/anIq6pU5KXTTFCDN@gpd4
> > 
> > If yes, I assume you would run something like:
> > 
> > export OMP_NUM_THREADS=88
> > numactl -C XXX --membind=0 ./benchmark/sgemm.goto 16384 16384 16384
> > 
> > Essentially you want to show that those 88 compute intensive tasks each
> > runs on his own core alone and so you get a higher TFLOPS value.
> > 
> > [...]
> 
> Yes, sure! I'll re-run some tests with that and share the results in a bit.
> 
> Thanks,
> -Andrea

I repeated the tests using the latest patch series [1] both with OpenBLAS and
NVPL (internal GEMM benchmark).

Kernels and test configuration
------------------------------

mainline: Linux 7.3.0-rc2
smt-pe0-prio: Linux 7.3.0-rc2 + patch series [1] applied

Both tests used:
 - 88 threads on NUMA node 0 (CPU list 0-87,176-263)
 - performance governor with cppc_cpufreq
 - same OpenBLAS binary and NVPL container image
 - metrics over 5 repetitions

Results
-------

Delta is (smt-pe0-prio / mainline - 1): higher is better.

 +---------------------+-------+---------------------+-----------------------+--------+
 | Throughput          | Runs  | mainline TFLOP/s    | smt-pe0-prio TFLOP/s  | Delta  |
 +---------------------+-------+---------------------+-----------------------+--------+
 | OpenBLAS            | 5 / 5 | 7.11876 +/- 0.06734 |  7.34669 +/- 0.01936  | +3.20% |
 | NVPL                | 5 / 5 | 9.64742 +/- 0.17311 | 10.29695 +/- 0.01786  | +6.73% |
 +---------------------+-------+----------------------+----------------------+--------+

Hardware statistics
-------------------

ST = single-thread mode
SMT = two-thread mode

Delta is (smt-pe0-prio / mainline - 1): lower is better.

OpenBLAS:
 +------------------------------+----------------------+----------------------+----------+
 | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
 +------------------------------+----------------------+----------------------+----------+
 | ST-to-SMT completed/run      | 10145.6 +/- 1835.2   | 1981.6 +/- 94.3      |  -80.47% |
 | SMT-to-ST completed/run      | 10342.6 +/- 1853.6   | 1946.2 +/- 93.1      |  -81.18% |
 | ST-to-SMT transitions/s      |   845.5 +/- 152.9    |  176.9 +/- 3.4       |  -79.08% |
 | SMT-to-ST transitions/s      |   861.9 +/- 154.5    |  173.7 +/- 1.5       |  -79.84% |
 | ST-to-SMT latency cycles/run | 15.785M +/- 3.315M   |  2.477M +/- 0.090M   |  -84.30% |
 | SMT-to-ST latency cycles/run |  9.545M +/- 1.762M   |  1.815M +/- 0.047M   |  -80.98% |
 +------------------------------+----------------------+----------------------+----------+

NVPL:
 +------------------------------+----------------------+----------------------+----------+
 | PMU metric                   | mainline             | smt-pe0-prio         | Delta    |
 +------------------------------+----------------------+----------------------+----------+
 | ST-to-SMT completed/run      | 7771.0 +/- 1312.2    | 2162.6 +/- 137.8     |  -72.17% |
 | SMT-to-ST completed/run      | 7759.8 +/- 1352.2    | 2135.2 +/- 108.0     |  -72.48% |
 | SMT-to-ST aborted/run        |    0.6 +/- 0.5       |    0.2 +/- 0.4       |  -66.67% |
 | ST-to-SMT transitions/s      |  777.1 +/- 131.2     |  251.5 +/- 4.8       |  -67.64% |
 | SMT-to-ST transitions/s      |  776.0 +/- 135.2     |  248.5 +/- 5.8       |  -67.98% |
 | ST-to-SMT latency cycles/run | 13.285M +/- 3.742M   |  2.971M +/- 0.296M   |  -77.64% |
 | SMT-to-ST latency cycles/run |  8.528M +/- 2.223M   |  2.287M +/- 0.126M   |  -73.18% |
 +------------------------------+----------------------+----------------------+----------+

Conclusion
----------

The patch leaves both workloads almost entirely in ST mode and substantially
reduces ST/SMT mode-transition churn.

Relative to mainline, completed ST-to-SMT transitions fall by 80.5% for OpenBLAS
and 72.2% for NVPL. This agrees with the throughput result: the scheduling
preference avoids repeatedly switching the active PE identity and allows cores
to remain in full-resource ST mode for longer intervals.

[1] https://lore.kernel.org/r/20260909062649.469633-1-arighi@nvidia.com

-Andrea