Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - NAS BT

[Bharata B Rao <bharata@amd.com>]

NAS Parallel Benchmark BT (Block Tri-diagonal solver), MPI version,
Class F modified, run with 16 ranks. Compute threads are bound to node 1 via:

  /usr/bin/mpirun -np 16 /usr/bin/numactl --cpunodebind=1 \
      NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the
benefit of promition), class E results in around 368G of memory which
overflows my toptier. Hence I wanted something in between these classes.
So I have  modified class F to the problem size of 768 which results in
around 160GB of memory.

System is a 3-node tiered setup: node 0 and node 1 are DRAM (CPUs),
node 2 is a CPU-less CXL lower tier (~252 GiB per node). Before the
measurement phase all provisioned memory (~161.4 GiB across ranks)
is migrated to the lower-tier CXL node 2. The ranks are then resumed
and measurement begins, so the run starts fully cold on slow memory.
This isolates the effect of hot-page promotion (node 2 -> node 1)
on a bandwidth/latency-sensitive workload.

Legend (kernel / promotion source)
----------------------------------------------------------------------
  A = base-numab0       Base kernel, NO tiering (NUMAB=0); reference
  B = base-numab2       Base kernel, tiering via NUMA Balancing
                        mode 2 (hint-fault driven promotion)
  C = pghot-hintfaults  pghot kernel, hint-fault source (NUMAB=2)
  D = pghot-hwhints     pghot kernel, HW-hint source (IBS memory
                        profiler); does NOT need NUMA Balancing

Table 1a - Runtime (lower is better)
----------------------------------------------------------------------
Metric                          A          B          C          D
----------------------------------------------------------------------
NPB Time (s)              7482.25    3306.81    3313.82    3506.55
Speedup vs A (time)          1.00       2.26       2.26       2.13

Table 1b - Throughput (higher is better)
----------------------------------------------------------------------
Metric                          A          B          C          D
----------------------------------------------------------------------
NPB Mop/s total          52305.50  118350.48  118100.13  111608.96
Mop/s ratio vs A             1.00       2.26       2.26       2.13
Mop/s ratio vs B             0.44       1.00       1.00       0.94
Peak prov. mem (GiB)       161.38     161.38     161.38     161.38

Table 2 - vmstat counter deltas (after - before measurement window)
----------------------------------------------------------------------
Counter                             A         B         C         D
----------------------------------------------------------------------
pgpromote_success                   0  42180536  42169996  39898316
pgpromote_candidate                 0         0         0         0
pgpromote_candidate_nrl             0  42180536  42169996  39899553
pgdemote_kswapd                     0         0         0         0
pgdemote_direct                     0         0         0         0
numa_pte_updates                    0  42250308  45027035         0
numa_hint_faults                    0  42182545  44119239         0
numa_pages_migrated                 0  42182543  42169996  39898316
pgmigrate_success                   0  42182543  42169996  39898316
pghot_recorded_accesses             0         0  44119239  40004334
pghot_recorded_hintfaults           0         0  44119239         0
pghot_recorded_hwhints              0         0         0 584209652
hwhint_total_events                 0         0         0 584209975
hwhint_dram_accesses                0         0         0 544195074
hwhint_extmem_accesses              0         0         0  40005074
hwhint_useful_events                0         0         0 584209652

Key observations
----------------------------------------------------------------------
Bottom line: hint-fault pghot reproduces mainline NUMAB=2 exactly,
while HW-hint pghot delivers ~94% of that performance with zero
NUMA-balancing overhead - a good trade-off where IBS is available.