Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - Graph500

[Bharata B Rao <bharata@amd.com>]

Benchmark : Graph500 reference BFS, SCALE=28, edgefactor=16, 128 ranks
Topology  : top-tier NUMA node=1 (CPUs+DRAM), lower-tier node=2 (mem-only/CXL)

Config legend
-------------
R1 : base  kernel, no tiering        (numa_balancing=0)
R2 : base  kernel, tiering           (numa_balancing=2 / NUMAB2)
R3 : pghot kernel, hintfaults source (pghot_enabled_sources=1,
     pghot_freq_threshold=1, numa_balancing=2, promotion=on)
R4 : pghot kernel, hwhints  source   (pghot_enabled_sources=2,
     pghot_freq_threshold=1, IBS mem-profiler on, numa_balancing=0,
     promotion=off)

Column legend
-------------
hmean   : harmonic_mean_TEPS   (primary Graph500 metric)
hstddev : harmonic_stddev_TEPS
median  : median_TEPS
bfs_t   : mean BFS time (seconds)
spdup   : speedup of hmean vs R1 baseline

Table 1: Performance
--------------------
+--------+------------+----------+------------+---------+--------+
| Config |   hmean    | hstddev  |   median   |  bfs_t  | spdup  |
|        |   (TEPS)   |  (TEPS)  |   (TEPS)   |  (sec)  |        |
+--------+------------+----------+------------+---------+--------+
| R1     | 5.469e+08  | 4.55e+05 | 5.468e+08  |  7.853  | 1.00x  |
| R2     | 1.442e+09  | 8.04e+07 | 1.491e+09  |  2.979  | 2.64x  |
| R3     | 1.362e+09  | 7.92e+07 | 1.515e+09  |  3.153  | 2.49x  |
| R4     | 1.463e+09  | 1.35e+07 | 1.487e+09  |  2.935  | 2.68x  |
+--------+------------+----------+------------+---------+--------+

Table 2: Relevant kernel counters (/proc/vmstat deltas over the run)
--------------------------------------------------------------------
Values are accumulated deltas (before -> after) for the whole run.

+----------------------------+-----------+-----------+-----------+-----------+
| Counter                    |    R1     |    R2     |    R3     |    R4     |
+----------------------------+-----------+-----------+-----------+-----------+
| numa_pte_updates           |         0 |  26675234 |  28872628 |         0 |
| numa_hint_faults           |         0 |  13747383 |  15501854 |         0 |
| numa_pages_migrated        |         0 |  13747376 |  13726437 |   3683579 |
| pgpromote_success          |         0 |  13747254 |  13726437 |   3683579 |
| pghot_recorded_accesses    |         0 |         0 |  15501778 |   3682112 |
| pghot_recorded_hintfaults  |         0 |         0 |  15501854 |         0 |
| pghot_recorded_hwhints     |         0 |         0 |         0 |  24291151 |
| hwhint_total_events        |         0 |         0 |         0 |  24291198 |
| hwhint_dram_accesses       |         0 |         0 |         0 |  19588681 |
| hwhint_extmem_accesses     |         0 |         0 |         0 |   3663926 |
| pgmigrate_success          |  26840188 |  40586709 |  40570940 |  30521808 |
+----------------------------+-----------+-----------+-----------+-----------+

Key findings
------------
1. Tiering helps Graph500.
2. pghot-hintfaults is on par with base NUMAB2 tiering.
3. pghot-hwhints (R4) matches NUMAB=2 performance with zero NUMA hint
   faults.