Re: [PATCH v5 00/36] Private Memory NUMA Nodes

[Arun George/Arun George <arun.george@samsung.com>]

Hi Gregory,

Thanks for sharing the working branch.

Some of us at Samsung are testing the series (both v4 & v5) on a 
compression capable CXL expander and are observing encouraging results.

Good things first! We feel that the isolation using private nodes and 
the capability selection using NODE_PRIVATE_CAP_* work functionally well 
for the compressed memory use cases.

On improvement points, we observed much higher 'page_faults', 
'allocation_stalls' etc. which contributed to the higher tail latencies 
in some tests. I hope these are already part of the optimization plans. 
The higher latencies might have resulted from the write_protection 
applied to the private node, I believe.

On the tools side, we used MLC and Taobench for the tests.

Note that our intention for this phase of experiments was to test only 
the 'private node' layer based isolation, and not the cram and below 
layers for the compressed memory. Therefore we did not enable/utilize 
the compression capability in the hardware for these set of experiments 
(enabling compression would require the cram level ballooning/memory 
shrinking and cxl level interfacing driver to the compression device). 
That would be different set of experiments where we would be testing the 
cram balloon shrinkers and our alternate algorithms (upstream targeted). 
So cram and below layers were used only for enumeration of private node 
regions for these tests and not for the run-time memory shrinkers.

======= Test Methodology ==============

We explored these 3 cases:

    1) (Baseline – existing CXL infra):  DRAM 16GB + CXL 32GB. Here we 
used the existing CXL driver infra to enable the device. No private node 
is involved. And compression is disabled on device.

    2) (private node in default write protected path): DRAM 16GB + CXL 
Private Node 32GB. Here private node infra is used to enable the device. 
And compression is disabled on device.

    3) (private node with no write protection): DRAM 16GB + CXL Private 
Node 32GB. Here private node infra is used to enable the device without 
the write protection/fencing enabled. Compression is disabled on device. 
We could not complete these runs as they resulted in kernel panics. 
Guess the code path is not stable yet for this (We had hoped that case 1 
and case 3 results would be similar). We also observed some unmovable 
page warnings logs in 'dmesg' before crash (might be related to the 
panic). Adding the log snippets at the end.

========= Results summary ===================

- TaoBench: ‘private node’ showed much higher latencies. We believe this 
could be due to the migration back into the DRAM for overwrites (due to 
write-protection) based on the kernel VM stats.

- MLC:  For lower inject delay, the private node shows better latency; 
but when inject delay goes high, the baseline is better.

Overall kernel VM stats suggested much higher ‘page faults, ‘allocation 
stalls’ etc. in ‘private node’ case which is corroborated with the results.

========== Setup ======================

CPU: 144 Cores (single socket), DRAM: 16GB, CXL: 32GB,
Kernel: 7.2.0-rc2+ (Private Node), 6.18.32 (Baseline)

Private node opt CAPs:
#define CRAM_NP_CAPS    (NODE_PRIVATE_CAP_RECLAIM |
                         NODE_PRIVATE_CAP_HOTUNPLUG | \
                         NODE_PRIVATE_CAP_DEMOTION |
                         NODE_PRIVATE_CAP_USER_NUMA | \
                         NODE_PRIVATE_CAP_NUMA_BALANCING |
                         NODE_PRIVATE_POLICY_WRITE_FENCE)

Enabled 'numa balancing' and 'demotion'.
   $ echo 1 > /proc/sys/kernel/numa_balancing
   $ echo true > /sys/kernel/mm/numa/demotion_enabled

========== TaoBench Experiments ====================

CMD:
$ python3 benchpress_cli.py run tao_bench_standalone -i '{"bind_mem": 0, 
"bind_cpu": 0, "memsize": 32, "test_time": 300, "warmup_time": 0, 
"clients_per_thread": 100, "set_get_ratio": "1:9"}'

1. (baseline) DRAM 16GB + CXL 32GB   ==>

$ grep -A8 "^ALL STATS" benchmark_metrics_27cfff0e/client_0.log

===================================================
Type  Avg. Latency    p50        p95        p99
----------------------------------------------------
Sets   2.75215     2.73500      4.57500     5.79100
Gets   2.84236     2.81500      4.76700     5.95100

  2. (private node in default write protected path)
      DRAM 16GB + CXL Private Node(uncompressed) 32GB   ===>

$ grep -A8 "^ALL STATS" benchmark_metrics_5181fda5/client_0.log

===================================================
Type  Avg. Latency    p50        p95        p99
----------------------------------------------------
Sets   5.58395     4.73500      12.35100     21.75900
Gets   4.83344     4.19100      10.43100     17.91900

Vmstat metrics comparison of both cases for Taobench
($cat /proc/vmstat):

-------------------------------------------------------------------
Metric           |  Baseline    |   CXL Private Node |   Ratio
-------------------------------------------------------------------
pgfault            37,121,808       171,512,616          4.6x
pgactivate          5,360,928       161,947,594         30.2x
pgreuse             4,346,396           407,397         10.7x less
pgrefill           88,775,854       436,486,518          4.9x
pgdemote_kswapd     5,933,439       129,586,750         21.8x
pgdemote_direct        24,837        35,505,114       1429x
allocstall_normal         469           153,600        327x
kswapd_low_wmark_hit_quickly 671         27,821         41.5x
pageoutrun              1,333            29,217         21.9x
numa_pte_updates   24,508,077                 0          na
numa_hint_faults   21,152,264                 0          na
numa_pages_migrated 3,330,897                 0          na
numa_hit           46,667,202       353,812,703          7.6x

---------- Takeaways -----

Private (CRAM) node performance is lower compared to a normal CXL memory 
allocation path. VM stats shows more page_faults and allocation_stalls 
on the private node case. Could it be the allocator waits for migration 
path to demote pages to private node? Or the actual hot pages in dram 
got demoted to private node to make space during overwrites? We will try 
further analysis on this.

========== MLC Experiments ====================

CMD:
   $./mlc --loaded_latency -j0 -c0 -b1g -k1-15 -W5 -r

Experiment Results:

1. (Baseline) DRAM 16GB + CXL 32GB

Inject  Latency Bandwidth
Delay   (ns)    MB/sec
================
  00000  686.18   40137.5
  00002  686.69   40124.0
  00008  709.74   40362.7
  00015  711.75   40350.0
  00050  710.72   40357.4
  00100  712.22   40316.0
  00200  695.29   40026.5
  00300  602.91   38845.3
  00400  482.28   36240.4
  00500  410.03   33606.6
  00700  275.13   26352.5
  01000  241.82   18851.7
  01300  230.91   14695.8
  01700  215.27   11403.2
  02500  217.76    7900.9
  03500  194.42    5787.0
  05000  192.17    4166.4
  09000  190.36    2475.0
  20000  188.09    1312.1

2. (private node in default write protected path)
     DRAM 16GB + CXL Private Node(Uncompressed) 32GB

Inject  Latency Bandwidth
Delay   (ns)    MB/sec
================
  00000  320.09   12186.2
  00002  543.42   22726.1
  00008  482.11   28385.2
  00015  488.26   26304.6
  00050  485.19   26898.4
  00100  477.65   24668.9
  00200  483.68   21273.8
  00300  486.44   17655.9
  00400  480.41   16470.0
  00500  481.30   13589.9
  00700  493.84    9960.8
  01000  488.66    8267.7
  01300  491.31    6433.6
  01700  476.97    6211.2
  02500  472.97    5037.0
  03500  466.05    3533.8
  05000  446.44    2783.1
  09000  417.03    1877.5
  20000  387.22    1035.3

Vmstat metrics comparison of both cases for MLC
($cat /proc/vmstat):

-------------------------------------------------------------------
Metric           |  Baseline    |   CXL Private Node |   Ratio
-------------------------------------------------------------------
pgfault           10,287,329         46540,782           4.6x
pgactivate           1141468        65,098,272          57x
pgrefill             260,699        47,592,751         182.5x
pgdemote_kswapd    1,085,783        26,063,754          24x
pgdemote_direct    4,562,539        15,809,485           3.5x
pgmigrate_fail        41,929         2,763,725          65.9x
allocstall_normal        161               425           2.6x
kswapd_low_wmark_hit_quickly  47           763          16.2x
pageoutrun                55               936          17x

Takeaway: Our MLC tests show a trade-off between the two configurations. 
During low inject delay, the private Node is faster. But when inject 
delay goes high, the baseline shows better latency. Could it be the TLB 
cache effects?

--------------------------------------------------------------------
Case 3 dmesg log snippet:

[   82.525741] page: refcount:1 mapcount:0 mapping:0000000000000000 
index:0x0 pfn:0x5f5800
[   82.525753] flags: 
0x17ffffc0002000(reserved|node=0|zone=2|lastcpupid=0x1fffff)
[   82.525762] raw: 0017ffffc0002000 ffefc85717d60008 ffefc85717d60008 
0000000000000000
[   82.525764] raw: 0000000000000000 0000000000000000 00000001ffffffff 
0000000000000000
[   82.525766] page dumped because: unmovable page
..............

We will update on the compression enabled experiments further.

~ Arun

On 23-07-2026 10:23 pm, Gregory Price wrote:
> On Thu, Jul 23, 2026 at 02:08:31PM +0530, Arun George/Arun George wrote:
>> On 21-07-2026 01:03 am, Gregory Price wrote:
>> We intend to test this series on a compression capable CXL expander
>> hardware. Since compressed ram example (cram) is not part of this
>> series, how do you suggest to do that? Do you have a version of cram
>> module compatible with this series?
>>
>> ~Arun
> 
> Hi Arun,
> > I plan to RFC the new setup for compressed ram this a bit later, but I
> will share my working branch with you for testing.
> 
> ~Gregory