Memory fragmentation and compaction test and evaluation

Memory fragmentation and compaction test and evaluation

[Karim Manaouil]

Hi folks,

With mTHP/THP, large block sizes, and increasing adoption of larger
pages in various subsystems (e.g. drm/ttm [1]), there is an increasing
pressure on physical memory contiguity and hence anti-fragmentation,
defragmentation and compaction. This brings attention to revisiting
existing mechanisms in the kernel, especially with the rapidly changing
hardware landscape in terms of memory technology, memory bandwidth,
CPU microarchitecture, CPU core count, interconnects and other factors.

However, it turns out it's extremely hard to evaluate fragmentation,
compaction and huge page availability. The main challenge rests on
reliable reproduction. We need tools and workloads. Specifically,
we need tools that:

	1. Reliably reproduce fragmentation on a system. Preferably
	   with varying degrees and some control over movable vs unmovable.

	2. We need tools/workloads that continously stress the system
	   and continously introduce fragmentation and challenge large
	   block allocation while other workloads are running,
	   preferably without interference between the two.

For memory fragmentation, a famous synthetic way that is widely adopted in
previous patches is allocating massive amout of anonymous/file memory while
an I/O metadata intensive workload is running on paralle to interleave
unmovable slab allocations from the filesystem with movable allocations, for
example by running "find /" or running fio in parallel as is the case with
mmtests/thpchallenge. This addresses point (1) to some extent.

Point (2) remains entirely un-addressed.

It is also important to have real world workloads.

I would love to see how other people address these challenges and how
people here test and evaluate fragmentation and compaction, especially
under sustained pressure. We can always start from a fragmented system
(for example mmtests/thpchallenge way) and run a workload to completion,
but that only tests how the system recovers initially and not how it
copes with continous stress.

Please share you experiences in this thread and share any workloads and
ideas for tools that you think could help in these sorts of development.

[1] https://lore.kernel.org/linux-mm/20260506033300.3534883-1-matthew.brost@intel.com/

Thanks
-- 
~karim