From: James Houghton <jthoughton@google.com>
To: Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>,
Muchun Song <muchun.song@linux.dev>,
Oscar Salvador <osalvador@suse.de>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>,
Nikos Nikoleris <nikos.nikoleris@arm.com>,
Linu Cherian <linu.cherian@arm.com>,
Mark Rutland <mark.rutland@arm.com>,
David Hildenbrand <david@kernel.org>,
Ryan Roberts <ryan.roberts@arm.com>,
Nanyong Sun <sunnanyong@huawei.com>, Yu Zhao <yuzhao@google.com>,
Frank van der Linden <fvdl@google.com>,
David Rientjes <rientjes@google.com>,
James Houghton <jthoughton@google.com>,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org
Subject: [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test
Date: Sun, 22 Nov 2026 18:44:48 +0000 [thread overview]
Message-ID: <20261122184453.2668189-15-jthoughton@google.com> (raw)
In-Reply-To: <20261122184453.2668189-1-jthoughton@google.com>
Add a script that stresses HugeTLB vmemmap optimization (HVO), in
particular on architectures that update the vmemmap in place while it
may be concurrently accessed:
- Check that optimizing N folios frees exactly N * (vmemmap pages per
folio - 1) vmemmap pages (per nr_memmap_pages and
nr_memmap_boot_pages), and that restoring them gives them back.
- Repeatedly optimize and restore folios by resizing the hugepage pool,
while concurrently reading struct pages through /proc/kpageflags and
/proc/kpagecount, compacting memory, and optionally reading
page_owner and offlining/onlining memory blocks.
- Do the same with fail_hugetlb_vmemmap_pte fault injection enabled,
if available, to exercise the rollback and partially-optimized folio
paths.
After each phase, the pool must shrink back to its original size and the
memmap accounting must return to its baseline. Finally, the kernel log
must not contain warnings or oopses.
Signed-off-by: James Houghton <jthoughton@google.com>
---
tools/testing/selftests/mm/Makefile | 2 +
.../selftests/mm/hugetlb_vmemmap_stress.sh | 380 ++++++++++++++++++
.../selftests/mm/ksft_hugetlb_vmemmap.sh | 4 +
tools/testing/selftests/mm/run_vmtests.sh | 4 +
4 files changed, 390 insertions(+)
create mode 100755 tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
create mode 100755 tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index 675ec57d47de..9ec348902743 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -150,6 +150,7 @@ TEST_PROGS += ksft_cow.sh
TEST_PROGS += ksft_gup_test.sh
TEST_PROGS += ksft_hmm.sh
TEST_PROGS += ksft_hugetlb.sh
+TEST_PROGS += ksft_hugetlb_vmemmap.sh
TEST_PROGS += ksft_hugevm.sh
TEST_PROGS += ksft_kmemleak_confirm.sh
TEST_PROGS += ksft_kmemleak_dedup.sh
@@ -182,6 +183,7 @@ TEST_FILES += test_hmm.sh
TEST_FILES += va_high_addr_switch.sh
TEST_FILES += charge_reserved_hugetlb.sh
TEST_FILES += hugetlb_reparenting_test.sh
+TEST_FILES += hugetlb_vmemmap_stress.sh
TEST_FILES += test_page_frag.sh
TEST_FILES += run_vmtests.sh
diff --git a/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
new file mode 100755
index 000000000000..b64fdb3ed761
--- /dev/null
+++ b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
@@ -0,0 +1,380 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Stress test for HugeTLB vmemmap optimization (HVO).
+#
+# Phases:
+# 1. accounting: allocate N hugepages, check that nr_memmap_pages +
+# nr_memmap_boot_pages drops by exactly N * (vmemmap pages per folio - 1),
+# then free them and check it returns to the baseline.
+# 2. stress: churn the pool (optimize/restore) while concurrently reading
+# struct pages (/proc/kpageflags, /proc/kpagecount), compacting memory,
+# and optionally reading page_owner and offlining/onlining memory.
+# 3. pte-inject: like 2, with fail_hugetlb_vmemmap_pte enabled; this hits
+# both the optimize rollback and the restore (partial HVO) paths.
+#
+# After each phase the pool is drained back to its original size, memmap
+# accounting must be back at the phase's baseline, and the kernel log must
+# not contain warnings/oopses.
+#
+# The fault-injection phases require CONFIG_FAIL_HUGETLB_VMEMMAP and
+# CONFIG_FAULT_INJECTION_DEBUG_FS, and are skipped otherwise.
+
+set -u
+
+KSFT_PASS=0
+KSFT_FAIL=1
+KSFT_SKIP=4
+
+size_kb=
+nr=16
+duration=60
+prob=20
+readers=4
+struct_page_size=64
+do_page_owner=0
+do_hotplug=0
+
+usage() {
+ cat <<EOF
+Usage: $0 [options]
+ -s KB hugepage size in kB (default: default hugepage size)
+ -n N number of hugepages to churn (default: $nr)
+ -t SEC duration of each stress phase (default: $duration)
+ -p PCT fault-injection probability in percent (default: $prob)
+ -j N number of struct page reader processes (default: $readers)
+ -S BYTES sizeof(struct page) (default: $struct_page_size)
+ -o also read /sys/kernel/debug/page_owner during stress
+ -m also offline/online memory blocks during stress (dissolves
+ free hugepages, exercising the restore path)
+
+For exact accounting checks, run on a hugepage size whose pool is
+initially empty (e.g. no boot-time reservations of that size).
+EOF
+ exit $KSFT_SKIP
+}
+
+while getopts "s:n:t:p:j:S:omh" opt; do
+ case $opt in
+ s) size_kb=$OPTARG ;;
+ n) nr=$OPTARG ;;
+ t) duration=$OPTARG ;;
+ p) prob=$OPTARG ;;
+ j) readers=$OPTARG ;;
+ S) struct_page_size=$OPTARG ;;
+ o) do_page_owner=1 ;;
+ m) do_hotplug=1 ;;
+ *) usage ;;
+ esac
+done
+
+log() { echo "# $*"; }
+skip() { echo "SKIP: $*"; exit $KSFT_SKIP; }
+
+failures=0
+fail() { echo "FAIL: $*"; failures=$((failures + 1)); }
+pass() { echo "PASS: $*"; }
+
+[ "$(id -u)" -eq 0 ] || skip "must be run as root"
+
+[ -r /proc/sys/vm/hugetlb_optimize_vmemmap ] ||
+ skip "HVO not supported (no vm.hugetlb_optimize_vmemmap)"
+[ "$(cat /proc/sys/vm/hugetlb_optimize_vmemmap)" = 1 ] ||
+ skip "HVO disabled (vm.hugetlb_optimize_vmemmap=0)"
+grep -q '^nr_memmap_pages ' /proc/vmstat ||
+ skip "no nr_memmap_pages in /proc/vmstat"
+
+[ -n "$size_kb" ] || size_kb=$(awk '/^Hugepagesize:/ {print $2}' /proc/meminfo)
+hp_dir=/sys/kernel/mm/hugepages/hugepages-${size_kb}kB
+[ -d "$hp_dir" ] || skip "no ${size_kb}kB hugepages"
+
+page_size=$(getconf PAGESIZE)
+vmemmap_pages=$(( size_kb * 1024 / page_size * struct_page_size / page_size ))
+freed_per_folio=$(( vmemmap_pages - 1 ))
+[ "$freed_per_folio" -gt 0 ] ||
+ skip "${size_kb}kB hugepages are not HVO-optimizable"
+
+dbgfs=/sys/kernel/debug
+mountpoint -q $dbgfs || mount -t debugfs none $dbgfs 2>/dev/null
+
+orig_nr=$(cat "$hp_dir/nr_hugepages")
+target_nr=$(( orig_nr + nr ))
+marker="hvo-stress-$$-$(date +%s)"
+tmpdir=$(mktemp -d)
+pids=()
+
+log "hugepage size ${size_kb}kB, page size $page_size"
+log "struct page size $struct_page_size"
+log "vmemmap pages/folio: $vmemmap_pages ($freed_per_folio freed by HVO)"
+log "pool: $orig_nr initially, churning between $orig_nr and $target_nr"
+[ "$orig_nr" -eq 0 ] ||
+ log "WARNING: pool not initially empty, accounting may be inexact"
+
+memmap_total() {
+ awk '/^nr_memmap_(boot_)?pages / {s += $2} END {print s + 0}' \
+ /proc/vmstat
+}
+
+set_nr() {
+ echo "$1" > "$hp_dir/nr_hugepages" 2>/dev/null
+ cat "$hp_dir/nr_hugepages"
+}
+
+# Shrink the pool back to orig_nr. Restore may transiently fail (e.g. with
+# fault injection active), so retry for a while.
+drain() {
+ local i cur surplus
+
+ for i in $(seq 20); do
+ # Pages whose vmemmap could not be restored are kept as free
+ # surplus pages, which shrinking nr_hugepages does not free.
+ # Writing the current size converts them back to persistent
+ # pages first.
+ set_nr "$(cat "$hp_dir/nr_hugepages")" > /dev/null
+ cur=$(set_nr "$orig_nr")
+ [ "$cur" -eq "$orig_nr" ] && return 0
+ sleep 1
+ done
+ surplus=$(cat "$hp_dir/surplus_hugepages")
+ fail "pool stuck at $cur hugepages ($surplus surplus)," \
+ "expected $orig_nr"
+ return 1
+}
+
+fa_dir() { echo "$dbgfs/$1"; }
+
+fa_enable() {
+ local d
+ d=$(fa_dir "$1")
+ echo 0 > "$d/verbose"
+ echo N > "$d/task-filter"
+ echo 1 > "$d/interval"
+ echo 1000000 > "$d/times"
+ echo "$prob" > "$d/probability"
+}
+
+# Disable and print the number of injected failures.
+fa_disable() {
+ local d left
+ d=$(fa_dir "$1")
+ echo 0 > "$d/probability"
+ left=$(cat "$d/times")
+ echo 0 > "$d/times"
+ echo $(( 1000000 - left ))
+}
+
+check_dmesg() {
+ local bad pat
+
+ pat='WARNING:|BUG[: ]|Oops|Unable to handle kernel'
+ pat+='|Internal error|KASAN:|UBSAN:'
+ pat+='|list_(add|del) corruption|page dumped because'
+ bad=$(dmesg | sed -n "/$marker/,\$p" | grep -E "$pat")
+ # Offlining a memory block that contains unmovable pages fails and
+ # dumps the offending page; that is expected with -m.
+ [ "$do_hotplug" -eq 1 ] &&
+ bad=$(echo "$bad" | grep -v 'page dumped because: unmovable page')
+ if [ -n "$bad" ]; then
+ fail "kernel log reports problems:"
+ echo "$bad" | head -20 | sed 's/^/# /'
+ fi
+}
+
+## Workers
+
+churn() {
+ while :; do
+ set_nr "$target_nr" > /dev/null
+ set_nr "$orig_nr" > /dev/null
+ done
+}
+
+kpage_reader() {
+ while :; do
+ dd if=/proc/kpageflags of=/dev/null bs=4M status=none
+ dd if=/proc/kpagecount of=/dev/null bs=4M status=none
+ done
+}
+
+compactor() {
+ while :; do
+ echo 1 > /proc/sys/vm/compact_memory
+ sleep 1
+ done
+}
+
+page_owner_reader() {
+ while :; do
+ cat $dbgfs/page_owner > /dev/null
+ done
+}
+
+hotplugger() {
+ local blk state removable
+
+ while :; do
+ for blk in /sys/devices/system/memory/memory*; do
+ state=$(cat "$blk/state" 2>/dev/null)
+ removable=$(cat "$blk/removable" 2>/dev/null || echo 1)
+ [ "$state" = online ] || continue
+ [ "$removable" = 1 ] || continue
+ echo "$blk" >> "$tmpdir/hotplug"
+ # A signal aborts a pending offline_pages(). Use
+ # --foreground so that the writer stays in the worker's
+ # process group and is killed by stop_workers().
+ timeout --foreground 10 \
+ sh -c "echo offline > $blk/state" 2>/dev/null
+ echo online > "$blk/state" 2>/dev/null
+ sleep 1
+ done
+ done
+}
+
+start_workers() {
+ local i
+
+ # Start each worker in its own process group (job control), so that
+ # stop_workers() can kill the commands it is running as well.
+ set -m
+ churn & pids+=($!)
+ for i in $(seq "$readers"); do
+ kpage_reader & pids+=($!)
+ done
+ compactor & pids+=($!)
+ if [ "$do_page_owner" -eq 1 ]; then
+ if [ -r $dbgfs/page_owner ]; then
+ page_owner_reader & pids+=($!)
+ else
+ log "page_owner not available, not reading it"
+ fi
+ fi
+ [ "$do_hotplug" -eq 1 ] && { hotplugger & pids+=($!); }
+ set +m
+}
+
+# Stop the workers, including any command they are running (e.g. a dd of
+# /proc/kpageflags), so that nothing touches struct pages after a phase ends.
+stop_workers() {
+ local pid i alive
+
+ [ "${#pids[@]}" -gt 0 ] || return 0
+ for pid in "${pids[@]}"; do
+ kill -TERM -- "-$pid" 2>/dev/null
+ done
+ for i in $(seq 30); do
+ alive=0
+ for pid in "${pids[@]}"; do
+ kill -0 -- "-$pid" 2>/dev/null && alive=1
+ done
+ [ "$alive" -eq 0 ] && break
+ sleep 1
+ done
+ if [ "$alive" -eq 1 ]; then
+ log "WARNING: workers still running, killing them"
+ for pid in "${pids[@]}"; do
+ kill -KILL -- "-$pid" 2>/dev/null
+ done
+ fi
+ # Reap the workers only once they are gone, so that this cannot block.
+ wait "${pids[@]}" 2>/dev/null
+ pids=()
+}
+
+cleanup() {
+ local blk t
+
+ stop_workers
+ for t in fail_hugetlb_vmemmap_pte; do
+ [ -d "$(fa_dir $t)" ] && fa_disable $t > /dev/null
+ done
+ if [ -f "$tmpdir/hotplug" ]; then
+ sort -u "$tmpdir/hotplug" | while read -r blk; do
+ [ "$(cat "$blk/state")" = online ] ||
+ echo online > "$blk/state" 2>/dev/null
+ done
+ fi
+ # drain() also frees free surplus pages left by failed restores.
+ drain > /dev/null
+ rm -rf "$tmpdir"
+}
+trap cleanup EXIT
+trap 'exit $KSFT_FAIL' INT TERM
+
+# Run a stress phase. $1: name, $2: fault attr to enable ("" for none).
+stress_phase() {
+ local name=$1 fa=$2 base after injected
+
+ if [ -n "$fa" ] && [ ! -d "$(fa_dir "$fa")" ]; then
+ echo "SKIP: $name (no $(fa_dir "$fa"))"
+ return
+ fi
+
+ log "phase $name: ${duration}s"
+ base=$(memmap_total)
+ [ -n "$fa" ] && fa_enable "$fa"
+ start_workers
+ sleep "$duration"
+ stop_workers
+ if [ -n "$fa" ]; then
+ injected=$(fa_disable "$fa")
+ log "$name: injected $injected failures"
+ [ "$injected" -gt 0 ] ||
+ log "WARNING: $name: no failures injected"
+ fi
+
+ drain || return
+ after=$(memmap_total)
+ if [ "$after" -ne "$base" ]; then
+ fail "$name: memmap pages $after after drain, expected $base"
+ else
+ pass "$name"
+ fi
+}
+
+accounting_phase() {
+ local base got added after expect
+
+ log "phase accounting"
+ base=$(memmap_total)
+ got=$(set_nr "$target_nr")
+ added=$(( got - orig_nr ))
+ if [ "$added" -le 0 ]; then
+ fail "accounting: could not allocate any ${size_kb}kB hugepages"
+ return
+ fi
+ [ "$added" -eq "$nr" ] || log "accounting: only allocated $added of $nr"
+
+ after=$(memmap_total)
+ expect=$(( base - added * freed_per_folio ))
+ if [ "$after" -ne "$expect" ]; then
+ fail "accounting: memmap pages $after after optimizing" \
+ "$added folios, expected $expect (baseline $base)"
+ else
+ pass "accounting: optimize freed $(( base - after ))" \
+ "vmemmap pages"
+ fi
+
+ drain || return
+ after=$(memmap_total)
+ if [ "$after" -ne "$base" ]; then
+ fail "accounting: memmap pages $after after restore," \
+ "expected $base"
+ else
+ pass "accounting: restore"
+ fi
+}
+
+echo "$marker" > /dev/kmsg
+
+accounting_phase
+stress_phase stress ""
+stress_phase pte-inject fail_hugetlb_vmemmap_pte
+
+check_dmesg
+
+if [ "$failures" -ne 0 ]; then
+ echo "FAILED: $failures check(s)"
+ exit $KSFT_FAIL
+fi
+echo "OK"
+exit $KSFT_PASS
diff --git a/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh b/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh
new file mode 100755
index 000000000000..905b75b2cfb4
--- /dev/null
+++ b/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: GPL-2.0
+
+./run_vmtests.sh -t hugetlb_vmemmap
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 973cb715b326..f439affe5d70 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -79,6 +79,8 @@ separated by spaces:
test transparent huge pages
- hugetlb
test hugetlbfs huge pages
+- hugetlb_vmemmap
+ test the hugetlb vmemmap optimization
- migration
invoke move_pages(2) to exercise the migration entry code
paths in the kernel
@@ -327,6 +329,8 @@ echo "$enable_soft_offline" > /proc/sys/vm/enable_soft_offline
CATEGORY="hugetlb" run_test ./hugetlb-read-hwpoison
fi
+CATEGORY="hugetlb_vmemmap" run_test ./hugetlb_vmemmap_stress.sh
+
if [ $VADDR64 -ne 0 ]; then
# va high address boundary switch test
CATEGORY="hugevm" run_test bash ./va_high_addr_switch.sh
--
2.56.1.809.ga0b61035b5-goog
next prev parent reply other threads:[~2026-11-22 18:45 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
2026-11-22 18:44 ` [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure James Houghton
2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
2026-11-22 18:44 ` [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization James Houghton
2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
2026-11-22 18:44 ` James Houghton [this message]
2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122184453.2668189-15-jthoughton@google.com \
--to=jthoughton@google.com \
--cc=akpm@linux-foundation.org \
--cc=catalin.marinas@arm.com \
--cc=david@kernel.org \
--cc=fvdl@google.com \
--cc=linu.cherian@arm.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mark.rutland@arm.com \
--cc=muchun.song@linux.dev \
--cc=nikos.nikoleris@arm.com \
--cc=osalvador@suse.de \
--cc=rientjes@google.com \
--cc=ryan.roberts@arm.com \
--cc=sunnanyong@huawei.com \
--cc=wangkefeng.wang@huawei.com \
--cc=will@kernel.org \
--cc=yuzhao@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®