[PATCH 00/10] mm/madvise: refactor cold and pageout page table walks

[PATCH 00/10] mm/madvise: refactor cold and pageout page table walks

[Gregory Price]

This code makes Lorenzo and Andrew sad every time I fix a bug in it,
and there's been [1,2,3,4] bugs of various severety in a single
userland reachable function in a single month.

Unfortunately this code is an unreviewable nest of... nesting.

Lets fix that.

The self tests are not intended to be merged en-masse, they are to
ensure the implementation details have not changed during the
refactor so that we may tear the refactored code apart for even
more nasty stuff.

The self tests originated as 28 individual commits, but it's been
truncated to a single commit for the first go around to reduce
noise / pressure on folks inboxes.

Some of the tests may even surprise and confuse!
Good! The code is confusing!

[1] https://lore.kernel.org/linux-mm/20260912110832.3203902-1-gourry@gourry.net/
[2] https://lore.kernel.org/linux-mm/20260912110540.3203010-1-gourry@gourry.net/
[3] https://lore.kernel.org/linux-mm/20260912034833.2952750-3-gourry@gourry.net/
[4] https://lore.kernel.org/linux-mm/20260817220810.1175596-3-gourry@gourry.net/

===
actual cover letter
===

MADV_COLD and MADV_PAGEOUT share a page-table walker that handles huge
PMDs, PTE-mapped large folios, ordinary PTEs, folio splitting and
deferred PAGEOUT reclaim in one function. The resulting control flow
makes page-table lock ownership and split retries difficult to audit.

This series adds 28 focused selftests before changing the implementation.
They cover full and partial advice at PMD and PTE granularity, shared and
pinned folios, active and unevictable folios, zero pages, swap entries, PTE
holes, file-page permissions, concurrent split attempts and NUMA migration.

The implementation separates huge-PMD handling, PTE batching, PTE-lock
ownership and shared walk setup. The existing periodic scheduling point is
preserved through the outer lock-ownership loop. Helpers shared only by
MADV_COLD and MADV_PAGEOUT use the madvise_lru prefix.

Some test data:

All 28 selftests passed before and after the refactor.  Some of the race
conditions are flakey due to being race conditions.

LTP madvise01 passed all 20 cases and process_madvise01 passed on both
implementations.

GCov data of ktests before and after refactor:
                         executable lines   branch edges taken
  before rewrite          179/181 (98.9%)    106/120 (88.3%)
  after rewrite           168/168 (100%)     112/118 (94.9%)

The last 6 branch edges remain untaken:
  - a large anonymous COW folio in an unauthorized private-file VMA
  - ZONE_DEVICE folios at PTE and PMD level
    (note that PMD zone_device is unreachable)
  - the warning edge for an invalid PMD softleaf
  - a pending fatal signal.

Gregory Price (10):
  selftests/mm: exercise MADV_COLD and MADV_PAGEOUT
  mm/madvise: name the shared LRU PMD callback
  mm/madvise: factor shared LRU folio handling
  mm/madvise: use the PMD softleaf validity helper
  mm/madvise: factor huge-PMD folio processing
  mm/madvise: separate huge PMDs from the PTE walk
  mm/madvise: separate PTE-batch folio processing
  mm/madvise: separate the PTL-held PTE scan
  mm/madvise: make cold and pageout PTE lock ownership explicit
  mm/madvise: share cold and pageout walk setup

 mm/madvise.c                               |  462 +++---
 tools/testing/selftests/mm/Makefile        |    2 +
 tools/testing/selftests/mm/ksft_madvise.sh |    4 +
 tools/testing/selftests/mm/madvise.c       | 1673 ++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |    5 +
 tools/testing/selftests/mm/vm_util.c       |    4 +-
 tools/testing/selftests/mm/vm_util.h       |    6 +
 7 files changed, 1934 insertions(+), 222 deletions(-)
 create mode 100755 tools/testing/selftests/mm/ksft_madvise.sh
 create mode 100644 tools/testing/selftests/mm/madvise.c

-- 
2.53.0-Meta

base-commit: 854157fc9d011b8dd5a1a86a237447e28d681eb9

[PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[Gregory Price]

MADV_COLD and MADV_PAGEOUT share a page-table walker covering ordinary
PTEs, PTE-mapped large folios and huge PMDs. Existing selftests provide
little coverage of its range, folio-state and permission decisions.

Add 28 cases ahead of a planned walker refactor. All cases check syscall
results and mapping contents, but they deliberately have different review
contracts.

This commit was originally 28 individual commits, but was squashed for
the initial RFC to limit noise during initial review. The question is
what set of self-tests should retain.

The eight user-visible contract tests verify:

  - MADV_COLD does not populate a PTE hole;
  - COLD and PAGEOUT reject a locked VMA;
  - full-range PAGEOUT swaps a PMD-mapped THP without corrupting it;
  - MADV_COLD does not fault in a swapped PTE;
  - PAGEOUT preserves an unevictable base page;
  - authorized file PAGEOUT evicts a clean file page;
  - unauthorized shared-file PAGEOUT leaves the page resident; and
  - private-file PAGEOUT reclaims a COW page but preserves a file page.

The eighteen implementation-detail tests record the current folio and
page-table behavior:

  - full and partial MADV_COLD on a PTE-mapped THP;
  - full, repeated and partial MADV_COLD on PMD-mapped THPs;
  - MADV_COLD on the shared zero page and huge zero page;
  - partial MADV_COLD on a shared PMD-mapped THP;
  - partial MADV_COLD on pinned PMD- and PTE-mapped THPs;
  - parallel partial MADV_COLD split attempts;
  - PAGEOUT on an unevictable THP;
  - MADV_COLD on active base-page and huge-PMD folios;
  - partial PAGEOUT on PMD- and PTE-mapped THPs;
  - partial and full MADV_COLD on shared PTE-mapped THPs; and
  - PAGEOUT filtering of a PMD-sized protected file folio.

These tests intentionally assert splitting, PMD/PTE mapping shape, folio
flags or folio sharing. They may need adjustment after a valid kernel
implementation change. Their purpose here is to detect unintended changes
during the refactor.

Two additional stress tests use implementation-specific setup while keeping
contract-level assertions:

  - MADV_COLD on a folio still queued in another CPU's LRU batch
  - MADV_PAGEOUT during NUMA migration of unevictable shmem THPs.

The stress tests require only successful calls and preserved contents.
They do not assert which internal path won. Tests requiring root, swap,
THP allocation, multiple CPUs or NUMA nodes report a skip when their
prerequisites are unavailable.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 tools/testing/selftests/mm/Makefile        |    2 +
 tools/testing/selftests/mm/ksft_madvise.sh |    4 +
 tools/testing/selftests/mm/madvise.c       | 1673 ++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |    5 +
 tools/testing/selftests/mm/vm_util.c       |    4 +-
 tools/testing/selftests/mm/vm_util.h       |    6 +
 6 files changed, 1692 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/mm/ksft_madvise.sh
 create mode 100644 tools/testing/selftests/mm/madvise.c

diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index beacc0f873049..8912aa85ca273 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -69,6 +69,7 @@ TEST_GEN_FILES += hugetlb-shm
 TEST_GEN_FILES += hugetlb-soft-offline
 TEST_GEN_FILES += khugepaged
 TEST_GEN_FILES += madv_populate
+TEST_GEN_FILES += madvise
 TEST_GEN_FILES += map_fixed_noreplace
 TEST_GEN_FILES += map_populate
 ifneq (,$(filter $(ARCH),arm64 riscv riscv64 x86 x86_64 loongarch32 loongarch64))
@@ -155,6 +156,7 @@ TEST_PROGS += ksft_kmemleak_dedup.sh
 TEST_PROGS += ksft_ksm.sh
 TEST_PROGS += ksft_ksm_numa.sh
 TEST_PROGS += ksft_madv_guard.sh
+TEST_PROGS += ksft_madvise.sh
 TEST_PROGS += ksft_madv_populate.sh
 TEST_PROGS += ksft_memfd_secret.sh
 TEST_PROGS += ksft_memory_failure.sh
diff --git a/tools/testing/selftests/mm/ksft_madvise.sh b/tools/testing/selftests/mm/ksft_madvise.sh
new file mode 100755
index 0000000000000..29cbbca800f2c
--- /dev/null
+++ b/tools/testing/selftests/mm/ksft_madvise.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: GPL-2.0
+
+./run_vmtests.sh -t madvise
diff --git a/tools/testing/selftests/mm/madvise.c b/tools/testing/selftests/mm/madvise.c
new file mode 100644
index 0000000000000..abde0b1c8a42e
--- /dev/null
+++ b/tools/testing/selftests/mm/madvise.c
@@ -0,0 +1,1673 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+
+#include <errno.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <sched.h>
+#include <stdbool.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <linux/mman.h>
+#include <linux/mempolicy.h>
+#include <sys/ipc.h>
+#include <sys/mman.h>
+#include <sys/shm.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/uio.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "kselftest.h"
+#include "vm_util.h"
+
+#ifndef MADV_COLLAPSE
+#define MADV_COLLAPSE 25
+#endif
+
+#define NR_CONCURRENT_THREADS 8
+#define NR_CONCURRENT_THPS 8
+#define NR_CONCURRENT_ROUNDS 400
+#define NR_MIGRATION_RACE_THPS 16
+#define NR_MIGRATION_RACE_ROUNDS 20
+#define NUMA_MASK_BITS 1024
+#define NUMA_MASK_LONGS (NUMA_MASK_BITS / (8 * sizeof(unsigned long)))
+
+static pthread_barrier_t concurrent_start_barrier;
+static pthread_barrier_t concurrent_done_barrier;
+static atomic_int concurrent_worker_errno;
+static char *concurrent_area;
+static size_t pmd_size;
+
+#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \
+	defined(SYS_migrate_pages)
+struct migration_pageout_data {
+	pthread_barrier_t start_barrier;
+	atomic_bool stop;
+	atomic_int calls;
+	atomic_int error;
+	char *mapping;
+	size_t size;
+};
+
+static bool find_two_memory_nodes(unsigned long *mask, int *node1, int *node2)
+{
+	int node;
+
+	if (syscall(SYS_get_mempolicy, NULL, mask, NUMA_MASK_BITS + 1, NULL,
+		    MPOL_F_MEMS_ALLOWED))
+		return false;
+
+	*node1 = *node2 = -1;
+	for (node = 0; node < NUMA_MASK_BITS; node++) {
+		if (!(mask[node / (8 * sizeof(*mask))] &
+		      (1UL << (node % (8 * sizeof(*mask))))))
+			continue;
+		if (*node1 < 0) {
+			*node1 = node;
+		} else {
+			*node2 = node;
+			return true;
+		}
+	}
+
+	return false;
+}
+
+static void numa_mask_set(unsigned long *mask, int node)
+{
+	mask[node / (8 * sizeof(*mask))] |=
+		1UL << (node % (8 * sizeof(*mask)));
+}
+
+static void *madvise_pageout_worker(void *arg)
+{
+	struct migration_pageout_data *data = arg;
+
+	pthread_barrier_wait(&data->start_barrier);
+	while (!atomic_load_explicit(&data->stop, memory_order_relaxed)) {
+		if (madvise(data->mapping, data->size, MADV_PAGEOUT) &&
+		    errno != EAGAIN) {
+			atomic_store(&data->error, errno);
+			break;
+		}
+		atomic_fetch_add(&data->calls, 1);
+	}
+
+	return NULL;
+}
+
+static int migrate_between_nodes(unsigned long *old_nodes,
+				 unsigned long *new_nodes)
+{
+	int i;
+
+	for (i = 0; i < NR_MIGRATION_RACE_ROUNDS; i++) {
+		unsigned long *tmp;
+
+		if (syscall(SYS_migrate_pages, 0, NUMA_MASK_BITS + 1,
+			    old_nodes, new_nodes) < 0)
+			return errno;
+		tmp = old_nodes;
+		old_nodes = new_nodes;
+		new_nodes = tmp;
+	}
+
+	return 0;
+}
+
+static int race_pageout_with_migration(char *mapping, size_t size,
+				       unsigned long *mask1,
+				       unsigned long *mask2)
+{
+	struct migration_pageout_data data = {
+		.mapping = mapping,
+		.size = size,
+	};
+	pthread_t thread;
+	int ret;
+
+	ret = pthread_barrier_init(&data.start_barrier, NULL, 2);
+	if (ret)
+		return ret;
+	ret = pthread_create(&thread, NULL, madvise_pageout_worker, &data);
+	if (ret) {
+		pthread_barrier_destroy(&data.start_barrier);
+		return ret;
+	}
+	pthread_barrier_wait(&data.start_barrier);
+
+	ret = migrate_between_nodes(mask1, mask2);
+	atomic_store(&data.stop, true);
+	pthread_join(thread, NULL);
+	pthread_barrier_destroy(&data.start_barrier);
+
+	if (ret)
+		return ret;
+	if (atomic_load(&data.error))
+		return atomic_load(&data.error);
+	return atomic_load(&data.calls) ? 0 : EIO;
+}
+#endif
+
+static char *map_aligned_pages(size_t size)
+{
+	char *mapping, *aligned;
+
+	mapping = mmap(NULL, size + pmd_size, PROT_READ | PROT_WRITE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+
+	aligned = (char *)(((uintptr_t)mapping + pmd_size - 1) &
+			   ~(pmd_size - 1));
+	if (aligned != mapping)
+		munmap(mapping, aligned - mapping);
+	if (aligned + size < mapping + size + pmd_size)
+		munmap(aligned + size,
+		       mapping + size + pmd_size - (aligned + size));
+
+	memset(aligned, 1, size);
+	return aligned;
+}
+
+/* MADV_COLLAPSE may fail transiently with EAGAIN. */
+static bool collapse_all(char *mapping, size_t size, int nr_hpages)
+{
+	int ret, retry;
+
+	for (retry = 0; retry < 10; retry++) {
+		ret = madvise(mapping, size, MADV_COLLAPSE);
+		if (!ret) {
+			if (check_huge_anon(mapping, size, nr_hpages, pmd_size))
+				return true;
+		} else if (errno != EAGAIN) {
+			return false;
+		}
+		usleep(10000);
+	}
+
+	return false;
+}
+
+static bool collapse_shmem(char *mapping, size_t size, int nr_hpages)
+{
+	int ret, retry;
+
+	for (retry = 0; retry < 10; retry++) {
+		ret = madvise(mapping, size, MADV_COLLAPSE);
+		if (!ret) {
+			if (check_huge_shmem(mapping, size, nr_hpages, pmd_size))
+				return true;
+		} else if (errno != EAGAIN) {
+			return false;
+		}
+		usleep(10000);
+	}
+
+	return false;
+}
+
+static char *map_unevictable_shmem(size_t size, int *shmid)
+{
+	char *reservation, *mapping, *aligned;
+
+	reservation = mmap(NULL, size + pmd_size, PROT_NONE,
+			   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (reservation == MAP_FAILED)
+		ksft_exit_fail_perror("mmap reservation");
+	aligned = (char *)(((uintptr_t)reservation + pmd_size - 1) &
+			   ~(pmd_size - 1));
+	munmap(reservation, size + pmd_size);
+
+	*shmid = shmget(IPC_PRIVATE, size, IPC_CREAT | 0600);
+	if (*shmid < 0)
+		ksft_exit_fail_perror("shmget");
+	if (shmctl(*shmid, SHM_LOCK, NULL)) {
+		shmctl(*shmid, IPC_RMID, NULL);
+		ksft_test_result_skip("could not lock a shmem segment\n");
+		return MAP_FAILED;
+	}
+	mapping = shmat(*shmid, aligned, 0);
+	if (mapping == (void *)-1)
+		ksft_exit_fail_perror("shmat");
+	if (mapping != aligned)
+		ksft_exit_fail_msg("shmat did not honor the aligned address\n");
+
+	return mapping;
+}
+
+static void unmap_unevictable_shmem(char *mapping, int shmid)
+{
+	shmctl(shmid, SHM_UNLOCK, NULL);
+	shmdt(mapping);
+	shmctl(shmid, IPC_RMID, NULL);
+}
+
+static char *map_aligned_file(int fd, size_t size, int flags)
+{
+	char *mapping, *aligned;
+
+	mapping = mmap(NULL, size + pmd_size, PROT_NONE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap reservation");
+	aligned = (char *)(((uintptr_t)mapping + pmd_size - 1) &
+			   ~(pmd_size - 1));
+	if (mmap(aligned, size, PROT_READ | PROT_WRITE, flags | MAP_FIXED,
+		 fd, 0) == MAP_FAILED)
+		ksft_exit_fail_perror("mmap file");
+	if (aligned != mapping)
+		munmap(mapping, aligned - mapping);
+	if (aligned + size < mapping + size + pmd_size)
+		munmap(aligned + size,
+		       mapping + size + pmd_size - (aligned + size));
+
+	return aligned;
+}
+
+static void split_pmd_mapping(char *mapping)
+{
+	const size_t page_size = getpagesize();
+
+	if (mprotect(mapping + page_size, page_size, PROT_READ) ||
+	    mprotect(mapping + page_size, page_size, PROT_READ | PROT_WRITE))
+		ksft_exit_fail_perror("mprotect");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("mprotect split the physical THP\n");
+}
+
+static void check_memory(char *mapping, size_t size)
+{
+	size_t offset;
+
+	for (offset = 0; offset < size; offset += getpagesize())
+		if (mapping[offset] != 1)
+			ksft_exit_fail_msg("memory changed at offset %zu\n", offset);
+}
+
+/*
+ * MADV_COLD on a full PTE-mapped THP must preserve the folio, while an
+ * operation on only half of it must split the folio. Neither operation may
+ * alter the mapping contents.
+ */
+static void test_pte_mapped_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+
+	split_pmd_mapping(mapping);
+	if (madvise(mapping, pmd_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("full MADV_COLD split a large folio\n");
+
+	if (madvise(mapping, pmd_size / 2, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
+		ksft_exit_fail_msg("partial MADV_COLD left a large folio\n");
+	check_memory(mapping, pmd_size);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("MADV_COLD handles a PTE-mapped THP\n");
+}
+
+/*
+ * A PTE walk must skip holes without populating them or overlooking the
+ * present pages on either side.
+ */
+static void test_madvise_cold_pte_hole(void)
+{
+	const size_t page_size = getpagesize();
+	const size_t size = 3 * page_size;
+	char *mapping;
+	int pagemap_fd;
+
+	mapping = mmap(NULL, size, PROT_READ | PROT_WRITE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+	mapping[0] = 1;
+	mapping[2 * page_size] = 1;
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+	if (pagemap_is_populated(pagemap_fd, mapping + page_size))
+		ksft_exit_fail_msg("PTE hole was populated before MADV_COLD\n");
+
+	if (madvise(mapping, size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (mapping[0] != 1 || mapping[2 * page_size] != 1)
+		ksft_exit_fail_msg("MADV_COLD changed populated pages\n");
+	if (pagemap_is_populated(pagemap_fd, mapping + page_size))
+		ksft_exit_fail_msg("MADV_COLD populated a PTE hole\n");
+
+	close(pagemap_fd);
+	munmap(mapping, size);
+	ksft_test_result_pass("MADV_COLD skips PTE holes\n");
+}
+
+/*
+ * A read fault on private anonymous memory may install the shared zero page.
+ * MADV_COLD must ignore that special PTE without replacing it or changing the
+ * mapping contents.
+ */
+static void test_madvise_cold_zero_page(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+	int pagemap_fd;
+
+	mapping = mmap(NULL, page_size, PROT_READ,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+	if (mapping[0])
+		ksft_exit_fail_msg("anonymous mapping is not zero-filled\n");
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+	if (!pagemap_is_populated(pagemap_fd, mapping))
+		ksft_exit_fail_msg("zero page is not populated\n");
+
+	if (madvise(mapping, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping))
+		ksft_exit_fail_msg("MADV_COLD changed the zero-page mapping\n");
+
+	close(pagemap_fd);
+	munmap(mapping, page_size);
+	ksft_test_result_pass("MADV_COLD skips the shared zero page\n");
+}
+
+/*
+ * A read fault may map the shared huge zero page with a PMD. MADV_COLD must
+ * ignore that special PMD without replacing it or changing the mapping.
+ */
+static void test_madvise_cold_huge_zero_page(void)
+{
+	char *reservation, *mapping;
+	unsigned long pfn;
+	uint64_t flags;
+	int pagemap_fd, kpageflags_fd;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to read page flags\n");
+		return;
+	}
+	reservation = mmap(NULL, 2 * pmd_size, PROT_NONE,
+			   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (reservation == MAP_FAILED)
+		ksft_exit_fail_perror("mmap reservation");
+	mapping = (char *)(((uintptr_t)reservation + pmd_size - 1) &
+			   ~(pmd_size - 1));
+	if (mmap(mapping, pmd_size, PROT_READ,
+		 MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) == MAP_FAILED)
+		ksft_exit_fail_perror("mmap huge zero page");
+	if (mapping != reservation)
+		munmap(reservation, mapping - reservation);
+	if (mapping + pmd_size < reservation + 2 * pmd_size)
+		munmap(mapping + pmd_size,
+		       reservation + 2 * pmd_size - (mapping + pmd_size));
+	if (madvise(mapping, pmd_size, MADV_HUGEPAGE))
+		ksft_exit_fail_perror("MADV_HUGEPAGE");
+	if (mapping[0])
+		ksft_exit_fail_msg("anonymous mapping is not zero-filled\n");
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
+	if (pagemap_fd < 0 || kpageflags_fd < 0)
+		ksft_exit_fail_perror("open page flags");
+	pfn = pagemap_get_pfn(pagemap_fd, mapping);
+	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags) ||
+	    !(flags & KPF_ZERO_PAGE)) {
+		close(kpageflags_fd);
+		close(pagemap_fd);
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("huge zero page is not available\n");
+		return;
+	}
+
+	if (madvise(mapping, pmd_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (mapping[0] || pagemap_get_pfn(pagemap_fd, mapping) != pfn)
+		ksft_exit_fail_msg("MADV_COLD changed the huge-zero-page mapping\n");
+
+	close(kpageflags_fd);
+	close(pagemap_fd);
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("MADV_COLD skips the huge zero page\n");
+}
+
+/*
+ * Neither hint is valid for a locked VMA: reclaiming it would violate the
+ * mlock contract, and merely aging it would serve no purpose.
+ */
+static void test_madvise_lru_locked_vma(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+
+	mapping = mmap(NULL, page_size, PROT_READ | PROT_WRITE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+	mapping[0] = 1;
+	if (mlock(mapping, page_size)) {
+		munmap(mapping, page_size);
+		ksft_test_result_skip("could not lock a page\n");
+		return;
+	}
+
+	errno = 0;
+	if (!madvise(mapping, page_size, MADV_COLD) || errno != EINVAL)
+		ksft_exit_fail_msg("MADV_COLD accepted a locked VMA\n");
+	errno = 0;
+	if (!madvise(mapping, page_size, MADV_PAGEOUT) || errno != EINVAL)
+		ksft_exit_fail_msg("MADV_PAGEOUT accepted a locked VMA\n");
+	if (mapping[0] != 1)
+		ksft_exit_fail_msg("madvise changed locked memory\n");
+
+	munlock(mapping, page_size);
+	munmap(mapping, page_size);
+	ksft_test_result_pass("COLD and PAGEOUT reject a locked VMA\n");
+}
+
+/*
+ * A full-range MADV_COLD operates directly on a huge PMD. Assert that aging
+ * the mapping preserves both the physical THP and its contents.
+ */
+static void test_full_pmd_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	if (madvise(mapping, pmd_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("full MADV_COLD split a PMD-mapped THP\n");
+	check_memory(mapping, pmd_size);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("full MADV_COLD preserves a PMD-mapped THP\n");
+}
+
+/*
+ * MADV_COLD is idempotent. A second request must handle an already-old huge
+ * PMD without splitting the THP or changing its contents.
+ */
+static void test_repeated_pmd_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	if (madvise(mapping, pmd_size, MADV_COLD) ||
+	    madvise(mapping, pmd_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("repeated MADV_COLD split a PMD-mapped THP\n");
+	check_memory(mapping, pmd_size);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("repeated MADV_COLD preserves a PMD-mapped THP\n");
+}
+
+/*
+ * MADV_COLD on half of a PMD-mapped THP must split the folio so the
+ * unadvised half is not aged as part of the THP.
+ */
+static void test_partial_pmd_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	if (madvise(mapping, pmd_size / 2, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
+		ksft_exit_fail_msg("partial MADV_COLD left a PMD-mapped THP\n");
+	check_memory(mapping, pmd_size);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("partial MADV_COLD splits a PMD-mapped THP\n");
+}
+
+static void pageout_half_thp(char *mapping)
+{
+	if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT))
+		ksft_exit_fail_perror("MADV_PAGEOUT");
+	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
+		ksft_exit_fail_msg("partial MADV_PAGEOUT left a large folio\n");
+	check_memory(mapping, pmd_size);
+}
+
+/*
+ * Full-range MADV_PAGEOUT can reclaim a PMD-mapped THP directly. Assert that
+ * the mapping is swapped without corrupting the folio contents.
+ */
+static void test_full_pmd_madvise_pageout(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+	int pagemap_fd, retry;
+	bool swapped = false;
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+
+	for (retry = 0; retry < 100; retry++) {
+		if (madvise(mapping, pmd_size, MADV_PAGEOUT))
+			ksft_exit_fail_perror("MADV_PAGEOUT");
+		if (pagemap_is_swapped(pagemap_fd, mapping)) {
+			swapped = true;
+			break;
+		}
+		usleep(10000);
+	}
+	if (swapped)
+		check_memory(mapping, pmd_size);
+
+	close(pagemap_fd);
+	munmap(mapping, pmd_size);
+	if (!swapped) {
+		ksft_test_result_skip("MADV_PAGEOUT did not swap the THP\n");
+		return;
+	}
+	ksft_test_result_pass("full MADV_PAGEOUT swaps a PMD-mapped THP\n");
+}
+
+/*
+ * A swapped PTE is non-present but not empty. MADV_COLD must skip the entry
+ * without faulting the page back in or changing its contents.
+ */
+static void test_madvise_cold_swapped_pte(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping = map_aligned_pages(page_size);
+	int pagemap_fd, retry;
+	bool swapped = false;
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+	for (retry = 0; retry < 100; retry++) {
+		if (madvise(mapping, page_size, MADV_PAGEOUT))
+			ksft_exit_fail_perror("MADV_PAGEOUT");
+		if (pagemap_is_swapped(pagemap_fd, mapping)) {
+			swapped = true;
+			break;
+		}
+		usleep(10000);
+	}
+	if (!swapped) {
+		close(pagemap_fd);
+		munmap(mapping, page_size);
+		ksft_test_result_skip("MADV_PAGEOUT did not swap the page\n");
+		return;
+	}
+
+	if (madvise(mapping, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!pagemap_is_swapped(pagemap_fd, mapping))
+		ksft_exit_fail_msg("MADV_COLD faulted in a swapped PTE\n");
+	if (mapping[0] != 1)
+		ksft_exit_fail_msg("swapped page contents changed\n");
+
+	close(pagemap_fd);
+	munmap(mapping, page_size);
+	ksft_test_result_pass("MADV_COLD skips a swapped PTE\n");
+}
+
+/*
+ * vmsplice() retains a reference to a THP in a pipe, preventing a partial
+ * MADV_COLD from splitting it. The advice must leave the folio intact while
+ * pinned, then split it normally after the pipe releases the reference.
+ */
+static void test_partial_pinned_madvise_cold(bool pte_mapped)
+{
+	const size_t page_size = getpagesize();
+	char *mapping = map_aligned_pages(pmd_size);
+	struct iovec iov = {
+		.iov_base = mapping,
+		.iov_len = page_size,
+	};
+	char *buffer;
+	int pipefd[2];
+	int retry;
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	if (pte_mapped)
+		split_pmd_mapping(mapping);
+	if (pipe(pipefd))
+		ksft_exit_fail_perror("pipe");
+	if (vmsplice(pipefd[1], &iov, 1, SPLICE_F_GIFT) != page_size) {
+		close(pipefd[0]);
+		close(pipefd[1]);
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("vmsplice could not retain a THP page\n");
+		return;
+	}
+
+	if (madvise(mapping, pmd_size / 2, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_COLD split a pinned THP\n");
+
+	buffer = malloc(page_size);
+	if (!buffer)
+		ksft_exit_fail_perror("malloc");
+	if (read(pipefd[0], buffer, page_size) != page_size)
+		ksft_exit_fail_perror("read pipe");
+	free(buffer);
+	close(pipefd[0]);
+	close(pipefd[1]);
+
+	for (retry = 0; retry < 10; retry++) {
+		if (madvise(mapping, pmd_size / 2, MADV_COLD))
+			ksft_exit_fail_perror("MADV_COLD");
+		if (check_large_folios(mapping, pmd_size, 0, pmd_size))
+			break;
+		usleep(10000);
+	}
+	if (retry == 10)
+		ksft_exit_fail_msg("MADV_COLD did not split an unpinned THP\n");
+	check_memory(mapping, pmd_size);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("partial MADV_COLD skips a pinned %s THP\n",
+			      pte_mapped ? "PTE-mapped" : "PMD-mapped");
+}
+
+static void test_partial_pinned_pmd_madvise_cold(void)
+{
+	test_partial_pinned_madvise_cold(false);
+}
+
+/*
+ * Pinning a PTE-mapped THP also prevents a partial MADV_COLD from splitting
+ * it. Once the pipe releases the reference, a serial retry must split it.
+ */
+static void test_partial_pinned_pte_madvise_cold(void)
+{
+	test_partial_pinned_madvise_cold(true);
+}
+
+static void *madvise_cold_worker(void *unused)
+{
+	int round, i;
+
+	(void)unused;
+	for (round = 0; round < NR_CONCURRENT_ROUNDS; round++) {
+		pthread_barrier_wait(&concurrent_start_barrier);
+		for (i = 0; i < NR_CONCURRENT_THPS; i++) {
+			if (madvise(concurrent_area + i * pmd_size,
+				    pmd_size / 2, MADV_COLD))
+				atomic_store(&concurrent_worker_errno, errno);
+		}
+		pthread_barrier_wait(&concurrent_done_barrier);
+	}
+
+	return NULL;
+}
+
+/*
+ * Concurrent partial MADV_COLD calls deliberately contend for each THP lock.
+ * Alternate PMD- and PTE-mapped rounds to exercise both split paths. A caller
+ * that loses the trylock race may skip that folio, so retry serially before
+ * asserting the stable interface: all calls succeed, the THPs remain
+ * splittable, and their contents are unchanged.
+ */
+static void test_concurrent_partial_madvise_cold(void)
+{
+	const size_t size = NR_CONCURRENT_THPS * pmd_size;
+	pthread_t threads[NR_CONCURRENT_THREADS];
+	int round, i;
+
+	concurrent_area = map_aligned_pages(size);
+	if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS)) {
+		munmap(concurrent_area, size);
+		ksft_test_result_skip("could not allocate PMD-sized THPs\n");
+		return;
+	}
+	if (pthread_barrier_init(&concurrent_start_barrier, NULL,
+				 NR_CONCURRENT_THREADS + 1) ||
+	    pthread_barrier_init(&concurrent_done_barrier, NULL,
+				 NR_CONCURRENT_THREADS + 1))
+		ksft_exit_fail_msg("pthread_barrier_init failed\n");
+	for (i = 0; i < NR_CONCURRENT_THREADS; i++)
+		if (pthread_create(&threads[i], NULL, madvise_cold_worker, NULL))
+			ksft_exit_fail_msg("pthread_create failed\n");
+
+	for (round = 0; round < NR_CONCURRENT_ROUNDS; round++) {
+		if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS))
+			ksft_exit_fail_msg("round %d: failed to form PMD THPs\n",
+					   round);
+		if (round & 1)
+			for (i = 0; i < NR_CONCURRENT_THPS; i++)
+				split_pmd_mapping(concurrent_area + i * pmd_size);
+		pthread_barrier_wait(&concurrent_start_barrier);
+		pthread_barrier_wait(&concurrent_done_barrier);
+
+		if (atomic_load(&concurrent_worker_errno)) {
+			errno = atomic_load(&concurrent_worker_errno);
+			ksft_exit_fail_perror("MADV_COLD");
+		}
+		for (i = 0; i < NR_CONCURRENT_THPS; i++) {
+			if (madvise(concurrent_area + i * pmd_size,
+				    pmd_size / 2, MADV_COLD))
+				ksft_exit_fail_perror("MADV_COLD retry");
+		}
+		if (!check_large_folios(concurrent_area, size, 0, pmd_size))
+			ksft_exit_fail_msg("round %d: PMD THP remained\n", round);
+	}
+	check_memory(concurrent_area, size);
+
+	for (i = 0; i < NR_CONCURRENT_THREADS; i++)
+		pthread_join(threads[i], NULL);
+	pthread_barrier_destroy(&concurrent_done_barrier);
+	pthread_barrier_destroy(&concurrent_start_barrier);
+	munmap(concurrent_area, size);
+	ksft_test_result_pass("concurrent partial MADV_COLD preserves memory\n");
+}
+
+/*
+ * SHM_LOCK makes a shmem folio unevictable without setting VM_LOCKED on this
+ * VMA. MADV_PAGEOUT must put an isolated folio back on its LRU rather than
+ * reclaiming it.
+ */
+static void test_madvise_pageout_unevictable(void)
+{
+	const size_t page_size = getpagesize();
+	unsigned long pfn;
+	uint64_t flags;
+	char *mapping;
+	int pagemap_fd, kpageflags_fd;
+	int shmid;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to read page flags\n");
+		return;
+	}
+	shmid = shmget(IPC_PRIVATE, page_size, IPC_CREAT | 0600);
+	if (shmid < 0)
+		ksft_exit_fail_perror("shmget");
+	if (shmctl(shmid, SHM_LOCK, NULL)) {
+		shmctl(shmid, IPC_RMID, NULL);
+		ksft_test_result_skip("could not lock a shmem segment\n");
+		return;
+	}
+	mapping = shmat(shmid, NULL, 0);
+	if (mapping == (void *)-1)
+		ksft_exit_fail_perror("shmat");
+	mapping[0] = 1;
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
+	if (pagemap_fd < 0 || kpageflags_fd < 0)
+		ksft_exit_fail_perror("open page flags");
+
+	/* Drain the LRU add batch so the locked mapping becomes unevictable. */
+	if (madvise(mapping, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	pfn = pagemap_get_pfn(pagemap_fd, mapping);
+	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags) ||
+	    !(flags & KPF_UNEVICTABLE))
+		ksft_exit_fail_msg("SHM_LOCK page is not unevictable\n");
+
+	if (madvise(mapping, page_size, MADV_PAGEOUT))
+		ksft_exit_fail_perror("MADV_PAGEOUT");
+	if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0] != 1)
+		ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable page\n");
+
+	close(kpageflags_fd);
+	close(pagemap_fd);
+	shmctl(shmid, SHM_UNLOCK, NULL);
+	shmdt(mapping);
+	shmctl(shmid, IPC_RMID, NULL);
+	ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable page\n");
+}
+
+/*
+ * Exercise the unevictable PAGEOUT path at PMD granularity. SHM_LOCK keeps
+ * the shmem THP resident without marking its VMA VM_LOCKED.
+ */
+static void test_madvise_pageout_unevictable_thp(void)
+{
+	char *mapping;
+	int pagemap_fd, shmid;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to lock a shmem segment\n");
+		return;
+	}
+	mapping = map_unevictable_shmem(pmd_size, &shmid);
+	if (mapping == MAP_FAILED)
+		return;
+	memset(mapping, 1, pmd_size);
+	if (!collapse_shmem(mapping, pmd_size, 1)) {
+		unmap_unevictable_shmem(mapping, shmid);
+		ksft_test_result_skip("could not allocate an unevictable THP\n");
+		return;
+	}
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+
+	if (madvise(mapping, pmd_size, MADV_PAGEOUT))
+		ksft_exit_fail_perror("MADV_PAGEOUT");
+	if (!pagemap_is_populated(pagemap_fd, mapping) ||
+	    !check_huge_shmem(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable THP\n");
+	check_memory(mapping, pmd_size);
+
+	close(pagemap_fd);
+	unmap_unevictable_shmem(mapping, shmid);
+	ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable THP\n");
+}
+
+/*
+ * migrate_pages() isolates folios from the LRU before replacing their page
+ * table entries. Race that interval against MADV_PAGEOUT on PMD-mapped THPs.
+ * Either operation may skip a folio owned by the other, but repeated calls
+ * must not report an error or corrupt the mapping.
+ */
+static void test_madvise_pageout_migration(void)
+{
+#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \
+	defined(SYS_migrate_pages)
+	unsigned long allowed[NUMA_MASK_LONGS] = {};
+	unsigned long mask1[NUMA_MASK_LONGS] = {};
+	unsigned long mask2[NUMA_MASK_LONGS] = {};
+	const size_t size = NR_MIGRATION_RACE_THPS * pmd_size;
+	char *mapping;
+	int node1, node2;
+	int error;
+	int shmid;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to lock a shmem segment\n");
+		return;
+	}
+	if (!find_two_memory_nodes(allowed, &node1, &node2)) {
+		ksft_test_result_skip("requires two allowed NUMA memory nodes\n");
+		return;
+	}
+
+	numa_mask_set(mask1, node1);
+	numa_mask_set(mask2, node2);
+	mapping = map_unevictable_shmem(size, &shmid);
+	if (mapping == MAP_FAILED)
+		return;
+	if (syscall(SYS_mbind, mapping, size, MPOL_BIND, mask1,
+		    NUMA_MASK_BITS + 1, 0)) {
+		unmap_unevictable_shmem(mapping, shmid);
+		ksft_test_result_skip("could not bind shmem to a NUMA node\n");
+		return;
+	}
+	memset(mapping, 1, size);
+	if (!collapse_shmem(mapping, size, NR_MIGRATION_RACE_THPS)) {
+		unmap_unevictable_shmem(mapping, shmid);
+		ksft_test_result_skip("could not allocate unevictable shmem THPs\n");
+		return;
+	}
+
+	error = race_pageout_with_migration(mapping, size, mask1, mask2);
+	if (error == ENOSYS) {
+		unmap_unevictable_shmem(mapping, shmid);
+		ksft_test_result_skip("NUMA migration is unavailable\n");
+		return;
+	}
+	if (error) {
+		errno = error;
+		ksft_exit_fail_perror("MADV_PAGEOUT/migrate_pages race");
+	}
+	check_memory(mapping, size);
+
+	unmap_unevictable_shmem(mapping, shmid);
+	ksft_test_result_pass("MADV_PAGEOUT races NUMA migration safely\n");
+#else
+	ksft_test_result_skip("NUMA migration system calls are unavailable\n");
+#endif
+}
+
+static bool activate_folio(char *mapping, char *drain, unsigned long pfn,
+			   int kpageflags_fd)
+{
+	struct iovec local = { .iov_base = drain, .iov_len = 1 };
+	struct iovec remote = { .iov_base = mapping, .iov_len = 1 };
+	uint64_t flags;
+	int retry;
+
+	for (retry = 0; retry < 10; retry++) {
+		if (process_vm_readv(getpid(), &local, 1, &remote, 1, 0) != 1)
+			ksft_exit_fail_perror("process_vm_readv");
+		/* Drain the activation batch without advising the target folio. */
+		if (madvise(drain, getpagesize(), MADV_COLD))
+			ksft_exit_fail_perror("MADV_COLD");
+		if (pageflags_get(pfn, kpageflags_fd, &flags))
+			ksft_exit_fail_perror("read kpageflags");
+		if (flags & KPF_ACTIVE)
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * Repeated GUP accesses promote an inactive folio on the traditional LRU.
+ * MADV_COLD must deactivate that folio while preserving its contents.
+ */
+static void test_madvise_cold_active_folio(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping = map_aligned_pages(page_size);
+	char *drain = map_aligned_pages(page_size);
+	cpu_set_t old_mask, mask;
+	int pagemap_fd, kpageflags_fd;
+	unsigned long pfn;
+	uint64_t flags;
+	int cpu;
+
+	if (geteuid()) {
+		munmap(drain, page_size);
+		munmap(mapping, page_size);
+		ksft_test_result_skip("requires root to read page flags\n");
+		return;
+	}
+	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("sched_getaffinity");
+	cpu = sched_getcpu();
+	if (cpu < 0)
+		ksft_exit_fail_perror("sched_getcpu");
+	CPU_ZERO(&mask);
+	CPU_SET(cpu, &mask);
+	if (sched_setaffinity(0, sizeof(mask), &mask))
+		ksft_exit_fail_perror("sched_setaffinity");
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
+	if (pagemap_fd < 0 || kpageflags_fd < 0)
+		ksft_exit_fail_perror("open page flags");
+	pfn = pagemap_get_pfn(pagemap_fd, mapping);
+	if (pfn == -1ul)
+		ksft_exit_fail_msg("could not read page PFN\n");
+
+	if (!activate_folio(mapping, drain, pfn, kpageflags_fd)) {
+		close(kpageflags_fd);
+		close(pagemap_fd);
+		sched_setaffinity(0, sizeof(old_mask), &old_mask);
+		munmap(drain, page_size);
+		munmap(mapping, page_size);
+		ksft_test_result_skip("could not activate a folio\n");
+		return;
+	}
+
+	if (madvise(mapping, page_size, MADV_COLD) ||
+	    madvise(drain, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (pageflags_get(pfn, kpageflags_fd, &flags))
+		ksft_exit_fail_perror("read kpageflags");
+	if ((flags & KPF_ACTIVE) || mapping[0] != 1)
+		ksft_exit_fail_msg("MADV_COLD did not deactivate the folio\n");
+
+	close(kpageflags_fd);
+	close(pagemap_fd);
+	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("restore affinity");
+	munmap(drain, page_size);
+	munmap(mapping, page_size);
+	ksft_test_result_pass("MADV_COLD deactivates an active folio\n");
+}
+
+struct remote_fault_data {
+	char *mapping;
+	pthread_barrier_t barrier;
+	int cpu;
+	int error;
+};
+
+static void *remote_fault_worker(void *arg)
+{
+	struct remote_fault_data *data = arg;
+	cpu_set_t mask;
+
+	CPU_ZERO(&mask);
+	CPU_SET(data->cpu, &mask);
+	if (sched_setaffinity(0, sizeof(mask), &mask))
+		data->error = errno;
+	else
+		data->mapping[0] = 1;
+	pthread_barrier_wait(&data->barrier);
+	pthread_barrier_wait(&data->barrier);
+	return NULL;
+}
+
+/*
+ * A folio faulted on another CPU can remain in that CPU's pending LRU batch.
+ * MADV_COLD drains only the calling CPU and must safely skip the non-LRU
+ * folio without changing the mapping.
+ */
+static void test_madvise_cold_remote_lru_batch(void)
+{
+	const size_t page_size = getpagesize();
+	struct remote_fault_data data = { .cpu = -1 };
+	char *mapping;
+	cpu_set_t old_mask, mask;
+	pthread_t thread;
+	unsigned long pfn;
+	uint64_t flags;
+	int pagemap_fd, kpageflags_fd;
+	int cpu, main_cpu = -1;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to read page flags\n");
+		return;
+	}
+	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("sched_getaffinity");
+	for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+		if (!CPU_ISSET(cpu, &old_mask))
+			continue;
+		if (main_cpu < 0) {
+			main_cpu = cpu;
+		} else {
+			data.cpu = cpu;
+			break;
+		}
+	}
+	if (main_cpu < 0 || data.cpu < 0) {
+		ksft_test_result_skip("requires two CPUs\n");
+		return;
+	}
+	CPU_ZERO(&mask);
+	CPU_SET(main_cpu, &mask);
+	if (sched_setaffinity(0, sizeof(mask), &mask))
+		ksft_exit_fail_perror("sched_setaffinity");
+
+	mapping = mmap(NULL, page_size, PROT_READ | PROT_WRITE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+	data.mapping = mapping;
+	if (pthread_barrier_init(&data.barrier, NULL, 2) ||
+	    pthread_create(&thread, NULL, remote_fault_worker, &data))
+		ksft_exit_fail_msg("could not start fault worker\n");
+	pthread_barrier_wait(&data.barrier);
+	if (data.error) {
+		errno = data.error;
+		ksft_exit_fail_perror("worker sched_setaffinity");
+	}
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
+	if (pagemap_fd < 0 || kpageflags_fd < 0)
+		ksft_exit_fail_perror("open page flags");
+	pfn = pagemap_get_pfn(pagemap_fd, mapping);
+	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags))
+		ksft_exit_fail_msg("could not read page flags\n");
+	if (flags & KPF_LRU) {
+		pthread_barrier_wait(&data.barrier);
+		pthread_join(thread, NULL);
+		pthread_barrier_destroy(&data.barrier);
+		close(kpageflags_fd);
+		close(pagemap_fd);
+		sched_setaffinity(0, sizeof(old_mask), &old_mask);
+		munmap(mapping, page_size);
+		ksft_test_result_skip("remote LRU batch was already drained\n");
+		return;
+	}
+
+	if (madvise(mapping, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (mapping[0] != 1 || !pagemap_is_populated(pagemap_fd, mapping))
+		ksft_exit_fail_msg("MADV_COLD changed a pending-LRU page\n");
+
+	pthread_barrier_wait(&data.barrier);
+	pthread_join(thread, NULL);
+	pthread_barrier_destroy(&data.barrier);
+	close(kpageflags_fd);
+	close(pagemap_fd);
+	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("restore affinity");
+	munmap(mapping, page_size);
+	ksft_test_result_pass("MADV_COLD skips a remote pending-LRU page\n");
+}
+
+/*
+ * Exercise the same active-folio transition through a huge PMD, where the
+ * page-table aging operation differs from the PTE implementation.
+ */
+static void test_pmd_madvise_cold_active_folio(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+	char *drain = map_aligned_pages(page_size);
+	cpu_set_t old_mask, mask;
+	int pagemap_fd, kpageflags_fd;
+	unsigned long pfn;
+	uint64_t flags;
+	int cpu, fd, retry;
+	bool active = false;
+
+	if (geteuid()) {
+		munmap(drain, page_size);
+		ksft_test_result_skip("requires root to read page flags\n");
+		return;
+	}
+	fd = memfd_create("madvise-active-thp", 0);
+	if (fd < 0)
+		ksft_exit_fail_perror("memfd_create");
+	if (ftruncate(fd, pmd_size))
+		ksft_exit_fail_perror("ftruncate");
+	mapping = map_aligned_file(fd, pmd_size, MAP_SHARED);
+	memset(mapping, 1, pmd_size);
+	if (msync(mapping, pmd_size, MS_SYNC) ||
+	    !collapse_shmem(mapping, pmd_size, 1)) {
+		munmap(drain, page_size);
+		munmap(mapping, pmd_size);
+		close(fd);
+		ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n");
+		return;
+	}
+	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("sched_getaffinity");
+	cpu = sched_getcpu();
+	if (cpu < 0)
+		ksft_exit_fail_perror("sched_getcpu");
+	CPU_ZERO(&mask);
+	CPU_SET(cpu, &mask);
+	if (sched_setaffinity(0, sizeof(mask), &mask))
+		ksft_exit_fail_perror("sched_setaffinity");
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
+	if (pagemap_fd < 0 || kpageflags_fd < 0)
+		ksft_exit_fail_perror("open page flags");
+	pfn = pagemap_get_pfn(pagemap_fd, mapping);
+	if (pfn == -1ul)
+		ksft_exit_fail_msg("could not read THP PFN\n");
+
+	for (retry = 0; retry < 10; retry++) {
+		if (pread(fd, drain, 1, 0) != 1)
+			ksft_exit_fail_perror("pread");
+		if (madvise(drain, page_size, MADV_COLD))
+			ksft_exit_fail_perror("MADV_COLD");
+		if (pageflags_get(pfn, kpageflags_fd, &flags))
+			ksft_exit_fail_perror("read kpageflags");
+		if (flags & KPF_ACTIVE) {
+			active = true;
+			break;
+		}
+	}
+	if (!active) {
+		close(kpageflags_fd);
+		close(pagemap_fd);
+		sched_setaffinity(0, sizeof(old_mask), &old_mask);
+		munmap(drain, page_size);
+		munmap(mapping, pmd_size);
+		close(fd);
+		ksft_test_result_skip("could not activate a THP\n");
+		return;
+	}
+
+	if (madvise(mapping, pmd_size, MADV_COLD) ||
+	    madvise(drain, page_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (pageflags_get(pfn, kpageflags_fd, &flags))
+		ksft_exit_fail_perror("read kpageflags");
+	if (flags & KPF_ACTIVE)
+		ksft_exit_fail_msg("MADV_COLD did not deactivate the THP\n");
+	if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_COLD split an active THP\n");
+	check_memory(mapping, pmd_size);
+
+	close(kpageflags_fd);
+	close(pagemap_fd);
+	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
+		ksft_exit_fail_perror("restore affinity");
+	munmap(drain, page_size);
+	munmap(mapping, pmd_size);
+	close(fd);
+	ksft_test_result_pass("MADV_COLD deactivates an active THP\n");
+}
+
+/*
+ * MADV_PAGEOUT on half of a PMD-mapped THP must split the folio before
+ * reclaim so the unadvised half is not reclaimed as part of the THP.
+ */
+static void test_partial_pmd_madvise_pageout(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	pageout_half_thp(mapping);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("partial MADV_PAGEOUT splits a PMD-mapped THP\n");
+}
+
+/*
+ * mprotect() replaces the huge PMD with PTEs without splitting the physical
+ * THP. MADV_PAGEOUT on half of that PTE-mapped THP must split the folio.
+ */
+static void test_partial_pte_madvise_pageout(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	split_pmd_mapping(mapping);
+	pageout_half_thp(mapping);
+
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("partial MADV_PAGEOUT splits a PTE-mapped THP\n");
+}
+
+static pid_t fork_waiting_child(int pipefd[2])
+{
+	pid_t pid;
+
+	if (pipe(pipefd))
+		ksft_exit_fail_perror("pipe");
+	pid = fork();
+	if (pid < 0)
+		ksft_exit_fail_perror("fork");
+	if (!pid) {
+		char byte;
+
+		close(pipefd[1]);
+		while (read(pipefd[0], &byte, 1) < 0 && errno == EINTR)
+			;
+		_exit(0);
+	}
+	close(pipefd[0]);
+	return pid;
+}
+
+/*
+ * fork() gives the THP another mapping, then mprotect() PTE-maps it in the
+ * parent. Assert that partial MADV_COLD leaves the shared physical THP and
+ * its contents intact.
+ */
+static void test_shared_pte_mapped_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+	int pipefd[2], status;
+	pid_t pid;
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	pid = fork_waiting_child(pipefd);
+
+	split_pmd_mapping(mapping);
+	if (madvise(mapping, pmd_size / 2, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_COLD split a shared large folio\n");
+	check_memory(mapping, pmd_size);
+
+	close(pipefd[1]);
+	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status))
+		ksft_exit_fail_msg("child process failed\n");
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("MADV_COLD skips a shared PTE-mapped THP\n");
+}
+
+/*
+ * A full-range PTE walk reaches the mapcount check rather than the partial
+ * folio split check. A second mapping must still prevent MADV_COLD from
+ * operating on the shared physical THP.
+ */
+static void test_full_shared_pte_mapped_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+	int pipefd[2], status;
+	pid_t pid;
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	pid = fork_waiting_child(pipefd);
+
+	split_pmd_mapping(mapping);
+	if (madvise(mapping, pmd_size, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_COLD split a shared large folio\n");
+	check_memory(mapping, pmd_size);
+
+	close(pipefd[1]);
+	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status))
+		ksft_exit_fail_msg("child process failed\n");
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("full MADV_COLD skips a shared PTE-mapped THP\n");
+}
+
+/*
+ * fork() gives a PMD-mapped THP another mapping. Assert that partial
+ * MADV_COLD leaves the shared physical THP and its contents intact.
+ */
+static void test_shared_pmd_madvise_cold(void)
+{
+	char *mapping = map_aligned_pages(pmd_size);
+	int pipefd[2], status;
+	pid_t pid;
+
+	if (!collapse_all(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
+		return;
+	}
+	pid = fork_waiting_child(pipefd);
+
+	if (madvise(mapping, pmd_size / 2, MADV_COLD))
+		ksft_exit_fail_perror("MADV_COLD");
+	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
+		ksft_exit_fail_msg("MADV_COLD split a shared PMD-mapped THP\n");
+	check_memory(mapping, pmd_size);
+
+	close(pipefd[1]);
+	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status))
+		ksft_exit_fail_msg("child process failed\n");
+	munmap(mapping, pmd_size);
+	ksft_test_result_pass("MADV_COLD skips a shared PMD-mapped THP\n");
+}
+
+static char *map_readonly_file(size_t size, int flags, int *fd)
+{
+	char template[] = "/tmp/madvise-pageout-XXXXXX";
+	char *mapping;
+
+	*fd = mkstemp(template);
+	if (*fd < 0)
+		ksft_exit_fail_perror("mkstemp");
+	if (unlink(template) || ftruncate(*fd, size) || fchmod(*fd, 0400))
+		ksft_exit_fail_perror("prepare file");
+	mapping = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, *fd, 0);
+	if (mapping == MAP_FAILED)
+		ksft_exit_fail_perror("mmap");
+
+	return mapping;
+}
+
+static bool pageout_until_evicted(char *mapping, size_t size, int pagemap_fd)
+{
+	int retry;
+
+	for (retry = 0; retry < 100; retry++) {
+		if (madvise(mapping, size, MADV_PAGEOUT))
+			ksft_exit_fail_perror("MADV_PAGEOUT");
+		if (!pagemap_is_populated(pagemap_fd, mapping))
+			return true;
+		usleep(10000);
+	}
+	return false;
+}
+
+/*
+ * The owner of a file may page out its clean page-cache pages. Assert that
+ * MADV_PAGEOUT removes the populated PTE and preserves the file contents.
+ */
+static void test_pageout_file(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+	int pagemap_fd;
+	bool evicted;
+	int fd;
+
+	mapping = map_readonly_file(page_size, MAP_PRIVATE, &fd);
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+	if (mapping[0])
+		ksft_exit_fail_msg("new file is not zero-filled\n");
+
+	evicted = pageout_until_evicted(mapping, page_size, pagemap_fd);
+	if (mapping[0])
+		ksft_exit_fail_msg("file mapping contents changed\n");
+
+	close(pagemap_fd);
+	munmap(mapping, page_size);
+	close(fd);
+	if (!evicted) {
+		ksft_test_result_skip("MADV_PAGEOUT did not evict the file page\n");
+		return;
+	}
+	ksft_test_result_pass("MADV_PAGEOUT evicts an authorized file page\n");
+}
+
+static int pageout_shared_file_without_permission(char *mapping,
+						  size_t page_size)
+{
+	int pagemap_fd;
+
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		return KSFT_FAIL;
+	if (setgid(65534) || setuid(65534))
+		return KSFT_FAIL;
+	if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping))
+		return KSFT_FAIL;
+	if (madvise(mapping, page_size, MADV_PAGEOUT))
+		return KSFT_FAIL;
+	if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0])
+		return KSFT_FAIL;
+	return KSFT_PASS;
+}
+
+/*
+ * A caller without file write permission may not page out a shared file
+ * mapping. Assert that MADV_PAGEOUT succeeds without evicting its file page.
+ */
+static void test_pageout_unauthorized_shared_file(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+	int fd, status;
+	pid_t pid;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to change credentials\n");
+		return;
+	}
+
+	mapping = map_readonly_file(page_size, MAP_SHARED, &fd);
+
+	pid = fork();
+	if (pid < 0)
+		ksft_exit_fail_perror("fork");
+	if (!pid)
+		_exit(pageout_shared_file_without_permission(mapping, page_size));
+
+	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
+	    WEXITSTATUS(status) != KSFT_PASS)
+		ksft_exit_fail_msg("unauthorized child evicted a shared file page\n");
+	munmap(mapping, page_size);
+	close(fd);
+	ksft_test_result_pass("MADV_PAGEOUT skips an unauthorized shared file\n");
+}
+
+/*
+ * An unauthorized private mapping may page out anonymous COW pages, but not
+ * its file folios. Assert that partial MADV_PAGEOUT filters a large shmem
+ * folio before attempting to split or reclaim it.
+ */
+static void test_pageout_anon_only_large_folio(void)
+{
+	char *mapping, *shared;
+	gid_t old_egid;
+	int fd;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to change credentials\n");
+		return;
+	}
+	fd = memfd_create("madvise-pageout-large", 0);
+	if (fd < 0)
+		ksft_exit_fail_perror("memfd_create");
+	if (ftruncate(fd, pmd_size))
+		ksft_exit_fail_perror("ftruncate");
+	shared = map_aligned_file(fd, pmd_size, MAP_SHARED);
+	memset(shared, 1, pmd_size);
+	if (msync(shared, pmd_size, MS_SYNC))
+		ksft_exit_fail_perror("msync");
+	munmap(shared, pmd_size);
+
+	mapping = map_aligned_file(fd, pmd_size, MAP_PRIVATE);
+	if (!collapse_shmem(mapping, pmd_size, 1)) {
+		munmap(mapping, pmd_size);
+		close(fd);
+		ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n");
+		return;
+	}
+	if (fchmod(fd, 0400))
+		ksft_exit_fail_perror("fchmod");
+
+	/* Retain saved uid 0 so this process can restore its credentials. */
+	old_egid = getegid();
+	if (setegid(65534) || seteuid(65534))
+		ksft_exit_fail_perror("drop privileges");
+	if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT))
+		ksft_exit_fail_perror("MADV_PAGEOUT");
+	if (seteuid(0) || setegid(old_egid))
+		ksft_exit_fail_perror("restore privileges");
+
+	if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size) ||
+	    mapping[0] != 1 || mapping[pmd_size - 1] != 1)
+		ksft_exit_fail_msg("MADV_PAGEOUT changed a protected large folio\n");
+	munmap(mapping, pmd_size);
+	close(fd);
+	ksft_test_result_pass("MADV_PAGEOUT filters a protected large folio\n");
+}
+
+/*
+ * A private file mapping can contain both anonymous COW and file-backed
+ * pages. As a caller without file write permission, assert that PAGEOUT swaps
+ * the COW page but leaves the file-backed page resident.
+ */
+static void test_pageout_anon_only(void)
+{
+	const size_t page_size = getpagesize();
+	char *mapping;
+	int pagemap_fd;
+	bool swapped;
+	int fd;
+
+	if (geteuid()) {
+		ksft_test_result_skip("requires root to change credentials\n");
+		return;
+	}
+
+	mapping = map_readonly_file(2 * page_size, MAP_PRIVATE, &fd);
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		ksft_exit_fail_perror("open pagemap");
+
+	if (setgid(65534) || setuid(65534))
+		ksft_exit_fail_perror("drop privileges");
+	if (mapping[0] || mapping[page_size])
+		ksft_exit_fail_msg("new file is not zero-filled\n");
+	mapping[0] = 1;
+	if (madvise(mapping, 2 * page_size, MADV_PAGEOUT))
+		ksft_exit_fail_perror("MADV_PAGEOUT");
+	if (!pagemap_is_populated(pagemap_fd, mapping + page_size))
+		ksft_exit_fail_msg("MADV_PAGEOUT evicted a protected file page\n");
+	swapped = pagemap_is_swapped(pagemap_fd, mapping);
+	if (mapping[0] != 1 || mapping[page_size])
+		ksft_exit_fail_msg("private file mapping contents changed\n");
+
+	close(pagemap_fd);
+	munmap(mapping, 2 * page_size);
+	close(fd);
+	if (!swapped) {
+		ksft_test_result_skip("MADV_PAGEOUT did not swap the COW page\n");
+		return;
+	}
+	ksft_test_result_pass("MADV_PAGEOUT filters private file pages\n");
+}
+
+int main(void)
+{
+	pmd_size = read_pmd_pagesize();
+
+	ksft_print_header();
+	ksft_set_plan(28);
+	if (!pmd_size)
+		ksft_exit_skip("PMD-sized THPs are not supported\n");
+
+	test_full_pmd_madvise_cold();
+	test_repeated_pmd_madvise_cold();
+	test_partial_pmd_madvise_cold();
+	test_pte_mapped_madvise_cold();
+	test_madvise_cold_pte_hole();
+	test_madvise_cold_zero_page();
+	test_madvise_cold_huge_zero_page();
+	test_madvise_lru_locked_vma();
+	test_shared_pmd_madvise_cold();
+	test_full_pmd_madvise_pageout();
+	test_madvise_cold_swapped_pte();
+	test_partial_pinned_pmd_madvise_cold();
+	test_partial_pinned_pte_madvise_cold();
+	test_concurrent_partial_madvise_cold();
+	test_madvise_pageout_unevictable();
+	test_madvise_pageout_unevictable_thp();
+	test_madvise_pageout_migration();
+	test_madvise_cold_active_folio();
+	test_madvise_cold_remote_lru_batch();
+	test_pmd_madvise_cold_active_folio();
+	test_partial_pmd_madvise_pageout();
+	test_partial_pte_madvise_pageout();
+	test_shared_pte_mapped_madvise_cold();
+	test_full_shared_pte_mapped_madvise_cold();
+	test_pageout_file();
+	test_pageout_unauthorized_shared_file();
+	test_pageout_anon_only_large_folio();
+	test_pageout_anon_only();
+	ksft_finished();
+}
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 6990485b1a9da..bc136ead10f91 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -51,6 +51,8 @@ separated by spaces:
 	hmm smoke tests
 - madv_guard
 	test madvise(2) MADV_GUARD_INSTALL and MADV_GUARD_REMOVE options
+- madvise
+	test MADV_COLD and MADV_PAGEOUT
 - madv_populate
 	test memadvise(2) MADV_POPULATE_{READ,WRITE} options
 - memfd_secret
@@ -327,6 +329,9 @@ CATEGORY="hmm" run_test bash ./test_hmm.sh smoke
 # MADV_GUARD_INSTALL and MADV_GUARD_REMOVE tests
 CATEGORY="madv_guard" run_test ./guard-regions
 
+# MADV_COLD and MADV_PAGEOUT tests
+CATEGORY="madvise" run_test ./madvise
+
 # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
 CATEGORY="madv_populate" run_test ./madv_populate
 
diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c
index 31d331c1c4521..af0a7f65bfd9f 100644
--- a/tools/testing/selftests/mm/vm_util.c
+++ b/tools/testing/selftests/mm/vm_util.c
@@ -373,8 +373,8 @@ static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages,
 	return thp == (nr_hpages * (hpage_size >> 10));
 }
 
-static bool check_large_folios(void *addr, size_t len, int nr_hpages,
-		uint64_t hpage_size)
+bool check_large_folios(void *addr, size_t len, int nr_hpages,
+			uint64_t hpage_size)
 {
 	int order = 0, pagesize = getpagesize();
 	unsigned int nr_pages = hpage_size / pagesize;
diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h
index 072a6c756c517..33ae68d721df9 100644
--- a/tools/testing/selftests/mm/vm_util.h
+++ b/tools/testing/selftests/mm/vm_util.h
@@ -22,10 +22,14 @@
 #define PM_SWAP                       BIT_ULL(62)
 #define PM_PRESENT                    BIT_ULL(63)
 
+#define KPF_LRU                       BIT_ULL(5)
+#define KPF_ACTIVE                    BIT_ULL(6)
 #define KPF_COMPOUND_HEAD             BIT_ULL(15)
 #define KPF_COMPOUND_TAIL             BIT_ULL(16)
+#define KPF_UNEVICTABLE               BIT_ULL(18)
 #define KPF_HWPOISON                  BIT_ULL(19)
 #define KPF_THP                       BIT_ULL(22)
+#define KPF_ZERO_PAGE                 BIT_ULL(24)
 /*
  * Ignore the checkpatch warning, we must read from x but don't want to do
  * anything with it in order to trigger a read page fault. We therefore must use
@@ -97,6 +101,8 @@ unsigned long rss_anon(void);
 bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
 bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
 bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
+bool check_large_folios(void *addr, size_t len, int nr_hpages,
+			uint64_t hpage_size);
 int64_t allocate_transhuge(void *ptr, int pagemap_fd);
 int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags);
 int gather_folio_orders(char *vaddr_start, size_t len,
-- 
2.53.0-Meta

Re: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[Lorenzo Stoakes (ARM)]

On Tue, Sep 22, 2026 at 07:58:21PM -0400, Gregory Price wrote:
> MADV_COLD and MADV_PAGEOUT share a page-table walker covering ordinary
> PTEs, PTE-mapped large folios and huge PMDs. Existing selftests provide
> little coverage of its range, folio-state and permission decisions.
>
> Add 28 cases ahead of a planned walker refactor. All cases check syscall
> results and mapping contents, but they deliberately have different review
> contracts.

Yikes that's a lot! :)

>
> This commit was originally 28 individual commits, but was squashed for
> the initial RFC to limit noise during initial review. The question is
> what set of self-tests should retain.
>
> The eight user-visible contract tests verify:
>
>   - MADV_COLD does not populate a PTE hole;
>   - COLD and PAGEOUT reject a locked VMA;
>   - full-range PAGEOUT swaps a PMD-mapped THP without corrupting it;
>   - MADV_COLD does not fault in a swapped PTE;
>   - PAGEOUT preserves an unevictable base page;
>   - authorized file PAGEOUT evicts a clean file page;
>   - unauthorized shared-file PAGEOUT leaves the page resident; and
>   - private-file PAGEOUT reclaims a COW page but preserves a file page.
>
> The eighteen implementation-detail tests record the current folio and
> page-table behavior:
>
>   - full and partial MADV_COLD on a PTE-mapped THP;
>   - full, repeated and partial MADV_COLD on PMD-mapped THPs;
>   - MADV_COLD on the shared zero page and huge zero page;
>   - partial MADV_COLD on a shared PMD-mapped THP;
>   - partial MADV_COLD on pinned PMD- and PTE-mapped THPs;
>   - parallel partial MADV_COLD split attempts;
>   - PAGEOUT on an unevictable THP;
>   - MADV_COLD on active base-page and huge-PMD folios;
>   - partial PAGEOUT on PMD- and PTE-mapped THPs;
>   - partial and full MADV_COLD on shared PTE-mapped THPs; and
>   - PAGEOUT filtering of a PMD-sized protected file folio.
>
> These tests intentionally assert splitting, PMD/PTE mapping shape, folio
> flags or folio sharing. They may need adjustment after a valid kernel
> implementation change. Their purpose here is to detect unintended changes
> during the refactor.
>
> Two additional stress tests use implementation-specific setup while keeping
> contract-level assertions:
>
>   - MADV_COLD on a folio still queued in another CPU's LRU batch
>   - MADV_PAGEOUT during NUMA migration of unevictable shmem THPs.
>
> The stress tests require only successful calls and preserved contents.
> They do not assert which internal path won. Tests requiring root, swap,
> THP allocation, multiple CPUs or NUMA nodes report a skip when their
> prerequisites are unavailable.

I think in general we want to avoid putting stress tests as part of the
selftests in general? Or at least if they might take a long time to run or
excessive memory usage, etc.

Though that might be benchmarks, as we have the THP stress tests so maybe it's
OK?

>
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>

In any case I really do think this should be split out into smaller parts, this
is a _gigantic_ change :)

Since you list a whole bunch of different test areas and you're exercising
different things you should be able to split this out logically across those I
think?

I like the idea of establishing some sort of invariants first before
implementing changes though in general.

> ---
>  tools/testing/selftests/mm/Makefile        |    2 +
>  tools/testing/selftests/mm/ksft_madvise.sh |    4 +
>  tools/testing/selftests/mm/madvise.c       | 1673 ++++++++++++++++++++
>  tools/testing/selftests/mm/run_vmtests.sh  |    5 +
>  tools/testing/selftests/mm/vm_util.c       |    4 +-
>  tools/testing/selftests/mm/vm_util.h       |    6 +
>  6 files changed, 1692 insertions(+), 2 deletions(-)
>  create mode 100755 tools/testing/selftests/mm/ksft_madvise.sh
>  create mode 100644 tools/testing/selftests/mm/madvise.c
>
> diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
> index beacc0f873049..8912aa85ca273 100644
> --- a/tools/testing/selftests/mm/Makefile
> +++ b/tools/testing/selftests/mm/Makefile
> @@ -69,6 +69,7 @@ TEST_GEN_FILES += hugetlb-shm
>  TEST_GEN_FILES += hugetlb-soft-offline
>  TEST_GEN_FILES += khugepaged
>  TEST_GEN_FILES += madv_populate
> +TEST_GEN_FILES += madvise
>  TEST_GEN_FILES += map_fixed_noreplace
>  TEST_GEN_FILES += map_populate
>  ifneq (,$(filter $(ARCH),arm64 riscv riscv64 x86 x86_64 loongarch32 loongarch64))
> @@ -155,6 +156,7 @@ TEST_PROGS += ksft_kmemleak_dedup.sh
>  TEST_PROGS += ksft_ksm.sh
>  TEST_PROGS += ksft_ksm_numa.sh
>  TEST_PROGS += ksft_madv_guard.sh
> +TEST_PROGS += ksft_madvise.sh
>  TEST_PROGS += ksft_madv_populate.sh
>  TEST_PROGS += ksft_memfd_secret.sh
>  TEST_PROGS += ksft_memory_failure.sh
> diff --git a/tools/testing/selftests/mm/ksft_madvise.sh b/tools/testing/selftests/mm/ksft_madvise.sh
> new file mode 100755
> index 0000000000000..29cbbca800f2c
> --- /dev/null
> +++ b/tools/testing/selftests/mm/ksft_madvise.sh
> @@ -0,0 +1,4 @@
> +#!/bin/sh -e
> +# SPDX-License-Identifier: GPL-2.0
> +
> +./run_vmtests.sh -t madvise
> diff --git a/tools/testing/selftests/mm/madvise.c b/tools/testing/selftests/mm/madvise.c
> new file mode 100644
> index 0000000000000..abde0b1c8a42e
> --- /dev/null
> +++ b/tools/testing/selftests/mm/madvise.c
> @@ -0,0 +1,1673 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#define _GNU_SOURCE
> +
> +#include <errno.h>
> +#include <fcntl.h>
> +#include <pthread.h>
> +#include <sched.h>
> +#include <stdbool.h>
> +#include <stdatomic.h>
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <linux/mman.h>
> +#include <linux/mempolicy.h>
> +#include <sys/ipc.h>
> +#include <sys/mman.h>
> +#include <sys/shm.h>
> +#include <sys/stat.h>
> +#include <sys/syscall.h>
> +#include <sys/uio.h>
> +#include <sys/wait.h>
> +#include <unistd.h>
> +
> +#include "kselftest.h"
> +#include "vm_util.h"
> +
> +#ifndef MADV_COLLAPSE
> +#define MADV_COLLAPSE 25
> +#endif
> +
> +#define NR_CONCURRENT_THREADS 8
> +#define NR_CONCURRENT_THPS 8
> +#define NR_CONCURRENT_ROUNDS 400
> +#define NR_MIGRATION_RACE_THPS 16
> +#define NR_MIGRATION_RACE_ROUNDS 20
> +#define NUMA_MASK_BITS 1024
> +#define NUMA_MASK_LONGS (NUMA_MASK_BITS / (8 * sizeof(unsigned long)))
> +
> +static pthread_barrier_t concurrent_start_barrier;
> +static pthread_barrier_t concurrent_done_barrier;
> +static atomic_int concurrent_worker_errno;
> +static char *concurrent_area;
> +static size_t pmd_size;
> +
> +#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \
> +	defined(SYS_migrate_pages)
> +struct migration_pageout_data {
> +	pthread_barrier_t start_barrier;
> +	atomic_bool stop;
> +	atomic_int calls;
> +	atomic_int error;
> +	char *mapping;
> +	size_t size;
> +};
> +
> +static bool find_two_memory_nodes(unsigned long *mask, int *node1, int *node2)
> +{
> +	int node;
> +
> +	if (syscall(SYS_get_mempolicy, NULL, mask, NUMA_MASK_BITS + 1, NULL,
> +		    MPOL_F_MEMS_ALLOWED))
> +		return false;
> +
> +	*node1 = *node2 = -1;
> +	for (node = 0; node < NUMA_MASK_BITS; node++) {
> +		if (!(mask[node / (8 * sizeof(*mask))] &
> +		      (1UL << (node % (8 * sizeof(*mask))))))
> +			continue;
> +		if (*node1 < 0) {
> +			*node1 = node;
> +		} else {
> +			*node2 = node;
> +			return true;
> +		}
> +	}
> +
> +	return false;
> +}
> +
> +static void numa_mask_set(unsigned long *mask, int node)
> +{
> +	mask[node / (8 * sizeof(*mask))] |=
> +		1UL << (node % (8 * sizeof(*mask)));
> +}
> +
> +static void *madvise_pageout_worker(void *arg)
> +{
> +	struct migration_pageout_data *data = arg;
> +
> +	pthread_barrier_wait(&data->start_barrier);
> +	while (!atomic_load_explicit(&data->stop, memory_order_relaxed)) {
> +		if (madvise(data->mapping, data->size, MADV_PAGEOUT) &&
> +		    errno != EAGAIN) {
> +			atomic_store(&data->error, errno);
> +			break;
> +		}
> +		atomic_fetch_add(&data->calls, 1);
> +	}
> +
> +	return NULL;
> +}
> +
> +static int migrate_between_nodes(unsigned long *old_nodes,
> +				 unsigned long *new_nodes)
> +{
> +	int i;
> +
> +	for (i = 0; i < NR_MIGRATION_RACE_ROUNDS; i++) {
> +		unsigned long *tmp;
> +
> +		if (syscall(SYS_migrate_pages, 0, NUMA_MASK_BITS + 1,
> +			    old_nodes, new_nodes) < 0)
> +			return errno;
> +		tmp = old_nodes;
> +		old_nodes = new_nodes;
> +		new_nodes = tmp;
> +	}
> +
> +	return 0;
> +}
> +
> +static int race_pageout_with_migration(char *mapping, size_t size,
> +				       unsigned long *mask1,
> +				       unsigned long *mask2)
> +{
> +	struct migration_pageout_data data = {
> +		.mapping = mapping,
> +		.size = size,
> +	};
> +	pthread_t thread;
> +	int ret;
> +
> +	ret = pthread_barrier_init(&data.start_barrier, NULL, 2);
> +	if (ret)
> +		return ret;
> +	ret = pthread_create(&thread, NULL, madvise_pageout_worker, &data);
> +	if (ret) {
> +		pthread_barrier_destroy(&data.start_barrier);
> +		return ret;
> +	}
> +	pthread_barrier_wait(&data.start_barrier);
> +
> +	ret = migrate_between_nodes(mask1, mask2);
> +	atomic_store(&data.stop, true);
> +	pthread_join(thread, NULL);
> +	pthread_barrier_destroy(&data.start_barrier);
> +
> +	if (ret)
> +		return ret;
> +	if (atomic_load(&data.error))
> +		return atomic_load(&data.error);
> +	return atomic_load(&data.calls) ? 0 : EIO;
> +}
> +#endif
> +
> +static char *map_aligned_pages(size_t size)
> +{
> +	char *mapping, *aligned;
> +
> +	mapping = mmap(NULL, size + pmd_size, PROT_READ | PROT_WRITE,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +
> +	aligned = (char *)(((uintptr_t)mapping + pmd_size - 1) &
> +			   ~(pmd_size - 1));
> +	if (aligned != mapping)
> +		munmap(mapping, aligned - mapping);
> +	if (aligned + size < mapping + size + pmd_size)
> +		munmap(aligned + size,
> +		       mapping + size + pmd_size - (aligned + size));
> +
> +	memset(aligned, 1, size);
> +	return aligned;
> +}
> +
> +/* MADV_COLLAPSE may fail transiently with EAGAIN. */
> +static bool collapse_all(char *mapping, size_t size, int nr_hpages)
> +{
> +	int ret, retry;
> +
> +	for (retry = 0; retry < 10; retry++) {
> +		ret = madvise(mapping, size, MADV_COLLAPSE);
> +		if (!ret) {
> +			if (check_huge_anon(mapping, size, nr_hpages, pmd_size))
> +				return true;
> +		} else if (errno != EAGAIN) {
> +			return false;
> +		}
> +		usleep(10000);
> +	}
> +
> +	return false;
> +}
> +
> +static bool collapse_shmem(char *mapping, size_t size, int nr_hpages)
> +{
> +	int ret, retry;
> +
> +	for (retry = 0; retry < 10; retry++) {
> +		ret = madvise(mapping, size, MADV_COLLAPSE);
> +		if (!ret) {
> +			if (check_huge_shmem(mapping, size, nr_hpages, pmd_size))
> +				return true;
> +		} else if (errno != EAGAIN) {
> +			return false;
> +		}
> +		usleep(10000);
> +	}
> +
> +	return false;
> +}
> +
> +static char *map_unevictable_shmem(size_t size, int *shmid)
> +{
> +	char *reservation, *mapping, *aligned;
> +
> +	reservation = mmap(NULL, size + pmd_size, PROT_NONE,
> +			   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (reservation == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap reservation");
> +	aligned = (char *)(((uintptr_t)reservation + pmd_size - 1) &
> +			   ~(pmd_size - 1));
> +	munmap(reservation, size + pmd_size);
> +
> +	*shmid = shmget(IPC_PRIVATE, size, IPC_CREAT | 0600);
> +	if (*shmid < 0)
> +		ksft_exit_fail_perror("shmget");
> +	if (shmctl(*shmid, SHM_LOCK, NULL)) {
> +		shmctl(*shmid, IPC_RMID, NULL);
> +		ksft_test_result_skip("could not lock a shmem segment\n");
> +		return MAP_FAILED;
> +	}
> +	mapping = shmat(*shmid, aligned, 0);
> +	if (mapping == (void *)-1)
> +		ksft_exit_fail_perror("shmat");
> +	if (mapping != aligned)
> +		ksft_exit_fail_msg("shmat did not honor the aligned address\n");
> +
> +	return mapping;
> +}
> +
> +static void unmap_unevictable_shmem(char *mapping, int shmid)
> +{
> +	shmctl(shmid, SHM_UNLOCK, NULL);
> +	shmdt(mapping);
> +	shmctl(shmid, IPC_RMID, NULL);
> +}
> +
> +static char *map_aligned_file(int fd, size_t size, int flags)
> +{
> +	char *mapping, *aligned;
> +
> +	mapping = mmap(NULL, size + pmd_size, PROT_NONE,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap reservation");
> +	aligned = (char *)(((uintptr_t)mapping + pmd_size - 1) &
> +			   ~(pmd_size - 1));
> +	if (mmap(aligned, size, PROT_READ | PROT_WRITE, flags | MAP_FIXED,
> +		 fd, 0) == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap file");
> +	if (aligned != mapping)
> +		munmap(mapping, aligned - mapping);
> +	if (aligned + size < mapping + size + pmd_size)
> +		munmap(aligned + size,
> +		       mapping + size + pmd_size - (aligned + size));
> +
> +	return aligned;
> +}
> +
> +static void split_pmd_mapping(char *mapping)
> +{
> +	const size_t page_size = getpagesize();
> +
> +	if (mprotect(mapping + page_size, page_size, PROT_READ) ||
> +	    mprotect(mapping + page_size, page_size, PROT_READ | PROT_WRITE))
> +		ksft_exit_fail_perror("mprotect");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("mprotect split the physical THP\n");
> +}
> +
> +static void check_memory(char *mapping, size_t size)
> +{
> +	size_t offset;
> +
> +	for (offset = 0; offset < size; offset += getpagesize())
> +		if (mapping[offset] != 1)
> +			ksft_exit_fail_msg("memory changed at offset %zu\n", offset);
> +}
> +
> +/*
> + * MADV_COLD on a full PTE-mapped THP must preserve the folio, while an
> + * operation on only half of it must split the folio. Neither operation may
> + * alter the mapping contents.
> + */
> +static void test_pte_mapped_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +
> +	split_pmd_mapping(mapping);
> +	if (madvise(mapping, pmd_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("full MADV_COLD split a large folio\n");
> +
> +	if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
> +		ksft_exit_fail_msg("partial MADV_COLD left a large folio\n");
> +	check_memory(mapping, pmd_size);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("MADV_COLD handles a PTE-mapped THP\n");
> +}
> +
> +/*
> + * A PTE walk must skip holes without populating them or overlooking the
> + * present pages on either side.
> + */
> +static void test_madvise_cold_pte_hole(void)
> +{
> +	const size_t page_size = getpagesize();
> +	const size_t size = 3 * page_size;
> +	char *mapping;
> +	int pagemap_fd;
> +
> +	mapping = mmap(NULL, size, PROT_READ | PROT_WRITE,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +	mapping[0] = 1;
> +	mapping[2 * page_size] = 1;
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +	if (pagemap_is_populated(pagemap_fd, mapping + page_size))
> +		ksft_exit_fail_msg("PTE hole was populated before MADV_COLD\n");
> +
> +	if (madvise(mapping, size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (mapping[0] != 1 || mapping[2 * page_size] != 1)
> +		ksft_exit_fail_msg("MADV_COLD changed populated pages\n");
> +	if (pagemap_is_populated(pagemap_fd, mapping + page_size))
> +		ksft_exit_fail_msg("MADV_COLD populated a PTE hole\n");
> +
> +	close(pagemap_fd);
> +	munmap(mapping, size);
> +	ksft_test_result_pass("MADV_COLD skips PTE holes\n");
> +}
> +
> +/*
> + * A read fault on private anonymous memory may install the shared zero page.
> + * MADV_COLD must ignore that special PTE without replacing it or changing the
> + * mapping contents.
> + */
> +static void test_madvise_cold_zero_page(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +	int pagemap_fd;
> +
> +	mapping = mmap(NULL, page_size, PROT_READ,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +	if (mapping[0])
> +		ksft_exit_fail_msg("anonymous mapping is not zero-filled\n");
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +	if (!pagemap_is_populated(pagemap_fd, mapping))
> +		ksft_exit_fail_msg("zero page is not populated\n");
> +
> +	if (madvise(mapping, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping))
> +		ksft_exit_fail_msg("MADV_COLD changed the zero-page mapping\n");
> +
> +	close(pagemap_fd);
> +	munmap(mapping, page_size);
> +	ksft_test_result_pass("MADV_COLD skips the shared zero page\n");
> +}
> +
> +/*
> + * A read fault may map the shared huge zero page with a PMD. MADV_COLD must
> + * ignore that special PMD without replacing it or changing the mapping.
> + */
> +static void test_madvise_cold_huge_zero_page(void)
> +{
> +	char *reservation, *mapping;
> +	unsigned long pfn;
> +	uint64_t flags;
> +	int pagemap_fd, kpageflags_fd;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to read page flags\n");
> +		return;
> +	}
> +	reservation = mmap(NULL, 2 * pmd_size, PROT_NONE,
> +			   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (reservation == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap reservation");
> +	mapping = (char *)(((uintptr_t)reservation + pmd_size - 1) &
> +			   ~(pmd_size - 1));
> +	if (mmap(mapping, pmd_size, PROT_READ,
> +		 MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap huge zero page");
> +	if (mapping != reservation)
> +		munmap(reservation, mapping - reservation);
> +	if (mapping + pmd_size < reservation + 2 * pmd_size)
> +		munmap(mapping + pmd_size,
> +		       reservation + 2 * pmd_size - (mapping + pmd_size));
> +	if (madvise(mapping, pmd_size, MADV_HUGEPAGE))
> +		ksft_exit_fail_perror("MADV_HUGEPAGE");
> +	if (mapping[0])
> +		ksft_exit_fail_msg("anonymous mapping is not zero-filled\n");
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
> +	if (pagemap_fd < 0 || kpageflags_fd < 0)
> +		ksft_exit_fail_perror("open page flags");
> +	pfn = pagemap_get_pfn(pagemap_fd, mapping);
> +	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags) ||
> +	    !(flags & KPF_ZERO_PAGE)) {
> +		close(kpageflags_fd);
> +		close(pagemap_fd);
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("huge zero page is not available\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, pmd_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (mapping[0] || pagemap_get_pfn(pagemap_fd, mapping) != pfn)
> +		ksft_exit_fail_msg("MADV_COLD changed the huge-zero-page mapping\n");
> +
> +	close(kpageflags_fd);
> +	close(pagemap_fd);
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("MADV_COLD skips the huge zero page\n");
> +}
> +
> +/*
> + * Neither hint is valid for a locked VMA: reclaiming it would violate the
> + * mlock contract, and merely aging it would serve no purpose.
> + */
> +static void test_madvise_lru_locked_vma(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +
> +	mapping = mmap(NULL, page_size, PROT_READ | PROT_WRITE,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +	mapping[0] = 1;
> +	if (mlock(mapping, page_size)) {
> +		munmap(mapping, page_size);
> +		ksft_test_result_skip("could not lock a page\n");
> +		return;
> +	}
> +
> +	errno = 0;
> +	if (!madvise(mapping, page_size, MADV_COLD) || errno != EINVAL)
> +		ksft_exit_fail_msg("MADV_COLD accepted a locked VMA\n");
> +	errno = 0;
> +	if (!madvise(mapping, page_size, MADV_PAGEOUT) || errno != EINVAL)
> +		ksft_exit_fail_msg("MADV_PAGEOUT accepted a locked VMA\n");
> +	if (mapping[0] != 1)
> +		ksft_exit_fail_msg("madvise changed locked memory\n");
> +
> +	munlock(mapping, page_size);
> +	munmap(mapping, page_size);
> +	ksft_test_result_pass("COLD and PAGEOUT reject a locked VMA\n");
> +}
> +
> +/*
> + * A full-range MADV_COLD operates directly on a huge PMD. Assert that aging
> + * the mapping preserves both the physical THP and its contents.
> + */
> +static void test_full_pmd_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	if (madvise(mapping, pmd_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("full MADV_COLD split a PMD-mapped THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("full MADV_COLD preserves a PMD-mapped THP\n");
> +}
> +
> +/*
> + * MADV_COLD is idempotent. A second request must handle an already-old huge
> + * PMD without splitting the THP or changing its contents.
> + */
> +static void test_repeated_pmd_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	if (madvise(mapping, pmd_size, MADV_COLD) ||
> +	    madvise(mapping, pmd_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("repeated MADV_COLD split a PMD-mapped THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("repeated MADV_COLD preserves a PMD-mapped THP\n");
> +}
> +
> +/*
> + * MADV_COLD on half of a PMD-mapped THP must split the folio so the
> + * unadvised half is not aged as part of the THP.
> + */
> +static void test_partial_pmd_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
> +		ksft_exit_fail_msg("partial MADV_COLD left a PMD-mapped THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("partial MADV_COLD splits a PMD-mapped THP\n");
> +}
> +
> +static void pageout_half_thp(char *mapping)
> +{
> +	if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT))
> +		ksft_exit_fail_perror("MADV_PAGEOUT");
> +	if (!check_large_folios(mapping, pmd_size, 0, pmd_size))
> +		ksft_exit_fail_msg("partial MADV_PAGEOUT left a large folio\n");
> +	check_memory(mapping, pmd_size);
> +}
> +
> +/*
> + * Full-range MADV_PAGEOUT can reclaim a PMD-mapped THP directly. Assert that
> + * the mapping is swapped without corrupting the folio contents.
> + */
> +static void test_full_pmd_madvise_pageout(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +	int pagemap_fd, retry;
> +	bool swapped = false;
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +
> +	for (retry = 0; retry < 100; retry++) {
> +		if (madvise(mapping, pmd_size, MADV_PAGEOUT))
> +			ksft_exit_fail_perror("MADV_PAGEOUT");
> +		if (pagemap_is_swapped(pagemap_fd, mapping)) {
> +			swapped = true;
> +			break;
> +		}
> +		usleep(10000);
> +	}
> +	if (swapped)
> +		check_memory(mapping, pmd_size);
> +
> +	close(pagemap_fd);
> +	munmap(mapping, pmd_size);
> +	if (!swapped) {
> +		ksft_test_result_skip("MADV_PAGEOUT did not swap the THP\n");
> +		return;
> +	}
> +	ksft_test_result_pass("full MADV_PAGEOUT swaps a PMD-mapped THP\n");
> +}
> +
> +/*
> + * A swapped PTE is non-present but not empty. MADV_COLD must skip the entry
> + * without faulting the page back in or changing its contents.
> + */
> +static void test_madvise_cold_swapped_pte(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping = map_aligned_pages(page_size);
> +	int pagemap_fd, retry;
> +	bool swapped = false;
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +	for (retry = 0; retry < 100; retry++) {
> +		if (madvise(mapping, page_size, MADV_PAGEOUT))
> +			ksft_exit_fail_perror("MADV_PAGEOUT");
> +		if (pagemap_is_swapped(pagemap_fd, mapping)) {
> +			swapped = true;
> +			break;
> +		}
> +		usleep(10000);
> +	}
> +	if (!swapped) {
> +		close(pagemap_fd);
> +		munmap(mapping, page_size);
> +		ksft_test_result_skip("MADV_PAGEOUT did not swap the page\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!pagemap_is_swapped(pagemap_fd, mapping))
> +		ksft_exit_fail_msg("MADV_COLD faulted in a swapped PTE\n");
> +	if (mapping[0] != 1)
> +		ksft_exit_fail_msg("swapped page contents changed\n");
> +
> +	close(pagemap_fd);
> +	munmap(mapping, page_size);
> +	ksft_test_result_pass("MADV_COLD skips a swapped PTE\n");
> +}
> +
> +/*
> + * vmsplice() retains a reference to a THP in a pipe, preventing a partial
> + * MADV_COLD from splitting it. The advice must leave the folio intact while
> + * pinned, then split it normally after the pipe releases the reference.
> + */
> +static void test_partial_pinned_madvise_cold(bool pte_mapped)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping = map_aligned_pages(pmd_size);
> +	struct iovec iov = {
> +		.iov_base = mapping,
> +		.iov_len = page_size,
> +	};
> +	char *buffer;
> +	int pipefd[2];
> +	int retry;
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	if (pte_mapped)
> +		split_pmd_mapping(mapping);
> +	if (pipe(pipefd))
> +		ksft_exit_fail_perror("pipe");
> +	if (vmsplice(pipefd[1], &iov, 1, SPLICE_F_GIFT) != page_size) {
> +		close(pipefd[0]);
> +		close(pipefd[1]);
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("vmsplice could not retain a THP page\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_COLD split a pinned THP\n");
> +
> +	buffer = malloc(page_size);
> +	if (!buffer)
> +		ksft_exit_fail_perror("malloc");
> +	if (read(pipefd[0], buffer, page_size) != page_size)
> +		ksft_exit_fail_perror("read pipe");
> +	free(buffer);
> +	close(pipefd[0]);
> +	close(pipefd[1]);
> +
> +	for (retry = 0; retry < 10; retry++) {
> +		if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +			ksft_exit_fail_perror("MADV_COLD");
> +		if (check_large_folios(mapping, pmd_size, 0, pmd_size))
> +			break;
> +		usleep(10000);
> +	}
> +	if (retry == 10)
> +		ksft_exit_fail_msg("MADV_COLD did not split an unpinned THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("partial MADV_COLD skips a pinned %s THP\n",
> +			      pte_mapped ? "PTE-mapped" : "PMD-mapped");
> +}
> +
> +static void test_partial_pinned_pmd_madvise_cold(void)
> +{
> +	test_partial_pinned_madvise_cold(false);
> +}
> +
> +/*
> + * Pinning a PTE-mapped THP also prevents a partial MADV_COLD from splitting
> + * it. Once the pipe releases the reference, a serial retry must split it.
> + */
> +static void test_partial_pinned_pte_madvise_cold(void)
> +{
> +	test_partial_pinned_madvise_cold(true);
> +}
> +
> +static void *madvise_cold_worker(void *unused)
> +{
> +	int round, i;
> +
> +	(void)unused;
> +	for (round = 0; round < NR_CONCURRENT_ROUNDS; round++) {
> +		pthread_barrier_wait(&concurrent_start_barrier);
> +		for (i = 0; i < NR_CONCURRENT_THPS; i++) {
> +			if (madvise(concurrent_area + i * pmd_size,
> +				    pmd_size / 2, MADV_COLD))
> +				atomic_store(&concurrent_worker_errno, errno);
> +		}
> +		pthread_barrier_wait(&concurrent_done_barrier);
> +	}
> +
> +	return NULL;
> +}
> +
> +/*
> + * Concurrent partial MADV_COLD calls deliberately contend for each THP lock.
> + * Alternate PMD- and PTE-mapped rounds to exercise both split paths. A caller
> + * that loses the trylock race may skip that folio, so retry serially before
> + * asserting the stable interface: all calls succeed, the THPs remain
> + * splittable, and their contents are unchanged.
> + */
> +static void test_concurrent_partial_madvise_cold(void)
> +{
> +	const size_t size = NR_CONCURRENT_THPS * pmd_size;
> +	pthread_t threads[NR_CONCURRENT_THREADS];
> +	int round, i;
> +
> +	concurrent_area = map_aligned_pages(size);
> +	if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS)) {
> +		munmap(concurrent_area, size);
> +		ksft_test_result_skip("could not allocate PMD-sized THPs\n");
> +		return;
> +	}
> +	if (pthread_barrier_init(&concurrent_start_barrier, NULL,
> +				 NR_CONCURRENT_THREADS + 1) ||
> +	    pthread_barrier_init(&concurrent_done_barrier, NULL,
> +				 NR_CONCURRENT_THREADS + 1))
> +		ksft_exit_fail_msg("pthread_barrier_init failed\n");
> +	for (i = 0; i < NR_CONCURRENT_THREADS; i++)
> +		if (pthread_create(&threads[i], NULL, madvise_cold_worker, NULL))
> +			ksft_exit_fail_msg("pthread_create failed\n");
> +
> +	for (round = 0; round < NR_CONCURRENT_ROUNDS; round++) {
> +		if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS))
> +			ksft_exit_fail_msg("round %d: failed to form PMD THPs\n",
> +					   round);
> +		if (round & 1)
> +			for (i = 0; i < NR_CONCURRENT_THPS; i++)
> +				split_pmd_mapping(concurrent_area + i * pmd_size);
> +		pthread_barrier_wait(&concurrent_start_barrier);
> +		pthread_barrier_wait(&concurrent_done_barrier);
> +
> +		if (atomic_load(&concurrent_worker_errno)) {
> +			errno = atomic_load(&concurrent_worker_errno);
> +			ksft_exit_fail_perror("MADV_COLD");
> +		}
> +		for (i = 0; i < NR_CONCURRENT_THPS; i++) {
> +			if (madvise(concurrent_area + i * pmd_size,
> +				    pmd_size / 2, MADV_COLD))
> +				ksft_exit_fail_perror("MADV_COLD retry");
> +		}
> +		if (!check_large_folios(concurrent_area, size, 0, pmd_size))
> +			ksft_exit_fail_msg("round %d: PMD THP remained\n", round);
> +	}
> +	check_memory(concurrent_area, size);
> +
> +	for (i = 0; i < NR_CONCURRENT_THREADS; i++)
> +		pthread_join(threads[i], NULL);
> +	pthread_barrier_destroy(&concurrent_done_barrier);
> +	pthread_barrier_destroy(&concurrent_start_barrier);
> +	munmap(concurrent_area, size);
> +	ksft_test_result_pass("concurrent partial MADV_COLD preserves memory\n");
> +}
> +
> +/*
> + * SHM_LOCK makes a shmem folio unevictable without setting VM_LOCKED on this
> + * VMA. MADV_PAGEOUT must put an isolated folio back on its LRU rather than
> + * reclaiming it.
> + */
> +static void test_madvise_pageout_unevictable(void)
> +{
> +	const size_t page_size = getpagesize();
> +	unsigned long pfn;
> +	uint64_t flags;
> +	char *mapping;
> +	int pagemap_fd, kpageflags_fd;
> +	int shmid;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to read page flags\n");
> +		return;
> +	}
> +	shmid = shmget(IPC_PRIVATE, page_size, IPC_CREAT | 0600);
> +	if (shmid < 0)
> +		ksft_exit_fail_perror("shmget");
> +	if (shmctl(shmid, SHM_LOCK, NULL)) {
> +		shmctl(shmid, IPC_RMID, NULL);
> +		ksft_test_result_skip("could not lock a shmem segment\n");
> +		return;
> +	}
> +	mapping = shmat(shmid, NULL, 0);
> +	if (mapping == (void *)-1)
> +		ksft_exit_fail_perror("shmat");
> +	mapping[0] = 1;
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
> +	if (pagemap_fd < 0 || kpageflags_fd < 0)
> +		ksft_exit_fail_perror("open page flags");
> +
> +	/* Drain the LRU add batch so the locked mapping becomes unevictable. */
> +	if (madvise(mapping, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	pfn = pagemap_get_pfn(pagemap_fd, mapping);
> +	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags) ||
> +	    !(flags & KPF_UNEVICTABLE))
> +		ksft_exit_fail_msg("SHM_LOCK page is not unevictable\n");
> +
> +	if (madvise(mapping, page_size, MADV_PAGEOUT))
> +		ksft_exit_fail_perror("MADV_PAGEOUT");
> +	if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0] != 1)
> +		ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable page\n");
> +
> +	close(kpageflags_fd);
> +	close(pagemap_fd);
> +	shmctl(shmid, SHM_UNLOCK, NULL);
> +	shmdt(mapping);
> +	shmctl(shmid, IPC_RMID, NULL);
> +	ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable page\n");
> +}
> +
> +/*
> + * Exercise the unevictable PAGEOUT path at PMD granularity. SHM_LOCK keeps
> + * the shmem THP resident without marking its VMA VM_LOCKED.
> + */
> +static void test_madvise_pageout_unevictable_thp(void)
> +{
> +	char *mapping;
> +	int pagemap_fd, shmid;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to lock a shmem segment\n");
> +		return;
> +	}
> +	mapping = map_unevictable_shmem(pmd_size, &shmid);
> +	if (mapping == MAP_FAILED)
> +		return;
> +	memset(mapping, 1, pmd_size);
> +	if (!collapse_shmem(mapping, pmd_size, 1)) {
> +		unmap_unevictable_shmem(mapping, shmid);
> +		ksft_test_result_skip("could not allocate an unevictable THP\n");
> +		return;
> +	}
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +
> +	if (madvise(mapping, pmd_size, MADV_PAGEOUT))
> +		ksft_exit_fail_perror("MADV_PAGEOUT");
> +	if (!pagemap_is_populated(pagemap_fd, mapping) ||
> +	    !check_huge_shmem(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	close(pagemap_fd);
> +	unmap_unevictable_shmem(mapping, shmid);
> +	ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable THP\n");
> +}
> +
> +/*
> + * migrate_pages() isolates folios from the LRU before replacing their page
> + * table entries. Race that interval against MADV_PAGEOUT on PMD-mapped THPs.
> + * Either operation may skip a folio owned by the other, but repeated calls
> + * must not report an error or corrupt the mapping.
> + */
> +static void test_madvise_pageout_migration(void)
> +{
> +#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \
> +	defined(SYS_migrate_pages)
> +	unsigned long allowed[NUMA_MASK_LONGS] = {};
> +	unsigned long mask1[NUMA_MASK_LONGS] = {};
> +	unsigned long mask2[NUMA_MASK_LONGS] = {};
> +	const size_t size = NR_MIGRATION_RACE_THPS * pmd_size;
> +	char *mapping;
> +	int node1, node2;
> +	int error;
> +	int shmid;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to lock a shmem segment\n");
> +		return;
> +	}
> +	if (!find_two_memory_nodes(allowed, &node1, &node2)) {
> +		ksft_test_result_skip("requires two allowed NUMA memory nodes\n");
> +		return;
> +	}
> +
> +	numa_mask_set(mask1, node1);
> +	numa_mask_set(mask2, node2);
> +	mapping = map_unevictable_shmem(size, &shmid);
> +	if (mapping == MAP_FAILED)
> +		return;
> +	if (syscall(SYS_mbind, mapping, size, MPOL_BIND, mask1,
> +		    NUMA_MASK_BITS + 1, 0)) {
> +		unmap_unevictable_shmem(mapping, shmid);
> +		ksft_test_result_skip("could not bind shmem to a NUMA node\n");
> +		return;
> +	}
> +	memset(mapping, 1, size);
> +	if (!collapse_shmem(mapping, size, NR_MIGRATION_RACE_THPS)) {
> +		unmap_unevictable_shmem(mapping, shmid);
> +		ksft_test_result_skip("could not allocate unevictable shmem THPs\n");
> +		return;
> +	}
> +
> +	error = race_pageout_with_migration(mapping, size, mask1, mask2);
> +	if (error == ENOSYS) {
> +		unmap_unevictable_shmem(mapping, shmid);
> +		ksft_test_result_skip("NUMA migration is unavailable\n");
> +		return;
> +	}
> +	if (error) {
> +		errno = error;
> +		ksft_exit_fail_perror("MADV_PAGEOUT/migrate_pages race");
> +	}
> +	check_memory(mapping, size);
> +
> +	unmap_unevictable_shmem(mapping, shmid);
> +	ksft_test_result_pass("MADV_PAGEOUT races NUMA migration safely\n");
> +#else
> +	ksft_test_result_skip("NUMA migration system calls are unavailable\n");
> +#endif
> +}
> +
> +static bool activate_folio(char *mapping, char *drain, unsigned long pfn,
> +			   int kpageflags_fd)
> +{
> +	struct iovec local = { .iov_base = drain, .iov_len = 1 };
> +	struct iovec remote = { .iov_base = mapping, .iov_len = 1 };
> +	uint64_t flags;
> +	int retry;
> +
> +	for (retry = 0; retry < 10; retry++) {
> +		if (process_vm_readv(getpid(), &local, 1, &remote, 1, 0) != 1)
> +			ksft_exit_fail_perror("process_vm_readv");
> +		/* Drain the activation batch without advising the target folio. */
> +		if (madvise(drain, getpagesize(), MADV_COLD))
> +			ksft_exit_fail_perror("MADV_COLD");
> +		if (pageflags_get(pfn, kpageflags_fd, &flags))
> +			ksft_exit_fail_perror("read kpageflags");
> +		if (flags & KPF_ACTIVE)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * Repeated GUP accesses promote an inactive folio on the traditional LRU.
> + * MADV_COLD must deactivate that folio while preserving its contents.
> + */
> +static void test_madvise_cold_active_folio(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping = map_aligned_pages(page_size);
> +	char *drain = map_aligned_pages(page_size);
> +	cpu_set_t old_mask, mask;
> +	int pagemap_fd, kpageflags_fd;
> +	unsigned long pfn;
> +	uint64_t flags;
> +	int cpu;
> +
> +	if (geteuid()) {
> +		munmap(drain, page_size);
> +		munmap(mapping, page_size);
> +		ksft_test_result_skip("requires root to read page flags\n");
> +		return;
> +	}
> +	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("sched_getaffinity");
> +	cpu = sched_getcpu();
> +	if (cpu < 0)
> +		ksft_exit_fail_perror("sched_getcpu");
> +	CPU_ZERO(&mask);
> +	CPU_SET(cpu, &mask);
> +	if (sched_setaffinity(0, sizeof(mask), &mask))
> +		ksft_exit_fail_perror("sched_setaffinity");
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
> +	if (pagemap_fd < 0 || kpageflags_fd < 0)
> +		ksft_exit_fail_perror("open page flags");
> +	pfn = pagemap_get_pfn(pagemap_fd, mapping);
> +	if (pfn == -1ul)
> +		ksft_exit_fail_msg("could not read page PFN\n");
> +
> +	if (!activate_folio(mapping, drain, pfn, kpageflags_fd)) {
> +		close(kpageflags_fd);
> +		close(pagemap_fd);
> +		sched_setaffinity(0, sizeof(old_mask), &old_mask);
> +		munmap(drain, page_size);
> +		munmap(mapping, page_size);
> +		ksft_test_result_skip("could not activate a folio\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, page_size, MADV_COLD) ||
> +	    madvise(drain, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (pageflags_get(pfn, kpageflags_fd, &flags))
> +		ksft_exit_fail_perror("read kpageflags");
> +	if ((flags & KPF_ACTIVE) || mapping[0] != 1)
> +		ksft_exit_fail_msg("MADV_COLD did not deactivate the folio\n");
> +
> +	close(kpageflags_fd);
> +	close(pagemap_fd);
> +	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("restore affinity");
> +	munmap(drain, page_size);
> +	munmap(mapping, page_size);
> +	ksft_test_result_pass("MADV_COLD deactivates an active folio\n");
> +}
> +
> +struct remote_fault_data {
> +	char *mapping;
> +	pthread_barrier_t barrier;
> +	int cpu;
> +	int error;
> +};
> +
> +static void *remote_fault_worker(void *arg)
> +{
> +	struct remote_fault_data *data = arg;
> +	cpu_set_t mask;
> +
> +	CPU_ZERO(&mask);
> +	CPU_SET(data->cpu, &mask);
> +	if (sched_setaffinity(0, sizeof(mask), &mask))
> +		data->error = errno;
> +	else
> +		data->mapping[0] = 1;
> +	pthread_barrier_wait(&data->barrier);
> +	pthread_barrier_wait(&data->barrier);
> +	return NULL;
> +}
> +
> +/*
> + * A folio faulted on another CPU can remain in that CPU's pending LRU batch.
> + * MADV_COLD drains only the calling CPU and must safely skip the non-LRU
> + * folio without changing the mapping.
> + */
> +static void test_madvise_cold_remote_lru_batch(void)
> +{
> +	const size_t page_size = getpagesize();
> +	struct remote_fault_data data = { .cpu = -1 };
> +	char *mapping;
> +	cpu_set_t old_mask, mask;
> +	pthread_t thread;
> +	unsigned long pfn;
> +	uint64_t flags;
> +	int pagemap_fd, kpageflags_fd;
> +	int cpu, main_cpu = -1;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to read page flags\n");
> +		return;
> +	}
> +	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("sched_getaffinity");
> +	for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
> +		if (!CPU_ISSET(cpu, &old_mask))
> +			continue;
> +		if (main_cpu < 0) {
> +			main_cpu = cpu;
> +		} else {
> +			data.cpu = cpu;
> +			break;
> +		}
> +	}
> +	if (main_cpu < 0 || data.cpu < 0) {
> +		ksft_test_result_skip("requires two CPUs\n");
> +		return;
> +	}
> +	CPU_ZERO(&mask);
> +	CPU_SET(main_cpu, &mask);
> +	if (sched_setaffinity(0, sizeof(mask), &mask))
> +		ksft_exit_fail_perror("sched_setaffinity");
> +
> +	mapping = mmap(NULL, page_size, PROT_READ | PROT_WRITE,
> +		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +	data.mapping = mapping;
> +	if (pthread_barrier_init(&data.barrier, NULL, 2) ||
> +	    pthread_create(&thread, NULL, remote_fault_worker, &data))
> +		ksft_exit_fail_msg("could not start fault worker\n");
> +	pthread_barrier_wait(&data.barrier);
> +	if (data.error) {
> +		errno = data.error;
> +		ksft_exit_fail_perror("worker sched_setaffinity");
> +	}
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
> +	if (pagemap_fd < 0 || kpageflags_fd < 0)
> +		ksft_exit_fail_perror("open page flags");
> +	pfn = pagemap_get_pfn(pagemap_fd, mapping);
> +	if (pfn == -1ul || pageflags_get(pfn, kpageflags_fd, &flags))
> +		ksft_exit_fail_msg("could not read page flags\n");
> +	if (flags & KPF_LRU) {
> +		pthread_barrier_wait(&data.barrier);
> +		pthread_join(thread, NULL);
> +		pthread_barrier_destroy(&data.barrier);
> +		close(kpageflags_fd);
> +		close(pagemap_fd);
> +		sched_setaffinity(0, sizeof(old_mask), &old_mask);
> +		munmap(mapping, page_size);
> +		ksft_test_result_skip("remote LRU batch was already drained\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (mapping[0] != 1 || !pagemap_is_populated(pagemap_fd, mapping))
> +		ksft_exit_fail_msg("MADV_COLD changed a pending-LRU page\n");
> +
> +	pthread_barrier_wait(&data.barrier);
> +	pthread_join(thread, NULL);
> +	pthread_barrier_destroy(&data.barrier);
> +	close(kpageflags_fd);
> +	close(pagemap_fd);
> +	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("restore affinity");
> +	munmap(mapping, page_size);
> +	ksft_test_result_pass("MADV_COLD skips a remote pending-LRU page\n");
> +}
> +
> +/*
> + * Exercise the same active-folio transition through a huge PMD, where the
> + * page-table aging operation differs from the PTE implementation.
> + */
> +static void test_pmd_madvise_cold_active_folio(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +	char *drain = map_aligned_pages(page_size);
> +	cpu_set_t old_mask, mask;
> +	int pagemap_fd, kpageflags_fd;
> +	unsigned long pfn;
> +	uint64_t flags;
> +	int cpu, fd, retry;
> +	bool active = false;
> +
> +	if (geteuid()) {
> +		munmap(drain, page_size);
> +		ksft_test_result_skip("requires root to read page flags\n");
> +		return;
> +	}
> +	fd = memfd_create("madvise-active-thp", 0);
> +	if (fd < 0)
> +		ksft_exit_fail_perror("memfd_create");
> +	if (ftruncate(fd, pmd_size))
> +		ksft_exit_fail_perror("ftruncate");
> +	mapping = map_aligned_file(fd, pmd_size, MAP_SHARED);
> +	memset(mapping, 1, pmd_size);
> +	if (msync(mapping, pmd_size, MS_SYNC) ||
> +	    !collapse_shmem(mapping, pmd_size, 1)) {
> +		munmap(drain, page_size);
> +		munmap(mapping, pmd_size);
> +		close(fd);
> +		ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n");
> +		return;
> +	}
> +	if (sched_getaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("sched_getaffinity");
> +	cpu = sched_getcpu();
> +	if (cpu < 0)
> +		ksft_exit_fail_perror("sched_getcpu");
> +	CPU_ZERO(&mask);
> +	CPU_SET(cpu, &mask);
> +	if (sched_setaffinity(0, sizeof(mask), &mask))
> +		ksft_exit_fail_perror("sched_setaffinity");
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags_fd = open("/proc/kpageflags", O_RDONLY);
> +	if (pagemap_fd < 0 || kpageflags_fd < 0)
> +		ksft_exit_fail_perror("open page flags");
> +	pfn = pagemap_get_pfn(pagemap_fd, mapping);
> +	if (pfn == -1ul)
> +		ksft_exit_fail_msg("could not read THP PFN\n");
> +
> +	for (retry = 0; retry < 10; retry++) {
> +		if (pread(fd, drain, 1, 0) != 1)
> +			ksft_exit_fail_perror("pread");
> +		if (madvise(drain, page_size, MADV_COLD))
> +			ksft_exit_fail_perror("MADV_COLD");
> +		if (pageflags_get(pfn, kpageflags_fd, &flags))
> +			ksft_exit_fail_perror("read kpageflags");
> +		if (flags & KPF_ACTIVE) {
> +			active = true;
> +			break;
> +		}
> +	}
> +	if (!active) {
> +		close(kpageflags_fd);
> +		close(pagemap_fd);
> +		sched_setaffinity(0, sizeof(old_mask), &old_mask);
> +		munmap(drain, page_size);
> +		munmap(mapping, pmd_size);
> +		close(fd);
> +		ksft_test_result_skip("could not activate a THP\n");
> +		return;
> +	}
> +
> +	if (madvise(mapping, pmd_size, MADV_COLD) ||
> +	    madvise(drain, page_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (pageflags_get(pfn, kpageflags_fd, &flags))
> +		ksft_exit_fail_perror("read kpageflags");
> +	if (flags & KPF_ACTIVE)
> +		ksft_exit_fail_msg("MADV_COLD did not deactivate the THP\n");
> +	if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_COLD split an active THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	close(kpageflags_fd);
> +	close(pagemap_fd);
> +	if (sched_setaffinity(0, sizeof(old_mask), &old_mask))
> +		ksft_exit_fail_perror("restore affinity");
> +	munmap(drain, page_size);
> +	munmap(mapping, pmd_size);
> +	close(fd);
> +	ksft_test_result_pass("MADV_COLD deactivates an active THP\n");
> +}
> +
> +/*
> + * MADV_PAGEOUT on half of a PMD-mapped THP must split the folio before
> + * reclaim so the unadvised half is not reclaimed as part of the THP.
> + */
> +static void test_partial_pmd_madvise_pageout(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	pageout_half_thp(mapping);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("partial MADV_PAGEOUT splits a PMD-mapped THP\n");
> +}
> +
> +/*
> + * mprotect() replaces the huge PMD with PTEs without splitting the physical
> + * THP. MADV_PAGEOUT on half of that PTE-mapped THP must split the folio.
> + */
> +static void test_partial_pte_madvise_pageout(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	split_pmd_mapping(mapping);
> +	pageout_half_thp(mapping);
> +
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("partial MADV_PAGEOUT splits a PTE-mapped THP\n");
> +}
> +
> +static pid_t fork_waiting_child(int pipefd[2])
> +{
> +	pid_t pid;
> +
> +	if (pipe(pipefd))
> +		ksft_exit_fail_perror("pipe");
> +	pid = fork();
> +	if (pid < 0)
> +		ksft_exit_fail_perror("fork");
> +	if (!pid) {
> +		char byte;
> +
> +		close(pipefd[1]);
> +		while (read(pipefd[0], &byte, 1) < 0 && errno == EINTR)
> +			;
> +		_exit(0);
> +	}
> +	close(pipefd[0]);
> +	return pid;
> +}
> +
> +/*
> + * fork() gives the THP another mapping, then mprotect() PTE-maps it in the
> + * parent. Assert that partial MADV_COLD leaves the shared physical THP and
> + * its contents intact.
> + */
> +static void test_shared_pte_mapped_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +	int pipefd[2], status;
> +	pid_t pid;
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	pid = fork_waiting_child(pipefd);
> +
> +	split_pmd_mapping(mapping);
> +	if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_COLD split a shared large folio\n");
> +	check_memory(mapping, pmd_size);
> +
> +	close(pipefd[1]);
> +	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
> +	    WEXITSTATUS(status))
> +		ksft_exit_fail_msg("child process failed\n");
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("MADV_COLD skips a shared PTE-mapped THP\n");
> +}
> +
> +/*
> + * A full-range PTE walk reaches the mapcount check rather than the partial
> + * folio split check. A second mapping must still prevent MADV_COLD from
> + * operating on the shared physical THP.
> + */
> +static void test_full_shared_pte_mapped_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +	int pipefd[2], status;
> +	pid_t pid;
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	pid = fork_waiting_child(pipefd);
> +
> +	split_pmd_mapping(mapping);
> +	if (madvise(mapping, pmd_size, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_COLD split a shared large folio\n");
> +	check_memory(mapping, pmd_size);
> +
> +	close(pipefd[1]);
> +	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
> +	    WEXITSTATUS(status))
> +		ksft_exit_fail_msg("child process failed\n");
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("full MADV_COLD skips a shared PTE-mapped THP\n");
> +}
> +
> +/*
> + * fork() gives a PMD-mapped THP another mapping. Assert that partial
> + * MADV_COLD leaves the shared physical THP and its contents intact.
> + */
> +static void test_shared_pmd_madvise_cold(void)
> +{
> +	char *mapping = map_aligned_pages(pmd_size);
> +	int pipefd[2], status;
> +	pid_t pid;
> +
> +	if (!collapse_all(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		ksft_test_result_skip("could not allocate a PMD-sized THP\n");
> +		return;
> +	}
> +	pid = fork_waiting_child(pipefd);
> +
> +	if (madvise(mapping, pmd_size / 2, MADV_COLD))
> +		ksft_exit_fail_perror("MADV_COLD");
> +	if (!check_large_folios(mapping, pmd_size, 1, pmd_size))
> +		ksft_exit_fail_msg("MADV_COLD split a shared PMD-mapped THP\n");
> +	check_memory(mapping, pmd_size);
> +
> +	close(pipefd[1]);
> +	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
> +	    WEXITSTATUS(status))
> +		ksft_exit_fail_msg("child process failed\n");
> +	munmap(mapping, pmd_size);
> +	ksft_test_result_pass("MADV_COLD skips a shared PMD-mapped THP\n");
> +}
> +
> +static char *map_readonly_file(size_t size, int flags, int *fd)
> +{
> +	char template[] = "/tmp/madvise-pageout-XXXXXX";
> +	char *mapping;
> +
> +	*fd = mkstemp(template);
> +	if (*fd < 0)
> +		ksft_exit_fail_perror("mkstemp");
> +	if (unlink(template) || ftruncate(*fd, size) || fchmod(*fd, 0400))
> +		ksft_exit_fail_perror("prepare file");
> +	mapping = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, *fd, 0);
> +	if (mapping == MAP_FAILED)
> +		ksft_exit_fail_perror("mmap");
> +
> +	return mapping;
> +}
> +
> +static bool pageout_until_evicted(char *mapping, size_t size, int pagemap_fd)
> +{
> +	int retry;
> +
> +	for (retry = 0; retry < 100; retry++) {
> +		if (madvise(mapping, size, MADV_PAGEOUT))
> +			ksft_exit_fail_perror("MADV_PAGEOUT");
> +		if (!pagemap_is_populated(pagemap_fd, mapping))
> +			return true;
> +		usleep(10000);
> +	}
> +	return false;
> +}
> +
> +/*
> + * The owner of a file may page out its clean page-cache pages. Assert that
> + * MADV_PAGEOUT removes the populated PTE and preserves the file contents.
> + */
> +static void test_pageout_file(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +	int pagemap_fd;
> +	bool evicted;
> +	int fd;
> +
> +	mapping = map_readonly_file(page_size, MAP_PRIVATE, &fd);
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +	if (mapping[0])
> +		ksft_exit_fail_msg("new file is not zero-filled\n");
> +
> +	evicted = pageout_until_evicted(mapping, page_size, pagemap_fd);
> +	if (mapping[0])
> +		ksft_exit_fail_msg("file mapping contents changed\n");
> +
> +	close(pagemap_fd);
> +	munmap(mapping, page_size);
> +	close(fd);
> +	if (!evicted) {
> +		ksft_test_result_skip("MADV_PAGEOUT did not evict the file page\n");
> +		return;
> +	}
> +	ksft_test_result_pass("MADV_PAGEOUT evicts an authorized file page\n");
> +}
> +
> +static int pageout_shared_file_without_permission(char *mapping,
> +						  size_t page_size)
> +{
> +	int pagemap_fd;
> +
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		return KSFT_FAIL;
> +	if (setgid(65534) || setuid(65534))
> +		return KSFT_FAIL;
> +	if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping))
> +		return KSFT_FAIL;
> +	if (madvise(mapping, page_size, MADV_PAGEOUT))
> +		return KSFT_FAIL;
> +	if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0])
> +		return KSFT_FAIL;
> +	return KSFT_PASS;
> +}
> +
> +/*
> + * A caller without file write permission may not page out a shared file
> + * mapping. Assert that MADV_PAGEOUT succeeds without evicting its file page.
> + */
> +static void test_pageout_unauthorized_shared_file(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +	int fd, status;
> +	pid_t pid;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to change credentials\n");
> +		return;
> +	}
> +
> +	mapping = map_readonly_file(page_size, MAP_SHARED, &fd);
> +
> +	pid = fork();
> +	if (pid < 0)
> +		ksft_exit_fail_perror("fork");
> +	if (!pid)
> +		_exit(pageout_shared_file_without_permission(mapping, page_size));
> +
> +	if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status) ||
> +	    WEXITSTATUS(status) != KSFT_PASS)
> +		ksft_exit_fail_msg("unauthorized child evicted a shared file page\n");
> +	munmap(mapping, page_size);
> +	close(fd);
> +	ksft_test_result_pass("MADV_PAGEOUT skips an unauthorized shared file\n");
> +}
> +
> +/*
> + * An unauthorized private mapping may page out anonymous COW pages, but not
> + * its file folios. Assert that partial MADV_PAGEOUT filters a large shmem
> + * folio before attempting to split or reclaim it.
> + */
> +static void test_pageout_anon_only_large_folio(void)
> +{
> +	char *mapping, *shared;
> +	gid_t old_egid;
> +	int fd;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to change credentials\n");
> +		return;
> +	}
> +	fd = memfd_create("madvise-pageout-large", 0);
> +	if (fd < 0)
> +		ksft_exit_fail_perror("memfd_create");
> +	if (ftruncate(fd, pmd_size))
> +		ksft_exit_fail_perror("ftruncate");
> +	shared = map_aligned_file(fd, pmd_size, MAP_SHARED);
> +	memset(shared, 1, pmd_size);
> +	if (msync(shared, pmd_size, MS_SYNC))
> +		ksft_exit_fail_perror("msync");
> +	munmap(shared, pmd_size);
> +
> +	mapping = map_aligned_file(fd, pmd_size, MAP_PRIVATE);
> +	if (!collapse_shmem(mapping, pmd_size, 1)) {
> +		munmap(mapping, pmd_size);
> +		close(fd);
> +		ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n");
> +		return;
> +	}
> +	if (fchmod(fd, 0400))
> +		ksft_exit_fail_perror("fchmod");
> +
> +	/* Retain saved uid 0 so this process can restore its credentials. */
> +	old_egid = getegid();
> +	if (setegid(65534) || seteuid(65534))
> +		ksft_exit_fail_perror("drop privileges");
> +	if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT))
> +		ksft_exit_fail_perror("MADV_PAGEOUT");
> +	if (seteuid(0) || setegid(old_egid))
> +		ksft_exit_fail_perror("restore privileges");
> +
> +	if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size) ||
> +	    mapping[0] != 1 || mapping[pmd_size - 1] != 1)
> +		ksft_exit_fail_msg("MADV_PAGEOUT changed a protected large folio\n");
> +	munmap(mapping, pmd_size);
> +	close(fd);
> +	ksft_test_result_pass("MADV_PAGEOUT filters a protected large folio\n");
> +}
> +
> +/*
> + * A private file mapping can contain both anonymous COW and file-backed
> + * pages. As a caller without file write permission, assert that PAGEOUT swaps
> + * the COW page but leaves the file-backed page resident.
> + */
> +static void test_pageout_anon_only(void)
> +{
> +	const size_t page_size = getpagesize();
> +	char *mapping;
> +	int pagemap_fd;
> +	bool swapped;
> +	int fd;
> +
> +	if (geteuid()) {
> +		ksft_test_result_skip("requires root to change credentials\n");
> +		return;
> +	}
> +
> +	mapping = map_readonly_file(2 * page_size, MAP_PRIVATE, &fd);
> +	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
> +	if (pagemap_fd < 0)
> +		ksft_exit_fail_perror("open pagemap");
> +
> +	if (setgid(65534) || setuid(65534))
> +		ksft_exit_fail_perror("drop privileges");
> +	if (mapping[0] || mapping[page_size])
> +		ksft_exit_fail_msg("new file is not zero-filled\n");
> +	mapping[0] = 1;
> +	if (madvise(mapping, 2 * page_size, MADV_PAGEOUT))
> +		ksft_exit_fail_perror("MADV_PAGEOUT");
> +	if (!pagemap_is_populated(pagemap_fd, mapping + page_size))
> +		ksft_exit_fail_msg("MADV_PAGEOUT evicted a protected file page\n");
> +	swapped = pagemap_is_swapped(pagemap_fd, mapping);
> +	if (mapping[0] != 1 || mapping[page_size])
> +		ksft_exit_fail_msg("private file mapping contents changed\n");
> +
> +	close(pagemap_fd);
> +	munmap(mapping, 2 * page_size);
> +	close(fd);
> +	if (!swapped) {
> +		ksft_test_result_skip("MADV_PAGEOUT did not swap the COW page\n");
> +		return;
> +	}
> +	ksft_test_result_pass("MADV_PAGEOUT filters private file pages\n");
> +}
> +
> +int main(void)
> +{
> +	pmd_size = read_pmd_pagesize();
> +
> +	ksft_print_header();
> +	ksft_set_plan(28);
> +	if (!pmd_size)
> +		ksft_exit_skip("PMD-sized THPs are not supported\n");
> +
> +	test_full_pmd_madvise_cold();
> +	test_repeated_pmd_madvise_cold();
> +	test_partial_pmd_madvise_cold();
> +	test_pte_mapped_madvise_cold();
> +	test_madvise_cold_pte_hole();
> +	test_madvise_cold_zero_page();
> +	test_madvise_cold_huge_zero_page();
> +	test_madvise_lru_locked_vma();
> +	test_shared_pmd_madvise_cold();
> +	test_full_pmd_madvise_pageout();
> +	test_madvise_cold_swapped_pte();
> +	test_partial_pinned_pmd_madvise_cold();
> +	test_partial_pinned_pte_madvise_cold();
> +	test_concurrent_partial_madvise_cold();
> +	test_madvise_pageout_unevictable();
> +	test_madvise_pageout_unevictable_thp();
> +	test_madvise_pageout_migration();
> +	test_madvise_cold_active_folio();
> +	test_madvise_cold_remote_lru_batch();
> +	test_pmd_madvise_cold_active_folio();
> +	test_partial_pmd_madvise_pageout();
> +	test_partial_pte_madvise_pageout();
> +	test_shared_pte_mapped_madvise_cold();
> +	test_full_shared_pte_mapped_madvise_cold();
> +	test_pageout_file();
> +	test_pageout_unauthorized_shared_file();
> +	test_pageout_anon_only_large_folio();
> +	test_pageout_anon_only();
> +	ksft_finished();
> +}
> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
> index 6990485b1a9da..bc136ead10f91 100755
> --- a/tools/testing/selftests/mm/run_vmtests.sh
> +++ b/tools/testing/selftests/mm/run_vmtests.sh
> @@ -51,6 +51,8 @@ separated by spaces:
>  	hmm smoke tests
>  - madv_guard
>  	test madvise(2) MADV_GUARD_INSTALL and MADV_GUARD_REMOVE options
> +- madvise
> +	test MADV_COLD and MADV_PAGEOUT
>  - madv_populate
>  	test memadvise(2) MADV_POPULATE_{READ,WRITE} options
>  - memfd_secret
> @@ -327,6 +329,9 @@ CATEGORY="hmm" run_test bash ./test_hmm.sh smoke
>  # MADV_GUARD_INSTALL and MADV_GUARD_REMOVE tests
>  CATEGORY="madv_guard" run_test ./guard-regions
>
> +# MADV_COLD and MADV_PAGEOUT tests
> +CATEGORY="madvise" run_test ./madvise
> +
>  # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
>  CATEGORY="madv_populate" run_test ./madv_populate
>
> diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c
> index 31d331c1c4521..af0a7f65bfd9f 100644
> --- a/tools/testing/selftests/mm/vm_util.c
> +++ b/tools/testing/selftests/mm/vm_util.c
> @@ -373,8 +373,8 @@ static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages,
>  	return thp == (nr_hpages * (hpage_size >> 10));
>  }
>
> -static bool check_large_folios(void *addr, size_t len, int nr_hpages,
> -		uint64_t hpage_size)
> +bool check_large_folios(void *addr, size_t len, int nr_hpages,
> +			uint64_t hpage_size)
>  {
>  	int order = 0, pagesize = getpagesize();
>  	unsigned int nr_pages = hpage_size / pagesize;
> diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h
> index 072a6c756c517..33ae68d721df9 100644
> --- a/tools/testing/selftests/mm/vm_util.h
> +++ b/tools/testing/selftests/mm/vm_util.h
> @@ -22,10 +22,14 @@
>  #define PM_SWAP                       BIT_ULL(62)
>  #define PM_PRESENT                    BIT_ULL(63)
>
> +#define KPF_LRU                       BIT_ULL(5)
> +#define KPF_ACTIVE                    BIT_ULL(6)
>  #define KPF_COMPOUND_HEAD             BIT_ULL(15)
>  #define KPF_COMPOUND_TAIL             BIT_ULL(16)
> +#define KPF_UNEVICTABLE               BIT_ULL(18)
>  #define KPF_HWPOISON                  BIT_ULL(19)
>  #define KPF_THP                       BIT_ULL(22)
> +#define KPF_ZERO_PAGE                 BIT_ULL(24)
>  /*
>   * Ignore the checkpatch warning, we must read from x but don't want to do
>   * anything with it in order to trigger a read page fault. We therefore must use
> @@ -97,6 +101,8 @@ unsigned long rss_anon(void);
>  bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
>  bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
>  bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
> +bool check_large_folios(void *addr, size_t len, int nr_hpages,
> +			uint64_t hpage_size);
>  int64_t allocate_transhuge(void *ptr, int pagemap_fd);
>  int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags);
>  int gather_folio_orders(char *vaddr_start, size_t len,
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

Re: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[Gregory Price]

On Wed, Sep 23, 2026 at 03:26:05PM +0100, Lorenzo Stoakes (ARM) wrote:
> 
> I think in general we want to avoid putting stress tests as part of the
> selftests in general? Or at least if they might take a long time to run or
> excessive memory usage, etc.
> 
> Though that might be benchmarks, as we have the THP stress tests so maybe it's
> OK?
> 
> >
> > Assisted-by: LLM
> > Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> 
> In any case I really do think this should be split out into smaller parts, this
> is a _gigantic_ change :)
>

Absolutely, but alas my poor email hosting throttles me if i send too
many emails - and sending a 38-part series where the important part is
the endpoint of the refactor moreso than the individual tests...
instant 24-hour timeout.

I'm working on a solution.  Need better hosting.

> Since you list a whole bunch of different test areas and you're exercising
> different things you should be able to split this out logically across those I
> think?

As mentioned, this was originally ~28 patches split out, but i think
ultimately only the contractual tests should be shipped.

Maybe there's an argument for keeping some of the implementation
details - they're fairly easy to find in the code below for individual
inspection.

Examples

contractual: These tests won't change behavior based on implementation
 - authorized file PAGEOUT evicts a clean file page;
 - unauthorized shared-file PAGEOUT leaves the page resident; and
 - private-file PAGEOUT reclaims a COW page but preserves a file page.

implementation detail: behavior can change without breaking a contract
 - full and partial MADV_COLD on a PTE-mapped THP

~Gregory

Re: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[Lorenzo Stoakes (ARM)]

On Wed, Sep 23, 2026 at 10:44:17AM -0400, Gregory Price wrote:
> On Wed, Sep 23, 2026 at 03:26:05PM +0100, Lorenzo Stoakes (ARM) wrote:
> >
> > I think in general we want to avoid putting stress tests as part of the
> > selftests in general? Or at least if they might take a long time to run or
> > excessive memory usage, etc.
> >
> > Though that might be benchmarks, as we have the THP stress tests so maybe it's
> > OK?
> >
> > >
> > > Assisted-by: LLM
> > > Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> >
> > In any case I really do think this should be split out into smaller parts, this
> > is a _gigantic_ change :)
> >
>
> Absolutely, but alas my poor email hosting throttles me if i send too
> many emails - and sending a 38-part series where the important part is
> the endpoint of the refactor moreso than the individual tests...
> instant 24-hour timeout.
>
> I'm working on a solution.  Need better hosting.

Thanks! But yeah damn indeed you do! :)

Though larger series can scare people off so maybe better to send the tests
separately?

>
> > Since you list a whole bunch of different test areas and you're exercising
> > different things you should be able to split this out logically across those I
> > think?
>
> As mentioned, this was originally ~28 patches split out, but i think
> ultimately only the contractual tests should be shipped.
>
> Maybe there's an argument for keeping some of the implementation
> details - they're fairly easy to find in the code below for individual
> inspection.
>
> Examples
>
> contractual: These tests won't change behavior based on implementation
>  - authorized file PAGEOUT evicts a clean file page;
>  - unauthorized shared-file PAGEOUT leaves the page resident; and
>  - private-file PAGEOUT reclaims a COW page but preserves a file page.
>
> implementation detail: behavior can change without breaking a contract
>  - full and partial MADV_COLD on a PTE-mapped THP

Yeah doesn't sound unreasonable.

>
> ~Gregory

--
Cheers, Lorenzo

Re: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[David Hildenbrand (Arm)]

On 9/23/26 16:46, Lorenzo Stoakes (ARM) wrote:
> On Wed, Sep 23, 2026 at 10:44:17AM -0400, Gregory Price wrote:
>> On Wed, Sep 23, 2026 at 03:26:05PM +0100, Lorenzo Stoakes (ARM) wrote:
>>>
>>> I think in general we want to avoid putting stress tests as part of the
>>> selftests in general? Or at least if they might take a long time to run or
>>> excessive memory usage, etc.
>>>
>>> Though that might be benchmarks, as we have the THP stress tests so maybe it's
>>> OK?
>>>
>>>
>>> In any case I really do think this should be split out into smaller parts, this
>>> is a _gigantic_ change :)
>>>
>>
>> Absolutely, but alas my poor email hosting throttles me if i send too
>> many emails - and sending a 38-part series where the important part is
>> the endpoint of the refactor moreso than the individual tests...
>> instant 24-hour timeout.
>>
>> I'm working on a solution.  Need better hosting.
> 
> Thanks! But yeah damn indeed you do! :)
> 
> Though larger series can scare people off so maybe better to send the tests
> separately?

In any case, we have to break this down into reviewable pieces. We neglected
selftests for too long.

... and there are some schlopped tests in there that we really shouldn't have as
they are way too special and hard to maintain.

Like this monstrosity:

+/*
+ * A folio faulted on another CPU can remain in that CPU's pending LRU batch.
+ * MADV_COLD drains only the calling CPU and must safely skip the non-LRU
+ * folio without changing the mapping.
+ */
+static void test_madvise_cold_remote_lru_batch(void)
+{

-- 
Cheers,

David

Re: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT

[Gregory Price]

On Thu, Sep 24, 2026 at 01:32:25PM +0200, David Hildenbrand (Arm) wrote:
> On 9/23/26 16:46, Lorenzo Stoakes (ARM) wrote:
> >>
> >> I'm working on a solution.  Need better hosting.
> > 
> > Thanks! But yeah damn indeed you do! :)
> > 
> > Though larger series can scare people off so maybe better to send the tests
> > separately?
> 
> In any case, we have to break this down into reviewable pieces. We neglected
> selftests for too long.
> 
> ... and there are some schlopped tests in there that we really shouldn't have as
> they are way too special and hard to maintain.
> 
> Like this monstrosity:
> 
> +/*
> + * A folio faulted on another CPU can remain in that CPU's pending LRU batch.
> + * MADV_COLD drains only the calling CPU and must safely skip the non-LRU
> + * folio without changing the mapping.
> + */
> +static void test_madvise_cold_remote_lru_batch(void)
> +{
>

Yeah the intent of these particular tests was to ensure I capture ever
weird branch in the code before i went refactoring it so I ensured there
was no loss of fidelity.

I think we can reasonably delete ~15-20 of these tests.  That said, the
code itself is not schlopp, but I believe in showing the work.

~Gregory

[PATCH 02/10] mm/madvise: name the shared LRU PMD callback

[Gregory Price]

The MADV_COLD and MADV_PAGEOUT page-walk callback is named after its PTE
implementation even though it is registered as a PMD callback and serves
both LRU operations.

Rename it to madvise_lru_pmd_entry() before separating the PMD and PTE
paths.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 00b1be655a8b5..83d54ab385da8 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -361,9 +361,8 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
 				     FPB_MERGE_YOUNG_DIRTY);
 }
 
-static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
-				unsigned long addr, unsigned long end,
-				struct mm_walk *walk)
+static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
+		unsigned long end, struct mm_walk *walk)
 {
 	struct madvise_walk_private *private = walk->private;
 	struct mmu_gather *tlb = private->tlb;
@@ -581,7 +580,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 }
 
 static const struct mm_walk_ops cold_walk_ops = {
-	.pmd_entry = madvise_cold_or_pageout_pte_range,
+	.pmd_entry = madvise_lru_pmd_entry,
 	.walk_lock = PGWALK_RDLOCK,
 };
 
-- 
2.53.0-Meta

Re: [PATCH 02/10] mm/madvise: name the shared LRU PMD callback

[Lorenzo Stoakes (ARM)]

On Tue, Sep 22, 2026 at 07:58:22PM -0400, Gregory Price wrote:
> The MADV_COLD and MADV_PAGEOUT page-walk callback is named after its PTE
> implementation even though it is registered as a PMD callback and serves
> both LRU operations.
>
> Rename it to madvise_lru_pmd_entry() before separating the PMD and PTE
> paths.
>
> No functional change intended.
>
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>

Oh yeah wow I was thinking 'reasonable in that it is the PTE range which the PMD
entry spans' but err, no, it does handle THP PMDs too so :)

And LRU is a resonable name, it does seem to me that these are the only madvise
operations that qualify so all makes sense.

So:

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

> ---
>  mm/madvise.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 00b1be655a8b5..83d54ab385da8 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -361,9 +361,8 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
>  				     FPB_MERGE_YOUNG_DIRTY);
>  }
>
> -static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> -				unsigned long addr, unsigned long end,
> -				struct mm_walk *walk)
> +static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
> +		unsigned long end, struct mm_walk *walk)
>  {
>  	struct madvise_walk_private *private = walk->private;
>  	struct mmu_gather *tlb = private->tlb;
> @@ -581,7 +580,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
>  }
>
>  static const struct mm_walk_ops cold_walk_ops = {
> -	.pmd_entry = madvise_cold_or_pageout_pte_range,
> +	.pmd_entry = madvise_lru_pmd_entry,
>  	.walk_lock = PGWALK_RDLOCK,
>  };
>
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

[PATCH 03/10] mm/madvise: factor shared LRU folio handling

[Gregory Price]

The huge-PMD and PTE paths duplicate folio filtering, reference clearing,
deactivation and pageout isolation. Keeping both copies synchronized
obscures the page-table-specific control flow.

Factor the common filtering and LRU operation into helpers. Keep the
lock-before-reference sequence visible at each split site because an early
reference can prevent split_folio() from succeeding.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 90 +++++++++++++++++++++++++---------------------------
 1 file changed, 43 insertions(+), 47 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 83d54ab385da8..c345fef23f15d 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -361,6 +361,39 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
 				     FPB_MERGE_YOUNG_DIRTY);
 }
 
+static inline void
+madvise_lru_folio(struct folio *folio, bool pageout,
+		struct list_head *folio_list)
+{
+	/*
+	 * Clear references before deactivating or reclaiming the folio. This can
+	 * make idle-page tracking miss recent accesses.
+	 */
+	folio_clear_referenced(folio);
+	folio_test_clear_young(folio);
+	if (folio_test_active(folio))
+		folio_set_workingset(folio);
+
+	if (!pageout) {
+		folio_deactivate(folio);
+		return;
+	}
+
+	if (!folio_isolate_lru(folio))
+		return;
+	if (folio_test_unevictable(folio))
+		folio_putback_lru(folio);
+	else
+		list_add(&folio->lru, folio_list);
+}
+
+static bool madvise_lru_folio_is_filtered(struct folio *folio,
+		bool pageout_anon_only)
+{
+	return folio_maybe_mapped_shared(folio) ||
+	       (pageout_anon_only && !folio_test_anon(folio));
+}
+
 static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		unsigned long end, struct mm_walk *walk)
 {
@@ -373,15 +406,14 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 	spinlock_t *ptl;
 	struct folio *folio = NULL;
 	LIST_HEAD(folio_list);
-	bool pageout_anon_only_filter;
 	unsigned int batch_count = 0;
+	bool pageout_anon_only;
 	int nr;
 
 	if (fatal_signal_pending(current))
 		return -EINTR;
-
-	pageout_anon_only_filter = pageout && !vma_is_anonymous(vma) &&
-					!can_do_file_pageout(vma);
+	pageout_anon_only = pageout && !vma_is_anonymous(vma) &&
+				       !can_do_file_pageout(vma);
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	if (pmd_trans_huge(*pmd)) {
@@ -407,11 +439,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		if (folio_is_zone_device(folio))
 			goto huge_unlock;
 
-		/* Do not interfere with other mappings of this folio */
-		if (folio_maybe_mapped_shared(folio))
-			goto huge_unlock;
-
-		if (pageout_anon_only_filter && !folio_test_anon(folio))
+		if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
 			goto huge_unlock;
 
 		if (next - addr != HPAGE_PMD_SIZE) {
@@ -437,19 +465,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 			tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
 		}
 
-		folio_clear_referenced(folio);
-		folio_test_clear_young(folio);
-		if (folio_test_active(folio))
-			folio_set_workingset(folio);
-		if (pageout) {
-			if (folio_isolate_lru(folio)) {
-				if (folio_test_unevictable(folio))
-					folio_putback_lru(folio);
-				else
-					list_add(&folio->lru, &folio_list);
-			}
-		} else
-			folio_deactivate(folio);
+		madvise_lru_folio(folio, pageout, &folio_list);
 huge_unlock:
 		spin_unlock(ptl);
 		if (pageout)
@@ -502,9 +518,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 			if (nr < folio_nr_pages(folio)) {
 				int err;
 
-				if (folio_maybe_mapped_shared(folio))
-					continue;
-				if (pageout_anon_only_filter && !folio_test_anon(folio))
+				if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
 					continue;
 				if (!folio_trylock(folio))
 					continue;
@@ -537,7 +551,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		    folio_mapcount(folio) != folio_nr_pages(folio))
 			continue;
 
-		if (pageout_anon_only_filter && !folio_test_anon(folio))
+		if (pageout_anon_only && !folio_test_anon(folio))
 			continue;
 
 		if (!pageout && pte_young(ptent)) {
@@ -546,25 +560,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 			tlb_remove_tlb_entries(tlb, pte, nr, addr);
 		}
 
-		/*
-		 * We are deactivating a folio for accelerating reclaiming.
-		 * VM couldn't reclaim the folio unless we clear PG_young.
-		 * As a side effect, it makes confuse idle-page tracking
-		 * because they will miss recent referenced history.
-		 */
-		folio_clear_referenced(folio);
-		folio_test_clear_young(folio);
-		if (folio_test_active(folio))
-			folio_set_workingset(folio);
-		if (pageout) {
-			if (folio_isolate_lru(folio)) {
-				if (folio_test_unevictable(folio))
-					folio_putback_lru(folio);
-				else
-					list_add(&folio->lru, &folio_list);
-			}
-		} else
-			folio_deactivate(folio);
+		madvise_lru_folio(folio, pageout, &folio_list);
 	}
 
 out:
@@ -651,8 +647,8 @@ static long madvise_pageout(struct madvise_behavior *madv_behavior)
 	 * owner nor write capable of the file. We allow private file mappings
 	 * further to pageout dirty anon pages.
 	 */
-	if (!vma_is_anonymous(vma) && (!can_do_file_pageout(vma) &&
-				(vma->vm_flags & VM_MAYSHARE)))
+	if (!vma_is_anonymous(vma) && !can_do_file_pageout(vma) &&
+	    (vma->vm_flags & VM_MAYSHARE))
 		return 0;
 
 	lru_add_drain();
-- 
2.53.0-Meta

Re: [PATCH 03/10] mm/madvise: factor shared LRU folio handling

[Lorenzo Stoakes (ARM)]

On Tue, Sep 22, 2026 at 07:58:23PM -0400, Gregory Price wrote:
> The huge-PMD and PTE paths duplicate folio filtering, reference clearing,
> deactivation and pageout isolation. Keeping both copies synchronized
> obscures the page-table-specific control flow.
>
> Factor the common filtering and LRU operation into helpers. Keep the
> lock-before-reference sequence visible at each split site because an early
> reference can prevent split_folio() from succeeding.
>
> No functional change intended.
>
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>

LGTM and AFAICT all the code is the same as before just less horrifying so:

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

> ---
>  mm/madvise.c | 90 +++++++++++++++++++++++++---------------------------
>  1 file changed, 43 insertions(+), 47 deletions(-)
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 83d54ab385da8..c345fef23f15d 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -361,6 +361,39 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
>  				     FPB_MERGE_YOUNG_DIRTY);
>  }
>
> +static inline void
> +madvise_lru_folio(struct folio *folio, bool pageout,
> +		struct list_head *folio_list)
> +{
> +	/*
> +	 * Clear references before deactivating or reclaiming the folio. This can
> +	 * make idle-page tracking miss recent accesses.
> +	 */
> +	folio_clear_referenced(folio);
> +	folio_test_clear_young(folio);
> +	if (folio_test_active(folio))
> +		folio_set_workingset(folio);
> +
> +	if (!pageout) {
> +		folio_deactivate(folio);
> +		return;
> +	}
> +
> +	if (!folio_isolate_lru(folio))
> +		return;
> +	if (folio_test_unevictable(folio))
> +		folio_putback_lru(folio);
> +	else
> +		list_add(&folio->lru, folio_list);
> +}
> +
> +static bool madvise_lru_folio_is_filtered(struct folio *folio,
> +		bool pageout_anon_only)
> +{
> +	return folio_maybe_mapped_shared(folio) ||
> +	       (pageout_anon_only && !folio_test_anon(folio));
> +}
> +
>  static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  		unsigned long end, struct mm_walk *walk)
>  {
> @@ -373,15 +406,14 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  	spinlock_t *ptl;
>  	struct folio *folio = NULL;
>  	LIST_HEAD(folio_list);
> -	bool pageout_anon_only_filter;
>  	unsigned int batch_count = 0;
> +	bool pageout_anon_only;
>  	int nr;
>
>  	if (fatal_signal_pending(current))
>  		return -EINTR;
> -
> -	pageout_anon_only_filter = pageout && !vma_is_anonymous(vma) &&
> -					!can_do_file_pageout(vma);
> +	pageout_anon_only = pageout && !vma_is_anonymous(vma) &&
> +				       !can_do_file_pageout(vma);
>
>  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
>  	if (pmd_trans_huge(*pmd)) {
> @@ -407,11 +439,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  		if (folio_is_zone_device(folio))
>  			goto huge_unlock;
>
> -		/* Do not interfere with other mappings of this folio */
> -		if (folio_maybe_mapped_shared(folio))
> -			goto huge_unlock;
> -
> -		if (pageout_anon_only_filter && !folio_test_anon(folio))
> +		if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
>  			goto huge_unlock;
>
>  		if (next - addr != HPAGE_PMD_SIZE) {
> @@ -437,19 +465,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  			tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
>  		}
>
> -		folio_clear_referenced(folio);
> -		folio_test_clear_young(folio);
> -		if (folio_test_active(folio))
> -			folio_set_workingset(folio);
> -		if (pageout) {
> -			if (folio_isolate_lru(folio)) {
> -				if (folio_test_unevictable(folio))
> -					folio_putback_lru(folio);
> -				else
> -					list_add(&folio->lru, &folio_list);
> -			}
> -		} else
> -			folio_deactivate(folio);
> +		madvise_lru_folio(folio, pageout, &folio_list);
>  huge_unlock:
>  		spin_unlock(ptl);
>  		if (pageout)
> @@ -502,9 +518,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  			if (nr < folio_nr_pages(folio)) {
>  				int err;
>
> -				if (folio_maybe_mapped_shared(folio))
> -					continue;
> -				if (pageout_anon_only_filter && !folio_test_anon(folio))
> +				if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
>  					continue;
>  				if (!folio_trylock(folio))
>  					continue;
> @@ -537,7 +551,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  		    folio_mapcount(folio) != folio_nr_pages(folio))
>  			continue;
>
> -		if (pageout_anon_only_filter && !folio_test_anon(folio))
> +		if (pageout_anon_only && !folio_test_anon(folio))
>  			continue;
>
>  		if (!pageout && pte_young(ptent)) {
> @@ -546,25 +560,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  			tlb_remove_tlb_entries(tlb, pte, nr, addr);
>  		}
>
> -		/*
> -		 * We are deactivating a folio for accelerating reclaiming.
> -		 * VM couldn't reclaim the folio unless we clear PG_young.
> -		 * As a side effect, it makes confuse idle-page tracking
> -		 * because they will miss recent referenced history.
> -		 */
> -		folio_clear_referenced(folio);
> -		folio_test_clear_young(folio);
> -		if (folio_test_active(folio))
> -			folio_set_workingset(folio);
> -		if (pageout) {
> -			if (folio_isolate_lru(folio)) {
> -				if (folio_test_unevictable(folio))
> -					folio_putback_lru(folio);
> -				else
> -					list_add(&folio->lru, &folio_list);
> -			}
> -		} else
> -			folio_deactivate(folio);
> +		madvise_lru_folio(folio, pageout, &folio_list);
>  	}
>
>  out:
> @@ -651,8 +647,8 @@ static long madvise_pageout(struct madvise_behavior *madv_behavior)
>  	 * owner nor write capable of the file. We allow private file mappings
>  	 * further to pageout dirty anon pages.
>  	 */
> -	if (!vma_is_anonymous(vma) && (!can_do_file_pageout(vma) &&
> -				(vma->vm_flags & VM_MAYSHARE)))
> +	if (!vma_is_anonymous(vma) && !can_do_file_pageout(vma) &&
> +	    (vma->vm_flags & VM_MAYSHARE))
>  		return 0;
>
>  	lru_add_drain();
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

[PATCH 04/10] mm/madvise: use the PMD softleaf validity helper

[Gregory Price]

A non-present huge PMD must contain a software leaf type supported at
PMD level. The open-coded check names the currently supported migration
and device-private entries instead of expressing that invariant.

Use pmd_is_valid_softleaf() so the validation follows the central
definition of valid PMD softleaf entries.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index c345fef23f15d..b31b877c2c130 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -427,8 +427,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 
 		orig_pmd = *pmd;
 		if (unlikely(!pmd_present(orig_pmd))) {
-			VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
-					!pmd_is_device_private_entry(orig_pmd));
+			VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd));
 			goto huge_unlock;
 		}
 
-- 
2.53.0-Meta

Re: [PATCH 04/10] mm/madvise: use the PMD softleaf validity helper

[Lorenzo Stoakes (ARM)]

On Tue, Sep 22, 2026 at 07:58:24PM -0400, Gregory Price wrote:
> A non-present huge PMD must contain a software leaf type supported at
> PMD level. The open-coded check names the currently supported migration
> and device-private entries instead of expressing that invariant.
>
> Use pmd_is_valid_softleaf() so the validation follows the central
> definition of valid PMD softleaf entries.
>
> No functional change intended.
>
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>

LGTM and I like the use of softlaf obviously :)

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

> ---
>  mm/madvise.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index c345fef23f15d..b31b877c2c130 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -427,8 +427,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>
>  		orig_pmd = *pmd;
>  		if (unlikely(!pmd_present(orig_pmd))) {
> -			VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
> -					!pmd_is_device_private_entry(orig_pmd));
> +			VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd));
>  			goto huge_unlock;
>  		}
>
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

[PATCH 05/10] mm/madvise: factor huge-PMD folio processing

[Gregory Price]

The huge-PMD branch combines PMD validation and lock ownership with folio
filtering, splitting, aging and isolation.

Move the folio-specific work into a helper called with the PMD lock held.
Return a locked and referenced folio only when the caller must drop the PMD
lock and split it.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 70 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 46 insertions(+), 24 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index b31b877c2c130..6b518f7f73651 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -394,6 +394,49 @@ static bool madvise_lru_folio_is_filtered(struct folio *folio,
 	       (pageout_anon_only && !folio_test_anon(folio));
 }
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static void madvise_cold_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		pmd_t *pmd, unsigned long addr, pmd_t orig_pmd)
+{
+	if (!pmd_young(orig_pmd))
+		return;
+
+	pmdp_invalidate(vma, addr, pmd);
+	orig_pmd = pmd_mkold(orig_pmd);
+	set_pmd_at(tlb->mm, addr, pmd, orig_pmd);
+	tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
+}
+
+/* Return a locked, referenced folio only when it must be split. */
+static struct folio *
+madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd,
+		unsigned long addr, unsigned long next, struct mm_walk *walk,
+		struct list_head *folio_list, bool pageout_anon_only)
+{
+	const struct madvise_walk_private *private = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	struct folio *folio;
+
+	folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
+	if (!folio || folio_is_zone_device(folio))
+		return NULL;
+	if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
+		return NULL;
+
+	if (next - addr != HPAGE_PMD_SIZE) {
+		if (!folio_trylock(folio))
+			return NULL;
+		folio_get(folio);
+		return folio;
+	}
+
+	if (!private->pageout)
+		madvise_cold_pmd(private->tlb, vma, pmd, addr, orig_pmd);
+	madvise_lru_folio(folio, private->pageout, folio_list);
+	return NULL;
+}
+#endif
+
 static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		unsigned long end, struct mm_walk *walk)
 {
@@ -431,22 +474,11 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 			goto huge_unlock;
 		}
 
-		folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
-		if (!folio)
-			goto huge_unlock;
-
-		if (folio_is_zone_device(folio))
-			goto huge_unlock;
-
-		if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
-			goto huge_unlock;
-
-		if (next - addr != HPAGE_PMD_SIZE) {
+		folio = madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next,
+				walk, &folio_list, pageout_anon_only);
+		if (folio) {
 			int err;
 
-			if (!folio_trylock(folio))
-				goto huge_unlock;
-			folio_get(folio);
 			spin_unlock(ptl);
 			err = split_folio(folio);
 			folio_unlock(folio);
@@ -455,16 +487,6 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 				goto regular_folio;
 			return 0;
 		}
-
-		if (!pageout && pmd_young(orig_pmd)) {
-			pmdp_invalidate(vma, addr, pmd);
-			orig_pmd = pmd_mkold(orig_pmd);
-
-			set_pmd_at(mm, addr, pmd, orig_pmd);
-			tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
-		}
-
-		madvise_lru_folio(folio, pageout, &folio_list);
 huge_unlock:
 		spin_unlock(ptl);
 		if (pageout)
-- 
2.53.0-Meta

Re: [PATCH 05/10] mm/madvise: factor huge-PMD folio processing

[Lorenzo Stoakes (ARM)]

On Tue, Sep 22, 2026 at 07:58:25PM -0400, Gregory Price wrote:
> The huge-PMD branch combines PMD validation and lock ownership with folio
> filtering, splitting, aging and isolation.
>
> Move the folio-specific work into a helper called with the PMD lock held.
> Return a locked and referenced folio only when the caller must drop the PMD
> lock and split it.
>
> No functional change intended.
>
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  mm/madvise.c | 70 ++++++++++++++++++++++++++++++++++------------------
>  1 file changed, 46 insertions(+), 24 deletions(-)
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index b31b877c2c130..6b518f7f73651 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -394,6 +394,49 @@ static bool madvise_lru_folio_is_filtered(struct folio *folio,
>  	       (pageout_anon_only && !folio_test_anon(folio));
>  }
>
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +static void madvise_cold_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
> +		pmd_t *pmd, unsigned long addr, pmd_t orig_pmd)
> +{
> +	if (!pmd_young(orig_pmd))
> +		return;
> +
> +	pmdp_invalidate(vma, addr, pmd);
> +	orig_pmd = pmd_mkold(orig_pmd);
> +	set_pmd_at(tlb->mm, addr, pmd, orig_pmd);
> +	tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
> +}
> +
> +/* Return a locked, referenced folio only when it must be split. */

I find it really weird that when it:

a. succeeds
b. mapped folio is missing/invalid/filtered

In both cases it returns NULL.

And it's also weirdly returning a folio in a kind of failure case, or it's
more like a defer-to-the-rest-of-the-code case I suppose.

I wonder if the split could be done as part of the function?

Then maybe have it return bool and document that true means it's fully
processed (invalid folio cases, success case), false means that it's been
split and the rest of the code should continue.

Awkward one actually.

> +static struct folio *
> +madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd,
> +		unsigned long addr, unsigned long next, struct mm_walk *walk,
> +		struct list_head *folio_list, bool pageout_anon_only)
> +{
> +	const struct madvise_walk_private *private = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	struct folio *folio;
> +
> +	folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
> +	if (!folio || folio_is_zone_device(folio))
> +		return NULL;
> +	if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
> +		return NULL;
> +
> +	if (next - addr != HPAGE_PMD_SIZE) {

NIT: Maybe could define above as:

	const bool spans_pmd = next - addr == HPAGE_PMD_SIZE;

And then make this:

	if (!spans_pmd)

?

> +		if (!folio_trylock(folio))
> +			return NULL;
> +		folio_get(folio);
> +		return folio;
> +	}
> +
> +	if (!private->pageout)
> +		madvise_cold_pmd(private->tlb, vma, pmd, addr, orig_pmd);
> +	madvise_lru_folio(folio, private->pageout, folio_list);
> +	return NULL;
> +}
> +#endif
> +
>  static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  		unsigned long end, struct mm_walk *walk)
>  {
> @@ -431,22 +474,11 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  			goto huge_unlock;
>  		}
>
> -		folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
> -		if (!folio)
> -			goto huge_unlock;
> -
> -		if (folio_is_zone_device(folio))
> -			goto huge_unlock;
> -
> -		if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
> -			goto huge_unlock;
> -
> -		if (next - addr != HPAGE_PMD_SIZE) {
> +		folio = madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next,
> +				walk, &folio_list, pageout_anon_only);
> +		if (folio) {
>  			int err;
>
> -			if (!folio_trylock(folio))
> -				goto huge_unlock;
> -			folio_get(folio);
>  			spin_unlock(ptl);
>  			err = split_folio(folio);
>  			folio_unlock(folio);
> @@ -455,16 +487,6 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
>  				goto regular_folio;
>  			return 0;
>  		}
> -
> -		if (!pageout && pmd_young(orig_pmd)) {
> -			pmdp_invalidate(vma, addr, pmd);
> -			orig_pmd = pmd_mkold(orig_pmd);
> -
> -			set_pmd_at(mm, addr, pmd, orig_pmd);
> -			tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
> -		}
> -
> -		madvise_lru_folio(folio, pageout, &folio_list);
>  huge_unlock:
>  		spin_unlock(ptl);
>  		if (pageout)
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

Re: [PATCH 05/10] mm/madvise: factor huge-PMD folio processing

[Gregory Price]

On Wed, Sep 23, 2026 at 05:43:59PM +0100, Lorenzo Stoakes (ARM) wrote:
> > +/* Return a locked, referenced folio only when it must be split. */
> 
> I find it really weird that when it:
> 
> a. succeeds
> b. mapped folio is missing/invalid/filtered
> 
> In both cases it returns NULL.
> 
> And it's also weirdly returning a folio in a kind of failure case, or it's
> more like a defer-to-the-rest-of-the-code case I suppose.
> 
> I wonder if the split could be done as part of the function?
> 
> Then maybe have it return bool and document that true means it's fully
> processed (invalid folio cases, success case), false means that it's been
> split and the rest of the code should continue.
> 
> Awkward one actually.

Yes this was an awkward one to futz around with.  I took a couple tries
at it and this is ultimately what fell out and passed the tests.

I think there's some tweaks that could be made here, but I err'd on the
side of "don't break shit" before I went twiddling.

It is at least easier to understand, but certainly this shows how poorly
the original code was structured.

> 
> > +static struct folio *
> > +madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd,
> > +		unsigned long addr, unsigned long next, struct mm_walk *walk,
> > +		struct list_head *folio_list, bool pageout_anon_only)
> > +{
> > +	const struct madvise_walk_private *private = walk->private;
> > +	struct vm_area_struct *vma = walk->vma;
> > +	struct folio *folio;
> > +
> > +	folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
> > +	if (!folio || folio_is_zone_device(folio))
> > +		return NULL;
> > +	if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
> > +		return NULL;
> > +
> > +	if (next - addr != HPAGE_PMD_SIZE) {
> 
> NIT: Maybe could define above as:
> 
> 	const bool spans_pmd = next - addr == HPAGE_PMD_SIZE;
> 
> And then make this:
> 
> 	if (!spans_pmd)
> 

seems reasonable. ack

~Gregory

Re: [PATCH 05/10] mm/madvise: factor huge-PMD folio processing

[Lorenzo Stoakes (ARM)]

On Wed, Sep 23, 2026 at 01:06:45PM -0400, Gregory Price wrote:
> On Wed, Sep 23, 2026 at 05:43:59PM +0100, Lorenzo Stoakes (ARM) wrote:
> > > +/* Return a locked, referenced folio only when it must be split. */
> >
> > I find it really weird that when it:
> >
> > a. succeeds
> > b. mapped folio is missing/invalid/filtered
> >
> > In both cases it returns NULL.
> >
> > And it's also weirdly returning a folio in a kind of failure case, or it's
> > more like a defer-to-the-rest-of-the-code case I suppose.
> >
> > I wonder if the split could be done as part of the function?
> >
> > Then maybe have it return bool and document that true means it's fully
> > processed (invalid folio cases, success case), false means that it's been
> > split and the rest of the code should continue.
> >
> > Awkward one actually.
>
> Yes this was an awkward one to futz around with.  I took a couple tries
> at it and this is ultimately what fell out and passed the tests.
>
> I think there's some tweaks that could be made here, but I err'd on the
> side of "don't break shit" before I went twiddling.
>
> It is at least easier to understand, but certainly this shows how poorly
> the original code was structured.

Yeah, if this is an intermediate state I won't necessarily insist but it just
feels so odd.

Maybe have a look at it doing the split in the function and see how that
sits?

>
> >
> > > +static struct folio *
> > > +madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd,
> > > +		unsigned long addr, unsigned long next, struct mm_walk *walk,
> > > +		struct list_head *folio_list, bool pageout_anon_only)
> > > +{
> > > +	const struct madvise_walk_private *private = walk->private;
> > > +	struct vm_area_struct *vma = walk->vma;
> > > +	struct folio *folio;
> > > +
> > > +	folio = vm_normal_folio_pmd(vma, addr, orig_pmd);
> > > +	if (!folio || folio_is_zone_device(folio))
> > > +		return NULL;
> > > +	if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
> > > +		return NULL;
> > > +
> > > +	if (next - addr != HPAGE_PMD_SIZE) {
> >
> > NIT: Maybe could define above as:
> >
> > 	const bool spans_pmd = next - addr == HPAGE_PMD_SIZE;
> >
> > And then make this:
> >
> > 	if (!spans_pmd)
> >
>
> seems reasonable. ack
>
> ~Gregory

--
Cheers, Lorenzo

Re: [PATCH 05/10] mm/madvise: factor huge-PMD folio processing

[Gregory Price]

On Wed, Sep 23, 2026 at 06:14:29PM +0100, Lorenzo Stoakes (ARM) wrote:
> >
> > I think there's some tweaks that could be made here, but I err'd on the
> > side of "don't break shit" before I went twiddling.
> >
> > It is at least easier to understand, but certainly this shows how poorly
> > the original code was structured.
> 
> Yeah, if this is an intermediate state I won't necessarily insist but it just
> feels so odd.
> 
> Maybe have a look at it doing the split in the function and see how that
> sits?
> 

There's an implied v2 for this series i think, i was going to look at
this commit a little more closely for sure.  There's a balance between
making a reviewable patch and fixing poorly structured code.

Let me take a crack at reworking it on top of the series, and then see
whether anything appreciable changes.  If so, then I'd prefer to keep
that in a separate commit so the behavioral change is very explicit.

Some of the sched() nonsense is going to look equally weird.

~Gregory

[PATCH 06/10] mm/madvise: separate huge PMDs from the PTE walk

[Gregory Price]

The PMD callback still owns huge-PMD locking, splitting and reclaim along
with its PTE-table loop. This obscures the transition between the two
paths.

Move huge-PMD lock ownership into a dedicated helper. Return false after a
requested PMD split succeeds so the callback continues at PTE level. Use
the PMD boundary already supplied by walk_pmd_range().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 85 +++++++++++++++++++++++++++++-----------------------
 1 file changed, 48 insertions(+), 37 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 6b518f7f73651..83b27258c9673 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -435,6 +435,52 @@ madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd,
 	madvise_lru_folio(folio, private->pageout, folio_list);
 	return NULL;
 }
+
+/* Return false when a requested split requires a PTE walk. */
+static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned long addr,
+		unsigned long next, struct mm_walk *walk,
+		bool pageout_anon_only)
+{
+	const struct madvise_walk_private *private = walk->private;
+	struct mmu_gather *tlb = private->tlb;
+	bool pageout = private->pageout;
+	LIST_HEAD(folio_list);
+	struct folio *folio = NULL;
+	spinlock_t *ptl;
+	pmd_t orig_pmd;
+
+	tlb_change_page_size(tlb, HPAGE_PMD_SIZE);
+	ptl = pmd_trans_huge_lock(pmd, walk->vma);
+	if (!ptl)
+		return true;
+
+	orig_pmd = *pmd;
+	if (unlikely(!pmd_present(orig_pmd))) {
+		VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd));
+	} else {
+		folio = madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next,
+				walk, &folio_list, pageout_anon_only);
+	}
+	spin_unlock(ptl);
+
+	if (folio) {
+		int err = split_folio(folio);
+
+		folio_unlock(folio);
+		folio_put(folio);
+		return err != 0;
+	}
+	if (pageout)
+		reclaim_pages(&folio_list);
+	return true;
+}
+#else
+static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned long addr,
+		unsigned long next, struct mm_walk *walk,
+		bool pageout_anon_only)
+{
+	return false;
+}
 #endif
 
 static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
@@ -458,44 +504,9 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 	pageout_anon_only = pageout && !vma_is_anonymous(vma) &&
 				       !can_do_file_pageout(vma);
 
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (pmd_trans_huge(*pmd)) {
-		pmd_t orig_pmd;
-		unsigned long next = pmd_addr_end(addr, end);
-
-		tlb_change_page_size(tlb, HPAGE_PMD_SIZE);
-		ptl = pmd_trans_huge_lock(pmd, vma);
-		if (!ptl)
-			return 0;
-
-		orig_pmd = *pmd;
-		if (unlikely(!pmd_present(orig_pmd))) {
-			VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd));
-			goto huge_unlock;
-		}
-
-		folio = madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next,
-				walk, &folio_list, pageout_anon_only);
-		if (folio) {
-			int err;
-
-			spin_unlock(ptl);
-			err = split_folio(folio);
-			folio_unlock(folio);
-			folio_put(folio);
-			if (!err)
-				goto regular_folio;
-			return 0;
-		}
-huge_unlock:
-		spin_unlock(ptl);
-		if (pageout)
-			reclaim_pages(&folio_list);
+	if (pmd_trans_huge(*pmd) &&
+	    madvise_lru_huge_pmd(pmd, addr, end, walk, pageout_anon_only))
 		return 0;
-	}
-
-regular_folio:
-#endif
 	tlb_change_page_size(tlb, PAGE_SIZE);
 restart:
 	start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
-- 
2.53.0-Meta

[PATCH 07/10] mm/madvise: separate PTE-batch folio processing

[Gregory Price]

The PTE loop combines page-table iteration with validation and
processing of each folio batch. This hides the batching rule for large
folios and the ownership transferred when a partial folio must be split.

Move one batch into a PTL-locked helper. A split candidate is returned
locked and referenced so the caller can release the PTE mapping before
calling split_folio().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 132 +++++++++++++++++++++++++--------------------------
 1 file changed, 65 insertions(+), 67 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 83b27258c9673..350b854ccc197 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -394,6 +394,54 @@ static bool madvise_lru_folio_is_filtered(struct folio *folio,
 	       (pageout_anon_only && !folio_test_anon(folio));
 }
 
+/* Return a split candidate locked and referenced for use after the PTL drop. */
+static struct folio *
+madvise_lru_pte_batch_locked(pte_t *pte, unsigned long addr,
+		unsigned long end, struct mm_walk *walk,
+		struct list_head *folio_list, bool pageout_anon_only, int *nr)
+{
+	const struct madvise_walk_private *private = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	struct folio *folio;
+	pte_t ptent;
+
+	*nr = 1;
+	ptent = ptep_get(pte);
+	if (pte_none(ptent) || !pte_present(ptent))
+		return NULL;
+
+	folio = vm_normal_folio(vma, addr, ptent);
+	if (!folio || folio_is_zone_device(folio))
+		return NULL;
+
+	/* Split PTE-mapped large folios before advising only part of them. */
+	if (folio_test_large(folio)) {
+		*nr = madvise_folio_pte_batch(addr, end, folio, pte, &ptent);
+		if (*nr < folio_nr_pages(folio)) {
+			if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
+				return NULL;
+			if (!folio_trylock(folio))
+				return NULL;
+			folio_get(folio);
+			return folio;
+		}
+	}
+
+	if (!folio_test_lru(folio) ||
+	    folio_mapcount(folio) != folio_nr_pages(folio))
+		return NULL;
+	if (pageout_anon_only && !folio_test_anon(folio))
+		return NULL;
+
+	if (!private->pageout && pte_young(ptent)) {
+		clear_young_dirty_ptes(vma, addr, pte, *nr, CYDP_CLEAR_YOUNG);
+		tlb_remove_tlb_entries(private->tlb, pte, *nr, addr);
+	}
+
+	madvise_lru_folio(folio, private->pageout, folio_list);
+	return NULL;
+}
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 static void madvise_cold_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		pmd_t *pmd, unsigned long addr, pmd_t orig_pmd)
@@ -491,7 +539,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 	bool pageout = private->pageout;
 	struct mm_struct *mm = tlb->mm;
 	struct vm_area_struct *vma = walk->vma;
-	pte_t *start_pte, *pte, ptent;
+	pte_t *start_pte, *pte;
 	spinlock_t *ptl;
 	struct folio *folio = NULL;
 	LIST_HEAD(folio_list);
@@ -515,9 +563,6 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 	flush_tlb_batched_pending(mm);
 	lazy_mmu_mode_enable();
 	for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) {
-		nr = 1;
-		ptent = ptep_get(pte);
-
 		if (++batch_count == SWAP_CLUSTER_MAX) {
 			batch_count = 0;
 			if (need_resched()) {
@@ -528,71 +573,24 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 			}
 		}
 
-		if (pte_none(ptent))
-			continue;
-
-		if (!pte_present(ptent))
-			continue;
-
-		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_zone_device(folio))
-			continue;
-
-		/*
-		 * If we encounter a large folio, only split it if it is not
-		 * fully mapped within the range we are operating on. Otherwise
-		 * leave it as is so that it can be swapped out whole. If we
-		 * fail to split a folio, leave it in place and advance to the
-		 * next pte in the range.
-		 */
-		if (folio_test_large(folio)) {
-			nr = madvise_folio_pte_batch(addr, end, folio, pte, &ptent);
-			if (nr < folio_nr_pages(folio)) {
-				int err;
-
-				if (madvise_lru_folio_is_filtered(folio, pageout_anon_only))
-					continue;
-				if (!folio_trylock(folio))
-					continue;
-				folio_get(folio);
-				lazy_mmu_mode_disable();
-				pte_unmap_unlock(start_pte, ptl);
-				start_pte = NULL;
-				err = split_folio(folio);
-				folio_unlock(folio);
-				folio_put(folio);
-				start_pte = pte =
-					pte_offset_map_lock(mm, pmd, addr, &ptl);
-				if (!start_pte)
-					break;
-				flush_tlb_batched_pending(mm);
-				lazy_mmu_mode_enable();
-				if (!err)
-					nr = 0;
-				continue;
-			}
-		}
-
-		/*
-		 * Do not interfere with other mappings of this folio and
-		 * non-LRU folio. If we have a large folio at this point, we
-		 * know it is fully mapped so if its mapcount is the same as its
-		 * number of pages, it must be exclusive.
-		 */
-		if (!folio_test_lru(folio) ||
-		    folio_mapcount(folio) != folio_nr_pages(folio))
-			continue;
-
-		if (pageout_anon_only && !folio_test_anon(folio))
+		folio = madvise_lru_pte_batch_locked(pte, addr, end, walk,
+				&folio_list, pageout_anon_only, &nr);
+		if (!folio)
 			continue;
 
-		if (!pageout && pte_young(ptent)) {
-			clear_young_dirty_ptes(vma, addr, pte, nr,
-					       CYDP_CLEAR_YOUNG);
-			tlb_remove_tlb_entries(tlb, pte, nr, addr);
-		}
-
-		madvise_lru_folio(folio, pageout, &folio_list);
+		lazy_mmu_mode_disable();
+		pte_unmap_unlock(start_pte, ptl);
+		start_pte = NULL;
+		if (!split_folio(folio))
+			nr = 0;
+		folio_unlock(folio);
+		folio_put(folio);
+		start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+		if (!start_pte)
+			break;
+		pte = start_pte;
+		flush_tlb_batched_pending(mm);
+		lazy_mmu_mode_enable();
 	}
 
 out:
-- 
2.53.0-Meta

[PATCH 08/10] mm/madvise: separate the PTL-held PTE scan

[Gregory Price]

The PTE loop mixes its PTL-protected scan with the surrounding lock drops
for rescheduling and large-folio splitting.

Move the scan into a helper that runs entirely under the PTE lock. It
returns a locked and referenced split candidate, or stops at the existing
scheduling boundary so the caller can drop the lock before yielding.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 66 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 39 insertions(+), 27 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 350b854ccc197..e3c3acfcd9b66 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -531,6 +531,28 @@ static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned long addr,
 }
 #endif
 
+static struct folio *
+madvise_lru_pte_range_locked(pte_t *pte, unsigned long *addr,
+		unsigned long end, struct mm_walk *walk,
+		struct list_head *folio_list, bool pageout_anon_only, int *nr,
+		unsigned int *batch_count)
+{
+	struct folio *folio;
+
+	for (; *addr < end; pte += *nr, *addr += *nr * PAGE_SIZE) {
+		if (++(*batch_count) == SWAP_CLUSTER_MAX) {
+			*batch_count = 0;
+			if (need_resched())
+				return NULL;
+		}
+		folio = madvise_lru_pte_batch_locked(pte, *addr, end, walk,
+				folio_list, pageout_anon_only, nr);
+		if (folio)
+			return folio;
+	}
+	return NULL;
+}
+
 static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		unsigned long end, struct mm_walk *walk)
 {
@@ -562,36 +584,26 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
 		goto out;
 	flush_tlb_batched_pending(mm);
 	lazy_mmu_mode_enable();
-	for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) {
-		if (++batch_count == SWAP_CLUSTER_MAX) {
-			batch_count = 0;
-			if (need_resched()) {
-				lazy_mmu_mode_disable();
-				pte_unmap_unlock(start_pte, ptl);
-				cond_resched();
-				goto restart;
-			}
-		}
-
-		folio = madvise_lru_pte_batch_locked(pte, addr, end, walk,
-				&folio_list, pageout_anon_only, &nr);
-		if (!folio)
-			continue;
-
+	folio = madvise_lru_pte_range_locked(pte, &addr, end, walk,
+			&folio_list, pageout_anon_only, &nr, &batch_count);
+	if (!folio && addr < end) {
 		lazy_mmu_mode_disable();
 		pte_unmap_unlock(start_pte, ptl);
-		start_pte = NULL;
-		if (!split_folio(folio))
-			nr = 0;
-		folio_unlock(folio);
-		folio_put(folio);
-		start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
-		if (!start_pte)
-			break;
-		pte = start_pte;
-		flush_tlb_batched_pending(mm);
-		lazy_mmu_mode_enable();
+		cond_resched();
+		goto restart;
 	}
+	if (!folio)
+		goto out;
+
+	lazy_mmu_mode_disable();
+	pte_unmap_unlock(start_pte, ptl);
+	start_pte = NULL;
+	if (!split_folio(folio))
+		nr = 0;
+	folio_unlock(folio);
+	folio_put(folio);
+	addr += nr * PAGE_SIZE;
+	goto restart;
 
 out:
 	if (start_pte) {
-- 
2.53.0-Meta

[PATCH 09/10] mm/madvise: make cold and pageout PTE lock ownership explicit

[Gregory Price]

The PMD callback still maps and unmaps the PTE table, drives split retries
and handles rescheduling after dispatching huge PMDs. This leaves PTE lock
ownership mixed with the page-walk callback.

Move that lifecycle into madvise_lru_pte_range(). Its outer loop drops the
lock before splitting or yielding and resumes at the current address,
leaving the PMD callback to select the huge-PMD or PTE path.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 94 ++++++++++++++++++++++++++--------------------------
 1 file changed, 47 insertions(+), 47 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index e3c3acfcd9b66..88a4a03dee04c 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -553,66 +553,66 @@ madvise_lru_pte_range_locked(pte_t *pte, unsigned long *addr,
 	return NULL;
 }
 
-static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
-		unsigned long end, struct mm_walk *walk)
+static void madvise_lru_pte_range(pmd_t *pmd, unsigned long addr,
+		unsigned long end, struct mm_walk *walk,
+		bool pageout_anon_only)
 {
-	struct madvise_walk_private *private = walk->private;
-	struct mmu_gather *tlb = private->tlb;
-	bool pageout = private->pageout;
-	struct mm_struct *mm = tlb->mm;
-	struct vm_area_struct *vma = walk->vma;
+	const struct madvise_walk_private *private = walk->private;
+	struct mm_struct *mm = private->tlb->mm;
+	LIST_HEAD(folio_list);
 	pte_t *start_pte, *pte;
 	spinlock_t *ptl;
-	struct folio *folio = NULL;
-	LIST_HEAD(folio_list);
+	struct folio *folio;
 	unsigned int batch_count = 0;
-	bool pageout_anon_only;
 	int nr;
 
-	if (fatal_signal_pending(current))
-		return -EINTR;
-	pageout_anon_only = pageout && !vma_is_anonymous(vma) &&
-				       !can_do_file_pageout(vma);
+	tlb_change_page_size(private->tlb, PAGE_SIZE);
+	while (addr < end) {
+		start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+		if (!start_pte)
+			break;
+		pte = start_pte;
+		flush_tlb_batched_pending(mm);
+		lazy_mmu_mode_enable();
+		folio = madvise_lru_pte_range_locked(pte, &addr, end, walk,
+				&folio_list, pageout_anon_only, &nr, &batch_count);
 
-	if (pmd_trans_huge(*pmd) &&
-	    madvise_lru_huge_pmd(pmd, addr, end, walk, pageout_anon_only))
-		return 0;
-	tlb_change_page_size(tlb, PAGE_SIZE);
-restart:
-	start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
-	if (!start_pte)
-		goto out;
-	flush_tlb_batched_pending(mm);
-	lazy_mmu_mode_enable();
-	folio = madvise_lru_pte_range_locked(pte, &addr, end, walk,
-			&folio_list, pageout_anon_only, &nr, &batch_count);
-	if (!folio && addr < end) {
 		lazy_mmu_mode_disable();
 		pte_unmap_unlock(start_pte, ptl);
-		cond_resched();
-		goto restart;
-	}
-	if (!folio)
-		goto out;
 
-	lazy_mmu_mode_disable();
-	pte_unmap_unlock(start_pte, ptl);
-	start_pte = NULL;
-	if (!split_folio(folio))
-		nr = 0;
-	folio_unlock(folio);
-	folio_put(folio);
-	addr += nr * PAGE_SIZE;
-	goto restart;
-
-out:
-	if (start_pte) {
-		lazy_mmu_mode_disable();
-		pte_unmap_unlock(start_pte, ptl);
+		if (!folio && addr < end) {
+			cond_resched();
+			continue;
+		}
+		if (folio) {
+			if (!split_folio(folio))
+				nr = 0;
+			folio_unlock(folio);
+			folio_put(folio);
+			addr += nr * PAGE_SIZE;
+		}
 	}
-	if (pageout)
+	if (private->pageout)
 		reclaim_pages(&folio_list);
 	cond_resched();
+}
+
+static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr,
+		unsigned long next, struct mm_walk *walk)
+{
+	const struct madvise_walk_private *private = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	bool pageout_anon_only;
+
+	if (fatal_signal_pending(current))
+		return -EINTR;
+	pageout_anon_only = private->pageout && !vma_is_anonymous(vma) &&
+			    !can_do_file_pageout(vma);
+
+	if (pmd_trans_huge(*pmd) &&
+	    madvise_lru_huge_pmd(pmd, addr, next, walk, pageout_anon_only))
+		return 0;
+	madvise_lru_pte_range(pmd, addr, next, walk, pageout_anon_only);
 
 	return 0;
 }
-- 
2.53.0-Meta

[PATCH 10/10] mm/madvise: share cold and pageout walk setup

[Gregory Price]

The MADV_COLD and MADV_PAGEOUT entry points duplicate the TLB and page-walk
setup while initializing equivalent private state.

Use one madvise_lru_vma_range() helper for the shared TLB lifecycle. Keep
operation-specific validation and policy in the two entry points.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/madvise.c | 49 +++++++++++++++++--------------------------------
 1 file changed, 17 insertions(+), 32 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 88a4a03dee04c..37e3d3a205a76 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -622,21 +622,19 @@ static const struct mm_walk_ops cold_walk_ops = {
 	.walk_lock = PGWALK_RDLOCK,
 };
 
-static void madvise_cold_page_range(struct mmu_gather *tlb,
-		struct madvise_behavior *madv_behavior)
-
+static void
+madvise_lru_vma_range(struct madvise_behavior *madv_behavior,
+		struct madvise_walk_private *walk_private)
 {
 	struct vm_area_struct *vma = madv_behavior->vma;
 	struct madvise_behavior_range *range = &madv_behavior->range;
-	struct madvise_walk_private walk_private = {
-		.pageout = false,
-		.tlb = tlb,
-	};
 
-	tlb_start_vma(tlb, vma);
+	tlb_gather_mmu(walk_private->tlb, madv_behavior->mm);
+	tlb_start_vma(walk_private->tlb, vma);
 	walk_page_range_vma(vma, range->start, range->end, &cold_walk_ops,
-			&walk_private);
-	tlb_end_vma(tlb, vma);
+			    walk_private);
+	tlb_end_vma(walk_private->tlb, vma);
+	tlb_finish_mmu(walk_private->tlb);
 }
 
 static inline bool can_madv_lru_vma(struct vm_area_struct *vma)
@@ -647,39 +645,28 @@ static inline bool can_madv_lru_vma(struct vm_area_struct *vma)
 static long madvise_cold(struct madvise_behavior *madv_behavior)
 {
 	struct vm_area_struct *vma = madv_behavior->vma;
-	struct mmu_gather tlb;
+	struct madvise_walk_private walk_private = {
+		.tlb = madv_behavior->tlb,
+		.pageout = false,
+	};
 
 	if (!can_madv_lru_vma(vma))
 		return -EINVAL;
 
 	lru_add_drain();
-	tlb_gather_mmu(&tlb, madv_behavior->mm);
-	madvise_cold_page_range(&tlb, madv_behavior);
-	tlb_finish_mmu(&tlb);
+	madvise_lru_vma_range(madv_behavior, &walk_private);
 
 	return 0;
 }
 
-static void madvise_pageout_page_range(struct mmu_gather *tlb,
-		struct vm_area_struct *vma,
-		struct madvise_behavior_range *range)
+static long madvise_pageout(struct madvise_behavior *madv_behavior)
 {
+	struct vm_area_struct *vma = madv_behavior->vma;
 	struct madvise_walk_private walk_private = {
+		.tlb = madv_behavior->tlb,
 		.pageout = true,
-		.tlb = tlb,
 	};
 
-	tlb_start_vma(tlb, vma);
-	walk_page_range_vma(vma, range->start, range->end, &cold_walk_ops,
-			    &walk_private);
-	tlb_end_vma(tlb, vma);
-}
-
-static long madvise_pageout(struct madvise_behavior *madv_behavior)
-{
-	struct mmu_gather tlb;
-	struct vm_area_struct *vma = madv_behavior->vma;
-
 	if (!can_madv_lru_vma(vma))
 		return -EINVAL;
 
@@ -694,9 +681,7 @@ static long madvise_pageout(struct madvise_behavior *madv_behavior)
 		return 0;
 
 	lru_add_drain();
-	tlb_gather_mmu(&tlb, madv_behavior->mm);
-	madvise_pageout_page_range(&tlb, vma, &madv_behavior->range);
-	tlb_finish_mmu(&tlb);
+	madvise_lru_vma_range(madv_behavior, &walk_private);
 
 	return 0;
 }
-- 
2.53.0-Meta