[PATCH v3 0/7] mm/page_owner: add bpf_iter and kfuncs

[PATCH v3 0/7] mm/page_owner: add bpf_iter and kfuncs

[Zhen Ni]

Hi all,

Introduction
============

page_owner records who allocated every page and is a core memory-debug
facility, but it currently reaches userspace as the debugfs flat dump:
slicing allocations by pid, comm, NUMA node, cgroup or stack, etc.,
means streaming and parsing the entire file in userspace, which is
expensive on large hosts and hard to query live.

Exposing the same walk to eBPF lets a program filter and aggregate in
the kernel and surface only the records it cares about. This series
adds that access as a seq_file bpf_iter target and an open-coded
iterator.

Implementation
==============

The page walk is factored into page_owner_next_eligible(), shared by
the read path and the iterators so eligibility stays identical. Two
kfuncs expose information that a BPF program cannot decode from the
snapshot alone: per-page memcg info including the full cgroup path,
and the stack trace behind a stack_depot handle. Each record is
snapshotted under the page_ext RCU lock, so consumers read a
consistent copy.

Testing
=======

The selftest exercises the seq_file and open-coded iterators, and
validates the stack_snprint and get_memcg_info kfuncs.

Changes in v3
=============
- Add a mem-hotplug read lock across the page scan.
- Drop the live struct page from the scan cursor and the bpf_iter context.
- Check the page_owner_inited static key in the iterator.
- Without scan->page, start at min_low_pfn - 1 and let next() pre-increment.
- Gate the kfuncs with KF_PERFMON.
- Enable BPF_ITER_RESCHED for the bpf_iter target.
- Check ENODATA/EOPNOTSUPP from <errno.h>.

Changes in v2
=============
- Route the zero-handle case through ext_put_continue.
- Drop the scan->pfn++.
- Fill page/po/pfn only when v is non-NULL.
- Implement page_owner_seq_stop().
- bpf_page_owner_stack_snprint() now takes the scan cursor.
- Buffer args renamed to buf__uninit/buf__sz.
- Removes the v1 get_nid() kfunc. The node is now stored in scan->nid.
- bpf_page_owner_get_memcg_info() now resolves the page with
  pfn_to_online_page() for an offline or removed pfn.
- Buffer/info args renamed to info__uninit/buf__uninit/buf__sz.
- Drop the get_nid kfunc coverage.
- Enable CONFIG_PAGE_OWNER in the bpf selftests config.

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-1-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-1-zhen.ni@easystack.cn/


Zhen Ni (7):
  mm/page_owner: extract page_owner_next_eligible() from read loop
  mm/page_owner: protect page access with mem-hotplug read lock
  mm/page_owner: add bpf_iter target "page_owner"
  mm/page_owner: add open-coded page_owner iterator
  mm/page_owner: record the NUMA node in the scan cursor
  mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc
  selftests/bpf: add tests for the page_owner bpf_iter target

 kernel/bpf/helpers.c                          |  10 +
 mm/Makefile                                   |   3 +
 mm/page_owner.c                               | 252 ++++++++++++------
 mm/page_owner.h                               |  60 +++++
 mm/page_owner_iter.c                          | 232 ++++++++++++++++
 tools/testing/selftests/bpf/config            |   2 +
 .../bpf/prog_tests/page_owner_iter.c          | 193 ++++++++++++++
 .../selftests/bpf/progs/bpf_iter_page_owner.c | 157 +++++++++++
 8 files changed, 826 insertions(+), 83 deletions(-)
 create mode 100644 mm/page_owner.h
 create mode 100644 mm/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c

-- 
2.20.1

[PATCH v3 1/7] mm/page_owner: extract page_owner_next_eligible() from read loop

[Zhen Ni]

Move the page scanning logic out of read_page_owner() into a dedicated
helper, in preparation for using it in the page_owner bpf_iter target.

The helper advances a struct page_owner_scan cursor to the next page
eligible for page_owner output and returns true on a hit. On a hit it
takes a snapshot of the page_owner record inside the page_ext RCU
window.

No change to the output is intended. handle is read from the snapshot,
taken before the non-zero check, so the validated and reported handle
are the same value.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Drop the page field in struct page_owner_scan.

Changes in v2:
- Route the zero-handle case through ext_put_continue.

v1: https://lore.kernel.org/linux-mm/20261009112308.240769-2-zhen.ni@easystack.cn/
v2: https://lore.kernel.org/linux-mm/20261013122125.843451-2-zhen.ni@easystack.cn/
---
 mm/page_owner.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)

diff --git a/mm/page_owner.c b/mm/page_owner.c
index fbbda7ba914b..92a92cdf74ec 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -36,6 +36,15 @@ struct page_owner {
 	pid_t free_tgid;
 };
 
+/*
+ * Cursor and per-page result of page_owner_next_eligible().
+ */
+struct page_owner_scan {
+	/* resume point */
+	unsigned long pfn;
+	struct page_owner po_snap;
+};
+
 struct stack {
 	struct stack_record *stack_record;
 	struct stack *next;
@@ -725,37 +734,24 @@ void __dump_page_owner(const struct page *page)
 	page_ext_put(page_ext);
 }
 
-static ssize_t
-read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
+/*
+ * Advance @scan to the next page eligible for page_owner output.
+ * On a hit, fill scan->pfn/po_snap and return true; the caller
+ * must advance scan->pfn past the hit before calling again to resume
+ * the scan.
+ */
+static bool page_owner_next_eligible(struct page_owner_scan *scan)
 {
-	unsigned long pfn;
-	struct page *page;
-	struct page_ext *page_ext;
-	struct page_owner *page_owner;
-	depot_stack_handle_t handle;
-	struct page_owner_filter_state *state = file->private_data;
+	unsigned long pfn = scan->pfn;
 
-	if (!static_branch_unlikely(&page_owner_inited))
-		return -EINVAL;
-
-	page = NULL;
-	if (*ppos == 0)
-		pfn = min_low_pfn;
-	else
-		pfn = *ppos;
 	/* Find a valid PFN or the start of a MAX_ORDER_NR_PAGES area */
 	while (!pfn_valid(pfn) && (pfn & (MAX_ORDER_NR_PAGES - 1)) != 0)
 		pfn++;
 
-	/* Find an allocated page */
 	for (; pfn < max_pfn; pfn++) {
-		/*
-		 * This temporary page_owner is required so
-		 * that we can avoid the context switches while holding
-		 * the rcu lock and copying the page owner information to
-		 * user through copy_to_user() or GFP_KERNEL allocations.
-		 */
-		struct page_owner page_owner_tmp;
+		struct page_owner *page_owner;
+		struct page_ext *page_ext;
+		struct page *page;
 
 		/*
 		 * If the new page is in a new MAX_ORDER_NR_PAGES area,
@@ -798,13 +794,45 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			goto ext_put_continue;
 
 		/*
-		 * Access to page_ext->handle isn't synchronous so we should
-		 * be careful to access it.
+		 * Take a snapshot of the page_owner record inside the RCU
+		 * window, then validate it, so that print_page_owner()
+		 * reads the checked handle from the same snapshot.
 		 */
-		handle = READ_ONCE(page_owner->handle);
-		if (!handle)
+		scan->pfn = pfn;
+		scan->po_snap = *page_owner;
+
+		if (!scan->po_snap.handle)
 			goto ext_put_continue;
 
+		page_ext_put(page_ext);
+		return true;
+
+ext_put_continue:
+		page_ext_put(page_ext);
+		cond_resched();
+	}
+
+	return false;
+}
+
+static ssize_t
+read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
+{
+	struct page_owner_scan scan;
+	struct page_owner_filter_state *state = file->private_data;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		return -EINVAL;
+
+	if (*ppos == 0)
+		scan.pfn = min_low_pfn;
+	else
+		scan.pfn = *ppos;
+
+	/* Find an allocated page */
+	while (page_owner_next_eligible(&scan)) {
+		struct page *page = pfn_to_page(scan.pfn);
+
 		if (state->nid_filter_enabled) {
 			int nid;
 			memdesc_flags_t page_flags = READ_ONCE(page->flags);
@@ -814,21 +842,20 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			 * VM_BUG_ON when accessing poisoned pages.
 			 */
 			if (page_flags.f == PAGE_POISON_PATTERN)
-				goto ext_put_continue;
+				goto skip_continue;
 			nid = memdesc_nid(&page_flags);
 			if (!node_isset(nid, state->nid_filter))
-				goto ext_put_continue;
+				goto skip_continue;
 		}
 
 		/* Record the next PFN to read in the file offset */
-		*ppos = pfn + 1;
+		*ppos = scan.pfn + 1;
 
-		page_owner_tmp = *page_owner;
-		page_ext_put(page_ext);
-		return print_page_owner(buf, count, pfn, page,
-				&page_owner_tmp, handle, state);
-ext_put_continue:
-		page_ext_put(page_ext);
+		return print_page_owner(buf, count, scan.pfn, page,
+				&scan.po_snap, scan.po_snap.handle, state);
+
+skip_continue:
+		scan.pfn++;
 		cond_resched();
 	}
 
-- 
2.20.1

Re: [PATCH v3 1/7] mm/page_owner: extract page_owner_next_eligible() from read loop

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4369 bytes --]

> mm/page_owner: extract page_owner_next_eligible() from read loop

TL;DR: The nid filter's page->flags read now happens outside the page_ext RCU
window, so it may fault on an unmapped vmemmap during memory hot-remove; later
patches fix it, so it is only a bisect window.

> diff --git a/mm/page_owner.c b/mm/page_owner.c
> index fbbda7ba914b..92a92cdf74ec 100644
> --- a/mm/page_owner.c
> +++ b/mm/page_owner.c

[ ... ]

> @@ -798,13 +794,45 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
>  			goto ext_put_continue;
>
> -		/*
> -		 * Access to page_ext->handle isn't synchronous so we should
> -		 * be careful to access it.
> +		/*
> +		 * Take a snapshot of the page_owner record inside the RCU
> +		 * window, then validate it, so that print_page_owner()
> +		 * reads the checked handle from the same snapshot.
>  		 */
> -		handle = READ_ONCE(page_owner->handle);
> -		if (!handle)
> +		scan->pfn = pfn;
> +		scan->po_snap = *page_owner;
> +
> +		if (!scan->po_snap.handle)
>  			goto ext_put_continue;
>
> +		page_ext_put(page_ext);
> +		return true;
> +
> +ext_put_continue:
> +		page_ext_put(page_ext);
> +		cond_resched();
> +	}
> +
> +	return false;
> +}

[ ... ]

> +	/* Find an allocated page */
> +	while (page_owner_next_eligible(&scan)) {
> +		struct page *page = pfn_to_page(scan.pfn);
> +
>  		if (state->nid_filter_enabled) {
>  			int nid;
>  			memdesc_flags_t page_flags = READ_ONCE(page->flags);

Can this READ_ONCE(page->flags) fault on an unmapped vmemmap?

In the parent commit the nid check ran between page_ext_get() and
page_ext_put(), so it was inside rcu_read_lock():

    page_ext = page_ext_get(page);   /* rcu_read_lock() */
    ...
    if (state->nid_filter_enabled) {
        memdesc_flags_t page_flags = READ_ONCE(page->flags);
        ...
            goto ext_put_continue;
    }
    ...
    ext_put_continue:
        page_ext_put(page_ext);      /* rcu_read_unlock() */

Now page_owner_next_eligible() calls page_ext_put() before it returns true,
and read_page_owner() reads page->flags with no RCU section, no
get_online_mems(), and no page reference held.

The RCU section mattered for memory hot-remove:

    offline_pages()
      memory_notify(MEM_OFFLINE)
        page_ext_callback()
          offline_page_ext()
            synchronize_rcu()

try_remove_memory() requires the blocks to be offline before
arch_remove_memory() -> ... -> section_deactivate() ->
depopulate_section_memmap() unmaps the vmemmap.  While a page_ext_get()
reader was active, that section's memmap could not be torn down.

After this patch, an offline plus remove can complete between page_ext_put()
and the READ_ONCE(page->flags), because synchronize_rcu() no longer waits
for this reader.  This needs CONFIG_PAGE_OWNER with page_owner=on,
CONFIG_MEMORY_HOTREMOVE, a nid filter set through page_owner_write(), and a
concurrent hot-remove of movable memory.

The commit message says "No change to the output is intended", but it does
not mention that the nid filter's struct page access moves out of the page_ext
RCU window.

For context, the parent already read struct page without a pin in
skip_buddy_pages() (PageBuddy() before page_ext_get()) and in
print_page_owner() (page->flags, pageblock flags and page->memcg_data after
page_ext_put()), so this widens an existing race rather than adding a new
class of problem.  It does turn one protected access into an unprotected
one, though.

This is fixed later in the series, so it is only a bisect window issue:

- 915f86c7862f ("mm/page_owner: protect page access with mem-hotplug read
  lock") re-resolves the page in read_page_owner() with pfn_to_online_page()
  under get_online_mems() before the nid check.

- 2cf991a93cbd ("mm/page_owner: record the NUMA node in the scan cursor")
  moves the nid computation back into page_owner_next_eligible(), "to avoid
  dereferencing a possibly-unmapped page".

Could the nid be read in page_owner_next_eligible() before page_ext_put(),
for example by storing it in the cursor?  Alternatively, could the
mem-hotplug protection patch be reordered ahead of this one?

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39593164040

[PATCH v3 2/7] mm/page_owner: protect page access with mem-hotplug read lock

[Zhen Ni]

The page_owner scan walks PFNs locklessly and dereferences the resolved
struct page in the scan itself and in print_page_owner(). With
CONFIG_MEMORY_HOTREMOVE, a concurrent remove_memory() can tear down the
section's vmemmap, so a dereference after a stale validity check may
fault.

Take a mem-hotplug read lock across each page's check-then-deref. In the
scan, gate every PFN with pfn_to_online_page() under the reader and drop
it before cond_resched(), so hotplug can still interleave between pages;
in print_page_owner(), re-resolve the page under a reader covering just
the print of one record.

The reader is cheap: get_online_mems()/put_online_mems() are just
per-CPU counter increments/decrements when no hotplug is active, so
they contend only with an actual add/remove. Taking the lock per-page
bounds the writer's wait to a single page's check-and-deref, keeping
hot-remove stalls brief while all accesses stay protected.

This makes the traversal crash-free. A recycled-but-present page can
still yield stale data -- the lockless scan is best-effort and
unchanged by this patch.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- New patch. Add the mem-hotplug protect.
---
 mm/page_owner.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/mm/page_owner.c b/mm/page_owner.c
index 92a92cdf74ec..8e72a85c663c 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -762,13 +762,22 @@ static bool page_owner_next_eligible(struct page_owner_scan *scan)
 			continue;
 		}
 
-		page = pfn_to_page(pfn);
+		/*
+		 * Hold a mem-hotplug reader across the page derefs so the
+		 * vmemmap isn't torn down mid-scan; released before
+		 * cond_resched().
+		 */
+		get_online_mems();
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto put_online_continue;
+
 		if (skip_buddy_pages(&pfn, page))
-			continue;
+			goto put_online_continue;
 
 		page_ext = page_ext_get(page);
 		if (unlikely(!page_ext))
-			continue;
+			goto put_online_continue;
 
 		/*
 		 * Some pages could be missed by concurrent allocation or free,
@@ -805,10 +814,13 @@ static bool page_owner_next_eligible(struct page_owner_scan *scan)
 			goto ext_put_continue;
 
 		page_ext_put(page_ext);
+		put_online_mems();
 		return true;
 
 ext_put_continue:
 		page_ext_put(page_ext);
+put_online_continue:
+		put_online_mems();
 		cond_resched();
 	}
 
@@ -831,7 +843,17 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 
 	/* Find an allocated page */
 	while (page_owner_next_eligible(&scan)) {
-		struct page *page = pfn_to_page(scan.pfn);
+		struct page *page;
+		ssize_t ret;
+
+		/*
+		 * The scan is lockless. Re-resolve the page under a mem-hotplug
+		 * read-side lock so that we never access a hot-removed section.
+		 */
+		get_online_mems();
+		page = pfn_to_online_page(scan.pfn);
+		if (!page)
+			goto skip_continue;
 
 		if (state->nid_filter_enabled) {
 			int nid;
@@ -851,10 +873,13 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 		/* Record the next PFN to read in the file offset */
 		*ppos = scan.pfn + 1;
 
-		return print_page_owner(buf, count, scan.pfn, page,
+		ret = print_page_owner(buf, count, scan.pfn, page,
 				&scan.po_snap, scan.po_snap.handle, state);
+		put_online_mems();
+		return ret;
 
 skip_continue:
+		put_online_mems();
 		scan.pfn++;
 		cond_resched();
 	}
-- 
2.20.1

Re: [PATCH v3 2/7] mm/page_owner: protect page access with mem-hotplug read lock

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4819 bytes --]

> mm/page_owner: protect page access with mem-hotplug read lock

TL;DR: read_page_owner() holds the mem-hotplug read lock across
copy_to_user(); a page fault there may deadlock against mmap_lock (lock-order
inversion with memremap_pages callers) and userspace can stall hot-remove
indefinitely.

> diff --git a/mm/page_owner.c b/mm/page_owner.c
> --- a/mm/page_owner.c
> +++ b/mm/page_owner.c

[ ... ]

> @@ -831,7 +843,17 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
>
>  	/* Find an allocated page */
>  	while (page_owner_next_eligible(&scan)) {
> -		struct page *page = pfn_to_page(scan.pfn);
> +		struct page *page;
> +		ssize_t ret;
> +
> +		/*
> +		 * The scan is lockless. Re-resolve the page under a mem-hotplug
> +		 * read-side lock so that we never access a hot-removed section.
> +		 */
> +		get_online_mems();
> +		page = pfn_to_online_page(scan.pfn);
> +		if (!page)
> +			goto skip_continue;

[ ... ]

> @@ -851,10 +873,13 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
>  		/* Record the next PFN to read in the file offset */
>  		*ppos = scan.pfn + 1;
>
> -		return print_page_owner(buf, count, scan.pfn, page,
> +		ret = print_page_owner(buf, count, scan.pfn, page,
>  				&scan.po_snap, scan.po_snap.handle, state);
> +		put_online_mems();
> +		return ret;
                ^^^^

Can this deadlock against mmap_lock?  read_page_owner() now holds the
mem_hotplug_lock reader across the whole print_page_owner() call, and that
includes the copy_to_user() of the formatted record:

print_page_owner()
    ...
    if (copy_to_user(buf, kbuf, ret))
            ret = -EFAULT;

Once kbuf is formatted, nothing in that copy touches struct page, so the
reader isn't needed there.  A user page fault on buf taken inside this
region takes current->mm->mmap_lock for read (on x86 a fault from
kernel-mode uaccess always does, see the 'goto lock_mmap' path in
arch/x86/mm/fault.c for !FAULT_FLAG_USER), and __might_fault() records that
dependency for lockdep.  That creates the lock order:

mem_hotplug_lock(R) -> mmap_lock(R)

The opposite order already exists in the tree.  Code that hotplugs device
memory while holding mmap_read_lock goes through memremap_pages() ->
pagemap_range() -> mem_hotplug_begin(), which does cpus_read_lock() and then
percpu_down_write(&mem_hotplug_lock):

nouveau_svmm_bind()
    mmap_read_lock(mm)
    nouveau_dmem_migrate_vma()
      nouveau_dmem_migrate_chunk()
        nouveau_dmem_migrate_copy_one()
          nouveau_dmem_page_alloc_locked()
            nouveau_dmem_chunk_alloc()
              memremap_pages()

lib/test_hmm.c:dmirror_migrate_to_device()
    mmap_read_lock(mm)
    dmirror_migrate_alloc_and_copy()
      dmirror_devmem_alloc_page()
        dmirror_allocate_chunk()
          memremap_pages()

With three threads sharing one mm, could this sequence occur?

T1 read(page_owner):
    get_online_mems(), then copy_to_user() faults on buf
T2 SVM bind ioctl:
    holds mmap_read_lock(mm), blocks in percpu_down_write(&mem_hotplug_lock)
    waiting for T1
T3 mmap()/munmap():
    queues on mmap_write_lock(mm) behind T2
T1 fault:
    mmap_read_lock(mm) blocks behind T3

None of these waits has a timeout.  While T2 waits it also holds
cpus_read_lock(), so CPU hotplug writers and every later get_online_mems()
caller stall too.

Even without that cycle, the reader can now be held for as long as
userspace wants.  A buf backed by userfaultfd or by a slow or remote file
mapping could stall memory hot-remove for an unbounded time.  Does that
match the commit message claim:

    "Taking the lock per-page bounds the writer's wait to a single page's
    check-and-deref, keeping hot-remove stalls brief while all accesses stay
    protected."

For comparison, ptdump_walk_pgd() already takes get_online_mems() followed
by mmap_write_lock(current->mm).  This new instance sits on a much more
commonly used debug path, and here the lock isn't needed around
copy_to_user() at all.

Would it work to format the record under the reader (or snapshot the
page-derived fields: pageblock migratetype, page->flags, memcg info), call
put_online_mems(), and only then call copy_to_user()?  The kmalloc() in
print_page_owner() could also move outside the locked region.

The last commit in the series (5b123f2cd336 "selftests/bpf: add tests for
the page_owner bpf_iter target") still calls print_page_owner() under the
reader, so the series does not address this later.

>
>  skip_continue:
> +		put_online_mems();
>  		scan.pfn++;
>  		cond_resched();
>  	}


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39593164040

[PATCH v3 3/7] mm/page_owner: add bpf_iter target "page_owner"

[Zhen Ni]

Register a bpf_iter target "page_owner" that walks the pages eligible
for page_owner output, with the same eligibility rules as the
/sys/kernel/debug/page_owner read path.

For each eligible page the attached BPF program receives the pfn and the
page_owner record, and can freely filter and format output.

The po_snap record in the scan cursor is taken inside the page_ext
RCU window and handed to the BPF program as a copy, so no RCU window
is needed at consumption time. Torn reads remain possible if the
record is concurrently updated while it is copied, but this is the
same best-effort behavior as the read path has always had.

The target is registered from pageowner_init() when both
CONFIG_PAGE_OWNER and CONFIG_BPF_SYSCALL are enabled; a registration
failure only disables the bpf_iter target.

Userspace tools such as bpftrace can attach to this target like any
other bpf_iter. For example, to count pages owned by init (pid 1):

    bpftrace -e 'iter:page_owner / ctx->po->pid == 1 /
                 { @ = sum(1 << ctx->po->order); }'

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Drop the page pointer from the BPF context.
- Enable BPF_ITER_RESCHED.
- Fix the bpftrace example in the commit message.

Changes in v2:
- Drop the scan->pfn++.
- Fill page/po/pfn only when v is non-NULL.
- Implement page_owner_seq_stop().

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-3-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-3-zhen.ni@easystack.cn/
---
 mm/Makefile          |   3 ++
 mm/page_owner.c      |  32 +++---------
 mm/page_owner.h      |  47 +++++++++++++++++
 mm/page_owner_iter.c | 118 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 175 insertions(+), 25 deletions(-)
 create mode 100644 mm/page_owner.h
 create mode 100644 mm/page_owner_iter.c

diff --git a/mm/Makefile b/mm/Makefile
index e7245cb88c66..d7307b5a964c 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -115,6 +115,9 @@ obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
 obj-$(CONFIG_DEBUG_RODATA_TEST) += rodata_test.o
 obj-$(CONFIG_DEBUG_VM_PGTABLE) += debug_vm_pgtable.o
 obj-$(CONFIG_PAGE_OWNER) += page_owner.o
+ifdef CONFIG_BPF_SYSCALL
+obj-$(CONFIG_PAGE_OWNER) += page_owner_iter.o
+endif
 obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o
 obj-$(CONFIG_ZSMALLOC)	+= zsmalloc.o
 obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o
diff --git a/mm/page_owner.c b/mm/page_owner.c
index 8e72a85c663c..ece3b69a2579 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -14,6 +14,7 @@
 #include <linux/sched/clock.h>
 
 #include "page_alloc.h"
+#include "page_owner.h"
 
 /*
  * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack)
@@ -21,30 +22,6 @@
  */
 #define PAGE_OWNER_STACK_DEPTH (16)
 
-struct page_owner {
-	unsigned short order;
-	short last_migrate_reason;
-	gfp_t gfp_mask;
-	depot_stack_handle_t handle;
-	depot_stack_handle_t free_handle;
-	u64 ts_nsec;
-	u64 free_ts_nsec;
-	char comm[TASK_COMM_LEN];
-	pid_t pid;
-	pid_t tgid;
-	pid_t free_pid;
-	pid_t free_tgid;
-};
-
-/*
- * Cursor and per-page result of page_owner_next_eligible().
- */
-struct page_owner_scan {
-	/* resume point */
-	unsigned long pfn;
-	struct page_owner po_snap;
-};
-
 struct stack {
 	struct stack_record *stack_record;
 	struct stack *next;
@@ -740,7 +717,7 @@ void __dump_page_owner(const struct page *page)
  * must advance scan->pfn past the hit before calling again to resume
  * the scan.
  */
-static bool page_owner_next_eligible(struct page_owner_scan *scan)
+bool page_owner_next_eligible(struct page_owner_scan *scan)
 {
 	unsigned long pfn = scan->pfn;
 
@@ -1221,6 +1198,11 @@ static int __init pageowner_init(void)
 			    &stack_fops);
 	debugfs_create_file("count_threshold", 0600, dir, NULL,
 			    &threshold_fops);
+
+#ifdef CONFIG_BPF_SYSCALL
+	if (page_owner_iter_register())
+		pr_warn("page_owner: failed to register bpf_iter target\n");
+#endif
 	return 0;
 }
 late_initcall(pageowner_init)
diff --git a/mm/page_owner.h b/mm/page_owner.h
new file mode 100644
index 000000000000..0aa7695bec82
--- /dev/null
+++ b/mm/page_owner.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * mm/ internal page_owner and page_owner_iter declarations
+ */
+
+#ifndef __MM_PAGE_OWNER_H
+#define __MM_PAGE_OWNER_H
+
+#include <linux/page_ext.h>
+#include <linux/sched.h>
+#include <linux/stackdepot.h>
+
+/*
+ * mm/page_owner.c
+ */
+struct page_owner {
+	unsigned short order;
+	short last_migrate_reason;
+	gfp_t gfp_mask;
+	depot_stack_handle_t handle;
+	depot_stack_handle_t free_handle;
+	u64 ts_nsec;
+	u64 free_ts_nsec;
+	char comm[TASK_COMM_LEN];
+	pid_t pid;
+	pid_t tgid;
+	pid_t free_pid;
+	pid_t free_tgid;
+};
+
+/*
+ * Cursor and per-page result of page_owner_next_eligible().
+ */
+struct page_owner_scan {
+	/* resume point */
+	unsigned long pfn;
+	struct page_owner po_snap;
+};
+
+bool page_owner_next_eligible(struct page_owner_scan *scan);
+
+/*
+ * mm/page_owner_iter.c
+ */
+int page_owner_iter_register(void);
+
+#endif /* __MM_PAGE_OWNER_H */
diff --git a/mm/page_owner_iter.c b/mm/page_owner_iter.c
new file mode 100644
index 000000000000..529b2c4a589a
--- /dev/null
+++ b/mm/page_owner_iter.c
@@ -0,0 +1,118 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <linux/btf_ids.h>
+#include <linux/memblock.h>
+#include <linux/mm.h>
+#include <linux/seq_file.h>
+
+#include "page_owner.h"
+
+struct bpf_iter__page_owner {
+	__bpf_md_ptr(struct bpf_iter_meta *, meta);
+	__bpf_md_ptr(struct page_owner *, po);
+	u64 pfn;
+};
+
+DEFINE_BPF_ITER_FUNC(page_owner, struct bpf_iter_meta *meta,
+		     struct page_owner *po, u64 pfn)
+
+static void *page_owner_seq_start(struct seq_file *seq, loff_t *posp)
+{
+	struct page_owner_scan *scan = seq->private;
+
+	if (*posp == 0)
+		scan->pfn = min_low_pfn;
+
+	if (!page_owner_next_eligible(scan))
+		return NULL;
+	return &scan->po_snap;
+}
+
+static void *page_owner_seq_next(struct seq_file *seq, void *v, loff_t *posp)
+{
+	struct page_owner_scan *scan = seq->private;
+
+	++*posp;
+	scan->pfn++;
+	if (!page_owner_next_eligible(scan))
+		return NULL;
+
+	return &scan->po_snap;
+}
+
+static int page_owner_prog_seq_show(struct bpf_prog *prog,
+				    struct bpf_iter_meta *meta, void *v)
+{
+	struct page_owner_scan *scan = meta->seq->private;
+	struct bpf_iter__page_owner ctx = {};
+
+	ctx.meta = meta;
+	if (v) {
+		/* @po: hit-time snapshot, no RCU window needed. */
+		ctx.po = &scan->po_snap;
+		ctx.pfn = scan->pfn;
+	}
+
+	return bpf_iter_run_prog(prog, &ctx);
+}
+
+static int page_owner_seq_show(struct seq_file *seq, void *v)
+{
+	struct bpf_iter_meta meta;
+	struct bpf_prog *prog;
+
+	meta.seq = seq;
+	prog = bpf_iter_get_info(&meta, false);
+	if (!prog)
+		return 0;
+
+	return page_owner_prog_seq_show(prog, &meta, v);
+}
+
+static void page_owner_seq_stop(struct seq_file *seq, void *v)
+{
+	if (!v) {
+		struct bpf_iter_meta meta = { .seq = seq };
+		struct bpf_prog *prog;
+
+		prog = bpf_iter_get_info(&meta, true);
+		if (prog)
+			page_owner_prog_seq_show(prog, &meta, NULL);
+	}
+}
+
+static const struct seq_operations page_owner_iter_seq_ops = {
+	.start	= page_owner_seq_start,
+	.next	= page_owner_seq_next,
+	.stop	= page_owner_seq_stop,
+	.show	= page_owner_seq_show,
+};
+
+static const struct bpf_iter_seq_info page_owner_iter_seq_info = {
+	.seq_ops		= &page_owner_iter_seq_ops,
+	.init_seq_private	= NULL,
+	.fini_seq_private	= NULL,
+	.seq_priv_size		= sizeof(struct page_owner_scan),
+};
+
+static struct bpf_iter_reg page_owner_iter_reg_info = {
+	.target			= "page_owner",
+	.ctx_arg_info_size	= 1,
+	.ctx_arg_info		= {
+		{ offsetof(struct bpf_iter__page_owner, po),
+		  PTR_TO_BTF_ID_OR_NULL },
+	},
+	.seq_info		= &page_owner_iter_seq_info,
+	.feature		= BPF_ITER_RESCHED,
+};
+
+BTF_ID_LIST(page_owner_btf_ids)
+BTF_ID(struct, page_owner)
+
+int page_owner_iter_register(void)
+{
+	page_owner_iter_reg_info.ctx_arg_info[0].btf_id =
+		page_owner_btf_ids[0];
+
+	return bpf_iter_reg_target(&page_owner_iter_reg_info);
+}
-- 
2.20.1

[PATCH v3 4/7] mm/page_owner: add open-coded page_owner iterator

[Zhen Ni]

Add an open-coded bpf_iter over the pages eligible for page_owner
output, reusing page_owner_next_eligible().

A sleepable BPF program drives it with
bpf_iter_page_owner_new()/next()/destroy(); each next() returns the
page_owner_scan. po_snap is copied inside the page_ext RCU window, so
the program needs no RCU section when it reads it.

Also add bpf_page_owner_stack_snprint(), which renders the stack trace
of the current page into a caller-provided buffer.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Gate the kfuncs with KF_PERFMON.
- Check the page_owner_inited static key in the iterator.
- Without scan->page, start at min_low_pfn - 1 and let next() pre-increment.
- bpf_page_owner_stack_snprint() rejects a NULL or zero-size buffer.

Changes in v2:
- bpf_page_owner_stack_snprint() now takes the scan cursor.
- Buffer args renamed to buf__uninit/buf__sz.

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-4-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-4-zhen.ni@easystack.cn/
---
 kernel/bpf/helpers.c |  9 +++++++
 mm/page_owner.h      |  3 +++
 mm/page_owner_iter.c | 64 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)

diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 712dca5a2c5b..0ba836f77957 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -4946,6 +4946,15 @@ BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
 BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
 BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
 #endif
+#ifdef CONFIG_PAGE_OWNER
+BTF_ID_FLAGS(func, bpf_iter_page_owner_new,
+	     KF_ITER_NEW | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_iter_page_owner_next,
+	     KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_iter_page_owner_destroy,
+	     KF_ITER_DESTROY | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_page_owner_stack_snprint, KF_SLEEPABLE | KF_PERFMON)
+#endif
 BTF_ID_FLAGS(func, __bpf_trap)
 BTF_ID_FLAGS(func, bpf_strcmp, KF_PERFMON);
 BTF_ID_FLAGS(func, bpf_strcasecmp, KF_PERFMON);
diff --git a/mm/page_owner.h b/mm/page_owner.h
index 0aa7695bec82..4ab513f17fc9 100644
--- a/mm/page_owner.h
+++ b/mm/page_owner.h
@@ -6,6 +6,7 @@
 #ifndef __MM_PAGE_OWNER_H
 #define __MM_PAGE_OWNER_H
 
+#include <linux/jump_label.h>
 #include <linux/page_ext.h>
 #include <linux/sched.h>
 #include <linux/stackdepot.h>
@@ -13,6 +14,8 @@
 /*
  * mm/page_owner.c
  */
+DECLARE_STATIC_KEY_FALSE(page_owner_inited);
+
 struct page_owner {
 	unsigned short order;
 	short last_migrate_reason;
diff --git a/mm/page_owner_iter.c b/mm/page_owner_iter.c
index 529b2c4a589a..4b9741a2de53 100644
--- a/mm/page_owner_iter.c
+++ b/mm/page_owner_iter.c
@@ -4,6 +4,7 @@
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/seq_file.h>
+#include <linux/stackdepot.h>
 
 #include "page_owner.h"
 
@@ -116,3 +117,66 @@ int page_owner_iter_register(void)
 
 	return bpf_iter_reg_target(&page_owner_iter_reg_info);
 }
+
+/* opaque to BPF programs */
+struct bpf_iter_page_owner {
+	__u64 __opaque[16];
+} __attribute__((aligned(8)));
+
+struct bpf_iter_page_owner_kern {
+	struct page_owner_scan scan;
+} __attribute__((aligned(8)));
+
+__bpf_kfunc_start_defs();
+
+__bpf_kfunc int bpf_iter_page_owner_new(struct bpf_iter_page_owner *it)
+{
+	struct bpf_iter_page_owner_kern *kit = (void *)it;
+
+	BUILD_BUG_ON(sizeof(*kit) > sizeof(*it));
+	BUILD_BUG_ON(__alignof__(*kit) != __alignof__(*it));
+
+	kit->scan.pfn = min_low_pfn - 1;
+	return 0;
+}
+
+__bpf_kfunc struct page_owner_scan *
+bpf_iter_page_owner_next(struct bpf_iter_page_owner *it)
+{
+	struct bpf_iter_page_owner_kern *kit = (void *)it;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		return NULL;
+
+	cond_resched();
+
+	kit->scan.pfn++;
+	if (!page_owner_next_eligible(&kit->scan))
+		return NULL;
+
+	return &kit->scan;
+}
+
+__bpf_kfunc void bpf_iter_page_owner_destroy(struct bpf_iter_page_owner *it)
+{
+}
+
+__bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan,
+					     char *buf__uninit, u32 buf__sz)
+{
+	int len = 0;
+
+	if (!buf__uninit || !buf__sz)
+		return 0;
+
+	if (static_branch_unlikely(&page_owner_inited))
+		len = stack_depot_snprint(scan->po_snap.handle, buf__uninit,
+					  buf__sz, 0);
+
+	if (!len)
+		buf__uninit[0] = '\0';
+
+	return len;
+}
+
+__bpf_kfunc_end_defs();
-- 
2.20.1

[PATCH v3 5/7] mm/page_owner: record the NUMA node in the scan cursor

[Zhen Ni]

Compute the page's NUMA node in page_owner_next_eligible() while the
page is still valid, and store it in scan->nid.

The debugfs nid filter reads scan->nid instead of recomputing it, and a
BPF program iterating the cursor reads the node straight from scan->nid.

Resolve nid at scan time to avoid dereferencing a possibly-unmapped page.
Keep it in the transient scan cursor rather than struct page_owner, so
nid does not add per-page memory overhead.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Add @scan->nid to the comment describing what is filled.

Changes in v2:
- Removes the v1 get_nid() kfunc. The node is now stored in scan->nid.

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-5-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-5-zhen.ni@easystack.cn/
---
 mm/page_owner.c      | 29 +++++++++++++++++------------
 mm/page_owner.h      |  1 +
 mm/page_owner_iter.c |  4 +++-
 3 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/mm/page_owner.c b/mm/page_owner.c
index ece3b69a2579..8bc15bcdc24e 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -711,9 +711,22 @@ void __dump_page_owner(const struct page *page)
 	page_ext_put(page_ext);
 }
 
+/*
+ * NUMA node of @page, bypassing PF_POISONED_CHECK() in page_to_nid() to
+ * avoid VM_BUG_ON on poisoned pages; -1 for poisoned pages.
+ */
+static int page_owner_get_nid(struct page *page)
+{
+	memdesc_flags_t page_flags = READ_ONCE(page->flags);
+
+	if (page_flags.f == PAGE_POISON_PATTERN)
+		return -1;
+	return memdesc_nid(&page_flags);
+}
+
 /*
  * Advance @scan to the next page eligible for page_owner output.
- * On a hit, fill scan->pfn/po_snap and return true; the caller
+ * On a hit, fill scan->pfn/po_snap/nid and return true; the caller
  * must advance scan->pfn past the hit before calling again to resume
  * the scan.
  */
@@ -790,6 +803,7 @@ bool page_owner_next_eligible(struct page_owner_scan *scan)
 		if (!scan->po_snap.handle)
 			goto ext_put_continue;
 
+		scan->nid = page_owner_get_nid(page);
 		page_ext_put(page_ext);
 		put_online_mems();
 		return true;
@@ -833,17 +847,8 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			goto skip_continue;
 
 		if (state->nid_filter_enabled) {
-			int nid;
-			memdesc_flags_t page_flags = READ_ONCE(page->flags);
-
-			/*
-			 * Bypass PF_POISONED_CHECK() in page_to_nid() to avoid
-			 * VM_BUG_ON when accessing poisoned pages.
-			 */
-			if (page_flags.f == PAGE_POISON_PATTERN)
-				goto skip_continue;
-			nid = memdesc_nid(&page_flags);
-			if (!node_isset(nid, state->nid_filter))
+			if (scan.nid < 0 ||
+			    !node_isset(scan.nid, state->nid_filter))
 				goto skip_continue;
 		}
 
diff --git a/mm/page_owner.h b/mm/page_owner.h
index 4ab513f17fc9..934bc9f84d6f 100644
--- a/mm/page_owner.h
+++ b/mm/page_owner.h
@@ -38,6 +38,7 @@ struct page_owner_scan {
 	/* resume point */
 	unsigned long pfn;
 	struct page_owner po_snap;
+	int nid;
 };
 
 bool page_owner_next_eligible(struct page_owner_scan *scan);
diff --git a/mm/page_owner_iter.c b/mm/page_owner_iter.c
index 4b9741a2de53..b7e0ce1dfa80 100644
--- a/mm/page_owner_iter.c
+++ b/mm/page_owner_iter.c
@@ -12,10 +12,11 @@ struct bpf_iter__page_owner {
 	__bpf_md_ptr(struct bpf_iter_meta *, meta);
 	__bpf_md_ptr(struct page_owner *, po);
 	u64 pfn;
+	int nid;
 };
 
 DEFINE_BPF_ITER_FUNC(page_owner, struct bpf_iter_meta *meta,
-		     struct page_owner *po, u64 pfn)
+		     struct page_owner *po, u64 pfn, int nid)
 
 static void *page_owner_seq_start(struct seq_file *seq, loff_t *posp)
 {
@@ -52,6 +53,7 @@ static int page_owner_prog_seq_show(struct bpf_prog *prog,
 		/* @po: hit-time snapshot, no RCU window needed. */
 		ctx.po = &scan->po_snap;
 		ctx.pfn = scan->pfn;
+		ctx.nid = scan->nid;
 	}
 
 	return bpf_iter_run_prog(prog, &ctx);
-- 
2.20.1

[PATCH v3 6/7] mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc

[Zhen Ni]

Expose per-page memcg information to BPF programs, so that a bpf_iter
over page_owner can aggregate and filter allocations by memcg.
Attributing an allocation to a specific cgroup needs its full cgroup
path, not just the memcg name.

Split memcg info collection out of print_page_owner_memcg() into
get_page_memcg_info(), which fills struct memcg_info and, when given a
caller-provided buffer, writes the full cgroup path via cgroup_path().
print_page_owner_memcg() now consumes the struct; this introduces no
functional change to the debugfs output.

The helper takes a single READ_ONCE snapshot within one RCU read-side
critical section, so the flags and cgroup path reflect a consistent
view of the page. cgroup_path() returns the path relative to the cgroup
hierarchy root, so memcg attribution works on both cgroup v1 and v2.

Because scan->pfn is selected by a lockless scan, the kfunc re-resolves
it with pfn_to_online_page() under get_online_mems() to avoid accessing
a hot-removed section. If the PFN is offline or removed, the kfunc
returns -ENODATA instead of dereferencing memcg_data.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Check the page_owner_inited static key.
- Resolve the page with pfn_to_online_page() under get_online_mems().

Changes in v2:
- bpf_page_owner_get_memcg_info() now resolves the page with
  pfn_to_online_page() for an offline or removed pfn.
- Buffer/info args renamed to info__uninit/buf__uninit/buf__sz.

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-6-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-6-zhen.ni@easystack.cn/
---
 kernel/bpf/helpers.c |  1 +
 mm/page_owner.c      | 85 ++++++++++++++++++++++++++++++++++----------
 mm/page_owner.h      |  9 +++++
 mm/page_owner_iter.c | 48 +++++++++++++++++++++++++
 4 files changed, 124 insertions(+), 19 deletions(-)

diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 0ba836f77957..bba9eeb076c3 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -4954,6 +4954,7 @@ BTF_ID_FLAGS(func, bpf_iter_page_owner_next,
 BTF_ID_FLAGS(func, bpf_iter_page_owner_destroy,
 	     KF_ITER_DESTROY | KF_SLEEPABLE | KF_PERFMON)
 BTF_ID_FLAGS(func, bpf_page_owner_stack_snprint, KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_page_owner_get_memcg_info, KF_SLEEPABLE | KF_PERFMON)
 #endif
 BTF_ID_FLAGS(func, __bpf_trap)
 BTF_ID_FLAGS(func, bpf_strcmp, KF_PERFMON);
diff --git a/mm/page_owner.c b/mm/page_owner.c
index 8bc15bcdc24e..7470b5d002be 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -538,16 +538,22 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m,
 
 #ifdef CONFIG_MEMCG
 /*
- * Looking for memcg information and print it out
+ * Collect memcg information of @page into @info; when @buf and a
+ * nonzero @size are given, also copy the full cgroup path of the
+ * charged memcg into @buf.
+ *
+ * Return: 0 if @page is charged to an attributable memcg; -ENODATA
+ * otherwise.
  */
-static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
-					 struct page *page)
+int get_page_memcg_info(struct page *page, struct memcg_info *info,
+			char *buf, int size)
 {
 	unsigned long memcg_data;
 	struct obj_cgroup *objcg;
 	struct mem_cgroup *memcg;
-	bool online;
-	char name[80];
+	int ret = -ENODATA;
+
+	*info = (struct memcg_info){};
 
 	rcu_read_lock();
 	memcg_data = READ_ONCE(page->memcg_data);
@@ -555,31 +561,68 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
 		goto out_unlock;
 
 	if (memcg_data & MEMCG_DATA_OBJEXTS) {
-		ret += scnprintf(kbuf + ret, count - ret,
-				"Slab cache page\n");
+		info->is_slab = true;
 		goto out_unlock;
 	}
 
 	objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
-	memcg = objcg ? obj_cgroup_memcg(objcg) : NULL;
+	if (!objcg)
+		goto out_unlock;
+
+	memcg = obj_cgroup_memcg(objcg);
 	if (!memcg)
 		goto out_unlock;
 
-	online = css_is_online(&memcg->css);
-	cgroup_name(memcg->css.cgroup, name, sizeof(name));
-	ret += scnprintf(kbuf + ret, count - ret,
-			"Charged %sto %smemcg %s\n",
-			(memcg_data & MEMCG_DATA_KMEM) ? "(via objcg) " : "",
-			online ? "" : "offline ",
-			name);
+	info->is_kmem = (memcg_data & MEMCG_DATA_KMEM) != 0;
+	info->is_online = css_is_online(&memcg->css);
+	cgroup_name(memcg->css.cgroup, info->name, sizeof(info->name));
+
+	if (buf && size)
+		cgroup_path(memcg->css.cgroup, buf, size);
+	ret = 0;
 out_unlock:
 	rcu_read_unlock();
 
+	if (ret < 0 && buf && size)
+		buf[0] = '\0';
+
+	return ret;
+}
+
+/*
+ * Print memcg information from memcg_info
+ */
+static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
+					 const struct memcg_info *info)
+{
+	if (!info)
+		return ret;
+
+	if (info->is_slab)
+		ret += scnprintf(kbuf + ret, count - ret,
+				"Slab cache page\n");
+
+	if (info->name[0])
+		ret += scnprintf(kbuf + ret, count - ret,
+				"Charged %sto %smemcg %s\n",
+				info->is_kmem ? "(via objcg) " : "",
+				info->is_online ? "" : "offline ",
+				info->name);
+
 	return ret;
 }
 #else
+int get_page_memcg_info(struct page *page, struct memcg_info *info,
+			char *buf, int size)
+{
+	*info = (struct memcg_info){};
+	if (buf && size)
+		buf[0] = '\0';
+	return -EOPNOTSUPP;
+}
+
 static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
-					 struct page *page)
+					 const struct memcg_info *info)
 {
 	return ret;
 }
@@ -589,7 +632,8 @@ static ssize_t
 print_page_owner(char __user *buf, size_t count, unsigned long pfn,
 		struct page *page, struct page_owner *page_owner,
 		depot_stack_handle_t handle,
-		struct page_owner_filter_state *state)
+		struct page_owner_filter_state *state,
+		const struct memcg_info *memcg_info)
 {
 	int ret, pageblock_mt, page_mt;
 	char *kbuf;
@@ -639,7 +683,7 @@ print_page_owner(char __user *buf, size_t count, unsigned long pfn,
 			migrate_reason_names[page_owner->last_migrate_reason]);
 	}
 
-	ret = print_page_owner_memcg(kbuf, count, ret, page);
+	ret = print_page_owner_memcg(kbuf, count, ret, memcg_info);
 
 	ret += snprintf(kbuf + ret, count - ret, "\n");
 	if (ret >= count)
@@ -822,6 +866,7 @@ static ssize_t
 read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 {
 	struct page_owner_scan scan;
+	struct memcg_info memcg_info;
 	struct page_owner_filter_state *state = file->private_data;
 
 	if (!static_branch_unlikely(&page_owner_inited))
@@ -855,8 +900,10 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 		/* Record the next PFN to read in the file offset */
 		*ppos = scan.pfn + 1;
 
+		get_page_memcg_info(page, &memcg_info, NULL, 0);
 		ret = print_page_owner(buf, count, scan.pfn, page,
-				&scan.po_snap, scan.po_snap.handle, state);
+				&scan.po_snap, scan.po_snap.handle, state,
+				&memcg_info);
 		put_online_mems();
 		return ret;
 
diff --git a/mm/page_owner.h b/mm/page_owner.h
index 934bc9f84d6f..65598415af73 100644
--- a/mm/page_owner.h
+++ b/mm/page_owner.h
@@ -41,7 +41,16 @@ struct page_owner_scan {
 	int nid;
 };
 
+struct memcg_info {
+	char name[80];
+	bool is_slab;
+	bool is_kmem;
+	bool is_online;
+};
+
 bool page_owner_next_eligible(struct page_owner_scan *scan);
+int get_page_memcg_info(struct page *page, struct memcg_info *info,
+			char *buf, int size);
 
 /*
  * mm/page_owner_iter.c
diff --git a/mm/page_owner_iter.c b/mm/page_owner_iter.c
index b7e0ce1dfa80..a649e1207e24 100644
--- a/mm/page_owner_iter.c
+++ b/mm/page_owner_iter.c
@@ -181,4 +181,52 @@ __bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan,
 	return len;
 }
 
+/**
+ * bpf_page_owner_get_memcg_info - Report the memcg charged for the current page
+ * @scan: page_owner scan handle.
+ * @info__uninit: output buffer for the memcg_info.
+ * @buf__uninit: optional output buffer for the full cgroup path.
+ * @buf__sz: size of @buf__uninit.
+ *
+ * Fills @info__uninit with the memcg charged for the page backing
+ * @scan->pfn.  If @buf__uninit and @buf__sz are provided, the full
+ * cgroup path is also copied out, so a BPF program can attribute a
+ * page_owner allocation to a specific cgroup.
+ *
+ * @scan->pfn is resolved to a struct page under get_online_mems(),
+ * so the page cannot be accessed after its memory section is offlined
+ * or removed.
+ *
+ * Returns:
+ * * 0 on success
+ * * -ENODATA if the page is not online or not charged to a memcg
+ * * -EOPNOTSUPP if built without CONFIG_MEMCG
+ */
+__bpf_kfunc int bpf_page_owner_get_memcg_info(struct page_owner_scan *scan,
+					      struct memcg_info *info__uninit,
+					      char *buf__uninit, u32 buf__sz)
+{
+	struct page *page;
+	int ret = -ENODATA;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		goto nodata;
+
+	get_online_mems();
+	page = pfn_to_online_page(scan->pfn);
+	if (!page) {
+		put_online_mems();
+		goto nodata;
+	}
+	ret = get_page_memcg_info(page, info__uninit, buf__uninit, buf__sz);
+	put_online_mems();
+	return ret;
+
+nodata:
+	*info__uninit = (struct memcg_info){};
+	if (buf__uninit && buf__sz)
+		buf__uninit[0] = '\0';
+	return ret;
+}
+
 __bpf_kfunc_end_defs();
-- 
2.20.1

[PATCH v3 7/7] selftests/bpf: add tests for the page_owner bpf_iter target

[Zhen Ni]

Add a sleepable BPF program and a userspace runner covering the seq_file
and open-coded page_owner iterators. The test checks:

  - seq_file walk, unfiltered and pid-filtered: records read back from
    the iterator fd match the lines the program printed.
  - open-coded iterator parity with the seq path.
  - the kfuncs' contracts: a non-empty stack trace, and memcg
    classification.

CONFIG_PAGE_OWNER is enabled in the bpf selftests config so the
program's page_owner BTF types are available. Skipped when debugfs
page_owner is unavailable.

Signed-off-by: Zhen Ni <zhen.ni@easystack.cn>
---
Changes in v3:
- Check the ENODATA/EOPNOTSUPP instead of local macros.

Changes in v2:
- Drop the get_nid kfunc coverage.
- Enable CONFIG_PAGE_OWNER in the bpf selftests config.

v2: https://lore.kernel.org/linux-mm/20261013122125.843451-7-zhen.ni@easystack.cn/
v1: https://lore.kernel.org/linux-mm/20261009112308.240769-7-zhen.ni@easystack.cn/
---
 tools/testing/selftests/bpf/config            |   2 +
 .../bpf/prog_tests/page_owner_iter.c          | 193 ++++++++++++++++++
 .../selftests/bpf/progs/bpf_iter_page_owner.c | 157 ++++++++++++++
 3 files changed, 352 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c

diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index ea7044f30adc..731bc3d79881 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -136,3 +136,5 @@ CONFIG_SMC_HS_CTRL_BPF=y
 CONFIG_DIBS=y
 CONFIG_DIBS_LO=y
 CONFIG_PM_WAKELOCKS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_PAGE_OWNER=y
diff --git a/tools/testing/selftests/bpf/prog_tests/page_owner_iter.c b/tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
new file mode 100644
index 000000000000..3684ca308935
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 */
+
+#include <test_progs.h>
+#include "bpf_iter_page_owner.skel.h"
+
+#define DEBUGFS_PAGE_OWNER	"/sys/kernel/debug/page_owner"
+
+static bool page_owner_available(void)
+{
+	return access(DEBUGFS_PAGE_OWNER, R_OK) == 0;
+}
+
+/* Count link_fd output lines; -1 on error. */
+static long iter_record_count(int link_fd)
+{
+	char *line = NULL;
+	size_t len = 0;
+	long count = 0;
+	int iter_fd;
+	FILE *f;
+
+	iter_fd = bpf_iter_create(link_fd);
+	if (!ASSERT_GE(iter_fd, 0, "bpf_iter_create"))
+		return -1;
+
+	f = fdopen(iter_fd, "r");
+	if (!ASSERT_OK_PTR(f, "fdopen")) {
+		close(iter_fd);
+		return -1;
+	}
+
+	for (;;) {
+		if (getline(&line, &len, f) > 0) {
+			count++;
+			continue;
+		}
+		if (!ferror(f))
+			break;		/* clean EOF */
+		/*
+		 * bpf_seq_read() returns -EAGAIN when a chunk scanned.
+		 * Not an error, just retry the read.
+		 */
+		if (errno == EAGAIN) {
+			clearerr(f);
+			continue;
+		}
+		ASSERT_OK(errno, "iter_read");
+		count = -1;
+		break;
+	}
+
+	free(line);
+	fclose(f);
+	return count;
+}
+
+/* Validate bpf_page_owner_get_memcg_info() results from the open-coded run. */
+static void check_memcg_skel(struct bpf_iter_page_owner *skel)
+{
+	fprintf(stderr,
+		"memcg: seen=%u charged=%u online=%u nodata=%u notsup=%u bad=%u first='%s' name='%s' flags=%u\n",
+		skel->bss->memcg_seen, skel->bss->memcg_charged,
+		skel->bss->memcg_online, skel->bss->memcg_nodata,
+		skel->bss->memcg_notsup, skel->bss->memcg_bad,
+		skel->bss->memcg_first_path, skel->bss->memcg_first_name,
+		skel->bss->memcg_first_flags);
+
+	if (!ASSERT_GT(skel->bss->memcg_seen, 0, "memcg_seen_gt_0"))
+		return;
+
+	/* all -EOPNOTSUPP: CONFIG_MEMCG is off */
+	if (skel->bss->memcg_notsup == skel->bss->memcg_seen) {
+		ASSERT_EQ(skel->bss->memcg_bad, 0, "memcg_stub_invariants");
+		return;
+	}
+
+	if (!ASSERT_EQ(skel->bss->memcg_bad, 0, "memcg_kfunc_invariants"))
+		return;
+
+	/* Every call is either charged or nodata */
+	if (!ASSERT_EQ(skel->bss->memcg_charged + skel->bss->memcg_nodata,
+		       skel->bss->memcg_seen, "memcg_ret_classified"))
+		return;
+
+	if (!skel->bss->memcg_first_saved)
+		return;
+
+	if (!ASSERT_EQ(skel->bss->memcg_first_path[0], '/',
+		       "memcg_first_path_abs"))
+		return;
+
+	if (skel->bss->memcg_first_name[0])
+		ASSERT_OK_PTR(strstr(skel->bss->memcg_first_path,
+				    skel->bss->memcg_first_name),
+			      "memcg_name_in_path");
+}
+
+void serial_test_page_owner_iter(void)
+{
+	struct bpf_iter_page_owner *skel = NULL;
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	long iter_count, unfiltered_count, oc_diff, pid;
+	unsigned int printed_before;
+	int link_fd, prog_fd;
+
+	if (!page_owner_available()) {
+		printf("%s:SKIP:page_owner debugfs unavailable "
+		       "(CONFIG_PAGE_OWNER + page_owner=on)\n", __func__);
+		test__skip();
+		return;
+	}
+
+	skel = bpf_iter_page_owner__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "bpf_iter_page_owner__open_and_load"))
+		return;
+
+	/* --- seq file iterator: unfiltered run --- */
+	skel->bss->filter_en = 0;
+
+	if (!ASSERT_OK(bpf_iter_page_owner__attach(skel), "skel_attach"))
+		goto destroy;
+
+	link_fd = bpf_link__fd(skel->links.dump_page_owner);
+	iter_count = iter_record_count(link_fd);
+	if (iter_count < 0)
+		goto destroy;
+	unfiltered_count = iter_count;
+
+	fprintf(stderr, "iter_count=%ld printed_lines=%u total_seen=%u\n",
+		iter_count, skel->bss->printed_lines, skel->bss->total_seen);
+
+	ASSERT_GT(iter_count, 0, "unfiltered_lines_gt_0");
+
+	ASSERT_EQ((long)skel->bss->printed_lines, iter_count,
+		  "lines_match_printed");
+
+	/* unfiltered: every visited page is printed, so the two must match */
+	ASSERT_EQ(skel->bss->total_seen, skel->bss->printed_lines,
+		  "unfiltered_all_printed");
+
+	/* --- seq file iterator: pid-filtered run --- */
+	pid = getpid();
+	skel->bss->filter_en = 1;
+	skel->bss->filter_pid = (pid_t)pid;
+
+	/* printed_lines is cumulative across runs: compare the delta */
+	printed_before = skel->bss->printed_lines;
+	iter_count = iter_record_count(link_fd);
+	if (iter_count < 0)
+		goto destroy;
+
+	fprintf(stderr,
+		"pid=%ld iter_count=%ld printed_delta=%u\n",
+		pid, iter_count, skel->bss->printed_lines - printed_before);
+
+	ASSERT_GT(iter_count, 0, "filtered_lines_gt_0");
+	ASSERT_EQ(skel->bss->printed_lines - printed_before, iter_count,
+		  "filtered_printed_match");
+
+	/* --- open-coded iterator parity --- */
+	prog_fd = bpf_program__fd(skel->progs.scan_open_coded);
+
+	ASSERT_OK(bpf_prog_test_run_opts(prog_fd, &opts), "prog_test_run");
+	fprintf(stderr,
+		"open_coded_seen=%u nid_seen=%u first_nid=%d "
+		"stack_bytes=%u stack=[\n%s]\n",
+		skel->bss->open_coded_seen, skel->bss->nid_seen,
+		skel->data->first_nid, skel->bss->stack_bytes,
+		skel->bss->stack_text);
+
+	/* same eligibility as the seq path: counts must be close */
+	ASSERT_GT(skel->bss->open_coded_seen, 0, "oc_seen_gt_0");
+	oc_diff = (long)skel->bss->open_coded_seen - unfiltered_count;
+	if (oc_diff < 0)
+		oc_diff = -oc_diff;
+	ASSERT_LE(oc_diff, unfiltered_count / 100 + 64,
+		  "oc_seen_close_to_iter");
+
+	/* the nid kfunc returned a valid node for every page */
+	ASSERT_EQ(skel->bss->nid_seen, skel->bss->open_coded_seen,
+		  "nid_valid_all");
+	ASSERT_GE(skel->data->first_nid, 0, "first_nid_valid");
+
+	ASSERT_GT(skel->bss->stack_bytes, 0, "stack_snprint_ok");
+	ASSERT_OK_PTR(strstr(skel->bss->stack_text, "+0x"), "stack_frames");
+	ASSERT_OK_PTR(strchr(skel->bss->stack_text, '\n'), "stack_lines");
+
+	check_memcg_skel(skel);
+
+destroy:
+	bpf_iter_page_owner__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c b/tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c
new file mode 100644
index 000000000000..1cd99956aff3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c
@@ -0,0 +1,157 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <errno.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u32 filter_en;
+__u32 filter_pid;
+__u32 total_seen;
+__u32 printed_lines;
+
+__u32 open_coded_seen;
+__u32 nid_seen;
+__s32 first_nid = -1;
+char stack_text[2048];
+__u32 stack_bytes;
+
+__u32 memcg_seen;
+__u32 memcg_charged;
+__u32 memcg_online;
+__u32 memcg_nodata;
+__u32 memcg_notsup;
+__u32 memcg_bad;
+__u32 memcg_first_saved;
+__u32 memcg_first_flags;	/* bit0 is_online, bit1 is_kmem */
+__u32 memcg_first_path_len;
+char memcg_pathbuf[4096];
+char memcg_first_path[4096];
+char memcg_first_name[80];
+
+#define MEMCG_F_ONLINE	(1 << 0)
+#define MEMCG_F_OBJCG	(1 << 1)
+
+extern int bpf_iter_page_owner_new(struct bpf_iter_page_owner *it) __ksym;
+extern struct page_owner_scan *
+bpf_iter_page_owner_next(struct bpf_iter_page_owner *it) __ksym;
+extern void bpf_iter_page_owner_destroy(struct bpf_iter_page_owner *it) __ksym;
+extern int bpf_page_owner_stack_snprint(struct page_owner_scan *scan,
+					char *buf__uninit, u32 buf__sz) __ksym;
+extern int bpf_page_owner_get_memcg_info(struct page_owner_scan *scan,
+					 struct memcg_info *info__uninit,
+					 char *buf__uninit, u32 buf__sz) __ksym;
+
+static __u32 copy_str(char *dst, const char *src, const __u32 max)
+{
+	__u32 i = 0;
+
+	while (i + 1 < max && src[i]) {
+		dst[i] = src[i];
+		i++;
+	}
+	dst[i] = '\0';
+	return i;
+}
+
+/*
+ * Exercise bpf_page_owner_get_memcg_info() until 32 pages charged to an
+ * ONLINE memcg have been seen, checking its per-return contract:
+ *  - 0:          buf holds an absolute cgroup path;
+ *  - -ENODATA:   buf was NUL-terminated;
+ *  - -EOPNOTSUPP: counted (CONFIG_MEMCG is off);
+ *
+ * The first online charged sample is saved for userspace to print and
+ * cross-check (name vs basename of path).
+ */
+static void check_memcg(struct page_owner_scan *scan)
+{
+	static struct memcg_info info;
+	int err;
+
+	err = bpf_page_owner_get_memcg_info(scan, &info, memcg_pathbuf,
+					    sizeof(memcg_pathbuf));
+	memcg_seen++;
+
+	if (err == 0) {
+		memcg_charged++;
+
+		if (memcg_pathbuf[0] != '/') {
+			memcg_bad++;
+			return;
+		}
+
+		if (!info.is_online)
+			return;
+		memcg_online++;
+
+		/* keep the first online sample for userspace */
+		if (!memcg_first_saved) {
+			memcg_first_path_len = copy_str(memcg_first_path,
+							memcg_pathbuf,
+							sizeof(memcg_first_path));
+			copy_str(memcg_first_name, info.name,
+				 sizeof(memcg_first_name));
+			memcg_first_flags = MEMCG_F_ONLINE |
+					    (info.is_kmem ? MEMCG_F_OBJCG : 0);
+			memcg_first_saved = 1;
+		}
+	} else if (err == -ENODATA) {
+		memcg_nodata++;
+		if (memcg_pathbuf[0] != '\0')
+			memcg_bad++;
+	} else if (err == -EOPNOTSUPP) {
+		memcg_notsup++;
+		if (memcg_pathbuf[0] != '\0')
+			memcg_bad++;
+	} else {
+		memcg_bad++;
+	}
+}
+
+SEC("syscall")
+int scan_open_coded(const void *ctx)
+{
+	struct bpf_iter_page_owner it;
+	struct page_owner_scan *scan;
+
+	bpf_iter_page_owner_new(&it);
+
+	while ((scan = bpf_iter_page_owner_next(&it))) {
+		open_coded_seen++;
+		if (scan->nid >= 0)
+			nid_seen++;
+		if (memcg_online < 32)
+			check_memcg(scan);
+		/* grab the stack of the 100th page for userspace to check */
+		if (open_coded_seen == 100) {
+			first_nid = scan->nid;
+			if (scan->po_snap.handle)
+				stack_bytes = bpf_page_owner_stack_snprint(
+					scan, stack_text, sizeof(stack_text));
+		}
+	}
+	bpf_iter_page_owner_destroy(&it);
+	return 0;
+}
+
+SEC("iter/page_owner")
+int dump_page_owner(struct bpf_iter__page_owner *ctx)
+{
+	struct page_owner *po = ctx->po;
+
+	if (!po)
+		return 0;
+
+	total_seen++;
+
+	if (filter_en && po->pid != (pid_t)filter_pid)
+		return 0;
+
+	BPF_SEQ_PRINTF(ctx->meta->seq, "pfn %llu pid %d comm %s\n",
+		       ctx->pfn, po->pid, po->comm);
+	printed_lines++;
+	return 0;
+}
-- 
2.20.1

Re: [PATCH v3 0/7] mm/page_owner: add bpf_iter and kfuncs

[Andrew Morton]

On Wed, 21 Oct 2026 19:24:30 +0800 Zhen Ni <zhen.ni@easystack.cn> wrote:

> page_owner records who allocated every page and is a core memory-debug
> facility, but it currently reaches userspace as the debugfs flat dump:
> slicing allocations by pid, comm, NUMA node, cgroup or stack, etc.,
> means streaming and parsing the entire file in userspace, which is
> expensive on large hosts and hard to query live.

Confused.  Your series 8570b65220e8 ("mm/page_owner: add print_mode
filter") addressed this significantly?