[PATCH 0/2] mm: kmemleak: batch the struct page scan

[PATCH 0/2] mm: kmemleak: batch the struct page scan

[Breno Leitao]

kmemleak walks the struct page array as a scan root and hands it to
scan_block() one page at a time.

scan_block() takes kmemleak_lock with interrupts disabled for the
duration of the call, so the scanner acquires the lock once per online
PFN in order to look at a single 64-byte struct page.

Batching adjacent pages up to that size keeps exactly the same pages
under the same maximum lock hold time, with MAX_SCAN_SIZE / pagesize
fewer acquisitions.

Patch 1 lifts the loop out of __kmemleak_scan() into scan_zone_pages()
with no functional change. Patch 2 does the batching, which is then
contained in that one function.

This improves the performance due to less atomic operations, which is
not a big deal on a regular machine, but, given kmemleak usually comes
with extra debug options, such as PROVE_LOCKING, DEBUG_SPINLOCK, etc.

For instance, measuring Meta's "debug kernel flavor" on an arm64 hosts,
this improve the scan time by 20%.

It is safe to get more work into scan_block(), given it has the
protections, added by commit eb11f56eeca560 ("mm/kmemleak: stop the task
stack scan early when interrupted")

MAX_SCAN_SIZE is also not a new maximum for a single scan_block() call.
kmemleak_scan_task_stacks() already hands it a whole task stack in one
go. On arm64 and x86_64 THREAD_SIZE is never below 16 KiB, four times
MAX_SCAN_SIZE, and it is 64 KiB on arm64 with 64K pages.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (2):
      mm: kmemleak: move the struct page scan into a helper
      mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

 mm/kmemleak.c | 70 +++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 47 insertions(+), 23 deletions(-)
---
base-commit: 3f2425f5b5bbbdd991ca9cdfd5502e68d8895998
change-id: 20260918-b4-kmemleak-page-scan-c7ac6b088c4f

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH 1/2] mm: kmemleak: move the struct page scan into a helper

[Breno Leitao]

The struct page scanning loop sits inline in __kmemleak_scan(), nested
three levels deep and sharing the caller's "stop" variable with the
zone walk around it. Move it into scan_zone_pages(), which scans one
zone and returns 1 if scanning should stop.

No functional change; this only makes room for changing how the pages
are handed to scan_block().

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 mm/kmemleak.c | 57 ++++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 34 insertions(+), 23 deletions(-)

diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 8fa409a4f9fb2..4040547a0af84 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -1855,6 +1855,39 @@ static void dedup_flush(struct xarray *dedup)
 	}
 }
 
+/*
+ * Scan the struct pages of a zone, skipping memory holes, pages that belong to
+ * another zone and pages that are not in use. Returns 1 if the scan should be
+ * stopped.
+ */
+static int scan_zone_pages(struct zone *zone)
+{
+	unsigned long start_pfn = zone->zone_start_pfn;
+	unsigned long end_pfn = zone_end_pfn(zone);
+	unsigned long pfn;
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn++) {
+		struct page *page = pfn_to_online_page(pfn);
+
+		if (!(pfn & 63))
+			cond_resched_tasks_rcu_qs();
+
+		if (!page)
+			continue;
+
+		/* only scan pages belonging to this zone */
+		if (page_zone(page) != zone)
+			continue;
+		/* only scan if page is in use */
+		if (page_count(page) == 0)
+			continue;
+		if (scan_block(page, page + 1, NULL))
+			return 1;
+	}
+
+	return 0;
+}
+
 /*
  * Scan data sections and all the referenced memory blocks allocated via the
  * kernel's standard allocators. This function must be called with the
@@ -1928,29 +1961,7 @@ static int __kmemleak_scan(bool full)
 	 */
 	get_online_mems();
 	for_each_populated_zone(zone) {
-		unsigned long start_pfn = zone->zone_start_pfn;
-		unsigned long end_pfn = zone_end_pfn(zone);
-		unsigned long pfn;
-
-		for (pfn = start_pfn; pfn < end_pfn; pfn++) {
-			struct page *page = pfn_to_online_page(pfn);
-
-			if (!(pfn & 63))
-				cond_resched_tasks_rcu_qs();
-
-			if (!page)
-				continue;
-
-			/* only scan pages belonging to this zone */
-			if (page_zone(page) != zone)
-				continue;
-			/* only scan if page is in use */
-			if (page_count(page) == 0)
-				continue;
-			stop = scan_block(page, page + 1, NULL);
-			if (stop)
-				break;
-		}
+		stop = scan_zone_pages(zone);
 		if (stop)
 			break;
 	}

-- 
2.53.0-Meta

[PATCH 2/2] mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

[Breno Leitao]

scan_zone_pages() scans the struct page array one page per scan_block()
call:

	if (scan_block(page, page + 1, NULL))

scan_block() takes kmemleak_lock with interrupts disabled for the
duration of the call, so this acquires the lock once per online PFN to
scan a single struct page.

Gather runs of adjacent eligible struct pages and pass each run to
scan_block() in one call, capped at MAX_SCAN_SIZE.

The longest kmemleak_lock is now held is MAX_SCAN_SIZE worth of words,
the same bound scan_large_block() already applies to the data sections
and the per-CPU areas.

This can make the scan faster. On an arm64 VM with 24 GiB and ~17 GiB
in use, the struct page phase goes from 4390k scan_block() calls down
to 69k for the same 4390k pages scanned, and the whole scan about 20%
faster (on debug kernel). The win is in the per-acquisition cost, so on
a kernel built without the lock debugging options the scan time is
unchanged. I don't think it will be a problem doing more on
scan_block(), given we have the scan_should_stop() protection.

Coverage is unchanged: a temporary assertion comparing the number of
eligible pages against the number actually passed to scan_block()
matched on every zone of every scan, including while memory was being
freed underneath the scan.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 mm/kmemleak.c | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 4040547a0af84..cb953df3b414a 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -1862,8 +1862,11 @@ static void dedup_flush(struct xarray *dedup)
  */
 static int scan_zone_pages(struct zone *zone)
 {
+	const unsigned int max_batch = MAX_SCAN_SIZE / sizeof(struct page);
 	unsigned long start_pfn = zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(zone);
+	struct page *first = NULL, *last = NULL;
+	unsigned int batch = 0;
 	unsigned long pfn;
 
 	for (pfn = start_pfn; pfn < end_pfn; pfn++) {
@@ -1872,19 +1875,29 @@ static int scan_zone_pages(struct zone *zone)
 		if (!(pfn & 63))
 			cond_resched_tasks_rcu_qs();
 
-		if (!page)
-			continue;
+		/* only scan in-use pages belonging to this zone */
+		if (page && (page_zone(page) != zone ||
+			     page_count(page) == 0))
+			page = NULL;
 
-		/* only scan pages belonging to this zone */
-		if (page_zone(page) != zone)
-			continue;
-		/* only scan if page is in use */
-		if (page_count(page) == 0)
+		if (page && first && page == last + 1 &&
+		    batch < max_batch) {
+			last = page;
+			batch++;
 			continue;
-		if (scan_block(page, page + 1, NULL))
+		}
+
+		if (first && scan_block(first, last + 1, NULL))
 			return 1;
+
+		first = page;
+		last = page;
+		batch = page ? 1 : 0;
 	}
 
+	if (first && scan_block(first, last + 1, NULL))
+		return 1;
+
 	return 0;
 }
 

-- 
2.53.0-Meta

Re: [PATCH 1/2] mm: kmemleak: move the struct page scan into a helper

[Catalin Marinas]

On Mon, Sep 21, 2026 at 05:30:26AM -0700, Breno Leitao wrote:
> The struct page scanning loop sits inline in __kmemleak_scan(), nested
> three levels deep and sharing the caller's "stop" variable with the
> zone walk around it. Move it into scan_zone_pages(), which scans one
> zone and returns 1 if scanning should stop.
> 
> No functional change; this only makes room for changing how the pages
> are handed to scan_block().
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH 2/2] mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

[Catalin Marinas]

On Mon, Sep 21, 2026 at 05:30:27AM -0700, Breno Leitao wrote:
> scan_zone_pages() scans the struct page array one page per scan_block()
> call:
> 
> 	if (scan_block(page, page + 1, NULL))
> 
> scan_block() takes kmemleak_lock with interrupts disabled for the
> duration of the call, so this acquires the lock once per online PFN to
> scan a single struct page.
> 
> Gather runs of adjacent eligible struct pages and pass each run to
> scan_block() in one call, capped at MAX_SCAN_SIZE.
> 
> The longest kmemleak_lock is now held is MAX_SCAN_SIZE worth of words,

Remove an 'is'. Also, MAX_SCAN_SIZE worth of bytes rather than words.

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH 2/2] mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

[Andrew Morton]

On Tue, 22 Sep 2026 10:51:46 +0100 Catalin Marinas <catalin.marinas@arm.com> wrote:

> On Mon, Sep 21, 2026 at 05:30:27AM -0700, Breno Leitao wrote:
> > scan_zone_pages() scans the struct page array one page per scan_block()
> > call:
> > 
> > 	if (scan_block(page, page + 1, NULL))
> > 
> > scan_block() takes kmemleak_lock with interrupts disabled for the
> > duration of the call, so this acquires the lock once per online PFN to
> > scan a single struct page.
> > 
> > Gather runs of adjacent eligible struct pages and pass each run to
> > scan_block() in one call, capped at MAX_SCAN_SIZE.
> > 
> > The longest kmemleak_lock is now held is MAX_SCAN_SIZE worth of words,
> 
> Remove an 'is'.

That was actually correct, but it doesn't read well.

> Also, MAX_SCAN_SIZE worth of bytes rather than words.

edited, thanks.

How's this?

: The longest kmemleak_lock hold duration is now MAX_SCAN_SIZE worth of
: bytes.  The same bound scan_large_block() already applies to the data
: sections and the per-CPU areas.

Re: [PATCH 2/2] mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches

[Catalin Marinas]

On Tue, Sep 22, 2026 at 06:39:21PM -0700, Andrew Morton wrote:
> On Tue, 22 Sep 2026 10:51:46 +0100 Catalin Marinas <catalin.marinas@arm.com> wrote:
> 
> > On Mon, Sep 21, 2026 at 05:30:27AM -0700, Breno Leitao wrote:
> > > scan_zone_pages() scans the struct page array one page per scan_block()
> > > call:
> > > 
> > > 	if (scan_block(page, page + 1, NULL))
> > > 
> > > scan_block() takes kmemleak_lock with interrupts disabled for the
> > > duration of the call, so this acquires the lock once per online PFN to
> > > scan a single struct page.
> > > 
> > > Gather runs of adjacent eligible struct pages and pass each run to
> > > scan_block() in one call, capped at MAX_SCAN_SIZE.
> > > 
> > > The longest kmemleak_lock is now held is MAX_SCAN_SIZE worth of words,
> > 
> > Remove an 'is'.
> 
> That was actually correct, but it doesn't read well.

Ah, yes, you can read it in a correct way as well ;).

> > Also, MAX_SCAN_SIZE worth of bytes rather than words.
> 
> edited, thanks.
> 
> How's this?
> 
> : The longest kmemleak_lock hold duration is now MAX_SCAN_SIZE worth of
> : bytes.  The same bound scan_large_block() already applies to the data
> : sections and the per-CPU areas.

Looks good. Thanks!

-- 
Catalin