mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
	David Hildenbrand <david@kernel.org>,
	Vlastimil Babka <vbabka@kernel.org>,
	Brendan Jackman <jackmanb@google.com>,
	Michal Hocko <mhocko@suse.com>,
	Suren Baghdasaryan <surenb@google.com>,
	Jason Wang <jasowang@redhat.com>,
	Andrea Arcangeli <aarcange@redhat.com>,
	Dev Jain <dev.jain@arm.com>,
	Geert Uytterhoeven <geert@linux-m68k.org>,
	Greg Ungerer <gerg@linux-m68k.org>,
	Gregory Price <gourry@gourry.net>,
	"Huang, Ying" <ying.huang@linux.alibaba.com>,
	Lance Yang <lance.yang@linux.dev>,
	Lorenzo Stoakes <ljs@kernel.org>,
	Magnus Lindholm <linmag7@gmail.com>,
	Matthew Wilcox <willy@infradead.org>,
	Miaohe Lin <linmiaohe@huawei.com>, Zi Yan <ziy@nvidia.com>,
	linux-mm@kvack.org, virtualization@lists.linux.dev,
	"Liam R. Howlett" <liam@infradead.org>,
	Mike Rapoport <rppt@kernel.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Brendan Jackman <brendan.jackman@linux.dev>
Subject: [PATCH v11 1/7] mm: add PG_zeroed page flag and buddy merge tracking
Date: Thu, 19 Nov 2026 07:02:32 -0500	[thread overview]
Message-ID: <a98c8a49744d30a5e603228d9cb17509261add90.1795089715.git.mst@redhat.com> (raw)
In-Reply-To: <cover.1795089715.git.mst@redhat.com>

Add PG_zeroed (aliased to PG_private_2) to track pages whose contents
are known to be zero.  Propagate it through buddy merges and splits.
Upon folio allocation, PG_zeroed is cleared and its value is transferred
to BIT(2) in lru.next.

This bit, in turn, is tested by the new folio_init_user(). If clear, the
helper zero-initializes the folio. If set, it skips clearing the
contents.

Clear the aliased flag on free so filesystem state cannot be mistaken
for zeroed contents.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: LLM
---
 include/linux/highmem.h    | 21 +++++++++++++++++++++
 include/linux/mm.h         | 32 +++++++++++++++++++++++++++-----
 include/linux/page-flags.h | 15 ++++++++++++++-
 mm/page_alloc.c            | 38 ++++++++++++++++++++++++++++++++------
 4 files changed, 94 insertions(+), 12 deletions(-)

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a4..70bbe93b8d669 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -303,6 +303,27 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,
 #endif
 }
 
+/**
+ * folio_init_user - Initialize a freshly allocated folio for userspace.
+ * @folio: The folio freshly allocated from the buddy allocator.
+ * @addr_hint: The address accessed by the user or the base address.
+ *
+ * The caller must not have written to the folio or repurposed folio->lru
+ * since allocation: the pre-zeroed marker is only valid until then.
+ * Preserve known-zero contents, but initialize KMSAN metadata that the
+ * allocator may have poisoned. Otherwise, clear the folio if required.
+ */
+static inline void folio_init_user(struct folio *folio, unsigned long addr_hint)
+{
+	if (folio_is_zeroed(folio)) {
+		kmsan_unpoison_memory(folio_address(folio), folio_size(folio));
+		return;
+	}
+
+	if (user_alloc_needs_zeroing())
+		folio_zero_user(folio, addr_hint);
+}
+
 #ifndef vma_alloc_zeroed_movable_folio
 /**
  * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6af3b7008f1ad..7c9ada8273815 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3325,6 +3325,26 @@ static inline void clear_page_pfmemalloc(struct page *page)
 	page->lru.next = NULL;
 }
 
+static inline bool page_is_zeroed(const struct page *page)
+{
+	return (uintptr_t)page->lru.next & BIT(2);
+}
+
+static inline bool folio_is_zeroed(const struct folio *folio)
+{
+	return (uintptr_t)folio->lru.next & BIT(2);
+}
+
+static inline void set_page_zeroed(struct page *page)
+{
+	page->lru.next = (void *)((uintptr_t)page->lru.next | BIT(2));
+}
+
+static inline void clear_page_zeroed(struct page *page)
+{
+	page->lru.next = (void *)((uintptr_t)page->lru.next & ~BIT(2));
+}
+
 /*
  * Can be called by the pagefault handler when it gets a VM_FAULT_OOM.
  */
@@ -5544,8 +5564,9 @@ enum mf_action_page_type {
 	MF_MSG_UNKNOWN,
 };
 
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 void folio_zero_user(struct folio *folio, unsigned long addr_hint);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 int copy_user_large_folio(struct folio *dst, struct folio *src,
 			  unsigned long addr_hint,
 			  struct vm_area_struct *vma);
@@ -5693,11 +5714,12 @@ int arch_lock_shadow_stack_status(struct task_struct *t, unsigned long status);
 
 /* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is
  * OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for
- * the head page of compound page and bit 1 for pfmemalloc page, as well as the
- * bits used for the DMA index. page_is_pfmemalloc() is checked in
- * __page_pool_put_page() to avoid recycling the pfmemalloc page.
+ * the head page of compound page, bit 1 for pfmemalloc page, and bit 2 for
+ * zeroed page, as well as the bits used for the DMA index.
+ * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid
+ * recycling the pfmemalloc page.
  */
-#define PP_MAGIC_MASK ~(PP_DMA_INDEX_MASK | 0x3UL)
+#define PP_MAGIC_MASK ~(PP_DMA_INDEX_MASK | 0x7UL)
 
 #ifdef CONFIG_PAGE_POOL
 static inline bool page_pool_page_is_pp(const struct page *page)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b0ddc652e76cc..39cd5911a2188 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -153,6 +153,9 @@ enum pageflags {
 	 */
 	PG_fscache = PG_private_2,	/* page backed by cache */
 
+	/* Page contents are known to be zero (host-zeroed or balloon) */
+	PG_zeroed = PG_private_2,
+
 	/* XEN */
 	/* Pinned in Xen as a read-only pagetable page. */
 	PG_pinned = PG_owner_priv_1,
@@ -686,6 +689,16 @@ FOLIO_FLAG_FALSE(idle)
  */
 __PAGEFLAG(Reported, reported, PF_NO_COMPOUND)
 
+/*
+ * PageZeroed() tracks pages whose contents are known to be zero.
+ * Set on free-list pages by the balloon driver or page reporting.
+ * The allocator uses this to skip redundant zeroing.
+ */
+__PAGEFLAG(Zeroed, zeroed, PF_HEAD)
+SETPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+CLEARPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+#define __PG_ZEROED (1UL << PG_zeroed)
+
 #ifdef CONFIG_MEMORY_HOTPLUG
 PAGEFLAG(VmemmapSelfHosted, vmemmap_self_hosted, PF_ANY)
 #else
@@ -1189,7 +1202,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page)
  * alloc-free cycle to prevent from reusing the page.
  */
 #define PAGE_FLAGS_CHECK_AT_PREP	\
-	((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK)
+	((PAGEFLAGS_MASK & ~(__PG_HWPOISON | __PG_ZEROED)) | LRU_GEN_MASK | LRU_REFS_MASK)
 
 /*
  * Flags stored in the second page of a compound page.  They may overlap
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6624293e4d787..927a288eb419b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -986,10 +986,21 @@ static inline void __free_one_page(struct page *page,
 			change_pageblock_range(buddy, order, migratetype);
 		}
 
-		combined_pfn = buddy_pfn & pfn;
-		page = page + (combined_pfn - pfn);
-		pfn = combined_pfn;
-		order++;
+		{
+			bool page_zeroed = PageZeroed(page);
+			bool buddy_zeroed = PageZeroed(buddy);
+
+			__ClearPageZeroed(page);
+			__ClearPageZeroed(buddy);
+
+			combined_pfn = buddy_pfn & pfn;
+			page = page + (combined_pfn - pfn);
+			pfn = combined_pfn;
+			order++;
+
+			if (page_zeroed && buddy_zeroed)
+				__SetPageZeroed(page);
+		}
 	}
 
 done_merging:
@@ -1378,7 +1389,7 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 					continue;
 				}
 			}
-			tail_page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+			tail_page->flags.f &= ~(PAGE_FLAGS_CHECK_AT_PREP | __PG_ZEROED);
 		}
 	}
 	if (folio_test_anon(folio)) {
@@ -1397,7 +1408,8 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 	}
 
 	page_cpupid_reset_last(page);
-	page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+	/* Clear the aliased PG_private_2 bit when freeing allocated pages. */
+	page->flags.f &= ~(PAGE_FLAGS_CHECK_AT_PREP | __PG_ZEROED);
 	page->private = 0;
 	reset_page_owner(page, order);
 	page_table_check_free(page, order);
@@ -1507,6 +1519,7 @@ static void split_large_buddy(struct zone *zone, struct page *page,
 			      unsigned long pfn, int order, fpi_t fpi)
 {
 	unsigned long end = pfn + (1 << order);
+	bool zeroed = PageZeroed(page);
 
 	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
 	/* Caller removed page from freelist, buddy info cleared! */
@@ -1518,6 +1531,8 @@ static void split_large_buddy(struct zone *zone, struct page *page,
 	do {
 		int mt = get_pfnblock_migratetype(page, pfn);
 
+		if (zeroed)
+			__SetPageZeroed(page);
 		__free_one_page(page, pfn, zone, order, mt, fpi);
 		pfn += 1 << order;
 		if (pfn == end)
@@ -1705,6 +1720,7 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,
 {
 	unsigned int size = 1 << high;
 	unsigned int nr_added = 0;
+	bool zeroed = PageZeroed(page);
 
 	while (high > low) {
 		high--;
@@ -1722,6 +1738,8 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,
 
 		__add_to_free_list(&page[size], zone, high, migratetype, false);
 		set_buddy_order(&page[size], high);
+		if (zeroed)
+			__SetPageZeroed(&page[size]);
 		nr_added += size;
 	}
 
@@ -1880,6 +1898,13 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
 		set_page_pfmemalloc(page);
 	else
 		clear_page_pfmemalloc(page);
+
+	if (PageZeroed(page)) {
+		__ClearPageZeroed(page);
+		set_page_zeroed(page);
+	} else {
+		clear_page_zeroed(page);
+	}
 }
 
 /*
@@ -3144,6 +3169,7 @@ int __isolate_free_page(struct page *page, unsigned int order)
 	}
 
 	del_page_from_free_list(page, zone, order, mt);
+	__ClearPageZeroed(page);
 
 	/*
 	 * Set the pageblock if the isolated page is at least half of a
-- 
MST


  reply	other threads:[~2026-11-19 12:02 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 12:02 [PATCH v11 0/7] mm/virtio: skip redundant zeroing of host-zeroed reported pages Michael S. Tsirkin
2026-11-19 12:02 ` Michael S. Tsirkin [this message]
2026-11-19 12:02 ` [PATCH v11 2/7] mm: page_reporting: support " Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 3/7] mm: skip zeroing in vma_alloc_anon_folio_pmd for pre-zeroed pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 4/7] mm: skip zeroing of pre-zeroed order-0 user pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 5/7] mm: skip zeroing in alloc_anon_folio for pre-zeroed pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 6/7] virtio_balloon: add DEVICE_INIT_REPORTED feature Michael S. Tsirkin
2026-11-19 12:03 ` [PATCH v11 7/7] mm: page_reporting: add flush parameter with page budget Michael S. Tsirkin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a98c8a49744d30a5e603228d9cb17509261add90.1795089715.git.mst@redhat.com \
    --to=mst@redhat.com \
    --cc=aarcange@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=brendan.jackman@linux.dev \
    --cc=david@kernel.org \
    --cc=dev.jain@arm.com \
    --cc=geert@linux-m68k.org \
    --cc=gerg@linux-m68k.org \
    --cc=gourry@gourry.net \
    --cc=hannes@cmpxchg.org \
    --cc=jackmanb@google.com \
    --cc=jasowang@redhat.com \
    --cc=lance.yang@linux.dev \
    --cc=liam@infradead.org \
    --cc=linmag7@gmail.com \
    --cc=linmiaohe@huawei.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=ljs@kernel.org \
    --cc=mhocko@suse.com \
    --cc=rppt@kernel.org \
    --cc=surenb@google.com \
    --cc=vbabka@kernel.org \
    --cc=virtualization@lists.linux.dev \
    --cc=willy@infradead.org \
    --cc=ying.huang@linux.alibaba.com \
    --cc=ziy@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®