From: "Michael S. Tsirkin" <mst@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Vlastimil Babka <vbabka@kernel.org>,
Brendan Jackman <jackmanb@google.com>,
Michal Hocko <mhocko@suse.com>,
Suren Baghdasaryan <surenb@google.com>,
Jason Wang <jasowang@redhat.com>,
Andrea Arcangeli <aarcange@redhat.com>,
Dev Jain <dev.jain@arm.com>,
Geert Uytterhoeven <geert@linux-m68k.org>,
Greg Ungerer <gerg@linux-m68k.org>,
Gregory Price <gourry@gourry.net>,
"Huang, Ying" <ying.huang@linux.alibaba.com>,
Lance Yang <lance.yang@linux.dev>,
Lorenzo Stoakes <ljs@kernel.org>,
Magnus Lindholm <linmag7@gmail.com>,
Matthew Wilcox <willy@infradead.org>,
Miaohe Lin <linmiaohe@huawei.com>, Zi Yan <ziy@nvidia.com>,
linux-mm@kvack.org, virtualization@lists.linux.dev,
"Liam R. Howlett" <liam@infradead.org>,
Mike Rapoport <rppt@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Brendan Jackman <brendan.jackman@linux.dev>
Subject: [PATCH v11 1/7] mm: add PG_zeroed page flag and buddy merge tracking
Date: Thu, 19 Nov 2026 07:02:32 -0500 [thread overview]
Message-ID: <a98c8a49744d30a5e603228d9cb17509261add90.1795089715.git.mst@redhat.com> (raw)
In-Reply-To: <cover.1795089715.git.mst@redhat.com>
Add PG_zeroed (aliased to PG_private_2) to track pages whose contents
are known to be zero. Propagate it through buddy merges and splits.
Upon folio allocation, PG_zeroed is cleared and its value is transferred
to BIT(2) in lru.next.
This bit, in turn, is tested by the new folio_init_user(). If clear, the
helper zero-initializes the folio. If set, it skips clearing the
contents.
Clear the aliased flag on free so filesystem state cannot be mistaken
for zeroed contents.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: LLM
---
include/linux/highmem.h | 21 +++++++++++++++++++++
include/linux/mm.h | 32 +++++++++++++++++++++++++++-----
include/linux/page-flags.h | 15 ++++++++++++++-
mm/page_alloc.c | 38 ++++++++++++++++++++++++++++++++------
4 files changed, 94 insertions(+), 12 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a4..70bbe93b8d669 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -303,6 +303,27 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,
#endif
}
+/**
+ * folio_init_user - Initialize a freshly allocated folio for userspace.
+ * @folio: The folio freshly allocated from the buddy allocator.
+ * @addr_hint: The address accessed by the user or the base address.
+ *
+ * The caller must not have written to the folio or repurposed folio->lru
+ * since allocation: the pre-zeroed marker is only valid until then.
+ * Preserve known-zero contents, but initialize KMSAN metadata that the
+ * allocator may have poisoned. Otherwise, clear the folio if required.
+ */
+static inline void folio_init_user(struct folio *folio, unsigned long addr_hint)
+{
+ if (folio_is_zeroed(folio)) {
+ kmsan_unpoison_memory(folio_address(folio), folio_size(folio));
+ return;
+ }
+
+ if (user_alloc_needs_zeroing())
+ folio_zero_user(folio, addr_hint);
+}
+
#ifndef vma_alloc_zeroed_movable_folio
/**
* vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6af3b7008f1ad..7c9ada8273815 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3325,6 +3325,26 @@ static inline void clear_page_pfmemalloc(struct page *page)
page->lru.next = NULL;
}
+static inline bool page_is_zeroed(const struct page *page)
+{
+ return (uintptr_t)page->lru.next & BIT(2);
+}
+
+static inline bool folio_is_zeroed(const struct folio *folio)
+{
+ return (uintptr_t)folio->lru.next & BIT(2);
+}
+
+static inline void set_page_zeroed(struct page *page)
+{
+ page->lru.next = (void *)((uintptr_t)page->lru.next | BIT(2));
+}
+
+static inline void clear_page_zeroed(struct page *page)
+{
+ page->lru.next = (void *)((uintptr_t)page->lru.next & ~BIT(2));
+}
+
/*
* Can be called by the pagefault handler when it gets a VM_FAULT_OOM.
*/
@@ -5544,8 +5564,9 @@ enum mf_action_page_type {
MF_MSG_UNKNOWN,
};
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
void folio_zero_user(struct folio *folio, unsigned long addr_hint);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
int copy_user_large_folio(struct folio *dst, struct folio *src,
unsigned long addr_hint,
struct vm_area_struct *vma);
@@ -5693,11 +5714,12 @@ int arch_lock_shadow_stack_status(struct task_struct *t, unsigned long status);
/* Mask used for checking in page_pool_page_is_pp() below. page->pp_magic is
* OR'ed with PP_SIGNATURE after the allocation in order to preserve bit 0 for
- * the head page of compound page and bit 1 for pfmemalloc page, as well as the
- * bits used for the DMA index. page_is_pfmemalloc() is checked in
- * __page_pool_put_page() to avoid recycling the pfmemalloc page.
+ * the head page of compound page, bit 1 for pfmemalloc page, and bit 2 for
+ * zeroed page, as well as the bits used for the DMA index.
+ * page_is_pfmemalloc() is checked in __page_pool_put_page() to avoid
+ * recycling the pfmemalloc page.
*/
-#define PP_MAGIC_MASK ~(PP_DMA_INDEX_MASK | 0x3UL)
+#define PP_MAGIC_MASK ~(PP_DMA_INDEX_MASK | 0x7UL)
#ifdef CONFIG_PAGE_POOL
static inline bool page_pool_page_is_pp(const struct page *page)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b0ddc652e76cc..39cd5911a2188 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -153,6 +153,9 @@ enum pageflags {
*/
PG_fscache = PG_private_2, /* page backed by cache */
+ /* Page contents are known to be zero (host-zeroed or balloon) */
+ PG_zeroed = PG_private_2,
+
/* XEN */
/* Pinned in Xen as a read-only pagetable page. */
PG_pinned = PG_owner_priv_1,
@@ -686,6 +689,16 @@ FOLIO_FLAG_FALSE(idle)
*/
__PAGEFLAG(Reported, reported, PF_NO_COMPOUND)
+/*
+ * PageZeroed() tracks pages whose contents are known to be zero.
+ * Set on free-list pages by the balloon driver or page reporting.
+ * The allocator uses this to skip redundant zeroing.
+ */
+__PAGEFLAG(Zeroed, zeroed, PF_HEAD)
+SETPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+CLEARPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+#define __PG_ZEROED (1UL << PG_zeroed)
+
#ifdef CONFIG_MEMORY_HOTPLUG
PAGEFLAG(VmemmapSelfHosted, vmemmap_self_hosted, PF_ANY)
#else
@@ -1189,7 +1202,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page)
* alloc-free cycle to prevent from reusing the page.
*/
#define PAGE_FLAGS_CHECK_AT_PREP \
- ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK)
+ ((PAGEFLAGS_MASK & ~(__PG_HWPOISON | __PG_ZEROED)) | LRU_GEN_MASK | LRU_REFS_MASK)
/*
* Flags stored in the second page of a compound page. They may overlap
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6624293e4d787..927a288eb419b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -986,10 +986,21 @@ static inline void __free_one_page(struct page *page,
change_pageblock_range(buddy, order, migratetype);
}
- combined_pfn = buddy_pfn & pfn;
- page = page + (combined_pfn - pfn);
- pfn = combined_pfn;
- order++;
+ {
+ bool page_zeroed = PageZeroed(page);
+ bool buddy_zeroed = PageZeroed(buddy);
+
+ __ClearPageZeroed(page);
+ __ClearPageZeroed(buddy);
+
+ combined_pfn = buddy_pfn & pfn;
+ page = page + (combined_pfn - pfn);
+ pfn = combined_pfn;
+ order++;
+
+ if (page_zeroed && buddy_zeroed)
+ __SetPageZeroed(page);
+ }
}
done_merging:
@@ -1378,7 +1389,7 @@ static __always_inline bool __free_pages_prepare(struct page *page,
continue;
}
}
- tail_page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ tail_page->flags.f &= ~(PAGE_FLAGS_CHECK_AT_PREP | __PG_ZEROED);
}
}
if (folio_test_anon(folio)) {
@@ -1397,7 +1408,8 @@ static __always_inline bool __free_pages_prepare(struct page *page,
}
page_cpupid_reset_last(page);
- page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ /* Clear the aliased PG_private_2 bit when freeing allocated pages. */
+ page->flags.f &= ~(PAGE_FLAGS_CHECK_AT_PREP | __PG_ZEROED);
page->private = 0;
reset_page_owner(page, order);
page_table_check_free(page, order);
@@ -1507,6 +1519,7 @@ static void split_large_buddy(struct zone *zone, struct page *page,
unsigned long pfn, int order, fpi_t fpi)
{
unsigned long end = pfn + (1 << order);
+ bool zeroed = PageZeroed(page);
VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
/* Caller removed page from freelist, buddy info cleared! */
@@ -1518,6 +1531,8 @@ static void split_large_buddy(struct zone *zone, struct page *page,
do {
int mt = get_pfnblock_migratetype(page, pfn);
+ if (zeroed)
+ __SetPageZeroed(page);
__free_one_page(page, pfn, zone, order, mt, fpi);
pfn += 1 << order;
if (pfn == end)
@@ -1705,6 +1720,7 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,
{
unsigned int size = 1 << high;
unsigned int nr_added = 0;
+ bool zeroed = PageZeroed(page);
while (high > low) {
high--;
@@ -1722,6 +1738,8 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,
__add_to_free_list(&page[size], zone, high, migratetype, false);
set_buddy_order(&page[size], high);
+ if (zeroed)
+ __SetPageZeroed(&page[size]);
nr_added += size;
}
@@ -1880,6 +1898,13 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
set_page_pfmemalloc(page);
else
clear_page_pfmemalloc(page);
+
+ if (PageZeroed(page)) {
+ __ClearPageZeroed(page);
+ set_page_zeroed(page);
+ } else {
+ clear_page_zeroed(page);
+ }
}
/*
@@ -3144,6 +3169,7 @@ int __isolate_free_page(struct page *page, unsigned int order)
}
del_page_from_free_list(page, zone, order, mt);
+ __ClearPageZeroed(page);
/*
* Set the pageblock if the isolated page is at least half of a
--
MST
next prev parent reply other threads:[~2026-11-19 12:02 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 12:02 [PATCH v11 0/7] mm/virtio: skip redundant zeroing of host-zeroed reported pages Michael S. Tsirkin
2026-11-19 12:02 ` Michael S. Tsirkin [this message]
2026-11-19 12:02 ` [PATCH v11 2/7] mm: page_reporting: support " Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 3/7] mm: skip zeroing in vma_alloc_anon_folio_pmd for pre-zeroed pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 4/7] mm: skip zeroing of pre-zeroed order-0 user pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 5/7] mm: skip zeroing in alloc_anon_folio for pre-zeroed pages Michael S. Tsirkin
2026-11-19 12:02 ` [PATCH v11 6/7] virtio_balloon: add DEVICE_INIT_REPORTED feature Michael S. Tsirkin
2026-11-19 12:03 ` [PATCH v11 7/7] mm: page_reporting: add flush parameter with page budget Michael S. Tsirkin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a98c8a49744d30a5e603228d9cb17509261add90.1795089715.git.mst@redhat.com \
--to=mst@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=brendan.jackman@linux.dev \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=geert@linux-m68k.org \
--cc=gerg@linux-m68k.org \
--cc=gourry@gourry.net \
--cc=hannes@cmpxchg.org \
--cc=jackmanb@google.com \
--cc=jasowang@redhat.com \
--cc=lance.yang@linux.dev \
--cc=liam@infradead.org \
--cc=linmag7@gmail.com \
--cc=linmiaohe@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=rppt@kernel.org \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=virtualization@lists.linux.dev \
--cc=willy@infradead.org \
--cc=ying.huang@linux.alibaba.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®