From: Muchun Song <songmuchun@bytedance.com>
To: Alexander Potapenko <glider@google.com>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Marco Elver <elver@google.com>,
Dmitry Vyukov <dvyukov@google.com>,
Chris Down <chris@chrisdown.name>,
David Hildenbrand <david@kernel.org>,
"Liam R . Howlett" <liam@infradead.org>,
Vlastimil Babka <vbabka@kernel.org>,
Mike Rapoport <rppt@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Muchun Song <muchun.song@linux.dev>,
Joerg Roedel <joerg.roedel@amd.com>,
Lu Baolu <baolu.lu@linux.intel.com>,
Pranjal Shrivastava <praan@google.com>,
Ketan <ketan.kishore@oss.qualcomm.com>,
Mostafa Saleh <smostafa@google.com>,
Sang-Heon Jeon <ekffu200098@gmail.com>,
Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
Sashiko <sashiko-bot@kernel.org>,
kasan-dev@googlegroups.com, workflows@vger.kernel.org,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, Muchun Song <songmuchun@bytedance.com>
Subject: [PATCH v3 7/7] kmsan: fix metadata loss during HugeTLB struct page reinitialization
Date: Thu, 19 Nov 2026 14:58:12 +0800 [thread overview]
Message-ID: <20261119065812.89959-8-songmuchun@bytedance.com> (raw)
In-Reply-To: <20261119065812.89959-1-songmuchun@bytedance.com>
KMSAN stores each page's shadow and origin pointers in struct page.
Boot-time gigantic HugeTLB pages are reinitialized after
kmsan_init_shadow() installs these pointers, dropping the mappings and
leaving the allocated metadata pages unreachable.
Move the pointers to page_ext, whose boot-time storage is available
before kmsan_init_shadow() and survives struct page reinitialization.
Use the inline unchecked lookup under notrace preemption so metadata
lookup cannot recurse through instrumented hooks.
This shrinks struct page from 80 to 64 bytes on x86-64 KMSAN builds,
making its size a power of two and allowing HugeTLB Vmemmap Optimization
with KMSAN.
Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-ac161ba4-30cc-4bd3-b93c-ab7af8a6f64f
Reviewed-by: Chris Down <chris@chrisdown.name>
Tested-by: Chris Down <chris@chrisdown.name>
---
v3:
- Collect Reviewed-by and Tested-by from Chris Down
v2:
- Update __mm_zero_struct_page() for the smaller struct page.
- Keep the unchecked lookup inline in KMSAN and leave page_ext.c
instrumented (suggested by Chris Down)
- Document page_ext metadata storage and KMSAN HVO support (suggested
by Chris Down)
- Use the common boot-time page_ext allocation instead of need_early
---
Documentation/dev-tools/kmsan.rst | 8 +++---
include/linux/kmsan.h | 2 ++
include/linux/mm.h | 12 +++------
include/linux/mm_types.h | 12 ---------
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 45 ++++++++++++++++++++++++++-----
mm/page_ext.c | 4 +++
7 files changed, 52 insertions(+), 32 deletions(-)
diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst
index 0dc668b183f6..5330d21d141d 100644
--- a/Documentation/dev-tools/kmsan.rst
+++ b/Documentation/dev-tools/kmsan.rst
@@ -376,13 +376,11 @@ Metadata allocation
There are several places in the kernel for which the metadata is stored.
-1. Each ``struct page`` instance contains two pointers to its shadow and
-origin pages::
+1. Each page frame has a corresponding ``struct page_metadata`` entry containing
+two pointers to its shadow and origin pages::
- struct page {
- ...
+ struct page_metadata {
struct page *shadow, *origin;
- ...
};
At boot-time, the kernel allocates shadow and origin pages for every available
diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..08fb79566235 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -15,12 +15,14 @@
#include <linux/types.h>
struct page;
+struct page_ext_operations;
struct kmem_cache;
struct task_struct;
struct scatterlist;
struct urb;
#ifdef CONFIG_KMSAN
+extern struct page_ext_operations kmsan_ops;
/**
* kmsan_task_create() - Initialize KMSAN state for the task.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 3578f53ea9cc..e04cd6289546 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -134,7 +134,7 @@ extern int mmap_rnd_compat_bits __read_mostly;
* define their own version of this macro in <asm/pgtable.h>
*/
#if BITS_PER_LONG == 64
-/* This function must be updated when the size of struct page grows above 96
+/* This function must be updated when the size of struct page grows above 80
* or reduces below 56. The idea that compiler optimizes out switch()
* statement, and only leaves move/store instructions. Also the compiler can
* combine write statements if they are both assignments and can be reordered,
@@ -145,18 +145,12 @@ static inline void __mm_zero_struct_page(struct page *page)
{
unsigned long *_pp = (void *)page;
- /* Check that struct page is either 56, 64, 72, 80, 88 or 96 bytes */
+ /* Check that struct page is either 56, 64, 72 or 80 bytes */
BUILD_BUG_ON(sizeof(struct page) & 7);
BUILD_BUG_ON(sizeof(struct page) < 56);
- BUILD_BUG_ON(sizeof(struct page) > 96);
+ BUILD_BUG_ON(sizeof(struct page) > 80);
switch (sizeof(struct page)) {
- case 96:
- _pp[11] = 0;
- fallthrough;
- case 88:
- _pp[10] = 0;
- fallthrough;
case 80:
_pp[9] = 0;
fallthrough;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..dc73f9f5a51b 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -208,18 +208,6 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-
-#ifdef CONFIG_KMSAN
- /*
- * KMSAN metadata for this page:
- * - shadow page: every bit indicates whether the corresponding
- * bit of the original page is initialized (0) or not (1);
- * - origin page: every 4 bytes contain an id of the stack trace
- * where the uninitialized value was created.
- */
- struct page *kmsan_shadow;
- struct page *kmsan_origin;
-#endif
} _struct_page_alignment;
/*
diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan
index cae1ddcc18e1..015e85c70793 100644
--- a/lib/Kconfig.kmsan
+++ b/lib/Kconfig.kmsan
@@ -10,6 +10,7 @@ config KMSAN
depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
depends on DEBUG_KERNEL && !KASAN && !KCSAN
depends on !PREEMPT_RT
+ select PAGE_EXTENSION
select STACKDEPOT
select STACKDEPOT_ALWAYS_INIT
help
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 300b48818af1..fa2a403d87c2 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -13,6 +13,7 @@
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
+#include <linux/page_ext.h>
#include <linux/slab.h>
#include <linux/smp.h>
#include <linux/stddef.h>
@@ -24,23 +25,55 @@
#define KMSAN_META_MASK_SHADOW BIT(0)
#define KMSAN_META_MASK_ORIGIN BIT(1)
+/*
+ * KMSAN metadata for this page:
+ * - shadow page: every bit indicates whether the corresponding
+ * bit of the original page is initialized (0) or not (1);
+ * - origin page: every 4 bytes contain an id of the stack trace
+ * where the uninitialized value was created.
+ */
struct page_metadata {
struct page *shadow;
struct page *origin;
};
+static __init bool need_page_metadata(void)
+{
+ return true;
+}
+
+struct page_ext_operations kmsan_ops = {
+ .size = sizeof(struct page_metadata),
+ .need = need_page_metadata,
+};
+
static inline struct page_metadata get_page_metadata(struct page *page)
{
- return (struct page_metadata) {
- .shadow = page->kmsan_shadow,
- .origin = page->kmsan_origin,
- };
+ struct page_metadata meta = {};
+ struct page_ext *page_ext;
+
+ /*
+ * Avoid instrumented preemption debug and tracing hooks while resolving
+ * KMSAN metadata, as they would recurse back into metadata lookup.
+ */
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
+ preempt_enable_no_resched_notrace();
+
+ return meta;
}
static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- page->kmsan_shadow = meta.shadow;
- page->kmsan_origin = meta.origin;
+ struct page_ext *page_ext;
+
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta;
+ preempt_enable_no_resched_notrace();
}
static inline void *shadow_ptr_for(struct page_metadata meta)
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 63af3d62ed74..985595bdf420 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kmsan.h>
#include <linux/slab.h>
/*
@@ -87,6 +88,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
#ifdef CONFIG_PAGE_TABLE_CHECK
&page_table_check_ops,
#endif
+#ifdef CONFIG_KMSAN
+ &kmsan_ops,
+#endif
#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
&page_iommu_debug_ops,
#endif
--
2.54.0
prev parent reply other threads:[~2026-11-19 7:01 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 6:58 [PATCH v3 0/7] KMSAN: Move per-page metadata out of struct page Muchun Song
2026-11-19 6:58 ` [PATCH v3 1/7] kmsan: encapsulate per-page metadata access Muchun Song
2026-11-19 6:58 ` [PATCH v3 2/7] mm/page_ext: fix client data alignment Muchun Song
2026-11-19 8:00 ` Mike Rapoport
2026-11-19 6:58 ` [PATCH v3 3/7] mm/page_ext: always use vmalloc for section page extensions Muchun Song
2026-11-19 6:58 ` [PATCH v3 4/7] mm/page_ext: use common allocation and initialization entry points Muchun Song
2026-11-19 6:58 ` [PATCH v3 5/7] mm/page_ext: allocate page extensions early for SPARSEMEM Muchun Song
2026-11-19 6:58 ` [PATCH v3 6/7] mm/page_ext: inline unchecked page extension lookup Muchun Song
2026-11-19 7:56 ` Mike Rapoport
2026-11-19 6:58 ` Muchun Song [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119065812.89959-8-songmuchun@bytedance.com \
--to=songmuchun@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=baolu.lu@linux.intel.com \
--cc=brendan.jackman@linux.dev \
--cc=chris@chrisdown.name \
--cc=corbet@lwn.net \
--cc=david@kernel.org \
--cc=dvyukov@google.com \
--cc=ekffu200098@gmail.com \
--cc=elver@google.com \
--cc=glider@google.com \
--cc=hannes@cmpxchg.org \
--cc=joerg.roedel@amd.com \
--cc=kasan-dev@googlegroups.com \
--cc=ketan.kishore@oss.qualcomm.com \
--cc=liam@infradead.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=muchun.song@linux.dev \
--cc=praan@google.com \
--cc=rdunlap@infradead.org \
--cc=rppt@kernel.org \
--cc=sashiko-bot@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=smostafa@google.com \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=workflows@vger.kernel.org \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®