From: Muchun Song <songmuchun@bytedance.com>
To: Alexander Potapenko <glider@google.com>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Marco Elver <elver@google.com>,
Dmitry Vyukov <dvyukov@google.com>,
Chris Down <chris@chrisdown.name>,
David Hildenbrand <david@kernel.org>,
"Liam R . Howlett" <liam@infradead.org>,
Vlastimil Babka <vbabka@kernel.org>,
Mike Rapoport <rppt@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Muchun Song <muchun.song@linux.dev>,
Joerg Roedel <joerg.roedel@amd.com>,
Lu Baolu <baolu.lu@linux.intel.com>,
Pranjal Shrivastava <praan@google.com>,
Ketan <ketan.kishore@oss.qualcomm.com>,
Mostafa Saleh <smostafa@google.com>,
Sang-Heon Jeon <ekffu200098@gmail.com>,
Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
Sashiko <sashiko-bot@kernel.org>,
kasan-dev@googlegroups.com, workflows@vger.kernel.org,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, Muchun Song <songmuchun@bytedance.com>
Subject: [PATCH v2 7/7] kmsan: fix metadata loss during HugeTLB struct page reinitialization
Date: Tue, 17 Nov 2026 20:06:27 +0800 [thread overview]
Message-ID: <20261117120627.50601-8-songmuchun@bytedance.com> (raw)
In-Reply-To: <20261117120627.50601-1-songmuchun@bytedance.com>
KMSAN stores each page's shadow and origin pointers in struct page.
Boot-time gigantic HugeTLB pages are reinitialized after
kmsan_init_shadow() installs these pointers, dropping the mappings and
leaving the allocated metadata pages unreachable.
Move the pointers to page_ext, whose boot-time storage is available
before kmsan_init_shadow() and survives struct page reinitialization.
Use the inline unchecked lookup under notrace preemption so metadata
lookup cannot recurse through instrumented hooks.
This shrinks struct page from 80 to 64 bytes on x86-64 KMSAN builds,
making its size a power of two and allowing HugeTLB Vmemmap Optimization
with KMSAN.
Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-ac161ba4-30cc-4bd3-b93c-ab7af8a6f64f
---
v2:
- Update __mm_zero_struct_page() for the smaller struct page.
- Keep the unchecked lookup inline in KMSAN and leave page_ext.c
instrumented (suggested by Chris Down)
- Document page_ext metadata storage and KMSAN HVO support (suggested
by Chris Down)
- Use the common boot-time page_ext allocation instead of need_early
---
Documentation/dev-tools/kmsan.rst | 8 +++---
include/linux/kmsan.h | 2 ++
include/linux/mm.h | 12 +++------
include/linux/mm_types.h | 12 ---------
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 45 ++++++++++++++++++++++++++-----
mm/page_ext.c | 4 +++
7 files changed, 52 insertions(+), 32 deletions(-)
diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst
index 0dc668b183f6..5330d21d141d 100644
--- a/Documentation/dev-tools/kmsan.rst
+++ b/Documentation/dev-tools/kmsan.rst
@@ -376,13 +376,11 @@ Metadata allocation
There are several places in the kernel for which the metadata is stored.
-1. Each ``struct page`` instance contains two pointers to its shadow and
-origin pages::
+1. Each page frame has a corresponding ``struct page_metadata`` entry containing
+two pointers to its shadow and origin pages::
- struct page {
- ...
+ struct page_metadata {
struct page *shadow, *origin;
- ...
};
At boot-time, the kernel allocates shadow and origin pages for every available
diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..08fb79566235 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -15,12 +15,14 @@
#include <linux/types.h>
struct page;
+struct page_ext_operations;
struct kmem_cache;
struct task_struct;
struct scatterlist;
struct urb;
#ifdef CONFIG_KMSAN
+extern struct page_ext_operations kmsan_ops;
/**
* kmsan_task_create() - Initialize KMSAN state for the task.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 77b5d450fe45..9349209cd2c0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -133,7 +133,7 @@ extern int mmap_rnd_compat_bits __read_mostly;
* define their own version of this macro in <asm/pgtable.h>
*/
#if BITS_PER_LONG == 64
-/* This function must be updated when the size of struct page grows above 96
+/* This function must be updated when the size of struct page grows above 80
* or reduces below 56. The idea that compiler optimizes out switch()
* statement, and only leaves move/store instructions. Also the compiler can
* combine write statements if they are both assignments and can be reordered,
@@ -144,18 +144,12 @@ static inline void __mm_zero_struct_page(struct page *page)
{
unsigned long *_pp = (void *)page;
- /* Check that struct page is either 56, 64, 72, 80, 88 or 96 bytes */
+ /* Check that struct page is either 56, 64, 72 or 80 bytes */
BUILD_BUG_ON(sizeof(struct page) & 7);
BUILD_BUG_ON(sizeof(struct page) < 56);
- BUILD_BUG_ON(sizeof(struct page) > 96);
+ BUILD_BUG_ON(sizeof(struct page) > 80);
switch (sizeof(struct page)) {
- case 96:
- _pp[11] = 0;
- fallthrough;
- case 88:
- _pp[10] = 0;
- fallthrough;
case 80:
_pp[9] = 0;
fallthrough;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..dc73f9f5a51b 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -208,18 +208,6 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-
-#ifdef CONFIG_KMSAN
- /*
- * KMSAN metadata for this page:
- * - shadow page: every bit indicates whether the corresponding
- * bit of the original page is initialized (0) or not (1);
- * - origin page: every 4 bytes contain an id of the stack trace
- * where the uninitialized value was created.
- */
- struct page *kmsan_shadow;
- struct page *kmsan_origin;
-#endif
} _struct_page_alignment;
/*
diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan
index cae1ddcc18e1..015e85c70793 100644
--- a/lib/Kconfig.kmsan
+++ b/lib/Kconfig.kmsan
@@ -10,6 +10,7 @@ config KMSAN
depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
depends on DEBUG_KERNEL && !KASAN && !KCSAN
depends on !PREEMPT_RT
+ select PAGE_EXTENSION
select STACKDEPOT
select STACKDEPOT_ALWAYS_INIT
help
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 300b48818af1..fa2a403d87c2 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -13,6 +13,7 @@
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
+#include <linux/page_ext.h>
#include <linux/slab.h>
#include <linux/smp.h>
#include <linux/stddef.h>
@@ -24,23 +25,55 @@
#define KMSAN_META_MASK_SHADOW BIT(0)
#define KMSAN_META_MASK_ORIGIN BIT(1)
+/*
+ * KMSAN metadata for this page:
+ * - shadow page: every bit indicates whether the corresponding
+ * bit of the original page is initialized (0) or not (1);
+ * - origin page: every 4 bytes contain an id of the stack trace
+ * where the uninitialized value was created.
+ */
struct page_metadata {
struct page *shadow;
struct page *origin;
};
+static __init bool need_page_metadata(void)
+{
+ return true;
+}
+
+struct page_ext_operations kmsan_ops = {
+ .size = sizeof(struct page_metadata),
+ .need = need_page_metadata,
+};
+
static inline struct page_metadata get_page_metadata(struct page *page)
{
- return (struct page_metadata) {
- .shadow = page->kmsan_shadow,
- .origin = page->kmsan_origin,
- };
+ struct page_metadata meta = {};
+ struct page_ext *page_ext;
+
+ /*
+ * Avoid instrumented preemption debug and tracing hooks while resolving
+ * KMSAN metadata, as they would recurse back into metadata lookup.
+ */
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
+ preempt_enable_no_resched_notrace();
+
+ return meta;
}
static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- page->kmsan_shadow = meta.shadow;
- page->kmsan_origin = meta.origin;
+ struct page_ext *page_ext;
+
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta;
+ preempt_enable_no_resched_notrace();
}
static inline void *shadow_ptr_for(struct page_metadata meta)
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 0f31e2da2611..274c3e778be6 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kmsan.h>
#include <linux/slab.h>
/*
@@ -87,6 +88,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
#ifdef CONFIG_PAGE_TABLE_CHECK
&page_table_check_ops,
#endif
+#ifdef CONFIG_KMSAN
+ &kmsan_ops,
+#endif
#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
&page_iommu_debug_ops,
#endif
--
2.54.0
next prev parent reply other threads:[~2026-11-17 12:07 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 12:06 [PATCH v2 0/7] KMSAN: Move per-page metadata out of struct page Muchun Song
2026-11-17 12:06 ` [PATCH v2 1/7] kmsan: encapsulate per-page metadata access Muchun Song
2026-11-17 12:06 ` [PATCH v2 2/7] mm/page_ext: fix client data alignment Muchun Song
2026-11-17 12:06 ` [PATCH v2 3/7] mm/page_ext: always use vmalloc for section page extensions Muchun Song
2026-11-17 12:06 ` [PATCH v2 4/7] mm/page_ext: use common allocation and initialization entry points Muchun Song
2026-11-17 12:06 ` [PATCH v2 5/7] mm/page_ext: allocate page extensions early for SPARSEMEM Muchun Song
2026-11-18 0:06 ` Chris Down
2026-11-17 12:06 ` [PATCH v2 6/7] mm/page_ext: inline unchecked page extension lookup Muchun Song
2026-11-17 12:06 ` Muchun Song [this message]
2026-11-18 0:06 ` [PATCH v2 0/7] KMSAN: Move per-page metadata out of struct page Chris Down
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117120627.50601-8-songmuchun@bytedance.com \
--to=songmuchun@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=baolu.lu@linux.intel.com \
--cc=brendan.jackman@linux.dev \
--cc=chris@chrisdown.name \
--cc=corbet@lwn.net \
--cc=david@kernel.org \
--cc=dvyukov@google.com \
--cc=ekffu200098@gmail.com \
--cc=elver@google.com \
--cc=glider@google.com \
--cc=hannes@cmpxchg.org \
--cc=joerg.roedel@amd.com \
--cc=kasan-dev@googlegroups.com \
--cc=ketan.kishore@oss.qualcomm.com \
--cc=liam@infradead.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=muchun.song@linux.dev \
--cc=praan@google.com \
--cc=rdunlap@infradead.org \
--cc=rppt@kernel.org \
--cc=sashiko-bot@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=smostafa@google.com \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=workflows@vger.kernel.org \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®