From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.216.51 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795071711; b=5T3XVNfCl4jOEBx5VqCVEixlf6DN9Vo2WRL7/vHh44Nd8lGOGO9N5DKic9Ble/3X5y8Q XlGy83CKLYNVKNs1FSNIGi5N7ODoosAkwZ27idwbHX4Uaij9Ta/WnO+3sdsBYH8zFVP2w nlHP4o/2+ThJQM/gR07eipdKNz3Oc5KCftnDzEROmb24vF2ceQaKq/wyfV5aTh3r54rYE jrXrouGnMjNaqzyHINNaPksPmuNFmAx/FxyDl/YLj5dhxLIHqLwL7mXngjQuK9hZi1OUM PY2bCJQEPLz9/PQfEnBZQKmMqUpVIP5uhdIsgeN9GDpZAjM3Ju7krO3AK6mN02kvYoA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795071711; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=Wa6FffWSifqy9byjf3TnSCSaUIR9pC+Hc3W0NorYeDA=; b=lB+QpPloCvEHa0uFr2NnlVBIDeDNLBzl8RjrV5XXsyWVMpIQU2kEwh+rEm42vGWIPFFS T40ZK3Ly13oLYS2PQwQbw2ACOMF2t8cqtwYrcaiFzqv7smYgX2VVwa+Jx/TPMOjeM1YFH 2+m4AzbDBOUTC5Gl/DLpgKuW3bkoe1+sRxxZ21Aa/d8As1FPEGFvzfH/Vxs27hBDEgcbT McwxlP33UqmDF7cEHyN4y/b7AQ7yrAtsmAuwdm/2UMVAk20HB8i0CFbrM9AEiFfi19W8K fFaYkEa7ro2sFEIReC2kikYge9wvSzfTORVViHoaVQAa7VFM/Yu2prAvQtssFl6xFDg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OWxGc8ul; arc=none smtp.remote-ip=209.85.216.51 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1974606A324 for ; Thu, 19 Nov 2026 07:01:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org B1974606A324 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OWxGc8ul Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-3adac5b29ebso1229007a91.2 for ; Wed, 18 Nov 2026 23:01:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1795071710; x=1795676510; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Wa6FffWSifqy9byjf3TnSCSaUIR9pC+Hc3W0NorYeDA=; b=OWxGc8ulAKiSoE29y2RfI3mhfCrDbDKt04aHs9njeS6LMiH8YaOX6ncJxzOtdqrCC7 pH6Wv7jy2hyhJLJ9u4NEfyMmzE/+gILHTOPrq37A4MOwyqz7ZhuaUgygMAf+KlCUzBNw /Pb57ziIKcNgY18Js14OZgu5Lbk+/ijfLrvhE0iImzsy5zJtY/1WCwNrn97koYob/d7a GlthzItKBlfAkUV0GSko4rxNTM9lC0u/yuqxPFl8vdFx2tNJU/5vByqOzSN+mP5di6eZ DcvzqKVTC8ysvcB7hDpSW26M9btMB9DNofHgMd0sZ+L9GkzcdZtXlajbz0G3FjZX/quL 4Qxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795071710; x=1795676510; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Wa6FffWSifqy9byjf3TnSCSaUIR9pC+Hc3W0NorYeDA=; b=Zhv/xOsmu2p/evBhNmfXz4G2NNMJozeQmREsm+q9TRv1nEOuPr1cVWA7ziheWL9jri np1V87h3/sYF2TdMtAUDtUZiFMpL2C7NWb+RLniCJen6XO1V+3V9BOJ2tsvENpQeTgzD XWIv4As7z/pujidtkfwGcv7prSPonv9rZMuc7NCr5N404Hx41B01Kjl4nTY00daU+qEU tYHt+cf+a+lUXGKLbU48qJCh1FkG5gIninz/+45R60/g+Kpa2bhX28zfq/++AwdDiXTn QfSdEnvcc8xmZwLWWYul67+nzaZlygwwrqKZMIIY7gsQIA0HbcR6QU0wE6QvO6/bpnNK dW7w== X-Forwarded-Encrypted: i=1; AL+hErGHYUNY13NKOeEOgki3agGfmwtSpSgQVqHQWtN4QhNsp/xaPDO1PUSWbgXJcbqEYvqgDh3CDetmGtSAKdY=@vger.kernel.org X-Gm-Message-State: APuU4Z9Bkohp+QwUJEVw5X2UjsCPI/c2NfbbHhItTP568WPRuEZGUGO/ K5ewk+Y/KdvdD2cJ4TEnjGLZXMmcUj0UlWDwTySAoi7uULDxBOEX50kLLUzLXimRUok= X-Gm-Gg: AZFdTZczBSoxXBA3nh88Lv4iAchO9ShaF4MoxqaiKMz2Z86KJ4lfKA2+dNpeToznuPP J9n3Ra6fro5P8urLaZFfGiaLg9/ITdAuBfTZq49mESnAX25qJJVd6TRMiWD1EEmqquAoVV690gi 40uYreAwfD+aE3V32LeZ0nzqOOhX5UtSMfdSG7vveDUwKI578B6tgnCEVnG9LeUp7fntpki49Uw P8qVvA0LBmb80wTqzWe2jDGZ2SIg4SShFOLyOenRQAqSPctgTzKL74bg5ocrWEh2DioHcJfSDxB qhpBTVuBEfvZvta7bMRr07yfVA1JquxG404cTPcUHpoCx/6vRO/GOWIZ+080R8jgkLfVShZd5pM jzjM+xGgJC8b2w1xKIvVeNrPcXMgNHfM9BbSSuxG7xElPtUIT/gUA0i3CeF6FxhERoKo217RBnl dEFtnnUO3WdyBRvlFk1USfTE/UJ7YKIBRjTx81BGp92FEP7O64Ppm0BS2PRxWf/U3xjLybl5KZF EghuMr1zW0iGaSVQYfx5G+p/S/H6eyEWMJFqYzfcAhFYPnTOYof35W77i2h49mil150oQ1POcnz YfoGq5UmwDhgD9F7LlXE6s22ff9jY0CanijuM+E8Hi2AMrgNKiS39XsqQ3A= X-Received: by 2002:a17:90a:ec88:b0:3b4:a63:d57a with SMTP id 98e67ed59e1d1-3b40a63da9fmr231139a91.1.1795071709630; Wed, 18 Nov 2026 23:01:49 -0800 (PST) Received: from G6L4RL2QG9 ([2408:8440:c38:5ca5:3175:932a:757b:716a]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b402e75223sm597488a91.4.2026.11.18.23.01.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Nov 2026 23:01:49 -0800 (PST) From: Muchun Song To: Alexander Potapenko , Andrew Morton Cc: Marco Elver , Dmitry Vyukov , Chris Down , David Hildenbrand , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Brendan Jackman , Zi Yan , Muchun Song , Joerg Roedel , Lu Baolu , Pranjal Shrivastava , Ketan , Mostafa Saleh , Sang-Heon Jeon , Jonathan Corbet , Shuah Khan , Randy Dunlap , Sashiko , kasan-dev@googlegroups.com, workflows@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v3 7/7] kmsan: fix metadata loss during HugeTLB struct page reinitialization Date: Thu, 19 Nov 2026 14:58:12 +0800 Message-ID: <20261119065812.89959-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261119065812.89959-1-songmuchun@bytedance.com> References: <20261119065812.89959-1-songmuchun@bytedance.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit KMSAN stores each page's shadow and origin pointers in struct page. Boot-time gigantic HugeTLB pages are reinitialized after kmsan_init_shadow() installs these pointers, dropping the mappings and leaving the allocated metadata pages unreachable. Move the pointers to page_ext, whose boot-time storage is available before kmsan_init_shadow() and survives struct page reinitialization. Use the inline unchecked lookup under notrace preemption so metadata lookup cannot recurse through instrumented hooks. This shrinks struct page from 80 to 64 bytes on x86-64 KMSAN builds, making its size a power of two and allowing HugeTLB Vmemmap Optimization with KMSAN. Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO") Signed-off-by: Muchun Song Reported-by: Sashiko Closes: https://sashiko.dev/#/bug/linux-ac161ba4-30cc-4bd3-b93c-ab7af8a6f64f Reviewed-by: Chris Down Tested-by: Chris Down --- v3: - Collect Reviewed-by and Tested-by from Chris Down v2: - Update __mm_zero_struct_page() for the smaller struct page. - Keep the unchecked lookup inline in KMSAN and leave page_ext.c instrumented (suggested by Chris Down) - Document page_ext metadata storage and KMSAN HVO support (suggested by Chris Down) - Use the common boot-time page_ext allocation instead of need_early --- Documentation/dev-tools/kmsan.rst | 8 +++--- include/linux/kmsan.h | 2 ++ include/linux/mm.h | 12 +++------ include/linux/mm_types.h | 12 --------- lib/Kconfig.kmsan | 1 + mm/kmsan/shadow.c | 45 ++++++++++++++++++++++++++----- mm/page_ext.c | 4 +++ 7 files changed, 52 insertions(+), 32 deletions(-) diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst index 0dc668b183f6..5330d21d141d 100644 --- a/Documentation/dev-tools/kmsan.rst +++ b/Documentation/dev-tools/kmsan.rst @@ -376,13 +376,11 @@ Metadata allocation There are several places in the kernel for which the metadata is stored. -1. Each ``struct page`` instance contains two pointers to its shadow and -origin pages:: +1. Each page frame has a corresponding ``struct page_metadata`` entry containing +two pointers to its shadow and origin pages:: - struct page { - ... + struct page_metadata { struct page *shadow, *origin; - ... }; At boot-time, the kernel allocates shadow and origin pages for every available diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index 7da9fd506b39..08fb79566235 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -15,12 +15,14 @@ #include struct page; +struct page_ext_operations; struct kmem_cache; struct task_struct; struct scatterlist; struct urb; #ifdef CONFIG_KMSAN +extern struct page_ext_operations kmsan_ops; /** * kmsan_task_create() - Initialize KMSAN state for the task. diff --git a/include/linux/mm.h b/include/linux/mm.h index 3578f53ea9cc..e04cd6289546 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -134,7 +134,7 @@ extern int mmap_rnd_compat_bits __read_mostly; * define their own version of this macro in */ #if BITS_PER_LONG == 64 -/* This function must be updated when the size of struct page grows above 96 +/* This function must be updated when the size of struct page grows above 80 * or reduces below 56. The idea that compiler optimizes out switch() * statement, and only leaves move/store instructions. Also the compiler can * combine write statements if they are both assignments and can be reordered, @@ -145,18 +145,12 @@ static inline void __mm_zero_struct_page(struct page *page) { unsigned long *_pp = (void *)page; - /* Check that struct page is either 56, 64, 72, 80, 88 or 96 bytes */ + /* Check that struct page is either 56, 64, 72 or 80 bytes */ BUILD_BUG_ON(sizeof(struct page) & 7); BUILD_BUG_ON(sizeof(struct page) < 56); - BUILD_BUG_ON(sizeof(struct page) > 96); + BUILD_BUG_ON(sizeof(struct page) > 80); switch (sizeof(struct page)) { - case 96: - _pp[11] = 0; - fallthrough; - case 88: - _pp[10] = 0; - fallthrough; case 80: _pp[9] = 0; fallthrough; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1ed35d258d2e..dc73f9f5a51b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -208,18 +208,6 @@ struct page { #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif - -#ifdef CONFIG_KMSAN - /* - * KMSAN metadata for this page: - * - shadow page: every bit indicates whether the corresponding - * bit of the original page is initialized (0) or not (1); - * - origin page: every 4 bytes contain an id of the stack trace - * where the uninitialized value was created. - */ - struct page *kmsan_shadow; - struct page *kmsan_origin; -#endif } _struct_page_alignment; /* diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan index cae1ddcc18e1..015e85c70793 100644 --- a/lib/Kconfig.kmsan +++ b/lib/Kconfig.kmsan @@ -10,6 +10,7 @@ config KMSAN depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT + select PAGE_EXTENSION select STACKDEPOT select STACKDEPOT_ALWAYS_INIT help diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index 300b48818af1..fa2a403d87c2 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -24,23 +25,55 @@ #define KMSAN_META_MASK_SHADOW BIT(0) #define KMSAN_META_MASK_ORIGIN BIT(1) +/* + * KMSAN metadata for this page: + * - shadow page: every bit indicates whether the corresponding + * bit of the original page is initialized (0) or not (1); + * - origin page: every 4 bytes contain an id of the stack trace + * where the uninitialized value was created. + */ struct page_metadata { struct page *shadow; struct page *origin; }; +static __init bool need_page_metadata(void) +{ + return true; +} + +struct page_ext_operations kmsan_ops = { + .size = sizeof(struct page_metadata), + .need = need_page_metadata, +}; + static inline struct page_metadata get_page_metadata(struct page *page) { - return (struct page_metadata) { - .shadow = page->kmsan_shadow, - .origin = page->kmsan_origin, - }; + struct page_metadata meta = {}; + struct page_ext *page_ext; + + /* + * Avoid instrumented preemption debug and tracing hooks while resolving + * KMSAN metadata, as they would recurse back into metadata lookup. + */ + preempt_disable_notrace(); + page_ext = __page_ext_lookup(page_to_pfn(page)); + if (page_ext) + meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops); + preempt_enable_no_resched_notrace(); + + return meta; } static inline void set_page_metadata(struct page *page, struct page_metadata meta) { - page->kmsan_shadow = meta.shadow; - page->kmsan_origin = meta.origin; + struct page_ext *page_ext; + + preempt_disable_notrace(); + page_ext = __page_ext_lookup(page_to_pfn(page)); + if (page_ext) + *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta; + preempt_enable_no_resched_notrace(); } static inline void *shadow_ptr_for(struct page_metadata meta) diff --git a/mm/page_ext.c b/mm/page_ext.c index 63af3d62ed74..985595bdf420 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -12,6 +12,7 @@ #include #include #include +#include #include /* @@ -87,6 +88,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif +#ifdef CONFIG_KMSAN + &kmsan_ops, +#endif #ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC &page_iommu_debug_ops, #endif -- 2.54.0