From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.215.173 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795071579; b=qojuBXmo8t2FbJSF1RPVXxpvCOY7kJvlJVA95atJmPok8OEujE7BvQUiJ7z5q3J71wO/ yJ+YblH8IWAQ8rPR8VUElsw9x/8zyLGNqHSUmgwuBwfjm1EoFntvvYQFScTRBhk05MrmF Z+V/vqQ1wMrLoSr9fNsdNbvXQ98hVvaTVRuurgJEy2tyduTfp9S/1p7+wVZUlMAP/0H+c gDlSKAZrN7lxjuCbTWvHbpwX5mLLvjd3Yx5Hx97fCWLBx3D7Lh9lz/CEb0l7QZy7Zl7w7 Pfovz8VOOvQC29nRiIOd/tHRsTc2xne9hlblJ+3KI3VXZuD/B2KnnAc/sRWF3fOEGIw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795071579; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=Ogi5GWES8zQkjfKNxeO+iNnJsM3bnylk9LsOSYEUFgo=; b=fQdswDW2bnYeeY1uiBBBujEigELt9lOZGl/IWxXZ0IBEw2LRhjutlTNTZ7evetNVHauL x0AS6dyEyfzPAAF10vHY17tyLaHhoJ7pTF8yB70n5rwTeZeI+ySpP9koxqmxEtrn3TtTJ 4QvpF3TwzqwsxyJsUajIGk/PcmaPLTCLMySXyovp0YDdCMp6pRLkdehG2WsK3NtQVW+Ru 2LHUEvWAkuASf8hzITkY2Bwr+7Vi0e8t4KtD9uYNV7LjR6gpNIhAgepH8YkEU8WgkZs2n SaxfLfL5SsLLqy6YYGo89LBw2x3DCOcYqUjJVp2xZWvcNAgna89fcIGyG90OhBIzzwg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CUHOUuAi; arc=none smtp.remote-ip=209.85.215.173 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45ABF606A33E for ; Thu, 19 Nov 2026 06:59:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 45ABF606A33E Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CUHOUuAi Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-cdba1b5ed8dso339098a12.1 for ; Wed, 18 Nov 2026 22:59:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1795071577; x=1795676377; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ogi5GWES8zQkjfKNxeO+iNnJsM3bnylk9LsOSYEUFgo=; b=CUHOUuAiPPu8G4zGY+9gfCbbxXqMcpIk3ZiJ/WOXd8JqIHjWc+ihgujztwz8sntflK Bs49ZJJFMjZ76F3hNcWA03svmaJqyQah5zLmW9p6febaxouS/M3GeD0ydc+ncC5sJgHM sgLw/wrO28GHsklOBBKhr7Z9wgV9WTID2deMfUxFa8DbegvIyblxSWIh7PfDHxepJ4hq 2UHtA85GdJ5CI8MBElLUbaGcA6UDV4Jldyc5/f91p3+yrkN6M1f3fNCnG4X6YezP71Bi GAsVZel/ZCKN3wHk1rd9kZMxm4SGGmLGfO9lAaZj2h4RF9S5deja0c41oiZKQJyNqFhJ T+gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795071577; x=1795676377; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Ogi5GWES8zQkjfKNxeO+iNnJsM3bnylk9LsOSYEUFgo=; b=ILdtk9XSx9d64ONJNez0G/lpvirhyrUxWtNiZDEuE8OaCpuktCCjaQQYO/fLgBoTjj D3BC+1bYR+kxUeOBVJWcn/3DxubponS9hjULEZJt4Ne3PoN4s3RGjAXHuDfbCoapnqew SNEdcIEqbIZZCt+E9Vx993XN5dkMn2BV5JaBZi9jQ0U9ghc89+i8GSQsDDa9yDEFQW7s kQttLF6h3pZU5jGnPiqnNsYp8PfHrCsWxS9wHXx1mrjFk7OFaPMhf3M2jKNsPJYiU1T0 OjXi2pnlvWeDULk/j2IGXXmPt5wT9Q58XbxRWrao4GMqBUwtr3kNCwNSxAGxcJ5jCz6v fuaA== X-Forwarded-Encrypted: i=1; AL+hErEoID5AA+/ogctCJZyL6JcW3qur883CVKk725919Nn6prfevG3ALMJpUuE7c0ZWvxefP8N8jy8nLsdCyJ0=@vger.kernel.org X-Gm-Message-State: APuU4Z8V/OTvrrN+IO0vMQ4rN89LkSMnejIUmqnHc1cyt+0yQBFHC6oB SXQpbP7Uc7G3jNPXICwnDZQia50XaCV1hv/JsNdlDJoyNmVTSixCcc9g6k+qxPQmusQ= X-Gm-Gg: AZFdTZdiNVOkKLI1rP5jVOSe7qtEA9JjkLU6a3TUcHz/xQnaHPOlBuOLtUIaFL7jrvn IF4LsO5bnFbkrycTEHqMhT7LwPSHpmqgs9DcYQVtyYTjeOIZidAHlq3iPR2NavDraKp3Un7gIjB YAm7iwMd7i4oV/VtWwZgCP7SZM7WsndmVGqttx6pWoT1WgdJbI1cs9LSxBT1pHOLHtetaoW76dj HCk71BWEN8/bPwo9ugYmnaAK5i/3QakRtc0KCU8+BaUZiWrTUcp05mJRjxj598s2TU/ESLzN8A/ OfcQ6b08J4uUBhzvU9LfINt88weV4N0qHbCUV0b/umwY5VRnD7RVFtK+MQY3INl/agMEpecaCjk pfJSHflkveBdtXl+CI/MUjhONucvDe5RvFkKRJkgkhZLpkFjQLrM6sUpvk1zPPBTaRrk+a3Ck0y 78CObjWltNciqzHuCyd0Emu6dhq8guwY8Q/1S6pVyRFfQyVnl7xPajMCATXOtYnHSSDj6+dwy3w vrYu00PsI1r9DZVSKbB/JXKtkraf1i37YJoJEtIIahFi929LzX+k0oxiMfMnIB1IoYv1mf00KpF mm7WBaB0JbSZrBjPrrkmPQLG+UkmlX1q1Wi4+cZfMmStI5oMPelIV+PcTqA= X-Received: by 2002:a17:90b:4b8b:b0:3a0:d7f2:5a58 with SMTP id 98e67ed59e1d1-3b409d9bfacmr158793a91.3.1795071577008; Wed, 18 Nov 2026 22:59:37 -0800 (PST) Received: from G6L4RL2QG9 ([2408:8440:c38:5ca5:3175:932a:757b:716a]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b402e75223sm597488a91.4.2026.11.18.22.59.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Nov 2026 22:59:36 -0800 (PST) From: Muchun Song To: Alexander Potapenko , Andrew Morton Cc: Marco Elver , Dmitry Vyukov , Chris Down , David Hildenbrand , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Brendan Jackman , Zi Yan , Muchun Song , Joerg Roedel , Lu Baolu , Pranjal Shrivastava , Ketan , Mostafa Saleh , Sang-Heon Jeon , Jonathan Corbet , Shuah Khan , Randy Dunlap , Sashiko , kasan-dev@googlegroups.com, workflows@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v3 1/7] kmsan: encapsulate per-page metadata access Date: Thu, 19 Nov 2026 14:58:06 +0800 Message-ID: <20261119065812.89959-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261119065812.89959-1-songmuchun@bytedance.com> References: <20261119065812.89959-1-songmuchun@bytedance.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit KMSAN directly accesses page->kmsan_shadow and page->kmsan_origin throughout the shadow implementation. This couples these users to storing the metadata pointers in struct page, requiring every access site to change when the storage moves elsewhere. Introduce helpers that get and set both metadata page pointers as a pair and convert the existing users to them. kmsan_get_shadow_origin_ptr() is the main metadata lookup path for compiler-instrumented memory accesses and requires both pointers. For page-backed addresses, resolve the backing page once and obtain the metadata pair through one lookup instead of calling the single-metadata path twice. This also avoids duplicate lookups after moving the metadata storage out of struct page. No functional change is intended. Signed-off-by: Muchun Song Reviewed-by: Chris Down Tested-by: Chris Down --- v3: - Collect Reviewed-by and Tested-by from Chris Down v2: - Use one page metadata lookup for shadow and origin pointers (suggested by Chris Down) --- mm/kmsan/shadow.c | 166 +++++++++++++++++++++++++++++++--------------- 1 file changed, 111 insertions(+), 55 deletions(-) diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index 0c88d89bf0d6..300b48818af1 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -20,29 +21,46 @@ #include "../vmalloc.h" #include "kmsan.h" -#define shadow_page_for(page) ((page)->kmsan_shadow) +#define KMSAN_META_MASK_SHADOW BIT(0) +#define KMSAN_META_MASK_ORIGIN BIT(1) -#define origin_page_for(page) ((page)->kmsan_origin) +struct page_metadata { + struct page *shadow; + struct page *origin; +}; -static void *shadow_ptr_for(struct page *page) +static inline struct page_metadata get_page_metadata(struct page *page) { - return page_address(shadow_page_for(page)); + return (struct page_metadata) { + .shadow = page->kmsan_shadow, + .origin = page->kmsan_origin, + }; } -static void *origin_ptr_for(struct page *page) +static inline void set_page_metadata(struct page *page, struct page_metadata meta) { - return page_address(origin_page_for(page)); + page->kmsan_shadow = meta.shadow; + page->kmsan_origin = meta.origin; } -static bool page_has_metadata(struct page *page) +static inline void *shadow_ptr_for(struct page_metadata meta) { - return shadow_page_for(page) && origin_page_for(page); + return page_address(meta.shadow); +} + +static inline void *origin_ptr_for(struct page_metadata meta) +{ + return page_address(meta.origin); +} + +static inline bool page_has_metadata(struct page_metadata meta) +{ + return meta.shadow && meta.origin; } static void set_no_shadow_origin_page(struct page *page) { - shadow_page_for(page) = NULL; - origin_page_for(page) = NULL; + set_page_metadata(page, (struct page_metadata) {}); } /* @@ -79,11 +97,56 @@ static struct page *virt_to_page_or_null(void *vaddr) return NULL; } +static __always_inline +struct shadow_origin_ptr __kmsan_get_metadata(void *address, unsigned int mask) +{ + struct shadow_origin_ptr ret = {}; + struct page_metadata meta; + struct page *page; + u64 addr = (u64)address; + u64 origin_addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE); + + if (kmsan_internal_is_vmalloc_addr(address) || + kmsan_internal_is_module_addr(address)) { + if (mask & KMSAN_META_MASK_SHADOW) + ret.shadow = (void *)vmalloc_meta(address, KMSAN_META_SHADOW); + if (mask & KMSAN_META_MASK_ORIGIN) + ret.origin = (void *)vmalloc_meta((void *)origin_addr, + KMSAN_META_ORIGIN); + return ret; + } + + if (mask & KMSAN_META_MASK_SHADOW) { + ret.shadow = arch_kmsan_get_meta_or_null(address, KMSAN_META_SHADOW); + if (!ret.shadow) + goto lookup_page; + } + if (mask & KMSAN_META_MASK_ORIGIN) + ret.origin = arch_kmsan_get_meta_or_null((void *)origin_addr, + KMSAN_META_ORIGIN); + if (ret.shadow || ret.origin) + return ret; + +lookup_page: + page = virt_to_page_or_null(address); + if (!page) + return ret; + meta = get_page_metadata(page); + if (!page_has_metadata(meta)) + return ret; + + if (mask & KMSAN_META_MASK_SHADOW) + ret.shadow = shadow_ptr_for(meta) + offset_in_page(addr); + if (mask & KMSAN_META_MASK_ORIGIN) + ret.origin = origin_ptr_for(meta) + offset_in_page(origin_addr); + + return ret; +} + struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size, bool store) { struct shadow_origin_ptr ret; - void *shadow; /* * Even if we redirect this memory access to the dummy page, it will @@ -95,12 +158,10 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size, goto return_dummy; KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(address, size)); - shadow = kmsan_get_metadata(address, KMSAN_META_SHADOW); - if (!shadow) + ret = __kmsan_get_metadata(address, KMSAN_META_MASK_SHADOW | + KMSAN_META_MASK_ORIGIN); + if (!ret.shadow) goto return_dummy; - - ret.shadow = shadow; - ret.origin = kmsan_get_metadata(address, KMSAN_META_ORIGIN); return ret; return_dummy: @@ -124,46 +185,36 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size, */ void *kmsan_get_metadata(void *address, bool is_origin) { - u64 addr = (u64)address, off; - struct page *page; - void *ret; - - if (is_origin) - addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE); - address = (void *)addr; - if (kmsan_internal_is_vmalloc_addr(address) || - kmsan_internal_is_module_addr(address)) - return (void *)vmalloc_meta(address, is_origin); - - ret = arch_kmsan_get_meta_or_null(address, is_origin); - if (ret) - return ret; - - page = virt_to_page_or_null(address); - if (!page) - return NULL; - if (!page_has_metadata(page)) - return NULL; - off = offset_in_page(addr); + struct shadow_origin_ptr ret; + unsigned int mask; - return (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off; + mask = is_origin ? KMSAN_META_MASK_ORIGIN : KMSAN_META_MASK_SHADOW; + ret = __kmsan_get_metadata(address, mask); + return is_origin ? ret.origin : ret.shadow; } void kmsan_copy_page_meta(struct page *dst, struct page *src) { + struct page_metadata dst_meta, src_meta; + if (!kmsan_enabled || kmsan_in_runtime()) return; - if (!dst || !page_has_metadata(dst)) + if (!dst) return; - if (!src || !page_has_metadata(src)) { + dst_meta = get_page_metadata(dst); + if (!page_has_metadata(dst_meta)) + return; + if (src) + src_meta = get_page_metadata(src); + if (!src || !page_has_metadata(src_meta)) { kmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE, /*checked*/ false); return; } kmsan_enter_runtime(); - __memcpy(shadow_ptr_for(dst), shadow_ptr_for(src), PAGE_SIZE); - __memcpy(origin_ptr_for(dst), origin_ptr_for(src), PAGE_SIZE); + __memcpy(shadow_ptr_for(dst_meta), shadow_ptr_for(src_meta), PAGE_SIZE); + __memcpy(origin_ptr_for(dst_meta), origin_ptr_for(src_meta), PAGE_SIZE); kmsan_leave_runtime(); } EXPORT_SYMBOL(kmsan_copy_page_meta); @@ -171,19 +222,18 @@ EXPORT_SYMBOL(kmsan_copy_page_meta); void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags) { bool initialized = (flags & __GFP_ZERO) || !kmsan_enabled; - struct page *shadow, *origin; + struct page_metadata meta; depot_stack_handle_t handle; int pages = 1 << order; if (!page) return; - shadow = shadow_page_for(page); - origin = origin_page_for(page); + meta = get_page_metadata(page); if (initialized) { - __memset(page_address(shadow), 0, PAGE_SIZE * pages); - __memset(page_address(origin), 0, PAGE_SIZE * pages); + __memset(shadow_ptr_for(meta), 0, PAGE_SIZE * pages); + __memset(origin_ptr_for(meta), 0, PAGE_SIZE * pages); return; } @@ -191,7 +241,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags) if (kmsan_in_runtime()) return; - __memset(page_address(shadow), -1, PAGE_SIZE * pages); + __memset(shadow_ptr_for(meta), -1, PAGE_SIZE * pages); kmsan_enter_runtime(); handle = kmsan_save_stack_with_flags(flags, /*extra_bits*/ 0); kmsan_leave_runtime(); @@ -200,7 +250,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags) * to just fill the origin pages with @handle. */ for (int i = 0; i < PAGE_SIZE * pages / sizeof(handle); i++) - ((depot_stack_handle_t *)page_address(origin))[i] = handle; + ((depot_stack_handle_t *)origin_ptr_for(meta))[i] = handle; } void kmsan_free_page(struct page *page, unsigned int order) @@ -238,8 +288,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, goto ret; } for (int i = 0; i < nr; i++) { - s_pages[i] = shadow_page_for(pages[i]); - o_pages[i] = origin_page_for(pages[i]); + struct page_metadata meta = get_page_metadata(pages[i]); + + s_pages[i] = meta.shadow; + o_pages[i] = meta.origin; } prot = PAGE_KERNEL; @@ -289,10 +341,12 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end) page = virt_to_page_or_null((char *)start + addr); shadow_p = virt_to_page((char *)shadow + addr); set_no_shadow_origin_page(shadow_p); - shadow_page_for(page) = shadow_p; origin_p = virt_to_page((char *)origin + addr); set_no_shadow_origin_page(origin_p); - origin_page_for(page) = origin_p; + set_page_metadata(page, (struct page_metadata) { + .shadow = shadow_p, + .origin = origin_p, + }); } } @@ -302,7 +356,9 @@ void kmsan_setup_meta(struct page *page, struct page *shadow, for (int i = 0; i < (1 << order); i++) { set_no_shadow_origin_page(&shadow[i]); set_no_shadow_origin_page(&origin[i]); - shadow_page_for(&page[i]) = &shadow[i]; - origin_page_for(&page[i]) = &origin[i]; + set_page_metadata(&page[i], (struct page_metadata) { + .shadow = &shadow[i], + .origin = &origin[i], + }); } } -- 2.54.0