From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B861830DD2F for ; Sun, 28 Jun 2026 21:45:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782683142; cv=none; b=CJpSTSsQcD0NiQyLDJ7wmEvTtskOuglotB3/w1rkevS/5SDyUrnk8OcUqRTubWrvcQeoe1OJOyzQgdQ2EmKFiDPtWzR/VY9aK9+TXBZe1MIpdcAaeXUnPQ6jn7NZ1J2CasJrHYYphKTEZK3pBq4WHEkG9bctdoxWQnKQ8wjN8zc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782683142; c=relaxed/simple; bh=5CwmCIIwBRF4rJtSkO3fSv7EygQhsTSRrEO6W+X+9sU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gwJLP+ZOFEZ1xh0DOd8JtGRGtA98oaMj4EFqjY6Z+vAHxlj8hbOaeobBUnsGQctnZywQSS8eCuNFKdU7JNWNCZWuQWGQF5tM8YDEInE/71JBxuLOqBJBoVJr6BzRyiv7DXZmBewC7smU4GcwClAh6KjGwiX5jcwX498atBLwDGY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=R6w8Jbyr; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=GlPsYnHb; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="R6w8Jbyr"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="GlPsYnHb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782683139; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=O2PF+qcJ3m8N9rKs3G2/wBo9RivbDKiHiu3dT3I/elw=; b=R6w8Jbyrz4A/euqa49fG8eRD+JQwgxxmKtodpDCq7eVZAnnG0zSrDvB5CJz5aoUVRT+WPJ 0JGYbi0S6bwLiy+7n6NVPwaOnzTkFFSemp0tZHx19piUSNBKSR3X7ibm/RlPGIc8AmR7ly 4OF0GOUOGlXfRxf8lNhLrL6p1Ka5pKM= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-546-4QNWTyu0Pz-lrXbsABH6zA-1; Sun, 28 Jun 2026 17:45:38 -0400 X-MC-Unique: 4QNWTyu0Pz-lrXbsABH6zA-1 X-Mimecast-MFC-AGG-ID: 4QNWTyu0Pz-lrXbsABH6zA_1782683137 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-47416fc1674so38844f8f.3 for ; Sun, 28 Jun 2026 14:45:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1782683137; x=1783287937; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=O2PF+qcJ3m8N9rKs3G2/wBo9RivbDKiHiu3dT3I/elw=; b=GlPsYnHbqZ91hvkNzpW0SAWQ+3CPENqRF8iI9jRdH4rAo4xCvtwJN3cALZWjGOFsRh ph7rPvo4JyrlAnqrFZk0mG1MWoVBaTre8drgTPQoIj+5Vt+QtYsBrWyELOmdzCruyZJz pFQLyjF41nGQxc0wzPp31cGMA9ec82H9Un4hdgSwiHAjuPlSF0+fgM6XuvJO7myideZo 3ZIsd0MXYD8NwHR8xPZSKBYX3RUeiCYT2N3XbYzGNyUii5sEczzXhcYKY3DTar9GMwbZ FqBagEwQGMzYmCqSzPANLPSxZck6hG9RSUB481XQAnOr3rC1WD98l6Nne5ThpU5KmDcX 1Jfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782683137; x=1783287937; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O2PF+qcJ3m8N9rKs3G2/wBo9RivbDKiHiu3dT3I/elw=; b=Svgk5X3wl75qEl16zkNFIIS8ZZ7kQbnWW3zYnp+UzdSzEitIm+vdBZSs8X+R1pIgai qSlkkele6zPcVCjmvYTbY8zeWJUaFFcWHZw1NM+px7A1wrbEuyTEqgafd+uWAGN+ZctL IfSHEKMwBWaDiEfOisJ4/4cBg08YzoE7snSykMqwZvl7dDThLGImW6K6LbA8oUloWbn1 K10e3lvIekptK2jYR+nvCqe7lHGAUf3v3gBboP/xTwyW/29/rs+daE2XXSA02SZpT7z7 wSrypJfKRUChtyBtoTASgEYmgLoC0cWaAlQ4wWopuUAP2Rip1QXFpRrQWwZIwE86K8h/ I01g== X-Gm-Message-State: AOJu0YzKkhrZWaKe5afumqnRkbySIWrMt9Xr296L95tBt7pdqo1C4+to L2/ZbqO3XMHv+kI562mJ66RhYgz7Rp96PxRdnuS9JNiH3hIjgwwHe9qjrD+JIK1h+UbxpjJ1dH7 /SFO1crj17pWwyJEwLYzropek2/nkRqWffg7xS2Y0DWHw7lQqua8RX/xPQzkR62lrUYGajHrmgM dh+WjbP2yuJr87qUGvN6lyowOcjA/r6Hyfb8Yw2WQ58pE= X-Gm-Gg: AfdE7cmoYwY8YHnzwvA/nLEpt511/7baxPmlMi0HpA5w/myzb4nqMhbDwJ+oY5b5AOq +2InMdeDv1hctsQJ9BCLH3QdDrJm22ml8gf/DEkddfLWaUqIFnrymAsYOtOHCmN9qtx5yT4a4I/ Xu+KIKbZ2dADfKAj5xVNV+xt9rh1k4SMolR4rhxLHcoBmVh+LAlFX4vMcAiFIk2VaIlvmMNQLSz lGnanNn2R1BLHNoobGv1FhdKz1BTPgrnuiYLS8Eyiu2sEDoZ5QCwdVVr3q7nE7LxoOyEeHsWwUu oHCnUyf3X7TNGcGQ2Xr/G7Z1NTvysCbD9UWQbhs8c3nRcxnG87Ni+XOJQXfOoCVUdtMR/xG+GWx R X-Received: by 2002:a05:6000:5c8:b0:472:8343:7f5b with SMTP id ffacd0b85a97d-472834380bcmr5815621f8f.17.1782683136941; Sun, 28 Jun 2026 14:45:36 -0700 (PDT) X-Received: by 2002:a05:6000:5c8:b0:472:8343:7f5b with SMTP id ffacd0b85a97d-472834380bcmr5815569f8f.17.1782683136320; Sun, 28 Jun 2026 14:45:36 -0700 (PDT) Received: from redhat.com ([31.187.78.70]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-473563c2df7sm5812233f8f.24.2026.06.28.14.45.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jun 2026 14:45:35 -0700 (PDT) Date: Sun, 28 Jun 2026 17:45:32 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: David Hildenbrand , Miaohe Lin , Naoya Horiguchi , Andrew Morton , Oscar Salvador , Andi Kleen , Hidehiro Kawai , Rik van Riel , Vlastimil Babka , Lorenzo Stoakes , "Liam R. Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , Hao Li , Kiryl Shutsemau , Byungchul Park , linux-mm@kvack.org, linux-cxl@vger.kernel.org, David Hildenbrand Subject: [PATCH 2/2] mm: wrap non-atomic page flag ops in RCU for HWPoison safety Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Mailer: git-send-email 2.51.2.2891.g4157995a80.dirty X-Mutt-Fcc: =sent Wrap all non-atomic page flag operations in rcu_read_lock/rcu_read_unlock so that synchronize_rcu() in memory_failure() can drain in-flight callers. This completes the RCU-based protection introduced in the previous patch. Add hwpoison_safe_set_bit/clear_bit wrappers in page-flags.h that bracket __set_bit/__clear_bit with rcu_read_lock/rcu_read_unlock. Convert all __SETPAGEFLAG/__CLEARPAGEFLAG macros and direct flag manipulation sites (page_alloc.c, huge_memory.c, memremap.c, mm_init.c, slub.c, mm.h) to use these wrappers. For batched flag clearing paths (page_alloc.c order > 0), use hwpoison_rcu_lock/unlock around the entire loop rather than per-operation locking. When !CONFIG_MEMORY_FAILURE, all wrappers compile to the bare operations with no overhead. Suggested-by: David Hildenbrand Signed-off-by: Michael S. Tsirkin Assisted-by: Claude:claude-opus-4-6 Assisted-by: Cursor:gpt-5.4-xhigh-fast --- include/linux/mm.h | 7 ++++ include/linux/page-flags.h | 81 +++++++++++++++++++++++++++++++++++--- mm/huge_memory.c | 2 + mm/memremap.c | 6 ++- mm/mm_init.c | 2 + mm/page_alloc.c | 4 ++ mm/slub.c | 2 +- 7 files changed, 97 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 06bbe9eba636..d52a5e90cad6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2341,6 +2341,7 @@ static inline int folio_last_cpupid(struct folio *folio) int folio_xchg_last_cpupid(struct folio *folio, int cpupid); +/* Caller must hold hwpoison_rcu_lock() */ static inline void page_cpupid_reset_last(struct page *page) { page->flags.f |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT; @@ -2503,8 +2504,10 @@ static inline struct zone *folio_zone(const struct folio *folio) #ifdef SECTION_IN_PAGE_FLAGS static inline void set_page_section(struct page *page, unsigned long section) { + hwpoison_rcu_lock_flags(&page->flags.f); page->flags.f &= ~(SECTIONS_MASK << SECTIONS_PGSHIFT); page->flags.f |= (section & SECTIONS_MASK) << SECTIONS_PGSHIFT; + hwpoison_rcu_unlock_flags(&page->flags.f); } static inline unsigned long memdesc_section(memdesc_flags_t mdf) @@ -2719,14 +2722,18 @@ static inline bool folio_is_longterm_pinnable(struct folio *folio) static inline void set_page_zone(struct page *page, enum zone_type zone) { + hwpoison_rcu_lock_flags(&page->flags.f); page->flags.f &= ~(ZONES_MASK << ZONES_PGSHIFT); page->flags.f |= (zone & ZONES_MASK) << ZONES_PGSHIFT; + hwpoison_rcu_unlock_flags(&page->flags.f); } static inline void set_page_node(struct page *page, unsigned long node) { + hwpoison_rcu_lock_flags(&page->flags.f); page->flags.f &= ~(NODES_MASK << NODES_PGSHIFT); page->flags.f |= (node & NODES_MASK) << NODES_PGSHIFT; + hwpoison_rcu_unlock_flags(&page->flags.f); } static inline void set_page_links(struct page *page, enum zone_type zone, diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7223f6f4e2b4..25eb5a77c92f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -9,6 +9,7 @@ #include #include #include +#include #ifndef __GENERATING_BOUNDS_H #include #include @@ -404,6 +405,68 @@ static unsigned long *folio_flags(struct folio *folio, unsigned n) #define FOLIO_HEAD_PAGE 0 #define FOLIO_SECOND_PAGE 1 +/* + * Non-atomic page flag operations (__set_bit, __clear_bit, flags &= ~mask) + * can race with atomic TestSetPageHWPoison() in memory_failure(). + * Wrap non-atomic ops in rcu_read_lock/rcu_read_unlock so that + * synchronize_rcu() in memory_failure() drains in-flight callers. + */ +#ifdef CONFIG_MEMORY_FAILURE +static __always_inline void +hwpoison_safe_set_bit(unsigned long nr, unsigned long *addr) +{ + rcu_read_lock(); + __set_bit(nr, addr); + rcu_read_unlock(); +} + +static __always_inline void +hwpoison_safe_clear_bit(unsigned long nr, unsigned long *addr) +{ + rcu_read_lock(); + __clear_bit(nr, addr); + rcu_read_unlock(); +} + +static __always_inline void hwpoison_rcu_lock_flags(unsigned long *addr) +{ + rcu_read_lock(); +} + +static __always_inline void hwpoison_rcu_unlock_flags(unsigned long *addr) +{ + rcu_read_unlock(); +} + +static __always_inline void hwpoison_rcu_lock(void) +{ + rcu_read_lock(); +} + +static __always_inline void hwpoison_rcu_unlock(void) +{ + rcu_read_unlock(); +} + +#else /* !CONFIG_MEMORY_FAILURE */ +static inline void +hwpoison_safe_set_bit(unsigned long nr, unsigned long *addr) +{ + __set_bit(nr, addr); +} + +static inline void +hwpoison_safe_clear_bit(unsigned long nr, unsigned long *addr) +{ + __clear_bit(nr, addr); +} + +static inline void hwpoison_rcu_lock_flags(unsigned long *addr) { } +static inline void hwpoison_rcu_unlock_flags(unsigned long *addr) { } +static inline void hwpoison_rcu_lock(void) { } +static inline void hwpoison_rcu_unlock(void) { } +#endif + /* * Macros to create function definitions for page flags */ @@ -421,11 +484,11 @@ static __always_inline void folio_clear_##name(struct folio *folio) \ #define __FOLIO_SET_FLAG(name, page) \ static __always_inline void __folio_set_##name(struct folio *folio) \ -{ __set_bit(PG_##name, folio_flags(folio, page)); } +{ hwpoison_safe_set_bit(PG_##name, folio_flags(folio, page)); } #define __FOLIO_CLEAR_FLAG(name, page) \ static __always_inline void __folio_clear_##name(struct folio *folio) \ -{ __clear_bit(PG_##name, folio_flags(folio, page)); } +{ hwpoison_safe_clear_bit(PG_##name, folio_flags(folio, page)); } #define FOLIO_TEST_SET_FLAG(name, page) \ static __always_inline bool folio_test_set_##name(struct folio *folio) \ @@ -458,12 +521,12 @@ static __always_inline void ClearPage##uname(struct page *page) \ #define __SETPAGEFLAG(uname, lname, policy) \ __FOLIO_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline void __SetPage##uname(struct page *page) \ -{ __set_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ hwpoison_safe_set_bit(PG_##lname, &policy(page, 1)->flags.f); } #define __CLEARPAGEFLAG(uname, lname, policy) \ __FOLIO_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline void __ClearPage##uname(struct page *page) \ -{ __clear_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ hwpoison_safe_clear_bit(PG_##lname, &policy(page, 1)->flags.f); } #define TESTSETFLAG(uname, lname, policy) \ FOLIO_TEST_SET_FLAG(lname, FOLIO_##policy) \ @@ -806,7 +869,7 @@ static inline bool PageUptodate(const struct page *page) static __always_inline void __folio_mark_uptodate(struct folio *folio) { smp_wmb(); - __set_bit(PG_uptodate, folio_flags(folio, 0)); + hwpoison_safe_set_bit(PG_uptodate, folio_flags(folio, 0)); } static __always_inline void folio_mark_uptodate(struct folio *folio) @@ -1166,6 +1229,14 @@ static __always_inline void ClearPageAnonExclusive(struct page *page) } static __always_inline void __ClearPageAnonExclusive(struct page *page) +{ + VM_BUG_ON_PGFLAGS(!PageAnon(page), page); + VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page); + hwpoison_safe_clear_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags.f); +} + +/* Caller must hold hwpoison_rcu_lock() */ +static __always_inline void ___ClearPageAnonExclusive(struct page *page) { VM_BUG_ON_PGFLAGS(!PageAnon(page), page); VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 970e077019b7..99f600459964 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3624,6 +3624,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order, * unreferenced sub-pages of an anonymous THP: we can simply drop * PG_anon_exclusive (-> PG_mappedtodisk) for these here. */ + hwpoison_rcu_lock_flags(&new_folio->flags.f); new_folio->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP; new_folio->flags.f |= (folio->flags.f & ((1L << PG_referenced) | @@ -3643,6 +3644,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order, #endif (1L << PG_dirty) | LRU_GEN_MASK | LRU_REFS_MASK)); + hwpoison_rcu_unlock_flags(&new_folio->flags.f); if (handle_hwpoison && page_range_has_hwpoisoned(new_head, new_nr_pages)) diff --git a/mm/memremap.c b/mm/memremap.c index 053842d45cb1..5c769dc4930c 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -425,8 +425,10 @@ void free_zone_device_folio(struct folio *folio) mem_cgroup_uncharge(folio); if (folio_test_anon(folio)) { + hwpoison_rcu_lock(); for (i = 0; i < nr; i++) - __ClearPageAnonExclusive(folio_page(folio, i)); + ___ClearPageAnonExclusive(folio_page(folio, i)); + hwpoison_rcu_unlock(); } /* @@ -494,7 +496,9 @@ void zone_device_page_init(struct page *page, struct dev_pagemap *pgmap, * blindly clear bits which could have set my order field here, * including page head. */ + hwpoison_rcu_lock_flags(&new_page->flags.f); new_page->flags.f &= ~0xffUL; /* Clear possible order, page head */ + hwpoison_rcu_unlock_flags(&new_page->flags.f); #ifdef NR_PAGES_IN_LARGE_FOLIO /* diff --git a/mm/mm_init.c b/mm/mm_init.c index f9f8e1af921c..881574dd14da 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -596,7 +596,9 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, set_page_links(page, zone, nid, pfn); init_page_count(page); atomic_set(&page->_mapcount, -1); + hwpoison_rcu_lock_flags(&page->flags.f); page_cpupid_reset_last(page); + hwpoison_rcu_unlock_flags(&page->flags.f); page_kasan_tag_reset(page); INIT_LIST_HEAD(&page->lru); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d49c254174da..60a4697ecb9a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1358,6 +1358,7 @@ __always_inline bool __free_pages_prepare(struct page *page, if (unlikely(order)) { int i; + hwpoison_rcu_lock(); if (compound) { page[1].flags.f &= ~PAGE_FLAGS_SECOND; #ifdef NR_PAGES_IN_LARGE_FOLIO @@ -1375,6 +1376,7 @@ __always_inline bool __free_pages_prepare(struct page *page, } (page + i)->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP; } + hwpoison_rcu_unlock(); } if (folio_test_anon(folio)) { mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); @@ -1391,8 +1393,10 @@ __always_inline bool __free_pages_prepare(struct page *page, return false; } + hwpoison_rcu_lock_flags(&page->flags.f); page_cpupid_reset_last(page); page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP; + hwpoison_rcu_unlock_flags(&page->flags.f); page->private = 0; reset_page_owner(page, order); page_table_check_free(page, order); diff --git a/mm/slub.c b/mm/slub.c index a2bf3756ca7d..fdfb3019b562 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -617,7 +617,7 @@ static inline void slab_set_pfmemalloc(struct slab *slab) static inline void __slab_clear_pfmemalloc(struct slab *slab) { - __clear_bit(SL_pfmemalloc, &slab->flags.f); + hwpoison_safe_clear_bit(SL_pfmemalloc, &slab->flags.f); } /* -- MST