From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21D9E331A76 for ; Thu, 30 Jul 2026 10:56:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785408983; cv=none; b=F5EK0/WggBoaNkY2bdERga1gvohg6nMOB25Xyb8ImeqZIgLqaWVOKwy8NtWhX/kRY+snVFJPTJL55oN0a/4jEaP330sKU3Fd7SysP+J3glyO/HlLwxHwolWZStzwIsL3CDr0iQLStNMHMl45gADhG7OphddjDD/vMYif/OdSsmE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785408983; c=relaxed/simple; bh=Bdhqr5S4tEK+aTxT4d4rhVG867HIHgX+zTIRmcaT1+g=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SSADK9h70PEXpc/jlWvBVTnwfpc3R+joi6ze3GWYMN6dNwb9Fquzj6qgRGGdaO54t9XtHBdjk+wTjv34mN0F42kaY0u3x/OxmM0cKyH8vfy44cpMCo/sdqgXrXGc/lm7z2Hms5/DFsg7zloX66TNIBwD7/SHW0MhsIey/TKHjX4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=e2C+CCdO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="e2C+CCdO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9795A1F00A3E; Thu, 30 Jul 2026 10:56:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785408981; bh=xVX/ZLhSYqzqIVhWf+VtDwCnqLiC5pfYqTBKkO89Rpk=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=e2C+CCdOQzN3wwcKwjjyQyL6Op1FIWTdiYoWKMQXzZjFghnnGfXoKDTJaK/VJDdqF I5efTmk8dkWBKHE0HnCBH5YulYIMO97pQC5i6LrMeh9v4odf1lj5pCjBSeE1qimdGN IKbSnn2tA8xuIUyLK8kf8lluXw0kz5jfh4MER5RdgfL/lhb04/JHzdEE6C8JHaP/Xu MxaI/e45ka+ZEQ7315z20WqXvmiIixyagDlG0epXzg+bjBeqgUwB2JSzbAVz4Svuvf 7q8RDqTdfFsfKYqxbk01yH0DJ3KxdAFuyRQ8h72dxJjJoA9VyH7C9u3R9DeS19XRHv QGfMZqPoF3w3Q== From: "Lorenzo Stoakes (ARM)" Date: Thu, 30 Jul 2026 11:55:48 +0100 Subject: [PATCH mm-hotfixes v2 2/2] mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org> References: <20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org> In-Reply-To: <20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org> To: Andrew Morton , David Hildenbrand , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Pankaj Raghav , Hannes Reinecke , Hugh Dickins , Yang Shi , Kiryl Shutsemau Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Hengbin Zhang , "Lorenzo Stoakes (ARM)" X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=7408; i=ljs@kernel.org; h=from:subject:message-id; bh=Bdhqr5S4tEK+aTxT4d4rhVG867HIHgX+zTIRmcaT1+g=; b=owGbwMvMwCV2fu7ZrsZH9SKMp9WSGLKydXdZnTXZfcJkuVZvpo7BB8O/si/dxM4etD4xd3Hc2 TU/J27O7ihlYRDjYpAVU2R5/kV8f5BI2LzOC/5uMHNYmUCGMHBxCsBEwmUZ/qka64mvfnBO1+bi FfVZ6SsfVJoF3llzbMqlXX3ZP6dMfBLG8D+D6ypf3wrtA4vfRNjxRfs+L7i3bMrNM56bgzuMzpx KvM4GAA== X-Developer-Key: i=ljs@kernel.org; a=openpgp; fpr=E7F417BF5214569E89D04F46CF9DCD8A81E27F14 Rather than mixing the refcounted and non-refcounted CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so it is clear what happens when this configuration option is set and what happens when it is not. Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only introduce the refcount, lock and shrinker if !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown, abstract the huge zero folio allocation from refcounting. Also change a BUG_ON() to WARN_ON_ONCE() while we're at it. No functional change intended. Signed-off-by: Lorenzo Stoakes (ARM) --- mm/huge_memory.c | 160 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 94 insertions(+), 66 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f0535721652..4d2c891b1968 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -78,10 +78,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc); static bool split_underused_thp = true; +#define HUGE_ZERO_UNSET_PFN (~0UL) +struct folio *huge_zero_folio __read_mostly; +unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN; +#ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO static atomic_t huge_zero_refcount; static DEFINE_SPINLOCK(huge_zero_lock); -struct folio *huge_zero_folio __read_mostly; -unsigned long huge_zero_pfn __read_mostly = ~0UL; +static struct shrinker *huge_zero_folio_shrinker; +#endif + unsigned long huge_anon_orders_always __read_mostly; unsigned long huge_anon_orders_madvise __read_mostly; unsigned long huge_anon_orders_inherit __read_mostly; @@ -223,23 +228,58 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, return orders; } -static bool get_huge_zero_folio(void) +static struct folio *alloc_huge_zero_folio(void) { struct folio *zero_folio; - /* Paired with atomic_set_release(). */ - if (likely(atomic_inc_not_zero(&huge_zero_refcount))) - return true; - zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) & ~__GFP_MOVABLE, HPAGE_PMD_ORDER); if (!zero_folio) { count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED); - return false; + return NULL; + } + folio_clear_large_rmappable(zero_folio); /* Explicitly not rmappable. */ + return zero_folio; +} + +#ifdef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO +static int __init huge_zero_init(void) +{ + huge_zero_folio = alloc_huge_zero_folio(); + if (!huge_zero_folio) { + pr_warn("Allocating persistent huge zero folio failed\n"); + } else { + huge_zero_pfn = folio_pfn(huge_zero_folio); + count_vm_event(THP_ZERO_PAGE_ALLOC); } - /* Ensure zero folio won't have large_rmappable flag set. */ - folio_clear_large_rmappable(zero_folio); + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + return huge_zero_folio; +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ +} +#else +static bool get_huge_zero_folio(void) +{ + struct folio *zero_folio; + + /* Paired with atomic_set_release(). */ + if (likely(atomic_inc_not_zero(&huge_zero_refcount))) + return true; + + zero_folio = alloc_huge_zero_folio(); + if (unlikely(!zero_folio)) + return false; /* Paired with critical section in shrink_huge_zero_folio_scan(). */ spin_lock(&huge_zero_lock); @@ -266,33 +306,7 @@ static void put_huge_zero_folio(void) * Counter should never go to zero here. Only shrinker can put * last reference. */ - BUG_ON(atomic_dec_and_test(&huge_zero_refcount)); -} - -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return huge_zero_folio; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - return READ_ONCE(huge_zero_folio); - - if (!get_huge_zero_folio()) - return NULL; - - if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); - - return READ_ONCE(huge_zero_folio); -} - -void mm_put_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); + WARN_ON_ONCE(atomic_dec_and_test(&huge_zero_refcount)); } static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, @@ -316,14 +330,53 @@ static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, zero_folio = huge_zero_folio; VM_WARN_ON_ONCE(!zero_folio); WRITE_ONCE(huge_zero_folio, NULL); - WRITE_ONCE(huge_zero_pfn, ~0UL); + WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN); } folio_put(zero_folio); return HPAGE_PMD_NR; } -static struct shrinker *huge_zero_folio_shrinker; +static int __init huge_zero_init(void) +{ + huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); + if (!huge_zero_folio_shrinker) { + shrinker_free(deferred_split_shrinker); + list_lru_destroy(&deferred_split_lru); + return -ENOMEM; + } + + huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; + huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; + shrinker_register(huge_zero_folio_shrinker); + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ + shrinker_free(huge_zero_folio_shrinker); +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + return READ_ONCE(huge_zero_folio); + + if (!get_huge_zero_folio()) + return NULL; + + if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); + + return READ_ONCE(huge_zero_folio); +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); +} +#endif /* CONFIG_PERSISTENT_HUGE_ZERO_FOLIO */ #ifdef CONFIG_SYSFS static ssize_t enabled_show(struct kobject *kobj, @@ -987,39 +1040,14 @@ static int __init thp_shrinker_init(void) deferred_split_shrinker->scan_objects = deferred_split_scan; shrinker_register(deferred_split_shrinker); - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) { - /* - * Bump the reference of the huge_zero_folio and do not - * initialize the shrinker. - * - * huge_zero_folio will always be NULL on failure. We assume - * that get_huge_zero_folio() will most likely not fail as - * thp_shrinker_init() is invoked early on during boot. - */ - if (!get_huge_zero_folio()) - pr_warn("Allocating persistent huge zero folio failed\n"); - return 0; - } - - huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); - if (!huge_zero_folio_shrinker) { - shrinker_free(deferred_split_shrinker); - list_lru_destroy(&deferred_split_lru); - return -ENOMEM; - } - - huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; - huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; - shrinker_register(huge_zero_folio_shrinker); - - return 0; + return huge_zero_init(); } static void __init thp_shrinker_exit(void) { - shrinker_free(huge_zero_folio_shrinker); shrinker_free(deferred_split_shrinker); list_lru_destroy(&deferred_split_lru); + huge_zero_shrinker_exit(); } static int __init hugepage_init(void) -- 2.55.0