From: James Houghton <jthoughton@google.com>
To: Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>,
Muchun Song <muchun.song@linux.dev>,
Oscar Salvador <osalvador@suse.de>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>,
Nikos Nikoleris <nikos.nikoleris@arm.com>,
Linu Cherian <linu.cherian@arm.com>,
Mark Rutland <mark.rutland@arm.com>,
David Hildenbrand <david@kernel.org>,
Ryan Roberts <ryan.roberts@arm.com>,
Nanyong Sun <sunnanyong@huawei.com>, Yu Zhao <yuzhao@google.com>,
Frank van der Linden <fvdl@google.com>,
David Rientjes <rientjes@google.com>,
James Houghton <jthoughton@google.com>,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org
Subject: [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
Date: Sun, 22 Nov 2026 18:44:37 +0000 [thread overview]
Message-ID: <20261122184453.2668189-4-jthoughton@google.com> (raw)
In-Reply-To: <20261122184453.2668189-1-jthoughton@google.com>
Right now it is assumed that the restore routine in vmemmap_remap_free()
will not fail; that is, if we fail to fully optimize a folio, it is
assumed that restoring the folio to its pre-HVO state is guaranteed to
succeed.
Although vmemmap restoration may never fail in practice, properly handle
such failure cases now. This will leave folios in a partially HVOed
state. In a follow-up patch, failure certainly becomes possible.
In the event we end up with partially-HVOed folios, make sure to:
1. Leave the HVO page flag in place.
2. Free any unused vmemmap pages.
Always pass the vmemmap_tail page to the restore routine to avoid
leaking the non-optimized vmemmap pages for a partially HVOed page.
Signed-off-by: James Houghton <jthoughton@google.com>
---
mm/hugetlb_vmemmap.c | 34 ++++++++++++++++++++++++++++------
1 file changed, 28 insertions(+), 6 deletions(-)
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 7be08c68c2f1..2e9cf8d39c89 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -274,12 +274,22 @@ static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
{
const struct page *page = (const struct page *)addr;
const struct folio *folio = page_folio(page);
- struct page *src = pte_page(ptep_get(pte)), *dst;
+ struct page *src = pte_page(ptep_get(pte)), *dst, *tail;
if (unlikely(!vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio))))
return 0;
- VM_WARN_ON_ONCE(folio_vmemmap_shared_tail_page(folio) != src);
+ tail = folio_vmemmap_shared_tail_page(folio);
+ if (WARN_ON_ONCE(!tail))
+ return -EINVAL;
+
+ /*
+ * The folio may be only partially optimized, so restore only PTEs
+ * currently pointing at the shared tail page.
+ */
+ if (src != tail)
+ return 0;
+
dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages);
set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
@@ -325,6 +335,7 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
return vmemmap_remap_range(start, end, &walk);
}
+#define VMEMMAP_REMAP_INCOMPLETE 1
/**
* vmemmap_optimize_folio - optimize the vmemmap backing of a folio
* @folio: folio whose vmemmap pages should be optimized.
@@ -340,7 +351,8 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
* Note: The caller must hold vmemmap_remap_rwsem. If TLB flushing is deferred,
* the lock must remain held until the final TLB flush.
*
- * Return: %0 on success, negative error code otherwise.
+ * Return: %0 on success, VMEMMAP_REMAP_INCOMPLETE if the folio is incompletely
+ * optimized, negative error code otherwise.
*/
static int vmemmap_optimize_folio(const struct folio *folio,
struct list_head *vmemmap_pages, unsigned long flags)
@@ -372,7 +384,8 @@ static int vmemmap_optimize_folio(const struct folio *folio,
return ret;
end = start + walk.nr_walked * PAGE_SIZE;
- vmemmap_restore_range(start, end, vmemmap_pages, 0);
+ if (vmemmap_restore_range(start, end, vmemmap_pages, 0))
+ return VMEMMAP_REMAP_INCOMPLETE;
return ret;
}
@@ -402,7 +415,12 @@ static int vmemmap_restore_folio(const struct folio *folio, unsigned long flags)
return ret;
guard(rwsem_read)(&vmemmap_remap_rwsem);
- return vmemmap_restore_range(start, end, &vmemmap_pages, flags);
+ ret = vmemmap_restore_range(start, end, &vmemmap_pages, flags);
+
+ /* Not all pages may have been consumed */
+ free_vmemmap_page_list(&vmemmap_pages);
+
+ return ret;
}
static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
@@ -526,7 +544,11 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
*/
folio_set_hugetlb_vmemmap_optimized(folio);
ret = vmemmap_optimize_folio(folio, vmemmap_pages, flags);
- if (ret)
+ /*
+ * If ret == VMEMMAP_REMAP_INCOMPLETE, the folio might be partially
+ * HVOed. Leave the HVO page folio flag in place.
+ */
+ if (ret < 0)
folio_clear_hugetlb_vmemmap_optimized(folio);
return ret;
--
2.56.1.809.ga0b61035b5-goog
next prev parent reply other threads:[~2026-11-22 18:45 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
2026-11-22 18:44 ` James Houghton [this message]
2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
2026-11-22 18:44 ` [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization James Houghton
2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
2026-11-22 18:44 ` [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test James Houghton
2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122184453.2668189-4-jthoughton@google.com \
--to=jthoughton@google.com \
--cc=akpm@linux-foundation.org \
--cc=catalin.marinas@arm.com \
--cc=david@kernel.org \
--cc=fvdl@google.com \
--cc=linu.cherian@arm.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mark.rutland@arm.com \
--cc=muchun.song@linux.dev \
--cc=nikos.nikoleris@arm.com \
--cc=osalvador@suse.de \
--cc=rientjes@google.com \
--cc=ryan.roberts@arm.com \
--cc=sunnanyong@huawei.com \
--cc=wangkefeng.wang@huawei.com \
--cc=will@kernel.org \
--cc=yuzhao@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®