* [PATCH v2 00/15] mm/hugetlb: various fixes
@ 2026-11-16 18:29 David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 01/15] hugetlbfs: fix integer overflow in fallocate() David Hildenbrand (Arm)
` (14 more replies)
0 siblings, 15 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
luojiajun, Josef Bacik, stable, Ackerley Tng,
Gregory Price (Meta),
Chris Mason, Daehyeon Ko
Hi,
v2 of the hugetlb fixes. There will be more at some point ... let's
drag these over the finishing line.
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
Changes in v2:
- Rebased on top of akpm/mm-stable
- Applied tags
- "mm/hugetlb: compare against persistent count when freeing lowmem folios"
and "mm/hugetlb: account temporary folios separately"
- Use const struct hstate pointers
- "mm/hugetlb: fix reservation leak on min_size= mounts"
- Clarify that this is only an accounting error in the description
- "mm/hugetlb: account restored reservations before freeing folios"
- Drop RFC
- Don't allocate surplus pages for restored reservations
- Free unused surplus pages directly instead of via free_huge_folio()
- Document available_huge_pages() and assert that hugetlb_lock is held
- "mm/hugetlb: delay anon folio rmap removal until after the TLB flush"
- Drop RFC
- Flush delayed rmaps in hugetlb_unmap_range() (due to rebasing)
- Move hugetlb_folio_restore_rsv_on_unmap() declaration to mm/internal.h
- Link to v1: https://patch.msgid.link/20261028-fixes_hugetlb-v1-0-a0a033db180f@kernel.org
---
Chris Mason (3):
hugetlbfs: do not grow a sealed file in mmap()
mm/hugetlb: keep surplus folios that are reserved
mm/hugetlb: fix reservation leak on min_size= mounts
David Hildenbrand (Arm) (10):
hugetlbfs: fix integer overflow in fallocate()
mm/hugetlb: compare against persistent count when freeing lowmem folios
mm/hugetlb: preserve reservations when shrinking on HIGHMEM
mm/hugetlb: account temporary folios separately
mm/hugetlb: fix mremap() racing with page migration
mm/hugetlb: serialize page-cache allocations with truncation
mm/memfd: recheck the page cache before allocating a hugetlb folio
mm/hugetlb: count again if the reserve map shrank
mm/hugetlb: account restored reservations before freeing folios
mm/hugetlb: delay anon folio rmap removal until after the TLB flush
Josef Bacik (2):
mm/hugetlb: keep surplus folios when the pool shrinks
hugetlbfs: fix unmapping folios from private VMAs
Documentation/mm/hugetlbfs_reserv.rst | 30 +--
arch/s390/include/asm/tlb.h | 5 +-
fs/hugetlbfs/inode.c | 36 ++-
include/asm-generic/tlb.h | 5 +-
include/linux/hugetlb.h | 17 ++
mm/hugetlb.c | 424 ++++++++++++++++++++++------------
mm/hugetlb_internal.h | 24 ++
mm/hugetlb_sysfs.c | 2 +-
mm/internal.h | 9 +
mm/memfd.c | 17 ++
mm/mmu_gather.c | 16 +-
11 files changed, 404 insertions(+), 181 deletions(-)
---
base-commit: fa4ad798f6f681031775266044ba1b0cf3c670bd
change-id: 20261027-fixes_hugetlb-2c4f02ff6304
Best regards,
--
David Hildenbrand (Arm) <david@kernel.org>
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 01/15] hugetlbfs: fix integer overflow in fallocate()
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 02/15] mm/hugetlb: compare against persistent count when freeing lowmem folios David Hildenbrand (Arm)
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
luojiajun, Josef Bacik, stable, Ackerley Tng,
Gregory Price (Meta)
hugetlbfs_fallocate() rounds the range end up by adding hpage_size - 1
to offset + len. vfs_fallocate()'s existing overflow check only makes
sure that offset + len does not overflow.
Once we overflow for valid endpoints near LLONG_MAX, we can get a
negative temporary result that is then converted to a very large
huge page index.
A one-byte allocation ending at LLONG_MAX can therefore consume all
available huge pages. Pages added beyond LLONG_MAX are skipped during
inode eviction, leaking them and triggering the BUG_ON() in clear_inode().
To fix it, calculate the end index in a simpler way that cannot
overflow. This also fixes the signed overflow reported by luojiajun in
2019.
Backporting notes: without commit a08c7193e4f18 ("mm/filemap: remove
hugetlb special casing in filemap.c"), remove_inode_hugepages() must
include the final huge page during truncate and eviction.
Reported-by: luojiajun <luojiajun3@huawei.com>
Closes: https://lore.kernel.org/all/1554775226-67213-1-git-send-email-luojiajun3@huawei.com/
Reported-by: Josef Bacik <josef@toxicpanda.com>
Fixes: 70c3547e36f5 ("hugetlbfs: add hugetlbfs_fallocate()")
Cc: stable@kernel.org
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: Oscar Salvador <osalvador@suse.de>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
fs/hugetlbfs/inode.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 02e0a65d596d8..9f502a456cdb0 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -722,11 +722,13 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
/*
* Default preallocate case.
- * For this range, start is rounded down and end is rounded up
- * as well as being converted to page offsets.
+ * vfs_fallocate() guarantees that offset + len is positive and does
+ * not overflow. Avoid overflows when calculating the end huge page
+ * index: set it to one past the index of the huge page containing the
+ * last byte.
*/
start = offset >> hpage_shift;
- end = (offset + len + hpage_size - 1) >> hpage_shift;
+ end = ((offset + len - 1) >> hpage_shift) + 1;
inode_lock(inode);
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 02/15] mm/hugetlb: compare against persistent count when freeing lowmem folios
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 01/15] hugetlbfs: fix integer overflow in fallocate() David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 03/15] mm/hugetlb: preserve reservations when shrinking on HIGHMEM David Hildenbrand (Arm)
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable
try_to_free_low() gets a target number of persistent folios, but
compares it with "nr_huge_pages", which includes surplus folios. As a
result, it can free lowmem folios below that target.
For example, with two persistent folios and one surplus folio, a target
of two can still end up freeing one persistent folio.
Compare against persistent_huge_pages() instead. Convert the macro to a
helper function and place it before try_to_free_low().
Fixes: 7893d1d505d5 ("hugetlb: Try to grow hugetlb pool for MAP_PRIVATE mappings")
Cc: stable@kernel.org
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 630c96cd1e880..349d45b932d9b 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1339,6 +1339,11 @@ static unsigned long available_huge_pages(struct hstate *h)
return h->free_huge_pages - h->resv_huge_pages;
}
+static unsigned long persistent_huge_pages(const struct hstate *h)
+{
+ return h->nr_huge_pages - h->surplus_huge_pages;
+}
+
static struct folio *dequeue_hugetlb_folio(struct hstate *h, gfp_t gfp_mask,
struct mempolicy_interpreted *mpoli)
{
@@ -3723,7 +3728,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count,
struct folio *folio, *next;
struct list_head *freel = &h->hugepage_freelists[i];
list_for_each_entry_safe(folio, next, freel, lru) {
- if (count >= h->nr_huge_pages)
+ if (count >= persistent_huge_pages(h))
goto out;
if (folio_test_highmem(folio))
continue;
@@ -3777,7 +3782,6 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed,
return 1;
}
-#define persistent_huge_pages(h) (h->nr_huge_pages - h->surplus_huge_pages)
static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
nodemask_t *nodes_allowed)
{
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 03/15] mm/hugetlb: preserve reservations when shrinking on HIGHMEM
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 01/15] hugetlbfs: fix integer overflow in fallocate() David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 02/15] mm/hugetlb: compare against persistent count when freeing lowmem folios David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 04/15] mm/hugetlb: account temporary folios separately David Hildenbrand (Arm)
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm)
try_to_free_low() drops the hugetlb_lock while freeing lowmem folios.
set_max_huge_pages() then resumes freeing folios using the
min_count computed before that unlock. Reservations created in between
can consequently lose their backing.
Collect lowmem folios on the caller's existing page_list instead. Keep
the hugetlb_lock held through both selection passes, and free the combined
list at the caller's existing unlock point.
Likely not stable-worthy.
Fixes: 1121828a0c21 ("hugetlb: call update_and_free_page without hugetlb_lock")
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 20 ++++++--------------
1 file changed, 6 insertions(+), 14 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 349d45b932d9b..0b071bef226ac 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3712,39 +3712,31 @@ static void __init report_hugepages(void)
#ifdef CONFIG_HIGHMEM
static void try_to_free_low(struct hstate *h, unsigned long count,
- nodemask_t *nodes_allowed)
+ nodemask_t *nodes_allowed, struct list_head *page_list)
{
int i;
- LIST_HEAD(page_list);
lockdep_assert_held(&hugetlb_lock);
if (hstate_is_gigantic(h))
return;
- /*
- * Collect pages to be freed on a list, and free after dropping lock
- */
+ /* Collect lowmem pages to free after dropping the lock. */
for_each_node_mask(i, *nodes_allowed) {
struct folio *folio, *next;
struct list_head *freel = &h->hugepage_freelists[i];
list_for_each_entry_safe(folio, next, freel, lru) {
if (count >= persistent_huge_pages(h))
- goto out;
+ return;
if (folio_test_highmem(folio))
continue;
remove_hugetlb_folio(h, folio, false);
- list_add(&folio->lru, &page_list);
+ list_add(&folio->lru, page_list);
}
}
-
-out:
- spin_unlock_irq(&hugetlb_lock);
- update_and_free_pages_bulk(h, &page_list);
- spin_lock_irq(&hugetlb_lock);
}
#else
static inline void try_to_free_low(struct hstate *h, unsigned long count,
- nodemask_t *nodes_allowed)
+ nodemask_t *nodes_allowed, struct list_head *page_list)
{
}
#endif
@@ -3939,7 +3931,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
}
min_count = h->resv_huge_pages + persistent_huge_pages(h) - persistent_free_count;
min_count = max(count, min_count);
- try_to_free_low(h, min_count, nodes_allowed);
+ try_to_free_low(h, min_count, nodes_allowed, &page_list);
/*
* Collect pages to be removed on list without dropping lock
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 04/15] mm/hugetlb: account temporary folios separately
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (2 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 03/15] mm/hugetlb: preserve reservations when shrinking on HIGHMEM David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 05/15] mm/hugetlb: keep surplus folios when the pool shrinks David Hildenbrand (Arm)
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable, Josef Bacik
Hugetlb migration might have to allocate a temporary folio through
alloc_migrate_hugetlb_folio(). Such folios are currently accounted in
"nr_huge_pages" but not in "surplus_huge_pages". persistent_huge_pages()
can mistake them as persistent huge pages, as it merely calculates the
difference.
If set_max_huge_pages() shrinks the pool at the wrong time, it will turn
hugetlb folios into surplus folios until persistent_huge_pages() returns
the desired number.
After the migration (successful or unsuccessful) either the new (temporary)
folio or the old folio gets freed: this will reduce "nr_huge_pages" but
not "surplus_huge_pages", to undo what was done when allocating a
temporary folio.
This can result in having more surplus folios than actually allocated
folios:
HugePages_Total: 0
HugePages_Surp: 1
From then on persistent_huge_pages() returns garbage.
To fix it, add internal temporary counters and subtract them when
calculating persistent pool sizes. Update persistent_huge_pages() and add
persistent_huge_pages_node() to make the code easier to read.
Count migration targets and rejected surplus allocations as temporary.
Transfer the per-node temporary count along with the temporary flag on
migration, such that folio freeing will always do the right thing.
Take care of calling account_new_hugetlb_folio() only after setting the
folios to be temporary.
For now, keep temporary folios in nr_huge_pages, so HugePages_Total and
Hugetlb continue to include them and overcommit accounting remains
unchanged. The new temporary counters are internal for now and are not
exported to userspace.
Fixes: ab5ac90aecf5 ("mm, hugetlb: do not rely on overcommit limit during migration")
Cc: stable@kernel.org
Reported-by: Josef Bacik <josef@toxicpanda.com>
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
include/linux/hugetlb.h | 3 +++
mm/hugetlb.c | 65 +++++++++++++++++++++++++++++--------------------
2 files changed, 42 insertions(+), 26 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 58f755e70e966..80ae65c7849b9 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -698,6 +698,8 @@ struct hstate {
unsigned long free_huge_pages;
unsigned long resv_huge_pages;
unsigned long surplus_huge_pages;
+ /* Included in nr_huge_pages; not exported separately to userspace. */
+ unsigned long temporary_huge_pages;
unsigned long nr_overcommit_huge_pages;
struct list_head hugepage_activelist;
struct list_head hugepage_freelists[MAX_NUMNODES];
@@ -705,6 +707,7 @@ struct hstate {
unsigned int nr_huge_pages_node[MAX_NUMNODES];
unsigned int free_huge_pages_node[MAX_NUMNODES];
unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+ unsigned int temporary_huge_pages_node[MAX_NUMNODES];
char name[HSTATE_NAME_LEN];
};
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 0b071bef226ac..d7c71025d166b 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -115,7 +115,7 @@ early_param(str, func##args)
/*
* Protects updates to hugepage_freelists, hugepage_activelist, nr_huge_pages,
- * free_huge_pages, and surplus_huge_pages.
+ * free_huge_pages, surplus_huge_pages, and temporary_huge_pages.
*/
__cacheline_aligned_in_smp DEFINE_SPINLOCK(hugetlb_lock);
@@ -1341,7 +1341,13 @@ static unsigned long available_huge_pages(struct hstate *h)
static unsigned long persistent_huge_pages(const struct hstate *h)
{
- return h->nr_huge_pages - h->surplus_huge_pages;
+ return h->nr_huge_pages - h->surplus_huge_pages - h->temporary_huge_pages;
+}
+
+static unsigned long persistent_huge_pages_node(const struct hstate *h, int nid)
+{
+ return h->nr_huge_pages_node[nid] - h->surplus_huge_pages_node[nid] -
+ h->temporary_huge_pages_node[nid];
}
static struct folio *dequeue_hugetlb_folio(struct hstate *h, gfp_t gfp_mask,
@@ -1426,6 +1432,10 @@ void remove_hugetlb_folio(struct hstate *h, struct folio *folio,
h->surplus_huge_pages--;
h->surplus_huge_pages_node[nid]--;
}
+ if (folio_test_hugetlb_temporary(folio)) {
+ h->temporary_huge_pages--;
+ h->temporary_huge_pages_node[nid]--;
+ }
/*
* We can only clear the hugetlb flag after allocating vmemmap
@@ -1768,9 +1778,15 @@ void free_huge_folio(struct folio *folio)
*/
static void account_new_hugetlb_folio(struct hstate *h, struct folio *folio)
{
+ const int nid = folio_nid(folio);
+
lockdep_assert_held(&hugetlb_lock);
h->nr_huge_pages++;
- h->nr_huge_pages_node[folio_nid(folio)]++;
+ h->nr_huge_pages_node[nid]++;
+ if (folio_test_hugetlb_temporary(folio)) {
+ h->temporary_huge_pages++;
+ h->temporary_huge_pages_node[nid]++;
+ }
}
void init_new_hugetlb_folio(struct folio *folio)
@@ -2117,13 +2133,6 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
return NULL;
spin_lock_irq(&hugetlb_lock);
- /*
- * nr_huge_pages needs to be adjusted within the same lock cycle
- * as surplus_pages, otherwise it might confuse
- * persistent_huge_pages() momentarily.
- */
- account_new_hugetlb_folio(h, folio);
-
/*
* We could have raced with the pool size change.
* Double check that and simply deallocate the new page
@@ -2133,11 +2142,18 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
*/
if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) {
folio_set_hugetlb_temporary(folio);
+ account_new_hugetlb_folio(h, folio);
spin_unlock_irq(&hugetlb_lock);
free_huge_folio(folio);
return NULL;
}
+ /*
+ * nr_huge_pages needs to be adjusted within the same lock cycle
+ * as surplus_pages, otherwise it might confuse
+ * persistent_huge_pages() momentarily.
+ */
+ account_new_hugetlb_folio(h, folio);
h->surplus_huge_pages++;
h->surplus_huge_pages_node[folio_nid(folio)]++;
@@ -2159,17 +2175,17 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas
if (!folio)
return NULL;
+ /*
+ * Count temporary folios separately so that pool resizing does not
+ * mistake them for persistent pages.
+ */
spin_lock_irq(&hugetlb_lock);
+ folio_set_hugetlb_temporary(folio);
account_new_hugetlb_folio(h, folio);
spin_unlock_irq(&hugetlb_lock);
/* fresh huge pages are frozen */
folio_ref_unfreeze(folio, 1);
- /*
- * We do not account these pages as surplus because they are only
- * temporary and will be released properly on the last reference
- */
- folio_set_hugetlb_temporary(folio);
return folio;
}
@@ -3761,8 +3777,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed,
}
} else {
for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
- if (h->surplus_huge_pages_node[node] <
- h->nr_huge_pages_node[node])
+ if (persistent_huge_pages_node(h, node))
goto found;
}
}
@@ -3811,9 +3826,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
if (nid != NUMA_NO_NODE) {
unsigned long old_count = count;
- count += persistent_huge_pages(h) -
- (h->nr_huge_pages_node[nid] -
- h->surplus_huge_pages_node[nid]);
+ count += persistent_huge_pages(h) - persistent_huge_pages_node(h, nid);
/*
* User may have specified a large count value which caused the
* above calculation to overflow. In this case, they wanted
@@ -7459,9 +7472,8 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio,
* be final while the old one will be freed so it takes over
* the temporary status.
*
- * Also note that we have to transfer the per-node surplus state
- * here as well otherwise the global surplus count will not match
- * the per-node's.
+ * Transfer the per-node temporary count to the old folio's node,
+ * and any surplus count in the opposite direction.
*/
if (folio_test_hugetlb_temporary(new_folio)) {
int old_nid = folio_nid(old_folio);
@@ -7470,13 +7482,14 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio,
folio_set_hugetlb_temporary(old_folio);
folio_clear_hugetlb_temporary(new_folio);
-
/*
- * There is no need to transfer the per-node surplus state
- * when we do not cross the node.
+ * There is no need to transfer the per-node state when we do
+ * not cross the node.
*/
if (new_nid != old_nid) {
spin_lock_irq(&hugetlb_lock);
+ h->temporary_huge_pages_node[new_nid]--;
+ h->temporary_huge_pages_node[old_nid]++;
if (h->surplus_huge_pages_node[old_nid]) {
h->surplus_huge_pages_node[old_nid]--;
h->surplus_huge_pages_node[new_nid]++;
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 05/15] mm/hugetlb: keep surplus folios when the pool shrinks
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (3 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 04/15] mm/hugetlb: account temporary folios separately David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 06/15] mm/hugetlb: fix mremap() racing with page migration David Hildenbrand (Arm)
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Josef Bacik, stable, Gregory Price (Meta)
From: Josef Bacik <josef@toxicpanda.com>
Surplus folios are accounted per node. When shrinking the pool,
set_max_huge_pages() removes free folios through
remove_pool_hugetlb_folio() and accounts them as persistent folios.
It can select a node whose folios are all surplus and leave its surplus
count larger than its total count. Free surplus folios can back
reservations created by gather_surplus_pages().
For example, with nr_overcommit_hugepages = 8 on two nodes:
(node columns show total/free/surplus)
total free surp node0 node1
mmap() of one page, bound to node 0 1 1 1 1 1 1 0 0 0
node1/nr_hugepages = 7 8 8 1 1 1 1 7 7 0
nr_hugepages = 6 7 7 1 0 0 1 7 7 0
munmap() 7 7 1 0 0 1 7 7 0
nr_hugepages = 0 1 1 1 0 0 1 1 1 0
After shrinking to six persistent folios, node 0 has no folios but its
surplus count is still one. Releasing the reservation cannot clear that
count, and shrinking the pool to zero leaves one free page on node 1.
A later write to node0/nr_hugepages underflows the unsigned per-node
persistent count: 0 - 1, resulting in allocation of many folios.
Use persistent_huge_pages_node() to skip nodes with no persistent folios,
as adjust_pool_surplus() does. This also excludes temporary folios.
Fixes: e4e574b767ba ("hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings")
Cc: stable@kernel.org
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
[ david: cleanup code and description ]
Acked-by: Muchun Song <muchun.song@linux.dev>
Tested-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index d7c71025d166b..3ffd87eb9e45f 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1959,17 +1959,25 @@ static struct folio *remove_pool_hugetlb_folio(struct hstate *h,
lockdep_assert_held(&hugetlb_lock);
for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
+ if (list_empty(&h->hugepage_freelists[node]))
+ continue;
+
/*
* If we're returning unused surplus pages, only examine
- * nodes with surplus pages.
+ * nodes with surplus pages. If we're shrinking the pool, only
+ * examine nodes with persistent folios.
*/
- if ((!acct_surplus || h->surplus_huge_pages_node[node]) &&
- !list_empty(&h->hugepage_freelists[node])) {
- folio = list_entry(h->hugepage_freelists[node].next,
- struct folio, lru);
- remove_hugetlb_folio(h, folio, acct_surplus);
- break;
+ if (acct_surplus) {
+ if (!h->surplus_huge_pages_node[node])
+ continue;
+ } else if (!persistent_huge_pages_node(h, node)) {
+ continue;
}
+
+ folio = list_first_entry(&h->hugepage_freelists[node],
+ struct folio, lru);
+ remove_hugetlb_folio(h, folio, acct_surplus);
+ break;
}
return folio;
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 06/15] mm/hugetlb: fix mremap() racing with page migration
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (4 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 05/15] mm/hugetlb: keep surplus folios when the pool shrinks David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 07/15] hugetlbfs: fix unmapping folios from private VMAs David Hildenbrand (Arm)
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable, Chris Mason
Migration of anon hugetlb folios walks the anon_vma, so the file's
i_mmap_rwsem does not serialize it against move_hugetlb_page_tables().
There are various issues with that. Most importantly, move_huge_pte()
releases both page table locks before the TLB is flushed. Migration can
then find the destination entry and finish migration, reusing the old
folio while the TLB still has entries cached.
Hold the anon_vma write lock across the move and the final TLB flush.
This also prevents concurrent rmap walks from missing entries moved
during rollback.
When we must allocate a destination page table, flush the TLB before
dropping the anon_vma lock, then reacquire it after allocation.
Dropping the lock only works around lockdep: hugetlb folios are not
reclaimed or migrated by allocation compaction, but their anon_vmas
share the lock class used by ordinary anonymous memory. Having a
separate lock class for hugetlb's anon_vma (like already done for
i_mmap_rwsem via hugetlbfs_i_mmap_rwsem_key) requires more thought.
There is a concern that this can block page migration of anonymous
folios quite a while ... which is not really different to what happens
to page-cache folios today. If ever a problem, we can temporarily
release the lock.
So let's keep it simple for now.
It is a shame that hugetlb has to implement everything core-mm implements
in a different way that just screams for problems.
Fixes: 550a7d60bd5e ("mm, hugepages: add mremap() support for hugepage backed vma")
Cc: stable@kernel.org
Reported-by: Chris Mason <mason@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3ffd87eb9e45f..4a24819733730 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5214,6 +5214,7 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
{
struct hstate *h = hstate_vma(vma);
struct address_space *mapping = vma->vm_file->f_mapping;
+ struct anon_vma *anon_vma = vma->anon_vma;
unsigned long sz = huge_page_size(h);
struct mm_struct *mm = vma->vm_mm;
unsigned long old_end = old_addr + len;
@@ -5237,6 +5238,8 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
/* Prevent race with file truncation */
hugetlb_vma_lock_write(vma);
i_mmap_lock_write(mapping);
+ if (anon_vma)
+ anon_vma_lock_write(anon_vma);
for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
const unsigned long offset_to_last_entry =
(old_addr | last_addr_mask) - old_addr;
@@ -5276,7 +5279,20 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
continue;
}
- dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz);
+ dst_pte = hugetlb_walk(new_vma, new_addr, sz);
+ if (!dst_pte) {
+ /*
+ * We can allocate memory under anon_vma lock with hugetlb,
+ * but lockdep does not know about that. So for now we simply
+ * flush the TLB and temporarily drop the lock.
+ */
+ tlb_flush_mmu_tlbonly(&tlb);
+ if (anon_vma)
+ anon_vma_unlock_write(anon_vma);
+ dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz);
+ if (anon_vma)
+ anon_vma_lock_write(anon_vma);
+ }
if (!dst_pte)
break;
@@ -5284,7 +5300,10 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
tlb_remove_huge_tlb_entry(h, &tlb, src_pte, old_addr);
}
+ /* Flush pending moves before allowing migration to walk the entries. */
tlb_flush_mmu_tlbonly(&tlb);
+ if (anon_vma)
+ anon_vma_unlock_write(anon_vma);
huge_pmd_unshare_flush(&tlb, vma);
mmu_notifier_invalidate_range_end(&range);
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 07/15] hugetlbfs: fix unmapping folios from private VMAs
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (5 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 06/15] mm/hugetlb: fix mremap() racing with page migration David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 08/15] mm/hugetlb: serialize page-cache allocations with truncation David Hildenbrand (Arm)
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Josef Bacik, stable, Daehyeon Ko
From: Josef Bacik <josef@toxicpanda.com>
hugetlb_unmap_file_folio() assumes vm_private_data points to a struct
hugetlb_vma_lock when hugetlb_vma_trylock_write() fails. Since hugetlb VMA
locking was extended to private VMAs, this can also happen when
vm_private_data contains a resv_map pointer; treating it as a
hugetlb_vma_lock corrupts memory and can cause crashes or hangs.
Only take the VMA lock when a shareable lock exists, as additional locking
is only needed for PMD table sharing. The fault mutex and i_mmap_rwsem
already provide the required protection for private VMAs.
Fix generated by an LLM and sent by Josef Bacik in private.
Fixes: bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs")
Cc: stable@kernel.org
Tested-by: Daehyeon Ko <4ncienth@gmail.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
[ david: simplify description and comment ]
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
fs/hugetlbfs/inode.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 9f502a456cdb0..1d0c35e639e80 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -401,6 +401,16 @@ static void hugetlb_unmap_file_folio(struct hstate *h,
if (!hugetlb_vma_maps_pfn(vma, v_start, pfn))
continue;
+ /*
+ * We only need additional VMA locking here for PMD table
+ * sharing, which synchronizes on the shareable lock.
+ */
+ if (!__vma_shareable_lock(vma)) {
+ unmap_hugepage_range(vma, v_start, v_end, NULL,
+ ZAP_FLAG_DROP_MARKER);
+ continue;
+ }
+
if (!hugetlb_vma_trylock_write(vma)) {
vma_lock = vma->vm_private_data;
/*
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 08/15] mm/hugetlb: serialize page-cache allocations with truncation
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (6 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 07/15] hugetlbfs: fix unmapping folios from private VMAs David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 09/15] mm/memfd: recheck the page cache before allocating a hugetlb folio David Hildenbrand (Arm)
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable, Josef Bacik
Truncation can remove an inode reservation while a fault is still
allocating its folio. UFFDIO_COPY and memfd pinning can also consume
reservations beyond the new i_size. This can underflow resv_huge_pages
and leave folios beyond EOF.
The main issue is that truncation takes the fault mutex for folios in
the page-cache, but it has no clue about faults that are still being
processed concurrently.
Matthew Wilcox raised mapping->invalidate_lock in a different upstream
hugetlb locking discussion:
"although I wonder if we still need the fault mutex array, given that we
now have mapping->invalidate_lock?"
While it likely cannot remove the mutex array entirely (concurrent page
faults have to sync ... ), we can at least use it to fix this problem.
Hold mapping->invalidate_lock shared across the size check, allocation
and page cache insertion or reservation rollback. Truncation takes it
exclusively around i_size_write() only for now: it waits for concurrent
allocations to finish updating the page-cache or rolling back reservations
before changing the size. Subsequent allocations see the updated i_size.
Let truncation release the lock for now before unmapping and removing
folios to preserve the existing lock ordering. Cached-folio faults and
private allocations do not acquire the new lock.
Use it to synchronize with truncation, keeping the fault mutexes to
serialize allocations for the same index.
Locking order is now:
fault mutex -> hugetlb VMA lock (read) -> invalidate lock (read) ->
folio lock -> PTE lock
And on truncation:
i_rwsem (write) -> invalidate lock (write)
Add hugetlb_pgoff_valid() to not repeat the same size check multiple
times. Memfd needs a bit of care, as it traditionally allowed partial
folios to be pinned: the size check in memfd_pin_folios() allows for
that.
Fixes: 1c9e8def43a3 ("userfaultfd: hugetlbfs: add UFFDIO_COPY support for shared mappings")
Fixes: 188a39725ad7 ("hugetlbfs: revert use i_mmap_rwsem to address page fault/truncate race")
Fixes: 89c1905d9c14 ("mm/gup: introduce memfd_pin_folios() for pinning memfd folios")
Cc: stable@kernel.org
Reported-by: Josef Bacik <josef@toxicpanda.com>
Link: https://lore.kernel.org/all/ajMfdQf-Fspwmstr@casper.infradead.org/
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
fs/hugetlbfs/inode.c | 15 +++++++++++----
include/linux/hugetlb.h | 12 ++++++++++++
mm/hugetlb.c | 45 +++++++++++++++++++++++++++++++++------------
mm/memfd.c | 8 ++++++++
4 files changed, 64 insertions(+), 16 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 1d0c35e639e80..7f64bdc07aa80 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -546,10 +546,9 @@ static void remove_inode_single_folio(struct hstate *h, struct inode *inode,
* truncation is indicated by end of range being LLONG_MAX
* In this case, we first scan the range and release found pages.
* After releasing pages, hugetlb_unreserve_pages cleans up region/reserve
- * maps and global counts. Page faults can race with truncation.
- * During faults, hugetlb_no_page() checks i_size before page allocation,
- * and again after obtaining page table lock. It will 'back out'
- * allocations in the truncated range.
+ * maps and global counts. hugetlb_vmtruncate() updates i_size under
+ * invalidate_lock to synchronize with new page cache allocations from
+ * page faults, uffd, etc.
* hole punch is indicated if end is not LLONG_MAX
* In the hole punch case we scan the range and release found pages.
* Only when releasing a page is the associated region/reserve map
@@ -623,7 +622,15 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset)
BUG_ON(offset & ~huge_page_mask(h));
pgoff = offset >> PAGE_SHIFT;
+ filemap_invalidate_lock(mapping);
i_size_write(inode, offset);
+ /*
+ * The size check now prevents new page-cache allocations in the
+ * truncated range. Existing folios and reservations are cleaned up
+ * below.
+ */
+ filemap_invalidate_unlock(mapping);
+
i_mmap_lock_write(mapping);
if (mapping_mapped(mapping))
hugetlb_vmdelete_list(mapping, pgoff, 0, ZAP_FLAG_DROP_MARKER);
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 80ae65c7849b9..1d2dd10074d7a 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -820,6 +820,18 @@ static inline unsigned huge_page_shift(struct hstate *h)
return h->order + PAGE_SHIFT;
}
+static inline bool hugetlb_pgoff_valid(struct address_space *mapping,
+ pgoff_t idx, bool partial)
+{
+ struct hstate *h = hstate_inode(mapping->host);
+ u64 size = i_size_read(mapping->host);
+
+ if (partial)
+ size = round_up(size, huge_page_size(h));
+
+ return idx < (size >> huge_page_shift(h));
+}
+
/**
* hugetlb_linear_page_index() - linear_page_index() but in hugetlb
* page size granularity.
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4a24819733730..3d8eebdca5136 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5887,6 +5887,7 @@ static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, unsigned
static vm_fault_t hugetlb_no_page(struct address_space *mapping,
struct vm_fault *vmf)
{
+ const bool pagecache_folio = vmf->vma->vm_flags & VM_MAYSHARE;
u32 hash = hugetlb_fault_mutex_hash(mapping, vmf->pgoff);
bool new_folio, new_anon_folio = false;
struct vm_area_struct *vma = vmf->vma;
@@ -5895,7 +5896,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
vm_fault_t ret = VM_FAULT_SIGBUS;
bool folio_locked = true;
struct folio *folio;
- unsigned long size;
pte_t new_pte;
/*
@@ -5917,8 +5917,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
new_folio = false;
folio = filemap_lock_hugetlb_folio(h, mapping, vmf->pgoff);
if (IS_ERR(folio)) {
- size = i_size_read(mapping->host) >> huge_page_shift(h);
- if (vmf->pgoff >= size)
+ if (!hugetlb_pgoff_valid(mapping, vmf->pgoff, /* partial= */ false))
goto out;
/* Check for page in userfault range */
if (userfaultfd_missing(vma)) {
@@ -5948,10 +5947,17 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
VM_UFFD_MISSING);
}
- if (!(vma->vm_flags & VM_MAYSHARE)) {
+ if (!pagecache_folio) {
ret = __vmf_anon_prepare(vmf);
if (unlikely(ret))
goto out;
+ } else {
+ /* Recheck EOF before consuming an inode reservation. */
+ filemap_invalidate_lock_shared(mapping);
+ if (!hugetlb_pgoff_valid(mapping, vmf->pgoff, /* partial= */ false)) {
+ filemap_invalidate_unlock_shared(mapping);
+ goto out;
+ }
}
folio = alloc_hugetlb_folio(vma, vmf->address, false);
@@ -5972,13 +5978,15 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
ret = vmf_error(PTR_ERR(folio));
else
ret = 0;
+ if (pagecache_folio)
+ filemap_invalidate_unlock_shared(mapping);
goto out;
}
folio_zero_user(folio, vmf->real_address);
__folio_mark_uptodate(folio);
new_folio = true;
- if (vma->vm_flags & VM_MAYSHARE) {
+ if (pagecache_folio) {
int err = hugetlb_add_to_page_cache(folio, mapping,
vmf->pgoff);
if (err) {
@@ -5993,8 +6001,10 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
folio);
folio_put(folio);
ret = VM_FAULT_SIGBUS;
+ filemap_invalidate_unlock_shared(mapping);
goto out;
}
+ filemap_invalidate_unlock_shared(mapping);
} else {
new_anon_folio = true;
folio_lock(folio);
@@ -6419,14 +6429,15 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
uffd_flags_t flags,
struct folio **foliop)
{
+ const int vm_shared = dst_vma->vm_flags & VM_SHARED;
+ const bool is_continue = uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE);
+ const bool alloc_pagecache_folio = vm_shared && !is_continue;
struct mm_struct *dst_mm = dst_vma->vm_mm;
- bool is_continue = uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE);
bool wp_enabled = (flags & MFILL_ATOMIC_WP);
struct hstate *h = hstate_vma(dst_vma);
struct address_space *mapping = dst_vma->vm_file->f_mapping;
pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr);
unsigned long size = huge_page_size(h);
- int vm_shared = dst_vma->vm_flags & VM_SHARED;
pte_t _dst_pte;
spinlock_t *ptl;
int ret = -ENOMEM;
@@ -6454,6 +6465,18 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
return 0;
}
+ if (alloc_pagecache_folio) {
+ filemap_invalidate_lock_shared(mapping);
+ if (!hugetlb_pgoff_valid(mapping, idx, /* partial= */ false)) {
+ if (*foliop) {
+ folio_put(*foliop);
+ *foliop = NULL;
+ }
+ ret = -EFAULT;
+ goto out;
+ }
+ }
+
if (is_continue) {
ret = -EFAULT;
folio = filemap_lock_hugetlb_folio(h, mapping, idx);
@@ -6551,11 +6574,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
WARN_ON_ONCE(!folio_test_uptodate(folio));
/* Add shared, newly allocated pages to the page cache. */
- if (vm_shared && !is_continue) {
- ret = -EFAULT;
- if (idx >= (i_size_read(mapping->host) >> huge_page_shift(h)))
- goto out_release_nounlock;
-
+ if (alloc_pagecache_folio) {
/*
* Serialization between remove_inode_hugepages() and
* hugetlb_add_to_page_cache() below happens through the
@@ -6626,6 +6645,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
spin_unlock(ptl);
ret = 0;
out:
+ if (alloc_pagecache_folio)
+ filemap_invalidate_unlock_shared(mapping);
return ret;
out_release_unlock:
spin_unlock(ptl);
diff --git a/mm/memfd.c b/mm/memfd.c
index 0e3ff3eebfa82..f03d12aa71f7a 100644
--- a/mm/memfd.c
+++ b/mm/memfd.c
@@ -96,6 +96,12 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
+ filemap_invalidate_lock_shared(memfd->f_mapping);
+ if (!hugetlb_pgoff_valid(memfd->f_mapping, idx, /* partial= */ true)) {
+ err = -EINVAL;
+ goto out_unlock;
+ }
+
nr_resv = hugetlb_reserve_pages(inode, idx, idx + 1, NULL, EMPTY_VMA_FLAGS);
if (nr_resv < 0) {
err = nr_resv;
@@ -131,6 +137,7 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
hugetlb_set_folio_subpool(folio, subpool_inode(inode));
folio_unlock(folio);
+ filemap_invalidate_unlock_shared(memfd->f_mapping);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
return folio;
}
@@ -138,6 +145,7 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
if (nr_resv > 0)
hugetlb_unreserve_pages(inode, idx, idx + 1, 0);
out_unlock:
+ filemap_invalidate_unlock_shared(memfd->f_mapping);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
return ERR_PTR(err);
}
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 09/15] mm/memfd: recheck the page cache before allocating a hugetlb folio
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (7 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 08/15] mm/hugetlb: serialize page-cache allocations with truncation David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 10/15] mm/hugetlb: count again if the reserve map shrank David Hildenbrand (Arm)
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Josef Bacik
A concurrent fault or PIN request can populate the page cache
after memfd_pin_folios() misses it in the page cache but before
memfd_alloc_folio() takes the fault mutex. The latter still tries to
allocate another folio and can return -ENOMEM even though the requested
folio is already cached.
In particular, hugetlb_reserve_pages() finds the existing reservation-map
entry and reserves nothing. If no global reservations remain,
alloc_hugetlb_folio_reserve() fails even when the pool has free pages,
effectively failing memfd_pin_folios().
Recheck the page cache before reserving or allocating. If the folio is
present, drop the lookup reference and return -EEXIST so
memfd_pin_folios() retries its lookup.
This is probably not worth backporting to stable.
Fixes: 89c1905d9c14 ("mm/gup: introduce memfd_pin_folios() for pinning memfd folios")
Reported-by: Josef Bacik <josef@toxicpanda.com>
Acked-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/memfd.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/memfd.c b/mm/memfd.c
index f03d12aa71f7a..276f4323b6bda 100644
--- a/mm/memfd.c
+++ b/mm/memfd.c
@@ -102,6 +102,15 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
goto out_unlock;
}
+ /* Let the caller retry if we raced with population. */
+ folio = filemap_get_folio(memfd->f_mapping,
+ idx << huge_page_order(h));
+ if (!IS_ERR(folio)) {
+ folio_put(folio);
+ err = -EEXIST;
+ goto out_unlock;
+ }
+
nr_resv = hugetlb_reserve_pages(inode, idx, idx + 1, NULL, EMPTY_VMA_FLAGS);
if (nr_resv < 0) {
err = nr_resv;
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 10/15] mm/hugetlb: count again if the reserve map shrank
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (8 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 09/15] mm/memfd: recheck the page cache before allocating a hugetlb folio David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 11/15] hugetlbfs: do not grow a sealed file in mmap() David Hildenbrand (Arm)
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable, Josef Bacik
hugetlb_reserve_pages() asks region_chg() how many pages of the range are
missing in the reserve map, charges the cgroup, the subpool and the pool
for that many, and then lets region_add() fill the gaps.
It copes with region_add() adding less, because a concurrent fault added
an entry in the meantime; however, it does not expect it to add more.
Since restore_reserve_on_error() can delete the entry of an allocation
that failed later on, entries also disappear without the inode lock that
mmap() holds. For a previously unreserved page, UFFDIO_COPY can do this
when the source is not resident: it allocates a folio, which adds the
entry, fails to copy with page faults disabled, and takes both back.
memfd_alloc_folio() does it when it has made a reservation and gets no
folio it may pin.
If region_chg() sees such an entry and region_add() does not, an entry is
added that nothing was charged for. When we later delete it,
we decrement resv_huge_pages and uncharge the cgroup once too often:
WARNING: mm/page_counter.c:60 at page_counter_uncharge+0xe2/0x160
Similarly, HugePages_Rsvd will underflow.
Tell region_add() how much has been accounted for and let it refuse to
add more. Convert it to a region_add_limited() for that purpose.
hugetlb_reserve_pages() then undoes what it has done, as for any other
failure of region_add(), and starts again.
Note that we move hugetlb_vma_lock_free() so we can retain the allocated
VMA lock object as we retry.
Limit retries to 10 to avoid livelock under repeated concurrent rollbacks.
Return -ENOMEM if the retries are exhausted.
Fixes: 846be08578ed ("mm/hugetlb: expand restore_reserve_on_error functionality")
Cc: stable@kernel.org
Reported-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 58 ++++++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 44 insertions(+), 14 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3d8eebdca5136..112fa25f36833 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -739,26 +739,36 @@ static int allocate_file_region_entries(struct resv_map *resv,
* region_del. The extra needed entries will be allocated.
*
* regions_needed is the out value provided by a previous call to region_chg.
+ * limit is what the caller has accounted for. If more huge pages would have
+ * to be added, because somebody has deleted from the range since region_chg,
+ * nothing is added and -EAGAIN is returned.
*
* Return the number of new huge pages added to the map. This number is greater
* than or equal to zero. If file_region entries needed to be allocated for
* this operation and we were not able to allocate, it returns -ENOMEM.
- * region_add of regions of length 1 never allocate file_regions and cannot
- * fail; region_chg will always allocate at least 1 entry and a region_add for
- * 1 page will only require at most 1 entry.
+ * Ranges of length 1 never require allocating file_regions and cannot fail
+ * if limit is at least 1; region_chg will always allocate at least 1 entry
+ * and adding 1 page will only require at most 1 entry.
*/
-static long region_add(struct resv_map *resv, long f, long t,
- long in_regions_needed, struct hstate *h,
- struct hugetlb_cgroup *h_cg)
+static long region_add_limited(struct resv_map *resv, long f, long t,
+ long in_regions_needed, struct hstate *h,
+ struct hugetlb_cgroup *h_cg, long limit)
{
long add = 0, actual_regions_needed = 0;
spin_lock(&resv->lock);
retry:
- /* Count how many regions are actually needed to execute this add. */
- add_reservation_in_range(resv, f, t, NULL, NULL,
- &actual_regions_needed);
+ /*
+ * Count how many regions are actually needed to execute this add.
+ * Fail if the number of huge pages to add exceeds limit.
+ */
+ add = add_reservation_in_range(resv, f, t, NULL, NULL,
+ &actual_regions_needed);
+ if (add > limit) {
+ spin_unlock(&resv->lock);
+ return -EAGAIN;
+ }
/*
* Check for sufficient descriptors in the cache to accommodate
@@ -794,6 +804,13 @@ static long region_add(struct resv_map *resv, long f, long t,
return add;
}
+static long region_add(struct resv_map *resv, long f, long t,
+ long in_regions_needed, struct hstate *h,
+ struct hugetlb_cgroup *h_cg)
+{
+ return region_add_limited(resv, f, t, in_regions_needed, h, h_cg, t - f);
+}
+
/*
* Examine the existing reserve map and determine how many
* huge pages in the specified range [f, t) are NOT currently
@@ -6847,7 +6864,7 @@ long hugetlb_reserve_pages(struct inode *inode,
struct resv_map *resv_map;
struct hugetlb_cgroup *h_cg = NULL;
long gbl_reserve, regions_needed = 0;
- int err;
+ int err, retries = 0;
/* This should never happen */
if (from > to) {
@@ -6869,6 +6886,7 @@ long hugetlb_reserve_pages(struct inode *inode,
if (vma_flags_test(&vma_flags, VMA_NORESERVE_BIT))
return 0;
+retry:
/*
* Shared mappings base their reservation on the number of pages that
* are already allocated on behalf of the file. Private mappings need
@@ -6947,7 +6965,8 @@ long hugetlb_reserve_pages(struct inode *inode,
* else has to be done for private mappings here
*/
if (!vma || vma_test(vma, VMA_MAYSHARE_BIT)) {
- add = region_add(resv_map, from, to, regions_needed, h, h_cg);
+ add = region_add_limited(resv_map, from, to, regions_needed, h,
+ h_cg, chg);
if (unlikely(add < 0)) {
hugetlb_acct_memory(h, -gbl_reserve);
@@ -7010,17 +7029,28 @@ long hugetlb_reserve_pages(struct inode *inode,
hugetlb_cgroup_uncharge_cgroup_rsvd(hstate_index(h),
chg * pages_per_huge_page(h), h_cg);
out_err:
- hugetlb_vma_lock_free(vma);
- if (!vma || vma_test(vma, VMA_MAYSHARE_BIT))
+ if (!vma || vma_test(vma, VMA_MAYSHARE_BIT)) {
/* Only call region_abort if the region_chg succeeded but the
* region_add failed or didn't run.
*/
if (chg >= 0 && add < 0)
region_abort(resv_map, from, to, regions_needed);
- if (vma && is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
+ /*
+ * A concurrent rollback removed entries after region_chg()
+ * counted the range, so we charged for too few pages and
+ * have to recount, recharge, and try again. Limit retries
+ * to avoid livelock under repeated rollbacks.
+ */
+ if (err == -EAGAIN) {
+ if (retries++ < 10)
+ goto retry;
+ err = -ENOMEM;
+ }
+ } else if (is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
kref_put(&resv_map->refs, resv_map_release);
set_vma_resv_map(vma, NULL);
}
+ hugetlb_vma_lock_free(vma);
return err;
}
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 11/15] hugetlbfs: do not grow a sealed file in mmap()
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (9 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 10/15] mm/hugetlb: count again if the reserve map shrank David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 12/15] mm/hugetlb: keep surplus folios that are reserved David Hildenbrand (Arm)
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Chris Mason, stable, Gregory Price (Meta)
From: Chris Mason <mason@kernel.org>
hugetlbfs_file_mmap() grows a file when a writable mapping extends beyond
EOF. We are missing seal checks similar to hugetlbfs_setattr() and
hugetlbfs_fallocate(), allowing a user to grow a file that has F_SEAL_GROW
set.
To fix it, skip the resize when F_SEAL_GROW is set; mmap() still succeeds
without growing the file.
We currently also allow CoW mappings to grow a file. Disallowing this,
maybe even only for read-only file descriptors, could break existing
userspace, so we have to be more careful. QEMU's file backend is an
example user that might rely on such things, and would suddenly start
getting SIGBUS when faulting in memory outside EOF.
So be careful and focus on F_SEAL_GROW for now.
Fix generated by an LLM and sent by Chris Mason in private.
Fixes: ff62a3421044 ("hugetlb: implement memfd sealing")
Cc: stable@kernel.org
Signed-off-by: Chris Mason <mason@kernel.org>
[ david: drop comment; simplify+extend description ]
Acked-by: Muchun Song <muchun.song@linux.dev>
Tested-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
fs/hugetlbfs/inode.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 7f64bdc07aa80..68761e1ed4d2e 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -157,7 +157,8 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
goto out;
ret = 0;
- if (vma_test(vma, VMA_WRITE_BIT) && inode->i_size < len)
+ if (vma_test(vma, VMA_WRITE_BIT) && inode->i_size < len &&
+ !(HUGETLBFS_I(inode)->seals & F_SEAL_GROW))
i_size_write(inode, len);
out:
inode_unlock(inode);
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 12/15] mm/hugetlb: keep surplus folios that are reserved
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (10 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 11/15] hugetlbfs: do not grow a sealed file in mmap() David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 13/15] mm/hugetlb: fix reservation leak on min_size= mounts David Hildenbrand (Arm)
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Chris Mason, stable, Gregory Price (Meta)
From: Chris Mason <mason@kernel.org>
free_huge_folio() may restore a reservation, but still returns the folio
to the page allocator if its node has surplus huge pages.
For example, start with no hugetlb folios and allow one surplus 2 MiB
folio:
1. Mount hugetlbfs with pagesize=2M,min_size=2M. This allocates and
reserves one surplus folio: free=1, reserved=1.
2. Create a 2 MiB file and write to a shared mapping of it. The fault
consumes the folio and its reservation: free=0, reserved=0.
3. Unmap the file and truncate it to zero. The mount's reservation is
restored, but the surplus folio is freed: free=0, reserved=1.
Besides losing a hugetlb folio that is required for a reservation, this
can also make available_huge_pages() underflow and cause other issues.
Keep the folio in the pool if it is needed to cover reservations,
including any reservation we just restored.
Fix generated by an LLM and sent by Chris Mason in private.
Fixes: 07443a85ad90 ("mm, hugetlb: return a reserved page to a reserved pool if failed")
Cc: stable@kernel.org
Signed-off-by: Chris Mason <mason@kernel.org>
[ david: simplify comment and description ]
Acked-by: Muchun Song <muchun.song@linux.dev>
Tested-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/hugetlb.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 112fa25f36833..2fd5a5e585ad5 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1778,8 +1778,9 @@ void free_huge_folio(struct folio *folio)
remove_hugetlb_folio(h, folio, false);
spin_unlock_irqrestore(&hugetlb_lock, flags);
update_and_free_hugetlb_folio(h, folio, true);
- } else if (h->surplus_huge_pages_node[nid]) {
- /* remove the page from active list */
+ } else if (h->surplus_huge_pages_node[nid] &&
+ h->free_huge_pages >= h->resv_huge_pages) {
+ /* Free this surplus folio that is not required globally. */
remove_hugetlb_folio(h, folio, true);
spin_unlock_irqrestore(&hugetlb_lock, flags);
update_and_free_hugetlb_folio(h, folio, true);
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 13/15] mm/hugetlb: fix reservation leak on min_size= mounts
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (11 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 12/15] mm/hugetlb: keep surplus folios that are reserved David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 14/15] mm/hugetlb: account restored reservations before freeing folios David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 15/15] mm/hugetlb: delay anon folio rmap removal until after the TLB flush David Hildenbrand (Arm)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Chris Mason, stable, Gregory Price (Meta)
From: Chris Mason <mason@kernel.org>
Failed allocations for CoW mappings can leak reservations on hugetlbfs
mounts with min_size= and no size=. The leaked reservations survive
unmount.
Note that this is only an accounting error and not a memory leak.
HPG_restore_reserve makes free_huge_folio() restore the global
reservation without returning it to the subpool. This assumes the
reserve map holds the reservation. However, a CoW mapping may have no
reserve map, or the map may already account for another folio at the
same index.
Commit 58db7c5fbe7d ("mm/hugetlb: fix avoid_reserve to allow taking
folio from subpool") allowed owner CoW allocations to consume subpool
reservations. Commit 51e1de00acdb ("mm/hugetlb: simplify
vma_has_reserves()") later extended this to CoW mappings without a
reserve map.
For example, with a subpool minimum of one page and a
MAP_PRIVATE|MAP_NORESERVE mapping:
subpool global
rsv_hpages resv_huge_pages
1. Mount 1 1
2. Allocate using subpool reserve 0 0
3. Free on error with flag set 0 1
4. Allocate without a reservation 0 1
5. Free and return to subpool 1 2
Call folio_clear_hugetlb_restore_reserve() for CoW mappings when
map_chg != MAP_CHG_REUSE, meaning no reserve-map reservation was consumed.
This lets free_huge_folio() return subpool reservations to the subpool
on error.
Do not set the flag again if region-cache allocation fails during
rollback: that would undo the flag clearing and skip returning the
reservation to the subpool. This also fixes the older case where setting
the flag invents a reservation for an unreserved allocation for a CoW
mapping.
The doc still mentions PagePrivate at various places: future work should
clean that up.
Fix generated by an LLM and sent by Chris Mason in private.
Fixes: 846be08578ed ("mm/hugetlb: expand restore_reserve_on_error functionality")
Fixes: 58db7c5fbe7d ("mm/hugetlb: fix avoid_reserve to allow taking folio from subpool")
Cc: stable@kernel.org
Signed-off-by: Chris Mason <mason@kernel.org>
[ david: simplify comments and improve description; updated Fixes ]
Tested-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
Documentation/mm/hugetlbfs_reserv.rst | 25 ++++++++++++++-----------
include/linux/hugetlb.h | 2 ++
mm/hugetlb.c | 34 +++++++++++++++-------------------
3 files changed, 31 insertions(+), 30 deletions(-)
diff --git a/Documentation/mm/hugetlbfs_reserv.rst b/Documentation/mm/hugetlbfs_reserv.rst
index a49115db18c76..fbca6625ca50d 100644
--- a/Documentation/mm/hugetlbfs_reserv.rst
+++ b/Documentation/mm/hugetlbfs_reserv.rst
@@ -201,10 +201,10 @@ The value returned from vma_needs_reservation() is generally
If a reservation does not exist, and there is a subpool associated with the
mapping the subpool is consulted to determine if it contains reservations.
If the subpool contains reservations, one can be used for this allocation.
-However, in every case the avoid_reserve argument overrides the use of
-a reservation for the allocation. After determining whether a reservation
-exists and can be used for the allocation, the routine dequeue_huge_page_vma()
-is called. This routine takes two arguments related to reservations:
+However, avoid_reserve only applies to reservations from the reservation map.
+After determining whether a reservation exists and can be used for the
+allocation, the routine dequeue_huge_page_vma() is called. This routine takes
+two arguments related to reservations:
- avoid_reserve, this is the same value/argument passed to
alloc_hugetlb_folio().
@@ -256,6 +256,10 @@ compensate. See the section
:ref:`Reservation Map Helper Routines <resv_map_helpers>` for more
information on these routines.
+For CoW mappings, alloc_hugetlb_folio() calls
+folio_clear_hugetlb_restore_reserve() if the reservation came from the subpool.
+This makes free_huge_folio() return the reservation to the subpool on error.
+
Instantiate Huge Pages
======================
@@ -284,14 +288,13 @@ be adjusted (see the section
:ref:`Consuming Reservations/Allocating a Huge Page <consume_resv>`
for information on how these are set).
-The routine first calls hugepage_subpool_put_pages() for the page. If this
-routine returns a value of 0 (which does not equal the value passed 1) it
-indicates reserves are associated with the subpool, and this newly free page
-must be used to keep the number of subpool reserves above the minimum size.
-Therefore, the global resv_huge_pages counter is incremented in this case.
+If folio_test_hugetlb_restore_reserve() is false, hugepage_subpool_put_pages()
+returns the page to the subpool. A return value of 0 means the page must restore
+a subpool reservation, so the global resv_huge_pages counter is incremented.
-If the PagePrivate flag was set in the page, the global resv_huge_pages counter
-will always be incremented.
+Otherwise, resv_huge_pages is incremented without calling
+hugepage_subpool_put_pages(): the restored reservation still counts as used in
+the subpool.
.. _sub_pool_resv:
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 1d2dd10074d7a..0d5414392d870 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -590,6 +590,8 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
* HPG_restore_reserve - Set when a hugetlb page consumes a reservation at
* allocation time. Cleared when page is fully instantiated. Free
* routine checks flag to restore a reservation on error paths.
+ * Cleared when allocating for a CoW mapping using a subpool
+ * reservation, which the free routine returns to the subpool.
* Synchronization: Examined or modified by code that knows it has
* the only reference to page. i.e. After allocation but before use
* or when the page is being freed.
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 2fd5a5e585ad5..6e79dedfc0fd6 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2635,7 +2635,7 @@ static long vma_del_reservation(struct hstate *h,
* 1) A reservation was in place and the folio consumed the reservation.
* hugetlb_restore_reserve is set in the folio.
* 2) No reservation was in place for the page, so hugetlb_restore_reserve is
- * not set. However, alloc_hugetlb_folio always updates the reserve map.
+ * not set. However, alloc_hugetlb_folio may have updated the reserve map.
*
* In case 1, free_huge_folio later in the error path will increment the
* global reserve count. But, free_huge_folio does not have enough context
@@ -2645,6 +2645,10 @@ static long vma_del_reservation(struct hstate *h,
* reserve map indicates there is a reservation present.
*
* In case 2, simply undo reserve map modifications done by alloc_hugetlb_folio.
+ *
+ * For CoW mappings, allocations using a subpool reservation leave the
+ * flag clear and the reserve map unchanged. free_huge_folio returns the
+ * reservation to the subpool.
*/
void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
unsigned long address, struct folio *folio)
@@ -2692,25 +2696,9 @@ void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
folio_set_hugetlb_restore_reserve(folio);
} else if (rc < 0) {
/*
- * Rare out of memory condition from
- * vma_needs_reservation call. Memory allocation is
- * only attempted if a new entry is needed. Therefore,
- * this implies there is not an entry in the
- * reserve map.
- *
- * For shared mappings, no entry in the map indicates
- * no reservation. We are done.
+ * Region cache allocation can fail even if an entry exists.
+ * Leave the flag and reserve map unchanged.
*/
- if (!(vma->vm_flags & VM_MAYSHARE))
- /*
- * For private mappings, no entry indicates
- * a reservation is present. Since we can
- * not add an entry, set hugetlb_restore_reserve
- * on the folio so reserve count will be
- * incremented when freed. This reserve will
- * be consumed on a subsequent allocation.
- */
- folio_set_hugetlb_restore_reserve(folio);
} else {
/*
* No reservation present, do nothing
@@ -3165,6 +3153,14 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
}
}
+ /*
+ * Return subpool reservations to the subpool on error. A CoW mapping
+ * might not have a reserve map, or it might already account for
+ * another folio at this index.
+ */
+ if (map_chg != MAP_CHG_REUSE && vma_is_cow_mapping(vma))
+ folio_clear_hugetlb_restore_reserve(folio);
+
return folio;
out_subpool_put:
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 14/15] mm/hugetlb: account restored reservations before freeing folios
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (12 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 13/15] mm/hugetlb: fix reservation leak on min_size= mounts David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 15/15] mm/hugetlb: delay anon folio rmap removal until after the TLB flush David Hildenbrand (Arm)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
stable, Josef Bacik
Unmapping an anonymous folio restores its private VMA reservation
immediately, but the global reservation count is only restored when the
folio is freed.
If any folio reference delays freeing, closing the VMA or refaulting can
decrement resv_huge_pages before it was incremented. This can underflow
the counter or hide another mapping's reservation, allowing an unreserved
allocation to consume its backing.
Increment resv_huge_pages when restoring the reserve map instead. Detach
the folio from its subpool, since the restored reservation now owns that
charge. Leave the restore-reserve flag clear so freeing the folio does not
repeat the accounting. The folio only becomes free after its last reference
is dropped.
Reservations can now temporarily outnumber free folios. Clamp available
pages to zero in that case, including for demotion, and limit releases of
pooled surplus pages to that availability so they cannot remove backing
for remaining reservations.
Use the same clamped availability when gathering surplus pages for a new
reservation. An existing deficit can be backed by unmapped folios that
are still referenced, so do not allocate replacement backing for it as
part of the new request. Otherwise, the request can allocate unnecessary
surplus pages or fail even when the overcommit limit has room for the new
reservation.
Free unused or aborted surplus allocations directly instead of passing
them to free_huge_folio(), which could retain them to cover the existing
deficit. These isolated folios have not backed any committed reservation.
Preserve pages promoted to persistent capacity by a concurrent pool resize,
and use the bulk freeing path to retain vmemmap restoration error handling.
Restore reservations even when surplus pages exist. free_huge_folio()
already retains pages needed to back them. It is unclear why that was
really added. The comment about i_mmap_lock_write can go as well, that
doesn't really help for anonymous folios.
Two existing problems remain:
1) Anonymous hugetlb migration temporarily drops the mapcount. A concurrent
unmap can observe zero and restore a reservation even though migration
will reinstall a mapping. i_mmap_lock_write() does not protect anonymous
folio mapcounts, so it does not prevent this race. This change fixes the
resulting counter underflow, but leaves premature restoration possible,
leading to (2). Maybe that anon_vma lock can help?
2) A fault can allocate a replacement folio using the restored
reservation before the old folio is freed. Repeated vmsplice(),
MADV_DONTNEED and refaults can retain multiple folios for one
reservation and consume pages backing other reservations. This was
already possible; counter accounting is now fixed, but early reuse
remains. Removing the surplus check also allows it after unmaps that
previously skipped restoration.
Deferring reservation-map restoration until the final put is tricky:
free_huge_folio() has no VMA context to update the private reserve map, and
that VMA may already have been destroyed.
Fixes: df7a6d1f6405 ("mm/hugetlb: restore the reservation if needed")
Cc: stable@kernel.org
Reported-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
Documentation/mm/hugetlbfs_reserv.rst | 5 +-
mm/hugetlb.c | 125 ++++++++++++++++++----------------
mm/hugetlb_internal.h | 24 +++++++
mm/hugetlb_sysfs.c | 2 +-
4 files changed, 95 insertions(+), 61 deletions(-)
diff --git a/Documentation/mm/hugetlbfs_reserv.rst b/Documentation/mm/hugetlbfs_reserv.rst
index fbca6625ca50d..4711029bd067d 100644
--- a/Documentation/mm/hugetlbfs_reserv.rst
+++ b/Documentation/mm/hugetlbfs_reserv.rst
@@ -34,7 +34,8 @@ resv_huge_pages
This is a global (per-hstate) count of reserved huge pages. Reserved
huge pages are only available to the task which reserved them.
Therefore, the number of huge pages generally available is computed
- as (``free_huge_pages - resv_huge_pages``).
+ as (``free_huge_pages - resv_huge_pages``), clamped to zero when
+ reservations outnumber free huge pages.
Reserve Map
A reserve map is described by the structure::
@@ -155,7 +156,7 @@ are enough free huge pages to accommodate the reservation. If there are,
the global reservation count resv_huge_pages is adjusted something like the
following::
- if (resv_needed <= (free_huge_pages - resv_huge_pages)
+ if (resv_needed <= available_huge_pages(h))
resv_huge_pages += resv_needed;
Note that the global lock hugetlb_lock is held when checking and adjusting
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6e79dedfc0fd6..94de63529cbc5 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1351,11 +1351,6 @@ static struct folio *dequeue_hugetlb_folio_nodemask(struct hstate *h, gfp_t gfp_
return NULL;
}
-static unsigned long available_huge_pages(struct hstate *h)
-{
- return h->free_huge_pages - h->resv_huge_pages;
-}
-
static unsigned long persistent_huge_pages(const struct hstate *h)
{
return h->nr_huge_pages - h->surplus_huge_pages - h->temporary_huge_pages;
@@ -2321,11 +2316,13 @@ static nodemask_t *policy_mbind_nodemask(gfp_t gfp)
*/
static long surplus_pages_needed(struct hstate *h, long delta, long allocated)
{
- long global_free = (long)h->free_huge_pages + allocated;
- long allowed_free = (long)allowed_mems_nr(h) + allocated;
+ const long available = min(available_huge_pages(h), allowed_mems_nr(h));
- return max((long)h->resv_huge_pages + delta - global_free,
- delta - allowed_free);
+ /*
+ * Restored reservations can still be backed by folios that were not
+ * freed yet.
+ */
+ return delta - available - allocated;
}
/*
@@ -2336,6 +2333,7 @@ static int gather_surplus_pages(struct hstate *h, long delta)
__must_hold(&hugetlb_lock)
{
LIST_HEAD(surplus_list);
+ LIST_HEAD(free_list);
struct folio *folio, *tmp;
int ret;
long i;
@@ -2415,14 +2413,26 @@ static int gather_surplus_pages(struct hstate *h, long delta)
enqueue_hugetlb_folio(h, folio);
}
free:
- spin_unlock_irq(&hugetlb_lock);
-
/*
- * Free unnecessary surplus pages to the buddy allocator.
- * Pages have no ref count, call free_huge_folio directly.
+ * These folios are not required for any reservation. Don't use
+ * free_huge_folio(), otherwise it might decide to keep them for
+ * reservations that were restored before the original folios were
+ * freed.
*/
- list_for_each_entry_safe(folio, tmp, &surplus_list, lru)
- free_huge_folio(folio);
+ list_for_each_entry_safe(folio, tmp, &surplus_list, lru) {
+ /*
+ * If the pool was resized concurrently, these folios might now
+ * count as persistent and we must keep them.
+ */
+ if (!h->surplus_huge_pages_node[folio_nid(folio)]) {
+ enqueue_hugetlb_folio(h, folio);
+ continue;
+ }
+ remove_hugetlb_folio(h, folio, true);
+ list_add(&folio->lru, &free_list);
+ }
+ spin_unlock_irq(&hugetlb_lock);
+ update_and_free_pages_bulk(h, &free_list);
spin_lock_irq(&hugetlb_lock);
return ret;
@@ -2453,7 +2463,8 @@ static void return_unused_surplus_pages(struct hstate *h,
* Part (or even all) of the reservation could have been backed
* by pre-allocated pages. Only free surplus pages.
*/
- nr_pages = min(unused_resv_pages, h->surplus_huge_pages);
+ nr_pages = min3(unused_resv_pages, h->surplus_huge_pages,
+ available_huge_pages(h));
/*
* We want to release as many surplus pages as possible, spread
@@ -5328,6 +5339,45 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
return len + old_addr - old_end;
}
+/* Must be called after rmap removal. */
+static void hugetlb_folio_restore_rsv_on_unmap(struct vm_area_struct *vma,
+ struct folio *folio)
+{
+ struct hstate *h = hstate_vma(vma);
+ unsigned long address;
+ long rc;
+
+ /*
+ * Restore only on the last unmap, otherwise the owner could reuse
+ * the reservation while the folio is still mapped by a child.
+ *
+ * TODO: Handle migration temporarily dropping the mapcount to zero.
+ * TODO: Prevent reservation reuse before the folio is freed.
+ */
+ if (!folio_test_anon(folio) || !__vma_private_lock(vma) ||
+ folio_mapped(folio))
+ return;
+
+ address = vma_anon_address(vma, folio->index, 1);
+ rc = vma_needs_reservation(h, vma, address);
+ if (!rc)
+ vma_end_reservation(h, vma, address);
+ if (rc <= 0)
+ return;
+
+ rc = vma_add_reservation(h, vma, address);
+ if (rc < 0)
+ return;
+
+ /* The restored reservation owns the subpool charge. */
+ hugetlb_set_folio_subpool(folio, NULL);
+
+ /* Account before the VMA can close or fault again. */
+ spin_lock_irq(&hugetlb_lock);
+ h->resv_huge_pages++;
+ spin_unlock_irq(&hugetlb_lock);
+}
+
static void hugetlb_unmap_range(struct mmu_gather *tlb,
struct vm_area_struct *vma, unsigned long start,
unsigned long end, struct folio *folio,
@@ -5341,7 +5391,6 @@ static void hugetlb_unmap_range(struct mmu_gather *tlb,
spinlock_t *ptl;
struct hstate *h = hstate_vma(vma);
unsigned long sz = huge_page_size(h);
- bool adjust_reservation;
unsigned long last_addr_mask;
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
@@ -5427,47 +5476,7 @@ static void hugetlb_unmap_range(struct mmu_gather *tlb,
hugetlb_remove_rmap(folio);
spin_unlock(ptl);
- /*
- * Restore the reservation for anonymous page, otherwise the
- * backing page could be stolen by someone. Restore only on the
- * last unmap, otherwise the owner could empty its resv map
- * while the folio is still mapped by a child. Note that holding
- * i_mmap_lock_write is needed to check the number of mappings.
- * If there we are freeing a surplus, do not set the restore
- * reservation bit.
- */
- adjust_reservation = false;
-
- spin_lock_irq(&hugetlb_lock);
- if (!h->surplus_huge_pages && __vma_private_lock(vma) &&
- !folio_mapped(folio) && folio_test_anon(folio)) {
- folio_set_hugetlb_restore_reserve(folio);
- /* Reservation to be adjusted after the spin lock */
- adjust_reservation = true;
- }
- spin_unlock_irq(&hugetlb_lock);
-
- /*
- * Adjust the reservation for the region that will have the
- * reserve restored. Keep in mind that vma_needs_reservation() changes
- * resv->adds_in_progress if it succeeds. If this is not done,
- * do_exit() will not see it, and will keep the reservation
- * forever.
- */
- if (adjust_reservation) {
- int rc = vma_needs_reservation(h, vma, address);
-
- if (rc < 0)
- /* Presumably allocate_file_region_entries failed
- * to allocate a file_region struct. Clear
- * hugetlb_restore_reserve so that global reserve
- * count will not be incremented by free_huge_folio.
- * Act as if we consumed the reservation.
- */
- folio_clear_hugetlb_restore_reserve(folio);
- else if (rc)
- vma_add_reservation(h, vma, address);
- }
+ hugetlb_folio_restore_rsv_on_unmap(vma, folio);
tlb_remove_page_size(tlb, folio_page(folio, 0),
folio_size(folio));
diff --git a/mm/hugetlb_internal.h b/mm/hugetlb_internal.h
index 1d2f870deccfe..201028e08a46a 100644
--- a/mm/hugetlb_internal.h
+++ b/mm/hugetlb_internal.h
@@ -10,6 +10,30 @@
#include <linux/hugetlb.h>
#include <linux/hugetlb_cgroup.h>
+/**
+ * available_huge_pages - number of free huge pages not needed for reservations
+ * @h: huge page size state
+ *
+ * Global number of free huge pages that are not needed for existing
+ * reservations, available for allocations without reservations. Things like
+ * NUMA placement, cpusets, mempolicy and the usability of individual folios can
+ * further restrict allocation.
+ *
+ * Reservations of anon folios may be restored before their backing folios are
+ * actually freed, so the number of reserved pages can be bigger than the
+ * number of free pages. In such a scenario, the result is clamped to 0.
+ *
+ * Return: The number of free huge pages not needed for reservations.
+ */
+static inline unsigned long available_huge_pages(const struct hstate *h)
+{
+ lockdep_assert_held(&hugetlb_lock);
+
+ if (h->resv_huge_pages > h->free_huge_pages)
+ return 0;
+ return h->free_huge_pages - h->resv_huge_pages;
+}
+
/*
* Check if the hstate represents gigantic pages but gigantic page
* runtime support is not available. This is a common condition used to
diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c
index c89e4c8475756..9e0c5d3d0fcb6 100644
--- a/mm/hugetlb_sysfs.c
+++ b/mm/hugetlb_sysfs.c
@@ -211,7 +211,7 @@ static ssize_t demote_store(struct kobject *kobj,
* Check for available pages to demote each time thorough the
* loop as demote_pool_huge_page will drop hugetlb_lock.
*/
- nr_available = h->free_huge_pages - h->resv_huge_pages;
+ nr_available = available_huge_pages(h);
if (nid != NUMA_NO_NODE)
nr_available = min(nr_available,
h->free_huge_pages_node[nid]);
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 15/15] mm/hugetlb: delay anon folio rmap removal until after the TLB flush
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
` (13 preceding siblings ...)
2026-11-16 18:29 ` [PATCH v2 14/15] mm/hugetlb: account restored reservations before freeing folios David Hildenbrand (Arm)
@ 2026-11-16 18:29 ` David Hildenbrand (Arm)
14 siblings, 0 replies; 16+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-16 18:29 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, Naoya Horiguchi, Andrew Morton,
Mel Gorman, Andy Whitcroft, Miaohe Lin, Michal Hocko,
Rik van Riel, Hugh Dickins, Baolin Wang, Vivek Kasireddy,
Dave Airlie, Jason Gunthorpe, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Jonathan Corbet, Shuah Khan, Randy Dunlap, Peter Xu,
Breno Leitao, Will Deacon, Aneesh Kumar K.V, Nick Piggin,
Peter Zijlstra, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Arnd Bergmann
Cc: linux-mm, linux-kernel, Gerd Hoffmann, linux-doc, linux-arch,
linux-s390, David Hildenbrand (Arm),
Chris Mason
After fork(), parent and child map anonymous folios read-only shared
by CoW. On a write fault, hugetlb_wp() reuses an anonymous page instead of
copying it if its mapcount is 1. As documented, switching to the safer
refcount would cause more harm than help with these scarce resources.
munmap() and MADV_DONTNEED unmap such folios and decrement the mapcount
before the TLB is flushed. Until then, another thread of the process can
still read the page through a stale TLB entry:
child: reader child: munmap() parent: writer
============= =============== ==============
reads the page
clears the entry
mapcount drops to 1
store to the page
hugetlb_wp() reuses it
reads the store
/* leak */
flushes the TLB
Similar observations are also known and possible with things like
vmsplice().
Extend the mmu_gather delayed-rmap machinery to handle hugetlb folios.
Defer anonymous hugetlb rmap removal until after the batched TLB flush.
Flush the TLB and remove pending rmaps when the batch fills and before
leaving the VMA, then drain the batch so later rmap flushes cannot process
the same entries again. This is very similar to how zap_present_ptes()
operates.
Skip deferral at exit. On UP, flush before removing the rmap. With
MMU_GATHER_NO_GATHER, clearing the entry already flushes the TLB.
Restore reservations after the actual rmap removal, preserving the existing
mapcount check.
This is very likely not stable-worthy, as hugetlb is commonly not used
for applications with different parent vs. child trust boundaries.
Fixes: 24669e58477e ("hugetlb: use mmu_gather instead of a temporary linked list for accumulating pages")
Reported-by: Chris Mason <mason@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
arch/s390/include/asm/tlb.h | 5 +++--
include/asm-generic/tlb.h | 5 +++--
mm/hugetlb.c | 39 ++++++++++++++++++++++++++++++++++-----
mm/internal.h | 9 +++++++++
mm/mmu_gather.c | 16 ++++++++++++----
5 files changed, 61 insertions(+), 13 deletions(-)
diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h
index 619fd41e710e9..3408ee400d23e 100644
--- a/arch/s390/include/asm/tlb.h
+++ b/arch/s390/include/asm/tlb.h
@@ -24,7 +24,7 @@
static inline void tlb_flush(struct mmu_gather *tlb);
static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,
- struct page *page, int page_size);
+ struct page *page, int page_size, bool delay_rmap);
static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb,
struct page *page, unsigned int nr_pages, bool delay_rmap);
@@ -45,8 +45,9 @@ static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb,
* s390 doesn't delay rmap removal.
*/
static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,
- struct page *page, int page_size)
+ struct page *page, int page_size, bool delay_rmap)
{
+ VM_WARN_ON_ONCE(delay_rmap);
free_folio_and_swap_cache(page_folio(page));
return false;
}
diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
index 3f4a180b48f3c..476fa3495eb29 100644
--- a/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -245,7 +245,8 @@ struct mmu_gather_batch {
*/
#define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH)
-extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size);
+bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
+ int page_size, bool delay_rmap);
bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page,
unsigned int nr_pages, bool delay_rmap);
@@ -464,7 +465,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
static inline void tlb_remove_page_size(struct mmu_gather *tlb,
struct page *page, int page_size)
{
- if (__tlb_remove_page_size(tlb, page, page_size))
+ if (__tlb_remove_page_size(tlb, page, page_size, false))
tlb_flush_mmu(tlb);
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 94de63529cbc5..ec9eaaa13ada3 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5340,7 +5340,7 @@ unsigned long move_hugetlb_page_tables(struct vm_area_struct *vma,
}
/* Must be called after rmap removal. */
-static void hugetlb_folio_restore_rsv_on_unmap(struct vm_area_struct *vma,
+void hugetlb_folio_restore_rsv_on_unmap(struct vm_area_struct *vma,
struct folio *folio)
{
struct hstate *h = hstate_vma(vma);
@@ -5401,6 +5401,8 @@ static void hugetlb_unmap_range(struct mmu_gather *tlb,
last_addr_mask = hugetlb_mask_last_page(h);
address = start;
for (; address < end; address += sz) {
+ bool delay_rmap = false;
+
ptep = hugetlb_walk(vma, address, sz);
if (!ptep) {
address |= last_addr_mask;
@@ -5473,19 +5475,46 @@ static void hugetlb_unmap_range(struct mmu_gather *tlb,
make_pte_marker(PTE_MARKER_UFFD_WP),
sz);
hugetlb_count_sub(pages_per_huge_page(h), mm);
- hugetlb_remove_rmap(folio);
+ /*
+ * hugetlb_wp() uses mapcount to decide whether an anonymous
+ * folio can be reused. Delay rmap removal until stale TLB
+ * entries are gone, unless the mm has no users left.
+ */
+ if (folio_test_anon(folio) && !tlb->fullmm) {
+ delay_rmap = tlb_delay_rmap(tlb);
+ /*
+ * Without delay-rmap support, just flush before
+ * dropping the PTL. Except with MMU_GATHER_NO_GATHER
+ * where clearing already flushed the TLB.
+ */
+ if (!delay_rmap && !IS_ENABLED(CONFIG_MMU_GATHER_NO_GATHER))
+ flush_hugetlb_tlb_range(vma, address, address + sz);
+ }
+ if (!delay_rmap)
+ hugetlb_remove_rmap(folio);
spin_unlock(ptl);
- hugetlb_folio_restore_rsv_on_unmap(vma, folio);
+ if (!delay_rmap)
+ hugetlb_folio_restore_rsv_on_unmap(vma, folio);
- tlb_remove_page_size(tlb, folio_page(folio, 0),
- folio_size(folio));
+ if (__tlb_remove_page_size(tlb, folio_page(folio, 0), sz,
+ delay_rmap)) {
+ tlb_flush_mmu_tlbonly(tlb);
+ tlb_flush_rmaps(tlb, vma);
+ tlb_flush_mmu(tlb);
+ }
/*
* If we were instructed to unmap a specific folio, we're done.
*/
if (folio_provided)
break;
}
+
+ if (tlb->delayed_rmap) {
+ tlb_flush_mmu_tlbonly(tlb);
+ tlb_flush_rmaps(tlb, vma);
+ tlb_flush_mmu(tlb);
+ }
}
void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
diff --git a/mm/internal.h b/mm/internal.h
index e3f92883bf238..2f1a7b9b2c055 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -614,6 +614,15 @@ void zap_vma_range_batched(struct mmu_gather *tlb,
struct vm_area_struct *vma, unsigned long addr,
unsigned long size, struct zap_details *details);
int zap_vma_for_reaping(struct vm_area_struct *vma);
+#ifdef CONFIG_HUGETLB_PAGE
+void hugetlb_folio_restore_rsv_on_unmap(struct vm_area_struct *vma,
+ struct folio *folio);
+#else
+static inline void hugetlb_folio_restore_rsv_on_unmap(struct vm_area_struct *vma,
+ struct folio *folio)
+{
+}
+#endif
int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio,
gfp_t gfp);
diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c
index 9f353f0e2ef4d..8728e1764fc5a 100644
--- a/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -15,6 +15,8 @@
#include <asm/tlb.h>
+#include "internal.h"
+
#ifndef CONFIG_MMU_GATHER_NO_GATHER
static bool tlb_next_batch(struct mmu_gather *tlb)
@@ -59,14 +61,19 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_
if (encoded_page_flags(enc) & ENCODED_PAGE_BIT_DELAY_RMAP) {
struct page *page = encoded_page_ptr(enc);
+ struct folio *folio = page_folio(page);
unsigned int nr_pages = 1;
if (unlikely(encoded_page_flags(enc) &
ENCODED_PAGE_BIT_NR_PAGES_NEXT))
nr_pages = encoded_nr_pages(pages[++i]);
- folio_remove_rmap_ptes(page_folio(page), page, nr_pages,
- vma);
+ if (folio_test_hugetlb(folio)) {
+ hugetlb_remove_rmap(folio);
+ hugetlb_folio_restore_rsv_on_unmap(vma, folio);
+ } else {
+ folio_remove_rmap_ptes(folio, page, nr_pages, vma);
+ }
}
}
}
@@ -211,9 +218,10 @@ bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page,
PAGE_SIZE);
}
-bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size)
+bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
+ int page_size, bool delay_rmap)
{
- return __tlb_remove_folio_pages_size(tlb, page, 1, false, page_size);
+ return __tlb_remove_folio_pages_size(tlb, page, 1, delay_rmap, page_size);
}
#endif /* MMU_GATHER_NO_GATHER */
--
2.56.1
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-11-16 18:32 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 18:29 [PATCH v2 00/15] mm/hugetlb: various fixes David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 01/15] hugetlbfs: fix integer overflow in fallocate() David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 02/15] mm/hugetlb: compare against persistent count when freeing lowmem folios David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 03/15] mm/hugetlb: preserve reservations when shrinking on HIGHMEM David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 04/15] mm/hugetlb: account temporary folios separately David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 05/15] mm/hugetlb: keep surplus folios when the pool shrinks David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 06/15] mm/hugetlb: fix mremap() racing with page migration David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 07/15] hugetlbfs: fix unmapping folios from private VMAs David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 08/15] mm/hugetlb: serialize page-cache allocations with truncation David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 09/15] mm/memfd: recheck the page cache before allocating a hugetlb folio David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 10/15] mm/hugetlb: count again if the reserve map shrank David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 11/15] hugetlbfs: do not grow a sealed file in mmap() David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 12/15] mm/hugetlb: keep surplus folios that are reserved David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 13/15] mm/hugetlb: fix reservation leak on min_size= mounts David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 14/15] mm/hugetlb: account restored reservations before freeing folios David Hildenbrand (Arm)
2026-11-16 18:29 ` [PATCH v2 15/15] mm/hugetlb: delay anon folio rmap removal until after the TLB flush David Hildenbrand (Arm)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®