[PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Mikhail Gavrilov]

On a box with a discrete GPU, lockdep reports a possible deadlock as soon
as kswapd shrinks the TTM page pool:

  WARNING: possible circular locking dependency detected
  7.3.0-rc3-f6e7b42bf05b+ #183 Tainted: G     U
  ------------------------------------------------------
  kswapd0/269 is trying to acquire lock:
  ((init_mm).mmap_lock){++++}-{4:4}, at: change_page_attr_set_clr+0x29a/0x4a0
  but task is already holding lock:
  (pool_shrink_rwsem){.+.+}-{4:4}, at: ttm_pool_shrink+0xb2/0x330 [ttm]
  Chain exists of:
    (init_mm).mmap_lock --> fs_reclaim --> pool_shrink_rwsem

The cycle is built from three edges:

  1) pool_shrink_rwsem -> (init_mm).mmap_lock

     The TTM shrinker restores the caching attribute of every page it
     frees, while holding pool_shrink_rwsem:

       ttm_pool_shrink()
       -> ttm_pool_dispose_list()
       -> ttm_pool_free_page()
       -> set_pages_wb()
       -> change_page_attr_set_clr()   [ init_mm mmap read lock ]

  2) fs_reclaim -> pool_shrink_rwsem

     The same shrinker, called from reclaim.

  3) (init_mm).mmap_lock -> fs_reclaim

     ioremap() installing a huge PUD mapping over an existing PMD table:

       ioremap_page_range()
       -> vmap_range_noflush()
       -> vmap_try_huge_pud()          [ init_mm mmap read lock ]
       -> pud_free_pmd_page()
       -> __get_free_page(GFP_KERNEL)  [ enters reclaim ]

Edge 3 is the one that should not exist.  Now that reclaim can acquire the
init_mm mmap lock, that lock must not be held over an allocation which can
enter reclaim.  This rule is stated by commit d5d8b8662e6e
("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
and honoured inside CPA itself, where split_large_page() drops the lock
around pagetable_alloc().  The vmap path took the same lock earlier, in
commit 26444eb71465
("mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF"),
and pud_free_pmd_page() still allocates its scratch page with GFP_KERNEL
underneath it.  Neither commit deadlocks on its own; together they close
the cycle.

Use GFP_NOWAIT for that page.  pud_free_pmd_page() already returns 0 when
the allocation fails, and its only caller, vmap_try_huge_pud(), then maps
at PMD granularity through the existing page table - exactly what it does
when its own mmap trylock fails.  So the failure path is not new, and a
failed allocation costs nothing but a smaller mapping.

This breaks the cycle at its source: no code holds the init_mm mmap lock
across a reclaiming allocation any more, so no shrinker-held lock can be
ordered against it.  The same cycle was reported from the i915 shrinker
with &vm->mutex in place of pool_shrink_rwsem.

Link: https://lore.kernel.org/all/80993b70-352f-4069-84c7-39a04c061e98@intel.com/
Fixes: d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
Cc: stable@vger.kernel.org
Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
---

#regzbot introduced: d5d8b8662e6e

Reproduced and verified on a Ryzen 9 7950X with a Radeon RX 7900 XTX
(Navi 31, 0000:03:00.0), lockdep and KASAN enabled.

On a kernel with a TTM driver bound the report above reproduces on
demand, no memory pressure needed:

  # cat /sys/kernel/debug/ttm/page_pool         # wc/uc rows non-zero
  # cat /sys/kernel/debug/ttm/page_pool_shrink

The second read runs the TTM shrinker with fs_reclaim held, so the
same cycle is reported from the reading task instead of kswapd.

Before (7.3-rc3-f6e7b42bf05b, #183): report within 105 s of boot,
2048 pool pages scanned.

After (same base plus this patch, #185): 1536 write-combined pages
scanned - the order-9 row of the pool went from 3 to 0 and total
node0 from 27650 to 26114 - no report, and /proc/lockdep_stats still
showed debug_locks: 1 afterwards.

 arch/x86/mm/pgtable.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index cb03f5a2b243..c79587962526 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -713,7 +713,7 @@ int pmd_clear_huge(pmd_t *pmd)
  * Context: The PUD range has been unmapped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  *
- * NOTE: Callers must allow a single page allocation.
+ * NOTE: Callers must allow a single non-blocking page allocation.
  */
 int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
@@ -722,7 +722,13 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 	int i;
 
 	pmd = pud_pgtable(*pud);
-	pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL);
+	/*
+	 * The only caller, vmap_try_huge_pud(), holds the init_mm mmap read
+	 * lock, which reclaim can take via set_memory_*().  Do not enter
+	 * reclaim from here.  Failing is fine: the caller then keeps the
+	 * existing PMD table instead of installing a huge PUD mapping.
+	 */
+	pmd_sv = (pmd_t *)__get_free_page(GFP_NOWAIT);
 	if (!pmd_sv)
 		return 0;
 
-- 
2.55.0

Re: [PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Mikhail Gavrilov]

Hi,

The lock cycle this patch breaks has meanwhile reached released stable
kernels: d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on
attribute changes to avoid UAF") went out in 7.2.7 and 6.18.53 on
Sep 21. Both trees already carry 26444eb71465 ("mm/vmalloc: acquire
init_mm lock on huge vmap to avoid ptdump UAF"), and pud_free_pmd_page()
there still allocates with GFP_KERNEL under the init_mm lock, so the
cycle quoted in the patch exists there too, not only in 7.3-rc.

The patch still applies cleanly to current mainline and next-20260922,
and the reproducer below the scissors of the original mail works
unchanged.

Dave, could this go in through x86/urgent for 7.3? Lorenzo, since the
rule it applies comes from your series, a review from you would help.
If either of you would rather fix it differently, I am happy to respin.

-- 
Thanks,
Mikhail

Re: [PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Pedro Falcato]

On Wed, Sep 16, 2026 at 11:22:22AM +0500, Mikhail Gavrilov wrote:
> On a box with a discrete GPU, lockdep reports a possible deadlock as soon
> as kswapd shrinks the TTM page pool:
> 
>   WARNING: possible circular locking dependency detected
>   7.3.0-rc3-f6e7b42bf05b+ #183 Tainted: G     U
>   ------------------------------------------------------
>   kswapd0/269 is trying to acquire lock:
>   ((init_mm).mmap_lock){++++}-{4:4}, at: change_page_attr_set_clr+0x29a/0x4a0
>   but task is already holding lock:
>   (pool_shrink_rwsem){.+.+}-{4:4}, at: ttm_pool_shrink+0xb2/0x330 [ttm]
>   Chain exists of:
>     (init_mm).mmap_lock --> fs_reclaim --> pool_shrink_rwsem
> 
> The cycle is built from three edges:
> 
>   1) pool_shrink_rwsem -> (init_mm).mmap_lock
> 
>      The TTM shrinker restores the caching attribute of every page it
>      frees, while holding pool_shrink_rwsem:
> 
>        ttm_pool_shrink()
>        -> ttm_pool_dispose_list()
>        -> ttm_pool_free_page()
>        -> set_pages_wb()
>        -> change_page_attr_set_clr()   [ init_mm mmap read lock ]
> 
>   2) fs_reclaim -> pool_shrink_rwsem
> 
>      The same shrinker, called from reclaim.
> 
>   3) (init_mm).mmap_lock -> fs_reclaim
> 
>      ioremap() installing a huge PUD mapping over an existing PMD table:
> 
>        ioremap_page_range()
>        -> vmap_range_noflush()
>        -> vmap_try_huge_pud()          [ init_mm mmap read lock ]
>        -> pud_free_pmd_page()
>        -> __get_free_page(GFP_KERNEL)  [ enters reclaim ]
> 
> Edge 3 is the one that should not exist.  Now that reclaim can acquire the
> init_mm mmap lock, that lock must not be held over an allocation which can
> enter reclaim.  This rule is stated by commit d5d8b8662e6e
> ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
> and honoured inside CPA itself, where split_large_page() drops the lock
> around pagetable_alloc().  The vmap path took the same lock earlier, in
> commit 26444eb71465
> ("mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF"),
> and pud_free_pmd_page() still allocates its scratch page with GFP_KERNEL
> underneath it.  Neither commit deadlocks on its own; together they close
> the cycle.
> 
> Use GFP_NOWAIT for that page.  pud_free_pmd_page() already returns 0 when
> the allocation fails, and its only caller, vmap_try_huge_pud(), then maps
> at PMD granularity through the existing page table - exactly what it does
> when its own mmap trylock fails.  So the failure path is not new, and a
> failed allocation costs nothing but a smaller mapping.
> 
> This breaks the cycle at its source: no code holds the init_mm mmap lock
> across a reclaiming allocation any more, so no shrinker-held lock can be
> ordered against it.  The same cycle was reported from the i915 shrinker
> with &vm->mutex in place of pool_shrink_rwsem.
> 
> Link: https://lore.kernel.org/all/80993b70-352f-4069-84c7-39a04c061e98@intel.com/
> Fixes: d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
> Cc: stable@vger.kernel.org
> Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
> ---
> 
> #regzbot introduced: d5d8b8662e6e
> 
> Reproduced and verified on a Ryzen 9 7950X with a Radeon RX 7900 XTX
> (Navi 31, 0000:03:00.0), lockdep and KASAN enabled.
> 
> On a kernel with a TTM driver bound the report above reproduces on
> demand, no memory pressure needed:
> 
>   # cat /sys/kernel/debug/ttm/page_pool         # wc/uc rows non-zero
>   # cat /sys/kernel/debug/ttm/page_pool_shrink
> 
> The second read runs the TTM shrinker with fs_reclaim held, so the
> same cycle is reported from the reading task instead of kswapd.
> 
> Before (7.3-rc3-f6e7b42bf05b, #183): report within 105 s of boot,
> 2048 pool pages scanned.
> 
> After (same base plus this patch, #185): 1536 write-combined pages
> scanned - the order-9 row of the pool went from 3 to 0 and total
> node0 from 27650 to 26114 - no report, and /proc/lockdep_stats still
> showed debug_locks: 1 afterwards.
> 
>  arch/x86/mm/pgtable.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> index cb03f5a2b243..c79587962526 100644
> --- a/arch/x86/mm/pgtable.c
> +++ b/arch/x86/mm/pgtable.c
> @@ -713,7 +713,7 @@ int pmd_clear_huge(pmd_t *pmd)
>   * Context: The PUD range has been unmapped and TLB purged.
>   * Return: 1 if clearing the entry succeeded. 0 otherwise.
>   *
> - * NOTE: Callers must allow a single page allocation.
> + * NOTE: Callers must allow a single non-blocking page allocation.
>   */
>  int pud_free_pmd_page(pud_t *pud, unsigned long addr)
>  {
> @@ -722,7 +722,13 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)
>  	int i;
>  
>  	pmd = pud_pgtable(*pud);
> -	pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL);
> +	/*
> +	 * The only caller, vmap_try_huge_pud(), holds the init_mm mmap read
> +	 * lock, which reclaim can take via set_memory_*().  Do not enter
> +	 * reclaim from here.  Failing is fine: the caller then keeps the
> +	 * existing PMD table instead of installing a huge PUD mapping.
> +	 */
> +	pmd_sv = (pmd_t *)__get_free_page(GFP_NOWAIT);
>  	if (!pmd_sv)
>  		return 0;

So, I'm really confuesd about this code. As in the original code. Copy-pasting
here:

        pmd_t *pmd, *pmd_sv;
        struct ptdesc *pt;
        int i;

        pmd = pud_pgtable(*pud);
        pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL);
        if (!pmd_sv)
                return 0;

We allocate a copy...

        for (i = 0; i < PTRS_PER_PMD; i++) {
                pmd_sv[i] = pmd[i];
                if (!pmd_none(pmd[i]))
                        pmd_clear(&pmd[i]);
        }

We, for some reason, save and clear the pmd.
        
        pud_clear(pud);

Then we clear the PUD entry...

        /* INVLPG to clear all paging-structure caches */
        flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);

Then we flush the TLB (and with it, translation caches).

        for (i = 0; i < PTRS_PER_PMD; i++) {
                if (!pmd_none(pmd_sv[i])) {
                        pt = page_ptdesc(pmd_page(pmd_sv[i]));
                        pagetable_dtor_free(pt);
                }
        }

        free_page((unsigned long)pmd_sv);

        pmd_free(&init_mm, pmd);

Then we free possible PTEs attached to the PMD (from our copy),
and free the PMD itself.


So, the question is: why the heck do we need a copy? PMD is still allocated
by the time we flush the TLB. Why doesn't a simple pud_clear() + flush_tlb +
free over the pmd Just Work? Am I missing something? The git log isn't
clueing me in.


All-in-all, I would much prefer not having a copy of the PMD at all. Perhaps,
if this isn't workable, then a linked list of PTEs would work. But I would rather
not have tricky logic at all.

-- 
Pedro

Re: [PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Mikhail Gavrilov]

On Wed, Sep 23, 2026 at 07:38:55PM +0100, Pedro Falcato wrote:
> So, the question is: why the heck do we need a copy? PMD is still allocated
> by the time we flush the TLB. Why doesn't a simple pud_clear() + flush_tlb +
> free over the pmd Just Work? Am I missing something? The git log isn't
> clueing me in.

I don't see anything you are missing.  The copy came with 5e0fb5df2ee8
("x86/mm: Add TLB purge to free pmd/pte page interfaces").  Its changelog
explains the flush but not the copy, and the only discussion of the copy
in that thread was Joerg objecting to the allocation and suggesting a
list_head on the stack instead:

  https://lore.kernel.org/all/20180529144438.GM18595@8bytes.org/

The existing code already frees the PMD table itself after pud_clear()
and that flush, so it already relies on the table being out of reach of
the page walker at that point.  If it is safe to free it then, it is
safe to read it then; clearing the PMD entries up front buys nothing,
because nothing is freed before the flush.  Nobody else writes to the
table either: vmap_try_huge_pud() only gets here for a range covering
the whole PUD, and ptdump is kept out by the init_mm lock the caller
holds.

> All-in-all, I would much prefer not having a copy of the PMD at all. Perhaps,
> if this isn't workable, then a linked list of PTEs would work. But I would rather
> not have tricky logic at all.

Agreed.  It also removes the allocation instead of weakening it, so no
fallback path is left behind.  I'll send a v2 that does pud_clear(), the
flush, and then frees the PTE tables straight from the detached PMD
table - the same order pmd_free_pte_page() already uses one level down.

Thanks for looking at it.

-- 
Thanks,
Mikhail

Re: [PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]

On 9/23/26 11:38, Pedro Falcato wrote:
> So, the question is: why the heck do we need a copy? PMD is still allocated
> by the time we flush the TLB. Why doesn't a simple pud_clear() + flush_tlb +
> free over the pmd Just Work? Am I missing something? The git log isn't
> clueing me in.

Yeah, the original changelog in here:

	commit 5e0fb5df2ee871b841f96f9cb6a7f2784e96aa4e
	Author: Toshi Kani <toshi.kani@hpe.com>
	Date:   Wed Jun 27 08:13:48 2018 -0600

	    x86/mm: Add TLB purge to free pmd/pte page interfaces

is a bit vague about what it is doing. I think there might have been
some confusion around speculation:

	speculation may cache pud/pmd entries (paging-structure
	caches) when they have P-bit set

Because, as far as I know, you can't have establish an entry in the TLB,
period, if Accessed==0. I think the changelog imagines a world where the
CPU is establishing TLB entries from Present=1,Accessed=0 PTEs. That
just doesn't happen.

There are only two ways the CPU can find a page table entry to cache in
the TLB and start setting Accessed bits:

 1. It walks down from CR3 and finds the entry
 2. It starts from a mid-level cache and finds the entry

This takes care of a walk from CR3 (#1):

	pud_clear(pud);

and this takes care of the mid-level caches (#2):

	/* INVLPG to clear all paging-structure caches */
	flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);

I don't think there's anything else to do. Right?

I think that means we can do something like the completely untested
attached patch.

Looks like Mikhail came to basically the same conclusion.

[-- Attachment #2: pud_free_pmd_page-simplify.patch --]
[-- Type: text/x-patch, Size: 1831 bytes --]



---

 b/arch/x86/mm/pgtable.c |   30 ++++++++++--------------------
 1 file changed, 10 insertions(+), 20 deletions(-)

diff -puN arch/x86/mm/pgtable.c~pud_free_pmd_page-simplify arch/x86/mm/pgtable.c
--- a/arch/x86/mm/pgtable.c~pud_free_pmd_page-simplify	2026-09-23 15:30:32.035610475 -0700
+++ b/arch/x86/mm/pgtable.c	2026-09-23 15:36:56.663747586 -0700
@@ -1,4 +1,4 @@
-// SPDX-License-Identifier: GPL-2.0
+pud_free_pmd_page-simplify// SPDX-License-Identifier: GPL-2.0
 #include <linux/mm.h>
 #include <linux/gfp.h>
 #include <linux/hugetlb.h>
@@ -711,44 +711,34 @@ int pmd_clear_huge(pmd_t *pmd)
  * @addr: Virtual address associated with PUD
  *
  * Context: The PUD range has been unmapped and TLB purged.
- * Return: 1 if clearing the entry succeeded. 0 otherwise.
  *
  * NOTE: Callers must allow a single page allocation.
  */
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+void pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
-	pmd_t *pmd, *pmd_sv;
 	struct ptdesc *pt;
+	pmd_t *pmd;
 	int i;
 
 	pmd = pud_pgtable(*pud);
-	pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL);
-	if (!pmd_sv)
-		return 0;
-
-	for (i = 0; i < PTRS_PER_PMD; i++) {
-		pmd_sv[i] = pmd[i];
-		if (!pmd_none(pmd[i]))
-			pmd_clear(&pmd[i]);
-	}
 
 	pud_clear(pud);
 
-	/* INVLPG to clear all paging-structure caches */
+	/*
+	 * 'pmd' and all its descendents are unreachable
+	 * via normal page walks. Make them unreachable
+	 * in cached mid-level walks too:
+	 */
 	flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);
 
 	for (i = 0; i < PTRS_PER_PMD; i++) {
-		if (!pmd_none(pmd_sv[i])) {
-			pt = page_ptdesc(pmd_page(pmd_sv[i]));
+		if (!pmd_none(pmd[i])) {
+			pt = page_ptdesc(pmd_page(pmd[i]));
 			pagetable_dtor_free(pt);
 		}
 	}
 
-	free_page((unsigned long)pmd_sv);
-
 	pmd_free(&init_mm, pmd);
-
-	return 1;
 }
 
 /**
_

Re: [PATCH] x86/mm: avoid a reclaiming allocation in pud_free_pmd_page()

[Mikhail Gavrilov]

On 9/23/26 15:53, Dave Hansen wrote:
> I think that means we can do something like the completely untested
> attached patch.
>
> Looks like Mikhail came to basically the same conclusion.

Yes - v2 does the same thing, and also drops the now-stale NOTE about
the page allocation.  It went out about 20 minutes before your mail:

  https://lore.kernel.org/all/20260923223116.20090-1-mikhail.v.gavrilov@gmail.com/

It was tested on a box that reaches this path at boot without any
instrumentation (from amdgpu_ttm_init()), plus the TTM shrinker
reproducer; the details are below the scissors there.

Two things in the attached sketch, in case it gets picked up as is:

 - The first hunk prepends the patch name to the SPDX line
   ("pud_free_pmd_page-simplify// SPDX-License-Identifier: ..."),
   which will not build.

 - Making it return void conflicts with the prototype in
   include/linux/pgtable.h, and vmap_try_huge_pud() still does
   "if (!pud_free_pmd_page(pud, addr))".  The arm64, riscv and powerpc
   versions also always return 1, so dropping the return value looks
   like a reasonable follow-up across architectures, but I kept v2
   x86-only and returning int so that it stays small for stable.

Your comment above the flush explains it better than the "INVLPG to
clear all paging-structure caches" one-liner.  If you prefer it, I can
send a v3 with it, or feel free to fold it in when applying.