[PATCH v2 0/3] KVM: arm64: Fix the hyp pool size on large machines, plus two hardening patches

[PATCH v2 0/3] KVM: arm64: Fix the hyp pool size on large machines, plus two hardening patches

[Xuheng Li via B4 Relay]

Three changes for issues we found in pKVM as part of our research
project.

Patch 1 fixes 32-bit size computations in hyp_early_alloc_contig()
and hyp_pool_init().  The hyp page-table and host stage-2 regions reach
4 GiB on machines with about 2 TiB of RAM; the early allocator then
carves them too small, so the two pools overlap and can hand out the
same page twice.

Patch 2 checks the host stage-2 PGD allocation for NULL before using
it.  With patch 1 applied that allocation cannot fail, so this only
turns a mis-sized pool into a clean -ENOMEM instead of a hyp panic.

Patch 3 makes hyp_map_walker() refuse to replace a valid block mapping
with a table, which would skip break-before-make and unmap the rest of
the block.  Nothing in mainline maps into an existing hyp block, so this
is hardening; Android's interface for EL2 modules to map pages can reach
it.

Signed-off-by: Xuheng Li <xuheng@cs.columbia.edu>
---
Changes in v2:
- Patch 1: also fix the same overflow in hyp_early_alloc_contig(), which
  makes hpool and host_s2_pool overlap (reported by the Sashiko review of
  v1); describe that in the commit message and add its Fixes: tag.
- Link to v1: https://lore.kernel.org/r/20261105-hyp-pool-and-map-v1-0-756bb5d8e067@cs.columbia.edu

---
Xuheng Li (3):
      KVM: arm64: Compute hyp early-allocation and pool sizes in 64 bits
      KVM: arm64: Check the host stage-2 PGD allocation for NULL
      KVM: arm64: Don't replace a valid hyp block mapping with a table

 arch/arm64/kvm/hyp/nvhe/early_alloc.c | 2 +-
 arch/arm64/kvm/hyp/nvhe/mem_protect.c | 3 +++
 arch/arm64/kvm/hyp/nvhe/page_alloc.c  | 6 +++---
 arch/arm64/kvm/hyp/pgtable.c          | 7 +++++++
 4 files changed, 14 insertions(+), 4 deletions(-)
---
base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
change-id: 20261105-hyp-pool-and-map-b5e7be9a1191

Best regards,
-- 
Xuheng Li <xuheng@cs.columbia.edu>

[PATCH v2 1/3] KVM: arm64: Compute hyp early-allocation and pool sizes in 64 bits

[Xuheng Li via B4 Relay]

From: Xuheng Li <xuheng@cs.columbia.edu>

hyp_early_alloc_contig() and hyp_pool_init() both take a page count as
an unsigned int and compute the size in bytes as nr_pages << PAGE_SHIFT
in 32 bits.  For 4 GiB or more (nr_pages >= 2^20 with 4 KiB pages) the
shift overflows.

Two of the regions that pKVM carves out at boot are sized from the
amount of memory: hyp_s1_pgtable_pages() for the hyp page tables (hpool)
and host_s2_pgtable_pages() for the host stage-2 pool both provision
page-granular tables for all of memory, about one page per 2 MiB of
RAM, so they reach 4 GiB on a machine with about 2 TiB of RAM.  Arm
systems of that size exist, for example AWS Graviton4 x8g.metal-48xl
instances (3 TiB) and Ampere Altra Max servers (up to 4 TiB per
socket).

On such a machine, divide_memory_pool() gets a truncated hyp page-table
region from hyp_early_alloc_contig(), so the host stage-2 region and the
FF-A proxy pages that follow it start inside the range that hpool will
own.  hyp_pool_init() then attaches all nr_pages pages of each pool to
its free lists, so hpool and host_s2_pool share pages and the same page
can be handed out twice, for example as a hyp stage-1 table and as a
host stage-2 table.  hyp_pool_init() also derives range_end and
max_order from the truncated size, so most of each pool is treated as
external order-0 pages; on a host whose IPA size makes the host stage-2
PGD a multi-page allocation (for example a 42-bit PARange), that
allocation fails and host_s2_zalloc_pages_exact() uses the NULL result.

Compute both sizes in 64 bits.

Fixes: e75960408723 ("KVM: arm64: Introduce an early Hyp page allocator")
Fixes: 8e17c66249e9 ("KVM: arm64: Introduce a Hyp buddy page allocator")
Cc: stable@vger.kernel.org
Signed-off-by: Xuheng Li <xuheng@cs.columbia.edu>
---
 arch/arm64/kvm/hyp/nvhe/early_alloc.c | 2 +-
 arch/arm64/kvm/hyp/nvhe/page_alloc.c  | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/early_alloc.c b/arch/arm64/kvm/hyp/nvhe/early_alloc.c
index 00de04153..f435f89be 100644
--- a/arch/arm64/kvm/hyp/nvhe/early_alloc.c
+++ b/arch/arm64/kvm/hyp/nvhe/early_alloc.c
@@ -23,7 +23,7 @@ unsigned long hyp_early_alloc_nr_used_pages(void)
 
 void *hyp_early_alloc_contig(unsigned int nr_pages)
 {
-	unsigned long size = (nr_pages << PAGE_SHIFT);
+	unsigned long size = (unsigned long)nr_pages << PAGE_SHIFT;
 	void *ret = (void *)cur;
 
 	if (!nr_pages)
diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c
index 57f86aa0f..e7c10d2f7 100644
--- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c
+++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c
@@ -233,16 +233,16 @@ int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages,
 		  unsigned int reserved_pages)
 {
 	phys_addr_t phys = hyp_pfn_to_phys(pfn);
+	u64 size = (u64)nr_pages << PAGE_SHIFT;
 	struct hyp_page *p;
 	int i;
 
 	hyp_spin_lock_init(&pool->lock);
-	pool->max_order = min(MAX_PAGE_ORDER,
-			      get_order(nr_pages << PAGE_SHIFT));
+	pool->max_order = min(MAX_PAGE_ORDER, get_order(size));
 	for (i = 0; i <= pool->max_order; i++)
 		INIT_LIST_HEAD(&pool->free_area[i]);
 	pool->range_start = phys;
-	pool->range_end = phys + (nr_pages << PAGE_SHIFT);
+	pool->range_end = phys + size;
 
 	/* Init the vmemmap portion */
 	p = hyp_phys_to_page(phys);

-- 
2.53.0

Re: [PATCH v2 1/3] KVM: arm64: Compute hyp early-allocation and pool sizes in 64 bits

[Fuad Tabba]

Hi Xuheng,

On Thu, 05 Nov 2026 22:59:48 +0000, Xuheng Li <xuheng@cs.columbia.edu> wrote:
> hyp_early_alloc_contig() and hyp_pool_init() both take a page count as
> an unsigned int and compute the size in bytes as nr_pages << PAGE_SHIFT
> in 32 bits.  For 4 GiB or more (nr_pages >= 2^20 with 4 KiB pages) the
> shift overflows.

Mostafa (Cc'd) posted a fix for the same truncation in July [1],
widening the parameters to unsigned long instead. Would you be OK
picking up his patch for v3, keeping him as the author and your
description and tags?

> Two of the regions that pKVM carves out at boot are sized from the
> amount of memory: hyp_s1_pgtable_pages() for the hyp page tables (hpool)
> and host_s2_pgtable_pages() for the host stage-2 pool both provision
> page-granular tables for all of memory, about one page per 2 MiB of
> RAM, so they reach 4 GiB on a machine with about 2 TiB of RAM.  Arm
> systems of that size exist, for example AWS Graviton4 x8g.metal-48xl
> instances (3 TiB) and Ampere Altra Max servers (up to 4 TiB per
> socket).

Could you also list the hyp vmemmap here? It's sized the same way and
overflows at 2 TiB too.

> On such a machine, divide_memory_pool() gets a truncated hyp page-table
> region from hyp_early_alloc_contig(), so the host stage-2 region and the
> FF-A proxy pages that follow it start inside the range that hpool will
> own.  hyp_pool_init() then attaches all nr_pages pages of each pool to
> its free lists, so hpool and host_s2_pool share pages and the same page
> can be handed out twice, for example as a hyp stage-1 table and as a
> host stage-2 table.  hyp_pool_init() also derives range_end and
> max_order from the truncated size, so most of each pool is treated as
> external order-0 pages; on a host whose IPA size makes the host stage-2
> PGD a multi-page allocation (for example a 42-bit PARange), that
> allocation fails and host_s2_zalloc_pages_exact() uses the NULL result.

Could you describe here what actually happens? I think pKVM init fails
before any page is handed out twice. Just below 2 TiB, hyp_pool_init()
for host_s2_pool hits the BUG_ON() in hyp_set_page_refcounted() on
hpool's page-table pages, and from 2 TiB up hyp_back_vmemmap() zeroes
the new hyp page tables while they're being built.

Cheers,
/fuad

[1] https://lore.kernel.org/all/20260715150125.2798116-1-smostafa@google.com/

Re: [PATCH v2 1/3] KVM: arm64: Compute hyp early-allocation and pool sizes in 64 bits

[Mostafa Saleh]

On Fri, Nov 06, 2026 at 11:31:57AM +0000, Fuad Tabba wrote:
> Hi Xuheng,
> 
> On Thu, 05 Nov 2026 22:59:48 +0000, Xuheng Li <xuheng@cs.columbia.edu> wrote:
> > hyp_early_alloc_contig() and hyp_pool_init() both take a page count as
> > an unsigned int and compute the size in bytes as nr_pages << PAGE_SHIFT
> > in 32 bits.  For 4 GiB or more (nr_pages >= 2^20 with 4 KiB pages) the
> > shift overflows.
> 
> Mostafa (Cc'd) posted a fix for the same truncation in July [1],
> widening the parameters to unsigned long instead. Would you be OK
> picking up his patch for v3, keeping him as the author and your
> description and tags?
> 

I was preparing v3 anyway, I can post it today.

Thanks,
Mostafa

> > Two of the regions that pKVM carves out at boot are sized from the
> > amount of memory: hyp_s1_pgtable_pages() for the hyp page tables (hpool)
> > and host_s2_pgtable_pages() for the host stage-2 pool both provision
> > page-granular tables for all of memory, about one page per 2 MiB of
> > RAM, so they reach 4 GiB on a machine with about 2 TiB of RAM.  Arm
> > systems of that size exist, for example AWS Graviton4 x8g.metal-48xl
> > instances (3 TiB) and Ampere Altra Max servers (up to 4 TiB per
> > socket).
> 
> Could you also list the hyp vmemmap here? It's sized the same way and
> overflows at 2 TiB too.
> 
> > On such a machine, divide_memory_pool() gets a truncated hyp page-table
> > region from hyp_early_alloc_contig(), so the host stage-2 region and the
> > FF-A proxy pages that follow it start inside the range that hpool will
> > own.  hyp_pool_init() then attaches all nr_pages pages of each pool to
> > its free lists, so hpool and host_s2_pool share pages and the same page
> > can be handed out twice, for example as a hyp stage-1 table and as a
> > host stage-2 table.  hyp_pool_init() also derives range_end and
> > max_order from the truncated size, so most of each pool is treated as
> > external order-0 pages; on a host whose IPA size makes the host stage-2
> > PGD a multi-page allocation (for example a 42-bit PARange), that
> > allocation fails and host_s2_zalloc_pages_exact() uses the NULL result.
> 
> Could you describe here what actually happens? I think pKVM init fails
> before any page is handed out twice. Just below 2 TiB, hyp_pool_init()
> for host_s2_pool hits the BUG_ON() in hyp_set_page_refcounted() on
> hpool's page-table pages, and from 2 TiB up hyp_back_vmemmap() zeroes
> the new hyp page tables while they're being built.
> 
> Cheers,
> /fuad
> 
> [1] https://lore.kernel.org/all/20260715150125.2798116-1-smostafa@google.com/

Re: [PATCH v2 1/3] KVM: arm64: Compute hyp early-allocation and pool sizes in 64 bits

[Xuheng Li]

Hi Mostafa, Fuad,

On Fri, Nov 06, 2026 at 02:02:40PM +0000, Mostafa Saleh wrote:
> I was preparing v3 anyway, I can post it today.

Thanks Mostafa, and thanks Fuad for the pointer and the correction.
I'll drop my patch 1 in favour of Mostafa's updated patch. Fuad's
description of what happens (pKVM init failing at the BUG_ON() in
hyp_set_page_refcounted() just below 2 TiB, and in hyp_back_vmemmap()
from 2 TiB up) is more accurate than mine, and the hyp vmemmap
overflows the same way. Mostafa, feel free to reuse any of my commit
message.

Thanks,
Xuheng

[PATCH v2 2/3] KVM: arm64: Check the host stage-2 PGD allocation for NULL

[Xuheng Li via B4 Relay]

From: Xuheng Li <xuheng@cs.columbia.edu>

host_s2_zalloc_pages_exact() passes the result of hyp_alloc_pages() to
hyp_split_page(hyp_virt_to_page(addr)) without checking it for NULL.
If the host stage-2 pool cannot supply the PGD, hyp_virt_to_page(NULL)
yields a bogus struct hyp_page, and hyp_split_page() reads and writes
through it, which ends in a hyp panic during pKVM initialisation
although kvm_pgtable_stage2_init() is prepared to fail with -ENOMEM.

The pool is sized so that this allocation, the first one made from it,
cannot fail, so nothing in mainline reaches this path once the pool
size is computed correctly.  Return NULL early anyway, so that a
mis-sized pool makes initialisation fail cleanly.

Signed-off-by: Xuheng Li <xuheng@cs.columbia.edu>
---
 arch/arm64/kvm/hyp/nvhe/mem_protect.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index d4c332a91..ee0848f2e 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -85,6 +85,9 @@ static void *host_s2_zalloc_pages_exact(size_t size)
 {
 	void *addr = hyp_alloc_pages(&host_s2_pool, get_order(size));
 
+	if (!addr)
+		return NULL;
+
 	hyp_split_page(hyp_virt_to_page(addr));
 
 	/*

-- 
2.53.0

Re: [PATCH v2 2/3] KVM: arm64: Check the host stage-2 PGD allocation for NULL

[Fuad Tabba]

Hi Xuheng,

On Thu, 05 Nov 2026 22:59:49 +0000, Xuheng Li <xuheng@cs.columbia.edu> wrote:
[...]
> The pool is sized so that this allocation, the first one made from it,
> cannot fail, so nothing in mainline reaches this path once the pool
> size is computed correctly.  Return NULL early anyway, so that a
> mis-sized pool makes initialisation fail cleanly.

Could you drop this patch? Once patch 1 is in nothing reaches it, and
before patch 1 pKVM init fails earlier on those machines.

The guest-side twin, guest_s2_zalloc_pages_exact(), is reachable, and
[1] fixes it by checking the PGD's alignment in __pkvm_init_vm().

Cheers,
/fuad

[1] https://lore.kernel.org/all/20261105075606.367005-2-fuad.tabba@linux.dev/

Re: [PATCH v2 2/3] KVM: arm64: Check the host stage-2 PGD allocation for NULL

[Xuheng Li]

Hi Fuad,

On Fri, Nov 06, 2026 at 11:39:13AM +0000, Fuad Tabba wrote:
> Could you drop this patch? Once patch 1 is in nothing reaches it, and
> before patch 1 pKVM init fails earlier on those machines.

Agreed, I'll drop this one. Thanks for the pointer to the guest-side
fix.

Thanks,
Xuheng

[PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Xuheng Li via B4 Relay]

From: Xuheng Li <xuheng@cs.columbia.edu>

When kvm_pgtable_hyp_map() is asked to map part of a range that a valid
block mapping already covers, hyp_map_walker_try_leaf() declines, as the
request cannot be a leaf at the block's level.  hyp_map_walker() then
allocates a zeroed table and stores it over the valid block entry.  The
walk descends into the new table and maps the requested pages, but the
rest of the block is now unmapped, and the valid block was replaced
without break-before-make.

No in-tree caller maps into an existing hyp block mapping today, so this
cannot be triggered in mainline.  Fail such a request with -EEXIST
instead, so that a future caller gets an error rather than silently
losing the rest of the block.

Signed-off-by: Xuheng Li <xuheng@cs.columbia.edu>
---
 arch/arm64/kvm/hyp/pgtable.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 9c83837b1..4c731b101 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -434,6 +434,13 @@ static int hyp_map_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	if (WARN_ON(ctx->level == KVM_PGTABLE_LAST_LEVEL))
 		return -EINVAL;
 
+	/*
+	 * Replacing a valid block with a table would need break-before-make,
+	 * and the new table would leave the rest of the block unmapped.
+	 */
+	if (WARN_ON(kvm_pte_valid(ctx->old)))
+		return -EEXIST;
+
 	childp = (kvm_pte_t *)mm_ops->zalloc_page(NULL);
 	if (!childp)
 		return -ENOMEM;

-- 
2.53.0

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Fuad Tabba]

Hi Xuheng,

On Thu, 05 Nov 2026 22:59:50 +0000, Xuheng Li <xuheng@cs.columbia.edu> wrote:
[...]
> No in-tree caller maps into an existing hyp block mapping today, so this
> cannot be triggered in mainline.  Fail such a request with -EEXIST
> instead, so that a future caller gets an error rather than silently
> losing the rest of the block.

Could you mention in the message that the WARN_ON() is fatal at EL2,
so a caller there gets a hyp panic rather than -EEXIST?
kvm_pgtable_hyp_map() is already documented as rejecting such a
mapping "with an error and a WARN()", which is a better reason than a
future caller.

Also, please don't respin series so fast. Give us time to digest the
series first :)

Cheers,
/fuad

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Vincent Donnefort]

On Fri, Nov 06, 2026 at 12:03:29PM +0000, Fuad Tabba wrote:
> Hi Xuheng,
> 
> On Thu, 05 Nov 2026 22:59:50 +0000, Xuheng Li <xuheng@cs.columbia.edu> wrote:
> [...]
> > No in-tree caller maps into an existing hyp block mapping today, so this
> > cannot be triggered in mainline.  Fail such a request with -EEXIST
> > instead, so that a future caller gets an error rather than silently
> > losing the rest of the block.
> 
> Could you mention in the message that the WARN_ON() is fatal at EL2,
> so a caller there gets a hyp panic rather than -EEXIST?
> kvm_pgtable_hyp_map() is already documented as rejecting such a
> mapping "with an error and a WARN()", which is a better reason than a
> future caller.
> 
> Also, please don't respin series so fast. Give us time to digest the
> series first :)
> 
> Cheers,
> /fuad
>

Actually, why making it fatal? All callers of kvm_pgtable_hyp_map() are checking
for the returned error.

-- 
Vincent

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Xuheng Li]

Hi Fuad,

On Fri, Nov 06, 2026 at 12:03:29PM +0000, Fuad Tabba wrote:
> Could you mention in the message that the WARN_ON() is fatal at EL2,
> so a caller there gets a hyp panic rather than -EEXIST?
> kvm_pgtable_hyp_map() is already documented as rejecting such a
> mapping "with an error and a WARN()", which is a better reason than a
> future caller.

Thanks. I'll reword the message to say that the WARN_ON() is fatal at
EL2, and point to kvm_pgtable_hyp_map()'s documented "error and a
WARN()" instead of a future caller. With patches 1 and 2 dropped, I'll
resend this one on its own once reviews have settled.

Thanks,
Xuheng

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Fuad Tabba]

On Fri, 6 Nov 2026 at 14:34, Xuheng Li <xuheng@cs.columbia.edu> wrote:
>
> Hi Fuad,
>
> On Fri, Nov 06, 2026 at 12:03:29PM +0000, Fuad Tabba wrote:
> > Could you mention in the message that the WARN_ON() is fatal at EL2,
> > so a caller there gets a hyp panic rather than -EEXIST?
> > kvm_pgtable_hyp_map() is already documented as rejecting such a
> > mapping "with an error and a WARN()", which is a better reason than a
> > future caller.
>
> Thanks. I'll reword the message to say that the WARN_ON() is fatal at
> EL2, and point to kvm_pgtable_hyp_map()'s documented "error and a
> WARN()" instead of a future caller. With patches 1 and 2 dropped, I'll
> resend this one on its own once reviews have settled.

Actually, like Vincent pointed out, why the WARN_ON(). The caller can
just handle the error.

Cheers,
/fuad

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Xuheng Li]

Hi Fuad, Vincent,

On Fri, Nov 06, 2026 at 02:34:55PM +0000, Fuad Tabba wrote:
> Actually, like Vincent pointed out, why the WARN_ON(). The caller can
> just handle the error.

Makes sense, I'll drop the WARN_ON() and just return -EEXIST.

One thing I noticed: the kernel-doc of kvm_pgtable_hyp_map() says that
a mapping over an existing one is "rejected with an error and a
WARN()", and the leaf case in hyp_map_walker_try_leaf() still warns.
Without the WARN_ON(), the comment would no longer match this case.
By the same reasoning, should the leaf case's WARN_ON() go too, in a
separate patch, with the comment reduced to "rejected with an error"?

Thanks,
Xuheng

Re: [PATCH v2 3/3] KVM: arm64: Don't replace a valid hyp block mapping with a table

[Vincent Donnefort]

On Fri, Nov 06, 2026 at 05:17:54PM +0000, Xuheng Li wrote:
> Hi Fuad, Vincent,
> 
> On Fri, Nov 06, 2026 at 02:34:55PM +0000, Fuad Tabba wrote:
> > Actually, like Vincent pointed out, why the WARN_ON(). The caller can
> > just handle the error.
> 
> Makes sense, I'll drop the WARN_ON() and just return -EEXIST.
> 
> One thing I noticed: the kernel-doc of kvm_pgtable_hyp_map() says that
> a mapping over an existing one is "rejected with an error and a
> WARN()", and the leaf case in hyp_map_walker_try_leaf() still warns.
> Without the WARN_ON(), the comment would no longer match this case.
> By the same reasoning, should the leaf case's WARN_ON() go too, in a
> separate patch, with the comment reduced to "rejected with an error"?
> 
> Thanks,
> Xuheng

You're going to make me regret my suggestion about removing the WARN_ON() :-) 

Indeed we should probably align the behaviour.

We can indeed drop the WARN_ON() from hyp_map_walker_try_leaf(). The if
(kvm_pte_valid()) you've added should catch this case anyway. 

We can keep the WARN_ON(ctx->level == KVM_PGTABLE_LAST_LEVEL). If hit, that is
an internal error...

I don't know if that warrant a different patch to be honnest... none of that is
fixing any existing bug. But no strong opinion.

-- 
Vincent