[PATCH 00/13] virtio_balloon, efi/unaccepted: support memory ballooning and free page reporting in CoCo guests

[PATCH 00/13] virtio_balloon, efi/unaccepted: support memory ballooning and free page reporting in CoCo guests

[Kevin Loughlin]

This series enables virtio-balloon free page reporting and memory balloon
inflation/deflation in Confidential Computing (CoCo) guests, starting
with architectural hooks implemented for AMD SEV variants, in order to
support dynamic CoCo guest resizing.

Today, memory ballooning and free page reporting are essentially unusable
in CoCo guests due to three main roadblocks:

1. SWIOTLB Bounce-Buffering on Free Page Reporting:
   Under VIRTIO_F_ACCESS_PLATFORM, virtqueue_add_inbuf() maps every
   scatterlist entry through the DMA API. For reporting_vq, these are
   multi-megabyte ranges of free buddy pages. The host never DMA-reads
   or writes the page contents; it only inspects the GPA to punch holes
   in backing memory. Running these ranges through SWIOTLB exhausts
   bounce buffer pools and wastes cycles copying gigabytes of
   unencrypted free memory.

2. Dynamic Memory Unacceptance & Re-acceptance:
   When the host reclaims guest physical pages via hole punching,
   the guest must transition those GPAs back to unaccepted/shared before
   the host reclaims them (rescinding PVALIDATE and issuing a GHCB Page
   State Change on SEV-SNP). Otherwise, any subsequent access triggers a
   fatal #NPF or #VC. While the EFI unaccepted memory subsystem handles
   boot-time lazy acceptance, it lacks an API or state tracking to
   dynamically unaccept memory at runtime or route unaccepted freed
   pages in the page allocator.

3. Acceptance Granularity (2 MiB) vs. Balloon Pages (4 KiB):
   On x86, the EFI unaccepted memory bitmap tracks state at 2 MiB
   (PMD_ORDER) granularity both to keep the bitmap compact and for
   performance: accepting and unaccepting in 2 MiB blocks preserves
   2 MiB host hugepage mappings (NPT/RMP on SEV-SNP and Secure EPT on
   TDX), avoids 512x per-4 KiB page state transition and validation
   overhead, and prevents RMP/SEPT size-mismatch splits. Because a
   2 MiB bitmap unit cannot be partially unaccepted, CoCo free page
   reporting and balloon inflation require 2 MiB blocks and do not
   fall back to 4 KiB allocations (though low-level PVALIDATE still
   falls back to 4 KiB instructions if the host RMP entry was already
   split). Standard virtio-balloon allocates 4 KiB pages and caps its
   PFN array at 256 PFNs (1 KiB descriptor), so CoCo guests must
   allocate 2 MiB compound pages while streaming PFNs in 256-PFN chunks
   to respect existing VMM descriptor limits (such as QEMU's 1024-byte
   cap).

   As a trade-off of requiring 2 MiB blocks rather than 4 KiB pages,
   the effectiveness of balloon inflation/deflation in CoCo guests
   depends on guest memory not being overly fragmented (which guest
   kernel command-line options like movablecore / ZONE_MOVABLE or
   background compaction can help mitigate). Even when unmovable
   allocations fragment some pageblocks, free page reporting already
   operates at this same 2 MiB (pageblock_order) granularity on x86_64
   and continues to reclaim contiguous free blocks opportunistically.

Key Design Elements in This Series
==================================
- Per-virtqueue DMA map ops (Patches 1-2):
  Extends virtio_ring to allow individual virtqueues to install custom
  struct virtio_map_ops. For reporting_vq, virtio_balloon installs a
  no-op GPA identity mapper so free page ranges are passed directly
  without SWIOTLB bounce buffers, while keeping the descriptor ring and
  any indirect descriptor tables securely mapped via the device-level
  DMA API.

- Dynamic unacceptance & SEV-SNP hooks (Patches 3-4):
  Adds arch_unaccept_memory() / snp_unaccept_memory() for SEV-SNP. Also
  adds exception fixup in vc_raw_handle_exception() for
  SVM_EXIT_PAGE_NOT_VALIDATED (0x404) and SVM_EXIT_NPF (0x400) at
  EX_TYPE_ZEROPAD instructions so speculative reads from
  load_unaligned_zeropad() that touch the edge of dynamically unaccepted
  memory are handled cleanly without kernel panics.

- Dynamic unaccepted memory tracking & MM routing (Patches 5-9):
  Anchors the EFI unaccepted table at base 0 when unaccepted memory is
  present, introduces mark_memory_unaccepted() with error reporting and
  proper locking, generalizes zone->unaccepted_pages to variable orders
  >= unaccepted_memory_order(), and updates mm/page_alloc,
  mm/page_isolation, mm/page_owner, and mm/page_reporting to route
  unaccepted pages back to zone->unaccepted_pages without corrupting
  them with page poisoning, KASAN, or init_on_free before re-acceptance.

  Generalizing zone->unaccepted_pages from MAX_PAGE_ORDER (4 MiB on x86)
  down to >= unaccepted_memory_order() (2 MiB) aligns allocator tracking
  with the underlying hardware validation/RMP/SEPT and EFI bitmap unit:
  even when boot-time lazy acceptance processes a MAX_PAGE_ORDER block,
  the hardware and bitmap validate it in 2 MiB chunks, and runtime
  allocations, pageblocks, and mm/page_reporting split accepted memory
  below MAX_PAGE_ORDER anyway.

  Note for reviewers: The three mm/ patches (Patches 7-9) are the most
  intricate part of this series due to the interactions between lazy
  acceptance, page isolation, and page reporting. Extra scrutiny from
  mm reviewers to ensure no page-allocator or pageblock corner cases
  were missed would be especially appreciated.

- 2 MiB compound ballooning with 256-PFN chunking (Patch 10):
  Allocates compound pages at PMD_ORDER in CoCo guests that require page
  state transitions, breaking them into 256-PFN chunks when queuing to
  the host so virtqueue descriptors remain at 1 KiB, preserving
  out-of-the-box compatibility with existing hypervisors.

- Balloon state transitions & CoCo ACCESS_PLATFORM (Patches 11-13):
  Calls mark_memory_unaccepted() on reported ranges and inflated pages
  before adding buffers to reporting_vq and kicking the virtqueue, routes
  deflated pages to unaccepted freelists, preserves
  VIRTIO_F_ACCESS_PLATFORM, disables VIRTIO_BALLOON_F_FREE_PAGE_HINT and
  unsupported VIRTIO_BALLOON_F_PAGE_POISON in CoCo guests, and rejects
  probing on CoCo platforms where balloon page state transitions are
  unsupported.

Why Both Free Page Reporting and Inflation/Deflation
====================================================
CoCo guests using lazy memory acceptance boot with their maximum RAM
declared as EFI_UNACCEPTED_MEMORY in the initial EFI memory map,
avoiding runtime ACPI/virtio memory hotplug. Prior to this series, lazy
acceptance was a one-way ratchet: once a workload spike caused the
guest to accept pages from zone->unaccepted_pages, those pages could
never be unaccepted or returned to the host.

Turning EFI unaccepted memory into a bidirectional lifecycle enables two
complementary mechanisms on initial boot RAM:
- Free Page Reporting (reporting_vq): Autonomously reports idle 2 MiB
  free buddy pages to the host for hole-punching, and lazily re-accepts
  them locally from zone->unaccepted_pages on demand without a host
  virtqueue round-trip.
- Balloon Inflation/Deflation (inflate_vq / deflate_vq): Allows VMMs
  that size guests without hotplugging to actively apply memory pressure
  (triggering guest reclaim) and enforce dynamic memory targets on boot
  RAM.

Architecture Support & Intel TDX Follow-Up
==========================================
The efi/unaccepted, mm/, and virtio_balloon changes in this series are
architecture-agnostic, with architecture hooks (arch_unaccept_memory())
wired up first for AMD SEV-SNP via snp_unaccept_memory() (alongside
VIRTIO_F_ACCESS_PLATFORM support for SEV-ES).

On Intel TDX, I think dynamic unacceptance should use
TDG.MEM.PAGE.RELEASE to transition private pages from MAPPED back to
PENDING so the host VMM can reclaim them (and the guest can later
re-accept them via the existing TDG.MEM.PAGE.ACCEPT path in
tdx_accept_memory()). Once TDG.MEM.PAGE.RELEASE support and
load_unaligned_zeropad() handling for dynamically unaccepted pages
(e.g., via direct-map unmapping under SEPT_VE_DISABLE) land upstream in
arch/x86/coco/tdx/, tdx_unaccept_memory() can slot directly into
arch_unaccept_memory().

Testing
=======
- AMD SEV-SNP Milan hardware (CoCo): Passed end-to-end memory pressure
  soak testing with live guest_memfd hole punching (FALLOC_FL_PUNCH_HOLE)
  and dynamic lazy re-acceptance under memory pressure on an internal
  hypervisor.
- Cloud Hypervisor & QEMU: Verified standard non-confidential
  ballooning remains compatible. End-to-end SEV-SNP balloon testing on
  upstream Cloud Hypervisor and QEMU has been limited so far (while
  both punch holes in guest_memfd on shared page-state transitions,
  QEMU's virtio-balloon still issues MADV_WILLNEED on shared HVA
  mappings during deflate); any testing help or suggestions on these
  hypervisors are greatly appreciated.

Note: An LLM coding assistant was used during the development of this
series to assist with code and commit message drafting and refactoring.
I have reviewed and verified all changes.

Kevin Loughlin (13):
  virtio_ring: support per-virtqueue DMA mapping ops
  virtio_balloon: add reporting_vq DMA map ops
  x86/coco/sev: add snp_unaccept_memory()
  x86/coco/sev: fix zeropad #VC on unaccepted memory
  efi/libstub: start unaccepted bitmap at address 0
  efi/unaccepted: add mark_memory_unaccepted()
  mm/page_alloc: allow lower-order unaccepted_pages
  mm/page_alloc: route unaccepted isolated pages
  mm/page_reporting: account for unaccepted memory
  virtio_balloon: add higher-order page inflation
  virtio_balloon: unaccept free reporting ranges
  virtio_balloon: unaccept inflated balloon pages
  virtio_balloon: enforce ACCESS_PLATFORM in CoCo

 Documentation/mm/physical_memory.rst          |   3 +-
 arch/x86/coco/sev/core.c                      |  13 +
 arch/x86/coco/sev/vc-handle.c                 |  29 ++
 arch/x86/include/asm/sev.h                    |   2 +
 arch/x86/include/asm/unaccepted_memory.h      |  14 +
 arch/x86/include/uapi/asm/svm.h               |   2 +
 .../firmware/efi/libstub/unaccepted_memory.c  |  45 +++-
 drivers/firmware/efi/unaccepted_memory.c      | 107 +++++++-
 drivers/virtio/virtio_balloon.c               | 214 +++++++++++++--
 drivers/virtio/virtio_ring.c                  | 136 +++++++---
 include/linux/balloon.h                       |   1 +
 include/linux/mm.h                            |  18 ++
 include/linux/mmzone.h                        |   2 +-
 include/linux/virtio.h                        |   4 +
 mm/balloon.c                                  |  67 +++--
 mm/page_alloc.c                               | 248 ++++++++++++-----
 mm/page_isolation.c                           |  44 +++-
 mm/page_owner.c                               |   7 +-
 mm/page_reporting.c                           |  34 ++-
 19 files changed, 824 insertions(+), 166 deletions(-)

-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 01/13] virtio_ring: support per-virtqueue DMA mapping ops

[Kevin Loughlin]

Allow individual virtqueues to override the device-wide DMA mapping
operations via virtqueue_set_map_ops(). When custom per-virtqueue
mapping operations are configured, vring_map_one_sg(), vring_unmap_one(),
vring_mapping_error(), and the DMA sync helpers dispatch through
vq->map_ops instead of the parent virtio_device DMA API path.

Replace vring_map_single() with vring_map_indirect() and
vring_unmap_indirect() using the device-level mapping path (vdev->map
or dma_map_page_attrs()) so that indirect descriptor tables continue to
be DMA-mapped for host VMM access even when a virtqueue overrides buffer
mapping via vq->map_ops.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_ring.c | 136 ++++++++++++++++++++++++-----------
 include/linux/virtio.h       |   4 ++
 2 files changed, 98 insertions(+), 42 deletions(-)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index db678f5a80e0..9c6a1e9f0a0b 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -261,6 +261,7 @@ struct vring_virtqueue {
 	bool we_own_ring;
 
 	union virtio_map map;
+	const struct virtio_map_ops *map_ops;
 
 #ifdef DEBUG
 	/* They're supposed to lock for us. */
@@ -477,16 +478,32 @@ static struct device *vring_dma_dev(const struct vring_virtqueue *vq)
 	return vq->map.dma_dev;
 }
 
+static inline const struct virtio_map_ops *vring_get_map_ops(const struct vring_virtqueue *vq)
+{
+	if (vq->map_ops)
+		return vq->map_ops;
+	return vq->vq.vdev->map;
+}
+
+void virtqueue_set_map_ops(struct virtqueue *_vq,
+			   const struct virtio_map_ops *map_ops)
+{
+	struct vring_virtqueue *vq = to_vvq(_vq);
+
+	vq->map_ops = map_ops;
+}
+EXPORT_SYMBOL_GPL(virtqueue_set_map_ops);
+
 static int vring_mapping_error(const struct vring_virtqueue *vq,
 			       dma_addr_t addr)
 {
-	struct virtio_device *vdev = vq->vq.vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
 	if (!vq->use_map_api)
 		return 0;
 
-	if (vdev->map)
-		return vdev->map->mapping_error(vq->map, addr);
+	if (map_ops)
+		return map_ops->mapping_error(vq->map, addr);
 	else
 		return dma_mapping_error(vring_dma_dev(vq), addr);
 }
@@ -530,15 +547,45 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist
 	return 0;
 }
 
-static dma_addr_t vring_map_single(const struct vring_virtqueue *vq,
-				   void *cpu_addr, size_t size,
-				   enum dma_data_direction direction)
+static int vring_map_indirect(const struct vring_virtqueue *vq, void *cpu_addr,
+			      size_t size, dma_addr_t *addr)
 {
-	if (!vq->use_map_api)
-		return (dma_addr_t)virt_to_phys(cpu_addr);
+	struct virtio_device *vdev = vq->vq.vdev;
+
+	if (!vq->use_map_api) {
+		kmsan_handle_dma(virt_to_phys(cpu_addr), size, DMA_TO_DEVICE);
+		*addr = (dma_addr_t)virt_to_phys(cpu_addr);
+		return 0;
+	}
+
+	if (vdev->map) {
+		*addr = vdev->map->map_page(vq->map, virt_to_page(cpu_addr),
+					    offset_in_page(cpu_addr), size,
+					    DMA_TO_DEVICE, 0);
+		if (vdev->map->mapping_error(vq->map, *addr))
+			return -ENOMEM;
+		return 0;
+	}
+
+	*addr = dma_map_page_attrs(vring_dma_dev(vq), virt_to_page(cpu_addr),
+				   offset_in_page(cpu_addr), size,
+				   DMA_TO_DEVICE, 0);
+	if (dma_mapping_error(vring_dma_dev(vq), *addr))
+		return -ENOMEM;
 
-	return virtqueue_map_single_attrs(&vq->vq, cpu_addr,
-					  size, direction, 0);
+	return 0;
+}
+
+static void vring_unmap_indirect(const struct vring_virtqueue *vq,
+				 dma_addr_t addr, size_t size)
+{
+	struct virtio_device *vdev = vq->vq.vdev;
+
+	if (vdev->map)
+		vdev->map->unmap_page(vq->map, addr, size, DMA_TO_DEVICE, 0);
+	else
+		dma_unmap_page_attrs(vring_dma_dev(vq), addr, size,
+				     DMA_TO_DEVICE, 0);
 }
 
 static void virtqueue_init(struct vring_virtqueue *vq, u32 num)
@@ -576,6 +623,8 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq,
 	if (flags & VRING_DESC_F_INDIRECT) {
 		if (!vq->use_map_api)
 			goto out;
+		vring_unmap_indirect(vq, extra->addr, extra->len);
+		goto out;
 	} else if (!vring_need_unmap_buffer(vq, extra))
 		goto out;
 
@@ -761,10 +810,11 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
 
 	if (indirect) {
 		/* Now that the indirect table is filled in, map it. */
-		dma_addr_t addr = vring_map_single(
-			vq, desc, total_sg * sizeof(struct vring_desc),
-			DMA_TO_DEVICE);
-		if (vring_mapping_error(vq, addr))
+		dma_addr_t addr;
+
+		if (vring_map_indirect(vq, desc,
+				       total_sg * sizeof(struct vring_desc),
+				       &addr))
 			goto unmap_release;
 
 		virtqueue_add_desc_split(vq, vq->split.vring.desc,
@@ -1370,6 +1420,7 @@ static struct virtqueue *__vring_new_virtqueue_split(unsigned int index,
 #endif
 	vq->map = map;
 	vq->use_map_api = vring_use_map_api(vdev);
+	vq->map_ops = NULL;
 
 	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
 		!context;
@@ -1488,6 +1539,8 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq,
 	if (flags & VRING_DESC_F_INDIRECT) {
 		if (!vq->use_map_api)
 			return;
+		vring_unmap_indirect(vq, extra->addr, extra->len);
+		return;
 	} else if (!vring_need_unmap_buffer(vq, extra))
 		return;
 
@@ -1585,10 +1638,9 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
 	}
 
 	/* Now that the indirect table is filled in, map it. */
-	addr = vring_map_single(vq, desc,
-			total_sg * sizeof(struct vring_packed_desc),
-			DMA_TO_DEVICE);
-	if (vring_mapping_error(vq, addr))
+	if (vring_map_indirect(vq, desc,
+			       total_sg * sizeof(struct vring_packed_desc),
+			       &addr))
 		goto unmap_release;
 
 	vq->packed.vring.desc[head].addr = cpu_to_le64(addr);
@@ -2620,6 +2672,7 @@ static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index,
 #endif
 	vq->map = map;
 	vq->use_map_api = vring_use_map_api(vdev);
+	vq->map_ops = NULL;
 
 	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
 		!context;
@@ -3064,7 +3117,7 @@ struct device *virtqueue_dma_dev(struct virtqueue *_vq)
 {
 	struct vring_virtqueue *vq = to_vvq(_vq);
 
-	if (vq->use_map_api && !_vq->vdev->map)
+	if (vq->use_map_api && !vring_get_map_ops(vq))
 		return vq->map.dma_dev;
 	else
 		return NULL;
@@ -3809,18 +3862,17 @@ EXPORT_SYMBOL_GPL(virtqueue_map_free_coherent);
  */
 dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
 				    struct page *page,
-				    unsigned long offset,
-				    size_t size,
+				    unsigned long offset, size_t size,
 				    enum dma_data_direction dir,
 				    unsigned long attrs)
 {
 	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
-	if (vdev->map)
-		return vdev->map->map_page(vq->map,
-					   page, offset, size,
-					   dir, attrs);
+	if (map_ops)
+		return map_ops->map_page(vq->map,
+					 page, offset, size,
+					 dir, attrs);
 
 	return dma_map_page_attrs(vring_dma_dev(vq),
 				  page, offset, size,
@@ -3842,11 +3894,11 @@ void virtqueue_unmap_page_attrs(const struct virtqueue *_vq,
 				unsigned long attrs)
 {
 	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
-	if (vdev->map)
-		vdev->map->unmap_page(vq->map,
-				      map_handle, size, dir, attrs);
+	if (map_ops)
+		map_ops->unmap_page(vq->map,
+				    map_handle, size, dir, attrs);
 	else
 		dma_unmap_page_attrs(vring_dma_dev(vq), map_handle,
 				     size, dir, attrs);
@@ -3941,13 +3993,13 @@ EXPORT_SYMBOL_GPL(virtqueue_map_mapping_error);
 bool virtqueue_map_need_sync(const struct virtqueue *_vq, dma_addr_t addr)
 {
 	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
 	if (!vq->use_map_api)
 		return false;
 
-	if (vdev->map)
-		return vdev->map->need_sync(vq->map, addr);
+	if (map_ops)
+		return map_ops->need_sync(vq->map, addr);
 	else
 		return dma_need_sync(vring_dma_dev(vq), addr);
 }
@@ -3971,14 +4023,14 @@ void virtqueue_map_sync_single_range_for_cpu(const struct virtqueue *_vq,
 					     enum dma_data_direction dir)
 {
 	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
 	if (!vq->use_map_api)
 		return;
 
-	if (vdev->map)
-		vdev->map->sync_single_for_cpu(vq->map,
-					       addr + offset, size, dir);
+	if (map_ops)
+		map_ops->sync_single_for_cpu(vq->map,
+					     addr + offset, size, dir);
 	else
 		dma_sync_single_range_for_cpu(vring_dma_dev(vq),
 					      addr, offset, size, dir);
@@ -4002,15 +4054,15 @@ void virtqueue_map_sync_single_range_for_device(const struct virtqueue *_vq,
 						enum dma_data_direction dir)
 {
 	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
+	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
 
 	if (!vq->use_map_api)
 		return;
 
-	if (vdev->map)
-		vdev->map->sync_single_for_device(vq->map,
-						  addr + offset,
-						  size, dir);
+	if (map_ops)
+		map_ops->sync_single_for_device(vq->map,
+						addr + offset,
+						size, dir);
 	else
 		dma_sync_single_range_for_device(vring_dma_dev(vq), addr,
 						 offset, size, dir);
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index f923e42cfd01..20fe333d6778 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -44,6 +44,7 @@ struct virtqueue {
 };
 
 struct vduse_vq_group;
+struct virtio_map_ops;
 
 union virtio_map {
 	/* Device that performs DMA */
@@ -294,6 +295,9 @@ void virtqueue_map_free_coherent(struct virtio_device *vdev,
 				 size_t size, void *vaddr,
 				 dma_addr_t dma_handle);
 
+void virtqueue_set_map_ops(struct virtqueue *_vq,
+			   const struct virtio_map_ops *map_ops);
+
 dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
 				    struct page *page,
 				    unsigned long offset,
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 02/13] virtio_balloon: add reporting_vq DMA map ops

[Kevin Loughlin]

In confidential computing guests where VIRTIO_F_ACCESS_PLATFORM is
negotiated, the standard DMA mapping layer rejects mapping unencrypted
or swiotlb-bounced scatterlists when reporting large free page blocks
through reporting_vq. However, the host VMM only inspects the guest
physical address and length of reported free page ranges in the
virtqueue descriptor table and never reads or writes the reported guest
memory payload.

When guest memory encryption is active (CC_ATTR_GUEST_MEM_ENCRYPT),
attach custom per-virtqueue mapping operations (balloon_reporting_map_ops)
to reporting_vq that translate scatterlist entries directly to guest
physical addresses without allocating swiotlb bounce buffers or
decrypting guest memory.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_balloon.c | 56 ++++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index ab3e3d887e00..11f74848fdc4 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -19,6 +19,7 @@
 #include <linux/wait.h>
 #include <linux/mm.h>
 #include <linux/page_reporting.h>
+#include <linux/cc_platform.h>
 
 /*
  * Balloon device works in 4K page units.  So each page is pointed to by
@@ -580,6 +581,55 @@ static void update_balloon_size_func(struct work_struct *work)
 		finish_wakeup_event(vb);
 }
 
+static dma_addr_t balloon_reporting_map_page(union virtio_map map,
+					     struct page *page,
+					     unsigned long offset, size_t size,
+					     enum dma_data_direction dir,
+					     unsigned long attrs)
+{
+	/*
+	 * Free page reporting transfers physical address and size metadata
+	 * via vring descriptors; the device/host never accesses or transfers
+	 * data on the page itself. Return the true physical address without
+	 * DMA mapping / bounce buffering.
+	 */
+	return (dma_addr_t)page_to_phys(page) + offset;
+}
+
+static void balloon_reporting_unmap_page(union virtio_map map,
+					 dma_addr_t map_handle, size_t size,
+					 enum dma_data_direction dir,
+					 unsigned long attrs)
+{
+}
+
+static void balloon_reporting_sync_single(union virtio_map map,
+					  dma_addr_t map_handle, size_t size,
+					  enum dma_data_direction dir)
+{
+}
+
+static bool balloon_reporting_need_sync(union virtio_map map,
+					dma_addr_t map_handle)
+{
+	return false;
+}
+
+static int balloon_reporting_mapping_error(union virtio_map map,
+					   dma_addr_t map_handle)
+{
+	return 0;
+}
+
+static const struct virtio_map_ops balloon_reporting_map_ops = {
+	.map_page		= balloon_reporting_map_page,
+	.unmap_page		= balloon_reporting_unmap_page,
+	.sync_single_for_cpu	= balloon_reporting_sync_single,
+	.sync_single_for_device	= balloon_reporting_sync_single,
+	.need_sync		= balloon_reporting_need_sync,
+	.mapping_error		= balloon_reporting_mapping_error,
+};
+
 static int init_vqs(struct virtio_balloon *vb)
 {
 	struct virtqueue_info vqs_info[VIRTIO_BALLOON_VQ_MAX] = {};
@@ -641,8 +691,12 @@ static int init_vqs(struct virtio_balloon *vb)
 	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
 		vb->free_page_vq = vqs[VIRTIO_BALLOON_VQ_FREE_PAGE];
 
-	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_REPORTING))
+	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_REPORTING)) {
 		vb->reporting_vq = vqs[VIRTIO_BALLOON_VQ_REPORTING];
+		if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+			virtqueue_set_map_ops(vb->reporting_vq,
+					      &balloon_reporting_map_ops);
+	}
 
 	return 0;
 }
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 03/13] x86/coco/sev: add snp_unaccept_memory()

[Kevin Loughlin]

Add snp_unaccept_memory() and wire it up to arch_unaccept_memory() and
arch_has_unaccept_memory() on AMD SEV-SNP guests.

When memory ranges are dynamically unaccepted (such as during
virtio-balloon inflation or free page reporting), snp_unaccept_memory()
invokes set_pages_state(..., SNP_PAGE_STATE_SHARED). This call rescinds
guest RMP validation via PVALIDATE(..., validate=false), handling 2MB
PMD pages with automatic fallback to 4KB pages on RMP size mismatch.
snp_unaccept_memory() -> set_pages_state() also issues a GHCB Page State
Change request so the hypervisor transitions the range from private to
shared symmetrically with snp_accept_memory().

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 arch/x86/coco/sev/core.c                 | 13 +++++++++++++
 arch/x86/include/asm/sev.h               |  2 ++
 arch/x86/include/asm/unaccepted_memory.h | 14 ++++++++++++++
 3 files changed, 29 insertions(+)

diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index eb2e853e950a..3f8809bae14a 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -432,6 +432,19 @@ void snp_accept_memory(phys_addr_t start, phys_addr_t end)
 	set_pages_state(vaddr, npages, SNP_PAGE_STATE_PRIVATE);
 }
 
+void snp_unaccept_memory(phys_addr_t start, phys_addr_t end)
+{
+	unsigned long vaddr, npages;
+
+	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
+		return;
+
+	vaddr = (unsigned long)__va(start);
+	npages = (end - start) >> PAGE_SHIFT;
+
+	set_pages_state(vaddr, npages, SNP_PAGE_STATE_SHARED);
+}
+
 static int vmgexit_ap_control(u64 event, struct sev_es_save_area *vmsa, u32 apic_id)
 {
 	bool create = event != SVM_VMGEXIT_AP_DESTROY;
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index c7ea9845351e..464a721f8889 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -514,6 +514,7 @@ bool snp_init(struct boot_params *bp);
 void snp_dmi_setup(void);
 int snp_issue_svsm_attest_req(u64 call_id, struct svsm_call *call, struct svsm_attest_call *input);
 void snp_accept_memory(phys_addr_t start, phys_addr_t end);
+void snp_unaccept_memory(phys_addr_t start, phys_addr_t end);
 u64 snp_get_unsupported_features(u64 status);
 u64 sev_get_status(void);
 void sev_show_status(void);
@@ -624,6 +625,7 @@ static inline int snp_issue_svsm_attest_req(u64 call_id, struct svsm_call *call,
 	return -ENOTTY;
 }
 static inline void snp_accept_memory(phys_addr_t start, phys_addr_t end) { }
+static inline void snp_unaccept_memory(phys_addr_t start, phys_addr_t end) { }
 static inline u64 snp_get_unsupported_features(u64 status) { return 0; }
 static inline u64 sev_get_status(void) { return 0; }
 static inline void sev_show_status(void) { }
diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h
index f5937e9866ac..931d1388973e 100644
--- a/arch/x86/include/asm/unaccepted_memory.h
+++ b/arch/x86/include/asm/unaccepted_memory.h
@@ -18,6 +18,20 @@ static inline void arch_accept_memory(phys_addr_t start, phys_addr_t end)
 	}
 }
 
+static inline bool arch_has_unaccept_memory(void)
+{
+	return cc_platform_has(CC_ATTR_GUEST_SEV_SNP);
+}
+
+static inline void arch_unaccept_memory(phys_addr_t start, phys_addr_t end)
+{
+	/* Platform-specific memory-unacceptance call goes here */
+	if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
+		snp_unaccept_memory(start, end);
+	else
+		WARN_ONCE(1, "Cannot unaccept memory: unsupported platform\n");
+}
+
 static inline struct efi_unaccepted_memory *efi_get_unaccepted_table(void)
 {
 	if (efi.unaccepted == EFI_INVALID_TABLE_ADDR)
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 04/13] x86/coco/sev: fix zeropad #VC on unaccepted memory

[Kevin Loughlin]

When dynamic memory unacceptance is active on AMD SEV-SNP, an accepted
4KB page may sit immediately adjacent to a 2MB block that has been
unaccepted via PVALIDATE(validate=false). Speculative unaligned loads
across the page boundary via load_unaligned_zeropad() (such as in
dentry path walk) trigger a hardware #VC exception with error code
SVM_EXIT_PAGE_NOT_VALIDATED (0x404) or SVM_EXIT_NPF (0x400) on the
unvalidated neighbor page.

Define SVM_EXIT_PAGE_NOT_VALIDATED (0x404) and invoke fixup_exception()
in vc_raw_handle_exception() when an SVM_EXIT_PAGE_NOT_VALIDATED or
SVM_EXIT_NPF #VC exception occurs in kernel mode at an EX_TYPE_ZEROPAD
instruction before dispatching to vc_handle_exitcode(), matching the
existing EX_TYPE_ZEROPAD handling in the Intel TDX #VE handler
(ve_raise_fault()).

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 arch/x86/coco/sev/vc-handle.c   | 29 +++++++++++++++++++++++++++++
 arch/x86/include/uapi/asm/svm.h |  2 ++
 2 files changed, 31 insertions(+)

diff --git a/arch/x86/coco/sev/vc-handle.c b/arch/x86/coco/sev/vc-handle.c
index f25d302efceb..1ed00771d87d 100644
--- a/arch/x86/coco/sev/vc-handle.c
+++ b/arch/x86/coco/sev/vc-handle.c
@@ -18,6 +18,7 @@
 #include <linux/io.h>
 #include <linux/psp-sev.h>
 #include <linux/efi.h>
+#include <linux/extable.h>
 #include <uapi/linux/sev-guest.h>
 
 #include <asm/init.h>
@@ -872,6 +873,29 @@ static __always_inline bool vc_from_invalid_context(struct pt_regs *regs)
 	return is_vc2_stack(sp) && !is_vc2_stack(prev_sp);
 }
 
+static bool vc_try_fixup_zeropad(struct es_em_ctxt *ctxt, struct pt_regs *regs,
+				 unsigned long error_code)
+{
+	unsigned long addr, fault_addr;
+
+	if ((error_code != SVM_EXIT_PAGE_NOT_VALIDATED &&
+	     error_code != SVM_EXIT_NPF) ||
+	    user_mode(regs) ||
+	    ex_get_fixup_type(regs->ip) != EX_TYPE_ZEROPAD)
+		return false;
+
+	addr = (unsigned long)insn_get_addr_ref(&ctxt->insn, regs);
+	if (addr == ~0ul)
+		return false;
+
+	/*
+	 * load_unaligned_zeropad() faults when an unaligned word load crosses
+	 * into an unaccepted page; round up to the crossed page boundary.
+	 */
+	fault_addr = (addr + sizeof(long) - 1) & PAGE_MASK;
+	return fixup_exception(regs, X86_TRAP_VC, error_code, fault_addr);
+}
+
 static bool vc_raw_handle_exception(struct pt_regs *regs, unsigned long error_code)
 {
 	struct ghcb_state state;
@@ -885,6 +909,11 @@ static bool vc_raw_handle_exception(struct pt_regs *regs, unsigned long error_co
 	vc_ghcb_invalidate(ghcb);
 	result = vc_init_em_ctxt(&ctxt, regs, error_code);
 
+	if (result == ES_OK && vc_try_fixup_zeropad(&ctxt, regs, error_code)) {
+		__sev_put_ghcb(&state);
+		return true;
+	}
+
 	if (result == ES_OK)
 		result = vc_handle_exitcode(&ctxt, ghcb, error_code);
 
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index e80676185092..38b8e0c2f5e2 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -101,6 +101,7 @@
 #define SVM_EXIT_AVIC_INCOMPLETE_IPI		0x401
 #define SVM_EXIT_AVIC_UNACCELERATED_ACCESS	0x402
 #define SVM_EXIT_VMGEXIT       0x403
+#define SVM_EXIT_PAGE_NOT_VALIDATED	0x404
 #define SVM_EXIT_PML_FULL	0x407
 
 /* SEV-ES software-defined VMGEXIT events */
@@ -237,6 +238,7 @@
 	{ SVM_EXIT_AVIC_INCOMPLETE_IPI,		"avic_incomplete_ipi" }, \
 	{ SVM_EXIT_AVIC_UNACCELERATED_ACCESS,   "avic_unaccelerated_access" }, \
 	{ SVM_EXIT_VMGEXIT,		"vmgexit" }, \
+	{ SVM_EXIT_PAGE_NOT_VALIDATED,	"page_not_validated" }, \
 	{ SVM_EXIT_PML_FULL,		"pml_full" }, \
 	{ SVM_VMGEXIT_MMIO_READ,	"vmgexit_mmio_read" }, \
 	{ SVM_VMGEXIT_MMIO_WRITE,	"vmgexit_mmio_write" }, \
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 05/13] efi/libstub: start unaccepted bitmap at address 0

[Kevin Loughlin]

When EFI_UNACCEPTED_MEMORY is present in the EFI memory map or the
platform supports dynamic memory unacceptance (such as AMD SEV-SNP),
start the EFI unaccepted memory bitmap at physical address 0
(phys_base = 0) instead of the lowest initial unaccepted address.

This ensures that pre-accepted low memory below the initial unaccepted
region is addressable in the bitmap so it can be dynamically marked
unaccepted and re-accepted later via virtio-balloon or free page
reporting. Non-confidential guests without EFI_UNACCEPTED_MEMORY
continue to skip allocating the table.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 .../firmware/efi/libstub/unaccepted_memory.c  | 45 ++++++++++++++-----
 1 file changed, 33 insertions(+), 12 deletions(-)

diff --git a/drivers/firmware/efi/libstub/unaccepted_memory.c b/drivers/firmware/efi/libstub/unaccepted_memory.c
index 757dbe734a47..fcb622ac0564 100644
--- a/drivers/firmware/efi/libstub/unaccepted_memory.c
+++ b/drivers/firmware/efi/libstub/unaccepted_memory.c
@@ -2,15 +2,28 @@
 
 #include <linux/efi.h>
 #include <asm/efi.h>
+#if defined(CONFIG_X86) && defined(CONFIG_AMD_MEM_ENCRYPT)
+#include <asm/sev.h>
+#endif
 #include "efistub.h"
 
 struct efi_unaccepted_memory *unaccepted_table;
 
+static bool efi_stub_has_dynamic_unaccepted_memory(void)
+{
+#if defined(CONFIG_X86) && defined(CONFIG_AMD_MEM_ENCRYPT)
+	return sev_get_status() & MSR_AMD64_SEV_SNP_ENABLED;
+#else
+	return false;
+#endif
+}
+
 efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc,
 					struct efi_boot_memmap *map)
 {
 	efi_guid_t unaccepted_table_guid = LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID;
-	u64 unaccepted_start = ULLONG_MAX, unaccepted_end = 0, bitmap_size;
+	bool has_unaccepted = efi_stub_has_dynamic_unaccepted_memory();
+	u64 unaccepted_end = 0, bitmap_size;
 	efi_status_t status;
 	int i;
 
@@ -30,19 +43,27 @@ efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc,
 		unsigned long m = (unsigned long)map->map;
 
 		d = efi_memdesc_ptr(m, map->desc_size, i);
-		if (d->type != EFI_UNACCEPTED_MEMORY)
-			continue;
-
-		unaccepted_start = min(unaccepted_start, d->phys_addr);
-		unaccepted_end = max(unaccepted_end,
-				     d->phys_addr + d->num_pages * PAGE_SIZE);
+		switch (d->type) {
+		case EFI_UNACCEPTED_MEMORY:
+			has_unaccepted = true;
+			fallthrough;
+		case EFI_LOADER_CODE:
+		case EFI_LOADER_DATA:
+		case EFI_BOOT_SERVICES_CODE:
+		case EFI_BOOT_SERVICES_DATA:
+		case EFI_CONVENTIONAL_MEMORY:
+			unaccepted_end = max(unaccepted_end,
+					     d->phys_addr +
+					     d->num_pages * EFI_PAGE_SIZE);
+			break;
+		default:
+			break;
+		}
 	}
 
-	if (unaccepted_start == ULLONG_MAX)
+	if (!has_unaccepted || unaccepted_end == 0)
 		return EFI_SUCCESS;
 
-	unaccepted_start = round_down(unaccepted_start,
-				      EFI_UNACCEPTED_UNIT_SIZE);
 	unaccepted_end = round_up(unaccepted_end, EFI_UNACCEPTED_UNIT_SIZE);
 
 	/*
@@ -59,7 +80,7 @@ efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc,
 	 * The bitmap will be populated in setup_e820() according to the memory
 	 * map after efi_exit_boot_services().
 	 */
-	bitmap_size = DIV_ROUND_UP(unaccepted_end - unaccepted_start,
+	bitmap_size = DIV_ROUND_UP(unaccepted_end,
 				   EFI_UNACCEPTED_UNIT_SIZE * BITS_PER_BYTE);
 
 	status = efi_bs_call(allocate_pool, EFI_ACPI_RECLAIM_MEMORY,
@@ -72,7 +93,7 @@ efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc,
 
 	unaccepted_table->version = 1;
 	unaccepted_table->unit_size = EFI_UNACCEPTED_UNIT_SIZE;
-	unaccepted_table->phys_base = unaccepted_start;
+	unaccepted_table->phys_base = 0;
 	unaccepted_table->size = bitmap_size;
 	memset(unaccepted_table->bitmap, 0, bitmap_size);
 
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 06/13] efi/unaccepted: add mark_memory_unaccepted()

[Kevin Loughlin]

Introduce mark_memory_unaccepted() to mark a physical memory range as
unaccepted in the EFI unaccepted memory bitmap. Use
unaccepted_memory_order() to query the page order matching the EFI
unaccepted memory unit_size granularity.

Synchronize mark_memory_unaccepted() with concurrent accept_memory()
callers via accepting_list and only invoke arch_unaccept_memory() on
sub-ranges that are currently marked accepted in the bitmap.

On platforms with dynamic unacceptance (arch_has_unaccept_memory()),
load_unaligned_zeropad() across page boundaries is recovered via
EX_TYPE_ZEROPAD exception fixup. Thus, skip extending accept_memory()
and range_contains_unaccepted_memory() by an extra unit_size guard
block to avoid corrupting the acceptance state of adjacent 2MB blocks.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/firmware/efi/unaccepted_memory.c | 107 ++++++++++++++++++++++-
 include/linux/mm.h                       |  12 +++
 2 files changed, 115 insertions(+), 4 deletions(-)

diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c
index 4a8ec8d6a571..128bc1b113cd 100644
--- a/drivers/firmware/efi/unaccepted_memory.c
+++ b/drivers/firmware/efi/unaccepted_memory.c
@@ -5,6 +5,8 @@
 #include <linux/spinlock.h>
 #include <linux/crash_dump.h>
 #include <linux/nmi.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
 #include <asm/unaccepted_memory.h>
 
 /* Protects unaccepted memory bitmap and accepting_list */
@@ -86,7 +88,16 @@ void accept_memory(phys_addr_t start, unsigned long size)
 	 *    if 'size+end' is aligned on a unit_size boundary. (immediately
 	 *    following this comment)
 	 */
-	if (!(end % unit_size))
+	/*
+	 * On platforms without implemented dynamic unacceptance support,
+	 * load_unaligned_zeropad() across a page boundary into unaccepted
+	 * memory causes a fatal VMM exit rather than a recoverable exception.
+	 * When dynamic unacceptance is supported (arch_has_unaccept_memory()),
+	 * load_unaligned_zeropad() is recovered via EX_TYPE_ZEROPAD exception
+	 * fixup, whereas extending the range by unit_size would accept
+	 * adjacent dynamically-unaccepted blocks.
+	 */
+	if (!arch_has_unaccept_memory() && !(end % unit_size))
 		end += unit_size;
 
 	/* Make sure not to overrun the bitmap */
@@ -191,10 +202,11 @@ bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size)
 	end -= unaccepted->phys_base;
 
 	/*
-	 * Also consider the unaccepted state of the *next* page. See fix #1 in
-	 * the comment on load_unaligned_zeropad() in accept_memory().
+	 * Also consider the unaccepted state of the *next* page on platforms
+	 * without implemented dynamic unacceptance support. See fix #1 in the
+	 * comment on load_unaligned_zeropad() in accept_memory().
 	 */
-	if (!(end % unit_size))
+	if (!arch_has_unaccept_memory() && !(end % unit_size))
 		end += unit_size;
 
 	/* Make sure not to overrun the bitmap */
@@ -215,6 +227,93 @@ bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size)
 	return ret;
 }
 
+unsigned int unaccepted_memory_order(void)
+{
+	struct efi_unaccepted_memory *unaccepted = efi_get_unaccepted_table();
+
+	if (unaccepted && unaccepted->unit_size >= PAGE_SIZE)
+		return ilog2(unaccepted->unit_size >> PAGE_SHIFT);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(unaccepted_memory_order);
+
+int mark_memory_unaccepted(phys_addr_t start, unsigned long size)
+{
+	struct efi_unaccepted_memory *unaccepted;
+	unsigned long range_start, range_end;
+	struct accept_range range, *entry;
+	phys_addr_t end = start + size;
+	unsigned long flags;
+	u64 unit_size;
+
+	if (!arch_has_unaccept_memory())
+		return -EOPNOTSUPP;
+
+	unaccepted = efi_get_unaccepted_table();
+	if (!unaccepted)
+		return -EOPNOTSUPP;
+
+	unit_size = unaccepted->unit_size;
+	if (WARN_ON_ONCE(!IS_ALIGNED(start, unit_size) ||
+			 !IS_ALIGNED(size, unit_size)))
+		return -EINVAL;
+
+	if (WARN_ON_ONCE(start < unaccepted->phys_base || end <= start))
+		return -ERANGE;
+
+	start -= unaccepted->phys_base;
+	end -= unaccepted->phys_base;
+
+	/* Make sure not to overrun the bitmap */
+	if (WARN_ON_ONCE(end > unaccepted->size * unit_size * BITS_PER_BYTE))
+		return -ERANGE;
+
+	range.start = start / unit_size;
+	range.end = end / unit_size;
+retry:
+	spin_lock_irqsave(&unaccepted_memory_lock, flags);
+
+	list_for_each_entry(entry, &accepting_list, list) {
+		if (entry->end <= range.start)
+			continue;
+		if (entry->start >= range.end)
+			continue;
+
+		spin_unlock_irqrestore(&unaccepted_memory_lock, flags);
+		goto retry;
+	}
+
+	list_add(&range.list, &accepting_list);
+
+	range_start = range.start;
+	for_each_clear_bitrange_from(range_start, range_end,
+				     unaccepted->bitmap, range.end) {
+		unsigned long phys_start, phys_end;
+		unsigned long len = range_end - range_start;
+
+		phys_start = range_start * unit_size + unaccepted->phys_base;
+		phys_end = range_end * unit_size + unaccepted->phys_base;
+
+		bitmap_set(unaccepted->bitmap, range_start, len);
+		spin_unlock(&unaccepted_memory_lock);
+
+		debug_pagealloc_map_pages(pfn_to_page(PHYS_PFN(phys_start)),
+					  (phys_end - phys_start) >> PAGE_SHIFT);
+		arch_unaccept_memory(phys_start, phys_end);
+
+		spin_lock(&unaccepted_memory_lock);
+	}
+
+	list_del(&range.list);
+
+	touch_softlockup_watchdog();
+
+	spin_unlock_irqrestore(&unaccepted_memory_lock, flags);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(mark_memory_unaccepted);
+
 #ifdef CONFIG_PROC_VMCORE
 static bool unaccepted_memory_vmcore_pfn_is_ram(struct vmcore_cb *cb,
 						unsigned long pfn)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6882ad400935..27bb40651c85 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -5362,6 +5362,8 @@ int set_anon_vma_name(unsigned long addr, unsigned long size,
 
 bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size);
 void accept_memory(phys_addr_t start, unsigned long size);
+unsigned int unaccepted_memory_order(void);
+int mark_memory_unaccepted(phys_addr_t start, unsigned long size);
 
 #else
 
@@ -5375,6 +5377,16 @@ static inline void accept_memory(phys_addr_t start, unsigned long size)
 {
 }
 
+static inline unsigned int unaccepted_memory_order(void)
+{
+	return 0;
+}
+
+static inline int mark_memory_unaccepted(phys_addr_t start, unsigned long size)
+{
+	return -EOPNOTSUPP;
+}
+
 #endif
 
 static inline bool pfn_is_unaccepted_memory(unsigned long pfn)
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 07/13] mm/page_alloc: allow lower-order unaccepted_pages

[Kevin Loughlin]

Currently, zone->unaccepted_pages assumes all unaccepted blocks are
MAX_PAGE_ORDER. However, dynamic memory unacceptance operates at
unaccepted_memory_order() granularity (typically 2 MiB / PMD_ORDER on
x86).

Introduce has_unaccepted_memory() to report whether lazy unaccepted
memory and dynamic unacceptance are active (returning false when booted
with accept_memory=eager), and generalize zone->unaccepted_pages from
MAX_PAGE_ORDER to orders >= unaccepted_memory_order() stored in
page_private(page):
- Update __free_unaccepted_locked() to verify that pageblocks are
  mergeable (moving blocks >= pageblock_order to MIGRATE_MOVABLE) and
  record the block order in page_private(page).
- Update __accept_page(), try_to_accept_memory_one(), and
  cond_accept_memory() to accept blocks of order page_private(page),
  and ensure cond_accept_memory() accepts at least (1 << order) pages
  when high-order watermark or rmqueue checks fail.
- Update prep_move_freepages_block(), set_migratetype_isolate(),
  isolate_single_pageblock(), and skip_buddy_pages() to handle
  variable-order PageUnaccepted blocks.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 Documentation/mm/physical_memory.rst     |   3 +-
 drivers/firmware/efi/unaccepted_memory.c |   2 +-
 include/linux/mm.h                       |   6 ++
 include/linux/mmzone.h                   |   2 +-
 mm/page_alloc.c                          | 127 ++++++++++++++++++-----
 mm/page_isolation.c                      |  44 +++++++-
 mm/page_owner.c                          |   7 +-
 7 files changed, 155 insertions(+), 36 deletions(-)

diff --git a/Documentation/mm/physical_memory.rst b/Documentation/mm/physical_memory.rst
index 2e67e8b23a99..e9fb554932d5 100644
--- a/Documentation/mm/physical_memory.rst
+++ b/Documentation/mm/physical_memory.rst
@@ -526,7 +526,8 @@ General
   ``zone_init_free_lists()``.
 
 ``unaccepted_pages``
-  The list of pages to be accepted. All pages on the list are ``MAX_PAGE_ORDER``.
+  The list of pages to be accepted. All pages on the list are at least
+  ``unaccepted_memory_order()`` and record their order in ``page_private()``.
   Defined only when ``CONFIG_UNACCEPTED_MEMORY`` is enabled.
 
 ``flags``
diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c
index 128bc1b113cd..769faf430690 100644
--- a/drivers/firmware/efi/unaccepted_memory.c
+++ b/drivers/firmware/efi/unaccepted_memory.c
@@ -246,7 +246,7 @@ int mark_memory_unaccepted(phys_addr_t start, unsigned long size)
 	unsigned long flags;
 	u64 unit_size;
 
-	if (!arch_has_unaccept_memory())
+	if (!has_unaccepted_memory())
 		return -EOPNOTSUPP;
 
 	unaccepted = efi_get_unaccepted_table();
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 27bb40651c85..619c71565dd6 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -5362,6 +5362,7 @@ int set_anon_vma_name(unsigned long addr, unsigned long size,
 
 bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size);
 void accept_memory(phys_addr_t start, unsigned long size);
+bool has_unaccepted_memory(void);
 unsigned int unaccepted_memory_order(void);
 int mark_memory_unaccepted(phys_addr_t start, unsigned long size);
 
@@ -5377,6 +5378,11 @@ static inline void accept_memory(phys_addr_t start, unsigned long size)
 {
 }
 
+static inline bool has_unaccepted_memory(void)
+{
+	return false;
+}
+
 static inline unsigned int unaccepted_memory_order(void)
 {
 	return 0;
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index d44b04589f48..ae4657dbb5a8 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1112,7 +1112,7 @@ struct zone {
 	struct free_area	free_area[NR_PAGE_ORDERS];
 
 #ifdef CONFIG_UNACCEPTED_MEMORY
-	/* Pages to be accepted. All pages on the list are MAX_PAGE_ORDER */
+	/* Pages to be accepted. All pages are >= unaccepted_memory_order() */
 	struct list_head	unaccepted_pages;
 
 	/* To be called once the last page in the zone is accepted */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 3322f9e75616..46ae8c6f3814 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -300,8 +300,10 @@ static unsigned long alloc_stall_warn_jiffies = INITIAL_JIFFIES;
 
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
-			       int alloc_flags);
-static bool __free_unaccepted(struct page *page);
+			       int alloc_flags, bool force);
+static bool __free_unaccepted(struct page *page, unsigned int order);
+static bool __free_unaccepted_locked(struct page *page, struct zone *zone,
+				     unsigned int order);
 
 int page_group_by_mobility_disabled __read_mostly;
 
@@ -1614,8 +1616,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 		atomic_long_add(nr_pages, &page_zone(page)->managed_pages);
 	}
 
-	if (page_contains_unaccepted(page, order)) {
-		if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
+	if (range_contains_unaccepted_memory(page_to_phys(page),
+					     PAGE_SIZE << order)) {
+		if (order >= unaccepted_memory_order() &&
+		    __free_unaccepted(page, order))
 			return;
 
 		accept_memory(page_to_phys(page), PAGE_SIZE << order);
@@ -2003,6 +2007,13 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page,
 				pfn += nr;
 				continue;
 			}
+			if (PageUnaccepted(page)) {
+				int nr = 1 << page_private(page);
+
+				*num_free += nr;
+				pfn += nr;
+				continue;
+			}
 			/*
 			 * We assume that pages that could be isolated for
 			 * migration are movable. But we don't actually try
@@ -3872,7 +3883,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 			continue;
 		}
 
-		cond_accept_memory(zone, order, alloc_flags);
+		cond_accept_memory(zone, order, alloc_flags, false);
 
 		/*
 		 * Detect whether the number of free pages is below high
@@ -3897,7 +3908,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		if (!zone_watermark_fast(zone, order, mark,
 				       ac->highest_zoneidx, alloc_flags,
 				       gfp_mask)) {
-			if (cond_accept_memory(zone, order, alloc_flags))
+			if (cond_accept_memory(zone, order, alloc_flags, true))
 				goto try_this_zone;
 
 			/*
@@ -3934,7 +3945,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 
 			return page;
 		} else {
-			if (cond_accept_memory(zone, order, alloc_flags))
+			if (cond_accept_memory(zone, order, alloc_flags, true))
 				goto try_this_zone;
 
 			/* Try again if zone has deferred pages */
@@ -5216,7 +5227,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 			goto failed;
 		}
 
-		cond_accept_memory(zone, 0, alloc_flags);
+		cond_accept_memory(zone, 0, alloc_flags, false);
 retry_this_zone:
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages - nr_populated;
 		if (zone_watermark_fast(zone, 0,  mark,
@@ -5225,7 +5236,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 			break;
 		}
 
-		if (cond_accept_memory(zone, 0, alloc_flags))
+		if (cond_accept_memory(zone, 0, alloc_flags, true))
 			goto retry_this_zone;
 
 		/* Try again if zone has deferred pages */
@@ -7867,6 +7878,7 @@ bool has_managed_zone(enum zone_type zone)
 }
 
 #ifdef CONFIG_UNACCEPTED_MEMORY
+#include <asm/unaccepted_memory.h>
 
 static bool lazy_accept = true;
 
@@ -7884,25 +7896,45 @@ static int __init accept_memory_parse(char *p)
 }
 early_param("accept_memory", accept_memory_parse);
 
+bool has_unaccepted_memory(void)
+{
+	return lazy_accept && efi_get_unaccepted_table() &&
+	       arch_has_unaccept_memory() &&
+	       unaccepted_memory_order() <= MAX_PAGE_ORDER;
+}
+EXPORT_SYMBOL_GPL(has_unaccepted_memory);
+
 static bool page_contains_unaccepted(struct page *page, unsigned int order)
 {
 	phys_addr_t start = page_to_phys(page);
 
+	/*
+	 * Unaccepted memory is tracked and freed in blocks of at least
+	 * unaccepted_memory_order(). Skipping lower-order frees avoids
+	 * acquiring unaccepted_memory_lock on normal page frees.
+	 */
+	if (!has_unaccepted_memory() || order < unaccepted_memory_order())
+		return false;
+
 	return range_contains_unaccepted_memory(start, PAGE_SIZE << order);
 }
 
 static void __accept_page(struct zone *zone, unsigned long *flags,
 			  struct page *page)
 {
+	unsigned int order = page_private(page);
+	unsigned int nr_pages = 1 << order;
+
 	list_del(&page->lru);
-	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
-	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
+	account_freepages(zone, -nr_pages, MIGRATE_MOVABLE);
+	__mod_zone_page_state(zone, NR_UNACCEPTED, -nr_pages);
 	__ClearPageUnaccepted(page);
+	set_page_private(page, 0);
 	spin_unlock_irqrestore(&zone->lock, *flags);
 
-	accept_memory(page_to_phys(page), PAGE_SIZE << MAX_PAGE_ORDER);
+	accept_memory(page_to_phys(page), PAGE_SIZE << order);
 
-	__free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL);
+	__free_pages_ok(page, order, FPI_TO_TAIL);
 }
 
 void accept_page(struct page *page)
@@ -7920,27 +7952,29 @@ void accept_page(struct page *page)
 	__accept_page(zone, &flags, page);
 }
 
-static bool try_to_accept_memory_one(struct zone *zone)
+static unsigned int try_to_accept_memory_one(struct zone *zone)
 {
 	unsigned long flags;
 	struct page *page;
+	unsigned int nr_pages;
 
 	spin_lock_irqsave(&zone->lock, flags);
 	page = list_first_entry_or_null(&zone->unaccepted_pages,
 					struct page, lru);
 	if (!page) {
 		spin_unlock_irqrestore(&zone->lock, flags);
-		return false;
+		return 0;
 	}
 
+	nr_pages = 1 << page_private(page);
 	/* Unlocks zone->lock */
 	__accept_page(zone, &flags, page);
 
-	return true;
+	return nr_pages;
 }
 
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
-			       int alloc_flags)
+			       int alloc_flags, bool force)
 {
 	long to_accept, wmark;
 	bool ret = false;
@@ -7957,7 +7991,7 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order,
 	/*
 	 * Watermarks have not been initialized yet.
 	 *
-	 * Accepting one MAX_PAGE_ORDER page to ensure progress.
+	 * Accepting one block to ensure progress.
 	 */
 	if (!wmark)
 		return try_to_accept_memory_one(zone);
@@ -7968,32 +8002,62 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order,
 		    __zone_watermark_unusable_free(zone, order, 0) -
 		    zone_page_state(zone, NR_UNACCEPTED));
 
+	if (force && to_accept < (1L << order))
+		to_accept = 1L << order;
+
 	while (to_accept > 0) {
-		if (!try_to_accept_memory_one(zone))
+		unsigned int accepted = try_to_accept_memory_one(zone);
+
+		if (!accepted)
 			break;
 		ret = true;
-		to_accept -= MAX_ORDER_NR_PAGES;
+		to_accept -= accepted;
 	}
 
 	return ret;
 }
 
-static bool __free_unaccepted(struct page *page)
+static bool __free_unaccepted_locked(struct page *page, struct zone *zone,
+				     unsigned int order)
+{
+	unsigned int nr_pages = 1 << order;
+	unsigned int step = max_t(unsigned int, 1, pageblock_nr_pages);
+	unsigned int i;
+
+	lockdep_assert_held(&zone->lock);
+
+	for (i = 0; i < nr_pages; i += step) {
+		int mt = get_pageblock_migratetype(page + i);
+
+		if (!migratetype_is_mergeable(mt))
+			return false;
+	}
+
+	if (order >= pageblock_order)
+		change_pageblock_range(page, order, MIGRATE_MOVABLE);
+
+	set_page_private(page, order);
+	list_add_tail(&page->lru, &zone->unaccepted_pages);
+	account_freepages(zone, nr_pages, MIGRATE_MOVABLE);
+	__mod_zone_page_state(zone, NR_UNACCEPTED, nr_pages);
+	__SetPageUnaccepted(page);
+	return true;
+}
+
+static bool __free_unaccepted(struct page *page, unsigned int order)
 {
 	struct zone *zone = page_zone(page);
 	unsigned long flags;
+	bool ret;
 
 	if (!lazy_accept)
 		return false;
 
 	spin_lock_irqsave(&zone->lock, flags);
-	list_add_tail(&page->lru, &zone->unaccepted_pages);
-	account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
-	__mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES);
-	__SetPageUnaccepted(page);
+	ret = __free_unaccepted_locked(page, zone, order);
 	spin_unlock_irqrestore(&zone->lock, flags);
 
-	return true;
+	return ret;
 }
 
 #else
@@ -8004,12 +8068,19 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order)
 }
 
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
-			       int alloc_flags)
+			       int alloc_flags, bool force)
 {
 	return false;
 }
 
-static bool __free_unaccepted(struct page *page)
+static bool __free_unaccepted_locked(struct page *page, struct zone *zone,
+				     unsigned int order)
+{
+	BUILD_BUG();
+	return false;
+}
+
+static bool __free_unaccepted(struct page *page, unsigned int order)
 {
 	BUILD_BUG();
 	return false;
diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 5aaad037384e..c85ce158a646 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -165,6 +165,36 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e
 	return NULL;
 }
 
+static struct page *find_unaccepted_page_in_pageblock(struct zone *zone,
+						      struct page *page)
+{
+#ifdef CONFIG_UNACCEPTED_MEMORY
+	unsigned long pb_start = pageblock_start_pfn(page_to_pfn(page));
+	unsigned long pb_end = pageblock_end_pfn(page_to_pfn(page));
+	unsigned long start_pfn, pfn, step;
+
+	lockdep_assert_held(&zone->lock);
+
+	if (list_empty(&zone->unaccepted_pages) ||
+	    unaccepted_memory_order() > MAX_PAGE_ORDER)
+		return NULL;
+
+	start_pfn = ALIGN_DOWN(pb_start, MAX_ORDER_NR_PAGES);
+	step = 1UL << unaccepted_memory_order();
+
+	for (pfn = start_pfn; pfn < pb_end; pfn += step) {
+		struct page *cand = pfn_to_online_page(pfn);
+
+		if (!cand || page_zone(cand) != zone || !PageUnaccepted(cand))
+			continue;
+
+		if (pfn + (1UL << page_private(cand)) > pb_start)
+			return cand;
+	}
+#endif
+	return NULL;
+}
+
 /*
  * This function set pageblock migratetype to isolate if no unmovable page is
  * present in [start_pfn, end_pfn). The pageblock must intersect with
@@ -175,10 +205,14 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode,
 {
 	struct zone *zone = page_zone(page);
 	struct page *unmovable;
+	struct page *unaccepted = NULL;
 	unsigned long check_unmovable_start, check_unmovable_end;
 
-	if (PageUnaccepted(page))
-		accept_page(page);
+retry:
+	if (unaccepted) {
+		accept_page(unaccepted);
+		unaccepted = NULL;
+	}
 
 	scoped_guard(spinlock_irqsave, &zone->lock) {
 		/*
@@ -189,6 +223,10 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode,
 		if (is_migrate_isolate_page(page))
 			return -EBUSY;
 
+		unaccepted = find_unaccepted_page_in_pageblock(zone, page);
+		if (unaccepted)
+			goto retry;
+
 		/*
 		 * FIXME: Now, memory hotplug doesn't call shrink_slab() by
 		 * itself. We just check MOVABLE pages.
@@ -383,7 +421,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn,
 		pfn = page_to_pfn(page);
 
 		if (PageUnaccepted(page)) {
-			pfn += MAX_ORDER_NR_PAGES;
+			accept_page(page);
 			continue;
 		}
 
diff --git a/mm/page_owner.c b/mm/page_owner.c
index cfc31c92d765..e3f3a019b0bc 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -459,10 +459,13 @@ static inline bool skip_buddy_pages(unsigned long *pfn, struct page *page)
 {
 	unsigned long order;
 
-	if (!PageBuddy(page))
+	if (PageBuddy(page))
+		order = buddy_order_unsafe(page);
+	else if (PageUnaccepted(page))
+		order = READ_ONCE(page->private);
+	else
 		return false;
 
-	order = buddy_order_unsafe(page);
 	if (order <= MAX_PAGE_ORDER) {
 		unsigned long new_pfn = *pfn + (1UL << order);
 		unsigned long boundary = ALIGN(*pfn + 1, MAX_ORDER_NR_PAGES);
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 08/13] mm/page_alloc: route unaccepted isolated pages

[Kevin Loughlin]

When free page reporting unaccepts isolated pages on confidential
computing guests, __putback_isolated_page() must not return those
unaccepted pages directly to zone->free_area; doing so would violate the
allocator invariant that all pages on zone->free_area are accepted.

Update __putback_isolated_page(), __free_pages_ok(),
__free_frozen_pages(), and free_unref_folios() to route dynamically
unaccepted pages to zone->unaccepted_pages via __free_unaccepted_locked()
(falling back to immediate acceptance for isolated or CMA pageblocks).

In addition:
- Extract free_pages_prepare_init() so post-acceptance poisoning,
  zeroing, KASAN tagging, arch_free_page(), and debug_pagealloc
  unmapping run after accept_memory() without touching unaccepted pages
  in __free_pages_prepare() or re-running metadata teardown in
  __accept_page().
- Pass FPI_SKIP_REPORT_NOTIFY when __accept_page() frees a freshly
  accepted block into the buddy allocator so it is not immediately
  re-queued for free page reporting before allocation.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 mm/page_alloc.c | 120 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 85 insertions(+), 35 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 46ae8c6f3814..c10b77304633 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1298,12 +1298,53 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
 
 #endif /* CONFIG_MEM_ALLOC_PROFILING */
 
+static inline bool should_free_unaccepted(struct page *page,
+					  unsigned int order,
+					  fpi_t fpi_flags)
+{
+	return !(fpi_flags & FPI_NOLOCK) &&
+	       page_contains_unaccepted(page, order);
+}
+
+static void free_pages_prepare_init(struct page *page, unsigned int order)
+{
+	bool skip_kasan_poison = should_skip_kasan_poison(page);
+	bool init = want_init_on_free();
+
+	kernel_poison_pages(page, 1 << order);
+
+	/*
+	 * As memory initialization might be integrated into KASAN,
+	 * KASAN poisoning and memory initialization code must be
+	 * kept together to avoid discrepancies in behavior.
+	 *
+	 * With hardware tag-based KASAN, memory tags must be set before the
+	 * page becomes unavailable via debug_pagealloc or arch_free_page.
+	 */
+	if (!skip_kasan_poison) {
+		kasan_poison_pages(page, order, init);
+
+		/* Memory is already initialized if KASAN did it internally. */
+		if (kasan_has_integrated_init())
+			init = false;
+	}
+	if (init)
+		clear_highpages_kasan_tagged(page, 1 << order);
+
+	/*
+	 * arch_free_page() can make the page's contents inaccessible.  s390
+	 * does this.  So nothing which can access the page's contents should
+	 * happen after this.
+	 */
+	arch_free_page(page, order);
+
+	debug_pagealloc_unmap_pages(page, 1 << order);
+}
+
 static __always_inline bool __free_pages_prepare(struct page *page,
 		unsigned int order, fpi_t fpi_flags)
 {
 	int bad = 0;
-	bool skip_kasan_poison = should_skip_kasan_poison(page);
-	bool init = want_init_on_free();
 	bool compound = PageCompound(page);
 	struct folio *folio = page_folio(page);
 
@@ -1411,34 +1452,10 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 					   PAGE_SIZE << order);
 	}
 
-	kernel_poison_pages(page, 1 << order);
-
-	/*
-	 * As memory initialization might be integrated into KASAN,
-	 * KASAN poisoning and memory initialization code must be
-	 * kept together to avoid discrepancies in behavior.
-	 *
-	 * With hardware tag-based KASAN, memory tags must be set before the
-	 * page becomes unavailable via debug_pagealloc or arch_free_page.
-	 */
-	if (!skip_kasan_poison) {
-		kasan_poison_pages(page, order, init);
-
-		/* Memory is already initialized if KASAN did it internally. */
-		if (kasan_has_integrated_init())
-			init = false;
-	}
-	if (init)
-		clear_highpages_kasan_tagged(page, 1 << order);
-
-	/*
-	 * arch_free_page() can make the page's contents inaccessible.  s390
-	 * does this.  So nothing which can access the page's contents should
-	 * happen after this.
-	 */
-	arch_free_page(page, order);
+	if (should_free_unaccepted(page, order, fpi_flags))
+		return true;
 
-	debug_pagealloc_unmap_pages(page, 1 << order);
+	free_pages_prepare_init(page, order);
 
 	return true;
 }
@@ -1572,14 +1589,31 @@ static void free_one_page(struct zone *zone, struct page *page,
 	__count_vm_events(PGFREE, 1 << order);
 }
 
+static bool handle_unaccepted_free(struct page *page, unsigned int order,
+				   fpi_t fpi_flags)
+{
+	if (!should_free_unaccepted(page, order, fpi_flags))
+		return false;
+
+	if (__free_unaccepted(page, order))
+		return true;
+
+	accept_memory(page_to_phys(page), PAGE_SIZE << order);
+	free_pages_prepare_init(page, order);
+	return false;
+}
+
 static void __free_pages_ok(struct page *page, unsigned int order,
 			    fpi_t fpi_flags)
 {
 	unsigned long pfn = page_to_pfn(page);
 	struct zone *zone = page_zone(page);
 
-	if (__free_pages_prepare(page, order, fpi_flags))
+	if (__free_pages_prepare(page, order, fpi_flags)) {
+		if (handle_unaccepted_free(page, order, fpi_flags))
+			return;
 		free_one_page(zone, page, pfn, order, fpi_flags);
+	}
 }
 
 void __meminit __free_pages_core(struct page *page, unsigned int order,
@@ -2955,7 +2989,8 @@ static void __free_frozen_pages(struct page *page, unsigned int order,
 	unsigned long pfn = page_to_pfn(page);
 	int migratetype;
 
-	if (!pcp_allowed_order(order)) {
+	if (!pcp_allowed_order(order) ||
+	    should_free_unaccepted(page, order, fpi_flags)) {
 		__free_pages_ok(page, order, fpi_flags);
 		return;
 	}
@@ -3024,9 +3059,13 @@ void free_unref_folios(struct folio_batch *folios)
 			continue;
 		/*
 		 * Free orders not handled on the PCP directly to the
-		 * allocator.
+		 * allocator, as well as any pages containing unaccepted memory.
 		 */
-		if (!pcp_allowed_order(order)) {
+		if (!pcp_allowed_order(order) ||
+		    should_free_unaccepted(&folio->page, order, FPI_NONE)) {
+			if (handle_unaccepted_free(&folio->page, order,
+						   FPI_NONE))
+				continue;
 			free_one_page(folio_zone(folio), &folio->page,
 				      pfn, order, FPI_NONE);
 			continue;
@@ -3187,6 +3226,16 @@ void __putback_isolated_page(struct page *page, unsigned int order, int mt)
 	/* zone lock should be held when this function is called */
 	lockdep_assert_held(&zone->lock);
 
+	if (page_contains_unaccepted(page, order)) {
+		if (__free_unaccepted_locked(page, zone, order))
+			return;
+		accept_memory(page_to_phys(page), PAGE_SIZE << order);
+		free_pages_prepare_init(page, order);
+		split_large_buddy(zone, page, page_to_pfn(page), order,
+				  FPI_SKIP_REPORT_NOTIFY | FPI_TO_TAIL);
+		return;
+	}
+
 	/* Return isolated page to tail of freelist. */
 	__free_one_page(page, page_to_pfn(page), zone, order, mt,
 			FPI_SKIP_REPORT_NOTIFY | FPI_TO_TAIL);
@@ -7933,8 +7982,9 @@ static void __accept_page(struct zone *zone, unsigned long *flags,
 	spin_unlock_irqrestore(&zone->lock, *flags);
 
 	accept_memory(page_to_phys(page), PAGE_SIZE << order);
-
-	__free_pages_ok(page, order, FPI_TO_TAIL);
+	free_pages_prepare_init(page, order);
+	free_one_page(zone, page, page_to_pfn(page), order,
+		      FPI_TO_TAIL | FPI_SKIP_REPORT_NOTIFY);
 }
 
 void accept_page(struct page *page)
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 09/13] mm/page_reporting: account for unaccepted memory

[Kevin Loughlin]

When lazy unaccepted memory is active, pages on zone->unaccepted_pages
are counted in NR_FREE_PAGES (MIGRATE_MOVABLE) as well as NR_UNACCEPTED.
Without accounting for NR_UNACCEPTED in free page reporting watermarks,
page_reporting_process_zone() and __isolate_free_page() treat
unaccepted pages as already-accepted free buddy pages and can over-drain
accepted free lists.

Account for zone_page_state(zone, NR_UNACCEPTED) in
__isolate_free_page(), page_reporting_cycle(), and
page_reporting_process_zone() (using promo_wmark_pages(zone) for free
page reporting watermarks and skipping non-mergeable migratetypes when
has_unaccepted_memory() is active), and enforce
unaccepted_memory_order() as the minimum page_reporting_order.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 mm/page_alloc.c     |  3 +++
 mm/page_reporting.c | 34 +++++++++++++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c10b77304633..541b2eaf34a4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3183,6 +3183,9 @@ int __isolate_free_page(struct page *page, unsigned int order)
 		 * exists.
 		 */
 		watermark = zone->_watermark[WMARK_MIN] + (1UL << order);
+#ifdef CONFIG_UNACCEPTED_MEMORY
+		watermark += zone_page_state(zone, NR_UNACCEPTED);
+#endif
 		if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA))
 			return 0;
 	}
diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index de587be17801..adacdb015b55 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -17,11 +17,14 @@ unsigned int page_reporting_order = PAGE_REPORTING_ORDER_UNSPECIFIED;
 
 static int page_order_update_notify(const char *val, const struct kernel_param *kp)
 {
+	unsigned int min_order = has_unaccepted_memory() ?
+				 unaccepted_memory_order() : 0;
+
 	/*
 	 * If param is set beyond this limit, order is set to default
 	 * pageblock_order value
 	 */
-	return  param_set_uint_minmax(val, kp, 0, MAX_PAGE_ORDER);
+	return param_set_uint_minmax(val, kp, min_order, MAX_PAGE_ORDER);
 }
 
 static const struct kernel_param_ops page_reporting_param_ops = {
@@ -206,6 +209,21 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,
 
 		/* Attempt to pull page from list and place in scatterlist */
 		if (*offset) {
+#ifdef CONFIG_UNACCEPTED_MEMORY
+			if (has_unaccepted_memory() ||
+			    !list_empty(&zone->unaccepted_pages)) {
+				unsigned long watermark;
+
+				watermark = promo_wmark_pages(zone) +
+					    zone_page_state(zone, NR_UNACCEPTED) +
+					    (1UL << order);
+				if (!zone_watermark_ok(zone, 0, watermark, 0,
+						       ALLOC_CMA)) {
+					next = page;
+					break;
+				}
+			}
+#endif
 			if (!__isolate_free_page(page, order)) {
 				next = page;
 				break;
@@ -275,6 +293,12 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev,
 	/* Generate minimum watermark to be able to guarantee progress */
 	watermark = low_wmark_pages(zone) +
 		    (prdev->capacity << page_reporting_order);
+#ifdef CONFIG_UNACCEPTED_MEMORY
+	if (has_unaccepted_memory() || !list_empty(&zone->unaccepted_pages))
+		watermark = promo_wmark_pages(zone) +
+			    zone_page_state(zone, NR_UNACCEPTED) +
+			    (prdev->capacity << page_reporting_order);
+#endif
 
 	/*
 	 * Cancel request if insufficient free memory or if we failed
@@ -289,6 +313,11 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev,
 			/* We do not pull pages from the isolate free list */
 			if (is_migrate_isolate(mt))
 				continue;
+#ifdef CONFIG_UNACCEPTED_MEMORY
+			if (has_unaccepted_memory() &&
+			    !migratetype_is_mergeable(mt))
+				continue;
+#endif
 
 			err = page_reporting_cycle(prdev, zone, order, mt,
 						   sgl, &offset);
@@ -384,6 +413,9 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)
 		else
 			page_reporting_order = pageblock_order;
 	}
+	if (has_unaccepted_memory() &&
+	    page_reporting_order < unaccepted_memory_order())
+		page_reporting_order = unaccepted_memory_order();
 
 	if (!prdev->capacity || prdev->capacity > PAGE_REPORTING_CAPACITY)
 		prdev->capacity = PAGE_REPORTING_CAPACITY;
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 10/13] virtio_balloon: add higher-order page inflation

[Kevin Loughlin]

In confidential computing guests requiring page state transitions,
unaccepted memory tracking operates at the EFI unaccepted memory unit
granularity (typically 2MB / order 9 on x86). Inflating the balloon
with 4KB pages would prevent unaccepting the backing 2MB unit.

Inflate balloon pages at unaccepted_memory_order() when
virtballoon_cc_needs_state_transition() is true, while retaining
order-0 allocations for non-CoCo guests and CoCo guests that do not
use unaccepted memory transitions. When a higher-order balloon page
spans more 4KB balloon PFNs than VIRTIO_BALLOON_ARRAY_PFNS_MAX (256),
transmit its PFNs to the host in VIRTIO_BALLOON_ARRAY_PFNS_MAX chunks.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_balloon.c | 87 ++++++++++++++++++++++++++-------
 include/linux/balloon.h         |  1 +
 mm/balloon.c                    | 67 +++++++++++++++++++------
 3 files changed, 121 insertions(+), 34 deletions(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 11f74848fdc4..d162c162b496 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -182,6 +182,13 @@ static void balloon_ack(struct virtqueue *vq)
 	wake_up(&vb->acked);
 }
 
+static inline bool virtballoon_cc_needs_state_transition(void)
+{
+	return IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) &&
+	       cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	       has_unaccepted_memory();
+}
+
 static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
 {
 	struct scatterlist sg;
@@ -243,24 +250,49 @@ static void set_page_pfns(struct virtio_balloon *vb,
 					  page_to_balloon_pfn(page) + i);
 }
 
+static void queue_page_pfns(struct virtio_balloon *vb, struct virtqueue *vq,
+			    struct page *page, unsigned int order)
+{
+	unsigned int nr_pages = 1 << order;
+	unsigned int i;
+
+	for (i = 0; i < nr_pages; i++) {
+		set_page_pfns(vb, vb->pfns + vb->num_pfns, page + i);
+		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
+		if (vb->num_pfns == ARRAY_SIZE(vb->pfns)) {
+			tell_host(vb, vq);
+			vb->num_pfns = 0;
+		}
+	}
+}
+
+static inline unsigned int virtballoon_preferred_order(void)
+{
+	if (virtballoon_cc_needs_state_transition())
+		return unaccepted_memory_order();
+	return 0;
+}
+
 static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
 {
-	unsigned int num_allocated_pages;
+	unsigned int order = virtballoon_preferred_order();
+	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
+	unsigned int num_allocated_pages = 0;
 	struct page *page, *next;
 	unsigned int num_pfns;
 	LIST_HEAD(pages);
 
-	/* We can only do one array worth at a time. */
-	num = min(num, ARRAY_SIZE(vb->pfns));
+	/* We can only do one array or preferred-order page worth at a time. */
+	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
 
 	for (num_pfns = 0; num_pfns < num;
-	     num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
-		page = balloon_page_alloc();
+	     num_pfns += page_pfns) {
+		page = balloon_pages_alloc(order);
 
 		if (!page) {
 			dev_info_ratelimited(&vb->vdev->dev,
 					     "Out of puff! Can't get %u pages\n",
-					     VIRTIO_BALLOON_PAGES_PER_PAGE);
+					     page_pfns);
 			/* Sleep for at least 1/5 of a second before retry. */
 			msleep(200);
 			break;
@@ -274,15 +306,17 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
 	vb->num_pfns = 0;
 
 	list_for_each_entry_safe(page, next, &pages, lru) {
+		unsigned int page_order = compound_order(page);
+		unsigned int pfns = (1 << page_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
+
 		list_del(&page->lru);
 		balloon_page_enqueue(&vb->vb_dev_info, page);
 
-		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
-		vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
-		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
+		vb->num_pages += pfns;
+		num_allocated_pages += pfns;
+		queue_page_pfns(vb, vb->inflate_vq, page, page_order);
 	}
 
-	num_allocated_pages = vb->num_pfns;
 	/* Did we get any? */
 	if (vb->num_pfns != 0)
 		tell_host(vb, vb->inflate_vq);
@@ -304,28 +338,42 @@ static void release_pages_balloon(struct virtio_balloon *vb,
 
 static unsigned int leak_balloon(struct virtio_balloon *vb, size_t num)
 {
-	unsigned int num_freed_pages;
+	unsigned int order = virtballoon_preferred_order();
+	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
+	unsigned int num_freed_pages = 0;
 	struct page *page;
 	struct balloon_dev_info *vb_dev_info = &vb->vb_dev_info;
 	LIST_HEAD(pages);
 
-	/* We can only do one array worth at a time. */
-	num = min(num, ARRAY_SIZE(vb->pfns));
+	/* We can only do one array or preferred-order page worth at a time. */
+	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
 
 	mutex_lock(&vb->balloon_lock);
 	/* We can't release more pages than taken */
 	num = min(num, (size_t)vb->num_pages);
-	for (vb->num_pfns = 0; vb->num_pfns < num;
-	     vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
+	vb->num_pfns = 0;
+	while (num_freed_pages < num) {
+		unsigned int dequeued_order;
+		unsigned int dequeued_pfns;
+
 		page = balloon_page_dequeue(vb_dev_info);
 		if (!page)
 			break;
-		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
+		dequeued_order = compound_order(page);
+		dequeued_pfns = (1 << dequeued_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
+
+		if (num_freed_pages + dequeued_pfns > num &&
+		    num_freed_pages > 0) {
+			balloon_page_enqueue(vb_dev_info, page);
+			break;
+		}
+
+		queue_page_pfns(vb, vb->deflate_vq, page, dequeued_order);
 		list_add(&page->lru, &pages);
-		vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
+		vb->num_pages -= dequeued_pfns;
+		num_freed_pages += dequeued_pfns;
 	}
 
-	num_freed_pages = vb->num_pfns;
 	/*
 	 * Note that if
 	 * virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
@@ -474,7 +522,8 @@ static inline s64 towards_target(struct virtio_balloon *vb)
 	 * Aligned up to guest page size to avoid inflating and deflating
 	 * balloon endlessly.
 	 */
-	target = ALIGN(num_pages, VIRTIO_BALLOON_PAGES_PER_PAGE);
+	target = ALIGN((s64)num_pages,
+		       VIRTIO_BALLOON_PAGES_PER_PAGE << virtballoon_preferred_order());
 	return target - vb->num_pages;
 }
 
diff --git a/include/linux/balloon.h b/include/linux/balloon.h
index ca5b15150f42..bc46e36b997f 100644
--- a/include/linux/balloon.h
+++ b/include/linux/balloon.h
@@ -58,6 +58,7 @@ struct balloon_dev_info {
 	bool adjust_managed_page_count;
 };
 
+struct page *balloon_pages_alloc(unsigned int order);
 struct page *balloon_page_alloc(void);
 void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
 		struct page *page);
diff --git a/mm/balloon.c b/mm/balloon.c
index 96a8f1e20bc6..e9941d08d8e9 100644
--- a/mm/balloon.c
+++ b/mm/balloon.c
@@ -27,9 +27,12 @@ static DEFINE_SPINLOCK(balloon_pages_lock);
 static void balloon_page_insert(struct balloon_dev_info *balloon,
 				       struct page *page)
 {
+	unsigned int i, nr_pages = 1U << compound_order(page);
+
 	lockdep_assert_held(&balloon_pages_lock);
-	__SetPageOffline(page);
-	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) {
+	for (i = 0; i < nr_pages; i++)
+		__SetPageOffline(page + i);
+	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page)) {
 		SetPageMovableOps(page);
 		set_page_private(page, (unsigned long)balloon);
 	}
@@ -45,20 +48,31 @@ static void balloon_page_insert(struct balloon_dev_info *balloon,
  */
 static void balloon_page_finalize(struct page *page)
 {
+	unsigned int i, nr_pages = 1U << compound_order(page);
+
 	lockdep_assert_held(&balloon_pages_lock);
-	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
+	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page))
 		set_page_private(page, 0);
-	/* PageOffline is sticky until the page is freed to the buddy. */
+	/*
+	 * PageOffline is sticky on the head page until freed to the buddy,
+	 * which resets the head page's page_type in __free_pages_prepare().
+	 * Clear PageOffline on tail pages here as __free_pages_prepare()
+	 * expects tail pages to have a default page_type.
+	 */
+	for (i = 1; i < nr_pages; i++)
+		__ClearPageOffline(page + i);
 }
 
 static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info,
 				     struct page *page)
 {
+	unsigned int order = compound_order(page);
+
 	balloon_page_insert(b_dev_info, page);
 	if (b_dev_info->adjust_managed_page_count)
-		adjust_managed_page_count(page, -1);
-	__count_vm_event(BALLOON_INFLATE);
-	inc_node_page_state(page, NR_BALLOON_PAGES);
+		adjust_managed_page_count(page, -(1L << order));
+	__count_vm_events(BALLOON_INFLATE, 1UL << order);
+	mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES, 1L << order);
 }
 
 /**
@@ -117,15 +131,18 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
 
 	spin_lock_irqsave(&balloon_pages_lock, flags);
 	list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) {
+		unsigned int order = compound_order(page);
+
 		if (n_pages == n_req_pages)
 			break;
 		list_del(&page->lru);
 		if (b_dev_info->adjust_managed_page_count)
-			adjust_managed_page_count(page, 1);
+			adjust_managed_page_count(page, 1L << order);
 		balloon_page_finalize(page);
-		__count_vm_event(BALLOON_DEFLATE);
+		__count_vm_events(BALLOON_DEFLATE, 1UL << order);
 		list_add(&page->lru, pages);
-		dec_node_page_state(page, NR_BALLOON_PAGES);
+		mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES,
+				    -(1L << order));
 		n_pages++;
 	}
 	spin_unlock_irqrestore(&balloon_pages_lock, flags);
@@ -135,8 +152,9 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
 EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
 
 /**
- * balloon_page_alloc - allocates a new page for insertion into the balloon
- *			page list.
+ * balloon_pages_alloc - allocates a new page of given order for insertion
+ *			 into the balloon page list.
+ * @order: allocation order
  *
  * Driver must call this function to properly allocate a new balloon page.
  * Driver must call balloon_page_enqueue before definitively removing the page
@@ -144,16 +162,35 @@ EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
  *
  * Return: struct page for the allocated page or NULL on allocation failure.
  */
-struct page *balloon_page_alloc(void)
+struct page *balloon_pages_alloc(unsigned int order)
 {
 	gfp_t gfp_flags = __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN;
 
-	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
+	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !order)
 		gfp_flags |= GFP_HIGHUSER_MOVABLE;
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	return alloc_page(gfp_flags);
+	if (order > 0)
+		gfp_flags |= __GFP_COMP;
+
+	return alloc_pages(gfp_flags, order);
+}
+EXPORT_SYMBOL_GPL(balloon_pages_alloc);
+
+/**
+ * balloon_page_alloc - allocates a new page for insertion into the balloon
+ *			page list.
+ *
+ * Driver must call this function to properly allocate a new balloon page.
+ * Driver must call balloon_page_enqueue before definitively removing the page
+ * from the guest system.
+ *
+ * Return: struct page for the allocated page or NULL on allocation failure.
+ */
+struct page *balloon_page_alloc(void)
+{
+	return balloon_pages_alloc(0);
 }
 EXPORT_SYMBOL_GPL(balloon_page_alloc);
 
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 11/13] virtio_balloon: unaccept free reporting ranges

[Kevin Loughlin]

In confidential computing guests with dynamic unaccepted memory
transitions, configure the free page reporting order to
unaccepted_memory_order(). Invoke mark_memory_unaccepted() on each
scatterlist range in virtballoon_free_page_report() before adding the
scatterlist to reporting_vq and kicking the host.

Unaccepting the pages prior to virtqueue_add_inbuf() and virtqueue_kick()
guarantees that the guest finishes PVALIDATE(validate=false) before the
host VMM can observe the reporting request in the virtqueue ring and
punch a hole in guest_memfd, eliminating any race where the guest
re-allocates and re-accepts a page before a deferred host hole-punch
executes.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_balloon.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index d162c162b496..e10bd792ef7c 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -213,7 +213,26 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i
 	struct virtio_balloon *vb =
 		container_of(pr_dev_info, struct virtio_balloon, pr_dev_info);
 	struct virtqueue *vq = vb->reporting_vq;
-	unsigned int unused, err;
+	unsigned int unused;
+	int err;
+
+	if (virtballoon_cc_needs_state_transition()) {
+		unsigned long unit_size = PAGE_SIZE << unaccepted_memory_order();
+		struct scatterlist *sgl;
+		unsigned int i;
+
+		for_each_sg(sg, sgl, nents, i) {
+			if (WARN_ON_ONCE(!IS_ALIGNED(sg_phys(sgl), unit_size) ||
+					 !IS_ALIGNED(sgl->length, unit_size)))
+				return -EINVAL;
+		}
+		for_each_sg(sg, sgl, nents, i) {
+			err = mark_memory_unaccepted(sg_phys(sgl), sgl->length);
+			if (err)
+				return err;
+			cond_resched();
+		}
+	}
 
 	/* We should always be able to add these buffers to an empty queue. */
 	err = virtqueue_add_inbuf(vq, sg, nents, vb, GFP_NOWAIT);
@@ -1142,8 +1161,11 @@ static int virtballoon_probe(struct virtio_device *vdev)
 		 * host's base page size. However, it needs more work to report
 		 * that value. The hard-coded order would be fine currently.
 		 */
+		if (virtballoon_cc_needs_state_transition())
+			vb->pr_dev_info.order = unaccepted_memory_order();
 #if defined(CONFIG_ARM64) && defined(CONFIG_ARM64_64K_PAGES)
-		vb->pr_dev_info.order = 5;
+		else
+			vb->pr_dev_info.order = 5;
 #endif
 
 		vb->pr_dev_info.capacity = capacity;
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 12/13] virtio_balloon: unaccept inflated balloon pages

[Kevin Loughlin]

When pages are allocated for balloon inflation in confidential
computing guests requiring page state transitions, mark the allocated
pages as unaccepted via mark_memory_unaccepted() before notifying the
host VMM. When those pages are later deflated and released via
put_page(), the page allocator routes them to zone->unaccepted_pages
for lazy re-acceptance on demand.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_balloon.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index e10bd792ef7c..217431e75751 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -329,6 +329,15 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
 		unsigned int pfns = (1 << page_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
 
 		list_del(&page->lru);
+		if (virtballoon_cc_needs_state_transition()) {
+			unsigned long page_size = (1UL << page_order) * PAGE_SIZE;
+			phys_addr_t paddr = page_to_phys(page);
+
+			if (mark_memory_unaccepted(paddr, page_size)) {
+				put_page(page);
+				continue;
+			}
+		}
 		balloon_page_enqueue(&vb->vb_dev_info, page);
 
 		vb->num_pages += pfns;
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 13/13] virtio_balloon: enforce ACCESS_PLATFORM in CoCo

[Kevin Loughlin]

In confidential computing environments (CC_ATTR_GUEST_MEM_ENCRYPT),
virtio devices require VIRTIO_F_ACCESS_PLATFORM to negotiate restricted
memory access and DMA bounce buffering for virtqueue rings and
metadata buffers.

Currently, virtballoon_validate() unconditionally clears
VIRTIO_F_ACCESS_PLATFORM, causing virtio_features_ok() to reject probing
the balloon device (-ENODEV) when restricted memory access is active.

Preserve VIRTIO_F_ACCESS_PLATFORM in virtballoon_validate() when
CC_ATTR_GUEST_MEM_ENCRYPT is active, clear VIRTIO_BALLOON_F_FREE_PAGE_HINT
in CoCo since free_page_vq maps MAX_PAGE_ORDER blocks through standard DMA
ops without unaccepting them. Reject probing (-ENODEV) on CoCo
platforms where balloon page state transitions are unsupported.

Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>
---
 drivers/virtio/virtio_balloon.c | 36 ++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 217431e75751..ae011636c07f 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -1311,6 +1311,14 @@ static int virtballoon_restore(struct virtio_device *vdev)
 }
 #endif
 
+static bool virtballoon_cc_supported(void)
+{
+	if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
+		return has_unaccepted_memory();
+
+	return cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT);
+}
+
 static int virtballoon_validate(struct virtio_device *vdev)
 {
 	/*
@@ -1318,7 +1326,17 @@ static int virtballoon_validate(struct virtio_device *vdev)
 	 * initialized. If we cannot do that then we should disable
 	 * page reporting as it could potentially change the contents
 	 * of our free pages.
+	 *
+	 * In encrypted guests without unaccepted memory (e.g. SEV-ES), the
+	 * hypervisor cannot populate discarded pages with guest-encrypted
+	 * poison/zero values and the guest does not re-initialize reported
+	 * pages on return. Clear VIRTIO_BALLOON_F_PAGE_POISON so reporting
+	 * is disabled when init_on_free or page_poison is active.
 	 */
+	if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    !has_unaccepted_memory())
+		__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_PAGE_POISON);
+
 	if (!want_init_on_free() && !page_poisoning_enabled_static())
 		__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_PAGE_POISON);
 	else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON))
@@ -1329,7 +1347,23 @@ static int virtballoon_validate(struct virtio_device *vdev)
 	 * virtqueue_add during page reporting.
 	 */
 	__virtio_clear_bit(vdev, VIRTIO_RING_F_INDIRECT_DESC);
-	__virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
+
+	/*
+	 * The balloon bypasses the DMA API for page addresses and cannot
+	 * support a physical or virtual IOMMU on non-CoCo guests. In CoCo
+	 * guests, VIRTIO_F_ACCESS_PLATFORM is required for bounce-buffering
+	 * virtqueue rings and metadata. Disable free page hinting in CoCo
+	 * guests as free_page_vq maps MAX_PAGE_ORDER blocks through standard
+	 * DMA ops, and reject probing on CoCo platforms where balloon page
+	 * state transitions are not supported.
+	 */
+	if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
+		__virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
+	} else {
+		if (!virtballoon_cc_supported())
+			return -ENODEV;
+		__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT);
+	}
 	return 0;
 }
 
-- 
2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 10/13] virtio_balloon: add higher-order page inflation

[Michael S. Tsirkin]

On Tue, Oct 27, 2026 at 02:56:13AM +0000, Kevin Loughlin wrote:
> In confidential computing guests requiring page state transitions,
> unaccepted memory tracking operates at the EFI unaccepted memory unit
> granularity (typically 2MB / order 9 on x86). Inflating the balloon
> with 4KB pages would prevent unaccepting the backing 2MB unit.
> 
> Inflate balloon pages at unaccepted_memory_order() when
> virtballoon_cc_needs_state_transition() is true, while retaining
> order-0 allocations for non-CoCo guests and CoCo guests that do not
> use unaccepted memory transitions. When a higher-order balloon page
> spans more 4KB balloon PFNs than VIRTIO_BALLOON_ARRAY_PFNS_MAX (256),
> transmit its PFNs to the host in VIRTIO_BALLOON_ARRAY_PFNS_MAX chunks.
> 
> Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>

I'm not too happy that this rides on the legacy PFN thing
which is after all limited to 44 bit addresses.

And given it is passing around 2MB chunks, then one 16 byte descriptor
covering all of 2MB would actually be more efficient than 512 4-byte
PFNs.


> ---
>  drivers/virtio/virtio_balloon.c | 87 ++++++++++++++++++++++++++-------
>  include/linux/balloon.h         |  1 +
>  mm/balloon.c                    | 67 +++++++++++++++++++------
>  3 files changed, 121 insertions(+), 34 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 11f74848fdc4..d162c162b496 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -182,6 +182,13 @@ static void balloon_ack(struct virtqueue *vq)
>  	wake_up(&vb->acked);
>  }
>  
> +static inline bool virtballoon_cc_needs_state_transition(void)
> +{
> +	return IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) &&
> +	       cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
> +	       has_unaccepted_memory();
> +}
> +
>  static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
>  {
>  	struct scatterlist sg;


Put all this, and preferred order, etc etc in kernel core somewhere,
and expose some sensible APIs that devices can use.



> @@ -243,24 +250,49 @@ static void set_page_pfns(struct virtio_balloon *vb,
>  					  page_to_balloon_pfn(page) + i);
>  }
>  
> +static void queue_page_pfns(struct virtio_balloon *vb, struct virtqueue *vq,
> +			    struct page *page, unsigned int order)
> +{
> +	unsigned int nr_pages = 1 << order;
> +	unsigned int i;
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		set_page_pfns(vb, vb->pfns + vb->num_pfns, page + i);
> +		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		if (vb->num_pfns == ARRAY_SIZE(vb->pfns)) {
> +			tell_host(vb, vq);
> +			vb->num_pfns = 0;
> +		}
> +	}
> +}
> +
> +static inline unsigned int virtballoon_preferred_order(void)
> +{
> +	if (virtballoon_cc_needs_state_transition())
> +		return unaccepted_memory_order();
> +	return 0;
> +}
> +
>  static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
>  {
> -	unsigned int num_allocated_pages;
> +	unsigned int order = virtballoon_preferred_order();
> +	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +	unsigned int num_allocated_pages = 0;
>  	struct page *page, *next;
>  	unsigned int num_pfns;
>  	LIST_HEAD(pages);
>  
> -	/* We can only do one array worth at a time. */
> -	num = min(num, ARRAY_SIZE(vb->pfns));
> +	/* We can only do one array or preferred-order page worth at a time. */
> +	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
>  
>  	for (num_pfns = 0; num_pfns < num;
> -	     num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
> -		page = balloon_page_alloc();
> +	     num_pfns += page_pfns) {
> +		page = balloon_pages_alloc(order);
>  
>  		if (!page) {
>  			dev_info_ratelimited(&vb->vdev->dev,
>  					     "Out of puff! Can't get %u pages\n",
> -					     VIRTIO_BALLOON_PAGES_PER_PAGE);
> +					     page_pfns);
>  			/* Sleep for at least 1/5 of a second before retry. */
>  			msleep(200);
>  			break;
> @@ -274,15 +306,17 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
>  	vb->num_pfns = 0;
>  
>  	list_for_each_entry_safe(page, next, &pages, lru) {
> +		unsigned int page_order = compound_order(page);
> +		unsigned int pfns = (1 << page_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +
>  		list_del(&page->lru);
>  		balloon_page_enqueue(&vb->vb_dev_info, page);
>  
> -		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
> -		vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
> -		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		vb->num_pages += pfns;
> +		num_allocated_pages += pfns;
> +		queue_page_pfns(vb, vb->inflate_vq, page, page_order);
>  	}
>  
> -	num_allocated_pages = vb->num_pfns;
>  	/* Did we get any? */
>  	if (vb->num_pfns != 0)
>  		tell_host(vb, vb->inflate_vq);
> @@ -304,28 +338,42 @@ static void release_pages_balloon(struct virtio_balloon *vb,
>  
>  static unsigned int leak_balloon(struct virtio_balloon *vb, size_t num)
>  {
> -	unsigned int num_freed_pages;
> +	unsigned int order = virtballoon_preferred_order();
> +	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +	unsigned int num_freed_pages = 0;
>  	struct page *page;
>  	struct balloon_dev_info *vb_dev_info = &vb->vb_dev_info;
>  	LIST_HEAD(pages);
>  
> -	/* We can only do one array worth at a time. */
> -	num = min(num, ARRAY_SIZE(vb->pfns));
> +	/* We can only do one array or preferred-order page worth at a time. */
> +	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
>  
>  	mutex_lock(&vb->balloon_lock);
>  	/* We can't release more pages than taken */
>  	num = min(num, (size_t)vb->num_pages);
> -	for (vb->num_pfns = 0; vb->num_pfns < num;
> -	     vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
> +	vb->num_pfns = 0;
> +	while (num_freed_pages < num) {
> +		unsigned int dequeued_order;
> +		unsigned int dequeued_pfns;
> +
>  		page = balloon_page_dequeue(vb_dev_info);
>  		if (!page)
>  			break;
> -		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
> +		dequeued_order = compound_order(page);
> +		dequeued_pfns = (1 << dequeued_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +
> +		if (num_freed_pages + dequeued_pfns > num &&
> +		    num_freed_pages > 0) {
> +			balloon_page_enqueue(vb_dev_info, page);
> +			break;
> +		}
> +
> +		queue_page_pfns(vb, vb->deflate_vq, page, dequeued_order);
>  		list_add(&page->lru, &pages);
> -		vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		vb->num_pages -= dequeued_pfns;
> +		num_freed_pages += dequeued_pfns;
>  	}
>  
> -	num_freed_pages = vb->num_pfns;
>  	/*
>  	 * Note that if
>  	 * virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
> @@ -474,7 +522,8 @@ static inline s64 towards_target(struct virtio_balloon *vb)
>  	 * Aligned up to guest page size to avoid inflating and deflating
>  	 * balloon endlessly.
>  	 */
> -	target = ALIGN(num_pages, VIRTIO_BALLOON_PAGES_PER_PAGE);
> +	target = ALIGN((s64)num_pages,
> +		       VIRTIO_BALLOON_PAGES_PER_PAGE << virtballoon_preferred_order());
>  	return target - vb->num_pages;
>  }
>  
> diff --git a/include/linux/balloon.h b/include/linux/balloon.h
> index ca5b15150f42..bc46e36b997f 100644
> --- a/include/linux/balloon.h
> +++ b/include/linux/balloon.h
> @@ -58,6 +58,7 @@ struct balloon_dev_info {
>  	bool adjust_managed_page_count;
>  };
>  
> +struct page *balloon_pages_alloc(unsigned int order);
>  struct page *balloon_page_alloc(void);
>  void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
>  		struct page *page);
> diff --git a/mm/balloon.c b/mm/balloon.c
> index 96a8f1e20bc6..e9941d08d8e9 100644
> --- a/mm/balloon.c
> +++ b/mm/balloon.c
> @@ -27,9 +27,12 @@ static DEFINE_SPINLOCK(balloon_pages_lock);
>  static void balloon_page_insert(struct balloon_dev_info *balloon,
>  				       struct page *page)
>  {
> +	unsigned int i, nr_pages = 1U << compound_order(page);
> +
>  	lockdep_assert_held(&balloon_pages_lock);
> -	__SetPageOffline(page);
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) {
> +	for (i = 0; i < nr_pages; i++)
> +		__SetPageOffline(page + i);
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page)) {
>  		SetPageMovableOps(page);
>  		set_page_private(page, (unsigned long)balloon);
>  	}
> @@ -45,20 +48,31 @@ static void balloon_page_insert(struct balloon_dev_info *balloon,
>   */
>  static void balloon_page_finalize(struct page *page)
>  {
> +	unsigned int i, nr_pages = 1U << compound_order(page);
> +
>  	lockdep_assert_held(&balloon_pages_lock);
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page))
>  		set_page_private(page, 0);
> -	/* PageOffline is sticky until the page is freed to the buddy. */
> +	/*
> +	 * PageOffline is sticky on the head page until freed to the buddy,
> +	 * which resets the head page's page_type in __free_pages_prepare().
> +	 * Clear PageOffline on tail pages here as __free_pages_prepare()
> +	 * expects tail pages to have a default page_type.
> +	 */
> +	for (i = 1; i < nr_pages; i++)
> +		__ClearPageOffline(page + i);
>  }
>  
>  static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info,
>  				     struct page *page)
>  {
> +	unsigned int order = compound_order(page);
> +
>  	balloon_page_insert(b_dev_info, page);
>  	if (b_dev_info->adjust_managed_page_count)
> -		adjust_managed_page_count(page, -1);
> -	__count_vm_event(BALLOON_INFLATE);
> -	inc_node_page_state(page, NR_BALLOON_PAGES);
> +		adjust_managed_page_count(page, -(1L << order));
> +	__count_vm_events(BALLOON_INFLATE, 1UL << order);
> +	mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES, 1L << order);
>  }
>  
>  /**
> @@ -117,15 +131,18 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
>  
>  	spin_lock_irqsave(&balloon_pages_lock, flags);
>  	list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) {
> +		unsigned int order = compound_order(page);
> +
>  		if (n_pages == n_req_pages)
>  			break;
>  		list_del(&page->lru);
>  		if (b_dev_info->adjust_managed_page_count)
> -			adjust_managed_page_count(page, 1);
> +			adjust_managed_page_count(page, 1L << order);
>  		balloon_page_finalize(page);
> -		__count_vm_event(BALLOON_DEFLATE);
> +		__count_vm_events(BALLOON_DEFLATE, 1UL << order);
>  		list_add(&page->lru, pages);
> -		dec_node_page_state(page, NR_BALLOON_PAGES);
> +		mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES,
> +				    -(1L << order));
>  		n_pages++;
>  	}
>  	spin_unlock_irqrestore(&balloon_pages_lock, flags);
> @@ -135,8 +152,9 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
>  EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
>  
>  /**
> - * balloon_page_alloc - allocates a new page for insertion into the balloon
> - *			page list.
> + * balloon_pages_alloc - allocates a new page of given order for insertion
> + *			 into the balloon page list.
> + * @order: allocation order
>   *
>   * Driver must call this function to properly allocate a new balloon page.
>   * Driver must call balloon_page_enqueue before definitively removing the page
> @@ -144,16 +162,35 @@ EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
>   *
>   * Return: struct page for the allocated page or NULL on allocation failure.
>   */
> -struct page *balloon_page_alloc(void)
> +struct page *balloon_pages_alloc(unsigned int order)
>  {
>  	gfp_t gfp_flags = __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN;
>  
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !order)
>  		gfp_flags |= GFP_HIGHUSER_MOVABLE;
>  	else
>  		gfp_flags |= GFP_HIGHUSER;
>  
> -	return alloc_page(gfp_flags);
> +	if (order > 0)
> +		gfp_flags |= __GFP_COMP;
> +
> +	return alloc_pages(gfp_flags, order);
> +}
> +EXPORT_SYMBOL_GPL(balloon_pages_alloc);
> +
> +/**
> + * balloon_page_alloc - allocates a new page for insertion into the balloon
> + *			page list.
> + *
> + * Driver must call this function to properly allocate a new balloon page.
> + * Driver must call balloon_page_enqueue before definitively removing the page
> + * from the guest system.
> + *
> + * Return: struct page for the allocated page or NULL on allocation failure.
> + */
> +struct page *balloon_page_alloc(void)
> +{
> +	return balloon_pages_alloc(0);
>  }
>  EXPORT_SYMBOL_GPL(balloon_page_alloc);
>  
> -- 
> 2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 10/13] virtio_balloon: add higher-order page inflation

[Michael S. Tsirkin]

On Tue, Oct 27, 2026 at 02:56:13AM +0000, Kevin Loughlin wrote:
> In confidential computing guests requiring page state transitions,
> unaccepted memory tracking operates at the EFI unaccepted memory unit
> granularity (typically 2MB / order 9 on x86). Inflating the balloon
> with 4KB pages would prevent unaccepting the backing 2MB unit.
> 
> Inflate balloon pages at unaccepted_memory_order() when
> virtballoon_cc_needs_state_transition() is true, while retaining
> order-0 allocations for non-CoCo guests and CoCo guests that do not
> use unaccepted memory transitions. When a higher-order balloon page
> spans more 4KB balloon PFNs than VIRTIO_BALLOON_ARRAY_PFNS_MAX (256),
> transmit its PFNs to the host in VIRTIO_BALLOON_ARRAY_PFNS_MAX chunks.
> 
> Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>

BTW I'd say *everyone* prefers large pages. The fact that balloon
currently breaks them up isn't nice to anyone. The only setup
where they might be helpful is severe fragmentation, but even then,
it is not clear we aren't contributing to the problem
by making the fragmentation worse.


> ---
>  drivers/virtio/virtio_balloon.c | 87 ++++++++++++++++++++++++++-------
>  include/linux/balloon.h         |  1 +
>  mm/balloon.c                    | 67 +++++++++++++++++++------
>  3 files changed, 121 insertions(+), 34 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 11f74848fdc4..d162c162b496 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -182,6 +182,13 @@ static void balloon_ack(struct virtqueue *vq)
>  	wake_up(&vb->acked);
>  }
>  
> +static inline bool virtballoon_cc_needs_state_transition(void)
> +{
> +	return IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) &&
> +	       cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
> +	       has_unaccepted_memory();
> +}
> +
>  static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
>  {
>  	struct scatterlist sg;
> @@ -243,24 +250,49 @@ static void set_page_pfns(struct virtio_balloon *vb,
>  					  page_to_balloon_pfn(page) + i);
>  }
>  
> +static void queue_page_pfns(struct virtio_balloon *vb, struct virtqueue *vq,
> +			    struct page *page, unsigned int order)
> +{
> +	unsigned int nr_pages = 1 << order;
> +	unsigned int i;
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		set_page_pfns(vb, vb->pfns + vb->num_pfns, page + i);
> +		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		if (vb->num_pfns == ARRAY_SIZE(vb->pfns)) {
> +			tell_host(vb, vq);
> +			vb->num_pfns = 0;
> +		}
> +	}
> +}
> +
> +static inline unsigned int virtballoon_preferred_order(void)
> +{
> +	if (virtballoon_cc_needs_state_transition())
> +		return unaccepted_memory_order();
> +	return 0;
> +}
> +
>  static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
>  {
> -	unsigned int num_allocated_pages;
> +	unsigned int order = virtballoon_preferred_order();
> +	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +	unsigned int num_allocated_pages = 0;
>  	struct page *page, *next;
>  	unsigned int num_pfns;
>  	LIST_HEAD(pages);
>  
> -	/* We can only do one array worth at a time. */
> -	num = min(num, ARRAY_SIZE(vb->pfns));
> +	/* We can only do one array or preferred-order page worth at a time. */
> +	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
>  
>  	for (num_pfns = 0; num_pfns < num;
> -	     num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
> -		page = balloon_page_alloc();
> +	     num_pfns += page_pfns) {
> +		page = balloon_pages_alloc(order);
>  
>  		if (!page) {
>  			dev_info_ratelimited(&vb->vdev->dev,
>  					     "Out of puff! Can't get %u pages\n",
> -					     VIRTIO_BALLOON_PAGES_PER_PAGE);
> +					     page_pfns);
>  			/* Sleep for at least 1/5 of a second before retry. */
>  			msleep(200);
>  			break;
> @@ -274,15 +306,17 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
>  	vb->num_pfns = 0;
>  
>  	list_for_each_entry_safe(page, next, &pages, lru) {
> +		unsigned int page_order = compound_order(page);
> +		unsigned int pfns = (1 << page_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +
>  		list_del(&page->lru);
>  		balloon_page_enqueue(&vb->vb_dev_info, page);
>  
> -		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
> -		vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
> -		vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		vb->num_pages += pfns;
> +		num_allocated_pages += pfns;
> +		queue_page_pfns(vb, vb->inflate_vq, page, page_order);
>  	}
>  
> -	num_allocated_pages = vb->num_pfns;
>  	/* Did we get any? */
>  	if (vb->num_pfns != 0)
>  		tell_host(vb, vb->inflate_vq);
> @@ -304,28 +338,42 @@ static void release_pages_balloon(struct virtio_balloon *vb,
>  
>  static unsigned int leak_balloon(struct virtio_balloon *vb, size_t num)
>  {
> -	unsigned int num_freed_pages;
> +	unsigned int order = virtballoon_preferred_order();
> +	unsigned int page_pfns = (1 << order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +	unsigned int num_freed_pages = 0;
>  	struct page *page;
>  	struct balloon_dev_info *vb_dev_info = &vb->vb_dev_info;
>  	LIST_HEAD(pages);
>  
> -	/* We can only do one array worth at a time. */
> -	num = min(num, ARRAY_SIZE(vb->pfns));
> +	/* We can only do one array or preferred-order page worth at a time. */
> +	num = min(num, max_t(size_t, ARRAY_SIZE(vb->pfns), page_pfns));
>  
>  	mutex_lock(&vb->balloon_lock);
>  	/* We can't release more pages than taken */
>  	num = min(num, (size_t)vb->num_pages);
> -	for (vb->num_pfns = 0; vb->num_pfns < num;
> -	     vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
> +	vb->num_pfns = 0;
> +	while (num_freed_pages < num) {
> +		unsigned int dequeued_order;
> +		unsigned int dequeued_pfns;
> +
>  		page = balloon_page_dequeue(vb_dev_info);
>  		if (!page)
>  			break;
> -		set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
> +		dequeued_order = compound_order(page);
> +		dequeued_pfns = (1 << dequeued_order) * VIRTIO_BALLOON_PAGES_PER_PAGE;
> +
> +		if (num_freed_pages + dequeued_pfns > num &&
> +		    num_freed_pages > 0) {
> +			balloon_page_enqueue(vb_dev_info, page);
> +			break;
> +		}
> +
> +		queue_page_pfns(vb, vb->deflate_vq, page, dequeued_order);
>  		list_add(&page->lru, &pages);
> -		vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
> +		vb->num_pages -= dequeued_pfns;
> +		num_freed_pages += dequeued_pfns;
>  	}
>  
> -	num_freed_pages = vb->num_pfns;
>  	/*
>  	 * Note that if
>  	 * virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
> @@ -474,7 +522,8 @@ static inline s64 towards_target(struct virtio_balloon *vb)
>  	 * Aligned up to guest page size to avoid inflating and deflating
>  	 * balloon endlessly.
>  	 */
> -	target = ALIGN(num_pages, VIRTIO_BALLOON_PAGES_PER_PAGE);
> +	target = ALIGN((s64)num_pages,
> +		       VIRTIO_BALLOON_PAGES_PER_PAGE << virtballoon_preferred_order());
>  	return target - vb->num_pages;
>  }
>  
> diff --git a/include/linux/balloon.h b/include/linux/balloon.h
> index ca5b15150f42..bc46e36b997f 100644
> --- a/include/linux/balloon.h
> +++ b/include/linux/balloon.h
> @@ -58,6 +58,7 @@ struct balloon_dev_info {
>  	bool adjust_managed_page_count;
>  };
>  
> +struct page *balloon_pages_alloc(unsigned int order);
>  struct page *balloon_page_alloc(void);
>  void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
>  		struct page *page);
> diff --git a/mm/balloon.c b/mm/balloon.c
> index 96a8f1e20bc6..e9941d08d8e9 100644
> --- a/mm/balloon.c
> +++ b/mm/balloon.c
> @@ -27,9 +27,12 @@ static DEFINE_SPINLOCK(balloon_pages_lock);
>  static void balloon_page_insert(struct balloon_dev_info *balloon,
>  				       struct page *page)
>  {
> +	unsigned int i, nr_pages = 1U << compound_order(page);
> +
>  	lockdep_assert_held(&balloon_pages_lock);
> -	__SetPageOffline(page);
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) {
> +	for (i = 0; i < nr_pages; i++)
> +		__SetPageOffline(page + i);
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page)) {
>  		SetPageMovableOps(page);
>  		set_page_private(page, (unsigned long)balloon);
>  	}
> @@ -45,20 +48,31 @@ static void balloon_page_insert(struct balloon_dev_info *balloon,
>   */
>  static void balloon_page_finalize(struct page *page)
>  {
> +	unsigned int i, nr_pages = 1U << compound_order(page);
> +
>  	lockdep_assert_held(&balloon_pages_lock);
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !PageCompound(page))
>  		set_page_private(page, 0);
> -	/* PageOffline is sticky until the page is freed to the buddy. */
> +	/*
> +	 * PageOffline is sticky on the head page until freed to the buddy,
> +	 * which resets the head page's page_type in __free_pages_prepare().
> +	 * Clear PageOffline on tail pages here as __free_pages_prepare()
> +	 * expects tail pages to have a default page_type.
> +	 */
> +	for (i = 1; i < nr_pages; i++)
> +		__ClearPageOffline(page + i);
>  }
>  
>  static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info,
>  				     struct page *page)
>  {
> +	unsigned int order = compound_order(page);
> +
>  	balloon_page_insert(b_dev_info, page);
>  	if (b_dev_info->adjust_managed_page_count)
> -		adjust_managed_page_count(page, -1);
> -	__count_vm_event(BALLOON_INFLATE);
> -	inc_node_page_state(page, NR_BALLOON_PAGES);
> +		adjust_managed_page_count(page, -(1L << order));
> +	__count_vm_events(BALLOON_INFLATE, 1UL << order);
> +	mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES, 1L << order);
>  }
>  
>  /**
> @@ -117,15 +131,18 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
>  
>  	spin_lock_irqsave(&balloon_pages_lock, flags);
>  	list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) {
> +		unsigned int order = compound_order(page);
> +
>  		if (n_pages == n_req_pages)
>  			break;
>  		list_del(&page->lru);
>  		if (b_dev_info->adjust_managed_page_count)
> -			adjust_managed_page_count(page, 1);
> +			adjust_managed_page_count(page, 1L << order);
>  		balloon_page_finalize(page);
> -		__count_vm_event(BALLOON_DEFLATE);
> +		__count_vm_events(BALLOON_DEFLATE, 1UL << order);
>  		list_add(&page->lru, pages);
> -		dec_node_page_state(page, NR_BALLOON_PAGES);
> +		mod_node_page_state(page_pgdat(page), NR_BALLOON_PAGES,
> +				    -(1L << order));
>  		n_pages++;
>  	}
>  	spin_unlock_irqrestore(&balloon_pages_lock, flags);
> @@ -135,8 +152,9 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
>  EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
>  
>  /**
> - * balloon_page_alloc - allocates a new page for insertion into the balloon
> - *			page list.
> + * balloon_pages_alloc - allocates a new page of given order for insertion
> + *			 into the balloon page list.
> + * @order: allocation order
>   *
>   * Driver must call this function to properly allocate a new balloon page.
>   * Driver must call balloon_page_enqueue before definitively removing the page
> @@ -144,16 +162,35 @@ EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
>   *
>   * Return: struct page for the allocated page or NULL on allocation failure.
>   */
> -struct page *balloon_page_alloc(void)
> +struct page *balloon_pages_alloc(unsigned int order)
>  {
>  	gfp_t gfp_flags = __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN;
>  
> -	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
> +	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION) && !order)
>  		gfp_flags |= GFP_HIGHUSER_MOVABLE;
>  	else
>  		gfp_flags |= GFP_HIGHUSER;
>  
> -	return alloc_page(gfp_flags);
> +	if (order > 0)
> +		gfp_flags |= __GFP_COMP;
> +
> +	return alloc_pages(gfp_flags, order);
> +}
> +EXPORT_SYMBOL_GPL(balloon_pages_alloc);
> +
> +/**
> + * balloon_page_alloc - allocates a new page for insertion into the balloon
> + *			page list.
> + *
> + * Driver must call this function to properly allocate a new balloon page.
> + * Driver must call balloon_page_enqueue before definitively removing the page
> + * from the guest system.
> + *
> + * Return: struct page for the allocated page or NULL on allocation failure.
> + */
> +struct page *balloon_page_alloc(void)
> +{
> +	return balloon_pages_alloc(0);
>  }
>  EXPORT_SYMBOL_GPL(balloon_page_alloc);
>  
> -- 
> 2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 01/13] virtio_ring: support per-virtqueue DMA mapping ops

[Michael S. Tsirkin]

On Tue, Oct 27, 2026 at 02:56:04AM +0000, Kevin Loughlin wrote:
> Allow individual virtqueues to override the device-wide DMA mapping
> operations via virtqueue_set_map_ops(). When custom per-virtqueue
> mapping operations are configured, vring_map_one_sg(), vring_unmap_one(),
> vring_mapping_error(), and the DMA sync helpers dispatch through
> vq->map_ops instead of the parent virtio_device DMA API path.
> 
> Replace vring_map_single() with vring_map_indirect() and
> vring_unmap_indirect() using the device-level mapping path (vdev->map
> or dma_map_page_attrs()) so that indirect descriptor tables continue to
> be DMA-mapped for host VMM access even when a virtqueue overrides buffer
> mapping via vq->map_ops.
> 
> Signed-off-by: Kevin Loughlin <kevinloughlin@google.com>

Please, don't. The automatic mappings inside virtio core are there
to keep simple things, simple.

Everyone doing something funky with mappings should use the premapped API.


> ---
>  drivers/virtio/virtio_ring.c | 136 ++++++++++++++++++++++++-----------
>  include/linux/virtio.h       |   4 ++
>  2 files changed, 98 insertions(+), 42 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index db678f5a80e0..9c6a1e9f0a0b 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -261,6 +261,7 @@ struct vring_virtqueue {
>  	bool we_own_ring;
>  
>  	union virtio_map map;
> +	const struct virtio_map_ops *map_ops;
>  
>  #ifdef DEBUG
>  	/* They're supposed to lock for us. */
> @@ -477,16 +478,32 @@ static struct device *vring_dma_dev(const struct vring_virtqueue *vq)
>  	return vq->map.dma_dev;
>  }
>  
> +static inline const struct virtio_map_ops *vring_get_map_ops(const struct vring_virtqueue *vq)
> +{
> +	if (vq->map_ops)
> +		return vq->map_ops;
> +	return vq->vq.vdev->map;
> +}
> +
> +void virtqueue_set_map_ops(struct virtqueue *_vq,
> +			   const struct virtio_map_ops *map_ops)
> +{
> +	struct vring_virtqueue *vq = to_vvq(_vq);
> +
> +	vq->map_ops = map_ops;
> +}
> +EXPORT_SYMBOL_GPL(virtqueue_set_map_ops);
> +
>  static int vring_mapping_error(const struct vring_virtqueue *vq,
>  			       dma_addr_t addr)
>  {
> -	struct virtio_device *vdev = vq->vq.vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
>  	if (!vq->use_map_api)
>  		return 0;
>  
> -	if (vdev->map)
> -		return vdev->map->mapping_error(vq->map, addr);
> +	if (map_ops)
> +		return map_ops->mapping_error(vq->map, addr);
>  	else
>  		return dma_mapping_error(vring_dma_dev(vq), addr);
>  }
> @@ -530,15 +547,45 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist
>  	return 0;
>  }
>  
> -static dma_addr_t vring_map_single(const struct vring_virtqueue *vq,
> -				   void *cpu_addr, size_t size,
> -				   enum dma_data_direction direction)
> +static int vring_map_indirect(const struct vring_virtqueue *vq, void *cpu_addr,
> +			      size_t size, dma_addr_t *addr)
>  {
> -	if (!vq->use_map_api)
> -		return (dma_addr_t)virt_to_phys(cpu_addr);
> +	struct virtio_device *vdev = vq->vq.vdev;
> +
> +	if (!vq->use_map_api) {
> +		kmsan_handle_dma(virt_to_phys(cpu_addr), size, DMA_TO_DEVICE);
> +		*addr = (dma_addr_t)virt_to_phys(cpu_addr);
> +		return 0;
> +	}
> +
> +	if (vdev->map) {
> +		*addr = vdev->map->map_page(vq->map, virt_to_page(cpu_addr),
> +					    offset_in_page(cpu_addr), size,
> +					    DMA_TO_DEVICE, 0);
> +		if (vdev->map->mapping_error(vq->map, *addr))
> +			return -ENOMEM;
> +		return 0;
> +	}
> +
> +	*addr = dma_map_page_attrs(vring_dma_dev(vq), virt_to_page(cpu_addr),
> +				   offset_in_page(cpu_addr), size,
> +				   DMA_TO_DEVICE, 0);
> +	if (dma_mapping_error(vring_dma_dev(vq), *addr))
> +		return -ENOMEM;
>  
> -	return virtqueue_map_single_attrs(&vq->vq, cpu_addr,
> -					  size, direction, 0);
> +	return 0;
> +}
> +
> +static void vring_unmap_indirect(const struct vring_virtqueue *vq,
> +				 dma_addr_t addr, size_t size)
> +{
> +	struct virtio_device *vdev = vq->vq.vdev;
> +
> +	if (vdev->map)
> +		vdev->map->unmap_page(vq->map, addr, size, DMA_TO_DEVICE, 0);
> +	else
> +		dma_unmap_page_attrs(vring_dma_dev(vq), addr, size,
> +				     DMA_TO_DEVICE, 0);
>  }
>  
>  static void virtqueue_init(struct vring_virtqueue *vq, u32 num)
> @@ -576,6 +623,8 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq,
>  	if (flags & VRING_DESC_F_INDIRECT) {
>  		if (!vq->use_map_api)
>  			goto out;
> +		vring_unmap_indirect(vq, extra->addr, extra->len);
> +		goto out;
>  	} else if (!vring_need_unmap_buffer(vq, extra))
>  		goto out;
>  
> @@ -761,10 +810,11 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
>  
>  	if (indirect) {
>  		/* Now that the indirect table is filled in, map it. */
> -		dma_addr_t addr = vring_map_single(
> -			vq, desc, total_sg * sizeof(struct vring_desc),
> -			DMA_TO_DEVICE);
> -		if (vring_mapping_error(vq, addr))
> +		dma_addr_t addr;
> +
> +		if (vring_map_indirect(vq, desc,
> +				       total_sg * sizeof(struct vring_desc),
> +				       &addr))
>  			goto unmap_release;
>  
>  		virtqueue_add_desc_split(vq, vq->split.vring.desc,
> @@ -1370,6 +1420,7 @@ static struct virtqueue *__vring_new_virtqueue_split(unsigned int index,
>  #endif
>  	vq->map = map;
>  	vq->use_map_api = vring_use_map_api(vdev);
> +	vq->map_ops = NULL;
>  
>  	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
>  		!context;
> @@ -1488,6 +1539,8 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq,
>  	if (flags & VRING_DESC_F_INDIRECT) {
>  		if (!vq->use_map_api)
>  			return;
> +		vring_unmap_indirect(vq, extra->addr, extra->len);
> +		return;
>  	} else if (!vring_need_unmap_buffer(vq, extra))
>  		return;
>  
> @@ -1585,10 +1638,9 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
>  	}
>  
>  	/* Now that the indirect table is filled in, map it. */
> -	addr = vring_map_single(vq, desc,
> -			total_sg * sizeof(struct vring_packed_desc),
> -			DMA_TO_DEVICE);
> -	if (vring_mapping_error(vq, addr))
> +	if (vring_map_indirect(vq, desc,
> +			       total_sg * sizeof(struct vring_packed_desc),
> +			       &addr))
>  		goto unmap_release;
>  
>  	vq->packed.vring.desc[head].addr = cpu_to_le64(addr);
> @@ -2620,6 +2672,7 @@ static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index,
>  #endif
>  	vq->map = map;
>  	vq->use_map_api = vring_use_map_api(vdev);
> +	vq->map_ops = NULL;
>  
>  	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
>  		!context;
> @@ -3064,7 +3117,7 @@ struct device *virtqueue_dma_dev(struct virtqueue *_vq)
>  {
>  	struct vring_virtqueue *vq = to_vvq(_vq);
>  
> -	if (vq->use_map_api && !_vq->vdev->map)
> +	if (vq->use_map_api && !vring_get_map_ops(vq))
>  		return vq->map.dma_dev;
>  	else
>  		return NULL;
> @@ -3809,18 +3862,17 @@ EXPORT_SYMBOL_GPL(virtqueue_map_free_coherent);
>   */
>  dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
>  				    struct page *page,
> -				    unsigned long offset,
> -				    size_t size,
> +				    unsigned long offset, size_t size,
>  				    enum dma_data_direction dir,
>  				    unsigned long attrs)
>  {
>  	const struct vring_virtqueue *vq = to_vvq(_vq);
> -	struct virtio_device *vdev = _vq->vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
> -	if (vdev->map)
> -		return vdev->map->map_page(vq->map,
> -					   page, offset, size,
> -					   dir, attrs);
> +	if (map_ops)
> +		return map_ops->map_page(vq->map,
> +					 page, offset, size,
> +					 dir, attrs);
>  
>  	return dma_map_page_attrs(vring_dma_dev(vq),
>  				  page, offset, size,
> @@ -3842,11 +3894,11 @@ void virtqueue_unmap_page_attrs(const struct virtqueue *_vq,
>  				unsigned long attrs)
>  {
>  	const struct vring_virtqueue *vq = to_vvq(_vq);
> -	struct virtio_device *vdev = _vq->vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
> -	if (vdev->map)
> -		vdev->map->unmap_page(vq->map,
> -				      map_handle, size, dir, attrs);
> +	if (map_ops)
> +		map_ops->unmap_page(vq->map,
> +				    map_handle, size, dir, attrs);
>  	else
>  		dma_unmap_page_attrs(vring_dma_dev(vq), map_handle,
>  				     size, dir, attrs);
> @@ -3941,13 +3993,13 @@ EXPORT_SYMBOL_GPL(virtqueue_map_mapping_error);
>  bool virtqueue_map_need_sync(const struct virtqueue *_vq, dma_addr_t addr)
>  {
>  	const struct vring_virtqueue *vq = to_vvq(_vq);
> -	struct virtio_device *vdev = _vq->vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
>  	if (!vq->use_map_api)
>  		return false;
>  
> -	if (vdev->map)
> -		return vdev->map->need_sync(vq->map, addr);
> +	if (map_ops)
> +		return map_ops->need_sync(vq->map, addr);
>  	else
>  		return dma_need_sync(vring_dma_dev(vq), addr);
>  }
> @@ -3971,14 +4023,14 @@ void virtqueue_map_sync_single_range_for_cpu(const struct virtqueue *_vq,
>  					     enum dma_data_direction dir)
>  {
>  	const struct vring_virtqueue *vq = to_vvq(_vq);
> -	struct virtio_device *vdev = _vq->vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
>  	if (!vq->use_map_api)
>  		return;
>  
> -	if (vdev->map)
> -		vdev->map->sync_single_for_cpu(vq->map,
> -					       addr + offset, size, dir);
> +	if (map_ops)
> +		map_ops->sync_single_for_cpu(vq->map,
> +					     addr + offset, size, dir);
>  	else
>  		dma_sync_single_range_for_cpu(vring_dma_dev(vq),
>  					      addr, offset, size, dir);
> @@ -4002,15 +4054,15 @@ void virtqueue_map_sync_single_range_for_device(const struct virtqueue *_vq,
>  						enum dma_data_direction dir)
>  {
>  	const struct vring_virtqueue *vq = to_vvq(_vq);
> -	struct virtio_device *vdev = _vq->vdev;
> +	const struct virtio_map_ops *map_ops = vring_get_map_ops(vq);
>  
>  	if (!vq->use_map_api)
>  		return;
>  
> -	if (vdev->map)
> -		vdev->map->sync_single_for_device(vq->map,
> -						  addr + offset,
> -						  size, dir);
> +	if (map_ops)
> +		map_ops->sync_single_for_device(vq->map,
> +						addr + offset,
> +						size, dir);
>  	else
>  		dma_sync_single_range_for_device(vring_dma_dev(vq), addr,
>  						 offset, size, dir);
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index f923e42cfd01..20fe333d6778 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -44,6 +44,7 @@ struct virtqueue {
>  };
>  
>  struct vduse_vq_group;
> +struct virtio_map_ops;
>  
>  union virtio_map {
>  	/* Device that performs DMA */
> @@ -294,6 +295,9 @@ void virtqueue_map_free_coherent(struct virtio_device *vdev,
>  				 size_t size, void *vaddr,
>  				 dma_addr_t dma_handle);
>  
> +void virtqueue_set_map_ops(struct virtqueue *_vq,
> +			   const struct virtio_map_ops *map_ops);
> +
>  dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
>  				    struct page *page,
>  				    unsigned long offset,
> -- 
> 2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 00/13] virtio_balloon, efi/unaccepted: support memory ballooning and free page reporting in CoCo guests

[Michael S. Tsirkin]

On Tue, Oct 27, 2026 at 02:56:03AM +0000, Kevin Loughlin wrote:
> This series enables virtio-balloon free page reporting and memory balloon
> inflation/deflation in Confidential Computing (CoCo) guests, starting
> with architectural hooks implemented for AMD SEV variants, in order to
> support dynamic CoCo guest resizing.
> 
> Today, memory ballooning and free page reporting are essentially unusable
> in CoCo guests due to three main roadblocks:


For me, the bigger question is not whether we can, but whether we
should. Considerations I would like to see addressed:

Security, e.g.: does knowing which memory is free leak information from
the guest? If no, why not?
Consider CONFIG_SHUFFLE_PAGE_ALLOCATOR, RANDSTRUCT, etc.

correctness, e.g.:  Linux has init on free (and default on windows), if
initialized pages are dropped and host can not be trusted, they have to be
re-initialized. Is this handled? I don't see it in code.

Also, performance, e.g.: How much does this re-initialization cost?

Management: it was simple with balloon off unconditionally.
Now we are in murkier waters with various features offering
various performance/security tradeoffs.
What if some guests want the tradeoffs but others don't?

Documentation: for all of the above.

-- 
MST