[PATCH RFC 0/6] virtio-mem: support memmap_on_memory

[PATCH RFC 0/6] virtio-mem: support memmap_on_memory

[Nikolaos Karaolidis]

virtio-mem adds memory without MHP_MEMMAP_ON_MEMORY. With the memory
onlined to ZONE_MOVABLE, a VM that grows to many times its boot size
can run short of unmovable memory. I ran into this while working on
a guest-initiated memory negotiation project.

David wrote last year that memmap_on_memory support for virtio-mem had
not been worth the effort, because virtio-mem adds memory in small
blocks [1]. Each block is small, yes, but the memmap adds up when
scaling to many blocks.

This picks up Hui Zhu's RFC from 2021 [2] and follows David's review of
it [3][4]:

1. Big Block Mode first (patch 3). Since commit 6b8f0798b85a
   ("mm/memory_hotplug: split memmap_on_memory requests across
   memblocks"), the core uses one altmap per memory block, so this also
   works for big blocks spanning several memory blocks.

2. alloc_contig_range() with pageblock granularity, so subblocks can be
   2 MiB: done since v5.19 (Zi Yan's series, 448b8ec3bf11).

3. Sub Block Mode only if the memmap spans complete subblocks (patch 5).
   With larger device blocks, the driver warns and continues without
   memmap_on_memory, as David suggested in 2023 [5].

The other review points on [2]:

- Plugging more than requested: if no more than the memmap subblocks
  remain to be plugged, the driver stops short of the requested size, so
  this doesn't happen.
- Error handling: if unplugging the memmap subblocks of a removed memory
  block fails, the block is left in VIRTIO_MEM_SBM_MB_PLUGGED and
  virtio_mem_cleanup_pending_mb() retries.

Georgi, Cc'ing you as you mentioned out-of-tree virtio-mem patches that
use memmap_on_memory [6].

Tested in QEMU/KVM (x86-64 guests) on v7.3: 128 MiB, 1 GiB
and 2 GiB memory blocks; device block sizes 2 MiB to 1 GiB;
ZONE_MOVABLE, kernel zones and offline blocks; force_bbm;
memory_hotplug.memmap_on_memory=force on a kernel without THP;
grow/fill/unplug cycles with memory held during unplug; 600
offline/online cycles from user space; random resizing under a memory
hog; injected failures when unplugging memmap subblocks; the
memory-hotplug selftest; virtio-mem as a module, reloaded. Also with
KASAN, lockdep, and DEBUG_VM. Built for x86-64, and the changed files
for arm64, riscv, and s390.

An LLM was used to review the series against the core, write the test
harness, and draft the cover letter and changelogs. I have reviewed all
of it.

[1] https://lore.kernel.org/all/1efcb368-fcdf-4bdd-8b94-a705b7806bc2@redhat.com/
[2] https://lore.kernel.org/all/20210623115839.3524-1-teawater@gmail.com/
[3] https://lore.kernel.org/all/CADFyXm7Ky+S2gq5ht2h_tX9bLsoOSuctCkG==VYAH0uUHGp+QQ@mail.gmail.com/
[4] https://lore.kernel.org/all/bc36b6fe-0d1a-6fe2-e8f3-49bbcaebcf41@redhat.com/
[5] https://lore.kernel.org/all/1bf30145-22a5-cc46-e583-25053460b105@redhat.com/
[6] https://lore.kernel.org/all/fb932d7f-0ba7-415d-ab36-e948d2bbb537@oss.qualcomm.com/

Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
Nikolaos Karaolidis (6):
      mm/memory_hotplug: add and export mhp_memmap_on_memory_nr_pages()
      virtio-mem: prepare memory notifier for a self-hosted memmap
      virtio-mem: support memmap_on_memory in Big Block Mode
      virtio-mem: move virtio_mem_sbm_try_remove_unplugged_mb()
      virtio-mem: support memmap_on_memory in Sub Block Mode
      Documentation/admin-guide/mm: document virtio-mem memmap_on_memory

 Documentation/admin-guide/mm/memory-hotplug.rst |  18 ++
 drivers/virtio/virtio_mem.c                     | 268 +++++++++++++++++-------
 include/linux/memory_hotplug.h                  |   6 +
 mm/memory_hotplug.c                             |  15 ++
 4 files changed, 236 insertions(+), 71 deletions(-)
---
base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
change-id: 20261103-virtio-mem-memmap-f3ec7382772d

Best regards,
--  
Nikolaos Karaolidis <nick@karaolidis.com>

[PATCH RFC 1/6] mm/memory_hotplug: add and export mhp_memmap_on_memory_nr_pages()

[Nikolaos Karaolidis]

virtio-mem wants to add memory with MHP_MEMMAP_ON_MEMORY. It has to
know at probe time how many pages at the start of each memory block
will host the memmap. Those pages never reach the buddy, so they must
not be fake-offlined, and in Sub Block Mode they must stay plugged while
the memory block is added.

memory_block_memmap_on_memory_pages() returns a non-zero value even if
the core would ignore MHP_MEMMAP_ON_MEMORY, and recomputing the value in
the driver would duplicate the padding of
"memory_hotplug.memmap_on_memory=force". Add a helper that returns the
number of memmap pages, or 0 if the flag would be ignored. Both inputs
are fixed after boot.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 include/linux/memory_hotplug.h |  6 ++++++
 mm/memory_hotplug.c            | 15 +++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index b39605d30896..1b261d85e69d 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -74,6 +74,7 @@ struct mhp_params {
 bool mhp_range_allowed(u64 start, u64 size, bool need_mapping);
 struct range mhp_get_pluggable_range(bool need_mapping);
 bool mhp_supports_memmap_on_memory(void);
+unsigned long mhp_memmap_on_memory_nr_pages(void);
 
 /*
  * Zone resizing functions
@@ -220,6 +221,11 @@ static inline bool mhp_supports_memmap_on_memory(void)
 	return false;
 }
 
+static inline unsigned long mhp_memmap_on_memory_nr_pages(void)
+{
+	return 0;
+}
+
 static inline void pgdat_kswapd_lock(pg_data_t *pgdat) {}
 static inline void pgdat_kswapd_unlock(pg_data_t *pgdat) {}
 static inline void pgdat_kswapd_lock_init(pg_data_t *pgdat) {}
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078a..800011ca194e 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1409,6 +1409,21 @@ bool mhp_supports_memmap_on_memory(void)
 }
 EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
 
+/**
+ * mhp_memmap_on_memory_nr_pages - self-hosted memmap pages per memory block
+ *
+ * Return: the number of pages at the start of each memory block that host its
+ * memmap when adding memory with MHP_MEMMAP_ON_MEMORY, including padding for
+ * "memory_hotplug.memmap_on_memory=force"; or 0 if the flag would be ignored.
+ */
+unsigned long mhp_memmap_on_memory_nr_pages(void)
+{
+	if (!mhp_supports_memmap_on_memory())
+		return 0;
+	return memory_block_memmap_on_memory_pages();
+}
+EXPORT_SYMBOL_GPL(mhp_memmap_on_memory_nr_pages);
+
 static void altmap_free(struct vmem_altmap *altmap)
 {
 	WARN_ONCE(altmap->alloc, "Altmap not fully unmapped");

-- 
2.55.0

[PATCH RFC 2/6] virtio-mem: prepare memory notifier for a self-hosted memmap

[Nikolaos Karaolidis]

With MHP_MEMMAP_ON_MEMORY, the core excludes the memmap at the start of
each memory block from the range passed to memory notifiers. The
virtio-mem notifier does not expect that:

* In Sub Block Mode, it rejects any range that is not exactly one
  aligned memory block with a WARN and NOTIFY_BAD.

* It adjusts offline_size by the notified size, while adding and
  removing memory use the full size. Every onlined memory block would
  leave its memmap size behind in offline_size, until the offline
  threshold stops further hotplug.

Track the memmap pages per memory block in vm->memmap_pages, expect the
notified range to start after the memmap in Sub Block Mode, and include
the memmap when adjusting offline_size.

vm->memmap_pages is always 0 for now, so there is no functional change.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 drivers/virtio/virtio_mem.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index e18dd736f2ec..ab06adbc0342 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -158,6 +158,9 @@ struct virtio_mem {
 	/* If set, the driver is in SBM, otherwise in BBM. */
 	bool in_sbm;
 
+	/* Pages at the start of each Linux memory block hosting its memmap. */
+	unsigned long memmap_pages;
+
 	union {
 		struct {
 			/* Id of the first memory block of this device. */
@@ -995,21 +998,23 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
 	struct memory_notify *mhp = arg;
 	const unsigned long start = PFN_PHYS(mhp->start_pfn);
 	const unsigned long size = PFN_PHYS(mhp->nr_pages);
+	const unsigned long memmap_size = PFN_PHYS(vm->memmap_pages);
 	int rc = NOTIFY_OK;
-	unsigned long id;
+	unsigned long id, mb_start;
 
 	if (!virtio_mem_overlaps_range(vm, start, size))
 		return NOTIFY_DONE;
 
 	if (vm->in_sbm) {
 		id = virtio_mem_phys_to_mb_id(start);
+		mb_start = virtio_mem_mb_id_to_phys(id);
 		/*
 		 * In SBM, we add memory in separate memory blocks - we expect
-		 * it to be onlined/offlined in the same granularity. Bail out
-		 * if this ever changes.
+		 * it to be onlined/offlined in the same granularity, excluding
+		 * a self-hosted memmap. Bail out if this ever changes.
 		 */
-		if (WARN_ON_ONCE(size != memory_block_size_bytes() ||
-				 !IS_ALIGNED(start, memory_block_size_bytes())))
+		if (WARN_ON_ONCE(start != mb_start + memmap_size ||
+				 size != memory_block_size_bytes() - memmap_size))
 			return NOTIFY_BAD;
 	} else {
 		id = virtio_mem_phys_to_bb_id(vm, start);
@@ -1062,7 +1067,8 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
 		if (vm->in_sbm)
 			virtio_mem_sbm_notify_offline(vm, id);
 
-		atomic64_add(size, &vm->offline_size);
+		/* Adding/removing memory accounts the memmap as well. */
+		atomic64_add(size + memmap_size, &vm->offline_size);
 		/*
 		 * Trigger the workqueue. Now that we have some offline memory,
 		 * maybe we can handle pending unplug requests.
@@ -1077,7 +1083,7 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
 		if (vm->in_sbm)
 			virtio_mem_sbm_notify_online(vm, id, mhp->start_pfn);
 
-		atomic64_sub(size, &vm->offline_size);
+		atomic64_sub(size + memmap_size, &vm->offline_size);
 		/*
 		 * Start adding more memory once we onlined half of our
 		 * threshold. Don't trigger if it's possibly due to our action

-- 
2.55.0

[PATCH RFC 3/6] virtio-mem: support memmap_on_memory in Big Block Mode

[Nikolaos Karaolidis]

virtio-mem adds memory without MHP_MEMMAP_ON_MEMORY. With hotplugged
memory onlined to ZONE_MOVABLE, its memmap is allocated from the kernel
zones. A small VM grown to many times its boot size can run out of
memory for unmovable allocations.

With MHP_MEMMAP_ON_MEMORY and memory_hotplug.memmap_on_memory, the core
allocates the memmap of each memory block from the memory block itself.
In Big Block Mode memory blocks are always added and removed as a
whole, and the core uses one altmap per memory block, so big blocks
spanning several memory blocks work.

Add virtio_mem.memmap_on_memory, off by default, which only takes
effect if memory_hotplug.memmap_on_memory is enabled as well, like the
opt-in knobs of dax/kmem and s390 standby memory. It is opt-in because:

* The self-hosted memmap (1/64 with 4 KiB pages) counts towards the
  plugged size of the device but is not available to Linux, and in Sub
  Block Mode the plugged size can end up slightly below the requested
  size. Whoever sizes the device should opt in.

* Gigantic pages can no longer be allocated from virtio-mem memory
  unless memory blocks are larger than a gigantic page.

If the core would ignore MHP_MEMMAP_ON_MEMORY, warn and continue
without it.

When unplugging a big block, skip the memmap of each memory block while
fake-offlining and when rolling back, because it is not managed by the
buddy.

Sub Block Mode is not supported yet, so warn and ignore the parameter.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 drivers/virtio/virtio_mem.c | 64 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 56 insertions(+), 8 deletions(-)

diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index ab06adbc0342..034d07207e44 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -40,6 +40,11 @@ module_param(bbm_block_size, ulong, 0444);
 MODULE_PARM_DESC(bbm_block_size,
 		 "Big Block size in bytes. Default is 0 (auto-detection).");
 
+static bool memmap_on_memory;
+module_param(memmap_on_memory, bool, 0444);
+MODULE_PARM_DESC(memmap_on_memory,
+		 "Allocate the memmap from hotplugged memory itself if memory_hotplug.memmap_on_memory is enabled. Default is 0.");
+
 /*
  * virtio-mem currently supports the following modes of operation:
  *
@@ -639,6 +644,7 @@ static bool virtio_mem_could_add_memory(struct virtio_mem *vm, uint64_t size)
 static int virtio_mem_add_memory(struct virtio_mem *vm, uint64_t addr,
 				 uint64_t size)
 {
+	mhp_t mhp_flags = MHP_MERGE_RESOURCE | MHP_NID_IS_MGID;
 	int rc;
 
 	/*
@@ -654,10 +660,13 @@ static int virtio_mem_add_memory(struct virtio_mem *vm, uint64_t addr,
 
 	dev_dbg(&vm->vdev->dev, "adding memory: 0x%llx - 0x%llx\n", addr,
 		addr + size - 1);
+	if (vm->memmap_pages)
+		mhp_flags |= MHP_MEMMAP_ON_MEMORY;
+
 	/* Memory might get onlined immediately. */
 	atomic64_add(size, &vm->offline_size);
 	rc = add_memory_driver_managed(vm->mgid, addr, size, vm->resource_name,
-				       MHP_MERGE_RESOURCE | MHP_NID_IS_MGID);
+				       mhp_flags);
 	if (rc) {
 		atomic64_sub(size, &vm->offline_size);
 		dev_warn(&vm->vdev->dev, "adding memory failed: %d\n", rc);
@@ -2154,6 +2163,20 @@ static int virtio_mem_sbm_unplug_request(struct virtio_mem *vm, uint64_t diff)
 	return rc;
 }
 
+/*
+ * A self-hosted memmap is not managed by the buddy and must not get
+ * fake-offlined. Returns the first pfn to process in the memory section
+ * starting at @pfn, which is beyond the section if the memmap covers it.
+ */
+static unsigned long virtio_mem_bbm_section_skip_memmap(struct virtio_mem *vm,
+							unsigned long pfn)
+{
+	const unsigned long mb_pfn = ALIGN_DOWN(pfn,
+						PHYS_PFN(memory_block_size_bytes()));
+
+	return max(pfn, mb_pfn + vm->memmap_pages);
+}
+
 /*
  * Try to offline and remove a big block from Linux and unplug it. Will fail
  * with -EBUSY if some memory is busy and cannot get unplugged.
@@ -2167,7 +2190,7 @@ static int virtio_mem_bbm_offline_remove_and_unplug_bb(struct virtio_mem *vm,
 	const unsigned long start_pfn = PFN_DOWN(virtio_mem_bb_id_to_phys(vm, bb_id));
 	const unsigned long nr_pages = PFN_DOWN(vm->bbm.bb_size);
 	unsigned long end_pfn = start_pfn + nr_pages;
-	unsigned long pfn;
+	unsigned long pfn, first_pfn, next_pfn;
 	struct page *page;
 	int rc;
 
@@ -2184,12 +2207,17 @@ static int virtio_mem_bbm_offline_remove_and_unplug_bb(struct virtio_mem *vm,
 	mutex_lock(&vm->hotplug_mutex);
 	virtio_mem_bbm_set_bb_state(vm, bb_id, VIRTIO_MEM_BBM_BB_FAKE_OFFLINE);
 
-	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
-		page = pfn_to_online_page(pfn);
+	for (pfn = start_pfn; pfn < end_pfn; pfn = next_pfn) {
+		next_pfn = pfn + PAGES_PER_SECTION;
+		first_pfn = virtio_mem_bbm_section_skip_memmap(vm, pfn);
+		if (first_pfn >= next_pfn)
+			continue;
+		page = pfn_to_online_page(first_pfn);
 		if (!page)
 			continue;
 
-		rc = virtio_mem_fake_offline(vm, pfn, PAGES_PER_SECTION);
+		rc = virtio_mem_fake_offline(vm, first_pfn,
+					     next_pfn - first_pfn);
 		if (rc) {
 			end_pfn = pfn;
 			goto rollback;
@@ -2213,11 +2241,15 @@ static int virtio_mem_bbm_offline_remove_and_unplug_bb(struct virtio_mem *vm,
 	return rc;
 
 rollback:
-	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
-		page = pfn_to_online_page(pfn);
+	for (pfn = start_pfn; pfn < end_pfn; pfn = next_pfn) {
+		next_pfn = pfn + PAGES_PER_SECTION;
+		first_pfn = virtio_mem_bbm_section_skip_memmap(vm, pfn);
+		if (first_pfn >= next_pfn)
+			continue;
+		page = pfn_to_online_page(first_pfn);
 		if (!page)
 			continue;
-		virtio_mem_fake_online(pfn, PAGES_PER_SECTION);
+		virtio_mem_fake_online(first_pfn, next_pfn - first_pfn);
 	}
 	virtio_mem_bbm_set_bb_state(vm, bb_id, VIRTIO_MEM_BBM_BB_ADDED);
 	mutex_unlock(&vm->hotplug_mutex);
@@ -2607,6 +2639,19 @@ static int virtio_mem_init_hotplug(struct virtio_mem *vm)
 					      vm->offline_threshold);
 	}
 
+	/* All memory blocks of a device must agree, decide once. */
+	if (memmap_on_memory) {
+		vm->memmap_pages = mhp_memmap_on_memory_nr_pages();
+		if (!vm->memmap_pages) {
+			dev_warn(&vm->vdev->dev,
+				 "memmap_on_memory is disabled or not supported by the core, ignoring\n");
+		} else if (vm->in_sbm) {
+			dev_warn(&vm->vdev->dev,
+				 "memmap_on_memory is only supported in BBM, ignoring\n");
+			vm->memmap_pages = 0;
+		}
+	}
+
 	dev_info(&vm->vdev->dev, "memory block size: 0x%lx",
 		 memory_block_size_bytes());
 	if (vm->in_sbm)
@@ -2615,6 +2660,9 @@ static int virtio_mem_init_hotplug(struct virtio_mem *vm)
 	else
 		dev_info(&vm->vdev->dev, "big block size: 0x%llx",
 			 (unsigned long long)vm->bbm.bb_size);
+	if (vm->memmap_pages)
+		dev_info(&vm->vdev->dev, "memmap on memory: 0x%llx bytes per memory block",
+			 (unsigned long long)PFN_PHYS(vm->memmap_pages));
 
 	/* create the parent resource for all memory */
 	rc = virtio_mem_create_resource(vm);

-- 
2.55.0

[PATCH RFC 4/6] virtio-mem: move virtio_mem_sbm_try_remove_unplugged_mb()

[Nikolaos Karaolidis]

Move it below virtio_mem_sbm_unplug_mb(), because the next patch makes
it call virtio_mem_sbm_unplug_sb().

No functional change.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 drivers/virtio/virtio_mem.c | 56 ++++++++++++++++++++++-----------------------
 1 file changed, 28 insertions(+), 28 deletions(-)

diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 034d07207e44..688c178c760f 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -791,34 +791,6 @@ static int virtio_mem_sbm_offline_and_remove_mb(struct virtio_mem *vm,
 	return virtio_mem_offline_and_remove_memory(vm, addr, size);
 }
 
-/*
- * Try (offlining and) removing memory from Linux in case all subblocks are
- * unplugged. Can be called on online and offline memory blocks.
- *
- * May modify the state of memory blocks in virtio-mem.
- */
-static int virtio_mem_sbm_try_remove_unplugged_mb(struct virtio_mem *vm,
-						  unsigned long mb_id)
-{
-	int rc;
-
-	/*
-	 * Once all subblocks of a memory block were unplugged, offline and
-	 * remove it.
-	 */
-	if (!virtio_mem_sbm_test_sb_unplugged(vm, mb_id, 0, vm->sbm.sbs_per_mb))
-		return 0;
-
-	/* offline_and_remove_memory() works for online and offline memory. */
-	mutex_unlock(&vm->hotplug_mutex);
-	rc = virtio_mem_sbm_offline_and_remove_mb(vm, mb_id);
-	mutex_lock(&vm->hotplug_mutex);
-	if (!rc)
-		virtio_mem_sbm_set_mb_state(vm, mb_id,
-					    VIRTIO_MEM_SBM_MB_UNUSED);
-	return rc;
-}
-
 /*
  * See virtio_mem_offline_and_remove_memory(): Try to offline and remove a
  * all Linux memory blocks covered by the big block.
@@ -1648,6 +1620,34 @@ static int virtio_mem_sbm_unplug_mb(struct virtio_mem *vm, unsigned long mb_id)
 	return virtio_mem_sbm_unplug_any_sb_raw(vm, mb_id, &nb_sb);
 }
 
+/*
+ * Try (offlining and) removing memory from Linux in case all subblocks are
+ * unplugged. Can be called on online and offline memory blocks.
+ *
+ * May modify the state of memory blocks in virtio-mem.
+ */
+static int virtio_mem_sbm_try_remove_unplugged_mb(struct virtio_mem *vm,
+						  unsigned long mb_id)
+{
+	int rc;
+
+	/*
+	 * Once all subblocks of a memory block were unplugged, offline and
+	 * remove it.
+	 */
+	if (!virtio_mem_sbm_test_sb_unplugged(vm, mb_id, 0, vm->sbm.sbs_per_mb))
+		return 0;
+
+	/* offline_and_remove_memory() works for online and offline memory. */
+	mutex_unlock(&vm->hotplug_mutex);
+	rc = virtio_mem_sbm_offline_and_remove_mb(vm, mb_id);
+	mutex_lock(&vm->hotplug_mutex);
+	if (!rc)
+		virtio_mem_sbm_set_mb_state(vm, mb_id,
+					    VIRTIO_MEM_SBM_MB_UNUSED);
+	return rc;
+}
+
 /*
  * Prepare tracking data for the next memory block.
  */

-- 
2.55.0

[PATCH RFC 5/6] virtio-mem: support memmap_on_memory in Sub Block Mode

[Nikolaos Karaolidis]

Forcing Big Block Mode to get a self-hosted memmap would raise the
plug/unplug granularity from 2 MiB to 128 MiB on x86-64.

A self-hosted memmap is written when the memory block is added and
never reaches the buddy, so the subblocks containing it ("memmap
subblocks") cannot be fake-offlined and must stay plugged while the
memory block is added.

Only support a memmap that spans complete subblocks, so that the memmap
subblocks never provide memory to Linux. That is the case if the
subblock size is not larger than the memmap, as with the 2 MiB default
device block size of QEMU on x86-64. Otherwise, for example with 4 MiB
device blocks, or with 32 MiB device blocks for a 4 KiB arm64 guest on
a 16 KiB host, warn and continue without memmap_on_memory; force_bbm
can be used instead.

Handle the memmap subblocks as part of every added memory block:

* Plug them together with the first other subblocks before adding the
  memory block. If no more than the memmap subblocks remain to be
  plugged, stop short of the requested size.

* Skip them when unplugging from an added memory block.

* Once all other subblocks are unplugged, remove the memory block and
  unplug its memmap subblocks, even if that unplugs more than requested.
  If that fails, keep the memory block in VIRTIO_MEM_SBM_MB_PLUGGED and
  let virtio_mem_cleanup_pending_mb() retry.

The plugged size therefore never stays above the requested size because
of the memmap, but can stay below it by up to the memmap subblocks of
one memory block. Adding or keeping a memory block only for its memmap
subblocks would provide no memory.

The memmap subblocks count towards the plugged size of the device, so a
fully plugged 128 MiB memory block adds 126 MiB to MemTotal.

In a 2 GiB x86-64 guest onlining to ZONE_MOVABLE, plugging 4 GiB took
about 64 MiB from the kernel zones without virtio_mem.memmap_on_memory
and less than 1 MiB with it.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 drivers/virtio/virtio_mem.c | 152 ++++++++++++++++++++++++++++++++------------
 1 file changed, 112 insertions(+), 40 deletions(-)

diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 688c178c760f..d6623f67b76b 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -54,6 +54,11 @@ MODULE_PARM_DESC(memmap_on_memory,
  *   Subblocks within a Linux memory block might either be plugged or unplugged.
  *   Memory is added/removed to Linux MM in Linux memory block granularity.
  *
+ *   With memmap_on_memory, the first subblock(s) of a Linux memory block
+ *   host its memmap ("memmap subblocks"). They stay plugged while the Linux
+ *   memory block is added to Linux, which requires at least one other plugged
+ *   subblock. "Fully plugged" and "partially plugged" below include them.
+ *
  * * Big Block Mode (BBM): A Big Block (BB) spans 1..X Linux memory blocks.
  *   Memory is added/removed to Linux MM in Big Block granularity.
  *
@@ -72,7 +77,10 @@ MODULE_PARM_DESC(memmap_on_memory,
 enum virtio_mem_sbm_mb_state {
 	/* Unplugged, not added to Linux. Can be reused later. */
 	VIRTIO_MEM_SBM_MB_UNUSED = 0,
-	/* (Partially) plugged, not added to Linux. Error on add_memory(). */
+	/*
+	 * (Partially) plugged, not added to Linux. Error on add_memory() or on
+	 * unplugging the memmap subblocks after removing it.
+	 */
 	VIRTIO_MEM_SBM_MB_PLUGGED,
 	/* Fully plugged, fully added to Linux, offline. */
 	VIRTIO_MEM_SBM_MB_OFFLINE,
@@ -179,6 +187,8 @@ struct virtio_mem {
 			uint64_t sb_size;
 			/* The number of subblocks per Linux memory block. */
 			uint32_t sbs_per_mb;
+			/* The number of memmap subblocks per memory block. */
+			u32 memmap_sbs;
 
 			/*
 			 * Some of the Linux memory blocks tracked as "partially
@@ -1567,15 +1577,16 @@ static int virtio_mem_bbm_plug_bb(struct virtio_mem *vm, unsigned long bb_id)
 
 /*
  * Unplug the desired number of plugged subblocks of a offline or not-added
- * memory block. Will fail if any subblock cannot get unplugged (instead of
- * skipping it).
+ * memory block, ignoring the subblocks below @first_sb. Will fail if any
+ * subblock cannot get unplugged (instead of skipping it).
  *
  * Will not modify the state of the memory block.
  *
  * Note: can fail after some subblocks were unplugged.
  */
 static int virtio_mem_sbm_unplug_any_sb_raw(struct virtio_mem *vm,
-					    unsigned long mb_id, uint64_t *nb_sb)
+					    unsigned long mb_id, int first_sb,
+					    uint64_t *nb_sb)
 {
 	int sb_id, count;
 	int rc;
@@ -1583,14 +1594,14 @@ static int virtio_mem_sbm_unplug_any_sb_raw(struct virtio_mem *vm,
 	sb_id = vm->sbm.sbs_per_mb - 1;
 	while (*nb_sb) {
 		/* Find the next candidate subblock */
-		while (sb_id >= 0 &&
+		while (sb_id >= first_sb &&
 		       virtio_mem_sbm_test_sb_unplugged(vm, mb_id, sb_id, 1))
 			sb_id--;
-		if (sb_id < 0)
+		if (sb_id < first_sb)
 			break;
 		/* Try to unplug multiple subblocks at a time */
 		count = 1;
-		while (count < *nb_sb && sb_id > 0 &&
+		while (count < *nb_sb && sb_id > first_sb &&
 		       virtio_mem_sbm_test_sb_plugged(vm, mb_id, sb_id - 1, 1)) {
 			count++;
 			sb_id--;
@@ -1607,7 +1618,7 @@ static int virtio_mem_sbm_unplug_any_sb_raw(struct virtio_mem *vm,
 }
 
 /*
- * Unplug all plugged subblocks of an offline or not-added memory block.
+ * Unplug all plugged subblocks of a not-added memory block.
  *
  * Will not modify the state of the memory block.
  *
@@ -1617,35 +1628,65 @@ static int virtio_mem_sbm_unplug_mb(struct virtio_mem *vm, unsigned long mb_id)
 {
 	uint64_t nb_sb = vm->sbm.sbs_per_mb;
 
-	return virtio_mem_sbm_unplug_any_sb_raw(vm, mb_id, &nb_sb);
+	return virtio_mem_sbm_unplug_any_sb_raw(vm, mb_id, 0, &nb_sb);
+}
+
+/*
+ * Unplug the memmap subblocks of a memory block removed from Linux and mark it
+ * unused. The memmap subblocks are deducted from @nb_sb (clamped), even if
+ * unplugging them fails. On error, the memory block stays
+ * VIRTIO_MEM_SBM_MB_PLUGGED and virtio_mem_cleanup_pending_mb() retries.
+ *
+ * Will modify the state of the memory block.
+ */
+static int virtio_mem_sbm_finish_remove_mb(struct virtio_mem *vm,
+					   unsigned long mb_id, uint64_t *nb_sb)
+{
+	const int count = vm->sbm.memmap_sbs;
+	int rc;
+
+	if (count) {
+		if (nb_sb)
+			*nb_sb -= min_t(uint64_t, *nb_sb, count);
+		virtio_mem_sbm_set_mb_state(vm, mb_id,
+					    VIRTIO_MEM_SBM_MB_PLUGGED);
+		rc = virtio_mem_sbm_unplug_sb(vm, mb_id, 0, count);
+		if (rc)
+			return rc;
+	}
+	virtio_mem_sbm_set_mb_state(vm, mb_id, VIRTIO_MEM_SBM_MB_UNUSED);
+	return 0;
 }
 
 /*
- * Try (offlining and) removing memory from Linux in case all subblocks are
- * unplugged. Can be called on online and offline memory blocks.
+ * Try (offlining and) removing memory from Linux in case all subblocks except
+ * the memmap subblocks are unplugged. Can be called on online and offline
+ * memory blocks.
  *
  * May modify the state of memory blocks in virtio-mem.
  */
 static int virtio_mem_sbm_try_remove_unplugged_mb(struct virtio_mem *vm,
-						  unsigned long mb_id)
+						  unsigned long mb_id,
+						  uint64_t *nb_sb)
 {
+	const int first_sb = vm->sbm.memmap_sbs;
 	int rc;
 
 	/*
 	 * Once all subblocks of a memory block were unplugged, offline and
-	 * remove it.
+	 * remove it. The memmap subblocks alone provide no memory to Linux.
 	 */
-	if (!virtio_mem_sbm_test_sb_unplugged(vm, mb_id, 0, vm->sbm.sbs_per_mb))
+	if (!virtio_mem_sbm_test_sb_unplugged(vm, mb_id, first_sb,
+					      vm->sbm.sbs_per_mb - first_sb))
 		return 0;
 
 	/* offline_and_remove_memory() works for online and offline memory. */
 	mutex_unlock(&vm->hotplug_mutex);
 	rc = virtio_mem_sbm_offline_and_remove_mb(vm, mb_id);
 	mutex_lock(&vm->hotplug_mutex);
-	if (!rc)
-		virtio_mem_sbm_set_mb_state(vm, mb_id,
-					    VIRTIO_MEM_SBM_MB_UNUSED);
-	return rc;
+	if (rc)
+		return rc;
+	return virtio_mem_sbm_finish_remove_mb(vm, mb_id, nb_sb);
 }
 
 /*
@@ -1686,7 +1727,7 @@ static int virtio_mem_sbm_plug_and_add_mb(struct virtio_mem *vm,
 	const int count = min_t(int, *nb_sb, vm->sbm.sbs_per_mb);
 	int rc;
 
-	if (WARN_ON_ONCE(!count))
+	if (WARN_ON_ONCE(count <= vm->sbm.memmap_sbs))
 		return -EINVAL;
 
 	/*
@@ -1806,6 +1847,9 @@ static int virtio_mem_sbm_plug_request(struct virtio_mem *vm, uint64_t diff)
 
 	/* Try to plug and add unused blocks */
 	virtio_mem_sbm_for_each_mb(vm, mb_id, VIRTIO_MEM_SBM_MB_UNUSED) {
+		/* Don't add a memory block just for its memmap subblocks. */
+		if (nb_sb <= vm->sbm.memmap_sbs)
+			return 0;
 		if (!virtio_mem_could_add_memory(vm, memory_block_size_bytes()))
 			return -ENOSPC;
 
@@ -1817,6 +1861,8 @@ static int virtio_mem_sbm_plug_request(struct virtio_mem *vm, uint64_t diff)
 
 	/* Try to prepare, plug and add new blocks */
 	while (nb_sb) {
+		if (nb_sb <= vm->sbm.memmap_sbs)
+			return 0;
 		if (!virtio_mem_could_add_memory(vm, memory_block_size_bytes()))
 			return -ENOSPC;
 
@@ -1954,9 +2000,10 @@ static int virtio_mem_sbm_unplug_any_sb_offline(struct virtio_mem *vm,
 						unsigned long mb_id,
 						uint64_t *nb_sb)
 {
+	const int first_sb = vm->sbm.memmap_sbs;
 	int rc;
 
-	rc = virtio_mem_sbm_unplug_any_sb_raw(vm, mb_id, nb_sb);
+	rc = virtio_mem_sbm_unplug_any_sb_raw(vm, mb_id, first_sb, nb_sb);
 
 	/* some subblocks might have been unplugged even on failure */
 	if (!virtio_mem_sbm_test_sb_plugged(vm, mb_id, 0, vm->sbm.sbs_per_mb))
@@ -1965,20 +2012,25 @@ static int virtio_mem_sbm_unplug_any_sb_offline(struct virtio_mem *vm,
 	if (rc)
 		return rc;
 
-	if (virtio_mem_sbm_test_sb_unplugged(vm, mb_id, 0, vm->sbm.sbs_per_mb)) {
+	if (virtio_mem_sbm_test_sb_unplugged(vm, mb_id, first_sb,
+					     vm->sbm.sbs_per_mb - first_sb)) {
 		/*
 		 * Remove the block from Linux - this should never fail.
 		 * Hinder the block from getting onlined by marking it
-		 * unplugged. Temporarily drop the mutex, so
-		 * any pending GOING_ONLINE requests can be serviced/rejected.
+		 * unplugged (or plugged, if memmap subblocks remain).
+		 * Temporarily drop the mutex, so any pending GOING_ONLINE
+		 * requests can be serviced/rejected.
 		 */
-		virtio_mem_sbm_set_mb_state(vm, mb_id,
+		virtio_mem_sbm_set_mb_state(vm, mb_id, first_sb ?
+					    VIRTIO_MEM_SBM_MB_PLUGGED :
 					    VIRTIO_MEM_SBM_MB_UNUSED);
 
 		mutex_unlock(&vm->hotplug_mutex);
 		rc = virtio_mem_sbm_remove_mb(vm, mb_id);
 		BUG_ON(rc);
 		mutex_lock(&vm->hotplug_mutex);
+
+		return virtio_mem_sbm_finish_remove_mb(vm, mb_id, nb_sb);
 	}
 	return 0;
 }
@@ -2040,27 +2092,33 @@ static int virtio_mem_sbm_unplug_any_sb_online(struct virtio_mem *vm,
 					       unsigned long mb_id,
 					       uint64_t *nb_sb)
 {
+	const int first_sb = vm->sbm.memmap_sbs;
+	const int count = vm->sbm.sbs_per_mb - first_sb;
 	int rc, sb_id;
 
-	/* If possible, try to unplug the complete block in one shot. */
-	if (*nb_sb >= vm->sbm.sbs_per_mb &&
-	    virtio_mem_sbm_test_sb_plugged(vm, mb_id, 0, vm->sbm.sbs_per_mb)) {
-		rc = virtio_mem_sbm_unplug_sb_online(vm, mb_id, 0,
-						     vm->sbm.sbs_per_mb);
+	/*
+	 * If possible, try to unplug the complete block (except the memmap
+	 * subblocks) in one shot.
+	 */
+	if (*nb_sb >= count &&
+	    virtio_mem_sbm_test_sb_plugged(vm, mb_id, first_sb, count)) {
+		rc = virtio_mem_sbm_unplug_sb_online(vm, mb_id, first_sb,
+						     count);
 		if (!rc) {
-			*nb_sb -= vm->sbm.sbs_per_mb;
+			*nb_sb -= count;
 			goto unplugged;
 		} else if (rc != -EBUSY)
 			return rc;
 	}
 
 	/* Fallback to single subblocks. */
-	for (sb_id = vm->sbm.sbs_per_mb - 1; sb_id >= 0 && *nb_sb; sb_id--) {
+	for (sb_id = vm->sbm.sbs_per_mb - 1; sb_id >= first_sb && *nb_sb;
+	     sb_id--) {
 		/* Find the next candidate subblock */
-		while (sb_id >= 0 &&
+		while (sb_id >= first_sb &&
 		       !virtio_mem_sbm_test_sb_plugged(vm, mb_id, sb_id, 1))
 			sb_id--;
-		if (sb_id < 0)
+		if (sb_id < first_sb)
 			break;
 
 		rc = virtio_mem_sbm_unplug_sb_online(vm, mb_id, sb_id, 1);
@@ -2072,7 +2130,7 @@ static int virtio_mem_sbm_unplug_any_sb_online(struct virtio_mem *vm,
 	}
 
 unplugged:
-	rc = virtio_mem_sbm_try_remove_unplugged_mb(vm, mb_id);
+	rc = virtio_mem_sbm_try_remove_unplugged_mb(vm, mb_id, nb_sb);
 	if (rc)
 		vm->sbm.have_unplugged_mb = 1;
 	/* Ignore errors, this is not critical. We'll retry later. */
@@ -2389,11 +2447,11 @@ static int virtio_mem_cleanup_pending_mb(struct virtio_mem *vm)
 
 	mutex_lock(&vm->hotplug_mutex);
 	virtio_mem_sbm_for_each_mb(vm, id, VIRTIO_MEM_SBM_MB_MOVABLE_PARTIAL)
-		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id);
+		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id, NULL);
 	virtio_mem_sbm_for_each_mb(vm, id, VIRTIO_MEM_SBM_MB_KERNEL_PARTIAL)
-		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id);
+		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id, NULL);
 	virtio_mem_sbm_for_each_mb(vm, id, VIRTIO_MEM_SBM_MB_OFFLINE_PARTIAL)
-		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id);
+		rc |= virtio_mem_sbm_try_remove_unplugged_mb(vm, id, NULL);
 	mutex_unlock(&vm->hotplug_mutex);
 
 	if (rc)
@@ -2642,12 +2700,23 @@ static int virtio_mem_init_hotplug(struct virtio_mem *vm)
 	/* All memory blocks of a device must agree, decide once. */
 	if (memmap_on_memory) {
 		vm->memmap_pages = mhp_memmap_on_memory_nr_pages();
-		if (!vm->memmap_pages) {
+		if (!vm->memmap_pages)
 			dev_warn(&vm->vdev->dev,
 				 "memmap_on_memory is disabled or not supported by the core, ignoring\n");
-		} else if (vm->in_sbm) {
+	}
+	/*
+	 * In SBM, only support a memmap that spans complete subblocks: the
+	 * memmap subblocks then never provide memory to Linux. As the core
+	 * keeps at least one pageblock of each memory block for the buddy,
+	 * at least one other subblock remains.
+	 */
+	if (vm->memmap_pages && vm->in_sbm) {
+		if (IS_ALIGNED(PFN_PHYS(vm->memmap_pages), vm->sbm.sb_size)) {
+			vm->sbm.memmap_sbs = PFN_PHYS(vm->memmap_pages) /
+					     vm->sbm.sb_size;
+		} else {
 			dev_warn(&vm->vdev->dev,
-				 "memmap_on_memory is only supported in BBM, ignoring\n");
+				 "memmap_on_memory requires the memmap to span complete subblocks, ignoring; consider force_bbm\n");
 			vm->memmap_pages = 0;
 		}
 	}
@@ -2663,6 +2732,9 @@ static int virtio_mem_init_hotplug(struct virtio_mem *vm)
 	if (vm->memmap_pages)
 		dev_info(&vm->vdev->dev, "memmap on memory: 0x%llx bytes per memory block",
 			 (unsigned long long)PFN_PHYS(vm->memmap_pages));
+	if (vm->in_sbm && vm->sbm.memmap_sbs)
+		dev_info(&vm->vdev->dev, "memmap subblocks per memory block: %u",
+			 vm->sbm.memmap_sbs);
 
 	/* create the parent resource for all memory */
 	rc = virtio_mem_create_resource(vm);

-- 
2.55.0

[PATCH RFC 6/6] Documentation/admin-guide/mm: document virtio-mem memmap_on_memory

[Nikolaos Karaolidis]

Document that virtio-mem only uses memory_hotplug.memmap_on_memory with
virtio_mem.memmap_on_memory=1.

Assisted-by: LLM
Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
---
 Documentation/admin-guide/mm/memory-hotplug.rst | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 0207f8725142..b0f895766fb5 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -454,6 +454,10 @@ The following module parameters are currently defined:
 				 architecture does not support the forced mode
 				 in all configurations.
 
+				 virtio-mem additionally requires
+				 ``virtio_mem.memmap_on_memory=1``. See
+				 `ZONE_MOVABLE Sizing Considerations`_.
+
 ``online_policy``		 read-write: Set the basic policy used for
 				 automatic zone selection when onlining memory
 				 blocks without specifying a target zone.
@@ -628,6 +632,20 @@ ZONE_MOVABLE, especially when fine-tuning zone ratios:
 - In certain DAX configurations the memory map for the device memory will be
   allocated from the kernel zones.
 
+- The memory map of hotplugged memory is allocated from the kernel zones,
+  unless ``memory_hotplug.memmap_on_memory`` is used (for virtio-mem, together
+  with ``virtio_mem.memmap_on_memory=1``).
+
+  With virtio-mem in Sub Block Mode, the subblocks containing the memory map of
+  a memory block stay plugged until the memory block is removed. They count
+  towards the plugged size of the device but are not available to Linux:
+  plugging 128 MiB into an empty 128 MiB memory block makes 126 MiB available.
+  The plugged size can stay below the requested size by up to these subblocks
+  of one memory block. Sub Block Mode only uses the self-hosted memory map if
+  it spans complete subblocks, for example with 128 MiB memory blocks and a
+  2 MiB device block size; otherwise, virtio-mem continues without it.
+  ``virtio_mem.force_bbm=1`` can be used instead.
+
 - KASAN can have a significant memory overhead, for example, consuming 1/8th of
   the total system memory size as (unmovable) tracking metadata.
 

-- 
2.55.0

Re: [PATCH RFC 0/6] virtio-mem: support memmap_on_memory

[David Hildenbrand (Arm)]

On 11/3/26 10:50, Nikolaos Karaolidis wrote:
> virtio-mem adds memory without MHP_MEMMAP_ON_MEMORY. With the memory
> onlined to ZONE_MOVABLE, a VM that grows to many times its boot size
> can run short of unmovable memory. I ran into this while working on
> a guest-initiated memory negotiation project.

Care to pitch the use case to me?

I'm all for making virtio-mem better suitable for some workloads, but my time is
limited.

Is this something that your use case would actually benefit from, or was that
project already discarded?

Why so much MOVABLE memory, and how do other kernel allocations (page tables
etc) cope with that?

-- 
Cheers,

David

Re: [PATCH RFC 0/6] virtio-mem: support memmap_on_memory

[Nick Karaolidis]

On Tuesday, November 3rd, 2026 at 4:29 PM, David Hildenbrand (Arm) <david@kernel.org> wrote:
> Care to pitch the use case to me?

Sure! The point of my project is better control over guest memory.

Each VM boots as small as possible. An agent in the guest throttles the
workload with memory.high before it OOMs and asks a daemon on the host
for more, the daemon grants it (or not) by resizing the virtio-mem
device, and the guest gives it back once it no longer needs it.

From there, one can set up VM priorities, revocation policies, etc etc.

> Is this something that your use case would actually benefit from,
> or was that project already discarded?

It is live, though on my own server for now, and nowhere near
datacenter scale. It's a 32 GiB box running two VMs, a CI runner and a
game server node.

Each used to have a static 32 GiB with a balloon because both workloads
could potentially use up to that much memory.

At the ceiling, the memmap for 30 GiB is about 480 MiB in the kernel
zones of a 2 GiB VM, on top of up to about 360 MiB of other unmovable
memory.

> Why so much MOVABLE memory,

Have you seen RAM prices lately? :)

In more seriousness, giving memory back only works if unplugging
reliably succeeds, so hotplugged memory is onlined movable.

Onlining some of it to the kernel zones, as auto-movable would, makes
that part hard to unplug again.

> and how do other kernel allocations (page tables etc) cope with
> that?

On said VMs, page tables peaked at 75 MiB, and page tables,
unreclaimable slab, kernel stacks, and percpu memory together at about
360 MiB. They scale with the workload, while memmap scales with the 
plugged size, which is why I started with it.

The other problem I hit was inodes. Their page cache ends up in
ZONE_MOVABLE, which reclaim for kernel allocations does not scan, so
they pile up in the kernel zones. In testing with 1 GiB of boot memory,
that meant OOM kills and usually a panic. Working on it turned up two
inode LRU bugs, which are now in 7.4-rc1:

  8dc4116041e2 ("fs: don't lose inodes that another LRU walker is isolating")
  8ee09f597763 ("fs: rotate inodes whose page cache could not be dropped")

I have a series for the pinning itself that I'll post separately.

My VMs have been running with all mentioned patches (with some minor
tweaks) applied for some 5 weeks.

Thanks!
Nikolaos