[PATCH v2] mm/hugetlb: allow crashkernel to reuse hugetlb CMA memory

[PATCH v2] mm/hugetlb: allow crashkernel to reuse hugetlb CMA memory

[Usama Arif]

crashkernel=size,cma reserves a separate CMA pool. On systems that
already reserve several gigabytes with hugetlb_cma= (which is common in
the Meta fleet), the crash kernel cannot use that existing memory as
part of its working set. With hugetlb_cma=6G, requesting another 2G
with crashkernel=2G,cma brings the total CMA reservation to 8G, none of
which can serve unmovable kernel allocations.

Use existing HugeTLB CMA memory first for crashkernel=size,cma and
reserve dedicated CMA areas only for the remainder. With hugetlb_cma=6G,
a 2G request reuses 2G without reserving more CMA. With hugetlb_cma=2G,
a 4G request reuses 2G and reserves another 2G.

The full HugeTLB CMA pool remains available to HugeTLB and movable
allocations during normal operation. After a crash, the capture kernel
can use the selected ranges as working memory. The conventional
reservation is still required to hold the kernel image and initrd.

The ,cma option already opts into excluding userspace memory from the
vmcore. This includes allocated HugeTLB pages in the selected ranges,
which the capture kernel can overwrite. The existing dump completeness
and pending DMA risks apply to these ranges as well.

Have architecture code record the requested CMA size and perform the
reservation in mm_core_init_early(), immediately after
hugetlb_cma_reserve(). The HugeTLB CMA areas are then available for reuse
while memblock can still reserve any remainder.

Use the existing four range slots for both reused and dedicated memory.
Limit reused memory to the page-aligned request, account for dedicated
CMA alignment, and report any shortfall. Ignore malformed optional sizes
and reject requests that would overflow page rounding.

Suggested-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
v1 -> v2:
- Use existing HugeTLB CMA memory first for crashkernel=size,cma and
  reserve dedicated CMA areas only for the remainder. (Kiryl)
---
 .../ABI/testing/sysfs-kernel-kexec-kdump      |  3 ++
 Documentation/admin-guide/kdump/kdump.rst     | 22 ++++++--
 .../admin-guide/kernel-parameters.txt         | 16 +++++-
 arch/arm64/mm/init.c                          |  2 +-
 arch/powerpc/include/asm/kexec.h              |  2 -
 arch/powerpc/kexec/core.c                     | 12 ++---
 arch/powerpc/mm/mem.c                         |  6 +--
 arch/riscv/mm/init.c                          |  2 +-
 arch/x86/kernel/setup.c                       |  2 +-
 include/linux/crash_reserve.h                 | 14 ++++-
 include/linux/hugetlb.h                       | 12 +++++
 kernel/crash_reserve.c                        | 51 ++++++++++++++-----
 mm/hugetlb_cma.c                              | 50 ++++++++++++++++++
 mm/mm_init.c                                  |  2 +
 14 files changed, 159 insertions(+), 37 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-kernel-kexec-kdump b/Documentation/ABI/testing/sysfs-kernel-kexec-kdump
index f59051b5d96d7..9197cbbcbe7c8 100644
--- a/Documentation/ABI/testing/sysfs-kernel-kexec-kdump
+++ b/Documentation/ABI/testing/sysfs-kernel-kexec-kdump
@@ -58,4 +58,7 @@ Description:	read only
 		the Contiguous Memory Allocator (CMA) area that are allocated
 		to the crash (kdump) kernel. It lists the start and end physical
 		addresses of CMA regions assigned for crashkernel use.
+		This includes existing HugeTLB CMA memory selected with
+		crashkernel=size,cma. These ranges remain usable
+		by the running kernel and are excluded from the vmcore.
 User:		kdump service
diff --git a/Documentation/admin-guide/kdump/kdump.rst b/Documentation/admin-guide/kdump/kdump.rst
index 7587caadbae1a..91779205d1456 100644
--- a/Documentation/admin-guide/kdump/kdump.rst
+++ b/Documentation/admin-guide/kdump/kdump.rst
@@ -313,9 +313,25 @@ crashkernel syntax
 
 4) crashkernel=size,cma
 
-	Reserve additional crash kernel memory from CMA. This reservation is
-	usable by the first system's userspace memory and kernel movable
-	allocations (memory balloon, zswap). Pages allocated from this memory
+	Reserve additional crash kernel memory from CMA. Existing HugeTLB CMA
+	memory is used first, and dedicated CMA areas are reserved for any
+	remaining amount. For example, this reuses 2G from the existing 6G pool
+	without reserving additional CMA memory::
+
+	    hugetlb_cma=6G crashkernel=256M crashkernel=2G,cma
+
+	With a smaller HugeTLB CMA pool, the remainder is reserved separately::
+
+	    hugetlb_cma=2G crashkernel=256M crashkernel=4G,cma
+
+	This reuses the existing 2G and reserves another 2G of dedicated CMA.
+	Both kinds of area use the same limit of four physical ranges. If there
+	is insufficient memory or too few range slots, the kernel reports the
+	shortfall. Normal CMA alignment applies to dedicated reservations.
+
+	The selected memory remains usable by the first system's userspace,
+	including HugeTLB allocations, and kernel movable allocations
+	(memory balloon, zswap). Pages allocated from this memory
 	range will not be included in the vmcore so this should not be used if
 	dumping of userspace memory is intended and it has to be expected that
 	some movable kernel pages may be missing from the dump.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..a1fb40e500097 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1100,7 +1100,21 @@ Kernel parameters
 			or memory reserved is below 4G.
 	crashkernel=size[KMG],cma
 			[KNL, X86, ARM64, RISCV, PPC] Reserve additional crash
-			kernel memory from CMA. This reservation is usable by
+			kernel memory from CMA. Existing hugetlb_cma memory is
+			used first, and dedicated CMA areas are reserved for any
+			remaining amount. For example:
+			  hugetlb_cma=6G crashkernel=256M crashkernel=2G,cma
+			reuses 2G without reserving more CMA, while:
+			  hugetlb_cma=2G crashkernel=256M crashkernel=4G,cma
+			reuses 2G and reserves another 2G of dedicated CMA.
+
+			Both kinds of area use the same limit of four physical
+			ranges. If the available memory or range slots cannot
+			cover the requested size, the kernel reports the
+			shortfall. Normal CMA alignment applies to dedicated
+			reservations.
+
+			The selected memory remains usable by
 			the first system's userspace memory and kernel movable
 			allocations (memory balloon, zswap). Pages allocated
 			from this memory range will not be included in the vmcore
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 4e7682dfe366c..a781cc60b4282 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -114,7 +114,7 @@ static void __init arch_reserve_crashkernel(void)
 		return;
 
 	reserve_crashkernel_generic(crash_size, crash_base, low_size, high);
-	reserve_crashkernel_cma(cma_size);
+	crashkernel_cma_set_size(cma_size);
 }
 
 static phys_addr_t __init max_zone_phys(phys_addr_t zone_limit)
diff --git a/arch/powerpc/include/asm/kexec.h b/arch/powerpc/include/asm/kexec.h
index e02710d6a2e1a..d3b5028102cdd 100644
--- a/arch/powerpc/include/asm/kexec.h
+++ b/arch/powerpc/include/asm/kexec.h
@@ -117,11 +117,9 @@ int setup_new_fdt_ppc64(const struct kimage *image, void *fdt, struct crash_mem
 #ifdef CONFIG_CRASH_RESERVE
 int __init overlaps_crashkernel(unsigned long start, unsigned long size);
 extern void arch_reserve_crashkernel(void);
-extern void kdump_cma_reserve(void);
 #else
 static inline void arch_reserve_crashkernel(void) {}
 static inline int overlaps_crashkernel(unsigned long start, unsigned long size) { return 0; }
-static inline void kdump_cma_reserve(void) { }
 #endif
 
 #if defined(CONFIG_CRASH_DUMP)
diff --git a/arch/powerpc/kexec/core.c b/arch/powerpc/kexec/core.c
index dc44f11be353e..1b70eb10beac2 100644
--- a/arch/powerpc/kexec/core.c
+++ b/arch/powerpc/kexec/core.c
@@ -62,8 +62,6 @@ void machine_kexec(struct kimage *image)
 
 #ifdef CONFIG_CRASH_RESERVE
 
-static unsigned long long crashk_cma_size;
-
 static unsigned long long __init get_crash_base(unsigned long long crash_base)
 {
 
@@ -107,6 +105,7 @@ static unsigned long long __init get_crash_base(unsigned long long crash_base)
 void __init arch_reserve_crashkernel(void)
 {
 	unsigned long long crash_size, crash_base, crash_end;
+	unsigned long long cma_size = 0;
 	unsigned long long kernel_start, kernel_size;
 	unsigned long long total_mem_sz;
 	int ret;
@@ -115,7 +114,7 @@ void __init arch_reserve_crashkernel(void)
 
 	/* use common parsing */
 	ret = parse_crashkernel(boot_command_line, total_mem_sz, &crash_size,
-				&crash_base, NULL, &crashk_cma_size, NULL);
+				&crash_base, NULL, &cma_size, NULL);
 
 	if (ret)
 		return;
@@ -133,12 +132,7 @@ void __init arch_reserve_crashkernel(void)
 	}
 
 	reserve_crashkernel_generic(crash_size, crash_base, 0, false);
-}
-
-void __init kdump_cma_reserve(void)
-{
-	if (crashk_cma_size)
-		reserve_crashkernel_cma(crashk_cma_size);
+	crashkernel_cma_set_size(cma_size);
 }
 
 int __init overlaps_crashkernel(unsigned long start, unsigned long size)
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index b617b69452cd6..d6e84dc1e24c5 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -272,12 +272,10 @@ void __init arch_mm_preinit(void)
 {
 
 	/*
-	 * Reserve large chunks of memory for use by CMA for kdump, fadump, KVM
-	 * and hugetlb. These must be called after pageblock_order is
-	 * initialised.
+	 * Reserve large chunks of memory for use by CMA for fadump and KVM.
+	 * These must be called after pageblock_order is initialised.
 	 */
 	fadump_cma_init();
-	kdump_cma_reserve();
 	kvm_cma_reserve();
 
 	/*
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index e822fce0af4b0..194045dc3fed6 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -1357,7 +1357,7 @@ static void __init arch_reserve_crashkernel(void)
 		return;
 
 	reserve_crashkernel_generic(crash_size, crash_base, low_size, high);
-	reserve_crashkernel_cma(cma_size);
+	crashkernel_cma_set_size(cma_size);
 }
 
 void __init paging_init(void)
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 19fc8e1714e64..8848e207780b5 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -627,7 +627,7 @@ static void __init arch_reserve_crashkernel(void)
 	}
 
 	reserve_crashkernel_generic(crash_size, crash_base, low_size, high);
-	reserve_crashkernel_cma(cma_size);
+	crashkernel_cma_set_size(cma_size);
 }
 
 static struct resource standard_io_resources[] = {
diff --git a/include/linux/crash_reserve.h b/include/linux/crash_reserve.h
index 30864d90d7f50..b11d7814979cc 100644
--- a/include/linux/crash_reserve.h
+++ b/include/linux/crash_reserve.h
@@ -5,6 +5,7 @@
 #include <linux/linkage.h>
 #include <linux/elfcore.h>
 #include <linux/elf.h>
+#include <linux/range.h>
 #ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION
 #include <asm/crash_reserve.h>
 #endif
@@ -31,7 +32,18 @@ int __init parse_crashkernel(char *cmdline, unsigned long long system_ram,
 		unsigned long long *low_size, unsigned long long *cma_size,
 		bool *high);
 
-void __init reserve_crashkernel_cma(unsigned long long cma_size);
+#if defined(CONFIG_CRASH_RESERVE) && defined(CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION)
+void __init crashkernel_cma_set_size(unsigned long long size);
+void __init reserve_crashkernel_cma(void);
+#else
+static inline void __init crashkernel_cma_set_size(unsigned long long size)
+{
+}
+
+static inline void __init reserve_crashkernel_cma(void)
+{
+}
+#endif
 
 #ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION
 #ifndef arch_add_crash_res_to_iomem
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 58f755e70e966..69732e4cc03e2 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -1349,12 +1349,24 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,
 	return ptl;
 }
 
+struct range;
+
 #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA)
 extern void __init hugetlb_cma_reserve(void);
+unsigned long long __init
+hugetlb_cma_add_crashkernel_ranges(unsigned long long size, struct range *ranges,
+				   int *nr_ranges, int max_ranges);
 #else
 static inline __init void hugetlb_cma_reserve(void)
 {
 }
+
+static inline unsigned long long
+hugetlb_cma_add_crashkernel_ranges(unsigned long long size, struct range *ranges,
+				   int *nr_ranges, int max_ranges)
+{
+	return 0;
+}
 #endif
 
 #ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
diff --git a/kernel/crash_reserve.c b/kernel/crash_reserve.c
index eee37a11380ce..53d76ba6b3c31 100644
--- a/kernel/crash_reserve.c
+++ b/kernel/crash_reserve.c
@@ -16,6 +16,7 @@
 #include <linux/kmemleak.h>
 #include <linux/cma.h>
 #include <linux/crash_reserve.h>
+#include <linux/hugetlb.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -340,9 +341,9 @@ int __init parse_crashkernel(char *cmdline,
 	 * optional CMA reservation
 	 * cma_base is ignored
 	 */
-	if (cma_size)
-		__parse_crashkernel(cmdline, 0, cma_size,
-			&cma_base, suffix_tbl[SUFFIX_CMA]);
+	if (cma_size && __parse_crashkernel(cmdline, 0, cma_size,
+					    &cma_base, suffix_tbl[SUFFIX_CMA]))
+		*cma_size = 0;
 #endif
 	if (!*crash_size)
 		ret = -EINVAL;
@@ -470,21 +471,41 @@ void __init reserve_crashkernel_generic(unsigned long long crash_size,
 }
 
 struct range crashk_cma_ranges[CRASHKERNEL_CMA_RANGES_MAX];
+static unsigned long long crashkernel_cma_size __initdata;
+
+void __init crashkernel_cma_set_size(unsigned long long cma_size)
+{
+	crashkernel_cma_size = cma_size;
+}
+
 #ifdef CRASHKERNEL_CMA
 int crashk_cma_cnt;
-void __init reserve_crashkernel_cma(unsigned long long cma_size)
+void __init reserve_crashkernel_cma(void)
 {
-	unsigned long long request_size = roundup(cma_size, PAGE_SIZE);
-	unsigned long long reserved_size = 0;
+	unsigned long long cma_size = crashkernel_cma_size;
+	unsigned long long request_size, reserved_size, reused_size;
 
-	if (!cma_size)
+	if (!cma_size || crashk_res.start >= crashk_res.end)
 		return;
 
+	if (cma_size > ULLONG_MAX - (PAGE_SIZE - 1)) {
+		pr_warn("crashkernel CMA size is too large\n");
+		return;
+	}
+	cma_size = roundup(cma_size, PAGE_SIZE);
+	reused_size = hugetlb_cma_add_crashkernel_ranges(cma_size,
+							 crashk_cma_ranges,
+							 &crashk_cma_cnt,
+							 CRASHKERNEL_CMA_RANGES_MAX);
+	reserved_size = reused_size;
+	request_size = cma_size - reused_size;
+
 	while (cma_size > reserved_size &&
 	       crashk_cma_cnt < CRASHKERNEL_CMA_RANGES_MAX) {
 
 		struct cma *res;
 
+		request_size = min(request_size, cma_size - reserved_size);
 		if (cma_declare_contiguous(0, request_size, 0, 0, 0, false,
 				       "crashkernel", &res)) {
 			/* reservation failed, try half-sized blocks */
@@ -500,21 +521,23 @@ void __init reserve_crashkernel_cma(unsigned long long cma_size)
 			crashk_cma_ranges[crashk_cma_cnt].start +
 			cma_get_size(res) - 1;
 		++crashk_cma_cnt;
-		reserved_size += request_size;
+		reserved_size += cma_get_size(res);
 	}
 
 	if (cma_size > reserved_size)
-		pr_warn("crashkernel CMA reservation failed: %lld MB requested, %lld MB reserved in %d ranges\n",
-			cma_size >> 20, reserved_size >> 20, crashk_cma_cnt);
+		pr_warn("crashkernel CMA reservation failed: %llu MB requested, %llu MB reused, %llu MB newly reserved in %d ranges\n",
+			cma_size >> 20, reused_size >> 20,
+			(reserved_size - reused_size) >> 20, crashk_cma_cnt);
 	else
-		pr_info("crashkernel CMA reserved: %lld MB in %d ranges\n",
-			reserved_size >> 20, crashk_cma_cnt);
+		pr_info("crashkernel CMA: %llu MB reused, %llu MB newly reserved in %d ranges\n",
+			reused_size >> 20, (reserved_size - reused_size) >> 20,
+			crashk_cma_cnt);
 }
 
 #else /* CRASHKERNEL_CMA */
-void __init reserve_crashkernel_cma(unsigned long long cma_size)
+void __init reserve_crashkernel_cma(void)
 {
-	if (cma_size)
+	if (crashkernel_cma_size)
 		pr_warn("crashkernel CMA reservation not supported\n");
 }
 #endif
diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index cd6a64986278a..cd6f64da31a2c 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -11,6 +11,8 @@
 #include <linux/memblock.h>
 #include <linux/math.h>
 #include <linux/math64.h>
+#include <linux/range.h>
+#include "cma.h"
 #include "internal.h"
 #include "hugetlb_cma.h"
 
@@ -387,6 +389,54 @@ void __init hugetlb_cma_reserve(void)
 		hugetlb_cma_size = 0;
 }
 
+/**
+ * hugetlb_cma_add_crashkernel_ranges - share HugeTLB CMA with the crash kernel
+ * @size: maximum number of bytes to share, aligned to PAGE_SIZE
+ * @ranges: crash kernel memory ranges
+ * @nr_ranges: number of entries already in @ranges, updated on return
+ * @max_ranges: capacity of @ranges
+ *
+ * The shared memory remains available to the HugeTLB allocator until a crash.
+ * Only physical ranges belonging to successfully reserved CMA areas are added.
+ *
+ * Return: number of bytes shared.
+ */
+unsigned long long __init
+hugetlb_cma_add_crashkernel_ranges(unsigned long long size, struct range *ranges,
+				   int *nr_ranges, int max_ranges)
+{
+	unsigned long long shared_size = 0;
+	int nid, i;
+
+	for_each_node_mask(nid, hugetlb_bootmem_nodes) {
+		struct cma *cma = hugetlb_cma[nid];
+
+		if (!cma)
+			continue;
+
+		for (i = 0; i < cma->nranges; i++) {
+			struct cma_memrange *cmr = &cma->ranges[i];
+			unsigned long long range_size;
+			struct range *range;
+
+			if (shared_size == size || *nr_ranges == max_ranges)
+				return shared_size;
+
+			range_size = min_t(unsigned long long,
+					   PFN_PHYS(cmr->count), size - shared_size);
+			range = &ranges[(*nr_ranges)++];
+			range->start = PFN_PHYS(cmr->base_pfn);
+			range->end = range->start + range_size - 1;
+			shared_size += range_size;
+
+			/* Keep shared pages reserved if CMA activation fails. */
+			cma_reserve_pages_on_error(cma);
+		}
+	}
+
+	return shared_size;
+}
+
 bool hugetlb_cma_exclusive_alloc(void)
 {
 	return hugetlb_cma_only;
diff --git a/mm/mm_init.c b/mm/mm_init.c
index a9d848c92678b..2367af05570bd 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -28,6 +28,7 @@
 #include <linux/swap.h>
 #include <linux/cma.h>
 #include <linux/crash_dump.h>
+#include <linux/crash_reserve.h>
 #include <linux/execmem.h>
 #include <linux/sizes.h>
 #include <linux/vmstat.h>
@@ -2686,6 +2687,7 @@ void __init mm_core_init_early(void)
 	free_area_init();
 
 	hugetlb_cma_reserve();
+	reserve_crashkernel_cma();
 	hugetlb_bootmem_alloc();
 
 	sparse_init();
-- 
2.53.0-Meta