From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D87B331EA9 for ; Fri, 12 Jun 2026 04:00:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781236827; cv=none; b=XTORbUfTu/5lydIquGmgbCRLwW0TTLH0v5KWY7/nCPC/L+zNungKMaLGDVycROBgkhLNJGXpWZL3+jbMciByFHbPEo9eL6Hg6NlD88wxXCPk6YZ6Y7PeNmpPX9lXanaByXBB6SNGoMQmhpA1+6nJ39zI6R/ad1rUgyTETxvR9vo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781236827; c=relaxed/simple; bh=udw/SeGPA6CD6uaMzNqCLS4SxbhM3Wn0KnnJN2byGmk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nfst7RbmqAuEByamjvR/bFosGzmDnGYzSQn5MPDeeCBnUWdRE7+zy+dr8X14sxlCiuoLGhB7iRM8DNmnRl/3i0Q81aD2uWdCGBiqCNYMltiVN5R7b8jVym4s2DRc5iGPqVlHWuqB3n0sgk6P9gGmKTHGMDCUujFLvdqYsrkV7gc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JRvu6aWT; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JRvu6aWT" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-842cd900ee0so267868b3a.2 for ; Thu, 11 Jun 2026 21:00:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1781236825; x=1781841625; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E/u6lKC7hQ/GhzoNdcnEA7gND1myfeYXkd1NgoaUr0Y=; b=JRvu6aWTlb1FcD1wZTLslEVh41FKusPzgGXO/duqya0mQXqzbTpWuOSC9ywhZ/yEO7 Ke4zLc1WA67Mqb1Sms0DspZRVEY3+hZER+gWpSTe3Ecb/V7tHTnhNVrbiGRv9Co3/RWy lirT0THZy6HOWFHg7G/fFqv9OFvhUKEk7d98W1LP/ro0R8CHxktJUfOKPeOAruPYjcIC 31DMIufoDiehqhhGAgCU/QrXwQNRaseiUyKoj6vhzenJv29dR+U6+qYcCat5oBaVXXPI TekvOFcqBtBCHLMasrNr/NifEV4lTfng0VSNoRrNMqqKoD/NvvYd6ITvfBKZhLlFUCsb DAHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781236825; x=1781841625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=E/u6lKC7hQ/GhzoNdcnEA7gND1myfeYXkd1NgoaUr0Y=; b=UaqNO1Bl0Mr4nha/1qNizNjk139kP8kmILzvS2MC6O+v8gz86ZkqqkgrZZWwnJ1Bey Hpp8qTlJ/BQvKdqfTKejdb+NknZ0DMHHgBGyPw5IldlWH8V+SO1SIsm6ahN+38JGg5Vn q+csJmaVkMSodetswhzgvh7C475l5o+dMNH0djc5wL7flg+4900JMjRE8mavtSvOgPsq vRT1y9syrWN5KxGFnq/odHUJB213buG3FTKOqgDqYBF27pK9PgH558IdWsNCpdce2JV4 RUIn6+BQmwk47jBYbvrHwLw3I8311r3VBygRAjYZ+QKr6q1w5tMClH5Iq7zlhSr6L+/E nPmg== X-Forwarded-Encrypted: i=1; AFNElJ9fJgTDtKWcgJKdkFeyEfhFD1Jiwv5mKQFawZ5dzBsi1fd/MzMcuoEWjc2IlYkGQyw18+zqbA7CCJuY9OE=@vger.kernel.org X-Gm-Message-State: AOJu0Yza99sOgi//OqK020Rb12vj9dyX4aV8z0luA0vh+ghHKBB6bQiv IZlDmB5ypJgSMFcc1IoNKeJNI6ewcFyDjjQn5Dc2TcHAHJqGno9afNQODRl77jmWFzE= X-Gm-Gg: Acq92OG2DkxjC5dUA9xY/sr3VQ1a5lhY8dthBIYv/2N1+3njNMTknRoeNnMDPI7f862 Ufi0akCisNgS0WNr/liIV2CyBcmpABSwrq6aJ+ZKbEbbDi7c1XrX1jElK8jNmP82R7GLIGLHBSt 1wTH+LiajrirTjhIqUKF3IPzcR3iJ3A16GdeY3fPK7fyU1gzt1ndEimgGwMt5gHeWEeuRKgq/65 7fhxjhAfrS0sy1Wa4D3KFFwFkiQ+In2DG0ct58ygJdwOfIpNcU8lTlqnogUU0LtctyMd+EpVz/b BpHokCyEjc7/j3xzo7Gip6IoVtLvbe8oxtA1wxgB3FyGt3sTFJ2yf7boVELnimslrhMJCMk6rky u4uTvPDVqbMTLblzP0NnO1xAM/Hx4UzWxmTuZW0PRIBoQNjjoYpqS2uXx9257vztZ12jtlCWmnb ChWrjm5kFO/9+TYFWZOyWiiNErDNTfFfDgJeygpDDbyUs= X-Received: by 2002:a05:6a00:114c:b0:842:4b88:20ee with SMTP id d2e1a72fcca58-8434d103dbemr974121b3a.44.1781236821951; Thu, 11 Jun 2026 21:00:21 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.99]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-8434ad03fdcsm643352b3a.24.2026.06.11.21.00.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 21:00:21 -0700 (PDT) From: Muchun Song To: Oscar Salvador , David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman Cc: Muchun Song , Mike Rapoport , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Nicholas Piggin , Christophe Leroy , Ritesh Harjani , "Aneesh Kumar K . V" , linuxppc-dev@lists.ozlabs.org, Mike Kravetz , Muchun Song Subject: [PATCH v4 14/19] mm/hugetlb: Free cross-zone bootmem gigantic pages after allocation Date: Fri, 12 Jun 2026 11:58:58 +0800 Message-ID: <20260612035903.2468601-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260612035903.2468601-1-songmuchun@bytedance.com> References: <20260612035903.2468601-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Now that hugetlb reservation runs after zone initialization, bootmem gigantic page allocation can detect pages that span multiple zones. Keep those cross-zone pages separate during allocation and free them after allocation completes, so later hugetlb initialization only sees zone-valid gigantic pages. This chooses to free cross-zone gigantic pages directly instead of retrying allocation. In practice, such cross-zone cases are expected to be very rare, so adding retry logic does not seem justified at this point. Keeping the handling simple also preserves the previous behavior. If similar real-world reports show up later, retry support can be reconsidered then. Signed-off-by: Muchun Song --- mm/hugetlb.c | 75 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5e557c05d80a..218fb1ca45f4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3060,12 +3060,15 @@ void *__init __alloc_bootmem_huge_page(struct hstate *h, int nid) static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) { + unsigned long pfn; + unsigned int nid_request = nid; struct huge_bootmem_page *m = arch_alloc_bootmem_huge_page(h, nid); if (!m) return false; - nid = early_pfn_to_nid(PHYS_PFN(__pa(m))); + pfn = PHYS_PFN(__pa(m)); + nid = early_pfn_to_nid(pfn); /* * Use the beginning of the huge page to store the huge_bootmem_page * struct (until gather_bootmem puts them into the mem_map). @@ -3073,22 +3076,38 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) * Put them into a private list first because mem_map is not up yet. */ INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[nid]); m->hstate = h; if (!hugetlb_early_cma(h)) { m->cma = NULL; m->flags = 0; } - /* - * Only initialize the head struct page in memmap_init_reserved_pages, - * rest of the struct pages will be initialized by the HugeTLB - * subsystem itself. - * The head struct page is used to get folio information by the HugeTLB - * subsystem like zone id and node id. - */ - memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), - huge_page_size(h) - PAGE_SIZE); + /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ + if (!hugetlb_early_cma(h) && + pfn_range_intersects_zones(nid, pfn, pages_per_huge_page(h))) { + /* + * If the allocated page is on a different node than requested + * (e.g., on PowerPC LPARs), put it on the requested node's list, + * because hugetlb_free_cross_zone_pages() only frees cross-zone + * pages belonging to the requested node. + */ + if (WARN_ON_ONCE(nid_request != NUMA_NO_NODE && nid != nid_request)) + list_add(&m->list, &huge_boot_pages[nid_request]); + else + list_add(&m->list, &huge_boot_pages[nid]); + } else { + list_add_tail(&m->list, &huge_boot_pages[nid]); + m->flags |= HUGE_BOOTMEM_ZONES_VALID; + /* + * Only initialize the head struct page in memmap_init_reserved_pages, + * rest of the struct pages will be initialized by the HugeTLB + * subsystem itself. + * The head struct page is used to get folio information by the HugeTLB + * subsystem like zone id and node id. + */ + memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), + huge_page_size(h) - PAGE_SIZE); + } return true; } @@ -3373,6 +3392,34 @@ void __init hugetlb_bootmem_struct_page_init(void) padata_do_multithreaded(&job); } +static unsigned long __init hugetlb_free_cross_zone_pages(struct hstate *h, int nid) +{ + unsigned long freed = 0; + struct huge_bootmem_page *m, *tmp; + + if (!hstate_is_gigantic(h)) + return freed; + + list_for_each_entry_safe(m, tmp, &huge_boot_pages[nid], list) { + if (m->flags & HUGE_BOOTMEM_ZONES_VALID) + break; + + list_del(&m->list); + memblock_free(m, huge_page_size(h)); + freed++; + } + + if (freed) { + char buf[32]; + + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_warn("HugeTLB: freed %lu cross-zone hugepages of size %s on node %d.\n", + freed, buf, nid); + } + + return freed; +} + static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { unsigned long i; @@ -3403,6 +3450,8 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) cond_resched(); } + i -= hugetlb_free_cross_zone_pages(h, nid); + if (!list_empty(&folio_list)) prep_and_add_allocated_folios(h, &folio_list); @@ -3476,6 +3525,7 @@ static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned l static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) { + int nid; unsigned long i; for (i = 0; i < h->max_huge_pages; ++i) { @@ -3484,6 +3534,9 @@ static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) cond_resched(); } + for_each_node(nid) + i -= hugetlb_free_cross_zone_pages(h, nid); + return i; } -- 2.54.0