From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 266DE395AFC for ; Fri, 12 Jun 2026 04:00:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781236829; cv=none; b=PxUbSYTwk7OfQ7NFwgQXFl+nLsdleu8cVasjzmh8OZHuhS79hQMjeORGwxOd18OYbtRWnCDwz5I8bysDbyFWxzUZ5M4wcIfKqNGXPPMERz9JC0uXj8ksyMJESJWuAeERfLh6HFE6umh1kiTDoP7SGSEm/OVT5ccyFl2Y989oG3U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781236829; c=relaxed/simple; bh=9Dj/ERIPLvIIFCDt+Qnl6vIguBpOSlgrmIGAsPoeGRg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rti8kngl3T607WOLElDX9z3ZdiWcS8IsSgCAGB72QgOGHIB8mlqhOOno7zp0TKM7xT0E3HjOxjOkNnvgjWFcA25iHVOt6heYEEjclXI1WBdawblvy1Uefhfm1FhYkPKcVhEkQDR8M/IW/+sDcjHIJvAzOK0XfDMaBeGw2QZAROI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RR9e5HDS; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RR9e5HDS" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-84275887a3fso1129693b3a.1 for ; Thu, 11 Jun 2026 21:00:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1781236827; x=1781841627; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z7F0jyR+NYrp4SN+/Gr3YYzk571idnjCyjkNavYszN4=; b=RR9e5HDSuEjFwO2DHFOFSPjb1raX/aeA8dUql6Y+l/9vPmTzmXYsvfQBL3m8LulDGB 7ub6Z5VHY+xagy8G9OJE/t65qBriHBwVhJZkzAQxKY0oHg2kp+OPaTOj81sOa8IziURE LiS/2FMcXdBByFNndapWugby0oh75/MfIjWtls9r++/J3Joqki/Q/m/i3wdwvdYHVLhE ao4u4KizwDrA+jQZUB1R6nJXeckuoX1lrGQO+ZLajQQfd4VqtyL4VTGLtGFI3IpswFFX i4DKtGn0IltubdPXfNaxQXZ0WsDlIuPNm703QEScDxLikjP9PxxQcHNTjNzuL12RCQuR 9RGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781236827; x=1781841627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=z7F0jyR+NYrp4SN+/Gr3YYzk571idnjCyjkNavYszN4=; b=kiDde0T+z/lR/kQqxthUh9wuvn1oymNmeLIZXCdzrD80A0OYmCzP2Exw1KKG1iqGSY BvieJtBLHF83PQXyuqhlLnls5aMukmKF11AuAwxqwtV9pF5ajK5vsB97EF/MnLgH1Suu gbMg6sXfzu64LKJ6ZIOAhOz/aW1sKW/XKRzzm3JpyGGhPts6ynm9y3oaFUm/S9u3TRBU 5ra01OJC/xDiMxSIJv5uA3j/mWbHgVm98GS+9meScSBpby3kxgfqIXvSGeZtSIrFBy1w 8KxBWrgQk5kBOvJKDi/sxKicB0RxU1wP+Juld5f7NUGJa/gsqfEhRi3iae6OkWaCKots IwLw== X-Forwarded-Encrypted: i=1; AFNElJ+b8cmfFFUTQfjOw4Jjr87EV7lPIqwTzIt2VAqc+vv77W6x6gWT00I5I9K2+POMiRYhw8wjLjoX/9V+ovM=@vger.kernel.org X-Gm-Message-State: AOJu0YxbZzSEyX0hv5tqxX+ag73cYReDniC1vTnuf/F13ShkqIUFa4v6 hVOeb6JsqpjTat36Ign5CTn3BftLg31tvTe35rDj0zjl7Z3DeVkLYGeZj0rgOW60wLA= X-Gm-Gg: Acq92OFrvyxyMTOcd2a0SweIy5a6a2tSIngfZpN69TtKC2oQUfvXeVj0elFVRlDeSVN xTNwwL9MBQonI3YyuJaTg7KQRfq77NcBBa/Qslj7NkV1uk3pfNnbS/2Xdwu5HdYu3DR+f3UwRH+ qeyLUdaDEOjJw8CvpZqghpbqJpcPWpOrNFcqwD1+r2pTC62olT7TY59NldYrVV8wiz65bdVjI1C fhzibrWskQ5WTOi1WtukE7JEZUmC7BDfbA1TmQTTWdhBR+OLy44atbtmZfo31XgAdOsU5fkTarP xTir+Sff3WMVrD8hsCnUeNTnJVIJvYeiDIQLvadGYYQRrngGI9zUk1oymvDiNOqXCV9pg/+ImgE BdcjdOUaS2Ps3km3g+tfxmMkLpvl/Ab4PQSN3ONIk4cjyZM62lo7lKjXT+xPo7QYI/49oRW504+ kLFJL9gc8cCsB457kHvjhbpURHrB2wMeR/mUy9h+Ahf8ShpexC+Qe6Vw== X-Received: by 2002:aa7:888f:0:b0:842:47f7:5f1 with SMTP id d2e1a72fcca58-8434978f969mr1354803b3a.28.1781236827034; Thu, 11 Jun 2026 21:00:27 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.99]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-8434ad03fdcsm643352b3a.24.2026.06.11.21.00.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jun 2026 21:00:26 -0700 (PDT) From: Muchun Song To: Oscar Salvador , David Hildenbrand , Andrew Morton , Madhavan Srinivasan , Michael Ellerman Cc: Muchun Song , Mike Rapoport , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Nicholas Piggin , Christophe Leroy , Ritesh Harjani , "Aneesh Kumar K . V" , linuxppc-dev@lists.ozlabs.org, Mike Kravetz , Muchun Song , Usama Arif Subject: [PATCH v4 15/19] mm/hugetlb_vmemmap: Move bootmem HVO setup to early init Date: Fri, 12 Jun 2026 11:58:59 +0800 Message-ID: <20260612035903.2468601-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260612035903.2468601-1-songmuchun@bytedance.com> References: <20260612035903.2468601-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Bootmem HugeTLB pages currently defer HVO setup to hugetlb_vmemmap_init_late(), because the optimization needs zone information. Now that zone initialization is available earlier, the bootmem HVO setup can be done directly from hugetlb_vmemmap_init_early(). This lets gigantic HugeTLB pages apply HVO as soon as they are allocated. Bootmem gigantic pages that span multiple zones are now filtered out when they are allocated, so the remaining bootmem gigantic pages seen by later hugetlb initialization are already zone-valid. As a result, hugetlb_vmemmap_init_late() no longer needs to handle bootmem HVO setup. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Usama Arif --- v3->v4: - Move pfn_to_zone() before hugetlb_vmemmap_init_early() instead of adding a forward declaration (suggested by Mike Rapoport) - Collect Acked-by from Mike Rapoport and Usama Arif --- mm/hugetlb_vmemmap.c | 93 ++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 68 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index ea6af85bfec1..ee4fbd5fed0d 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -745,6 +745,20 @@ static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m) return true; } +static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) +{ + struct zone *zone; + enum zone_type zone_type; + + for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { + zone = &NODE_DATA(nid)->node_zones[zone_type]; + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -752,6 +766,7 @@ void __init hugetlb_vmemmap_init_early(int nid) { unsigned long psize, paddr, section_size; unsigned long ns, i, pnum, pfn, nr_pages; + unsigned long start, end; struct huge_bootmem_page *m = NULL; void *map; @@ -761,6 +776,8 @@ void __init hugetlb_vmemmap_init_early(int nid) section_size = (1UL << PA_SECTION_SHIFT); list_for_each_entry(m, &huge_boot_pages[nid], list) { + struct zone *zone; + if (!vmemmap_should_optimize_bootmem_page(m)) continue; @@ -769,6 +786,14 @@ void __init hugetlb_vmemmap_init_early(int nid) paddr = virt_to_phys(m); pfn = PHYS_PFN(paddr); map = pfn_to_page(pfn); + start = (unsigned long)map; + end = start + hugetlb_vmemmap_size(m->hstate); + zone = pfn_to_zone(nid, pfn); + + if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), + zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) + panic("Failed to allocate memmap for HugeTLB page\n"); + memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SIZE)); pnum = pfn_to_section_nr(pfn); ns = psize / section_size; @@ -784,76 +809,8 @@ void __init hugetlb_vmemmap_init_early(int nid) } } -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) -{ - struct zone *zone; - enum zone_type zone_type; - - for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { - zone = &NODE_DATA(nid)->node_zones[zone_type]; - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; -} - void __init hugetlb_vmemmap_init_late(int nid) { - struct huge_bootmem_page *m, *tm; - unsigned long phys, nr_pages, start, end; - unsigned long pfn, nr_mmap; - struct zone *zone = NULL; - struct hstate *h; - void *map; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return; - - list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { - if (!(m->flags & HUGE_BOOTMEM_HVO)) - continue; - - phys = virt_to_phys(m); - h = m->hstate; - pfn = PHYS_PFN(phys); - nr_pages = pages_per_huge_page(h); - map = pfn_to_page(pfn); - start = (unsigned long)map; - end = start + nr_pages * sizeof(struct page); - - if (!hugetlb_bootmem_page_zones_valid(nid, m)) { - /* - * Oops, the hugetlb page spans multiple zones. - * Remove it from the list, and populate it normally. - */ - list_del(&m->list); - - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - - memblock_phys_free(phys, huge_page_size(h)); - continue; - } - - if (!zone || !zone_spans_pfn(zone, pfn)) - zone = pfn_to_zone(nid, pfn); - if (WARN_ON_ONCE(!zone)) - continue; - - if (vmemmap_populate_hvo(start, end, huge_page_order(h), zone, - HUGETLB_VMEMMAP_RESERVE_SIZE) < 0) { - /* Fallback if HVO population fails */ - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - } else { - m->flags |= HUGE_BOOTMEM_ZONES_VALID; - nr_mmap = HUGETLB_VMEMMAP_RESERVE_SIZE; - } - - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - } } #endif -- 2.54.0