From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FBD725B08D for ; Wed, 13 May 2026 13:08:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677737; cv=none; b=mCPBL0A5hLWB24Dfi+mNE/98a0KDsRiTABz678O/hSwdJ72Bf/JCGYYB7sX7OOCfJ4BL7RytQakPJdmSLauW+sqBFCtgaTkJxr6EEZH6kUbwkH/Ml51HRpMxWEjtzeyYpGZHo9Z/2d51hfz7yOmQ+Pm2olYI7+No/DgEh9gKR0o= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677737; c=relaxed/simple; bh=m9PYSf0N9zNFHMLHSpi/NK3N/H04nmx0abDnoXp185g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SMu/Ip8MHu85uIiJKow0TL4Pku+y9WcTqeCKTuryS3H5tKaGg92Vqxppz1WM3iX3WqoFYAiU3V+VzE6nkQj/MvSXXU54zJ8/wCguOifjKYoUCAUsdKctPv4chOEf5esvL0fH2EzApnnUcgXDJKOKjBw+co7PQ1uLAlvg4k9p6bY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BSXsuPLc; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BSXsuPLc" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-366087480d8so6040143a91.3 for ; Wed, 13 May 2026 06:08:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677736; x=1779282536; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NFXyqH1Cou6po67hW+Tbj9/tmuAtnCuDUU0al6ZhZaU=; b=BSXsuPLc1fgOW6KoU9Aj07XNolIeoJSCjXefKKpVENdJZxeoro6XHMfVT6X0BIFU8d WPnEJbkq5h/jZMriiXZK6kAuLNSzd9Y1oisZM/q7E7+x3ZABteot0+3VqxaQkt5cI1K/ UEPWPDhLriUc/eZEEJnJR9l63kHpsxfusiVqS2zpis0YX8jval25QHJUDgoYmo++EShI 8gBcrhuB4pJcukTx9s1gc+glddZuukGPCBBQzujLDCXTWysJsaF1SSjwuY1rltf7tNjT 39nVDDeS09evVx4T7bmLjdwl9ZsSCEfjSImPdjW51mGomgtwJlEGCxBmj50N+xdSvEE3 QwUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677736; x=1779282536; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=NFXyqH1Cou6po67hW+Tbj9/tmuAtnCuDUU0al6ZhZaU=; b=iruXP5Pt/biwvLkpDalUdK8zpZPqgsuidz/tUTITTPx59PBtsRwP6eUL/Qq/QIYvZl 5KVKkHUK9Ruv32YGy1fQHJI/jHWr/uyJ7mN/M9Pt4Z5ZLbBVMS9zFBskvcg+eMNYZW5A UCXlmsAi5dtMivvQGMgbq0rVMhZeKzt9T8e6btV3x0KT5+8xTCFBr+R7F5VcNyL+DREh ckUeB0HgDlssMI3cFiiF7y0l1+KJTVKtbBlgGiv+PiKBIEo9uxre1dNtnSHHzP0ONSHH zQsPQJlDZE9KMLU1+h/vOYAmZwzilQNYOyGrd9SlSZHPaW5kzu7JxS6NNMBvk67bYtXS 5+8A== X-Forwarded-Encrypted: i=1; AFNElJ+foLXjRW60mXz00dXlTyjlJtDiWwpXJYvSv3shNW8CFv1A7P7G9DGdkcruW/ymqOSYm5T+hBb0oupaZ2A=@vger.kernel.org X-Gm-Message-State: AOJu0YzSBb1+cDDKUGKHOzVYAsZ7AXcu3OzlvYsKYFsMG43F6Xqtxt0U af0MpkX1veA8QGtuRj/k8OAVyDvV7otSNqhyga+VzFQUOfY0hs4bZdZwotyGwb2JT8U= X-Gm-Gg: Acq92OFjmlDze8zA6BHJwwLRo64k1UhGFEk1cV4HG3agY5Dx9oE/uBB8eMWmdYa9RoS kLdTk10bRlExLj2TrW+a4DjYxHE6a1mLtdvB7K+1EEiU5U/N3z3RE0Oaz17O2kx0tDcTXfNljCD 5/ofwADxVL5C0eYgw3aquf7DDmO78RM/EkwwPPo6CeoEdWicJoMWEaXD98fnEKyGRJa8adklON9 ezdZqxCsjwVj7R0ANGiQsg/NFnpwAz7oL9VvFV2b1GwbZE1UIaO0TFDjGbGiFI70sFmzEvV+g6v SspvectWYj5SJziydPaVmRRqneUFIKhcW7hzCOWEo/y/YWcZa8dI1W3L8RUJXApwoDe8bWSomLj 6V2TPhrUEIsR5UHOJiHkYRJ/wfk24oUiP8Fo5synk/RbPeVqSnrq+cNQwwJd6u3qWybSu8F4RSL f5UgDiunQ8c8XeUywuCWtMP0H1gy1FnqfJQHgjDEMWUqHrDWfakLNH5pWkWr8= X-Received: by 2002:a17:90a:d00e:b0:368:b724:6d53 with SMTP id 98e67ed59e1d1-368f398cc9dmr3518409a91.4.1778677735539; Wed, 13 May 2026 06:08:55 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:55 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 15/69] mm/hugetlb_vmemmap: Move bootmem HVO setup to early init Date: Wed, 13 May 2026 21:04:43 +0800 Message-ID: <20260513130542.35604-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Bootmem HugeTLB pages currently defer HVO setup to hugetlb_vmemmap_init_late(), because the optimization needs zone information. Now that zone initialization is available earlier, the bootmem HVO setup can be done directly from hugetlb_vmemmap_init_early(). This lets gigantic HugeTLB pages apply HVO as soon as they are allocated. Bootmem gigantic pages that span multiple zones are now filtered out when they are allocated, so the remaining bootmem gigantic pages seen by later hugetlb initialization are already zone-valid. As a result, hugetlb_vmemmap_init_late() no longer needs to handle bootmem HVO setup. Signed-off-by: Muchun Song --- mm/hugetlb_vmemmap.c | 67 +++++++++----------------------------------- 1 file changed, 13 insertions(+), 54 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4f58cd940f61..e2251bc47444 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -745,6 +745,8 @@ static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m) return true; } +static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn); + /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -752,6 +754,7 @@ void __init hugetlb_vmemmap_init_early(int nid) { unsigned long psize, paddr, section_size; unsigned long ns, i, pnum, pfn, nr_pages; + unsigned long start, end; struct huge_bootmem_page *m = NULL; void *map; @@ -761,6 +764,8 @@ void __init hugetlb_vmemmap_init_early(int nid) section_size = (1UL << PA_SECTION_SHIFT); list_for_each_entry(m, &huge_boot_pages[nid], list) { + struct zone *zone; + if (!vmemmap_should_optimize_bootmem_page(m)) continue; @@ -769,6 +774,14 @@ void __init hugetlb_vmemmap_init_early(int nid) paddr = virt_to_phys(m); pfn = PHYS_PFN(paddr); map = pfn_to_page(pfn); + start = (unsigned long)map; + end = start + hugetlb_vmemmap_size(m->hstate); + zone = pfn_to_zone(nid, pfn); + + if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), + zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) + panic("Failed to allocate memmap for HugeTLB page\n"); + memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SIZE)); pnum = pfn_to_section_nr(pfn); ns = psize / section_size; @@ -800,60 +813,6 @@ static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) void __init hugetlb_vmemmap_init_late(int nid) { - struct huge_bootmem_page *m, *tm; - unsigned long phys, nr_pages, start, end; - unsigned long pfn, nr_mmap; - struct zone *zone = NULL; - struct hstate *h; - void *map; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return; - - list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { - if (!(m->flags & HUGE_BOOTMEM_HVO)) - continue; - - phys = virt_to_phys(m); - h = m->hstate; - pfn = PHYS_PFN(phys); - nr_pages = pages_per_huge_page(h); - map = pfn_to_page(pfn); - start = (unsigned long)map; - end = start + nr_pages * sizeof(struct page); - - if (!hugetlb_bootmem_page_zones_valid(nid, m)) { - /* - * Oops, the hugetlb page spans multiple zones. - * Remove it from the list, and populate it normally. - */ - list_del(&m->list); - - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - - memblock_phys_free(phys, huge_page_size(h)); - continue; - } - - if (!zone || !zone_spans_pfn(zone, pfn)) - zone = pfn_to_zone(nid, pfn); - if (WARN_ON_ONCE(!zone)) - continue; - - if (vmemmap_populate_hvo(start, end, huge_page_order(h), zone, - HUGETLB_VMEMMAP_RESERVE_SIZE) < 0) { - /* Fallback if HVO population fails */ - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - } else { - m->flags |= HUGE_BOOTMEM_ZONES_VALID; - nr_mmap = HUGETLB_VMEMMAP_RESERVE_SIZE; - } - - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - } } #endif -- 2.54.0