From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7BE33B4404 for ; Thu, 18 Jun 2026 08:48:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772482; cv=none; b=BiZCnlcgX2/zzEmRKiB2sDBo9ko5WMLAQ5BcSRM+1WHaQzMhdmTLYEQau9TAAZE7vHohv9VtDPBcSacyjN0XEjB2UYY93H6vxBRqPheFw1+zH6iFPRCV0qiiN2tKRd4NZM/C/tHSuRZmv1+rHhmsesEnPvPj8OkdZHGvcMoJsWo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772482; c=relaxed/simple; bh=0jA4bWRxCS9RYlViZzqAf2RUbzV0nYU610uVSvrK/KQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RRjBNmTD0vWcoon/ID4m9oL9GpUCG8IRQHqPpZ4VVwI1ceEZovlnx+eipGEpSB2Kn7UvTjQnodaCq9+WTnOq0xVKaRYvn8nMVgmcR2VdLh7eF2QmVqcoIcxeXeNqSq0YZ5+/eUpPJftG/RT/UP90gWMaG3c0owhPGN5q9QjF6UA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rL/BMpGU; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rL/BMpGU" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-36bcbd7821fso221883a91.2 for ; Thu, 18 Jun 2026 01:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772480; x=1782377280; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OaoSxKyClWE3jnYRtJaf7Jb5FKch1UvGKeE4Hl6seUo=; b=rL/BMpGUW2/XCSN2HMJEfIp4nX9xWx3d/67hiN9fjc266ksze/Cd1XLf5iX5t9Yvjm oSFM8C3N1T8z8mwc8f3SmS97rCsGwSM+yghtthrhd0WCtSJnKefRrW20goaftRkF1ncL 59d7AGCJqbYfaThdWD0Mj9PQLHayo6NSr7sM/c29/pjDPv6P8Xth6t4p5bFIPLb4RtnZ eIla7aneNFVirHct1gejnJzeSPqKw6jGiKEjKok+FMXptbdn1ANjMZcxWIOpfhx7A73d ai4bC/jCQTSkWz1PEw3TXBT/ydIjLyVbnKuTwGVroV/TybpSQW8RiXocXgxVm9fO+JFP 7d7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772480; x=1782377280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OaoSxKyClWE3jnYRtJaf7Jb5FKch1UvGKeE4Hl6seUo=; b=aMP1de90LJ/KR1c4yjiSPIK6nImzHPfU2adbxGXtDjpMH07JM+PM3WhDhUHY/64VXs AYpMTFGI9prHZB9ZRyheHHTAUQZfQThFQhGdwcgt63Lol8BnuL0nHNaPRz6AuJee8/Cb hjkkgiPYEPuJqM+Tz7xySKOPmlymfoRTH2NFrxIp11wPN9596cTuNXJlWRVqxRfEEDLt S1tdIoFhis8/MMvGGWT2AlwJDkU/4+X+Tf628Ayr+NKsQBtb8XpSVCYiYtoJF2RYruwh 5ZmSYUJMyxa/m93XpPgGYJeQ21vrH0HV1YItTD4A148zwtCgPe3CAQJSXH369i/rHO+M 6pSw== X-Forwarded-Encrypted: i=1; AFNElJ9Pvc9vCB9vnBTSttqgIvsqzL3Q5moZ0tLfnEV4r1BPcEtri0gZa+aIYd5yktXk75hqNKXCLTDOv9cLDaY=@vger.kernel.org X-Gm-Message-State: AOJu0Yzpv4Lut+YzdA1PXa+dAsoYbkCZ/9tKcDtYVfFIx6KPu4p4mvR9 2XCXDHGMvc0DlfZkpulsYmYxkeqmNWqJJRorBBUh/9xvGgqWJpUWO6Fz X-Gm-Gg: AfdE7clmiPISLtDUCJVuyNak+a7xWIVnBdMP+VFZoj067p2wP4eNNisYefGVf5r2na4 RzqNexANsZWlP9LzpM2Q3STnqkHnPi19+uDBiI6Bl6pOPJQDZtKyXLp1WQ2iO4PFhL8VzqebdTQ bOqAaeL+GrtEj3d29ouNJG5yX7OYbEYFSoZNunvB5nFsyTPVDXxD+0UuFbEbCUHbSkwlB7Cx9ss ygWBVlkx2piinLWaAsbgy6l9R+XC6r6UILvl7+/lj7nn31uL6/FMUiQsz0fyWRI5lBT1d21XslW gYYG02LK77d6unncwvmBSNSC9cEAtTqdbRaRbvCzj/XMVrKii/U+vl7pciPmsmZBH1Xqll+TgYd RXl3wPl0jYHnPewUHRQWGuk4xOwcs86RCTx7ndJ+FVNCtogIYYlE6hQpxkaDzutuCsHtWW7g+TV WUfXCjNzesdwR/gvAAXOWQZeixQ6lt9zOh448= X-Received: by 2002:a17:90b:1dcd:b0:36d:f28a:c5ee with SMTP id 98e67ed59e1d1-37c9ed08579mr7793228a91.25.1781772479946; Thu, 18 Jun 2026 01:47:59 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:59 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 5/6] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Thu, 18 Jun 2026 16:47:25 +0800 Message-Id: <20260618084726.1070022-6-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: "Barry Song (Xiaomi)" In many cases, the pages passed to vmap() may include high-order pages. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually—even pages inside a high-order block are handled one by one. This patch detects physically contiguous pages (regardless of whether they are compound or non-compound) by scanning with num_pages_contiguous(), and maps them as a single contiguous block whenever possible. The mapping order is determined by taking the minimum of the contiguous page count and the pfn alignment, allowing graceful degradation when pfn alignment is less than the contiguous range. Pages with the same page_shift are coalesced and mapped via vmap_pages_range_noflush_walk() to avoid page table rewalk. As users typically allocate memory in descending orders (e.g. 8 → 4 → 0), once an order-0 page is encountered, we stop scanning for contiguous pages since subsequent pages are likely order-0 as well. Signed-off-by: Barry Song (Xiaomi) Co-developed-by: Dev Jain Signed-off-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- mm/vmalloc.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 85 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 253e017130e09..fffb885cb2158 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3545,6 +3545,89 @@ void vunmap(const void *addr) } EXPORT_SYMBOL(vunmap); +static inline unsigned int vm_shift(pgprot_t prot, unsigned long size) +{ + if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE) + return PMD_SHIFT; + + return arch_vmap_pte_supported_shift(size); +} + +static inline int get_vmap_batch_order(struct page **pages, + pgprot_t prot, unsigned int max_steps, unsigned int idx) +{ + unsigned int nr_contig; + int order; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return 0; + + nr_contig = num_pages_contiguous(&pages[idx], max_steps); + if (nr_contig < 2) + return 0; + + order = ilog2(nr_contig); + + /* Limit order by pfn alignment */ + order = min_t(int, order, __ffs(page_to_pfn(pages[idx]))); + + if (vm_shift(prot, PAGE_SIZE << order) == PAGE_SHIFT) + return 0; + + return order; +} + +static int vmap_batched(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + unsigned int count = (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift = 0, idx = 0; + unsigned long start = addr, map_addr = addr; + int err; + + err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + if (err) + goto out; + + for (unsigned int i = 0; i < count; ) { + unsigned int shift = PAGE_SHIFT + + get_vmap_batch_order(pages, prot, count - i, i); + + if (!i) + prev_shift = shift; + + if (shift != prev_shift) { + err = vmap_pages_range_noflush_walk(map_addr, addr, + prot, pages + idx, prev_shift); + if (err) + goto out; + prev_shift = shift; + map_addr = addr; + idx = i; + } + + /* + * Once small pages are encountered, the remaining pages + * are likely small as well. + */ + if (shift == PAGE_SHIFT) + break; + + addr += 1UL << shift; + i += 1U << (shift - PAGE_SHIFT); + } + + /* Remaining */ + if (map_addr < end) + err = vmap_pages_range_noflush_walk(map_addr, end, + prot, pages + idx, prev_shift); + +out: + flush_cache_vmap(start, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3588,8 +3671,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; addr = (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_batched(addr, addr + size, pgprot_nx(prot), + pages) < 0) { vunmap(area->addr); return NULL; } -- 2.34.1