From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C3E8451062 for ; Wed, 15 Jul 2026 12:08:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117342; cv=none; b=DEEwd6qgkLR0E6GtM8/jDK5KWwNZbFdtd3Fz847MYO51u93hI+TjtGU7sf7rC26Xmb1MrvIeQ+MDpDK1P0sqc9NcBq1UT2EQzZ6FHTsUYPASQ0QMZf1b0DN0ofxrI+lm5wvAQqQED/zKKR3xZUPa8KNymumnnb7fbhzgZDTXuRI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117342; c=relaxed/simple; bh=RXjyYv08xorg4GntxMS8ChvKvuX96S5jNDbgjdUo19A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RyX6/WrYUhmaKmCzziToD+mRkxdHu7r3iXDjxQoPwmqZtj/fMZPzygcvluwHB3VomIQsMBUZqwdha8UH/kmDJUgjaq9HXweBglQZNtG1Q7BI+W6Ml2b8dNoYPcLgi/FoRtF1eHUvqibGznUm86GmvF5dmxVTYVmxzsh5sMnrA48= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=sZmMVW04; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="sZmMVW04" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2cee9b74ee1so19310365ad.3 for ; Wed, 15 Jul 2026 05:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117336; x=1784722136; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=93J8dJ31kBsJOKH3R1rbCLn8YwyTsJsj4AeCzglR+bg=; b=sZmMVW04ZwKc9A0WKkeQcgPaDN03j/I4fBDzP60vn4nh35DjNoi99Zclo3/hbBJ4bV ZnaxHPf6A9is7gcEfrBEr6HSEt0qwFgsBkDcFzwpxLjIMIkeN2/zIWmWGXAX+QG2uX7P PVIu4Yn7S6KhbLw2jBZZM8XRN/oglXmOf43+7ItAiyP8T54bNtZ+qJ35sEDQ/FwANoni JSq35XJXTCAkpQqLzxCq3P6BP8aZH1hpLNra7/FK3qwBv2KNQWLIHPS/WXmfb9GKGBhk 7pG6CI2OeBx8WdchZUEemudDyQqzd//ZLBb8l9h96xot/21M4zMDnK3PhCc2fyuvl1ko 4Fwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117336; x=1784722136; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=93J8dJ31kBsJOKH3R1rbCLn8YwyTsJsj4AeCzglR+bg=; b=khzoeID4J0UNThhKY5z0hoIPQ3wsPS2gytoLtF3drhWpNEEB6a82GgYpqi60RnUSn/ M19EI8JHdvXEgyugtWOAciVzU3QrrQa8pgPtFmoNsdUEyP/psQcoTIRAJ5VTmXb2p6wu q9nkLk51lf0V5jvQtiheaNcajhA/n11ipcFzcrVTbI8qDsoZTTWY9GeJTXljcPhWXl8+ JW24ePVgChMKYHmJbPF8X2Gnwkoy/Jw2uqVHz62M29+i/4vbMPtyFSPT2eeOT7Ubei8s K/bI4n8yuRNd9aPUIg/PZJMUc7f3rHmjFRfSGjJzLjcKO+wTR2cgGvnxD0TfiVEOTxt3 tLxQ== X-Forwarded-Encrypted: i=1; AHgh+RppTyuLdhI06Eds2tvEA6RL+J8lS1eM3J+DQssyJNKaHOFNBMhMu7hPZ0V3bnL+lVivlYgOHeqSqU/n4Lc=@vger.kernel.org X-Gm-Message-State: AOJu0Yx9XhlsY/QnBLzFX5q95bOpL3HGCvCaTtphWqdo7ZBoHEqx5Dv/ PucrNCQyTPQkwA3BfETWUQKFxAdLq05WfeVw1MLgtoSqd/7IB67hlHOM X-Gm-Gg: AfdE7cnQeSnECaDs8gM4gZrNuRZ3mnj9qOZ46ZfHg6ZmyeppHNvEocvQaxjiyLYwX4X oBB0Cs2JJ6CR/5HsNWlbB5PS8IajbbGWcZPTzCpCaemt4XjMRP2Zck/skJYP9OsznNsAwAZwvCV v2jggmfAEIADxXX574Qd0BNooJL/Op7tZ1qmI/7zdJ+jGLxNNu64LMmGa+HTfV1huFDJfdPKhRA bQ8GZBki+y6NWO2MlCYjzdDhQqnvy2LfmxQxqDCKeiflOW6mdmr615JCitjneovVVacCK4jezOh pzL9T6W85vAJ+VvxshcTalJhq41OtARlsry4BQWAQWRy105HJ6fG6WWLY66Trt1d4PGW9d7p1Rv 5WjFMp9VCtj/58VvRv7htOGHb4TLfxPgckDCsEIOhJ+/7bd4a+Jlak/Yy9/V0WcoiehIembHDg7 0DNsSOSYvIyy1V/hDNOFV58DdNgRW/XG9Iy9E= X-Received: by 2002:a17:902:f686:b0:2ca:3df2:919a with SMTP id d9443c01a7336-2cf03dca19fmr22554225ad.33.1784117336405; Wed, 15 Jul 2026 05:08:56 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:56 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 6/7] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Wed, 15 Jul 2026 20:08:12 +0800 Message-Id: <20260715120813.3609949-7-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: "Barry Song (Xiaomi)" In many cases, the pages passed to vmap() may include high-order pages. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually—even pages inside a high-order block are handled one by one. This patch detects physically contiguous pages (regardless of whether they are compound or non-compound) by scanning with num_pages_contiguous(), and maps them as a single contiguous block whenever possible. The mapping order is determined by taking the minimum of the contiguous page count and the pfn alignment, allowing graceful degradation when pfn alignment is less than the contiguous range. Pages with the same page_shift are coalesced and mapped via vmap_pages_range_noflush_walk() to avoid page table rewalk. As users typically allocate memory in descending orders (e.g. 8 → 4 → 0), once an order-0 page is encountered, we stop scanning for contiguous pages since subsequent pages are likely order-0 as well. Signed-off-by: Barry Song (Xiaomi) Co-developed-by: Dev Jain Signed-off-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 80 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a1e025120e9df..92f9cd5e9def5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3557,6 +3557,84 @@ static inline unsigned int vm_shift(pgprot_t prot, unsigned long size) return arch_vmap_pte_supported_shift(size); } +static inline int get_vmap_batch_order(struct page **pages, + pgprot_t prot, unsigned int max_steps, unsigned int idx) +{ + unsigned long pfn; + unsigned int nr_contig; + int order; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return 0; + + nr_contig = num_pages_contiguous(&pages[idx], max_steps); + if (nr_contig < 2) + return 0; + + order = ilog2(nr_contig); + pfn = page_to_pfn(pages[idx]); + + /* Limit order by pfn alignment */ + if (pfn > 0) + order = min_t(int, order, __ffs(pfn)); + + if (vm_shift(prot, PAGE_SIZE << order) == PAGE_SHIFT) + return 0; + + return order; +} + +static int vmap_pages_range_batched(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + unsigned int count = (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift = 0, idx = 0; + unsigned long map_addr = addr, batch_end = addr; + int err; + + err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + if (err) + goto out; + + for (unsigned int i = 0; i < count; ) { + unsigned int shift = PAGE_SHIFT + + get_vmap_batch_order(pages, prot, count - i, i); + + if (!i) + prev_shift = shift; + + if (shift != prev_shift) { + err = vmap_pages_range_noflush_walk(map_addr, batch_end, + prot, pages + idx, prev_shift); + if (err) + goto out; + prev_shift = shift; + map_addr = batch_end; + idx = i; + } + + /* + * Once small pages are encountered, the remaining pages + * are likely small as well. + */ + if (shift == PAGE_SHIFT) + break; + + batch_end += 1UL << shift; + i += 1U << (shift - PAGE_SHIFT); + } + + /* Remaining */ + if (map_addr < end) + err = vmap_pages_range_noflush_walk(map_addr, end, + prot, pages + idx, prev_shift); + +out: + flush_cache_vmap(addr, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3600,8 +3678,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; addr = (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_pages_range_batched(addr, addr + size, pgprot_nx(prot), + pages) < 0) { vunmap(area->addr); return NULL; } -- 2.34.1