From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1760D3112C0 for ; Thu, 18 Jun 2026 08:47:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772456; cv=none; b=bul4LgCoGTXpCJjk0DPagjqFxzNJPyU9glNJPDhrq5ZKd6Va8ptPnxSRybvX3wOvaLqr7PVtw5M5b4hW1xsYMjSIppAd8ix10ZlclEAjWYdNTHq2+dXohGEg+e+bl/D5T7OJFcMAQdHJmoruwc7zNeml2vDIYIOIoWSanOkF6qs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772456; c=relaxed/simple; bh=hNMoa77+kRqxGl5D2xkbi43kcThA80dQA+2fOnkFhJ8=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Content-Type; b=nLMbJj2NB+H3swJzoWBSNJqUX2lcBEl+A7IRDLJzMAj/4jAwYZtuMbvavuhniHAOHzFYchzza3XpX+NjXPuvWRUDH0hVw0qAhODPr2PscX4TP40J0CahUdlbp5rnHns7Du8qYfcVqs65ZtSH3CXKYQ6OhQCBjeNN8O5Ub0WEUQU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=D4BU87ug; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="D4BU87ug" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-36baeec21dcso344846a91.3 for ; Thu, 18 Jun 2026 01:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772454; x=1782377254; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ytp+bdNRdXUdbPQ9Ezf0ymEFpvBnhLVGkSee+aNNB40=; b=D4BU87ugEzrShu1axvyuCO+zSJB0jTLxnXD7V8wL1pUCmWteVm36j/v7C0fbMhYmei 4pdXQsJkHILn2bXiH4ROwb/asFN5IkGJxWbkMryTZblodwQS9/emXgfOeQlNEvFt9eaB WFBD4hVuJFICd1reNXhU43MnVseqaAseFfd+FoOzZRD3iYpJVFnJRBGXHLKWeXoBhisG nqZ+KgmIN4L+Pz9liqcvk//1lN/NVgrpEX6Oa5pO1mQiLIBBqZi2r8jN8LTEJmDerT2e CX4cE/lU8ia1MIwdoBYSAsiRecBYmk8cyRxUqRDD9PFfnciQ3XR7GefwJpjw0RYIJHVr ro7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772454; x=1782377254; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ytp+bdNRdXUdbPQ9Ezf0ymEFpvBnhLVGkSee+aNNB40=; b=FUwSt6bPZ2+XeXSLUbYLwPerQBaAr9gcD2GJNOHfBLWS4nUzGCQGRM23x9N13AG2lV eQDHOlZjLXn0DYZx4vuyOUQBxojSBFhS4mZp8GJ+4Fwl0mWQ+ng+A4gVcZCO/XgTg0SG cdj+lfyEp+Wn464dSDPDQ6Uk7ALoWWacCYzbmNgUtK17TID+IKTU50dVsIinJfvmFFUg kJihoiKLNskLy+ZFc6oqCfpqyDJcA17D6RN4W12u1eFvf+iMykfTVNquSdshP0gJCze3 cx2NWiUn4nnNvbNi4L5m9K5W6rtChTKRCrkEftwfIQDspLpHMPQh5ZVf+L/qzynCssL5 WLhA== X-Forwarded-Encrypted: i=1; AFNElJ/C7JL1wI6Z7ZfDyuxGlRx9Vn3ltxoFpkuy+nhRN2zaHmVplc2Ty4sMSCrSz5pDsVeD5hDZAjCdhiefn6c=@vger.kernel.org X-Gm-Message-State: AOJu0YwoqcXXsGbXw3bjGKYoYpzIjHeKha/CdrrjMnPDQWVWADAUfu45 tzlfBptShaGIXi97qL3NtqY5582iCQpwfz2pdxV+zqv+aDMacFrRmla4 X-Gm-Gg: AfdE7cnr0WwdpBr/jEJEkOJ8kyvsYC9xqoozxLhPf/JRiNCcTskXD5tV/kTyZ0RCuFG DHwKebosoMyEjGxVDgUeDdcNiyh3w+Z/P8I0+Pr90FeJWgdh4vSZsdsSb7wwzXr4ssVxgpvpVxx jbYiOKSE1kDcAQyXigExAbDRSx4stlQNwEj5csWjSgoK8JrOi5+1ARK2C7n8cn/da964nYkEQkx 8nvmrI05Cc46UIz7tc98mX7NSEHA7hYDSGzmG3TKsXpTax6i2a2g7xN7ac1juUIyOnbotsyX+Ys auSumQRDQk82uEfMofSPBALdVQk1MyihmIE9XtGflK3nXKdC0pDQ4qqC4Cp26AC1X7BnhCFWAwu lptRnQJ7BPCj4hOQtO7RdnF9B7ICrgC0J34xub0VI45dqKlLIe2WzohnspmG2fkdde8+41YSnVJ MNGj/n60OqlYwSPH+tLKjnpLVxQnQP4nkRImk= X-Received: by 2002:a17:90a:c2cc:b0:35b:9894:f6f9 with SMTP id 98e67ed59e1d1-37c9e9d41a7mr6843844a91.18.1781772454417; Thu, 18 Jun 2026 01:47:34 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:33 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 0/6] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Date: Thu, 18 Jun 2026 16:47:20 +0800 Message-Id: <20260618084726.1070022-1-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This patchset accelerates ioremap, vmalloc, and vmap when the memory is physically fully or partially contiguous. Two techniques are used: 1. Avoid page table rewalk when setting PTEs/PMDs for multiple memory segments 2. Use batched mappings wherever possible in both vmalloc and ARM64 layers Besides accelerating the mapping path, this also enables large mappings (PMD and cont-PTE) for vmap, which are currently not supported. Patches 1-2 extend ARM64 vmalloc CONT-PTE mapping to support multiple CONT-PTE regions instead of just one. Patch 3 extracts a common helper vmap_set_ptes() that consolidates PTE mapping logic between the ioremap and vmalloc/vmap paths, handling both CONT_PTE and regular PTE mappings. This prepares for the next patch. Patch 4 extends the page table walk path to support page shifts other than PAGE_SHIFT and eliminates the page table rewalk for huge vmalloc mappings. The function is renamed from vmap_small_pages_range_noflush() to vmap_pages_range_noflush_walk(). Patches 5-6 add huge vmap support for contiguous pages, including support for non-compound pages with pfn alignment verification. On the RK3588 8-core ARM64 SoC, with tasks pinned to a little core and the performance CPUfreq policy enabled, benchmark results: * ioremap(1 MB): 1.35x faster (3407 ns -> 2526 ns) * vmalloc(1 MB) mapping time (excluding allocation) with VM_ALLOW_HUGE_VMAP: 1.42x faster (5.00 us -> 3.53us) * vmap(100MB) with order-8 pages: 8.3x faster (1235 us -> 149 us) Many thanks to Xueyuan Chen for his testing efforts on RK3588 boards. Changes since v3: - Squash vmap_pte_range() loop variable fix into patch 4 (patch 3, 4) - Use shift >= PMD_SHIFT and fix *nr increment in vmap_pages_pmd_range() (patch 4) - Pass page_shift directly without capping at PMD_SHIFT (patch 4, 5) - Add vm_shift() helper and pass pgprot_t to get_vmap_batch_order() (patch 5) - Use min(order, __ffs(pfn)) for graceful pfn alignment degradation, replacing IS_ALIGNED check (patch 5) - Remove irrelevant ioremap_max_page_shift early-exit (patch 5) - Add __get_vm_area_node_aligned_caller() wrapper, rename to vmap_get_aligned_vm_area() (patch 6) Changes since v2: - Use __fls instead of fls in arch_vmap_pte_range_map_size (patch 2) - Add WARN_ON checks in vmap_pages_pmd_range (patch 4) - Fix flush_cache_vmap to use saved start address instead of the already-advanced addr (patch 5) - Rename __vmap_huge() to vmap_batched() (patch 5) - Add caller parameter and unroll while(1) loop (patch 5) - Squash patch 7 into patch 5 (stop scanning for compound pages after encountering small pages) Changes since v1: - Fix condition order and use PMD_SIZE instead of CONT_PMD_SIZE in patch 1 (Dev Jain) - Squash patch 3+4 and patch 5+7 (Dev Jain) - Replace "zigzag" with "page table rewalk" in commit messages (Dev Jain) - Rename vmap_small_pages_range_noflush() to vmap_pages_range_noflush_walk() (Dev Jain) - Extract vmap_set_ptes() as a new patch to consolidate PTE mapping logic between vmap_pte_range() and vmap_pages_pte_range(), handling both CONT_PTE and regular mappings (Mike Rapoport) - Support non-compound pages in get_vmap_batch_order() by falling back to physical contiguity scanning with pfn alignment check (Dev Jain, Uladzislau Rezki) - In get_vmap_batch_order(), filter out orders that the architecture cannot batch by checking arch_vmap_pte_supported_shift() directly. This avoids overhead for orders 1-3 on ARM64 CONT_PTE with 4K pages. (patch 5) Barry Song (Xiaomi) (5): arm64/hugetlb: Extend batching of multiple CONT_PTE in a single PTE setup arm64/vmalloc: Allow arch_vmap_pte_range_map_size to batch multiple CONT_PTE mm/vmalloc: Extend page table walk to support larger page_shift sizes and eliminate page table rewalk mm/vmalloc: map contiguous pages in batches for vmap() if possible mm/vmalloc: align vm_area so vmap() can batch mappings Wen Jiang (1): mm/vmalloc: Extract vmap_set_ptes() to consolidate PTE mapping logic arch/arm64/include/asm/vmalloc.h | 6 +- arch/arm64/mm/hugetlbpage.c | 10 ++ mm/vmalloc.c | 247 +++++++++++++++++++++++++------ 3 files changed, 213 insertions(+), 50 deletions(-) -- 2.34.1