From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 179E2395252 for ; Thu, 26 Feb 2026 08:45:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772095528; cv=none; b=ZoLVlodx/P7GOeraaSLB6/VFCA9A3UqEyHDrLJyjICDNzQyEo9+P7BZCu0Lk2HFPEeVOgwFcA3aSTGmTkpo286cKAKQuHrbCabiSza3tYb0eURWUnuPAzv7bOwgSL6MU6WAHVT0PLNJpIJ0x2b1pBiY0qhgp92qoaWuIP5r/x9U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772095528; c=relaxed/simple; bh=VZcxxFVqJcMGG2ouLxoT83wvf9CTkC6U1PLHlgUfKJ4=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=C1uXR7xNJNCo2D16iIxYw0BeJSEfaa9ODlufRcE6wMES1dAqbfUIKtGIwDTBdSBSWE25zAiUWbjJHNdHC06V0TDgfwepHRcKo82bVqezF59L3HoVs//vrKhI13OoJrjjBG+tCUHvzVoneotN3xYXZMDkNdHXrG50hat8oMkkv40= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7A6C51516; Thu, 26 Feb 2026 00:45:15 -0800 (PST) Received: from [10.164.19.28] (unknown [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 00D4F3F7BD; Thu, 26 Feb 2026 00:45:15 -0800 (PST) Message-ID: Date: Thu, 26 Feb 2026 14:15:13 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [LSF/MM/BPF TOPIC] Per-process page size To: Kalesh Singh Cc: lsf-pc@lists.linux-foundation.org, ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, ardb@kernel.org, willy@infradead.org, hughd@google.com, baolin.wang@linux.alibaba.com, akpm@linux-foundation.org, david@kernel.org, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, =?UTF-8?Q?Mateusz_Ma=C4=87kowski?= , =?UTF-8?Q?Adrian_Barna=C5=9B?= , Marcin Szymczyk References: <20260217145026.3880286-1-dev.jain@arm.com> From: Dev Jain Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 26/02/26 1:10 pm, Kalesh Singh wrote: > On Tue, Feb 17, 2026 at 6:50 AM Dev Jain wrote: >> >> Hi everyone, >> >> We propose per-process page size on arm64. Although the proposal is for >> arm64, perhaps the concept can be extended to other arches, thus the >> generic topic name. >> >> ------------- >> INTRODUCTION >> ------------- >> While mTHP has brought the performance of many workloads running on an arm64 4K >> kernel closer to that of the performance on an arm64 64K kernel, a performance >> gap still remains. This is attributed to a combination of greater number of >> pgtable levels, less reach within the walk cache and higher data cache footprint >> for pgtable memory. At the same time, 64K is not suitable for general >> purpose environments due to it's significantly higher memory footprint. >> >> To solve this, we have been experimenting with a concept called "per-process >> page size". This breaks the historic assumption of a single page size for the >> entire system: a process will now operate on a page size ABI that is greater >> than or equal to the kernel's page size. This is enabled by a key architectural >> feature on Arm: the separation of user and kernel page tables. >> >> This can also lead to a future of a single kernel image instead of 4K, 16K >> and 64K images. >> >> -------------- >> CURRENT DESIGN >> -------------- >> The design is based on one core idea; most of the kernel continues to believe >> there is only one page size in use across the whole system. That page size is >> the size selected at compile-time, as is done today. But every process (more >> accurately mm_struct) has a page size ABI which is one of the 3 page sizes >> (4K, 16K or 64K) as long as that page size is greater than or equal to the >> kernel page size (kernel page size is the macro PAGE_SIZE). >> >> Pagesize selection >> ------------------ >> A process' selected page size ABI comes into force at execve() time and >> remains fixed until the process exits or until the next execve(). Any forked >> processes inherit the page size of their parent. >> The personality() mechanism already exists for similar cases, so we propose >> to extend it to enable specifying the required page size. >> >> There are 3 layers to the design. The first two are not arch-dependent, >> and makes Linux support a per-process pagesize ABI. The last layer is >> arch-specific. >> >> 1. ABI adapter >> -------------- >> A translation layer is added at the syscall boundary to convert between the >> process page size and the kernel page size. This effectively means enforcing >> alignment requirements for addresses passed to syscalls and ensuring that >> quantities passed as “number of pages” are interpreted relative to the process >> page size and not the kernel page size. In this way the process has the illusion >> that it is working in units of its page size, but the kernel is working in >> units of the kernel page size. >> >> 2. Generic Linux MM enlightenment >> --------------------------------- >> We enlighten the Linux MM code to always hand out memory in the granularity >> of process pages. Most of this work is greatly simplified because of the >> existing mTHP allocation paths, and the ongoing support for large folios >> across different areas of the kernel. The process order will be used as the >> hard minimum mTHP order to allocate. >> >> File memory >> ----------- >> For a growing list of compliant file systems, large folios can already be >> stored in the page cache. There is even a mechanism, introduced to support >> filesystems with block sizes larger than the system page size, to set a >> hard-minimum size for folios on a per-address-space basis. This mechanism >> will be reused and extended to service the per-process page size requirements. >> >> One key reason that the 64K kernel currently consumes considerably more memory >> than the 4K kernel is that Linux systems often have lots of small >> configuration files which each require a page in the page cache. But these >> small files are (likely) only used by certain processes. So, we prefer to >> continue to cache those using a 4K page. >> Therefore, if a process with a larger page size maps a file whose pagecache >> contains smaller folios, we drop them and re-read the range with a folio >> order at least that of the process order. >> >> 3. Translation from Linux pagetable to native pagetable >> ------------------------------------------------------- >> Assume the case of a kernel pagesize of 4K and app pagesize of 64K. >> Now that enlightenment is done, it is guaranteed that every single mapping >> in the 4K pagetable (which we call the Linux pagetable) is of granularity >> at least 64K. In the arm64 MM code, we maintain a "native" pagetable per >> mm_struct, which is based off a 64K geometry. Because of the guarantee >> aforementioned, any pagetable operation on the Linux pagetable >> (set_ptes, clear_flush_ptes, modify_prot_start_ptes, etc) is going to happen >> at a granularity of at least 16 PTEs - therefore we can translate this >> operation to modify a single PTE entry in the native pagetable. >> Given that enlightenment may miss corner cases, we insert a warning in the >> architecture code - on being presented with an operation not translatable >> into a native operation, we fallback to the Linux pagetable, thus losing >> the benefits borne out of the pagetable geometry but keeping >> the emulation intact. >> >> ----------------------- >> What we want to discuss >> ----------------------- >> - Are there other arches which could benefit from this? >> - What level of compatibility we can achieve - is it even possible to >> contain userspace within the emulated ABI? >> - Rough edges of compatibility layer - pfnmaps, ksm, procfs, etc. For >> example, what happens when a 64K process opens a procfs file of >> a 4K process? >> - native pgtable implementation - perhaps inspiration can be taken >> from other arches with an involved pgtable logic (ppc, s390)? >> > > Hi Dev, Ryan, > > I'd be very interested in joining this discussion at LSF/MM. Thanks Kalesh for your interest! > > On Android, we have a separate but very related use case: we emulate a > larger userspace page size on x86, primarily to allow app developers > to test their apps for 16KB compatibility using x86 emulators [1]. > > Similar to your proposed "ABI adapter" layer, our approach works by > enforcing a larger 16KB granularity and alignment on the VMAs to > emulate the userspace page size, while the underlying kernel still > operates on a 4KB granularity [2]. > > In our emulation experience, we've run into a few specific rough edges: > > 1. mmap and SIGBUS: Enforcing a larger VMA granularity means that > mapping files can easily extend the VMA beyond the end of the file's > valid offset. When userspace touches this padded area, the 4KB filemap > fault cannot resolve to a valid index, resulting in a SIGBUS that > applications aren't expecting. You did mention in the other email the links below, and I went ahead to compare :) I was puzzled to see some sort of VMA padding approach in your patches. OTOH our approach pads anonymous pages. So for example, if a 64K process maps a 12K sized file, we will map 52K/4K = 13 anonymous pages into the 64K-aligned VMA. Implementation-wise, we detect such a condition in filemap_fault and return VM_FAULT_NEED_ANONPAGE, and redirect that to do_anonymous_page to map 4K pages. > > 2. userfaultfd: This inherently operates at the strict PTE granularity > of the underlying kernel (4KB). Hiding this from a userspace that > expects a 16KB/64KB fault granularity while the kernel still operates > on 4KB granularity is messy ... Indeed. We will have to fault in 16 4K pages. > > 3. pagemap and PFN interfaces: As you noted with procfs, interfaces > that expose or consume PFNs are problematic. Userspace tools reading > /proc/pid/pagemap, /proc/kpagecount, /proc/kpageflags, > /proc/kpagecgroup, and /sys/kernel/mm/page_idle/bitmap calculate > offsets based on the userspace page size ABI, but the kernel returns > 4KB PFNs which breaks such users. > > > It would be great to explore if we can align on a unified approach to > solve these. > > [1] https://developer.android.com/guide/practices/page-sizes#16kb-emulator > [2] https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic > > Thanks, > Kalesh > >> ------------- >> Key Attendees >> ------------- >> - Ryan Roberts (co-presenter) >> - mm folks (David Hildenbrand, Matthew Wilcox, Liam Howlett, Lorenzo Stoakes, >> and many others) >> - arch folks >>