[LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

No, there's no new hardware (that I know of). I want to explore what page size
means.

The kernel uses the same value - PAGE_SIZE - for two things:

  - the order-0 buddy allocation size;

  - the granularity of virtual address space mapping;

I think we can benefit from separating these two meanings and allowing
order-0 allocations to be larger than the virtual address space covered by a
PTE entry.

The main motivation is scalability. Managing memory on multi-terabyte
machines in 4k is suboptimal, to say the least.

Potential benefits of the approach (assuming 64k pages):

  - The order-0 page size cuts struct page overhead by a factor of 16. From
    ~1.6% of RAM to ~0.1%;

  - TLB wins on machines with TLB coalescing as long as mapping is naturally
    aligned;

  - Order-5 allocation is 2M, resulting in less pressure on the zone lock;

  - 1G pages are within possibility for the buddy allocator - order-14
    allocation. It can open the road to 1G THPs.

  - As with THP, fewer pages - less pressure on the LRU lock;

  - ...

The trade-off is memory waste (similar to what we have on architectures with
native 64k pages today) and complexity, mostly in the core-MM code.

== Design considerations ==

I want to split PAGE_SIZE into two distinct values:

  - PTE_SIZE defines the virtual address space granularity;

  - PG_SIZE defines the size of the order-0 buddy allocation;

PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
requires conversion, and keep existing code working while conversion is in
progress.

The same split happens for other page-related macros: mask, shift,
alignment helpers, etc.

PFNs are in PTE_SIZE units.

The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
units.

Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
for userspace. But we might want to communicate PG_SIZE to userspace to
get the optimal results for userspace that cares.

PTE_SIZE granularity requires a substantial rework of page fault and VMA
handling:

  - A struct page pointer and pgprot_t are not enough to create a PTE entry.
    We also need the offset within the page we are creating the PTE for.

  - Since the VMA start can be aligned arbitrarily with respect to the
    underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
    which is in PTE_SIZE units.

  - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
    misaligned cases;

Page faults into file mappings are relatively simple to handle as we
always have the page cache to refer to. So you can map only the part of the
page that fits in the page table, similarly to fault-around.

Anonymous and file-CoW faults should also be simple as long as the VMA is
aligned to PG_SIZE in both the virtual address space and with respect to
vm_pgoff. We might waste some memory on the ends of the VMA, but it is
tolerable.

Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
pages across a page table boundary. In the worst case, a page is mapped across
a PGD entry boundary and PTEs for the page have to be put in two separate
subtrees of page tables.

A naive implementation would map different pages on different sides of a
page table boundary and accept the waste of one page per page table crossing.
The hope is that misaligned mappings are rare, but this is suboptimal.

mremap(2) is the ultimate stress test for the design.

On x86, page tables are allocated from the buddy allocator and if PG_SIZE
is greater than 4 KB, we need a way to pack multiple page tables into a
single page. We could use the slab allocator for this, but it would
require relocating the page-table metadata out of struct page.

Things I have not thought much about yet:

  - Accounting for wasted memory;

  - rmap;

  - mapcount;

  - A lot of arch-specific code;

  - <insert my blind spot here>;

== Status ==

I have a POC implementation on top of v6.17:

git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git pte_size

It is WIP and full of hacks I am trying to make sense of now.

It compiles with my minimalistic kernel config and can boot to a shell with
both 16k and 64k base page sizes. The shell doesn't crash immediately, but
sometimes I wonder why :P

The patchset is large:

 378 files changed, 3348 insertions(+), 3102 deletions(-)

and it is far from being complete.

== Goals ==

I want to get feedback for the overall design and possible ways to
upstream.

My plan is to submit an RFC-quality patchset before the summit.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Peter Zijlstra]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> 
> The kernel uses the same value - PAGE_SIZE - for two things:
> 
>   - the order-0 buddy allocation size;
> 
>   - the granularity of virtual address space mapping;
> 
> I think we can benefit from separating these two meanings and allowing
> order-0 allocations to be larger than the virtual address space covered by a
> PTE entry.

Didn't AA do this a decade ago or somesuch?

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Peter Zijlstra]

On Thu, Feb 19, 2026 at 04:17:29PM +0100, Peter Zijlstra wrote:
> On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >   - the order-0 buddy allocation size;
> > 
> >   - the granularity of virtual address space mapping;
> > 
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> 
> Didn't AA do this a decade ago or somesuch?

  https://lwn.net/Articles/240914/

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 04:20:45PM +0100, Peter Zijlstra wrote:
> On Thu, Feb 19, 2026 at 04:17:29PM +0100, Peter Zijlstra wrote:
> > On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > > No, there's no new hardware (that I know of). I want to explore what page size
> > > means.
> > > 
> > > The kernel uses the same value - PAGE_SIZE - for two things:
> > > 
> > >   - the order-0 buddy allocation size;
> > > 
> > >   - the granularity of virtual address space mapping;
> > > 
> > > I think we can benefit from separating these two meanings and allowing
> > > order-0 allocations to be larger than the virtual address space covered by a
> > > PTE entry.
> > 
> > Didn't AA do this a decade ago or somesuch?
> 
>   https://lwn.net/Articles/240914/

Oh, 2007. It predates me in kernel. Will read up. Thanks!

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Pedro Falcato]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> 
> The kernel uses the same value - PAGE_SIZE - for two things:
> 
>   - the order-0 buddy allocation size;
> 
>   - the granularity of virtual address space mapping;
> 
> I think we can benefit from separating these two meanings and allowing
> order-0 allocations to be larger than the virtual address space covered by a
> PTE entry.
>

Doesn't this idea make less sense these days, with mTHP? Simply by toggling one
of the entries in /sys/kernel/mm/transparent_hugepage.
 
> The main motivation is scalability. Managing memory on multi-terabyte
> machines in 4k is suboptimal, to say the least.
> 
> Potential benefits of the approach (assuming 64k pages):
> 
>   - The order-0 page size cuts struct page overhead by a factor of 16. From
>     ~1.6% of RAM to ~0.1%;
> 
>   - TLB wins on machines with TLB coalescing as long as mapping is naturally
>     aligned;
> 
>   - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> 
>   - 1G pages are within possibility for the buddy allocator - order-14
>     allocation. It can open the road to 1G THPs.
> 
>   - As with THP, fewer pages - less pressure on the LRU lock;

We could perhaps add a way to enforce a min_order globally on the page cache,
as a way to address it.

There are some points there which aren't addressed by mTHP work in any way
(1G THPs for one), others which are being addressed separately (memdesc work
trying to cut down on struct page overhead).

(I also don't understand your point about order-5 allocation, AFAIK pcp will
cache up to COSTLY_ORDER (3) and PMD order, but I'm probably not seeing the
full picture)


-- 
Pedro

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 03:33:47PM +0000, Pedro Falcato wrote:
> On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >   - the order-0 buddy allocation size;
> > 
> >   - the granularity of virtual address space mapping;
> > 
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> >
> 
> Doesn't this idea make less sense these days, with mTHP? Simply by toggling one
> of the entries in /sys/kernel/mm/transparent_hugepage.

mTHP is still best effort. This is way you don't need to care about
fragmentation, you will get your 64k page as long as you have free
memory.

> > The main motivation is scalability. Managing memory on multi-terabyte
> > machines in 4k is suboptimal, to say the least.
> > 
> > Potential benefits of the approach (assuming 64k pages):
> > 
> >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> >     ~1.6% of RAM to ~0.1%;
> > 
> >   - TLB wins on machines with TLB coalescing as long as mapping is naturally
> >     aligned;
> > 
> >   - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > 
> >   - 1G pages are within possibility for the buddy allocator - order-14
> >     allocation. It can open the road to 1G THPs.
> > 
> >   - As with THP, fewer pages - less pressure on the LRU lock;
> 
> We could perhaps add a way to enforce a min_order globally on the page cache,
> as a way to address it.

Raising min_order is not free. I puts more pressure on page allocator.

> There are some points there which aren't addressed by mTHP work in any way
> (1G THPs for one), others which are being addressed separately (memdesc work
> trying to cut down on struct page overhead).
> 
> (I also don't understand your point about order-5 allocation, AFAIK pcp will
> cache up to COSTLY_ORDER (3) and PMD order, but I'm probably not seeing the
> full picture)

With higher base page size, page allocator doesn't need to do as much
work to merge/split buddy pages. So serving the same 2M as order-5 is
cheaper than order-9.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/19/26 16:50, Kiryl Shutsemau wrote:
> On Thu, Feb 19, 2026 at 03:33:47PM +0000, Pedro Falcato wrote:
>> On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
>>> No, there's no new hardware (that I know of). I want to explore what page size
>>> means.
>>>
>>> The kernel uses the same value - PAGE_SIZE - for two things:
>>>
>>>    - the order-0 buddy allocation size;
>>>
>>>    - the granularity of virtual address space mapping;
>>>
>>> I think we can benefit from separating these two meanings and allowing
>>> order-0 allocations to be larger than the virtual address space covered by a
>>> PTE entry.
>>>
>>
>> Doesn't this idea make less sense these days, with mTHP? Simply by toggling one
>> of the entries in /sys/kernel/mm/transparent_hugepage.
> 
> mTHP is still best effort. This is way you don't need to care about
> fragmentation, you will get your 64k page as long as you have free
> memory.
> 
>>> The main motivation is scalability. Managing memory on multi-terabyte
>>> machines in 4k is suboptimal, to say the least.
>>>
>>> Potential benefits of the approach (assuming 64k pages):
>>>
>>>    - The order-0 page size cuts struct page overhead by a factor of 16. From
>>>      ~1.6% of RAM to ~0.1%;
>>>
>>>    - TLB wins on machines with TLB coalescing as long as mapping is naturally
>>>      aligned;
>>>
>>>    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
>>>
>>>    - 1G pages are within possibility for the buddy allocator - order-14
>>>      allocation. It can open the road to 1G THPs.
>>>
>>>    - As with THP, fewer pages - less pressure on the LRU lock;
>>
>> We could perhaps add a way to enforce a min_order globally on the page cache,
>> as a way to address it.
> 
> Raising min_order is not free. I puts more pressure on page allocator.
> 
>> There are some points there which aren't addressed by mTHP work in any way
>> (1G THPs for one), others which are being addressed separately (memdesc work
>> trying to cut down on struct page overhead).
>>
>> (I also don't understand your point about order-5 allocation, AFAIK pcp will
>> cache up to COSTLY_ORDER (3) and PMD order, but I'm probably not seeing the
>> full picture)
> 
> With higher base page size, page allocator doesn't need to do as much
> work to merge/split buddy pages. So serving the same 2M as order-5 is
> cheaper than order-9.

I think the idea is that if most of your allocations (anon + pagecache) 
are 64k instead of 4k, on average, you'll just naturally do less merging 
splitting.

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Pedro Falcato]

On Thu, Feb 19, 2026 at 04:53:10PM +0100, David Hildenbrand (Arm) wrote:
> On 2/19/26 16:50, Kiryl Shutsemau wrote:
> > On Thu, Feb 19, 2026 at 03:33:47PM +0000, Pedro Falcato wrote:
> > > On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > > > No, there's no new hardware (that I know of). I want to explore what page size
> > > > means.
> > > > 
> > > > The kernel uses the same value - PAGE_SIZE - for two things:
> > > > 
> > > >    - the order-0 buddy allocation size;
> > > > 
> > > >    - the granularity of virtual address space mapping;
> > > > 
> > > > I think we can benefit from separating these two meanings and allowing
> > > > order-0 allocations to be larger than the virtual address space covered by a
> > > > PTE entry.
> > > > 
> > > 
> > > Doesn't this idea make less sense these days, with mTHP? Simply by toggling one
> > > of the entries in /sys/kernel/mm/transparent_hugepage.
> > 
> > mTHP is still best effort. This is way you don't need to care about
> > fragmentation, you will get your 64k page as long as you have free
> > memory.
> > 
> > > > The main motivation is scalability. Managing memory on multi-terabyte
> > > > machines in 4k is suboptimal, to say the least.
> > > > 
> > > > Potential benefits of the approach (assuming 64k pages):
> > > > 
> > > >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> > > >      ~1.6% of RAM to ~0.1%;
> > > > 
> > > >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> > > >      aligned;
> > > > 
> > > >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > > > 
> > > >    - 1G pages are within possibility for the buddy allocator - order-14
> > > >      allocation. It can open the road to 1G THPs.
> > > > 
> > > >    - As with THP, fewer pages - less pressure on the LRU lock;
> > > 
> > > We could perhaps add a way to enforce a min_order globally on the page cache,
> > > as a way to address it.
> > 
> > Raising min_order is not free. I puts more pressure on page allocator.
> > 
> > > There are some points there which aren't addressed by mTHP work in any way
> > > (1G THPs for one), others which are being addressed separately (memdesc work
> > > trying to cut down on struct page overhead).
> > > 
> > > (I also don't understand your point about order-5 allocation, AFAIK pcp will
> > > cache up to COSTLY_ORDER (3) and PMD order, but I'm probably not seeing the
> > > full picture)
> > 
> > With higher base page size, page allocator doesn't need to do as much
> > work to merge/split buddy pages. So serving the same 2M as order-5 is
> > cheaper than order-9.
> 
> I think the idea is that if most of your allocations (anon + pagecache) are
> 64k instead of 4k, on average, you'll just naturally do less merging
> splitting.

Yep. That plus slab_min_order would hopefully yield a system where 90%+
(depending on how your filesystem's buffer cache works) allocations are 64K.

-- 
Pedro

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/19/26 16:08, Kiryl Shutsemau wrote:
> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> 
> The kernel uses the same value - PAGE_SIZE - for two things:
> 
>    - the order-0 buddy allocation size;
> 
>    - the granularity of virtual address space mapping;
> 
> I think we can benefit from separating these two meanings and allowing
> order-0 allocations to be larger than the virtual address space covered by a
> PTE entry.
> 
> The main motivation is scalability. Managing memory on multi-terabyte
> machines in 4k is suboptimal, to say the least.
> 
> Potential benefits of the approach (assuming 64k pages):
> 
>    - The order-0 page size cuts struct page overhead by a factor of 16. From
>      ~1.6% of RAM to ~0.1%;
> 
>    - TLB wins on machines with TLB coalescing as long as mapping is naturally
>      aligned;
> 
>    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> 
>    - 1G pages are within possibility for the buddy allocator - order-14
>      allocation. It can open the road to 1G THPs.
> 
>    - As with THP, fewer pages - less pressure on the LRU lock;
> 
>    - ...
> 
> The trade-off is memory waste (similar to what we have on architectures with
> native 64k pages today) and complexity, mostly in the core-MM code.
> 
> == Design considerations ==
> 
> I want to split PAGE_SIZE into two distinct values:
> 
>    - PTE_SIZE defines the virtual address space granularity;
> 
>    - PG_SIZE defines the size of the order-0 buddy allocation;
> 
> PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> requires conversion, and keep existing code working while conversion is in
> progress.
> 
> The same split happens for other page-related macros: mask, shift,
> alignment helpers, etc.
> 
> PFNs are in PTE_SIZE units.
> 
> The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> units.
> 
> Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> for userspace. But we might want to communicate PG_SIZE to userspace to
> get the optimal results for userspace that cares.
> 
> PTE_SIZE granularity requires a substantial rework of page fault and VMA
> handling:
> 
>    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
>      We also need the offset within the page we are creating the PTE for.
> 
>    - Since the VMA start can be aligned arbitrarily with respect to the
>      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
>      which is in PTE_SIZE units.
> 
>    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
>      misaligned cases;
> 
> Page faults into file mappings are relatively simple to handle as we
> always have the page cache to refer to. So you can map only the part of the
> page that fits in the page table, similarly to fault-around.
> 
> Anonymous and file-CoW faults should also be simple as long as the VMA is
> aligned to PG_SIZE in both the virtual address space and with respect to
> vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> tolerable.
> 
> Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> pages across a page table boundary. In the worst case, a page is mapped across
> a PGD entry boundary and PTEs for the page have to be put in two separate
> subtrees of page tables.
> 
> A naive implementation would map different pages on different sides of a
> page table boundary and accept the waste of one page per page table crossing.
> The hope is that misaligned mappings are rare, but this is suboptimal.
> 
> mremap(2) is the ultimate stress test for the design.
> 
> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> is greater than 4 KB, we need a way to pack multiple page tables into a
> single page. We could use the slab allocator for this, but it would
> require relocating the page-table metadata out of struct page.

When discussing per-process page sizes with Ryan and Dev, I mentioned 
that having a larger emulated page size could be interesting for other 
architectures as well.

That is, we would emulate a 64K page size on Intel for user space as 
well, but let the OS work with 4K pages.

We'd only allocate+map large folios into user space + pagecache, but 
still allow for page tables etc. to not waste memory.

So "most" of your allocations in the system would actually be at least 
64k, reducing zone lock contention etc.


It doesn't solve all the problems you wanted to tackle on your list 
(e.g., "struct page" overhead, which will be sorted out by memdescs).

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
> On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >    - the order-0 buddy allocation size;
> > 
> >    - the granularity of virtual address space mapping;
> > 
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> > 
> > The main motivation is scalability. Managing memory on multi-terabyte
> > machines in 4k is suboptimal, to say the least.
> > 
> > Potential benefits of the approach (assuming 64k pages):
> > 
> >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> >      ~1.6% of RAM to ~0.1%;
> > 
> >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> >      aligned;
> > 
> >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > 
> >    - 1G pages are within possibility for the buddy allocator - order-14
> >      allocation. It can open the road to 1G THPs.
> > 
> >    - As with THP, fewer pages - less pressure on the LRU lock;
> > 
> >    - ...
> > 
> > The trade-off is memory waste (similar to what we have on architectures with
> > native 64k pages today) and complexity, mostly in the core-MM code.
> > 
> > == Design considerations ==
> > 
> > I want to split PAGE_SIZE into two distinct values:
> > 
> >    - PTE_SIZE defines the virtual address space granularity;
> > 
> >    - PG_SIZE defines the size of the order-0 buddy allocation;
> > 
> > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > requires conversion, and keep existing code working while conversion is in
> > progress.
> > 
> > The same split happens for other page-related macros: mask, shift,
> > alignment helpers, etc.
> > 
> > PFNs are in PTE_SIZE units.
> > 
> > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > units.
> > 
> > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > for userspace. But we might want to communicate PG_SIZE to userspace to
> > get the optimal results for userspace that cares.
> > 
> > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > handling:
> > 
> >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> >      We also need the offset within the page we are creating the PTE for.
> > 
> >    - Since the VMA start can be aligned arbitrarily with respect to the
> >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> >      which is in PTE_SIZE units.
> > 
> >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> >      misaligned cases;
> > 
> > Page faults into file mappings are relatively simple to handle as we
> > always have the page cache to refer to. So you can map only the part of the
> > page that fits in the page table, similarly to fault-around.
> > 
> > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > aligned to PG_SIZE in both the virtual address space and with respect to
> > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > tolerable.
> > 
> > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > pages across a page table boundary. In the worst case, a page is mapped across
> > a PGD entry boundary and PTEs for the page have to be put in two separate
> > subtrees of page tables.
> > 
> > A naive implementation would map different pages on different sides of a
> > page table boundary and accept the waste of one page per page table crossing.
> > The hope is that misaligned mappings are rare, but this is suboptimal.
> > 
> > mremap(2) is the ultimate stress test for the design.
> > 
> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
> 
> When discussing per-process page sizes with Ryan and Dev, I mentioned that
> having a larger emulated page size could be interesting for other
> architectures as well.
> 
> That is, we would emulate a 64K page size on Intel for user space as well,
> but let the OS work with 4K pages.
> 
> We'd only allocate+map large folios into user space + pagecache, but still
> allow for page tables etc. to not waste memory.
> 
> So "most" of your allocations in the system would actually be at least 64k,
> reducing zone lock contention etc.

I am not convinced emulation would help zone lock contention. I expect
contention to be higher if page allocator would see a mix of 4k and 64k
requests. It sounds like constant split/merge under the lock.

> It doesn't solve all the problems you wanted to tackle on your list (e.g.,
> "struct page" overhead, which will be sorted out by memdescs).

I don't think we can serve 1G pages out of buddy allocator with 4k
order-0. And without it, I don't see how to get to a viable 1G THPs.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/19/26 16:54, Kiryl Shutsemau wrote:
> On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
>> On 2/19/26 16:08, Kiryl Shutsemau wrote:
>>> No, there's no new hardware (that I know of). I want to explore what page size
>>> means.
>>>
>>> The kernel uses the same value - PAGE_SIZE - for two things:
>>>
>>>     - the order-0 buddy allocation size;
>>>
>>>     - the granularity of virtual address space mapping;
>>>
>>> I think we can benefit from separating these two meanings and allowing
>>> order-0 allocations to be larger than the virtual address space covered by a
>>> PTE entry.
>>>
>>> The main motivation is scalability. Managing memory on multi-terabyte
>>> machines in 4k is suboptimal, to say the least.
>>>
>>> Potential benefits of the approach (assuming 64k pages):
>>>
>>>     - The order-0 page size cuts struct page overhead by a factor of 16. From
>>>       ~1.6% of RAM to ~0.1%;
>>>
>>>     - TLB wins on machines with TLB coalescing as long as mapping is naturally
>>>       aligned;
>>>
>>>     - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
>>>
>>>     - 1G pages are within possibility for the buddy allocator - order-14
>>>       allocation. It can open the road to 1G THPs.
>>>
>>>     - As with THP, fewer pages - less pressure on the LRU lock;
>>>
>>>     - ...
>>>
>>> The trade-off is memory waste (similar to what we have on architectures with
>>> native 64k pages today) and complexity, mostly in the core-MM code.
>>>
>>> == Design considerations ==
>>>
>>> I want to split PAGE_SIZE into two distinct values:
>>>
>>>     - PTE_SIZE defines the virtual address space granularity;
>>>
>>>     - PG_SIZE defines the size of the order-0 buddy allocation;
>>>
>>> PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
>>> requires conversion, and keep existing code working while conversion is in
>>> progress.
>>>
>>> The same split happens for other page-related macros: mask, shift,
>>> alignment helpers, etc.
>>>
>>> PFNs are in PTE_SIZE units.
>>>
>>> The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
>>> units.
>>>
>>> Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
>>> for userspace. But we might want to communicate PG_SIZE to userspace to
>>> get the optimal results for userspace that cares.
>>>
>>> PTE_SIZE granularity requires a substantial rework of page fault and VMA
>>> handling:
>>>
>>>     - A struct page pointer and pgprot_t are not enough to create a PTE entry.
>>>       We also need the offset within the page we are creating the PTE for.
>>>
>>>     - Since the VMA start can be aligned arbitrarily with respect to the
>>>       underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
>>>       which is in PTE_SIZE units.
>>>
>>>     - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
>>>       misaligned cases;
>>>
>>> Page faults into file mappings are relatively simple to handle as we
>>> always have the page cache to refer to. So you can map only the part of the
>>> page that fits in the page table, similarly to fault-around.
>>>
>>> Anonymous and file-CoW faults should also be simple as long as the VMA is
>>> aligned to PG_SIZE in both the virtual address space and with respect to
>>> vm_pgoff. We might waste some memory on the ends of the VMA, but it is
>>> tolerable.
>>>
>>> Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
>>> pages across a page table boundary. In the worst case, a page is mapped across
>>> a PGD entry boundary and PTEs for the page have to be put in two separate
>>> subtrees of page tables.
>>>
>>> A naive implementation would map different pages on different sides of a
>>> page table boundary and accept the waste of one page per page table crossing.
>>> The hope is that misaligned mappings are rare, but this is suboptimal.
>>>
>>> mremap(2) is the ultimate stress test for the design.
>>>
>>> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
>>> is greater than 4 KB, we need a way to pack multiple page tables into a
>>> single page. We could use the slab allocator for this, but it would
>>> require relocating the page-table metadata out of struct page.
>>
>> When discussing per-process page sizes with Ryan and Dev, I mentioned that
>> having a larger emulated page size could be interesting for other
>> architectures as well.
>>
>> That is, we would emulate a 64K page size on Intel for user space as well,
>> but let the OS work with 4K pages.
>>
>> We'd only allocate+map large folios into user space + pagecache, but still
>> allow for page tables etc. to not waste memory.
>>
>> So "most" of your allocations in the system would actually be at least 64k,
>> reducing zone lock contention etc.
> 
> I am not convinced emulation would help zone lock contention. I expect
> contention to be higher if page allocator would see a mix of 4k and 64k
> requests. It sounds like constant split/merge under the lock.

If most your allocations are larger, then there isn't that much 
splitting/merging.

There will be some for the < 64k allocations of course, but when all 
user space+page cache is >= 64 then the split/merge + zone lock should 
be heavily reduced.

> 
>> It doesn't solve all the problems you wanted to tackle on your list (e.g.,
>> "struct page" overhead, which will be sorted out by memdescs).
> 
> I don't think we can serve 1G pages out of buddy allocator with 4k
> order-0. And without it, I don't see how to get to a viable 1G THPs.

Zi Yan was one working on this, and I think we had ideas on how to make 
that work in the long run.

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Zi Yan]

On 19 Feb 2026, at 11:09, David Hildenbrand (Arm) wrote:

> On 2/19/26 16:54, Kiryl Shutsemau wrote:
>> On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
>>> On 2/19/26 16:08, Kiryl Shutsemau wrote:
>>>> No, there's no new hardware (that I know of). I want to explore what page size
>>>> means.
>>>>
>>>> The kernel uses the same value - PAGE_SIZE - for two things:
>>>>
>>>>     - the order-0 buddy allocation size;
>>>>
>>>>     - the granularity of virtual address space mapping;
>>>>
>>>> I think we can benefit from separating these two meanings and allowing
>>>> order-0 allocations to be larger than the virtual address space covered by a
>>>> PTE entry.
>>>>
>>>> The main motivation is scalability. Managing memory on multi-terabyte
>>>> machines in 4k is suboptimal, to say the least.
>>>>
>>>> Potential benefits of the approach (assuming 64k pages):
>>>>
>>>>     - The order-0 page size cuts struct page overhead by a factor of 16. From
>>>>       ~1.6% of RAM to ~0.1%;
>>>>
>>>>     - TLB wins on machines with TLB coalescing as long as mapping is naturally
>>>>       aligned;
>>>>
>>>>     - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
>>>>
>>>>     - 1G pages are within possibility for the buddy allocator - order-14
>>>>       allocation. It can open the road to 1G THPs.
>>>>
>>>>     - As with THP, fewer pages - less pressure on the LRU lock;
>>>>
>>>>     - ...
>>>>
>>>> The trade-off is memory waste (similar to what we have on architectures with
>>>> native 64k pages today) and complexity, mostly in the core-MM code.
>>>>
>>>> == Design considerations ==
>>>>
>>>> I want to split PAGE_SIZE into two distinct values:
>>>>
>>>>     - PTE_SIZE defines the virtual address space granularity;
>>>>
>>>>     - PG_SIZE defines the size of the order-0 buddy allocation;
>>>>
>>>> PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
>>>> requires conversion, and keep existing code working while conversion is in
>>>> progress.
>>>>
>>>> The same split happens for other page-related macros: mask, shift,
>>>> alignment helpers, etc.
>>>>
>>>> PFNs are in PTE_SIZE units.
>>>>
>>>> The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
>>>> units.
>>>>
>>>> Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
>>>> for userspace. But we might want to communicate PG_SIZE to userspace to
>>>> get the optimal results for userspace that cares.
>>>>
>>>> PTE_SIZE granularity requires a substantial rework of page fault and VMA
>>>> handling:
>>>>
>>>>     - A struct page pointer and pgprot_t are not enough to create a PTE entry.
>>>>       We also need the offset within the page we are creating the PTE for.
>>>>
>>>>     - Since the VMA start can be aligned arbitrarily with respect to the
>>>>       underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
>>>>       which is in PTE_SIZE units.
>>>>
>>>>     - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
>>>>       misaligned cases;
>>>>
>>>> Page faults into file mappings are relatively simple to handle as we
>>>> always have the page cache to refer to. So you can map only the part of the
>>>> page that fits in the page table, similarly to fault-around.
>>>>
>>>> Anonymous and file-CoW faults should also be simple as long as the VMA is
>>>> aligned to PG_SIZE in both the virtual address space and with respect to
>>>> vm_pgoff. We might waste some memory on the ends of the VMA, but it is
>>>> tolerable.
>>>>
>>>> Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
>>>> pages across a page table boundary. In the worst case, a page is mapped across
>>>> a PGD entry boundary and PTEs for the page have to be put in two separate
>>>> subtrees of page tables.
>>>>
>>>> A naive implementation would map different pages on different sides of a
>>>> page table boundary and accept the waste of one page per page table crossing.
>>>> The hope is that misaligned mappings are rare, but this is suboptimal.
>>>>
>>>> mremap(2) is the ultimate stress test for the design.
>>>>
>>>> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
>>>> is greater than 4 KB, we need a way to pack multiple page tables into a
>>>> single page. We could use the slab allocator for this, but it would
>>>> require relocating the page-table metadata out of struct page.
>>>
>>> When discussing per-process page sizes with Ryan and Dev, I mentioned that
>>> having a larger emulated page size could be interesting for other
>>> architectures as well.
>>>
>>> That is, we would emulate a 64K page size on Intel for user space as well,
>>> but let the OS work with 4K pages.
>>>
>>> We'd only allocate+map large folios into user space + pagecache, but still
>>> allow for page tables etc. to not waste memory.
>>>
>>> So "most" of your allocations in the system would actually be at least 64k,
>>> reducing zone lock contention etc.
>>
>> I am not convinced emulation would help zone lock contention. I expect
>> contention to be higher if page allocator would see a mix of 4k and 64k
>> requests. It sounds like constant split/merge under the lock.
>
> If most your allocations are larger, then there isn't that much splitting/merging.
>
> There will be some for the < 64k allocations of course, but when all user space+page cache is >= 64 then the split/merge + zone lock should be heavily reduced.
>
>>
>>> It doesn't solve all the problems you wanted to tackle on your list (e.g.,
>>> "struct page" overhead, which will be sorted out by memdescs).
>>
>> I don't think we can serve 1G pages out of buddy allocator with 4k
>> order-0. And without it, I don't see how to get to a viable 1G THPs.
>
> Zi Yan was one working on this, and I think we had ideas on how to make that work in the long run.

Right. The idea is to add super pageblock (or whatever name), which consists of N consecutive
pageblocks, so that anti fragmentation can work at larger granularity, e.g., 1GB, to create
free pages. Whether 1GB free pages from memory compaction need to go into buddy allocator
or not is debatable.

--
Best Regards,
Yan, Zi

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
>> When discussing per-process page sizes with Ryan and Dev, I mentioned that
>> having a larger emulated page size could be interesting for other
>> architectures as well.
>> That is, we would emulate a 64K page size on Intel for user space as well,
>> but let the OS work with 4K pages.
>> We'd only allocate+map large folios into user space + pagecache, but still
>> allow for page tables etc. to not waste memory.
>> So "most" of your allocations in the system would actually be at least 64k,
>> reducing zone lock contention etc.

Am Do, Feb 19, 2026 um 15:54:56 +0000, Kiryl Shutsemau schrieb:
> I am not convinced emulation would help zone lock contention. I expect
> contention to be higher if page allocator would see a mix of 4k and 64k
> requests. It sounds like constant split/merge under the lock.

Presumably because the higher-order allocations would lack the per-CPU
special-case handling for order 0? IIRC LLFree is lock-free at all
allocation orders, which would likely help the zone lock specifically.
I suppose there's a sense I'm hoping my memory isn't out-of-date, but I
would still be glad to see the improvement. Extending per-CPU freelists
to small nonzero orders z.B. 4 might also help zone locks to allocate
64 KiB case likely around as much as using PAGE_MMUSHIFT == 4.


On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
>> It doesn't solve all the problems you wanted to tackle on your list (e.g.,
>> "struct page" overhead, which will be sorted out by memdescs).

Am Do, Feb 19, 2026 um 15:54:56 +0000, Kiryl Shutsemau schrieb:
> I don't think we can serve 1G pages out of buddy allocator with 4k
> order-0. And without it, I don't see how to get to a viable 1G THPs.

memdescs are too recent for me to know about, but there is a comment to
make about 1 GiB superpages: doubly exponential decays in availability
are going to overwhelm the effects of any reasonable valud of
PAGE_MMUSHIFT (your ilog(PG_SIZE/PTE_SIZE) which has no macro for it)
long before 1 GiB. Bumping a small constant off the grand total of
allocation orders isn't going to be an adequate strategy. ISTR Rik van
Riel had code going round to partition RAM into something akin to
pre-formed large z.B. 1 GiB superpages that's a far more meaningful
approach to 1 GiB superpages than reducing the allocation order for
1 GiB from 18 to 14 or 12. The double exponential makes orders above
maybe 2 or 3 require migration or compaction and there's a cut-off above
which you're going to need to be either be fresh off a virgin boot or to
do something slow bordering on swsusp-style mass eviction.

If memdescs, presumably from „memory descriptors“, are something like
memory extents, I wholeheartedly endorse moves to more extent-based or
extent-like memory management and can quibble about the details later.

You can make an argument that scaling may need to take special steps to
reduce the TLB entries for the smallest page sizes so they don't
outcompete the real datasets when they proliferate. But buddy allocating
(LLFree should be used instead) isn't going to be appreciably any better
at orders 14 (PAGE_MMUSHIFT == 4) or 12 (PAGE_MMUSHIFT == 6) than 18, as
with double exponential decays, it dropped to effectively zero onf.
vmtl. 5 or 6, and values of PAGE_SIZE much larger than 64-256 KiB would
likely require far more extensive efforts to address regressions driven
by bitblitting latency and internal fragmentation than simple things like
tail packing the page cache. If you're starting from an
MMUPAGE_SIZE/PTE_SIZE of 64 KiB z.B. ppc64, it only really changes the
story to cutting 14 to 10 or 8, but the net result of that with
PAGE_MMUSHIFT == 4 is a PAGE_SIZE of 1 MiB and for PAGE_MMUSHIFT == 6 of
4 MiB, which are clearly beyond the pale of what'll be tolerable for
insb. network latencies for high-performance scenarios.

There is a case where those handfuls of buddy bitmap orders makes a
difference: if you're trying to split the difference between z.B. the
MMUPAGE_SHIFT/ilog2(PTE_SIZE) and PMD_SHIFT. That's not reaching 1 GiB,
though. That's the only real benefit possible to offer in the trivial
case of the sparse page size spectrum. Dense page size spectra like
MIPS, LoongArch, RISC-V Svnapot, and Fujitsu M12 SPARC (I presume IA64
has sadly been completely discontinued at this point) are more of where
the benefits an argument can be made for are concentrated. In the
desperation of the sparse page size spectrum case z.B. x86 page size
growth increment / Sprungweite of 9 or some Apple M5 ARM systems' 11,
trimming orders off of it might be worth it even after something in the
vein of Rik's large superpage / „huge page“ -sized arenas, but with some
devoted to the first superpage size as that's too many orders off from
MMUPAGE_SIZE for naïve handling to work well.

To be clear, I am in favour of page clustering as per your pte_size,
Andrea's hard-page-size, my pgcl (not genuinely distinct from Hugh's
except via my being a far lesser coder than he), and Hugh's larpage.
My objection is that the rationale you've given isn't sustainable.


-- nyc

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 04:39:34PM +0100, David Hildenbrand (Arm) wrote:
> On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >    - the order-0 buddy allocation size;
> > 
> >    - the granularity of virtual address space mapping;
> > 
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> > 
> > The main motivation is scalability. Managing memory on multi-terabyte
> > machines in 4k is suboptimal, to say the least.
> > 
> > Potential benefits of the approach (assuming 64k pages):
> > 
> >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> >      ~1.6% of RAM to ~0.1%;
> > 
> >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> >      aligned;
> > 
> >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > 
> >    - 1G pages are within possibility for the buddy allocator - order-14
> >      allocation. It can open the road to 1G THPs.
> > 
> >    - As with THP, fewer pages - less pressure on the LRU lock;
> > 
> >    - ...
> > 
> > The trade-off is memory waste (similar to what we have on architectures with
> > native 64k pages today) and complexity, mostly in the core-MM code.
> > 
> > == Design considerations ==
> > 
> > I want to split PAGE_SIZE into two distinct values:
> > 
> >    - PTE_SIZE defines the virtual address space granularity;
> > 
> >    - PG_SIZE defines the size of the order-0 buddy allocation;
> > 
> > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > requires conversion, and keep existing code working while conversion is in
> > progress.
> > 
> > The same split happens for other page-related macros: mask, shift,
> > alignment helpers, etc.
> > 
> > PFNs are in PTE_SIZE units.
> > 
> > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > units.
> > 
> > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > for userspace. But we might want to communicate PG_SIZE to userspace to
> > get the optimal results for userspace that cares.
> > 
> > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > handling:
> > 
> >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> >      We also need the offset within the page we are creating the PTE for.
> > 
> >    - Since the VMA start can be aligned arbitrarily with respect to the
> >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> >      which is in PTE_SIZE units.
> > 
> >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> >      misaligned cases;
> > 
> > Page faults into file mappings are relatively simple to handle as we
> > always have the page cache to refer to. So you can map only the part of the
> > page that fits in the page table, similarly to fault-around.
> > 
> > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > aligned to PG_SIZE in both the virtual address space and with respect to
> > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > tolerable.
> > 
> > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > pages across a page table boundary. In the worst case, a page is mapped across
> > a PGD entry boundary and PTEs for the page have to be put in two separate
> > subtrees of page tables.
> > 
> > A naive implementation would map different pages on different sides of a
> > page table boundary and accept the waste of one page per page table crossing.
> > The hope is that misaligned mappings are rare, but this is suboptimal.
> > 
> > mremap(2) is the ultimate stress test for the design.
> > 
> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
> 
> When discussing per-process page sizes with Ryan and Dev, I mentioned that
> having a larger emulated page size could be interesting for other
> architectures as well.
>
> That is, we would emulate a 64K page size on Intel for user space as well,
> but let the OS work with 4K pages.

Just to clarify, do you want it to be enforced on userspace ABI.
Like, all mappings are 64k aligned?

> We'd only allocate+map large folios into user space + pagecache, but still
> allow for page tables etc. to not waste memory.

Waste of memory for page table is solvable and pretty straight forward.
Most of such cases can be solve mechanically by switching to slab.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

>> When discussing per-process page sizes with Ryan and Dev, I mentioned that
>> having a larger emulated page size could be interesting for other
>> architectures as well.
>>
>> That is, we would emulate a 64K page size on Intel for user space as well,
>> but let the OS work with 4K pages.
> 
> Just to clarify, do you want it to be enforced on userspace ABI.
> Like, all mappings are 64k aligned?

Right, see the proposal from Dev on the list.

 From user-space POV, the pagesize would be 64K for these emulated 
processes. That is, VMAs must be suitable aligned etc.

One key thing I think is that you could run such emulated-64k process 
(that actually support it!) with 4k processes on the same machine, like 
Arm is considering.

You would have no weird "vma crosses base pages" handling, which is just 
rather nasty and makes my head hurt.

> 
>> We'd only allocate+map large folios into user space + pagecache, but still
>> allow for page tables etc. to not waste memory.
> 
> Waste of memory for page table is solvable and pretty straight forward.
> Most of such cases can be solve mechanically by switching to slab.

Well, yes, like Willy says, there are already similar custom solutions 
for s390x and ppc.

Pasha talked recently about the memory waste of 16k kernel stacks and 
how we would want to reduce that to 4k. In your proposal, it would be 
64k, unless you somehow manage to allocate multiple kernel stacks from 
the same 64k page. My head hurts thinking about whether that could work, 
maybe it could (no idea about guard pages in there, though).


Let's take a look at the history of page size usage on Arm (people can 
feel free to correct me):

(1) Most distros were using 64k on Arm.

(2) People realized that 64k was suboptimal many use cases (memory
     waste for stacks, pagecache, etc) and started to switch to 4k. I
     remember that mostly HPC-centric users sticked to 64k, but there was
     also demand from others to be able to stay on 64k.

(3) Arm improved performance on a 4k kernel by adding cont-pte support,
     trying to get closer to 64k native performance.

(4) Achieving 64k native performance is hard, which is why per-process
     page sizes are being explored to get the best out of both worlds
     (use 64k page size only where it really matters for performance).

Arm clearly has the added benefit of actually benefiting from hardware 
support for 64k.

IIUC, what you are proposing feels a bit like traveling back in time 
when it comes to the memory waste problem that Arm users encountered.

Where do you see the big difference to 64k on Arm in your proposal? 
Would you currently also be running 64k Arm in production and the memory 
waste etc is acceptable?

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Fri, Feb 20, 2026 at 11:24:37AM +0100, David Hildenbrand (Arm) wrote:
> > > When discussing per-process page sizes with Ryan and Dev, I mentioned that
> > > having a larger emulated page size could be interesting for other
> > > architectures as well.
> > > 
> > > That is, we would emulate a 64K page size on Intel for user space as well,
> > > but let the OS work with 4K pages.
> > 
> > Just to clarify, do you want it to be enforced on userspace ABI.
> > Like, all mappings are 64k aligned?
> 
> Right, see the proposal from Dev on the list.
> 
> From user-space POV, the pagesize would be 64K for these emulated processes.
> That is, VMAs must be suitable aligned etc.

Well, it will drastically limit the adoption. We have too much legacy
stuff on x86.

> > > We'd only allocate+map large folios into user space + pagecache, but still
> > > allow for page tables etc. to not waste memory.
> > 
> > Waste of memory for page table is solvable and pretty straight forward.
> > Most of such cases can be solve mechanically by switching to slab.
> 
> Well, yes, like Willy says, there are already similar custom solutions for
> s390x and ppc.
> 
> Pasha talked recently about the memory waste of 16k kernel stacks and how we
> would want to reduce that to 4k. In your proposal, it would be 64k, unless
> you somehow manage to allocate multiple kernel stacks from the same 64k
> page. My head hurts thinking about whether that could work, maybe it could
> (no idea about guard pages in there, though).

Kernel stack is allocated from vmalloc. I think mapping them with
sub-page granularity should be doable.

BTW, do you see any reason why slab-allocated stack wouldn't work for
large base page sizes? There's no requirement for it be aligned to page
or PTE, right?

> Let's take a look at the history of page size usage on Arm (people can feel
> free to correct me):
> 
> (1) Most distros were using 64k on Arm.
> 
> (2) People realized that 64k was suboptimal many use cases (memory
>     waste for stacks, pagecache, etc) and started to switch to 4k. I
>     remember that mostly HPC-centric users sticked to 64k, but there was
>     also demand from others to be able to stay on 64k.
> 
> (3) Arm improved performance on a 4k kernel by adding cont-pte support,
>     trying to get closer to 64k native performance.
> 
> (4) Achieving 64k native performance is hard, which is why per-process
>     page sizes are being explored to get the best out of both worlds
>     (use 64k page size only where it really matters for performance).
> 
> Arm clearly has the added benefit of actually benefiting from hardware
> support for 64k.
> 
> IIUC, what you are proposing feels a bit like traveling back in time when it
> comes to the memory waste problem that Arm users encountered.
> 
> Where do you see the big difference to 64k on Arm in your proposal? Would
> you currently also be running 64k Arm in production and the memory waste etc
> is acceptable?

That's the point. I don't see a big difference to 64k Arm. I want to
bring this option to x86: at some machine size it makes sense trade
memory consumption for scalability. I am targeting it to machines with
over 2TiB of RAM.

BTW, we do run 64k Arm in our fleet. There's some growing pains, but it
looks good in general We have no plans to switch to 4k (or 16k) at the
moment. 512M THPs also look good on some workloads.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/20/26 13:07, Kiryl Shutsemau wrote:
> On Fri, Feb 20, 2026 at 11:24:37AM +0100, David Hildenbrand (Arm) wrote:
>>>
>>> Just to clarify, do you want it to be enforced on userspace ABI.
>>> Like, all mappings are 64k aligned?
>>
>> Right, see the proposal from Dev on the list.
>>
>>  From user-space POV, the pagesize would be 64K for these emulated processes.
>> That is, VMAs must be suitable aligned etc.
> 
> Well, it will drastically limit the adoption. We have too much legacy
> stuff on x86.

I'd assume that many applications nowadays can deal with differing page 
sizes (thanks to some other architectures paving the way).

But yes, some real legacy stuff, or stuff that ever only cared about 
intel still hardcodes pagesize=4k.

In Meta's fleet, I'd be quite interesting how much conversion there 
would have to be done.

For legacy apps, you could still run them as 4k pagesize on the same 
system, of course.

> 
>>>
>>> Waste of memory for page table is solvable and pretty straight forward.
>>> Most of such cases can be solve mechanically by switching to slab.
>>
>> Well, yes, like Willy says, there are already similar custom solutions for
>> s390x and ppc.
>>
>> Pasha talked recently about the memory waste of 16k kernel stacks and how we
>> would want to reduce that to 4k. In your proposal, it would be 64k, unless
>> you somehow manage to allocate multiple kernel stacks from the same 64k
>> page. My head hurts thinking about whether that could work, maybe it could
>> (no idea about guard pages in there, though).
> 
> Kernel stack is allocated from vmalloc. I think mapping them with
> sub-page granularity should be doable.

I still have to wrap my head around the sub-page mapping here as well. 
It's scary.

Re mapcount: I think if any part of the page is mapped, it would be 
considered mapped -> mapcount += 1.

> 
> BTW, do you see any reason why slab-allocated stack wouldn't work for
> large base page sizes? There's no requirement for it be aligned to page
> or PTE, right?

I'd assume that would work. Devil is in the detail with these things 
before we have memdescs.

E.g., page table have a dedicated type (PGTY_table) and store separate 
metadata in the ptdesc. For kernel stack there was once a proposal to 
have a type but it is not upstream.

> 
>> Let's take a look at the history of page size usage on Arm (people can feel
>> free to correct me):
>>
>> (1) Most distros were using 64k on Arm.
>>
>> (2) People realized that 64k was suboptimal many use cases (memory
>>      waste for stacks, pagecache, etc) and started to switch to 4k. I
>>      remember that mostly HPC-centric users sticked to 64k, but there was
>>      also demand from others to be able to stay on 64k.
>>
>> (3) Arm improved performance on a 4k kernel by adding cont-pte support,
>>      trying to get closer to 64k native performance.
>>
>> (4) Achieving 64k native performance is hard, which is why per-process
>>      page sizes are being explored to get the best out of both worlds
>>      (use 64k page size only where it really matters for performance).
>>
>> Arm clearly has the added benefit of actually benefiting from hardware
>> support for 64k.
>>
>> IIUC, what you are proposing feels a bit like traveling back in time when it
>> comes to the memory waste problem that Arm users encountered.
>>
>> Where do you see the big difference to 64k on Arm in your proposal? Would
>> you currently also be running 64k Arm in production and the memory waste etc
>> is acceptable?
> 
> That's the point. I don't see a big difference to 64k Arm. I want to
> bring this option to x86: at some machine size it makes sense trade
> memory consumption for scalability. I am targeting it to machines with
> over 2TiB of RAM.
> 
> BTW, we do run 64k Arm in our fleet. There's some growing pains, but it
> looks good in general We have no plans to switch to 4k (or 16k) at the
> moment. 512M THPs also look good on some workloads.

Okay, that's valuable information, thanks!

Being able to remove the sub-page mapping part (or being able to just 
hide it somewhere deep down in arch code) would make this a lot easier 
to digest.

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kalesh Singh]

On Fri, Feb 20, 2026 at 8:30 AM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 2/20/26 13:07, Kiryl Shutsemau wrote:
> > On Fri, Feb 20, 2026 at 11:24:37AM +0100, David Hildenbrand (Arm) wrote:
> >>>
> >>> Just to clarify, do you want it to be enforced on userspace ABI.
> >>> Like, all mappings are 64k aligned?
> >>
> >> Right, see the proposal from Dev on the list.
> >>
> >>  From user-space POV, the pagesize would be 64K for these emulated processes.
> >> That is, VMAs must be suitable aligned etc.
> >
> > Well, it will drastically limit the adoption. We have too much legacy
> > stuff on x86.
>
> I'd assume that many applications nowadays can deal with differing page
> sizes (thanks to some other architectures paving the way).
>
> But yes, some real legacy stuff, or stuff that ever only cared about
> intel still hardcodes pagesize=4k.

I think most issues will stem from linkers setting the default ELF
segment alignment (max-page-size) for x86 to 4096. So those ELFs will
not load correctly or at all on the larger emulated granularity.

-- Kalesh

>
> In Meta's fleet, I'd be quite interesting how much conversion there
> would have to be done.
>
> For legacy apps, you could still run them as 4k pagesize on the same
> system, of course.
>
> >
> >>>
> >>> Waste of memory for page table is solvable and pretty straight forward.
> >>> Most of such cases can be solve mechanically by switching to slab.
> >>
> >> Well, yes, like Willy says, there are already similar custom solutions for
> >> s390x and ppc.
> >>
> >> Pasha talked recently about the memory waste of 16k kernel stacks and how we
> >> would want to reduce that to 4k. In your proposal, it would be 64k, unless
> >> you somehow manage to allocate multiple kernel stacks from the same 64k
> >> page. My head hurts thinking about whether that could work, maybe it could
> >> (no idea about guard pages in there, though).
> >
> > Kernel stack is allocated from vmalloc. I think mapping them with
> > sub-page granularity should be doable.
>
> I still have to wrap my head around the sub-page mapping here as well.
> It's scary.
>
> Re mapcount: I think if any part of the page is mapped, it would be
> considered mapped -> mapcount += 1.
>
> >
> > BTW, do you see any reason why slab-allocated stack wouldn't work for
> > large base page sizes? There's no requirement for it be aligned to page
> > or PTE, right?
>
> I'd assume that would work. Devil is in the detail with these things
> before we have memdescs.
>
> E.g., page table have a dedicated type (PGTY_table) and store separate
> metadata in the ptdesc. For kernel stack there was once a proposal to
> have a type but it is not upstream.
>
> >
> >> Let's take a look at the history of page size usage on Arm (people can feel
> >> free to correct me):
> >>
> >> (1) Most distros were using 64k on Arm.
> >>
> >> (2) People realized that 64k was suboptimal many use cases (memory
> >>      waste for stacks, pagecache, etc) and started to switch to 4k. I
> >>      remember that mostly HPC-centric users sticked to 64k, but there was
> >>      also demand from others to be able to stay on 64k.
> >>
> >> (3) Arm improved performance on a 4k kernel by adding cont-pte support,
> >>      trying to get closer to 64k native performance.
> >>
> >> (4) Achieving 64k native performance is hard, which is why per-process
> >>      page sizes are being explored to get the best out of both worlds
> >>      (use 64k page size only where it really matters for performance).
> >>
> >> Arm clearly has the added benefit of actually benefiting from hardware
> >> support for 64k.
> >>
> >> IIUC, what you are proposing feels a bit like traveling back in time when it
> >> comes to the memory waste problem that Arm users encountered.
> >>
> >> Where do you see the big difference to 64k on Arm in your proposal? Would
> >> you currently also be running 64k Arm in production and the memory waste etc
> >> is acceptable?
> >
> > That's the point. I don't see a big difference to 64k Arm. I want to
> > bring this option to x86: at some machine size it makes sense trade
> > memory consumption for scalability. I am targeting it to machines with
> > over 2TiB of RAM.
> >
> > BTW, we do run 64k Arm in our fleet. There's some growing pains, but it
> > looks good in general We have no plans to switch to 4k (or 16k) at the
> > moment. 512M THPs also look good on some workloads.
>
> Okay, that's valuable information, thanks!
>
> Being able to remove the sub-page mapping part (or being able to just
> hide it somewhere deep down in arch code) would make this a lot easier
> to digest.
>
> --
> Cheers,
>
> David
>

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/20/26 20:33, Kalesh Singh wrote:
> On Fri, Feb 20, 2026 at 8:30 AM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 2/20/26 13:07, Kiryl Shutsemau wrote:
>>>
>>> Well, it will drastically limit the adoption. We have too much legacy
>>> stuff on x86.
>>
>> I'd assume that many applications nowadays can deal with differing page
>> sizes (thanks to some other architectures paving the way).
>>
>> But yes, some real legacy stuff, or stuff that ever only cared about
>> intel still hardcodes pagesize=4k.
> 
> I think most issues will stem from linkers setting the default ELF
> segment alignment (max-page-size) for x86 to 4096. So those ELFs will
> not load correctly or at all on the larger emulated granularity.

Right, I assume that they will have to be thought about that, and 
possibly, some binaries/libraries recompiled.

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Mon, Feb 23, 2026 at 12:04:10PM +0100, David Hildenbrand (Arm) wrote:
> On 2/20/26 20:33, Kalesh Singh wrote:
> > On Fri, Feb 20, 2026 at 8:30 AM David Hildenbrand (Arm)
> > <david@kernel.org> wrote:
> > > 
> > > On 2/20/26 13:07, Kiryl Shutsemau wrote:
> > > > 
> > > > Well, it will drastically limit the adoption. We have too much legacy
> > > > stuff on x86.
> > > 
> > > I'd assume that many applications nowadays can deal with differing page
> > > sizes (thanks to some other architectures paving the way).
> > > 
> > > But yes, some real legacy stuff, or stuff that ever only cared about
> > > intel still hardcodes pagesize=4k.
> > 
> > I think most issues will stem from linkers setting the default ELF
> > segment alignment (max-page-size) for x86 to 4096. So those ELFs will
> > not load correctly or at all on the larger emulated granularity.
> 
> Right, I assume that they will have to be thought about that, and possibly,
> some binaries/libraries recompiled.

I think backward compatibility is important and I believe we can get
there without ABI break. And optimize from there.

BTW, x86-64 SysV ABI allows for 64k page size:

	Systems are permitted to use any power-of-two page size between
	4KB and 64KB, inclusive.

But it doesn't work in practice.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/23/26 12:13, Kiryl Shutsemau wrote:
> On Mon, Feb 23, 2026 at 12:04:10PM +0100, David Hildenbrand (Arm) wrote:
>> On 2/20/26 20:33, Kalesh Singh wrote:
>>> On Fri, Feb 20, 2026 at 8:30 AM David Hildenbrand (Arm)
>>> <david@kernel.org> wrote:
>>>
>>> I think most issues will stem from linkers setting the default ELF
>>> segment alignment (max-page-size) for x86 to 4096. So those ELFs will
>>> not load correctly or at all on the larger emulated granularity.
>>
>> Right, I assume that they will have to be thought about that, and possibly,
>> some binaries/libraries recompiled.
> 
> I think backward compatibility is important and I believe we can get
> there without ABI break. And optimize from there.
> 
> BTW, x86-64 SysV ABI allows for 64k page size:
> 
> 	Systems are permitted to use any power-of-two page size between
> 	4KB and 64KB, inclusive.
> 
> But it doesn't work in practice.

Even in well controlled environments you would run in a hyperscaler?

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Mon, Feb 23, 2026 at 12:27:33PM +0100, David Hildenbrand (Arm) wrote:
> On 2/23/26 12:13, Kiryl Shutsemau wrote:
> > On Mon, Feb 23, 2026 at 12:04:10PM +0100, David Hildenbrand (Arm) wrote:
> > > On 2/20/26 20:33, Kalesh Singh wrote:
> > > > On Fri, Feb 20, 2026 at 8:30 AM David Hildenbrand (Arm)
> > > > <david@kernel.org> wrote:
> > > > 
> > > > I think most issues will stem from linkers setting the default ELF
> > > > segment alignment (max-page-size) for x86 to 4096. So those ELFs will
> > > > not load correctly or at all on the larger emulated granularity.
> > > 
> > > Right, I assume that they will have to be thought about that, and possibly,
> > > some binaries/libraries recompiled.
> > 
> > I think backward compatibility is important and I believe we can get
> > there without ABI break. And optimize from there.
> > 
> > BTW, x86-64 SysV ABI allows for 64k page size:
> > 
> > 	Systems are permitted to use any power-of-two page size between
> > 	4KB and 64KB, inclusive.
> > 
> > But it doesn't work in practice.
> 
> Even in well controlled environments you would run in a hyperscaler?

I have not invested much time into investigating this.

I intentionally targeted compatible version assuming it will be better
received by upstream. I want it to be usable outside specially cured
userspace. 64k might not be good fit for a desktop, but 16k can be a
different story.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Dave Hansen]

On 2/23/26 03:27, David Hildenbrand (Arm) wrote:
...
>> BTW, x86-64 SysV ABI allows for 64k page size:
>>
>>     Systems are permitted to use any power-of-two page size between
>>     4KB and 64KB, inclusive.
>>
>> But it doesn't work in practice.
> 
> Even in well controlled environments you would run in a hyperscaler?

I think what Kirill is trying to say is that "it breaks userspace". ;)

A hyperscaler (or other "embedded" environment) might be willing or able
to go fix up userspace breakage. I would suspect our high frequency
trading friends would be all over this if it shaved a microsecond off
their receive times.

The more important question is what it breaks and how badly it breaks
things. 5-level paging, for instance, broke some JITs that historically
used the new (>48) upper virtual address bits for metadata. The gains
from 5-level paging were big enough and the userspace breakage was
confined and fixable enough that 5-level paging was viable.

I'm not sure which side a larger base page side will fall on, though. Is
it going to be an out-of-tree hack that a few folks use, or will it be
more like 5-level paging and be good enough that it goes into mainline?

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 2/23/26 16:14, Dave Hansen wrote:
> On 2/23/26 03:27, David Hildenbrand (Arm) wrote:
> ...
>>> BTW, x86-64 SysV ABI allows for 64k page size:
>>>
>>>      Systems are permitted to use any power-of-two page size between
>>>      4KB and 64KB, inclusive.
>>>
>>> But it doesn't work in practice.
>>
>> Even in well controlled environments you would run in a hyperscaler?
> 
> I think what Kirill is trying to say is that "it breaks userspace". ;)

Yes. Probably similar to Intel proposing an actual 64k page size. 
Expected. :)

> 
> A hyperscaler (or other "embedded" environment) might be willing or able
> to go fix up userspace breakage. I would suspect our high frequency
> trading friends would be all over this if it shaved a microsecond off
> their receive times.
> 
> The more important question is what it breaks and how badly it breaks
> things. 5-level paging, for instance, broke some JITs that historically
> used the new (>48) upper virtual address bits for metadata. The gains
> from 5-level paging were big enough and the userspace breakage was
> confined and fixable enough that 5-level paging was viable.
> 
> I'm not sure which side a larger base page side will fall on, though. Is
> it going to be an out-of-tree hack that a few folks use, or will it be
> more like 5-level paging and be good enough that it goes into mainline?

Just thinking about VMAs spanning partial pages makes me shiver. Or A 
single page spanning multiple VMAs.

I haven't seen the code yet, but I am certain that I will not like it.

I'm happy to be proven wrong :)

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Mon, Feb 23, 2026 at 04:31:56PM +0100, David Hildenbrand (Arm) wrote:
> On 2/23/26 16:14, Dave Hansen wrote:
> > On 2/23/26 03:27, David Hildenbrand (Arm) wrote:
> > ...
> > > > BTW, x86-64 SysV ABI allows for 64k page size:
> > > > 
> > > >      Systems are permitted to use any power-of-two page size between
> > > >      4KB and 64KB, inclusive.
> > > > 
> > > > But it doesn't work in practice.
> > > 
> > > Even in well controlled environments you would run in a hyperscaler?
> > 
> > I think what Kirill is trying to say is that "it breaks userspace". ;)
> 
> Yes. Probably similar to Intel proposing an actual 64k page size. Expected.
> :)
> 
> > 
> > A hyperscaler (or other "embedded" environment) might be willing or able
> > to go fix up userspace breakage. I would suspect our high frequency
> > trading friends would be all over this if it shaved a microsecond off
> > their receive times.
> > 
> > The more important question is what it breaks and how badly it breaks
> > things. 5-level paging, for instance, broke some JITs that historically
> > used the new (>48) upper virtual address bits for metadata. The gains
> > from 5-level paging were big enough and the userspace breakage was
> > confined and fixable enough that 5-level paging was viable.
> > 
> > I'm not sure which side a larger base page side will fall on, though. Is
> > it going to be an out-of-tree hack that a few folks use, or will it be
> > more like 5-level paging and be good enough that it goes into mainline?
> 
> Just thinking about VMAs spanning partial pages makes me shiver. Or A
> single page spanning multiple VMAs.

Hate to break it to you, but we have it now upstream :P

THP can span multiple VMAs. And can be partially mapped.

The only new thing is that we allow this for order-0 page now. And you
cannot realistically recover wasted memory -- no deferred split.

> I haven't seen the code yet, but I am certain that I will not like it.
> 
> I'm happy to be proven wrong :)

I will do my best, but no promises :)

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

>>
>> Just thinking about VMAs spanning partial pages makes me shiver. Or A
>> single page spanning multiple VMAs.
> 
> Hate to break it to you, but we have it now upstream :P
> 
> THP can span multiple VMAs. And can be partially mapped.

Single mapcount, single anon-exclusive flag.

Completely different story :P

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Lorenzo Stoakes]

On Mon, Feb 23, 2026 at 04:31:56PM +0100, David Hildenbrand (Arm) wrote:
> On 2/23/26 16:14, Dave Hansen wrote:
> > On 2/23/26 03:27, David Hildenbrand (Arm) wrote:
> > ...
> > > > BTW, x86-64 SysV ABI allows for 64k page size:
> > > >
> > > >      Systems are permitted to use any power-of-two page size between
> > > >      4KB and 64KB, inclusive.
> > > >
> > > > But it doesn't work in practice.
> > >
> > > Even in well controlled environments you would run in a hyperscaler?
> >
> > I think what Kirill is trying to say is that "it breaks userspace". ;)
>
> Yes. Probably similar to Intel proposing an actual 64k page size. Expected.
> :)
>
> >
> > A hyperscaler (or other "embedded" environment) might be willing or able
> > to go fix up userspace breakage. I would suspect our high frequency
> > trading friends would be all over this if it shaved a microsecond off
> > their receive times.
> >
> > The more important question is what it breaks and how badly it breaks
> > things. 5-level paging, for instance, broke some JITs that historically
> > used the new (>48) upper virtual address bits for metadata. The gains
> > from 5-level paging were big enough and the userspace breakage was
> > confined and fixable enough that 5-level paging was viable.
> >
> > I'm not sure which side a larger base page side will fall on, though. Is
> > it going to be an out-of-tree hack that a few folks use, or will it be
> > more like 5-level paging and be good enough that it goes into mainline?
>
> Just thinking about VMAs spanning partial pages makes me shiver. Or A single
> page spanning multiple VMAs.

Yeah agree, we're not doing this.

It's already a nightmare to deal with per-page anonexclusive vs. per-folio
pretty much everything else, and we shouldn't have allowed that to be a thing,
but now we have to live with it.

>
> I haven't seen the code yet, but I am certain that I will not like it.

If the code tries to implement anything that even resembles some sub-base-page
metadata then that's just not something that's going to land.

Handling VMA vs. folio state coherently is _already_ painful and difficult.

Piling on more complexity because we theoretically could feels rather along the
lines of 'let's just keep adding features and not worrying about where we end
up', which is I think a bit of an anti-pattern in the kernel in general.

>
> I'm happy to be proven wrong :)

Sure me also if I'm missing something, but what's discussed here is... worrying.

>
> --
> Cheers,
>
> David

Thanks, Lorenzo

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Laight]

On Mon, 23 Feb 2026 07:14:39 -0800
Dave Hansen <dave.hansen@intel.com> wrote:

> On 2/23/26 03:27, David Hildenbrand (Arm) wrote:
> ...
> >> BTW, x86-64 SysV ABI allows for 64k page size:
> >>
> >>     Systems are permitted to use any power-of-two page size between
> >>     4KB and 64KB, inclusive.
> >>
> >> But it doesn't work in practice.  
> > 
> > Even in well controlled environments you would run in a hyperscaler?  
> 
> I think what Kirill is trying to say is that "it breaks userspace". ;)

With a 4k physical page what stops you dynamically splitting the 64k a
'struct page' references into 16 4k pages (using an extra dynamically
allocated structure)?
I'm not thinking it would happen that often, but it would solve the
problem of 4k aligned .data and (probably) mmap() of small files.

If the cpu supports TLB coalescing there could easily be a net gain
using 64k pages for most of a program binary.

	David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Right, see the proposal from Dev on the list.
> From user-space POV, the pagesize would be 64K for these emulated processes.
> That is, VMAs must be suitable aligned etc.
> One key thing I think is that you could run such emulated-64k process (that
> actually support it!) with 4k processes on the same machine, like Arm is
> considering.
> You would have no weird "vma crosses base pages" handling, which is just
> rather nasty and makes my head hurt.

While it had to be written, it was a straightforward extension of
converting ->vm_pgoff to being in MMUPAGE_SIZE (Kiryl's PTE_SIZE) units,
recovering offsets into pages usw. Nothing seemed terribly hard to
understand about it. Really, the code as it stands rarely approaches
what it's doing in such a way that it broadly surveys the layout of vmas
and pages and be conscious of whether pages spanned vmas or multiple
vmas were representing maps of of fragments of pages, so I'm not
convinced there's anything particularly unique to be afraid of here.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Well, yes, like Willy says, there are already similar custom solutions for
> s390x and ppc.
> Pasha talked recently about the memory waste of 16k kernel stacks and how we
> would want to reduce that to 4k. In your proposal, it would be 64k, unless
> you somehow manage to allocate multiple kernel stacks from the same 64k
> page. My head hurts thinking about whether that could work, maybe it could
> (no idea about guard pages in there, though).

Never mind wasting memory; just getting it booting in 2003 needed
adjusting things so the THREAD_SIZE stayed constant and there were
things in the core kernel that got a divide by zero or some such when
THREAD_SIZE was too large. The effort for the fix in 2003 was just
getting adequate diagnostics, not the stack allocation code itself.
The mechanically assisted forward port did something that got it booting
that, upon critical examination, I don't like and uses a lot of space.
vmalloc and guard pages don't have anything special about them either,
apart from taking work to implement, which I've done before as part of
what appears to be a now non-extant patch kit, stack_paranoia.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Let's take a look at the history of page size usage on Arm (people can feel
> free to correct me):
> (1) Most distros were using 64k on Arm.
> (2) People realized that 64k was suboptimal many use cases (memory
>     waste for stacks, pagecache, etc) and started to switch to 4k. I
>     remember that mostly HPC-centric users sticked to 64k, but there was
>     also demand from others to be able to stay on 64k.
> (3) Arm improved performance on a 4k kernel by adding cont-pte support,
>     trying to get closer to 64k native performance.
> (4) Achieving 64k native performance is hard, which is why per-process
>     page sizes are being explored to get the best out of both worlds
>     (use 64k page size only where it really matters for performance).

There are ways to deal with internal fragmentation, like tail packing
for the pagecache, switching to slab for a lot of things, potentially
even pagetables, so they don't grow with PAGE_SIZE usw. Stacks might
take more work than the average kernel data structure, but I've
implemented changes to stack allocation before at points when the
vmalloc and size varying usw. usf. stack bits weren't in mainline.
Bitblitting latencies are also issues to try to keep bounded that I've
seen little discussion of here yet.

I suppose the bad news is that I didn't get the chance to actually work
on the internal fragmentation mitigation strategies I wrote out plans
for 23 years ago. They weren't that involved, though.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Arm clearly has the added benefit of actually benefiting from hardware
> support for 64k.
> IIUC, what you are proposing feels a bit like traveling back in time when it
> comes to the memory waste problem that Arm users encountered.
> Where do you see the big difference to 64k on Arm in your proposal? Would
> you currently also be running 64k Arm in production and the memory waste etc
> is acceptable?

My original 2003 intentions for the use of Hugh's ABI compatibility
technique were to make page clustering as a solution for 32-bit large
memory „forward-looking to 64-bit“, where at the time, 64-bit meant
IA64, by using it to provide a guarantee that small superpage
allocations would never fail due to external fragmentation, which design
goal came from above. While some colouring benefits usw. were
anticipated as byproducts, this idea about how to handle small
superpages was the primary way it was conceived of as addressing 64-bit
concerns, perhaps in part by dint of IA64-centrism, or potentially a
kind of inheritance of a requirement for an equivalent of the
very-differently-implemented mechanism for the same from HP-UX. It also
seemed rather safe to set the goal of „raising the floor“ so the very
smallest TLB entries wouldn't proliferate, and the smallest page sizes
consume a lesser fraction of available TLB entries.

Proposals for trying to use the technique as some kind of fully general
method for reaching for 1 GiB if not larger pages seem ill-conceived to
me. Those kinds of issues demand changes akin to transforming the
problem so it's in an entirely lesser complexity class, not adding a
small constant to the methods already in place attempting to deal with
large-scale external fragmentation. I liked Rik's partitioning memory
into arenas of pre-constructed 1 GiB or otherwise very large superpages
as a strategy for that, for instance.

I generally care enough about architectural coverage that my recent
mechanically assisted efforts maintained an 80+ -cell regression testing
matrix of 20+ architectures with PAGE_MMUSHIFT values of 0, 2, 4 and 6.
ARM notably represented an extra architecture variant via LPAE. The arch
code I cry about was not having got the chance to use the recent MIPS
PageGrain extension to provide a 1 KiB MMUPAGE_SIZE and a PAGE_MMUSHIFT
of 8 for demonstration purposes. If I had got that done, I might have
even emailed Babaoğlu, Joy, and possibly even McKusick about it, though
Karels is reputedly the most likely suspect for who outside of Babaoğlu
and Joy might have committed CLSIZE u.a. to 3BSD's VAX VM code in 1979.

I made at least earnest attempts to cover every architecture in the
kernel. There was one I couldn't find a toolchain for that just wired
all of the MMUPAGE macros to be identical to their PAGE analogues and
hoped at least built (arc?), but otherwise, I was trying to cover them
all, even if at the moment only in QEMU. I hope I can put a personal
many-architecture testing lab like I had in the '00s together for
myself again at some point, given the chance, but they'll have to be
small things like for embedded this time with how flats in Berlin are.


-- nyc

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kalesh Singh]

On Thu, Feb 19, 2026 at 7:39 AM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> >
> > The kernel uses the same value - PAGE_SIZE - for two things:
> >
> >    - the order-0 buddy allocation size;
> >
> >    - the granularity of virtual address space mapping;
> >
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> >
> > The main motivation is scalability. Managing memory on multi-terabyte
> > machines in 4k is suboptimal, to say the least.
> >
> > Potential benefits of the approach (assuming 64k pages):
> >
> >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> >      ~1.6% of RAM to ~0.1%;
> >
> >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> >      aligned;
> >
> >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> >
> >    - 1G pages are within possibility for the buddy allocator - order-14
> >      allocation. It can open the road to 1G THPs.
> >
> >    - As with THP, fewer pages - less pressure on the LRU lock;
> >
> >    - ...
> >
> > The trade-off is memory waste (similar to what we have on architectures with
> > native 64k pages today) and complexity, mostly in the core-MM code.
> >
> > == Design considerations ==
> >
> > I want to split PAGE_SIZE into two distinct values:
> >
> >    - PTE_SIZE defines the virtual address space granularity;
> >
> >    - PG_SIZE defines the size of the order-0 buddy allocation;
> >
> > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > requires conversion, and keep existing code working while conversion is in
> > progress.
> >
> > The same split happens for other page-related macros: mask, shift,
> > alignment helpers, etc.
> >
> > PFNs are in PTE_SIZE units.
> >
> > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > units.
> >
> > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > for userspace. But we might want to communicate PG_SIZE to userspace to
> > get the optimal results for userspace that cares.
> >
> > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > handling:
> >
> >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> >      We also need the offset within the page we are creating the PTE for.
> >
> >    - Since the VMA start can be aligned arbitrarily with respect to the
> >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> >      which is in PTE_SIZE units.
> >
> >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> >      misaligned cases;
> >
> > Page faults into file mappings are relatively simple to handle as we
> > always have the page cache to refer to. So you can map only the part of the
> > page that fits in the page table, similarly to fault-around.
> >
> > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > aligned to PG_SIZE in both the virtual address space and with respect to
> > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > tolerable.
> >
> > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > pages across a page table boundary. In the worst case, a page is mapped across
> > a PGD entry boundary and PTEs for the page have to be put in two separate
> > subtrees of page tables.
> >
> > A naive implementation would map different pages on different sides of a
> > page table boundary and accept the waste of one page per page table crossing.
> > The hope is that misaligned mappings are rare, but this is suboptimal.
> >
> > mremap(2) is the ultimate stress test for the design.
> >
> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
>
> When discussing per-process page sizes with Ryan and Dev, I mentioned
> that having a larger emulated page size could be interesting for other
> architectures as well.
>
> That is, we would emulate a 64K page size on Intel for user space as
> well, but let the OS work with 4K pages.
>
> We'd only allocate+map large folios into user space + pagecache, but
> still allow for page tables etc. to not waste memory.
>
> So "most" of your allocations in the system would actually be at least
> 64k, reducing zone lock contention etc.
>
>
> It doesn't solve all the problems you wanted to tackle on your list
> (e.g., "struct page" overhead, which will be sorted out by memdescs).

Hi Kiryl,

I'd be interested to discuss this at LSFMM.

On Android, we have a separate but related use case: we emulate the
userspace page size on x86, primarily to enable app developers to
conduct compatibility testing of their apps for 16KB Android devices.
[1]

It mainly works by enforcing a larger granularity on the VMAs to
emulate a userspace page size, somewhat similar to what David
mentioned, while the underlying kernel still operates on a 4KB
granularity. [2]

IIUC the current design would not enfore the larger granularity /
alignment for VMAs to avoid breaking ABI. However, I'd be interest to
discuss whether it can be extended to cover this usecase as well.

[1]  https://developer.android.com/guide/practices/page-sizes#16kb-emulator
[2] https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic

Thanks,
Kalesh




>
> --
> Cheers,
>
> David
>

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 03:24:37PM -0800, Kalesh Singh wrote:
> On Thu, Feb 19, 2026 at 7:39 AM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
> >
> > On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > > No, there's no new hardware (that I know of). I want to explore what page size
> > > means.
> > >
> > > The kernel uses the same value - PAGE_SIZE - for two things:
> > >
> > >    - the order-0 buddy allocation size;
> > >
> > >    - the granularity of virtual address space mapping;
> > >
> > > I think we can benefit from separating these two meanings and allowing
> > > order-0 allocations to be larger than the virtual address space covered by a
> > > PTE entry.
> > >
> > > The main motivation is scalability. Managing memory on multi-terabyte
> > > machines in 4k is suboptimal, to say the least.
> > >
> > > Potential benefits of the approach (assuming 64k pages):
> > >
> > >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> > >      ~1.6% of RAM to ~0.1%;
> > >
> > >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> > >      aligned;
> > >
> > >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > >
> > >    - 1G pages are within possibility for the buddy allocator - order-14
> > >      allocation. It can open the road to 1G THPs.
> > >
> > >    - As with THP, fewer pages - less pressure on the LRU lock;
> > >
> > >    - ...
> > >
> > > The trade-off is memory waste (similar to what we have on architectures with
> > > native 64k pages today) and complexity, mostly in the core-MM code.
> > >
> > > == Design considerations ==
> > >
> > > I want to split PAGE_SIZE into two distinct values:
> > >
> > >    - PTE_SIZE defines the virtual address space granularity;
> > >
> > >    - PG_SIZE defines the size of the order-0 buddy allocation;
> > >
> > > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > > requires conversion, and keep existing code working while conversion is in
> > > progress.
> > >
> > > The same split happens for other page-related macros: mask, shift,
> > > alignment helpers, etc.
> > >
> > > PFNs are in PTE_SIZE units.
> > >
> > > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > > units.
> > >
> > > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > > for userspace. But we might want to communicate PG_SIZE to userspace to
> > > get the optimal results for userspace that cares.
> > >
> > > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > > handling:
> > >
> > >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> > >      We also need the offset within the page we are creating the PTE for.
> > >
> > >    - Since the VMA start can be aligned arbitrarily with respect to the
> > >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> > >      which is in PTE_SIZE units.
> > >
> > >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> > >      misaligned cases;
> > >
> > > Page faults into file mappings are relatively simple to handle as we
> > > always have the page cache to refer to. So you can map only the part of the
> > > page that fits in the page table, similarly to fault-around.
> > >
> > > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > > aligned to PG_SIZE in both the virtual address space and with respect to
> > > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > > tolerable.
> > >
> > > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > > pages across a page table boundary. In the worst case, a page is mapped across
> > > a PGD entry boundary and PTEs for the page have to be put in two separate
> > > subtrees of page tables.
> > >
> > > A naive implementation would map different pages on different sides of a
> > > page table boundary and accept the waste of one page per page table crossing.
> > > The hope is that misaligned mappings are rare, but this is suboptimal.
> > >
> > > mremap(2) is the ultimate stress test for the design.
> > >
> > > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > > is greater than 4 KB, we need a way to pack multiple page tables into a
> > > single page. We could use the slab allocator for this, but it would
> > > require relocating the page-table metadata out of struct page.
> >
> > When discussing per-process page sizes with Ryan and Dev, I mentioned
> > that having a larger emulated page size could be interesting for other
> > architectures as well.
> >
> > That is, we would emulate a 64K page size on Intel for user space as
> > well, but let the OS work with 4K pages.
> >
> > We'd only allocate+map large folios into user space + pagecache, but
> > still allow for page tables etc. to not waste memory.
> >
> > So "most" of your allocations in the system would actually be at least
> > 64k, reducing zone lock contention etc.
> >
> >
> > It doesn't solve all the problems you wanted to tackle on your list
> > (e.g., "struct page" overhead, which will be sorted out by memdescs).
> 
> Hi Kiryl,
> 
> I'd be interested to discuss this at LSFMM.
> 
> On Android, we have a separate but related use case: we emulate the
> userspace page size on x86, primarily to enable app developers to
> conduct compatibility testing of their apps for 16KB Android devices.
> [1]
> 
> It mainly works by enforcing a larger granularity on the VMAs to
> emulate a userspace page size, somewhat similar to what David
> mentioned, while the underlying kernel still operates on a 4KB
> granularity. [2]
> 
> IIUC the current design would not enfore the larger granularity /
> alignment for VMAs to avoid breaking ABI. However, I'd be interest to
> discuss whether it can be extended to cover this usecase as well.

I don't want to break ABI, but might add a knob (maybe personality(2) ?)
for enforcement to see what breaks.

In general, I would prefer to advertise a new value to userspace that
would mean preferred virtual address space granularity.

> 
> [1]  https://developer.android.com/guide/practices/page-sizes#16kb-emulator
> [2] https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic
> 
> Thanks,
> Kalesh
> 
> 
> 
> 
> >
> > --
> > Cheers,
> >
> > David
> >

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kalesh Singh]

On Fri, Feb 20, 2026 at 4:10 AM Kiryl Shutsemau <kas@kernel.org> wrote:
>
> On Thu, Feb 19, 2026 at 03:24:37PM -0800, Kalesh Singh wrote:
> > On Thu, Feb 19, 2026 at 7:39 AM David Hildenbrand (Arm)
> > <david@kernel.org> wrote:
> > >
> > > On 2/19/26 16:08, Kiryl Shutsemau wrote:
> > > > No, there's no new hardware (that I know of). I want to explore what page size
> > > > means.
> > > >
> > > > The kernel uses the same value - PAGE_SIZE - for two things:
> > > >
> > > >    - the order-0 buddy allocation size;
> > > >
> > > >    - the granularity of virtual address space mapping;
> > > >
> > > > I think we can benefit from separating these two meanings and allowing
> > > > order-0 allocations to be larger than the virtual address space covered by a
> > > > PTE entry.
> > > >
> > > > The main motivation is scalability. Managing memory on multi-terabyte
> > > > machines in 4k is suboptimal, to say the least.
> > > >
> > > > Potential benefits of the approach (assuming 64k pages):
> > > >
> > > >    - The order-0 page size cuts struct page overhead by a factor of 16. From
> > > >      ~1.6% of RAM to ~0.1%;
> > > >
> > > >    - TLB wins on machines with TLB coalescing as long as mapping is naturally
> > > >      aligned;
> > > >
> > > >    - Order-5 allocation is 2M, resulting in less pressure on the zone lock;
> > > >
> > > >    - 1G pages are within possibility for the buddy allocator - order-14
> > > >      allocation. It can open the road to 1G THPs.
> > > >
> > > >    - As with THP, fewer pages - less pressure on the LRU lock;
> > > >
> > > >    - ...
> > > >
> > > > The trade-off is memory waste (similar to what we have on architectures with
> > > > native 64k pages today) and complexity, mostly in the core-MM code.
> > > >
> > > > == Design considerations ==
> > > >
> > > > I want to split PAGE_SIZE into two distinct values:
> > > >
> > > >    - PTE_SIZE defines the virtual address space granularity;
> > > >
> > > >    - PG_SIZE defines the size of the order-0 buddy allocation;
> > > >
> > > > PAGE_SIZE is only defined if PTE_SIZE == PG_SIZE. It will flag which code
> > > > requires conversion, and keep existing code working while conversion is in
> > > > progress.
> > > >
> > > > The same split happens for other page-related macros: mask, shift,
> > > > alignment helpers, etc.
> > > >
> > > > PFNs are in PTE_SIZE units.
> > > >
> > > > The buddy allocator and page cache (as well as all I/O) operate in PG_SIZE
> > > > units.
> > > >
> > > > Userspace mappings are maintained with PTE_SIZE granularity. No ABI changes
> > > > for userspace. But we might want to communicate PG_SIZE to userspace to
> > > > get the optimal results for userspace that cares.
> > > >
> > > > PTE_SIZE granularity requires a substantial rework of page fault and VMA
> > > > handling:
> > > >
> > > >    - A struct page pointer and pgprot_t are not enough to create a PTE entry.
> > > >      We also need the offset within the page we are creating the PTE for.
> > > >
> > > >    - Since the VMA start can be aligned arbitrarily with respect to the
> > > >      underlying page, vma->vm_pgoff has to be changed to vma->vm_pteoff,
> > > >      which is in PTE_SIZE units.
> > > >
> > > >    - The page fault handler needs to handle PTE_SIZE < PG_SIZE, including
> > > >      misaligned cases;
> > > >
> > > > Page faults into file mappings are relatively simple to handle as we
> > > > always have the page cache to refer to. So you can map only the part of the
> > > > page that fits in the page table, similarly to fault-around.
> > > >
> > > > Anonymous and file-CoW faults should also be simple as long as the VMA is
> > > > aligned to PG_SIZE in both the virtual address space and with respect to
> > > > vm_pgoff. We might waste some memory on the ends of the VMA, but it is
> > > > tolerable.
> > > >
> > > > Misaligned anonymous and file-CoW faults are a pain. Specifically, mapping
> > > > pages across a page table boundary. In the worst case, a page is mapped across
> > > > a PGD entry boundary and PTEs for the page have to be put in two separate
> > > > subtrees of page tables.
> > > >
> > > > A naive implementation would map different pages on different sides of a
> > > > page table boundary and accept the waste of one page per page table crossing.
> > > > The hope is that misaligned mappings are rare, but this is suboptimal.
> > > >
> > > > mremap(2) is the ultimate stress test for the design.
> > > >
> > > > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > > > is greater than 4 KB, we need a way to pack multiple page tables into a
> > > > single page. We could use the slab allocator for this, but it would
> > > > require relocating the page-table metadata out of struct page.
> > >
> > > When discussing per-process page sizes with Ryan and Dev, I mentioned
> > > that having a larger emulated page size could be interesting for other
> > > architectures as well.
> > >
> > > That is, we would emulate a 64K page size on Intel for user space as
> > > well, but let the OS work with 4K pages.
> > >
> > > We'd only allocate+map large folios into user space + pagecache, but
> > > still allow for page tables etc. to not waste memory.
> > >
> > > So "most" of your allocations in the system would actually be at least
> > > 64k, reducing zone lock contention etc.
> > >
> > >
> > > It doesn't solve all the problems you wanted to tackle on your list
> > > (e.g., "struct page" overhead, which will be sorted out by memdescs).
> >
> > Hi Kiryl,
> >
> > I'd be interested to discuss this at LSFMM.
> >
> > On Android, we have a separate but related use case: we emulate the
> > userspace page size on x86, primarily to enable app developers to
> > conduct compatibility testing of their apps for 16KB Android devices.
> > [1]
> >
> > It mainly works by enforcing a larger granularity on the VMAs to
> > emulate a userspace page size, somewhat similar to what David
> > mentioned, while the underlying kernel still operates on a 4KB
> > granularity. [2]
> >
> > IIUC the current design would not enfore the larger granularity /
> > alignment for VMAs to avoid breaking ABI. However, I'd be interest to
> > discuss whether it can be extended to cover this usecase as well.
>
> I don't want to break ABI, but might add a knob (maybe personality(2) ?)
> for enforcement to see what breaks.

I think personality(2) may be too late? By the time a process invokes
it, the initial userspace mappings (executable, linker for init, etc)
are already established with the default granularity.

To handle this, I've been using an early_param to enforce the larger
VMA alignment system-wide right from boot.

Perhaps, something for global enforcement (Kconfig/early param) and a
prctl/personality flag for per-process opt in?

>
> In general, I would prefer to advertise a new value to userspace that
> would mean preferred virtual address space granularity.

This makes sense for maintaining ABI compatibility. Userspace
allocators might want to optimize their layouts to match PG_SIZE while
still being able to operate at PTE_SIZE when needed.

-- Kalesh

>
> >
> > [1]  https://developer.android.com/guide/practices/page-sizes#16kb-emulator
> > [2] https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic
> >
> > Thanks,
> > Kalesh
> >
> >
> >
> >
> > >
> > > --
> > > Cheers,
> > >
> > > David
> > >
>
> --
>   Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Dave Hansen]

On 2/19/26 07:08, Kiryl Shutsemau wrote:
>   - The order-0 page size cuts struct page overhead by a factor of 16. From
>     ~1.6% of RAM to ~0.1%;

First of all, this looks like fun. Nice work! I'm not opposed at all in
concept to cleaning up things and doing the logical separation you
described to split buddy granularity and mapping granularity. That seems
like a worthy endeavor and some of the union/#define tricks look like a
likely viable way to do it incrementally.

But I don't think there's going to be a lot of memory savings in the
end. Maybe this would bring the mem= hyperscalers back into the fold and
have them actually start using 'struct page' again for their VM memory.
Dunno.

But, let's look at my kernel directory and round the file sizes up to
4k, 16k and 64k:

find .  -printf '%s\n' | while read size; do echo	\
		$(((size + 0x0fff) & 0xfffff000))	\
		$(((size + 0x3fff) & 0xffffc000))	\
		$(((size + 0xffff) & 0xffff0000));
done

... and add them all up:

11,297,648 KB - on disk
11,297,712 KB - in a 4k page cache
12,223,488 KB - in a 16k page cache
16,623,296 KB - in a 64k page cache

So a 64k page cache eats ~5GB of extra memory for a kernel tree (well,
_my_ kernel tree). In other words, if you are looking for memory savings
on my laptop, you'll need ~300GB of RAM before 'struct page' overhead
overwhelms the page cache bloat from a single kernel tree.

The whole kernel obviously isn't in the page cache all at the same time.
The page cache across the system is also obviously different than a
kernel tree, but you get the point.

That's not to diminish how useful something like this might be,
especially for folks that are sensitive to 'struct page' overhead or
allocator performance.

But, it will mostly be getting better performance at the _cost_ of
consuming more RAM, not saving RAM.

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> On 2/19/26 07:08, Kiryl Shutsemau wrote:
> >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> >     ~1.6% of RAM to ~0.1%;
> ...
> But, it will mostly be getting better performance at the _cost_ of
> consuming more RAM, not saving RAM.

That's fair.

The problem with struct page memory consumption is that it is static and
cannot be reclaimed. You pay the struct page tax no matter what.

Page cache rounding overhead can be large, but a motivated userspace can
keep it under control by avoiding splitting a dataset into many small
files. And this memory is reclaimable.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Liam R. Howlett]

* Kiryl Shutsemau <kas@kernel.org> [260219 17:05]:
> On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> > On 2/19/26 07:08, Kiryl Shutsemau wrote:
> > >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> > >     ~1.6% of RAM to ~0.1%;
> > ...
> > But, it will mostly be getting better performance at the _cost_ of
> > consuming more RAM, not saving RAM.
> 
> That's fair.
> 
> The problem with struct page memory consumption is that it is static and
> cannot be reclaimed. You pay the struct page tax no matter what.
> 
> Page cache rounding overhead can be large, but a motivated userspace can
> keep it under control by avoiding splitting a dataset into many small
> files. And this memory is reclaimable.
> 

But we are in reclaim a lot more these days.  As I'm sure you are aware,
we are trying to maximize the resources (both cpu and ram) of any
machine powered on.  Entering reclaim will consume the cpu time and will
affect other tasks.

Especially with multiple workload machines, the tendency is to have a
primary focus with the lower desired work being killed, if necessary.
Reducing the overhead just means more secondary tasks, or a bigger
footprint of the ones already executing.

Increasing the memory pressure will degrade the primary workload more
frequently, even if we recover enough to avoid OOMing the secondary.

While in the struct page tax world, the secondary task would be killed
after a shorter (and less frequently executed) reclaim comes up short.
So, I would think that we would be degrading the primary workload in an
attempt to keep the secondary alive?  Maybe I'm over-simplifying here?

Near the other end of the spectrum, we have chromebooks that are
constantly in reclaim, even with 4k pages.  I guess these machines would
be destine to maintain the same page size they use today.  That is, this
solution for the struct page tax is only useful if you have a lot of
memory.  But then again, that's where the bookkeeping costs become hard
to take.

Thanks,
Liam

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 10:28:20PM -0500, Liam R. Howlett wrote:
> * Kiryl Shutsemau <kas@kernel.org> [260219 17:05]:
> > On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> > > On 2/19/26 07:08, Kiryl Shutsemau wrote:
> > > >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> > > >     ~1.6% of RAM to ~0.1%;
> > > ...
> > > But, it will mostly be getting better performance at the _cost_ of
> > > consuming more RAM, not saving RAM.
> > 
> > That's fair.
> > 
> > The problem with struct page memory consumption is that it is static and
> > cannot be reclaimed. You pay the struct page tax no matter what.
> > 
> > Page cache rounding overhead can be large, but a motivated userspace can
> > keep it under control by avoiding splitting a dataset into many small
> > files. And this memory is reclaimable.
> > 
> 
> But we are in reclaim a lot more these days.  As I'm sure you are aware,
> we are trying to maximize the resources (both cpu and ram) of any
> machine powered on.  Entering reclaim will consume the cpu time and will
> affect other tasks.
> 
> Especially with multiple workload machines, the tendency is to have a
> primary focus with the lower desired work being killed, if necessary.
> Reducing the overhead just means more secondary tasks, or a bigger
> footprint of the ones already executing.
> 
> Increasing the memory pressure will degrade the primary workload more
> frequently, even if we recover enough to avoid OOMing the secondary.
> 
> While in the struct page tax world, the secondary task would be killed
> after a shorter (and less frequently executed) reclaim comes up short.
> So, I would think that we would be degrading the primary workload in an
> attempt to keep the secondary alive?  Maybe I'm over-simplifying here?

I am not sure I fully follow your point.

Sizing tasks and scheduling tasks between machines is hard in general.
I don't think the balance between struct page tax and page cache
rounding overhead is going to be the primary factor.

> Near the other end of the spectrum, we have chromebooks that are
> constantly in reclaim, even with 4k pages.  I guess these machines would
> be destine to maintain the same page size they use today.  That is, this
> solution for the struct page tax is only useful if you have a lot of
> memory.  But then again, that's where the bookkeeping costs become hard
> to take.

Smaller machines are not target for 64k pages. They will not benefit
from them.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Liam R. Howlett]

* Kiryl Shutsemau <kas@kernel.org> [260220 07:33]:
> On Thu, Feb 19, 2026 at 10:28:20PM -0500, Liam R. Howlett wrote:
> > * Kiryl Shutsemau <kas@kernel.org> [260219 17:05]:
> > > On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> > > > On 2/19/26 07:08, Kiryl Shutsemau wrote:
> > > > >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> > > > >     ~1.6% of RAM to ~0.1%;
> > > > ...
> > > > But, it will mostly be getting better performance at the _cost_ of
> > > > consuming more RAM, not saving RAM.
> > > 
> > > That's fair.
> > > 
> > > The problem with struct page memory consumption is that it is static and
> > > cannot be reclaimed. You pay the struct page tax no matter what.
> > > 
> > > Page cache rounding overhead can be large, but a motivated userspace can
> > > keep it under control by avoiding splitting a dataset into many small
> > > files. And this memory is reclaimable.
> > > 
> > 
> > But we are in reclaim a lot more these days.  As I'm sure you are aware,
> > we are trying to maximize the resources (both cpu and ram) of any
> > machine powered on.  Entering reclaim will consume the cpu time and will
> > affect other tasks.
> > 
> > Especially with multiple workload machines, the tendency is to have a
> > primary focus with the lower desired work being killed, if necessary.
> > Reducing the overhead just means more secondary tasks, or a bigger
> > footprint of the ones already executing.
> > 
> > Increasing the memory pressure will degrade the primary workload more
> > frequently, even if we recover enough to avoid OOMing the secondary.
> > 
> > While in the struct page tax world, the secondary task would be killed
> > after a shorter (and less frequently executed) reclaim comes up short.
> > So, I would think that we would be degrading the primary workload in an
> > attempt to keep the secondary alive?  Maybe I'm over-simplifying here?
> 
> I am not sure I fully follow your point.
> 
> Sizing tasks and scheduling tasks between machines is hard in general.
> I don't think the balance between struct page tax and page cache
> rounding overhead is going to be the primary factor.

I think there are more trade offs than what you listed.  It's still
probably worth doing, but I wanted to know if you though that this would
cause us to spend more time in reclaim, which seems to be implied above.
So, another trade-off might be all the reclaim penalty being paid more
frequently?

...

Thanks,
Liam

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Fri, Feb 20, 2026 at 10:17:45AM -0500, Liam R. Howlett wrote:
> * Kiryl Shutsemau <kas@kernel.org> [260220 07:33]:
> > On Thu, Feb 19, 2026 at 10:28:20PM -0500, Liam R. Howlett wrote:
> > > * Kiryl Shutsemau <kas@kernel.org> [260219 17:05]:
> > > > On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> > > > > On 2/19/26 07:08, Kiryl Shutsemau wrote:
> > > > > >   - The order-0 page size cuts struct page overhead by a factor of 16. From
> > > > > >     ~1.6% of RAM to ~0.1%;
> > > > > ...
> > > > > But, it will mostly be getting better performance at the _cost_ of
> > > > > consuming more RAM, not saving RAM.
> > > > 
> > > > That's fair.
> > > > 
> > > > The problem with struct page memory consumption is that it is static and
> > > > cannot be reclaimed. You pay the struct page tax no matter what.
> > > > 
> > > > Page cache rounding overhead can be large, but a motivated userspace can
> > > > keep it under control by avoiding splitting a dataset into many small
> > > > files. And this memory is reclaimable.
> > > > 
> > > 
> > > But we are in reclaim a lot more these days.  As I'm sure you are aware,
> > > we are trying to maximize the resources (both cpu and ram) of any
> > > machine powered on.  Entering reclaim will consume the cpu time and will
> > > affect other tasks.
> > > 
> > > Especially with multiple workload machines, the tendency is to have a
> > > primary focus with the lower desired work being killed, if necessary.
> > > Reducing the overhead just means more secondary tasks, or a bigger
> > > footprint of the ones already executing.
> > > 
> > > Increasing the memory pressure will degrade the primary workload more
> > > frequently, even if we recover enough to avoid OOMing the secondary.
> > > 
> > > While in the struct page tax world, the secondary task would be killed
> > > after a shorter (and less frequently executed) reclaim comes up short.
> > > So, I would think that we would be degrading the primary workload in an
> > > attempt to keep the secondary alive?  Maybe I'm over-simplifying here?
> > 
> > I am not sure I fully follow your point.
> > 
> > Sizing tasks and scheduling tasks between machines is hard in general.
> > I don't think the balance between struct page tax and page cache
> > rounding overhead is going to be the primary factor.
> 
> I think there are more trade offs than what you listed.  It's still
> probably worth doing, but I wanted to know if you though that this would
> cause us to spend more time in reclaim, which seems to be implied above.
> So, another trade-off might be all the reclaim penalty being paid more
> frequently?

I am not sure.

Kernel would need to do less work in reclaim per unit of memory.
Depending on workloads you might see less allocation events and
therefore less frequent reclaim.

It's all too hand-wavy at the stage.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> First of all, this looks like fun. Nice work! I'm not opposed at all in
> concept to cleaning up things and doing the logical separation you
> described to split buddy granularity and mapping granularity. That seems
> like a worthy endeavor and some of the union/#define tricks look like a
> likely viable way to do it incrementally.

My 2003 pgcl code was the funnest kernel hacking I've ever done. You
were there (or at least involved; I don't remember in enough detail to
even be sure if we were together in person) when it got the first public
boot on 32-bit x86 with 64 GiB RAM, and even noted that my earlier work
on shrinking the size of struct page had got things booting with
ridiculously little lowmem left over after mem_map when you were
flagging me down for 16 nodes being available to be glued together.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> But I don't think there's going to be a lot of memory savings in the
> end. Maybe this would bring the mem= hyperscalers back into the fold and
> have them actually start using 'struct page' again for their VM memory.
> Dunno.

For 32-bit it was only really an issue because it was lowmem. From day
one back in 2003 it was always a matter of trading off internal for
external fragmentation, and there was even some worry about losses from
internal fragmentation potentially exhausting lowmem under load. It's in
a way like being able to adjust the frontier between internal and
external fragmentation more than really saving memory at all. Apart from
lowmem exhaustion on 32-bit and maybe being so large and heavily used
that it competes with userspace for cache and TLB, the struct page
array's memory consumption is mostly irrelevant.

strictly speaking, struct page as presently conceived isn't necessary at
all. There are mostly microkernels out there that do no per-page memory
bookkeeping. The basic pattern is that the commitment to what kind of
bookkeeping to do is mostly deferred until it gets allocated and put in
use, at which point things like slab memory, pagecache memory,
pagetables and such all have different ways of tracking everything. The
basic allocators can do a variety of things, too. In-band metadata IOW
using the memory the allocator is saying is technically free to hold
metadata is one of the tricks, adaptively switching between radix tree
or bitmapped things, using address calculation to only need things like
a pair of bits for each radix tree node's entries, and maybe even bit
twiddling lockfree code etc. all start getting in the mix like LLFree:

https://www.usenix.org/conference/atc23/presentation/wrenger
https://www.sra.uni-hannover.de/Publications/2023/llfree-atc23/
https://github.com/luhsra/llfree-c

You still get a couple of bits per page, but the delayed metadata
allocation getting less internal fragmentation(!) from forcing too many
disparate metadata types into a union isn't saving anywhere near as much
as the disappearance of the centralised, readily-identifiable struct
page array might suggest, so there's more than enough left over to
swamp those bits. Having slab, pagecache, pagetable etc. metadata all
unmixed with each other still means they have significant metadata
footprints that are significant fractions of what a struct page array
would be. The real advantage for memory hyperscalars is that splitting
them those ways enables more locality of reference.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> But, let's look at my kernel directory and round the file sizes up to
> 4k, 16k and 64k:
> find .  -printf '%s\n' | while read size; do echo	\
> 		$(((size + 0x0fff) & 0xfffff000))	\
> 		$(((size + 0x3fff) & 0xffffc000))	\
> 		$(((size + 0xffff) & 0xffff0000));
> done
> ... and add them all up:
> 11,297,648 KB - on disk
> 11,297,712 KB - in a 4k page cache
> 12,223,488 KB - in a 16k page cache
> 16,623,296 KB - in a 64k page cache

4 KiB being so close to disc is surprising esp. since disc is usually
doing tail packing atop 512 B blocks. The rest is unsurprising.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> So a 64k page cache eats ~5GB of extra memory for a kernel tree (well,
> _my_ kernel tree). In other words, if you are looking for memory savings
> on my laptop, you'll need ~300GB of RAM before 'struct page' overhead
> overwhelms the page cache bloat from a single kernel tree.
> The whole kernel obviously isn't in the page cache all at the same time.
> The page cache across the system is also obviously different than a
> kernel tree, but you get the point.
> That's not to diminish how useful something like this might be,
> especially for folks that are sensitive to 'struct page' overhead or
> allocator performance.
> But, it will mostly be getting better performance at the _cost_ of
> consuming more RAM, not saving RAM.

Given that you don't remember the project at all, I can assume the
IBM-internal whitepaper where I described doing tail packing for small
files in the pagecache as part of an array of strategies to mitigate
internal fragmentation and latency issues expected to arise for large
values of PAGE_MMUSHIFT is also forgotten. I wrote some things I thought
worthwhile there and I was hoping you'd be able to at least corroborate
the one-time, now-former existence of the thing and maybe some
smatterings of its contents since it's no longer extant.

Or please do let me know if I've jogged your memory or whatever.


-- nyc

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Hillf Danton]

On Sun, 19 Jul 2026 02:51:08 +0200 Nadia Chambers wrote:
>On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
>> So a 64k page cache eats ~5GB of extra memory for a kernel tree (well,
>> _my_ kernel tree). In other words, if you are looking for memory savings
>> on my laptop, you'll need ~300GB of RAM before 'struct page' overhead
>> overwhelms the page cache bloat from a single kernel tree.
>> The whole kernel obviously isn't in the page cache all at the same time.
>> The page cache across the system is also obviously different than a
>> kernel tree, but you get the point.
>> That's not to diminish how useful something like this might be,
>> especially for folks that are sensitive to 'struct page' overhead or
>> allocator performance.
>> But, it will mostly be getting better performance at the _cost_ of
>> consuming more RAM, not saving RAM.
>
> Given that you don't remember the project at all, I can assume the
> IBM-internal whitepaper where I described doing tail packing for small
> files in the pagecache as part of an array of strategies to mitigate
> internal fragmentation and latency issues expected to arise for large
> values of PAGE_MMUSHIFT is also forgotten. I wrote some things I thought
> worthwhile there and I was hoping you'd be able to at least corroborate
> the one-time, now-former existence of the thing and maybe some
> smatterings of its contents since it's no longer extant.
>
I dont think this discussion could make forward progress without clearly
drawing the rule of thumb that which page size is supposed to match what
RAM size, with the supported page size put aside now, because I dont know
why 128K page is needed in the xbc/yz environment.

	RAM size	page size
	32G		4K
	512G		128K
	4T		2M

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Dave Hansen]

On 2/19/26 07:08, Kiryl Shutsemau wrote:
...
> The patchset is large:
> 
>  378 files changed, 3348 insertions(+), 3102 deletions(-)

A few notes about the diffstats:

$ git diff v6.17..HEAD arch/x86 | diffstat | tail -1
 105 files changed, 874 insertions(+), 843 deletions(-)
$ git diff v6.17..HEAD mm | diffstat | tail -1
 53 files changed, 1136 insertions(+), 1069 deletions(-)

The vast, vast majority of this seems to be the renames. Stuff like:

> -               new = round_down(new, PAGE_SIZE);
> +               new = round_down(new, PTE_SIZE);

or even less worrying:

> -int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
> +int set_direct_map_valid_noflush(struct page *page, unsigned numpages, bool valid);

That stuff obviously needs to be audited but it's far less concerning
than the logic changes.

So just for review sanity, if you go forward with this, I'd very much
appreciate a strong separation of the purely mechanical bits from any
logic changes.

> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> is greater than 4 KB, we need a way to pack multiple page tables into a
> single page. We could use the slab allocator for this, but it would
> require relocating the page-table metadata out of struct page.

Others mentioned this, but I think this essentially gates what you are
doing behind a full tree conversion over to ptdescs.

The most useful thing we can do with this series is look at it and
decide what _other_ things need to get done before the tree could
possibly go in that direction, like ptdesc or a the disambiguation
between PTE_SIZE and PG_SIZE that you've kicked off here.

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 09:30:36AM -0800, Dave Hansen wrote:
> On 2/19/26 07:08, Kiryl Shutsemau wrote:
> ...
> > The patchset is large:
> > 
> >  378 files changed, 3348 insertions(+), 3102 deletions(-)
> 
> A few notes about the diffstats:
> 
> $ git diff v6.17..HEAD arch/x86 | diffstat | tail -1
>  105 files changed, 874 insertions(+), 843 deletions(-)
> $ git diff v6.17..HEAD mm | diffstat | tail -1
>  53 files changed, 1136 insertions(+), 1069 deletions(-)
> 
> The vast, vast majority of this seems to be the renames. Stuff like:
> 
> > -               new = round_down(new, PAGE_SIZE);
> > +               new = round_down(new, PTE_SIZE);
> 
> or even less worrying:
> 
> > -int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
> > +int set_direct_map_valid_noflush(struct page *page, unsigned numpages, bool valid);
> 
> That stuff obviously needs to be audited but it's far less concerning
> than the logic changes.
> 
> So just for review sanity, if you go forward with this, I'd very much
> appreciate a strong separation of the purely mechanical bits from any
> logic changes.

That's the plan. That's the only way I can keep myself sane :P

> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
> 
> Others mentioned this, but I think this essentially gates what you are
> doing behind a full tree conversion over to ptdescs.

I have not followed ptdescs closely. Need to catch up.

For PoC, I will just waste full order-0 page for page table. Packing is
not required for correctness.

> The most useful thing we can do with this series is look at it and
> decide what _other_ things need to get done before the tree could
> possibly go in that direction, like ptdesc or a the disambiguation
> between PTE_SIZE and PG_SIZE that you've kicked off here.

Right.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Dave Hansen]

On 2/19/26 14:14, Kiryl Shutsemau wrote:
>> Others mentioned this, but I think this essentially gates what you are
>> doing behind a full tree conversion over to ptdescs.
> I have not followed ptdescs closely. Need to catch up.
> 
> For PoC, I will just waste full order-0 page for page table. Packing is
> not required for correctness.

Yeah, I guess padding it out is ugly but effective.

I was trying to figure out how it would apply to the KPTI pgd because we
just flip bit 12 to switch between user and kernel PGDs. But I guess the
8k of PGDs in the current allocation will fit fine in 128k, so it's
weird but functional.

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Matthew Wilcox]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> is greater than 4 KB, we need a way to pack multiple page tables into a
> single page. We could use the slab allocator for this, but it would
> require relocating the page-table metadata out of struct page.

Have you looked at the s390/ppc implementations (yes, they're different,
no, that sucks)?  slab seems like the wrong approach to me.

There's a third approach that I've never looked at which is to allocate
the larger size, then just use it for N consecutive entries.

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 05:47:22PM +0000, Matthew Wilcox wrote:
> On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > On x86, page tables are allocated from the buddy allocator and if PG_SIZE
> > is greater than 4 KB, we need a way to pack multiple page tables into a
> > single page. We could use the slab allocator for this, but it would
> > require relocating the page-table metadata out of struct page.
> 
> Have you looked at the s390/ppc implementations (yes, they're different,
> no, that sucks)? 

No, will check it out tomorrow.

> slab seems like the wrong approach to me.

I was the first thing that came to mind. I have not put much time into
it

> There's a third approach that I've never looked at which is to allocate
> the larger size, then just use it for N consecutive entries.

Yeah, that's a possible way. We would need to populate 16 page table
entries of the parent page table. But you don't need to care about
fragmentation within the page.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
>> On x86, page tables are allocated from the buddy allocator and if PG_SIZE
>> is greater than 4 KB, we need a way to pack multiple page tables into a
>> single page. We could use the slab allocator for this, but it would
>> require relocating the page-table metadata out of struct page.

Am Do, Feb 19, 2026 um 17:47:22 +0000, Matthew Wilcox schrieb:
> Have you looked at the s390/ppc implementations (yes, they're different,
> no, that sucks)?  slab seems like the wrong approach to me.

Yes, they both required fair amounts of debugging. I don't remember what
went wrong in s390 early boot. ppc64 mm/slub.c item-within-page indices
ended up notably overflowing 16-bit counters. PA-RISC had some issues
surrounding a race with setup or teardown of I think accounting data
structures for some IO bus among other things. SPARC had an interesting
issue because its page order growth increment / Sprungweite was 3 where
the PAGE_MMUSHIFT increment used in testing was 2. Something odd
happened with SHMLBA on a bunch of architectures and I can't remember
whether there were hardware factors involved or if it was just include
and config messes leaking PAGE_SIZE.


Am Do, Feb 19, 2026 um 17:47:22 +0000, Matthew Wilcox schrieb:
> There's a third approach that I've never looked at which is to allocate
> the larger size, then just use it for N consecutive entries.

Strangely, I think the mechanical assistant did that for MIPS and few or
no others. It was simpler in various ways, but I didn't like the space or
locking overheads. Given the chance, I'll likely redo it bes. if I ever
get to demo MIPS' 1 KiB PageGrain extension running high performance
with a near-VAX minimum MMU/TLB mapping granularity.

My energies are more likely to be directed elsewhere if anywhere at all
from what you're telling me, though.


-- nyc

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Laight]

On Thu, 19 Feb 2026 15:08:51 +0000
Kiryl Shutsemau <kas@kernel.org> wrote:

> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> 
> The kernel uses the same value - PAGE_SIZE - for two things:
> 
>   - the order-0 buddy allocation size;
> 
>   - the granularity of virtual address space mapping;

Also the 'random' buffers that are PAGE_SIZE rather than 4k.

I also wonder how is affects mmap of kernel memory and the alignement
of PCIe windows (etc).

	David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Fri, Feb 20, 2026 at 09:04:09AM +0000, David Laight wrote:
> On Thu, 19 Feb 2026 15:08:51 +0000
> Kiryl Shutsemau <kas@kernel.org> wrote:
> 
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >   - the order-0 buddy allocation size;
> > 
> >   - the granularity of virtual address space mapping;
> 
> Also the 'random' buffers that are PAGE_SIZE rather than 4k.

Yeah, in some places we use PAGE_SIZE just because without any reason.

> I also wonder how is affects mmap of kernel memory and the alignement
> of PCIe windows (etc).

Kernel, as userspace, is free to map memory PTE_SIZE granularity.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

On Thu, 19 Feb 2026 15:08:51 +0000 Kiryl Shutsemau <kas@kernel.org> wrote:
>> No, there's no new hardware (that I know of). I want to explore what
>> page size means.
>> The kernel uses the same value - PAGE_SIZE - for two things:
>>   - the order-0 buddy allocation size;
>>   - the granularity of virtual address space mapping;

Am Fr, Feb 20, 2026 um 09:04:09 +0000, David Laight schrieb:
> Also the 'random' buffers that are PAGE_SIZE rather than 4k.
> I also wonder how is affects mmap of kernel memory and the alignement
> of PCIe windows (etc).

It doesn't. Any adequate implementation has to sweep drivers and update
them to use MMUPAGE_SIZE or other address calculations that are still
correct for device memory, buffers need sizing so they work usw.

Yes, those things are there. Sweeping them is integral to implementing
page clustering, whether it was Hugh, me, Andrea or Kiryl.


-- nyc

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Matthew Wilcox]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> 
> The kernel uses the same value - PAGE_SIZE - for two things:
> 
>   - the order-0 buddy allocation size;
> 
>   - the granularity of virtual address space mapping;
> 
> I think we can benefit from separating these two meanings and allowing
> order-0 allocations to be larger than the virtual address space covered by a
> PTE entry.

I actually want to go in the other direction.  I once came up with a
name -- POTAM -- which stands for Power Of Two Allocator with Metadata.
The use case was something like XFS's buffer cache where we want a
filesystem block size of data (so 0.5KiB to 64KiB) with some metadata
attached (xfs_buf is 664 bytes with debugging enabled!)

I set this aside to work on folios, but folios offer a back door to
unifying this with the buddy allocator.  It's a long road, but here's
a sketch:

First, we separate memdescs from pages.  I believe this lets us shrink
struct page down to 8 bytes (previously presented as various LSFMMs).

Second, we get rid of 'page' in things like sglist and bvec.  This is
already in progress for various other good reasons.

Third (this bit is new), we replace memmap with something like a maple
tree.  That lets us lookup memdescs by physical address (typically
a memdesc will contain either the physical or virtual address of the
memory it controls).

Fourth, we change the unit of the lookup in the maple tree from being
a PFN to being address / 512 (or whatever size we want to use as our
minimum).

Now we can have memdescs for an arbitrary power of two which means we
can ditch all the awful code from ppc/s390 page table handling where
they try to share one memdesc between several different page tables.

It's going to be "fun" avoiding allocation deadlocks where we want to
rebalance the maple tree containing the memdescs ... that's a five year
away problem.

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Wed, Apr 29, 2026 at 03:39:18PM +0100, Matthew Wilcox wrote:
> On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> > No, there's no new hardware (that I know of). I want to explore what page size
> > means.
> > 
> > The kernel uses the same value - PAGE_SIZE - for two things:
> > 
> >   - the order-0 buddy allocation size;
> > 
> >   - the granularity of virtual address space mapping;
> > 
> > I think we can benefit from separating these two meanings and allowing
> > order-0 allocations to be larger than the virtual address space covered by a
> > PTE entry.
> 
> I actually want to go in the other direction.  I once came up with a
> name -- POTAM -- which stands for Power Of Two Allocator with Metadata.

... of the House Targaryen, the First of Her Name!

> The use case was something like XFS's buffer cache where we want a
> filesystem block size of data (so 0.5KiB to 64KiB) with some metadata
> attached (xfs_buf is 664 bytes with debugging enabled!)
> 
> I set this aside to work on folios, but folios offer a back door to
> unifying this with the buddy allocator.  It's a long road, but here's
> a sketch:
> 
> First, we separate memdescs from pages.  I believe this lets us shrink
> struct page down to 8 bytes (previously presented as various LSFMMs).
> 
> Second, we get rid of 'page' in things like sglist and bvec.  This is
> already in progress for various other good reasons.
> 
> Third (this bit is new), we replace memmap with something like a maple
> tree.  That lets us lookup memdescs by physical address (typically
> a memdesc will contain either the physical or virtual address of the
> memory it controls).
> 
> Fourth, we change the unit of the lookup in the maple tree from being
> a PFN to being address / 512 (or whatever size we want to use as our
> minimum).
> 
> Now we can have memdescs for an arbitrary power of two which means we
> can ditch all the awful code from ppc/s390 page table handling where
> they try to share one memdesc between several different page tables.

I had similar, but less ambitious idea. Can we get this functionality
from slab?

Maybe having a kind of kmem_cache that would allow to have metadata for
each allocated object. It will be backed by two slabs: one for actual
object and one for metadata. Plus some glue that allows to translate
object->metadata (not sure if reverse is required). If both object and
metadata are power-of-2 it should be doable: pointer to metadata in slab
page plus some math.

But I have not thought much about the idea yet.

Your idea is much bigger and I don't understand implications yet. It
seems redefining basis of memory allocation in kernel. Do we still have
page allocator? Where page allocator ends and slab begins?

But sounds fun to discuss it next week!

> It's going to be "fun" avoiding allocation deadlocks where we want to
> rebalance the maple tree containing the memdescs ... that's a five year
> away problem.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

> Fourth, we change the unit of the lookup in the maple tree from being
> a PFN to being address / 512 (or whatever size we want to use as our
> minimum).
> 
> Now we can have memdescs for an arbitrary power of two which means we
> can ditch all the awful code from ppc/s390 page table handling where
> they try to share one memdesc between several different page tables.

Having a memdesc describe a smaller piece than a page sounds like the way to
approach this cleanly. A lot of work, but sounds like the right long-term vision
also when it comes to having folios that are smaller than a page.

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Thu, Feb 19, 2026 at 03:08:58PM +0000, Kiryl Shutsemau wrote:
> == Status ==
> 
> I have a POC implementation on top of v6.17:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git pte_size

Just a heads up.

I've updated the tree. It is on top of v7.0 now.

I can boot my VM fully (not only to shell as before) on all three page
size 4k, 16k and 64k. And it doesn't crash immediately.

Page fault handler is the most challenging part of the design. I wanted
to flash it out to RFC quality before the summit, but it is not there
yet. But it started getting a shape.

Volume of the code changes makes it hard to deal with the patchset:

 1530 files changed, 12675 insertions(+), 8669 deletions(-)

Most of it mechanical, but still.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 5/1/26 20:00, Kiryl Shutsemau wrote:
> On Thu, Feb 19, 2026 at 03:08:58PM +0000, Kiryl Shutsemau wrote:
>> == Status ==
>>
>> I have a POC implementation on top of v6.17:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git pte_size
> 
> Just a heads up.
> 
> I've updated the tree. It is on top of v7.0 now.
> 
> I can boot my VM fully (not only to shell as before) on all three page
> size 4k, 16k and 64k. And it doesn't crash immediately.
> 
> Page fault handler is the most challenging part of the design. I wanted
> to flash it out to RFC quality before the summit, but it is not there
> yet. But it started getting a shape.
> 
> Volume of the code changes makes it hard to deal with the patchset:
> 
>  1530 files changed, 12675 insertions(+), 8669 deletions(-)

I think you know the answer already :P

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Kiryl Shutsemau]

On Fri, May 01, 2026 at 08:02:28PM +0200, David Hildenbrand (Arm) wrote:
> On 5/1/26 20:00, Kiryl Shutsemau wrote:
> > On Thu, Feb 19, 2026 at 03:08:58PM +0000, Kiryl Shutsemau wrote:
> >> == Status ==
> >>
> >> I have a POC implementation on top of v6.17:
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git pte_size
> > 
> > Just a heads up.
> > 
> > I've updated the tree. It is on top of v7.0 now.
> > 
> > I can boot my VM fully (not only to shell as before) on all three page
> > size 4k, 16k and 64k. And it doesn't crash immediately.
> > 
> > Page fault handler is the most challenging part of the design. I wanted
> > to flash it out to RFC quality before the summit, but it is not there
> > yet. But it started getting a shape.
> > 
> > Volume of the code changes makes it hard to deal with the patchset:
> > 
> >  1530 files changed, 12675 insertions(+), 8669 deletions(-)
> 
> I think you know the answer already :P

:/

Let me present the case at least. Most of this is cocci change and most
of + difference is tests.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[David Hildenbrand (Arm)]

On 5/1/26 20:12, Kiryl Shutsemau wrote:
> On Fri, May 01, 2026 at 08:02:28PM +0200, David Hildenbrand (Arm) wrote:
>> On 5/1/26 20:00, Kiryl Shutsemau wrote:
>>>
>>> Just a heads up.
>>>
>>> I've updated the tree. It is on top of v7.0 now.
>>>
>>> I can boot my VM fully (not only to shell as before) on all three page
>>> size 4k, 16k and 64k. And it doesn't crash immediately.
>>>
>>> Page fault handler is the most challenging part of the design. I wanted
>>> to flash it out to RFC quality before the summit, but it is not there
>>> yet. But it started getting a shape.
>>>
>>> Volume of the code changes makes it hard to deal with the patchset:
>>>
>>>  1530 files changed, 12675 insertions(+), 8669 deletions(-)
>>
>> I think you know the answer already :P
> 
> :/
> 
> Let me present the case at least. 

Oh, absolutely, looking forward to it!

-- 
Cheers,

David

Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers]

On Thu, Feb 19, 2026 at 03:08:51PM +0000, Kiryl Shutsemau wrote:
> No, there's no new hardware (that I know of). I want to explore what page size
> means.
> The kernel uses the same value - PAGE_SIZE - for two things:
>   - the order-0 buddy allocation size;
>   - the granularity of virtual address space mapping;
> I think we can benefit from separating these two meanings and allowing
> order-0 allocations to be larger than the virtual address space covered by a
> PTE entry.
> The main motivation is scalability. Managing memory on multi-terabyte
> machines in 4k is suboptimal, to say the least.

I apologise for my delay responding. I've been out of the loop a while.

As far as I know, the ABI compatibility algorithms are original to Hugh
Dickins from his 2001 larpage patch. I forward ported it in 2003 and
renamed it to „pgcl“ because „large pages“ had a major nameclash with
the „large pages“ of various superpaging efforts. Babaoğlu and Joy in
1979 introduced CLSIZE as part of their VM work on the VAX, though I've
heard suggestions it might have actually been McKusick or Karels, hence
the terminology „page clustering“ and the use of the abbreviation „pgcl“
from taking the „page“ abbreviation „pg“ from „NBPG“ and the „cluster“
abbreviation „cl“ from „CLSIZE“, as there is some lineage and priority
to their efforts, though they did not preserve the 512 B -aligned VAX
ABI (corrections from McKusick are also welcome, though this mailing
list might be a bit far afield for casually cc:'ing him).

The Linux parts of the superpaging efforts renamed superpages to „huge
pages“, though the well-known existing literature and code from e.g.
Talluri & Hill (1992) and Navarro, Iyer, Druschel & Cox (2001), where the
Cox of the second paper is even the Alan Cox known, in addition to other
things, for his extensive contributions to Linux. I'm unsure of the
origin of the nomenclature „huge pages“, but suspect Rohit Seth, from
whom I've not seen recent posts to Linux mailing lists, though like all
the other historical contributors, it would also be good to hear from
him in this thread. I strongly suspect the ABI compatibility algorithms
were original to Hugh and have cc:'d him in case he wants clarify the
origins or correct any of my commentary involving his work or otherwise
comment on things.

My 2003 code can be found at
https://mirrors.edge.kernel.org/pub/linux/kernel/people/wli/vm/pgcl/
I don't know of a link to Hugh Dickins' code from 2001, but it was
a couple of years earlier and far better-written than my forward ports or
attempts thereat of his code.

Scalability is worth being more specific about. Trimming a constant
factor is potentially worthwhile, but often one might want to go about
trying to make a difference in the computational complexity. In my 2003
IBM-internal whitepaper (dhansen had to have seen it, but it may not
have been memorable enough) and some commentaries on the subject since,
I've described it differently from scalability per se, varying by
whether the page size spectrum is dense or sparse:

	1. The sparse case is perhaps trivial, where the log of the
		clustering factor (your ilog2(PG_SIZE/PTE_SIZE) without
		a named macro for it that I know of, Hugh's
		PAGE_MMUSHIFT) gets shaved off the allocation order of
		the first nontrivial superpage/„huge page“ size, and
		maybe still-larger ones, but the small constant
		subtracted becomes progressively less significant as
		they continue their growth. I think there is
		doubly-exponential drop-off in availability as page
		orders increase, barring tactics to combat it.
		Empirically, orders up to maybe 2 or 3 aren't too bad,
		so choosing a PAGE_MMUSHIFT so the first nontrivial
		superpage size is order 2 or 3 can be reasonable, though
		with page size spectra as sparse as e.g. x86-64
		(„Sprungfaktor“ might be worth borrowing from German) of
		2**9 == 512, it's already 256 KiB, which might start
		raising zeroing latency and internal fragmentation of
		concerns, which can be algorithmically addressed via
		e.g. tail packing the pagecache into MMUPAGE_SIZE
		-aligned fragments for the perhaps common case of many
		small files and slab allocating many things now
		allocated as pages and more.

	2. The dense page size spectrum case can go about mapping
		individual PAGE_SIZE/PG_SIZE -sized allocation units
		with many different translation sizes. This provides a
		guarantee that so long as the size and alignment allow,
		the superpages/„huge pages“ up to PAGE_SIZE/PG_SIZE will
		always successfully be allocated without fear of
		external fragmentation. This guarantee was of particular
		industrial importance in the „forward-looking to
		64-bit“ aspects for proposing it as the solution to
		32-bit large memory's struct page array virtualspace
		footprint problems, in part because at the time „64-bit“
		meant IA64, which meant a dense page size spectrum with
		a Sprungfaktor of 4 like MIPS and the VHPT inverted
		pagetable, perhaps more like SPARC. (Here, XKVA, later
		done by Ingo as „4/4“, was the „32-bit hack“ to be
		avoided according to the advice I was given and the page
		clustering strategy was intended to be the
		forward-looking solution to the struct page array.) The
		interim since I worked on it has been long enough that
		priorities differ now.

These more specific motivations may have more force than „scalability.“

As far as completeness, in discussions about mechanical coding assistance
of a certain type, I used page clustering as an example of something
some of the algorithms might have trouble with. While trying on real x86
laptop hardware was late and not the subject of the majority of the
thought, it was able to get that up to running a GUI and starting up
some Electron apps in the start-up at login list before anything goes
wrong. The regression testing matrix in QEMU was 20 architectures with
PAGE_MMUSHIFT values of 0, 2, 4, and 6, for 80 cells, with the
regression tests being LTP and something called mm-stress, which is I
think mostly a script running various arguments for stress-ng. I don't
think QEMU cuts it and some sort of real hardware would be far better to
use if one can put together the multi-architecture testing lab with
terminal concentrators, serial consoles, and remote power controls etc.
Brewing up a kernel de novo as a vehicle for it all almost seemed like a
good idea at one point. The Coq, CBMC etc. models they produced are also
a bit weak for checking it all, and I was particularly dissatisfied with
their not directly representing user address spaces in the models, never
mind the concrete C (and I guess Rust is in the mix these days too)
rather often leaving code unswept right beside code it just swept, and
the stack top code looked off, too, and environments weren't being
inherited properly, but it seemed better to approach by extending the
verification than directly fixing it by hand. I had started trying to
add Dartagnan and some other Coq etc. packages in on it that are also
kernel-relevant before certain disruptions, but it got renamed to Dat3M
and some unusual Java things are ongoing with it that make it harder to
get interoperating the rest of the Coq-based tools. Perhaps the
inevitable and always obligatory human review has to expand to make up
for where formal modelling and the thus-far-not-entirely-well-grounded
machine reasoning systems cannot yet tread. While there's been something
of an interruption or disruption so that it's disorganised, the residue
or detritus of the experimentation with mechanically-generated code
ended up at git@git.sr.ht:~nadiayvette/linux-pgcl which is rather far
from a finished product, but if we're posting things that merely get to
a shell that doesn't crash immediately but does eventually, well, it's
of that kind.

In 2003 I thought it was by far the most fun project I ever worked on.
It would be great if I could contribute to efforts along these lines.


-- nyc