Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers <nadia.yvette.chambers@ik.me>]

On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> First of all, this looks like fun. Nice work! I'm not opposed at all in
> concept to cleaning up things and doing the logical separation you
> described to split buddy granularity and mapping granularity. That seems
> like a worthy endeavor and some of the union/#define tricks look like a
> likely viable way to do it incrementally.

My 2003 pgcl code was the funnest kernel hacking I've ever done. You
were there (or at least involved; I don't remember in enough detail to
even be sure if we were together in person) when it got the first public
boot on 32-bit x86 with 64 GiB RAM, and even noted that my earlier work
on shrinking the size of struct page had got things booting with
ridiculously little lowmem left over after mem_map when you were
flagging me down for 16 nodes being available to be glued together.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> But I don't think there's going to be a lot of memory savings in the
> end. Maybe this would bring the mem= hyperscalers back into the fold and
> have them actually start using 'struct page' again for their VM memory.
> Dunno.

For 32-bit it was only really an issue because it was lowmem. From day
one back in 2003 it was always a matter of trading off internal for
external fragmentation, and there was even some worry about losses from
internal fragmentation potentially exhausting lowmem under load. It's in
a way like being able to adjust the frontier between internal and
external fragmentation more than really saving memory at all. Apart from
lowmem exhaustion on 32-bit and maybe being so large and heavily used
that it competes with userspace for cache and TLB, the struct page
array's memory consumption is mostly irrelevant.

strictly speaking, struct page as presently conceived isn't necessary at
all. There are mostly microkernels out there that do no per-page memory
bookkeeping. The basic pattern is that the commitment to what kind of
bookkeeping to do is mostly deferred until it gets allocated and put in
use, at which point things like slab memory, pagecache memory,
pagetables and such all have different ways of tracking everything. The
basic allocators can do a variety of things, too. In-band metadata IOW
using the memory the allocator is saying is technically free to hold
metadata is one of the tricks, adaptively switching between radix tree
or bitmapped things, using address calculation to only need things like
a pair of bits for each radix tree node's entries, and maybe even bit
twiddling lockfree code etc. all start getting in the mix like LLFree:

https://www.usenix.org/conference/atc23/presentation/wrenger
https://www.sra.uni-hannover.de/Publications/2023/llfree-atc23/
https://github.com/luhsra/llfree-c

You still get a couple of bits per page, but the delayed metadata
allocation getting less internal fragmentation(!) from forcing too many
disparate metadata types into a union isn't saving anywhere near as much
as the disappearance of the centralised, readily-identifiable struct
page array might suggest, so there's more than enough left over to
swamp those bits. Having slab, pagecache, pagetable etc. metadata all
unmixed with each other still means they have significant metadata
footprints that are significant fractions of what a struct page array
would be. The real advantage for memory hyperscalars is that splitting
them those ways enables more locality of reference.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> But, let's look at my kernel directory and round the file sizes up to
> 4k, 16k and 64k:
> find .  -printf '%s\n' | while read size; do echo	\
> 		$(((size + 0x0fff) & 0xfffff000))	\
> 		$(((size + 0x3fff) & 0xffffc000))	\
> 		$(((size + 0xffff) & 0xffff0000));
> done
> ... and add them all up:
> 11,297,648 KB - on disk
> 11,297,712 KB - in a 4k page cache
> 12,223,488 KB - in a 16k page cache
> 16,623,296 KB - in a 64k page cache

4 KiB being so close to disc is surprising esp. since disc is usually
doing tail packing atop 512 B blocks. The rest is unsurprising.


On Thu, Feb 19, 2026 at 09:08:57AM -0800, Dave Hansen wrote:
> So a 64k page cache eats ~5GB of extra memory for a kernel tree (well,
> _my_ kernel tree). In other words, if you are looking for memory savings
> on my laptop, you'll need ~300GB of RAM before 'struct page' overhead
> overwhelms the page cache bloat from a single kernel tree.
> The whole kernel obviously isn't in the page cache all at the same time.
> The page cache across the system is also obviously different than a
> kernel tree, but you get the point.
> That's not to diminish how useful something like this might be,
> especially for folks that are sensitive to 'struct page' overhead or
> allocator performance.
> But, it will mostly be getting better performance at the _cost_ of
> consuming more RAM, not saving RAM.

Given that you don't remember the project at all, I can assume the
IBM-internal whitepaper where I described doing tail packing for small
files in the pagecache as part of an array of strategies to mitigate
internal fragmentation and latency issues expected to arise for large
values of PAGE_MMUSHIFT is also forgotten. I wrote some things I thought
worthwhile there and I was hoping you'd be able to at least corroborate
the one-time, now-former existence of the thing and maybe some
smatterings of its contents since it's no longer extant.

Or please do let me know if I've jogged your memory or whatever.


-- nyc