Re: Improve page refcount scalability

[Pedro Falcato <pfalcato@suse.de>]

On Tue, Oct 13, 2026 at 06:01:15PM -0700, Linus Torvalds wrote:
> On Tue, 13 Oct 2026 at 17:10, Harry Yoo <harry@kernel.org> wrote:
> >
> > There should be a limitation on how much memory can be submitted to RCU
> > by the page cache, otherwise will be a corner case where we have too
> > much unreclaimable memory waiting for the RCU subsystem to free them.
> 
> There is. In fact, I tried it a few different ways... It's not quite
> what you suggest, but try it out.
> 
> > For the same reason, when a process tries to create RCU-delay-freed
> > pages much faster than RCU can free memory, the kernel needs to limit
> > that.
> 
> That was actually my initial approach - making the places that set the
> PG_rcu_delay bit throttle themselves when there were lots of such
> pages in flight.
> 
> Except I worried about slowing down simple things like truncating big
> files: particularly since I believe firmly that slowing that down is
> the *wrong* thing to do from a memory pressure standpoint, because
> you'd be slowing down the very thing that is actively freeing pages -
> it's just that it then takes a (short) while for those pages to get to
> a usable state.
> 
> So what I did instead was to actively then expedite RCU when there
> were a lot of pages in flight.

Right, this makes sense.

> 
> That worked quite well, but it hit a problem when you had lots of CPUs
> doing it: since an RCU quiet period needs to happen on *each* CPU, you
> still end up wanting to at some point wait for things: you need to
> synchronize not _one_ thread freeing things, but synchronize across
> all the CPU's.
> 
> But again: the good part really is that this "lots and lots of pages
> in flight" only happens when there are literally lots and lots of
> pages that are very much about to be free'd. So you are in a fairly
> good position when it happens, you just need to *understand* the
> position you are in.
> 
> So the other part is to come at the issue from the other end: when you
> have trouble allocating pages, but see that there are a ton of pages
> in flight due to PG_rcu_free, you basically do the same thing from the
> other end, and do synchronize_rcu_expedited().
> 
> And in testing, that worked really well under even insane loads where
> concurrent hole punching or truncate calls would generate gigabytes of
> PG_rcu_free activity each second.
> 
> Now, it turns out that Kiryl also suggested another approach to the
> pages becoming PG_rcu_free in the first place: instead of being able
> to easily trigger gigabytes of such free pages by just truncating a
> large file - which marked all those pages PG_rcu_free whether that was
> useful or not - the current version actually comes at *that* the other
> way too: mark only pages that are hot and are *worth* the PG_rcu_free
> treatment (because they get read over and over again) with PG_rcu_free
> in the first place, so that you don't generate that flood of
> RCU-free'd pages in the first place, and now the PG_rcu_free flood is
> in a way throttled by the *use* of said pages.
> 
> So there's effectively three different things working together:
> 
>  - mark a folio PG_rcu_free only when it's hot and keeps being accessed
> 
>    (See commits dabc522cee5b: "mm: have folio_mark_accessed() return
> whether the folio is hot" and  bb5908b733a1: "mm/filemap: set
> PG_rcu_free on the folios that reads keep coming back to" for details
> - both are pretty tiny and not really all that complicated)

I'm not sure this is a good criteria for checking whether a folio should
be RCU-ified. Namely:
- you don't need that many accesses to get an active, referenced folio. LRU
batching makes it a little harder, but not by much.
- rewinding along a folio doesn't tell you much other than the reader
went back, for some reason. It's also natively racy so funny things like
multiple pread()'s to the same fd can easily confuse this mechanism into
marking this as hot.

How many pages (as a %) are you seeing as marked PG_rcu_free across a boot/
realistic workload?

> 
>  - proactively expedite RCU grace periods when there are a lot of of
> pages pending
> 
>    (See commit 2ce8df4627fe: "mm: expedite the grace period when a CPU
> has many deferred frees")
> 
>  - teach the page allocator about the outstanding pages ratehr that failing
> 
>    (See commit 82fc13ff3a1e mm: "count the pages waiting for a grace
> period and free them before OOM")
> 
> and it really does seem to work quite smoothly. But it's not like I
> can guarantee it's perfect.
> 
> All those commits are fairly trivial. Of the ones listed above, that
> "count the pages waiting for a grace period and free them before OOM"
> is the biggest by far, but it's not really particularly complex or
> subtle.
> 
> The actual biggest patches in the series are:
> 
>  - 65401f4102b9 ("mm/filemap: read from the page cache without folio
> references")
> 
>    This really is not very big or complicated, although it *does* have
> subtle ordering issues with i_size as pointed out by Kiryl. It's
> mostly really just a simplified version of my patch from two and a
> half years ago. And it's simplified because now the RCU freeing makes
> everything more natural.
> 
>  - df0b133495c9 ("mm: free PG_rcu_free folios only after an RCU grace period")
> 
>    This is the one that actually runs on the page freeing side and
> adds a page that is marked PG_rcu_free to a list and sets up the whole
> RCU machinery (ok, there are actually two per-cpu lists: the 'pending'
> list and the 'batch' list, and a RCU grace period will free the
> current batch and move the pending list to be the next batch - not
> rocket science).
> 
> and even those "big" patches aren't really all that big. Both are less
> than 200 lines of code, and while I wouldn't call either of them
> _trivial_, both are very much doing one clear thing and there's
> nothing really *strange* about them either.
> 
> So it's ten patches, none of them particularly hairy. I really would
> encourage you to just take a look and take it for a try.
> 
> Now, I *will* say that obviously RCU-freeing pages is an absolutely
> conceptually *enormous* change to the VM. But it's in no way at all a
> new thing: we've been doing the same thing for kmalloc() for decades.

Well, yes, but generally you have a lot more memory pinned as page cache
than kmalloc. In my current laptop (fairly standard desktop system) that
ratio is about 20x (more page cache than slab).

In any case, I was fairly interested in the problem back then, but I don't
think a solution that may fundamentally alter the VM system is worth it. At
least until we can find an actual real-world use case where performance
substancially changes (i.e not a microbenchmark). Until then, these patches
look to be more trouble than they're worth.

Regarding the RCU'ing, I'm wondering if we could find a similar system where
some type of RCU stability exists, but without delaying the actual frees to
RCU. Like TYPESAFE_BY_RCU, but for folios (where they keep their order and
some other interesting properties like direct map status, (un)acceptability)
such that the old solution would be more palatable. In any case, talk is cheap,
and I definitely have no code :)

(or, and this is another random idea that is currently not feasible due to
$raisins, but the folio refcount could be sharded per-cpu or so; this should
avoid most contention at the expense of the (relatively) rare truncate/reclaim
path)

-- 
Pedro