Re: Path forward for Virtualized Swap?

[Gregory Price <gourry@gourry.net>]

On Sat, Sep 26, 2026 at 04:44:07AM +0800, Kairui Song wrote:
> On Fri, Sep 25, 2026 at 12:11:55PM +0800, Gregory Price wrote:
> > 2) something has to pay the price of virtualization - period.
> > 
> >    No matter how we cut this issue, something in the stack is going
> >    to implement a method of virtualizing a swap entry.  Whether that's
> >    a virtualization layer that virtualizes every backend, or some kind
> >    of franken-swap device that lets a non-physical swap backend look
> >    like a physical swap back end without charging physical swap.
> > 
> >    If the history of virtualization has taught us anything it's that
> >    the overhead of a virtualization layer is *almost* never so extreme
> >    that it isn't worth the utility of the layer
> >    ( unless you virtualize the wrong things :] )
> 
> Hmmm... I mean there are tons and tons of projects trying to get
> rid of the overhead of virtualization, if you are referring to
> virtialization as a broad topic..
> 
> With careful design we can miminize the overhead for sure, and
> that's exactly why we need to do the design carefully :D
>

Oh absolutely, overhead is enemy number #1 - but we still deploy page
tables and virtual machines because the utility is just that high.

I suspect the level of virtualization we're talking about here, which
amounts to a single token representing many possible backends, is an
incredibly small overhead if designed correctly.

> > 
> > I struggle to think of a scenario that cannot be described
> > compositionally this way - and that is without adding any kind of
> > "tier" logic.
> 
> That sounds interesting, but I think this is already a mini
> version of swap tiering :D
>

Sort of.  The key difference is that the actual resource being consumed
has vastly differently levels of availability and at least an order of
magnitude (if not 2-3) difference in latency.  So while the tails can be
very bad, the P50s are generally not that severe.

We do this same thing in memory tiering - we treat socket-hosted memory
as the same tier, simply because it's not worth differentiating.

But we still prefer locality within that tier.  So i could imagine
memory-backed swap tiers having locality that is separately grouped
from storage-backed swap tiers.

zsmalloc actually encodes a mild version of this - preferring to
compress onto the node the folio is being swapped out from 

https://lore.kernel.org/linux-mm/20250402204416.3435994-1-nphamcs@gmail.com/

So in fact zsmalloc already has a tiering concept built-in.

> And not sure if the implementation will looks odd, e.g. like if you
> adjust that midway, you get different slot of different compression
> rates.?
> 

It is not an intuitive set of knobs, but then i also don't think it's
particularly complicated math.  But yes, there are some odd looking
configs.

> That data is very helpful info! But it could be very hardware dependent,
> especially with certain mitigations and hardwares, the faulting
> process may slowdown by a lot. This is the right picture for most
> cases, and for another example on one of our certain instance of VM, I
> observed this (high pressure, batch fault):
> 
>                     component                      µs/page
> page fault handling (directly calibrated)          ~1.92
> swap machinery (slot, swap cache, rmap, zram, etc) ~1.04
> lzo decompression                                  ~0.32
> 
> Kind of crazy that decompression only take a very minor part
> of it.  While storage based SWAP goes through SPDK so it changes
> dramatically according to the host, setup or spec.
>
> BTW CRAM looks really nice from the data of yours :) !
>

Funny enough this is exactly the data needed to argue that CRAM should
not be a swap backend - you're offloading the cheapest part of the
process.

The actual win comes from allowing the compressed memory to stay mapped
while Read-only, which means you don't have faults.  I'll call this
"ghost-pressure" - the data is in-memory, but you have to take a fault
because it's software-compressed.  Letting it remain mapped R/O solves
the entire fault/swap machinery piece.

We cannot safely map R/W however, as you could have catestrophic loss of
compression ratio.  (I digress, but hopefully we can chat more at LPC)

> > That shows up as stalls and PSI pretty quickly and can be dealt with
> > by monitoring - but I can see the desire to just have them OOM.
> 
> Right, monitoring is less effective from somehting that is directly
> in the kernel.
> 

Well.... I think this is probably an organizational philosophy.

We are pretty solidly effective with PSI + proactive reclaim keeping
our machines chugging along nicely.  I think this is where a lot of
confusion is coming from - because the PSI data is very effective.

> > > The second one is a policy change,
> > 
> > Disagree.
> > 
> > If zswap does not charge a physical slot, then it is correct to stop
> > charging the counter based on the historic definition - it just was
> 
> Hmm, but the historical defination is the logical entry, no?
>

I think this is the entire debate, right?

Historically I think it's clear that it is a physical slot, from the
time it was proposed it was talked about in terms of consumption, and
Johannes' original definition even said "physical" (for some reason
this did not make it into the docs).

The zswap-charging-swap was a function of zswap reserving a writeback
slot - which means it *was* consuming a physical slot and it was being
accounted accordingly.

If zswap no longer reserves a writeback slot by default, it's not being
consumed, and so not charging it is correct.

But if you read the docs as swap being a logical counter as opposed to a
physical counter, without the context of zswap reserving writeback, then
I can understand how we got here.

The question is how do we move foward.  Continuing to charge swap breaks
us, but not charging swap breaks you.  At the end of the day, someone is
changing the counter they're using (or methodology).

> > I would push back on tiering - this seems like more complexity than the
> > requirement warrants.  We may find memory-backed and storage-backed swap
> > want different solutions in the "tiering" worlds.
> 
> Hmm, I mean in general swap are page fault (swap entry) based, that makes
> them similiar and that's the common point?
>

As above, I think even in memory-tiering we have this separation of
reasoning about different logical tiers of memory (VRAM, DRAM, CXL) but
among those tiers we do not differentiation except maybe based on
locality of accessor (socket0 prefers cxl on its own CXL complex over
cxl on socket1's complex to avoid the extra NUMA hop).

I can see there being a differentiation here as well (memory backed vs
storage backed).

> > 
> > e.g. it may never make sense to give zswap/zram/xswap a lower priority
> > than a physical swap layer, and you may never really want to compose
> > them with one another, so why does it need to be considered?
> 
> Making ZRAM have a lower priority could make sense... Well I haven't
> actually see it being used that way, but first there are users of
> multiple ZRAMs indeed, and e.g. if you put offline batch tasks on the
> same machine with high priority tasks, and the offline batch task only
> want to finish their work, however slow is fine, and the user want to
> have lower impact on CPU usage from them, with a limited storage space.
> Well I have to say this looks odd and just a theory, but we have see
> too many bizzard usage of the Linux kernel, there are even people
> using deprecated GPU's VRAM as SWAP for CPU's RAM, and hey they
> still make sense, you can easily found popular community projects
> based on that.

This is what I mean by I would push back on a tiering solution.  We
haven't even figured this out for memory alone - let alone a component
that composes both memory and storage.

What's the chance we get it right?  I think unlikely.

~Gregory