Re: Path forward for Virtualized Swap?

[Kairui Song <ryncsn@gmail.com>]

On Fri, Sep 25, 2026 at 12:11:55PM +0800, Gregory Price wrote:
> On Fri, Sep 25, 2026 at 09:15:17PM +0800, Kairui Song wrote:
> > 
> 
> First, thank you for the very nice email, this provides a lot of
> missing context to the discussion and clarifies the issue.
> 
> I really appreciate you taking the time to spell it out.
> 
> I've given it a read thrice over to make sure i haven't missed,
> so please don't take my trimming for skipping something, if you feel
> I missed something important let me know.

Thanks for taking the time reading this :)

> 
> > 
> > First a little bit off topic, I'll be really happy if we can make
> > both compressed memory and swap as separate counters (I even once
> > tried to implement a zpool accounting to account compressed memory
> > in some unified way, but, well, zpool got killed before I post
> > that :P), or at least a way to do that, e.g. something like nokmem.
> > Due to our real usage:
> > 
> 
> I think we all want the same thing, just in mildly different shapes.
> I really see this breaking into two core issues:
> 
> 1) memory.swap is overloaded, and we disagree on its semantics
> 
>    I think there is a strong use case for both detaching zswap/swap
>    and for retaining swap as a logical limit.  You lay it out fairly
>    well here that you are replacing a memsw SLO mechanism with the
>    counters available in memcg, and I think we need to give this
>    consideration as something more than simply luck.

Yeah there are still effort trying to push memsw into V2 upstreamly,
I even got a suggestion to hold a topic on that last year at LSFMM
but didn't find enough time :P

> 2) something has to pay the price of virtualization - period.
> 
>    No matter how we cut this issue, something in the stack is going
>    to implement a method of virtualizing a swap entry.  Whether that's
>    a virtualization layer that virtualizes every backend, or some kind
>    of franken-swap device that lets a non-physical swap backend look
>    like a physical swap back end without charging physical swap.
> 
>    If the history of virtualization has taught us anything it's that
>    the overhead of a virtualization layer is *almost* never so extreme
>    that it isn't worth the utility of the layer
>    ( unless you virtualize the wrong things :] )

Hmmm... I mean there are tons and tons of projects trying to get
rid of the overhead of virtualization, if you are referring to
virtialization as a broad topic..

With careful design we can miminize the overhead for sure, and
that's exactly why we need to do the design carefully :D

> 
> I also think these problems are mildly orthogonal.
> 
> We can solve the memory.swap problem without having to solve the
> virtualization problem at the same time.
> 
> The virtualization issue is much more compositionally complex - we
> need to account not only for ordering (tiers, priorities, etc), but
> also writeback mechanics and migrations.
> 
> Quoting roman:
> 
>  > I don't think it's reasonable to add a new interface, but having a
>  > patch/config option or even a mount option which changes the semantics
>  > of memory.swap.max to the v1-like behavior should be ok.
> 
> It seems we're having the same discussion.  We have users who want
> them decoupled, and users who want them tied together. I think boot
> or build option that switches this behavior is the worst of both
> worlds when the counters compose cleanly:
> 
>    memory.max = max physical ram you can use
>    zswap.max  = max physical ram zswap can use
>    pswap.max  = max physical storage swap can use
>    swap.max   = max logical swap storage a container can use
> 
> > But with compression as a fixed part in memory.current, first the
> > compression rate is totally uncontrollable, both the user and us
> > will be fully *unaware* of how much memory they can *actually* use,
> > that makes the planning really awkward. memory.max stops being the
> > bound of what we planned or sold, anything compressed lets the raw
> > footprint go past it by however much the compression ratio happens
> > to give, so what we oversold is bounded by the workload's data and
> > not by anything we configure. We can substract the zswap reading
> > though with adaption, however it's hard to change the performance,
> > OOM behavior or reclaim behavior:
> > 
> 
> I do in fact this is solved by splitting the counters.
> 
> Consider:
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 0
>   swap.max   = 2GB
> 
> This looks silly on its face - but it's not. This says limit total
> swappable capacity to 25% of ram, and you can only offload 25% of
> your total logical memory sapce - i don't care how much it compresses.
> 
> You could offload 2GB of anon at whatever compression ratio you get
> and fill the dead space with page cache.  There's utility.
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 0
>   swap.max   = 4GB
> 
> Limit overall compression ratio to 2:1 and 25% of memory size.  If you
> happen to compress more, you can eat some page cache, but you're never
> getting more than 4GB of anon swapped.
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 0
>   swap.max   = 8GB
> 
> Limit compression ratio to 4:1 and 25% of memory size.  Same as above,
> you can fill the slack space with page cache.
> 
>   memory.max = 8GB
>   zswap.max  = 0
>   pswap.max  = 4GB
>   swap.max   = 4GB
> 
> Limit logical and physical swap to 4GB
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 4GB
>   swap.max   = 8GB
> 
> Limit zswap to 25% of memory - maximum of 4:1 compression ratio
> if you don't hit that ratio use up to 4GB of physical swap.
> if compression hits 2:1 and you can use all of your physical swap.
> but you cannot swap out more than 8GB of logical memory
> 
>   memory.max = 8GB
>   zswap.max  = 4GB
>   pswap.max  = 2GB
>   swap.max   = 8GB
> 
> Limit zswap to 50% of memory, and a 2:1 compression ratio overall, but
> if you don't hit that ratio use up to 2GB of physical swap.
> 
> In all of these scenarios swap.max is exact same counter it has been for
> you, and what we get out of this is:
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 8GB
>   swap.max   = max
> 
> I don't care how much you compress, you're limited to 2GB of ram, have
> at it - and feel free to use another 8GB of storage. If your PSI gets
> too high we'll do you a favor and OOM kill you so you don't spin
> endlessly churning your swap.
> 
>   memory.max = 8GB
>   zswap.max  = 2GB
>   pswap.max  = 0
>   swap.max   = max
> 
> Only zswap up to 25% of your memory, i don't care how much to squeeze.
> 
> I struggle to think of a scenario that cannot be described
> compositionally this way - and that is without adding any kind of
> "tier" logic.

That sounds interesting, but I think this is already a mini
version of swap tiering :D

And not sure if the implementation will looks odd, e.g. like if you
adjust that midway, you get different slot of different compression
rates.?

> > And if the memory compression is really fully transparent (not
> > doable by software), yeah, that's great as there is nothing to do
> > with reclaim.  But, for now, we have to go through page fault / folio
> > allocation / map it again.
> 
> Transparent compression creates a completely different accounting
> problem.  You lose ALL ability to see the per-page and per-memcg
> compression ratio, and as a result you can have 1 container jam
> 300 billion zero pages into 1 page, and another container spew
> /dev/random to its memory - the result is the 300-billion zero-page
> consumer is the one that will get killed for over-consumption.
> 
> Unless hardware provides a way to attach a token to a particular page
> (either by out of band reporting or some architecture extension that
> reports this as it is written), this problem won't get solved.
> 
> Until then, don't count on having anything more than a logical-view
> of offloaded compressed ram consumption.

Right, thanks for the info, I don't have any transparent compression
hardware so not aware of how that part works, the software based
one is being very commonly used though.

> 
> (this one of a few reasons it should not be serviced by the swap
>  subsystem, but I digress).
> 
> > So For example, if we already have
> > memory.max == memory.current or under high pressure, then now
> > doing any read from the compressed part would need to some
> > require further eviction first to make place for the decompressed
> > new data, this is not like any kind of "real" memory, something
> > feels not right here.
> >
> 
> This is in fact how numa balancing works on tiered systems.  You eat a
> fault to migrate a page to the socket, and with tiered counters you are
> required to drive reclaim to get that promotion. (Presently this isn't
> true of global pressure, migrations will happily fail).
> 
> In practice, we find this is acceptable but could be better if we
> allowed the promotions to occur asynchronously outside the fault path,
> but this has other trade offs (loss of locality information).
> 
> Anyway, this is exactly like real memory, just a bit more stringent
> (a "Swap migration" cannot be allowed to fail, while a promotion can).
> 
> I have also experimented with this particular issue in compressed-ram
> forcing promote-on-write instead of letting the hardware-compressed
> memory be written to uncontended (which is a RAS nightmare).
> 
> In practice, if you are decent about proactively reclaiming your
> workloads, or if fairness is enforced across workloads, this kind of
> issue is not as big an issue as it looks. Fault time can be minimized.
> 
> Some partially simulated data:
> 
>                       Cost of a page fault
> backend                   p50       p90       p99	max
> disk swap-in (NVMe)       59 µs     70 µs    145 µs     1.63 ms
> swap-cache hit          1.13 µs   1.15 µs   1.25 µs     79.9 µs
> zswap-in (zstd)           26 µs     31 µs     44 µs     0.27 ms
> cram write-fault        2.10 µs   2.24 µs   2.43 µs       70 µs
>   (promotion)
> cram read               0.13 µs   0.14 µs   0.14 µs     14.7 µs
>   (read-in-place)

...

> 
> Generally I disagree but the data above shows you are technically
> correct for extreme outliars (zswap-in max vs disk swap-in p50).
> 

That data is very helpful info! But it could be very hardware dependent,
especially with certain mitigations and hardwares, the faulting
process may slowdown by a lot. This is the right picture for most
cases, and for another example on one of our certain instance of VM, I
observed this (high pressure, batch fault):

                    component                      µs/page
page fault handling (directly calibrated)          ~1.92
swap machinery (slot, swap cache, rmap, zram, etc) ~1.04
lzo decompression                                  ~0.32

Kind of crazy that decompression only take a very minor part
of it.  While storage based SWAP goes through SPDK so it changes
dramatically according to the host, setup or spec.

BTW CRAM looks really nice from the data of yours :) !

> But I can understand why the point here is that you might want to limit
> the total amount of swappable memory, regardless of where it ends up.
> 
> The real issue is when your hot (or warm) memory exceeds your real
> memory limit - at which point you start entering a churn cycle where
> every swap-in swaps out a hot page.
> 
> That shows up as stalls and PSI pretty quickly and can be dealt with
> by monitoring - but I can see the desire to just have them OOM.

Right, monitoring is less effective from somehting that is directly
in the kernel.

> 
> > I also want to separate two things that I think got bundled together
> > here: not requiring a physical slot behind a compressed entry, and not
> > charging the raw size to the swap counter. The first one is great, yeah,
> > and it's exactly the part we want, it's what makes compression usable
> > without provisioning disk.
> 
> agree
> 
> > The second one is a policy change,
> 
> Disagree.
> 
> If zswap does not charge a physical slot, then it is correct to stop
> charging the counter based on the historic definition - it just was

Hmm, but the historical defination is the logical entry, no?

> lucky happenstance that the swap counter and logical accounting matched
> up until now.
> 
> That said, I think there's a strong argument that breaking that coupling
> may in fact break users - but the answer may not be to do nothing, but
> to simply give physical swap its own counter and begrudingly change the
> contract.
> 
> I can hear Johannes rolling his eyes at me from 3 states away :]
> 
> > maybe it's not needed for the first stage, charging a cgroup for the
> > memories it has offloaded doesn't require any slot to exist behind them.
> > If someone wants to run memory compression with no disk at all,
> > memory.swap.max defaults to max, so that still works fine, right?
> >
> 
> It becomes confusing, because we also want to use the swap counter to
> control physical consumption.  To run with only zswap w/ swap.max you
> must have no physical disk limit.
> 
> It is the exact opposite issue of yours.
> 
> > And I'm not asking the compression layer to have an opinion on how
> > much compression or ratio is too much, just think we need a sane limit
> > for container schedulers to sets on top of it.
> >
> 
> I don't think it's unreasonable.  This just seems like a distraction if
> we can come to a conclusion where both scenarios can be composed.
> 
> > memory.max: control the raw usage of application.
> > memory.swap.current/max: controls the offloaded size/limit of a application.
> > With (memory.max + memory.swap.max) <= planned usage (and this is memsw).
> > 
> > We can keep the compressed part in memory.current of course, as already
> > did in upstream with ZSWAP, and things can be further adapted, we
> > still have:
> > 
> > memory.max: control the real usage of application.
> > memory.swap.current/max: controls the offloaded size/limit of a application.
> > With (memory.max - <compress memory> + memory.swap.max) <= planned usage.
> > 
> 
> This makes me wonder whether a pswap counter functionally deprecates
> zram, since the hard memory consumption limit becomes composable
> (putting aside the implementation details).
> 
> i.e. is there a zram scenario which cannot be composed with something
> like zswap/pswap/swap
> 
> > Just for reference. Maybe a seperate counter, tiering, is a better idea
> > than changing the swap counter?
> 
> I would push back on tiering - this seems like more complexity than the
> requirement warrants.  We may find memory-backed and storage-backed swap
> want different solutions in the "tiering" worlds.

Hmm, I mean in general swap are page fault (swap entry) based, that makes
them similiar and that's the common point?

> 
> e.g. it may never make sense to give zswap/zram/xswap a lower priority
> than a physical swap layer, and you may never really want to compose
> them with one another, so why does it need to be considered?

Making ZRAM have a lower priority could make sense... Well I haven't
actually see it being used that way, but first there are users of
multiple ZRAMs indeed, and e.g. if you put offline batch tasks on the
same machine with high priority tasks, and the offline batch task only
want to finish their work, however slow is fine, and the user want to
have lower impact on CPU usage from them, with a limited storage space.
Well I have to say this looks odd and just a theory, but we have see
too many bizzard usage of the Linux kernel, there are even people
using deprecated GPU's VRAM as SWAP for CPU's RAM, and hey they
still make sense, you can easily found popular community projects
based on that.