From: nakixii <nakixii@gmail.com>
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org,
Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Chris Li <chrisl@kernel.org>, Kairui Song <kasong@tencent.com>,
Nhat Pham <nphamcs@gmail.com>, Yosry Ahmed <yosry@kernel.org>,
Minchan Kim <minchan@kernel.org>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Barry Song <baohua@kernel.org>, Hugh Dickins <hughd@google.com>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Mike Rapoport <rppt@kernel.org>, Peter Xu <peterx@redhat.com>,
Suren Baghdasaryan <surenb@google.com>
Subject: [RFC PATCH 06/16] mm: memcomp: store shmem and tmpfs pages
Date: Mon, 16 Nov 2026 19:01:12 +0800 [thread overview]
Message-ID: <20261116110122.309748-7-nakixii@gmail.com> (raw)
In-Reply-To: <20261116110122.309748-1-nakixii@gmail.com>
Anonymous token PTEs do not cover shmem contents held by a page-cache
mapping. Shmem needs an exceptional entry that survives eviction and can
be replaced atomically when the contents are faulted back.
Store eligible single-page shmem folios in memcomp and place their
tokens in the page-cache xarray. Reserve per-key slot storage with cache
ownership distinct from an anonymous producer's resident-record
ownership. Prepare payload and accounting before activating the slot
references that the published shadow will own.
Keep a lookup reference while allocating, charging and decompressing a
refault folio. Confirm that the expected shadow remains installed before
committing the page-cache replacement. Only the winning replacement
consumes the cache owner; an aborted load retains it. Balance inode
accounting, memory charges and temporary references on every exit.
Publish a zero shadow's owner count only when the expected xarray
replacement succeeds. Keep that cache-owner ledger separate from
anonymous zero PTEs, and do not allocate slot metadata for a zero
shadow.
Retain ordinary shmem swap as a fallback. A successful in-memory store
needs no swap device, and truncate or swapin races can retry without
publishing contents from a stale shadow.
Signed-off-by: nakixii <nakixii@gmail.com>
---
include/linux/memcomp.h | 80 +++++++++++++
mm/Kconfig | 3 +-
mm/memcomp.c | 46 ++++++++
mm/memcomp/api.h | 34 ++++++
mm/memcomp/cache_folio.h | 70 ++++++++++++
mm/memcomp/cache_store.h | 70 ++++++++++++
mm/memcomp/consumers.h | 55 +++++++++
mm/shmem.c | 238 +++++++++++++++++++++++++++++++++++++++
8 files changed, 595 insertions(+), 1 deletion(-)
create mode 100644 mm/memcomp/cache_folio.h
create mode 100644 mm/memcomp/cache_store.h
diff --git a/include/linux/memcomp.h b/include/linux/memcomp.h
index 20b553d12151f7bc0202c25d52e4f6832c6ae0b6..20e81dee9062b97274910a1ed3451dd0089ce235 100644
--- a/include/linux/memcomp.h
+++ b/include/linux/memcomp.h
@@ -103,6 +103,9 @@ void memcomp_entry_resident_put(struct memcomp_ref *ref);
/* Consumes and clears the held guard on every return. */
int memcomp_restore_resident(struct vm_fault *vmf, struct memcomp_ref *ref);
void memcomp_invalidate(unsigned long token);
+void memcomp_invalidate_cache(unsigned long token);
+/* Called only by the winning zero-cache XA replacement while locked. */
+void memcomp_publish_cache_zero(void);
void memcomp_invalidate_nr(unsigned long token, unsigned int nr_pages);
void memcomp_invalidate_folio(struct folio *folio);
bool memcomp_release_folio(struct folio *folio);
@@ -125,6 +128,30 @@ void memcomp_entry_clear_refresh_pending(unsigned long token);
bool memcomp_entry_get(unsigned long token);
bool memcomp_entry_present(unsigned long token);
+/*
+ * Page-cache variants for shmem/tmpfs pages. A zero store result prepares
+ * a value only: its winning XA replacement must publish the cache owner
+ * before unlocking. An aborted zero preparation creates no owner to drop.
+ */
+bool memcomp_store_cache(struct folio *folio, unsigned long *token);
+/*
+ * Pin rejects token0 with an empty guard. Explicit zero load consumes no
+ * owner; the winning shadow replacement drops its cache owner separately.
+ * Pin must be resolved with exactly one abort or commit call. Commit also
+ * drops the shadow owner and is valid only after replacing the expected
+ * shadow in the page cache. Failed pins clear the output; abort and commit
+ * consume and clear the guard. Load keeps the guard on every return.
+ */
+bool memcomp_load_folio_pin(unsigned long token, struct memcomp_ref *ref);
+int memcomp_load_folio(struct memcomp_ref *ref, struct folio *folio);
+int memcomp_load_folio_zero(struct folio *folio);
+void memcomp_load_folio_abort(struct memcomp_ref *ref);
+void memcomp_load_folio_commit(struct memcomp_ref *ref);
+
+/* Check if a page-cache xa_value is a memcomp entry. */
+bool is_memcomp_shadow(void *shadow);
+unsigned long memcomp_shadow_token(void *shadow);
+
#else
static inline bool memcomp_available(void)
@@ -225,6 +252,14 @@ static inline void memcomp_invalidate(unsigned long token)
{
}
+static inline void memcomp_invalidate_cache(unsigned long token)
+{
+}
+
+static inline void memcomp_publish_cache_zero(void)
+{
+}
+
static inline void memcomp_invalidate_nr(unsigned long token,
unsigned int nr_pages)
{
@@ -301,6 +336,51 @@ static inline bool memcomp_entry_present(unsigned long token)
return false;
}
+static inline bool memcomp_store_cache(struct folio *folio,
+ unsigned long *token)
+{
+ return false;
+}
+
+static inline bool memcomp_load_folio_pin(unsigned long token,
+ struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+ return false;
+}
+
+static inline int memcomp_load_folio(struct memcomp_ref *ref,
+ struct folio *folio)
+{
+ *ref = (struct memcomp_ref) { };
+ return -ENOENT;
+}
+
+static inline int memcomp_load_folio_zero(struct folio *folio)
+{
+ return -ENOENT;
+}
+
+static inline void memcomp_load_folio_abort(struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+}
+
+static inline void memcomp_load_folio_commit(struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+}
+
+static inline bool is_memcomp_shadow(void *shadow)
+{
+ return false;
+}
+
+static inline unsigned long memcomp_shadow_token(void *shadow)
+{
+ return 0;
+}
+
#endif
#endif
diff --git a/mm/Kconfig b/mm/Kconfig
index d88c27e0235bf8ac79e05e009b8a4625d9d0c693..61356d7e375b4ce4bcf93b5d37108deb8dd655bd 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -195,7 +195,8 @@ config MEMCOMP
select XXHASH
default n
help
- Enable direct in-memory compression for reclaimed anonymous pages.
+ Enable direct in-memory compression for reclaimed anonymous and
+ shmem/tmpfs pages.
Pages are compressed into RAM and mapped through a special
non-swap PTE entry type. This does not require swap block I/O as
diff --git a/mm/memcomp.c b/mm/memcomp.c
index 8944fa081f3e35bfacf00d3a1c09c8bb32a9db18..07242de736f6771d4a6d74a6182b2cad4c9d150d 100644
--- a/mm/memcomp.c
+++ b/mm/memcomp.c
@@ -2234,6 +2234,35 @@ memcomp_prepare_payload_locked(struct memcomp_reclaim_ctx **ctxp,
return 0;
}
+static int memcomp_prepare_payload(const void *src, unsigned int src_len,
+ unsigned int *length,
+ unsigned long *handle,
+ u8 *pool_idx,
+ enum memcomp_payload_mode mode,
+ bool allow_provisional_raw)
+{
+ struct memcomp_reclaim_ctx *ctx;
+ unsigned long element;
+ int ret;
+
+ /* Same-filled shmem pages do not need a compression workspace. */
+ if (page_same_filled(src, &element)) {
+ *length = 0;
+ *handle = element;
+ return 0;
+ }
+
+ ctx = memcomp_get_comp_ctx(mode, NULL, MEMCOMP_RECLAIM_DIRECT);
+ if (!ctx)
+ return -EBUSY;
+ ret = memcomp_prepare_payload_locked(&ctx, src, src_len, length,
+ handle, pool_idx,
+ allow_provisional_raw, NULL);
+ WARN_ON_ONCE(!ctx);
+ memcomp_put_comp_ctx(ctx);
+ return ret;
+}
+
static __always_inline int
memcomp_arch_prepare_to_swap(struct folio *folio)
{
@@ -2242,6 +2271,9 @@ memcomp_arch_prepare_to_swap(struct folio *folio)
/* Allocate @nr_pages consecutive tokens, skipping fragmented free ranges. */
+#include "memcomp/cache_store.h"
+#include "memcomp/cache_folio.h"
+
/*
* Core store - compress @nr_pages pages and insert them into the global
* entries xarray under consecutive tokens. Returns 0 with *token set
@@ -2455,6 +2487,16 @@ static bool memcomp_map_resident(struct vm_area_struct *vma,
#endif
/* Check if a page-cache xa_value is a memcomp entry. */
+bool is_memcomp_shadow(void *shadow)
+{
+ return xa_is_value(shadow) &&
+ is_memcomp_entry(radix_to_swp_entry(shadow));
+}
+
+unsigned long memcomp_shadow_token(void *shadow)
+{
+ return swp_offset(radix_to_swp_entry(shadow));
+}
/*
* Shmem/tmpfs variant: compress a folio and return the token. This does not
@@ -2468,6 +2510,10 @@ static bool memcomp_map_resident(struct vm_area_struct *vma,
* abort. The caller handles locking, page-cache insertion and rmap.
*/
/* Explicit zero transport: no sentinel guard or owner consumption. */
+int memcomp_load_folio_zero(struct folio *folio)
+{
+ return mc16_cache_load_zero(folio);
+}
/* Fill an existing token span after every mapping has been removed. */
diff --git a/mm/memcomp/api.h b/mm/memcomp/api.h
index 5e49d8484f0a303533f5ae04e0477960775c3512..4cc192fba3af9cd858f5752a8e5e931ef4e3af8f 100644
--- a/mm/memcomp/api.h
+++ b/mm/memcomp/api.h
@@ -29,6 +29,16 @@ void memcomp_invalidate_nr(unsigned long token, unsigned int nr)
mc16_pte_put_nr(token, nr);
}
+void memcomp_publish_cache_zero(void)
+{
+ mc16_zero_cache_get(1);
+}
+
+void memcomp_invalidate_cache(unsigned long token)
+{
+ mc16_cache_drop(token);
+}
+
bool memcomp_entry_present(unsigned long token)
{
return mc16_entry_present(token);
@@ -146,4 +156,28 @@ int memcomp_restore_resident(struct vm_fault *vmf, struct memcomp_ref *ref)
return (ret & VM_FAULT_SIGBUS) ? -EFAULT : 0;
}
+bool memcomp_store_cache(struct folio *folio, unsigned long *token)
+{
+ return mc16_store_cache_folio(folio, token);
+}
+
+bool memcomp_load_folio_pin(unsigned long token, struct memcomp_ref *ref)
+{
+ return mc16_cache_pin(token, ref);
+}
+
+int memcomp_load_folio(struct memcomp_ref *ref, struct folio *folio)
+{
+ return mc16_cache_load(ref, folio);
+}
+
+void memcomp_load_folio_abort(struct memcomp_ref *ref)
+{
+ mc16_cache_abort(ref);
+}
+
+void memcomp_load_folio_commit(struct memcomp_ref *ref)
+{
+ mc16_cache_commit(ref);
+}
#endif /* _MM_MEMCOMP_API_H */
diff --git a/mm/memcomp/cache_folio.h b/mm/memcomp/cache_folio.h
new file mode 100644
index 0000000000000000000000000000000000000000..b0b9f154b42f02f54720f44f5c1e15dd2bf4140a
--- /dev/null
+++ b/mm/memcomp/cache_folio.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _MM_MEMCOMP_CACHE_FOLIO_H
+#define _MM_MEMCOMP_CACHE_FOLIO_H
+
+#include <linux/highmem.h>
+
+/* Include after cache_store.h and memcomp_arch_prepare_to_swap(). */
+#define mc16_cache_folio_arch_prepare memcomp_arch_prepare_to_swap
+
+/*
+ * Caller holds the order-0 source folio lock throughout preparation and XA
+ * replacement. Output is valid only on success and unchanged on failure.
+ * Positive success transfers the initial cache ref, without another get.
+ * Caller drops it if the expected XA replacement loses.
+ * Zero success is preparation only: the winning XA replacement must publish
+ * one zero cache owner under its XA lock. A losing zero preparation owes no put.
+ */
+static inline bool mc16_store_cache_folio(struct folio *folio,
+ unsigned long *token)
+{
+ struct memcomp_ref initial = { };
+#ifdef CONFIG_MEMCOMP_DEBUG_STATS
+ unsigned int nr_pages = folio ? folio_nr_pages(folio) : 1;
+#endif
+ void *src;
+ int ret;
+
+ if (!token || !folio || !folio_test_locked(folio) ||
+ folio_test_large(folio) || folio_mapped(folio) ||
+ folio_test_mlocked(folio) || folio_test_writeback(folio) ||
+ folio_maybe_dma_pinned(folio))
+ goto fail_eligibility;
+ if (!memcomp_is_enabled())
+ goto fail_payload;
+ /* The arch predicate includes tags, unlike a byte-only zero scan. */
+ if (pages_identical(&folio->page, ZERO_PAGE(0))) {
+ *token = 0;
+ goto success;
+ }
+ src = kmap_local_folio(folio, 0);
+ ret = mc16_cache_store(src, folio, &initial);
+ kunmap_local(src);
+ if (ret)
+ goto fail_payload;
+ /* The arch hook keys metadata off page_swap_entry(). */
+ folio->swap = make_memcomp_entry(initial.token);
+ ret = mc16_cache_folio_arch_prepare(folio);
+ folio->swap.val = 0;
+ if (ret) {
+ mc16_ref_put(&initial);
+ goto fail_arch;
+ }
+ *token = initial.token;
+success:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_store_ok_pages);
+ return true;
+
+fail_eligibility:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_eligibility_fail_pages);
+ goto fail;
+fail_payload:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_payload_fail_pages);
+ goto fail;
+fail_arch:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_arch_fail_pages);
+fail:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_store_fail_pages);
+ return false;
+}
+#endif /* _MM_MEMCOMP_CACHE_FOLIO_H */
diff --git a/mm/memcomp/cache_store.h b/mm/memcomp/cache_store.h
new file mode 100644
index 0000000000000000000000000000000000000000..ca2d6097b491aeaaaf2807e367d37ec3f6b8cc8b
--- /dev/null
+++ b/mm/memcomp/cache_store.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _MM_MEMCOMP_CACHE_STORE_H
+#define _MM_MEMCOMP_CACHE_STORE_H
+
+/*
+ * Prepare one cache payload, retaining its initial cache reference in out.
+ * The caller keeps the source/account folio alive and transfers that reference
+ * only after arch preparation and the expected page-cache replacement succeed.
+ * This lower-level helper does not choose the separate slotless-zero path.
+ */
+/* Private reservation -> counted initial cache ownership, infallibly. */
+static inline void mc16_cache_publish(struct memcomp_ref *ref, unsigned int bytes)
+{
+ memcomp_stored_bytes_inc();
+ memcomp_compressed_bytes_add(bytes);
+ mc16_account_state(ref, true);
+ mc16_cache_activate(ref);
+}
+
+static inline int mc16_cache_store(const void *src, struct folio *account,
+ struct memcomp_ref *out)
+{
+ struct mc16_initial_payload payload = { };
+ struct obj_cgroup *objcg;
+ unsigned long handle = 0;
+ unsigned int bytes = 0;
+ u8 pool = 0;
+ int ret;
+
+ if (!out)
+ return -EINVAL;
+ *out = (struct memcomp_ref) { };
+ if (!src || !account)
+ return -EINVAL;
+ if (!memcomp_is_enabled())
+ return -EOPNOTSUPP;
+ /* Independent source owner, never a borrowed registry key across X1/X2. */
+ objcg = memcomp_objcg_get(account);
+ ret = memcomp_prepare_payload(src, PAGE_SIZE, &bytes, &handle, &pool,
+ MEMCOMP_PAYLOAD_BEST_EFFORT, false);
+ if (ret)
+ goto out_owner;
+ if (!memcomp_payloads_net_reclaimable(&bytes, 1)) {
+ if (bytes == PAGE_SIZE)
+ memcomp_debug_inc(&memcomp_raw_reject_net);
+ ret = -E2BIG;
+ goto out_payload;
+ }
+ ret = memcomp_objcg_charge(objcg, GFP_NOWAIT | __GFP_NOWARN, bytes);
+ if (ret)
+ goto out_payload;
+ payload = (struct mc16_initial_payload) { .word = handle, .bytes = bytes };
+ ret = mc16_reserve_cache(objcg, folio_nid(account), &payload, 1, out);
+ if (ret) {
+ memcomp_objcg_uncharge(objcg, bytes);
+ goto out_payload;
+ }
+ /* No pending/resident/OWNER transition or record/member RMW occurs. */
+ mc16_cache_publish(out, bytes);
+ ret = 0;
+ goto out_owner;
+out_payload:
+ if (bytes)
+ memcomp_zs_free(pool, handle);
+out_owner:
+ if (objcg)
+ memcomp_objcg_put(objcg);
+ return ret;
+}
+#endif /* _MM_MEMCOMP_CACHE_STORE_H */
diff --git a/mm/memcomp/consumers.h b/mm/memcomp/consumers.h
index 929c046bf98d52c878e727e14d859d81c871b743..03aef9ffecb6df83b6314df43739677120fdd8f8 100644
--- a/mm/memcomp/consumers.h
+++ b/mm/memcomp/consumers.h
@@ -46,19 +46,74 @@ static inline void mc16_pte_put_nr(unsigned long token, unsigned int nr)
}
/* The caller removed one cache owner, independently of any PTE owner. */
+static inline void mc16_cache_drop(unsigned long token)
+{
+ struct memcomp_ref ref;
+
+ if (!token) {
+ mc16_zero_cache_put(1);
+ return;
+ }
+ if (mc16_get(token, &ref))
+ mc16_ref_put_count(&ref, 2); /* lookup + removed cache owner */
+}
/* Empty output on failure, including token0; caller handles zero explicitly. */
+static inline bool mc16_cache_pin(unsigned long token, struct memcomp_ref *ref)
+{
+ return mc16_get(token, ref);
+}
/*
* Keep this same guard through load and the caller's xarray transaction.
* Failure leaves both the guard and cache owner intact for abort/retry.
*/
+static inline int mc16_cache_load(struct memcomp_ref *ref, struct folio *folio)
+{
+ void *dst;
+ int ret;
+
+ if (!ref || !ref->slot)
+ return -ENOENT;
+ if (!folio || folio_test_large(folio) || !folio_test_locked(folio))
+ return -EINVAL;
+ dst = kmap_local_folio(folio, 0);
+ ret = mc16_load_page(ref, dst);
+ kunmap_local(dst);
+ if (!ret) {
+ __folio_mark_uptodate(folio);
+ flush_dcache_folio(folio);
+ memcomp_arch_restore(ref->token, folio);
+ }
+ return ret;
+}
+
+static inline void mc16_cache_abort(struct memcomp_ref *ref)
+{
+ if (ref && ref->slot)
+ mc16_ref_put(ref);
+}
/* Only the winner replacing the expected cache shadow may consume its owner. */
+static inline void mc16_cache_commit(struct memcomp_ref *ref)
+{
+ if (ref && ref->slot)
+ mc16_ref_put_count(ref, 2); /* same guard + cache owner */
+}
/*
* Explicit token0 branch: no guard, lookup or owner consumption. After the
* expected zero shadow is replaced successfully, caller must cache_drop(0).
* An aborted xarray transaction leaves the zero cache owner unchanged.
*/
+static inline int mc16_cache_load_zero(struct folio *folio)
+{
+ if (!folio || folio_test_large(folio) || !folio_test_locked(folio))
+ return -EINVAL;
+ clear_highpage(folio_page(folio, 0));
+ __folio_mark_uptodate(folio);
+ flush_dcache_folio(folio);
+ memcomp_arch_restore(0, folio);
+ return 0;
+}
#endif /* _MM_MEMCOMP_CONSUMERS_H */
diff --git a/mm/shmem.c b/mm/shmem.c
index 5960e5035f983571fb806313855f74c61abf8354..a04e68d445d37384b02b56e791586d09b392fcfa 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -80,6 +80,9 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uuid.h>
#include <linux/quotaops.h>
#include <linux/rcupdate_wait.h>
+#ifdef CONFIG_MEMCOMP
+#include <linux/memcomp.h>
+#endif
#include <linux/uaccess.h>
@@ -847,6 +850,15 @@ static void shmem_delete_from_page_cache(struct folio *folio, void *radswap)
xa_lock_irq(&mapping->i_pages);
error = shmem_replace_entry(mapping, folio->index, folio, radswap);
+#ifdef CONFIG_MEMCOMP
+ if (!error) {
+ swp_entry_t entry = radix_to_swp_entry(radswap);
+
+ /* Publish ownership before any reader can remove this shadow. */
+ if (is_memcomp_entry(entry) && !swp_offset(entry))
+ memcomp_publish_cache_zero();
+ }
+#endif
folio->mapping = NULL;
mapping->nrpages -= nr;
shmem_update_stats(folio, -nr);
@@ -864,14 +876,29 @@ static long shmem_free_swap(struct address_space *mapping,
pgoff_t index, void *radswap)
{
int order = xa_get_order(&mapping->i_pages, index);
+#ifdef CONFIG_MEMCOMP
+ swp_entry_t entry = radix_to_swp_entry(radswap);
+ long nr_pages = is_memcomp_entry(entry) ? 1 : 1 << order;
+#endif
void *old;
old = xa_cmpxchg_irq(&mapping->i_pages, index, radswap, NULL, 0);
if (old != radswap)
return 0;
+#ifdef CONFIG_MEMCOMP
+ if (is_memcomp_entry(entry))
+ memcomp_invalidate_cache(swp_offset(entry));
+ else
+ free_swap_and_cache_nr(entry, nr_pages);
+#else
free_swap_and_cache_nr(radix_to_swp_entry(radswap), 1 << order);
+#endif
+#ifdef CONFIG_MEMCOMP
+ return nr_pages;
+#else
return 1 << order;
+#endif
}
/*
@@ -893,8 +920,23 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping,
xas_for_each(&xas, page, max) {
if (xas_retry(&xas, page))
continue;
+#ifdef CONFIG_MEMCOMP
+ if (xa_is_value(page)) {
+ swp_entry_t entry = radix_to_swp_entry(page);
+
+ if (is_memcomp_entry(entry)) {
+ if (memcomp_entry_stat(swp_offset(entry),
+ NULL, NULL))
+ swapped += PAGE_SIZE;
+ } else {
+ swapped += (1UL << xas_get_order(&xas)) <<
+ PAGE_SHIFT;
+ }
+ }
+#else
if (xa_is_value(page))
swapped += 1 << xas_get_order(&xas);
+#endif
if (xas.xa_index == max)
break;
if (need_resched()) {
@@ -904,7 +946,11 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping,
}
rcu_read_unlock();
+#ifdef CONFIG_MEMCOMP
+ return swapped;
+#else
return swapped << PAGE_SHIFT;
+#endif
}
/*
@@ -932,6 +978,15 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)
if (!swapped)
return 0;
+#ifdef CONFIG_MEMCOMP
+ /*
+ * info->swapped counts a memcomp entry as one page, which is what
+ * shmem_partial_swap_usage() also reports for it, so the whole-object
+ * answer is still exact without the walk. Restoring this matters:
+ * smaps_rollup is polled continuously, and the walk is an RCU pass
+ * over every entry of the mapping with an xa_load per memcomp one.
+ */
+#endif
if (!vma->vm_pgoff && vma->vm_end - vma->vm_start >= inode->i_size)
return swapped << PAGE_SHIFT;
@@ -1321,6 +1376,10 @@ static int shmem_find_swap_entries(struct address_space *mapping,
continue;
entry = radix_to_swp_entry(folio);
+#ifdef CONFIG_MEMCOMP
+ if (is_memcomp_entry(entry))
+ continue;
+#endif
/*
* swapin error entries can be found in the mapping. But they're
* deliberately ignored here as we've done everything we can do.
@@ -1459,7 +1518,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
swp_entry_t swap;
pgoff_t index;
+#ifdef CONFIG_MEMCOMP
+ unsigned long mc_token;
+#endif
int nr_pages;
+#ifdef CONFIG_MEMCOMP
+ bool memcomp_attempted = false;
+#endif
bool split = false;
/*
@@ -1475,8 +1540,84 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
if (WARN_ON_ONCE((info->flags & VM_LOCKED) || sbinfo->noswap))
goto redirty;
+#ifdef CONFIG_MEMCOMP
+ /*
+ * Only an uptodate folio. The !uptodate handling below is what gives
+ * shmem_fallocate() its backpressure - it counts the folio in
+ * shmem_falloc->nr_unswapped and redirties it so the fallocation can
+ * give up with -ENOMEM - and what zeroes a folio whose contents were
+ * never initialised. Jumping straight to the store skipped both, so a
+ * fallocate that should have failed ran on to OOM, and on a kernel
+ * booted with init_on_alloc=0 the uninitialised page became file
+ * contents. A folio that arrives !uptodate takes the swap path, and
+ * the retries below reach the store once it has been made uptodate.
+ */
+ if (memcomp_available() && !folio_test_large(folio) &&
+ folio_test_uptodate(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+ goto try_swap;
+
+try_memcomp:
+ memcomp_attempted = true;
+ if (!folio_test_large(folio)) {
+ if (memcomp_store_cache(folio, &mc_token)) {
+ swp_entry_t mc_entry = make_memcomp_entry(mc_token);
+
+ mutex_lock(&shmem_swaplist_mutex);
+ if (list_empty(&info->swaplist))
+ list_add(&info->swaplist, &shmem_swaplist);
+ mutex_unlock(&shmem_swaplist_mutex);
+
+ /*
+ * Raise swapped before nrpages is lowered, as the
+ * swap path below does. shmem_recalc_inode()
+ * compensates for exactly that order - see the
+ * comment on its `swapped > 0` clause - so calling it
+ * after shmem_delete_from_page_cache() has already
+ * dropped nrpages makes it count the page as freed
+ * and hand a block back that nothing released.
+ */
+ shmem_recalc_inode(inode, 0, 1);
+ shmem_delete_from_page_cache(folio,
+ swp_to_radix_entry(mc_entry));
+ if (folio_test_dirty(folio))
+ folio_clear_dirty(folio);
+ /*
+ * Leave the token on the folio. The folio now has no
+ * mapping and one reference, and shrink_folio_list()
+ * only credits nr_reclaimed for a folio it frees on
+ * its free_it: path. Its recovery arm for exactly
+ * this shape keys on page_swap_entry(), so without a
+ * token the folio takes keep_locked, is dropped
+ * unaccounted by move_folios_to_lru(), and direct
+ * reclaim reports no progress for work it did - which
+ * __alloc_pages_slowpath() counts toward OOM.
+ *
+ * The entry is owned by the page-cache xarray, not by
+ * the folio: memcomp_do_store() was passed no
+ * owner folio, so it carries MC_SHMEM rather than
+ * MC_OWNER and the memcomp_invalidate_folio() that
+ * arm performs drops nothing. This is the same
+ * handover memcomp_reserve() makes for anonymous
+ * folios.
+ */
+ folio->swap = mc_entry;
+ folio_unlock(folio);
+ return 0;
+ }
+ }
+
+try_swap:
+ /* This is the regular shmem fallback, not memcomp writeback. */
+#endif
if (!total_swap_pages)
goto redirty;
+#ifdef CONFIG_MEMCOMP
+ if (memcomp_attempted)
+ memcomp_count_normal_swap_fallback();
+#endif
/*
* If CONFIG_THP_SWAP is not enabled, the large folio should be
@@ -1541,6 +1682,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
swap = folio_alloc_swap(folio);
if (!swap.val) {
+#ifdef CONFIG_MEMCOMP
+ if (!memcomp_attempted && memcomp_available() &&
+ !folio_test_large(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+#endif
if (nr_pages > 1)
goto try_split;
@@ -1573,6 +1721,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
mutex_unlock(&shmem_swaplist_mutex);
put_swap_folio(folio, swap);
+#ifdef CONFIG_MEMCOMP
+ if (!memcomp_attempted && memcomp_available() &&
+ !folio_test_large(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+#endif
redirty:
folio_mark_dirty(folio);
if (wbc->for_reclaim)
@@ -2110,6 +2265,89 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
if (is_poisoned_swp_entry(swap))
return -EIO;
+#ifdef CONFIG_MEMCOMP
+
+ if (is_memcomp_entry(swap)) {
+ struct memcomp_ref mc_ref = { };
+ unsigned long token = swp_offset(swap);
+
+ if (!shmem_confirm_swap(mapping, index, swap))
+ return -EEXIST;
+
+ if (shmem_inode_acct_blocks(inode, 1))
+ return -ENOSPC;
+
+ folio = shmem_alloc_folio(gfp, 0, info, index);
+ if (!folio) {
+ shmem_inode_unacct_blocks(inode, 1);
+ return -ENOMEM;
+ }
+ __folio_set_locked(folio);
+ __folio_set_swapbacked(folio);
+ nr_pages = folio_nr_pages(folio);
+
+ error = mem_cgroup_charge(folio, fault_mm, gfp & GFP_RECLAIM_MASK);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ return error;
+ }
+
+ /*
+ * Allocation and memcg charge may reclaim or wait for OOM.
+ * Pin only after both, then recheck the page-cache entry.
+ */
+ if (token)
+ memcomp_load_folio_pin(token, &mc_ref);
+ if ((token && !mc_ref.slot) ||
+ !shmem_confirm_swap(mapping, index, swap)) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ if (mc_ref.slot)
+ memcomp_load_folio_abort(&mc_ref);
+ return -EEXIST;
+ }
+
+ error = token ? memcomp_load_folio(&mc_ref, folio) :
+ memcomp_load_folio_zero(folio);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ if (!shmem_confirm_swap(mapping, index, swap))
+ error = -EEXIST;
+ memcomp_load_folio_abort(&mc_ref);
+ return error;
+ }
+
+ error = shmem_add_to_page_cache(folio, mapping, index,
+ swp_to_radix_entry(swap), gfp);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ memcomp_load_folio_abort(&mc_ref);
+ return error == -EEXIST ? -EEXIST : error;
+ }
+
+ shmem_recalc_inode(inode, 1, -1);
+
+ if (sgp == SGP_WRITE)
+ folio_mark_accessed(folio);
+
+ folio_mark_dirty(folio);
+ if (token)
+ memcomp_load_folio_commit(&mc_ref);
+ else
+ memcomp_invalidate_cache(0);
+ folio_add_lru(folio);
+
+ *foliop = folio;
+ return 0;
+ }
+#endif
si = get_swap_device(swap);
if (!si) {
next prev parent reply other threads:[~2026-11-16 11:04 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 11:01 [RFC PATCH 00/16] mm: add in-memory compression of reclaimed pages nakixii
2026-11-16 11:01 ` [RFC PATCH 01/16] mm: userfaultfd: check the exclusivity of the page being moved nakixii
2026-11-16 11:01 ` [RFC PATCH 02/16] mm: add a swap entry type for in-memory compressed pages nakixii
2026-11-16 11:01 ` [RFC PATCH 03/16] mm: memcomp: compress reclaimed anonymous pages in memory nakixii
2026-11-16 11:01 ` [RFC PATCH 04/16] mm: userfaultfd: let UFFDIO_MOVE move memcomp entries nakixii
2026-11-16 11:01 ` [RFC PATCH 05/16] mm: memcomp: charge compressed pages to the source memcg nakixii
2026-11-16 11:01 ` nakixii [this message]
2026-11-16 11:01 ` [RFC PATCH 07/16] mm: memcomp: write cold pages back to swap nakixii
2026-11-16 11:01 ` [RFC PATCH 08/16] mm: memcomp: compress foreground batches asynchronously nakixii
2026-11-16 11:01 ` [RFC PATCH 09/16] mm: memcomp: batch unmap reserved large folios nakixii
2026-11-16 11:01 ` [RFC PATCH 10/16] mm: memcomp: queue restored large folios for splitting nakixii
2026-11-16 11:01 ` [RFC PATCH 11/16] mm: memcomp: account user eviction and refault nakixii
2026-11-16 11:01 ` [RFC PATCH 12/16] mm: memcomp: report precise token residency nakixii
2026-11-16 11:01 ` [RFC PATCH 13/16] mm: memcomp: add KUnit tests nakixii
2026-11-16 11:01 ` [RFC PATCH 14/16] selftests/mm: add a memcomp pageout and refault test nakixii
2026-11-16 11:01 ` [RFC PATCH 15/16] mm: memcomp: add the design doc nakixii
2026-11-16 11:01 ` [RFC PATCH 16/16] mm: memcomp: add the admin guide nakixii
2026-11-16 11:14 ` [RFC PATCH 00/16] mm: add in-memory compression of reclaimed pages Mike Rapoport
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116110122.309748-7-nakixii@gmail.com \
--to=nakixii@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=baohua@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=chrisl@kernel.org \
--cc=david@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=kasong@tencent.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=nphamcs@gmail.com \
--cc=peterx@redhat.com \
--cc=roman.gushchin@linux.dev \
--cc=rppt@kernel.org \
--cc=senozhatsky@chromium.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=yosry@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®