[PATCH v2 0/2] mm/compaction: stop repeating failed async compaction on every THP fault

[Qiliang Yuan <odys.yuan@gmail.com>]

Mooncake, the KV cache store used with vLLM and SGLang, backs each
rank's host segment with MADV_HUGEPAGE memory and registers it with
the EFA NIC, which long-term pins it. Its commit c3fa13ecaf10 [1]
reports:

  "under the default local NUMA policy every THP fault on a full node
   runs direct compaction before falling back. With 8 ranks x 190GB
   this made registration take ~38 minutes."

Mooncake now binds the segment MPOL_INTERLEAVE across all allowed
nodes, so that no node fills before the others. That gives up
node-local placement, and a task that sets its own memory policy
still gets the stall.

A THP fault on a MADV_HUGEPAGE VMA tries the local node first, with a
single round of async direct compaction, before falling back to other
nodes. A failed async run is never deferred, so when the free memory
of the local node is scattered between pages that can't be migrated,
such as long-term pins, every 2M fault scans the zone again and fails.
A node that is simply full doesn't pay this: compaction_suitable()
turns compaction away before it scans anything.

Patch 1 splits the deferral state by migration mode. Patch 2 defers
async compaction on its own state after it fails, leaving sync
compaction as it is.

Populating a 4 GiB MADV_HUGEPAGE buffer on a node fragmented by pinned
memory, direct compactions drop from 649-1473 to 9-41 and the time
spent in compaction from 118.6-264.2 ms to 2.6-5.1 ms, with no fewer
THPs on the local node. Details are in patch 2.

[1] https://github.com/kvcache-ai/Mooncake/commit/c3fa13ecaf1038df933dcb784ff8ffe026381100

Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
---
V1 -> V2:
- Describe the Mooncake report in the cover letter
- Emit mm_compaction_defer_reset for both states (Sashiko)
- Reset the pageblock skip hints only from sync compaction (Sashiko)
- Rebase on v7.4-rc1 and rerun the tests

v1: https://lore.kernel.org/r/20261001-bug-mm-thp-async-compact-defer-v1-0-0174c7923430@gmail.com

---
Qiliang Yuan (2):
      mm/compaction: keep compaction deferral state per migration mode
      mm/compaction: defer failed async direct compaction

 include/linux/mmzone.h            |  7 ++--
 include/trace/events/compaction.h | 27 +++++++------
 mm/compaction.c                   | 82 ++++++++++++++++++++++-----------------
 3 files changed, 66 insertions(+), 50 deletions(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261001-bug-mm-thp-async-compact-defer-b83351bfb25d

Best regards,
-- 
Qiliang Yuan <odys.yuan@gmail.com>