mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v6 0/4] mm/huge_memory: copy PMDs at fork like the PTE path does
@ 2026-11-19 13:44 Konstantin Khorenko
  2026-11-19 13:44 ` [PATCH v6 1/4] mm/rmap: duplicate a PMD mapping as one in folio_dup_file_rmap_pmd() Konstantin Khorenko
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Konstantin Khorenko @ 2026-11-19 13:44 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand (Arm),
	Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
	Dev Jain, Barry Song, Lance Yang, Usama Arif, Kiryl Shutsemau,
	Peter Xu
  Cc: Alex Williamson, kvm, Loïc Molinari, Lorenzo Stoakes,
	Shuah Khan, linux-kselftest, dri-devel, Madhavan Srinivasan,
	Ritesh Harjani (IBM),
	linuxppc-dev, linux-mm, linux-kernel, Konstantin Khorenko,
	Jason Gunthorpe, stable, Jose A. Perez de Azpillaga, Sashiko

v6 is rebased onto mm/core for-next, as David asked, and patch 2 was
finished on the way: it now checks the page cache refcount limit before
it copies a file folio, as copy_present_ptes() and copy_huge_pud() do
since the refcount overflow fixes. So patch 2 changed after the
Reviewed-by tags it carries.

copy_huge_pmd() decides what to copy at fork by its own rules, apart
from copy_present_ptes(): a lockless check for special PMDs, and a skip
of every file THP. Following David's review of v3, this series makes it
decide the way the PTE path does, under the page table lock, with
vm_normal_page_pmd(). That fixes the two bugs v1-v3 were about. The
patches:

1. folio_dup_file_rmap_pmd() duplicated a PMD mapping as HPAGE_PMD_NR
   PTE mappings. It had no caller until patch 2.
2. File THPs are copied at fork instead of being refaulted by the
   child, as their PTEs would be.
3. The special-PMD read moves under the lock. That closes the race with
   a driver zap (vfio-pci, panthor) and the missing page table deposit
   on ppc64 hash that Sashiko found in v1.
4. A selftest for 1 and 2.

Stable
======

The race (since v6.12) and the ppc64 hash NULL deref (since v7.1) are
fixed in patch 3, which only works on top of the copy rework in patch
2, and patch 2 is the first caller of the helper patch 1 fixes. So
patches 1-3 carry Cc: stable, as David suggested on v4. Patch 2 needs
folio_ref_count_over_limit() from 26b8f029ce522 ("mm: limit pagecache
folio refcount when mapping pages"). They will not apply to 6.12 as
they are, I will do the backports.

The related refcount bug in copy_huge_pud() for device DAX is fixed
in for-next by d90bdc02e36ec ("mm/huge_memory: properly handle
refcounted PUD mappings on fork").

Testing
=======

v6 on mm/core for-next (v7.4-rc3 based), x86_64 with 4 NUMA nodes, the
series with CONFIG_DEBUG_VM, KASAN and lockdep:

- fork_file_thp (patch 4) skips on for-next, where fork leaves file
  THPs to a refault, and fails in both cases without patch 1, where
  ShmemPmdMapped drops when the child exits and the kernel reports
  "Bad page cache";
- with the series it passes, also under migrate_pages() of the parent
  and under reclaim in a small memcg, where some runs skip; with other
  processes churning PMD-mapped shmem it passes or skips, never fails;
- cow, guard-regions, uffd-unit-tests, uffd-wp-mremap,
  split_huge_page_test, mremap_test, soft-dirty, pagemap_ioctl,
  migration, madv_populate, transhuge-stress and khugepaged pass as on
  for-next.

The runs below are from v4, on v7.3-rc7 in a KVM guest with 4 vCPUs,
x86_64. Patches 1-3 changed since only by the device private CoW check
and the refcount limit check.

Race with a driver zap. The guest's QXL device is bound to vfio-pci in
no-IOMMU mode, BAR0 of 64 MB mapped with PMDs. The reproducer, in the v5
cover letter, forks while a second thread clears PCI_COMMAND_MEMORY
through the vfio config region after a random delay of up to 1 ms; the
child waits for the write to finish and reads one byte of each 2 MB of
the BAR. A child that reads anything without SIGBUS kept a stale
mapping. I have not hit it without help, so a test-only debugfs knob
adds an mdelay(2) between copy_pmd_range()'s lockless pmd_is_huge() and
the PMD locks:

  +-----------------------+-------+-------+----------------+
  | kernel                | delay | forks | stale children |
  +-----------------------+-------+-------+----------------+
  | 7.3-rc7               | none  | 20900 | 0              |
  | 7.3-rc7               | 2 ms  | 150   | 128            |
  | 7.3-rc7 + this series | none  | 3000  | 0              |
  | 7.3-rc7 + this series | 2 ms  | 400   | 0              |
  +-----------------------+-------+-------+----------------+

Ritesh tested v5 on ppc64 in hash mode: without the series the child
dereferences a NULL table in __hash_page_thp(), with it it does not.
Before that, on x86 a test-only hack made arch_needs_pgtable_deposit()
true and mremap move the table, as on ppc64 hash. Without the series
the first child of the reproducer oopses at exit in
pgtable_trans_huge_withdraw(); with it the reproducer and the mm
selftests pass.

Cost
====

fork() now copies file THPs instead of leaving them to a refault, which
costs fork() a reference, an rmap update and a counter per huge PMD and
saves the child a fault per huge PMD it touches. fork() of a process
with 4 GB of shmem THPs mapped MAP_PRIVATE and an anon_vma on the VMA,
x86_64 bare metal with 4 NUMA nodes, the benchmark bound to node 0,
both kernels v7.4-rc1 (v5) with the same config, medians of 2000 forks,
range over the rounds; the exec is of /bin/true:

  +-----------------+------------+-----------------+--------------+
  | kernel          | fork()     | child reads all | fork()+exec  |
  +-----------------+------------+-----------------+--------------+
  | v7.4-rc1        | 135-141 us | 3820-4036 us    | 559-567 us   |
  | v7.4-rc1+series | 429-440 us | 1007-1036 us    | 1190-1226 us |
  +-----------------+------------+-----------------+--------------+

A child that execs right away, or touches little, pays about 0.3 us per
huge PMD for the copy and the teardown. One that faults a PMD in itself
pays about 1.8 us for it, so the copy wins once the child touches more
than about a sixth of them.

Only a narrow case pays it, a file THP that is:

- mapped MAP_PRIVATE;
- in a VMA written to at least once, so it has an anon_vma;
- a PMD-sized folio in the page cache, mapped by one PMD;
- itself never written to.

For example:

- a big data file mapped MAP_PRIVATE that is mostly read and patched
  in a few places;
- a private mapping of a memfd or tmpfs file.

Notes
=====

- drm_gem_shmem and mshv_vtl map special PMDs with
  vmf_insert_pfn_pmd() as well.
- copy_huge_pud() still differs from the PMD tail in that it does not
  mark a shared PUD clean in the child or handle uffd bits.

The reproducer is in the v5 cover letter:
https://patch.msgid.link/20261104-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v5-0-c56d0d26f970@virtuozzo.com

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
Changes in v6:
- Rebase onto mm/core for-next (David).
- Patch 2: do not copy a file folio over the page cache refcount limit,
  as copy_present_ptes() and copy_huge_pud() now do; fail fork() with
  -ENOMEM only in a VM_MIXEDMAP VMA.
- Based on mm/core for-next, which still has read_pmd_pagesize(), so
  patch 4 keeps it; on mm-new it needs s/read_pmd_pagesize/pmd_psize/,
  as Jose noted on v5.
- Collect the tags from v5, Tested-by from Ritesh on patch 3.
- Cover: new Cost table, measured on bare metal with a fork()+exec
  column, and the cases that pay for the copy (Andrew). The reproducer
  is only linked now.
- Link to v5: https://patch.msgid.link/20261104-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v5-0-c56d0d26f970@virtuozzo.com

Changes in v5:
- Rebase onto v7.4-rc1.
- Patches 1-3: Cc: stable (David, Lorenzo).
- Patch 1: drop the list of consequences, the helper had no caller
  (Zi Yan).
- Patch 2: the device private write path gets the CoW check of
  copy_nonpresent_pte() (David). Drop fuse DAX as the example of
  VM_MIXEDMAP, it no longer sets the flag. Say on the Cc: stable line
  that the next patch needs it.
- Patch 4: up to three attempts, each on a fresh file. Fail only if
  every attempt sees ShmemPmdMapped drop by exactly the parent's share,
  pass on the first clean one, skip if none is clean: other processes
  change the counter and reclaim or migration take PMDs from the parent
  (Kiryl). Keep every mapping until exit: after the bug, an unmap
  drops the counter once more, which hid the bug from the second case.
  Use check_huge_file(), check_huge_shmem() is gone.
- Cover: v5 test results, box tables.
- Link to v4: https://patch.msgid.link/20261021-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v4-0-66f00b7a564c@virtuozzo.com

Changes in v4:
- Rework after David's review: copy_huge_pmd() decides under the PTL
  with vm_normal_page_pmd(), like copy_present_ptes(), and copies file
  THPs (patches 2, 3). The pmd_none() recheck and the separate deposit
  patch are gone.
- New patch 1: fix folio_dup_file_rmap_pmd(), whose first caller is
  patch 2.
- New selftest, patch 4.
- Move Assisted-by from the patches to the cover letter, after commit
  1e035297ffdad ("docs: remove Assisted-by references").
- Link to v3: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v3-0-85eaec8417f6@virtuozzo.com

Changes in v3:
- Patch 1: recheck the PMD with pmd_none(), as only a zap can change
  it and a zap leaves it none (Lance Yang).
- Link to v2: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v2-0-e1548488fa68@virtuozzo.com

Changes in v2:
- Add patch 2, the page table deposit for ppc64 hash, reported by
  Sashiko.
- Patch 1: name panthor instead of drm_gem_shmem as the driver whose
  stale child mapping reaches freed pages: drm_gem_shmem keeps the
  pages of a purged object until the last mapping goes away.
- Link to v1: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v1-1-d225c60bc316@virtuozzo.com

---
Konstantin Khorenko (4):
      mm/rmap: duplicate a PMD mapping as one in folio_dup_file_rmap_pmd()
      mm/huge_memory: copy file THPs at fork like copy_present_ptes() does
      mm/huge_memory: read a special PMD under the lock in copy_huge_pmd()
      selftests/mm: test fork of PMD-mapped shmem THPs

 MAINTAINERS                                |   1 +
 include/linux/rmap.h                       |   2 +-
 mm/huge_memory.c                           |  96 ++++++-----
 tools/testing/selftests/mm/Makefile        |   1 +
 tools/testing/selftests/mm/fork_file_thp.c | 256 +++++++++++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |   1 +
 6 files changed, 312 insertions(+), 45 deletions(-)
---
base-commit: f7d95bf74091373fad7273db4557df9b5a902335
change-id: 20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-bdf7efb612c2

Best regards,
--  
Konstantin Khorenko,
Virtuozzo Linux Kernel Team


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-19 18:32 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 13:44 [PATCH v6 0/4] mm/huge_memory: copy PMDs at fork like the PTE path does Konstantin Khorenko
2026-11-19 13:44 ` [PATCH v6 1/4] mm/rmap: duplicate a PMD mapping as one in folio_dup_file_rmap_pmd() Konstantin Khorenko
2026-11-19 13:44 ` [PATCH v6 2/4] mm/huge_memory: copy file THPs at fork like copy_present_ptes() does Konstantin Khorenko
2026-11-19 13:44 ` [PATCH v6 3/4] mm/huge_memory: read a special PMD under the lock in copy_huge_pmd() Konstantin Khorenko
2026-11-19 13:44 ` [PATCH v6 4/4] selftests/mm: test fork of PMD-mapped shmem THPs Konstantin Khorenko
2026-11-19 18:32   ` Andrew Morton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®