[RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo

[Luigi Rizzo <lrizzo@google.com>]

This is v2 of the RFC previously posted as DMA_PMD [0] (renamed to avoid
confusion with CPU page table PMD levels).

PISA stands for "Pre-mapped IOVA Superpage Allocator" and as the name
says, it is a specialized memory allocator. Invoked via __GFP_PISA* flag
to alloc_pages(), it returns pages ("PISA memory") that make DMA-related
operations on that memory a lot more efficient, both in software and
in hardware.

Specifically, PISA memory has the following properties:

- it is backed by 2MB contiguous physical pages, so the IOMMU can
  use large leaf mappings and significantly reduce IOTLB pressure see [1]

- IOMMU mappings are recycled while PISA memory is used for IO buffers,
  thus greatly reducing IOVA allocation and dma map/unmap costs.
  This includes cases (e.g. overcommitted VMs or Confidential Compute)
  where they would trigger expensive VM exits or bounce buffer copies.

- PISA memory is returned to the buddy allocator in background, and
  only after all existing IOMMU and IOTLB mappings have been flushed.
  This removes a significant security problem in DMA-FQ, which instead returns 
  pages before flushing the IOTLB, without the performance penalty of
  strict iommu (DMA) where the flushes are synchronous.

The pitch line remains:

  PISA has the performance of identity mapping, guarantees that only active
  I/O buffers have IOMMU/IOTLB mappings (unlike DMA-FQ), removes bounce-buffer
  copies in Confidential Computing VMs, and integrates transparently with
  existing kernel subsystems and drivers.

=== KEY IDEA: A SPECIALIZED MEMORY ALLOCATOR

For avoidance of doubt, PISA is NOT a new buffer cache, page_pool
replacement, slab allocator, or custom netdev feature. The kernel already
has plenty of specialized caches and pools (page_pool, socket page_frags,
SLUB, page cache) built on top of the buddy allocator.

Instead, PISA sits *below* those existing subsystems as a specialized
memory provider modeled after alloc_pages() and __free_pages_prepare().
It enforces a few properties on the memory it dispenses so that IOMMU
mapping, unmapping, and CoCo page decryption become nearly free:

1. PISA allocates order-N blocks out of physically contiguous 2MB pages that
   are mapped once per IOMMU domain using 2MB IOMMU leaf entries in a
   reserved per-domain IOVA window (and decrypted once upfront in CoCo
   guests). Subsequent dma_map_*() calls resolve the IOVA with a simple
   window offset calculation, and dma_unmap_*() is a no-op.

2. Freed blocks are intercepted in __free_pages_prepare() and recycled
   back into their PISA allocator while their 2MB IOMMU mappings remain
   active. A 2MB page is returned to the buddy allocator only when all
   of its blocks are idle, its IOMMU mappings have been removed from all
   domains, a synchronous IOTLB flush has completed, and an RCU grace
   period has elapsed.

3. Callers allocate from per-NUMA-node PISA allocators using standard page
   allocator calls with __GFP_PISA_RW (DMA_BIDIRECTIONAL) or __GFP_PISA_RO
   (DMA_TO_DEVICE, mapped IOMMU_READ-only so devices cannot overwrite
   shared/adjacent data), falling back to the buddy allocator on failure.

Because PISA operates at the page allocator and DMA mapping boundary,
existing clients require minimal changes. For example, the entire patch
integrating page_pool with PISA (patch 13/25) is just 3 lines:

  --- a/net/core/page_pool.c
  +++ b/net/core/page_pool.c
  @@ -16,6 +16,7 @@
   #include <net/page_pool/helpers.h>
   #include <net/page_pool/memory_provider.h>
   #include <net/xdp.h>
  +#include <linux/pisa.h>
  @@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(...)
   	if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
   		gfp |= __GFP_NOWARN;
  +	if (dma_map)
  +		gfp |= pisa_rx_gfp(pool->p.dev);

Similarly:
- Coherent rings/queues (dma_alloc_attrs()) are served from the per-node
  order-0 PISA allocator when dev->pisa_coherent is enabled; no driver
  changes needed.
- Socket TX page fragments (skb_page_frag_refill()) use a per-task
  page_frag fed by __GFP_PISA_RO when the egress device has pisa_tx_bufs
  enabled; no driver changes needed.
- Small TX skb heads (kmalloc_reserve()) use a parallel
  skbuff_small_head_pisa SLUB cache backed by zeroed, read-only PISA
  folios (SLAB_CACHE_PISA_RO) when pisa_tx_skb_heads is enabled. This
  eliminates the per-driver TX header bounce-buffer patches from v1.
- TX MSG_ZEROCOPY, O_DIRECT, and buffered disk I/O migrates or allocates
  order-0 user and page-cache folios from per-node PISA allocators so
  zero-copy network sends and storage I/O also use 2MB IOMMU mappings.

All features are strictly opt-in per device via
/sys/devices/.../pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads} (and
pisa.user_max_4K_pages for user/page-cache folios). Until a device opts in,
PISA is gated off by a static key and allocates no metadata.

=== CHANGES SINCE v1 [0]

I have make use and applied several comments and suggestions received on
the previous version. The changes are massive so perhaps it makes sense
to look at the new code in isolation, unless you have already revised
in detail the previous version.

A high level summary is the following:

- Renamed DMA_PMD to PISA (CONFIG_PISA, include/linux/pisa.h,
  drivers/iommu/pisa-*.c, /sys/devices/.../pisa_*).

- Hooked __GFP_PISA_RO and __GFP_PISA_RW into __alloc_frozen_pages_noprof()
  and alloc_pages_bulk_noprof(), replacing custom pool calls in page_pool
  and socket TX refill.

- Added lockless per-CPU block caches (pisa_enable_pcp()) in front of the
  per-node order-0 RW and SKB_FRAG_PAGE_ORDER RO allocators.

- Added SLAB_CACHE_PISA_RO in SLUB and skbuff_small_head_pisa for small
  TX skb heads, dropping the per-driver TX header bounce-buffer patches.

- Added PISA support for MSG_ZEROCOPY user pages (madvise(MADV_PISA) and
  automatic upgrade on send) and block/page-cache I/O (O_DIRECT and
  buffered reads/writes), plus age/waste-based compaction and scrub
  migration (pisa.wasted_pct, pisa.old_ms, pisa.scrub_migrate).

- Made metadata table initialization and static key activation lazy on
  first device opt-in.

=== PERFORMANCE BENEFITS

Your mileage may vary. Enabling the IOMMU may have no throughput impact
until system components (bus, IOTLB, CPU) become overloaded. Aside from
throughput reduction, one key metric is IOTLB effectiveness. Below are SMMU
performance counters on a large ARM64 system with 2x200G NICs running
bidirectional traffic:

    === DMA-FQ MODE (total throughput ~440Gbps)
    25,757,488      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
    16,634,941      smmuv3_pmcg_*/event=0x81/   IOTLB misses

    === PISA on top of strict DMA (total throughput ~745Gbps)
    22,693,218      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
        30,536      smmuv3_pmcg_*/event=0x81/   IOTLB misses

The 2MB mappings make IOTLB misses almost non-existent because the working
set of IOTLB entries is reduced by a factor of ~512.

=== NOTES

[0] v1 (DMA_PMD):
    https://lore.kernel.org/all/20261003212241.3432303-1-lrizzo@google.com/

[1] IOMMU IMPACT AND IOTLB THRASHING
Documented in detail in Documentation/core-api/pisa.rst. In short, there
are three main costs with the IOMMU:
- CPU cost for dma_map/unmap
- CPU cost and latency for synchronous IOTLB flush (required for security)
- IOTLB thrashing when the working set of 4KB pages exceeds IOTLB capacity,
  causing IOMMU page walks that backpressure PCIe/bus traffic (30..60%
  throughput drop on high-speed NICs). See also:
  https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/

[2] WHY NOT USE PAGE_POOL FOR EVERYTHING
page_pool is a consumer of PISA (via __GFP_PISA_RW), not a replacement:
- page_pool does not split 2MB pages into order-N blocks or manage shared
  2MB IOMMU mappings across domains;
- page_pool pages cannot be managed by arbitrary get_page()/put_page()
  callers (socket TX fragments, SLUB slabs, user/page-cache folios);
- page_pool requires a single NAPI consumer with BH disabled and a known
  device at allocation time, whereas socket TX, SLUB, and dma_alloc_attrs()
  run in process/preemptible context across arbitrary devices.

Luigi Rizzo (25):
  iommu/dma: Introduce CONFIG_PISA and metadata table
  iommu/dma: Add PISA allocator lifecycle and page recycle hook
  iommu/dma: Add per-NUMA-node PISA page reservoir
  iommu/dma: Add PISA block allocation
  iommu/dma: Add global cap and shrinker for PISA pages
  iommu/dma: Reserve a per-domain IOVA window for PISA pages
  iommu/dma: Release PISA domain mappings on domain teardown
  iommu/dma: Use per-domain IOVA window to map PISA memory
  driver core: Add per-device pisa_* sysfs attributes for PISA optin
  iommu/dma: Use PISA for coherent DMA allocations
  iommu/dma: Add per-CPU block caches to PISA allocators
  net/core: Use per-node PISA allocators for skb_page_frag_refill()
  net/core: Use PISA for page_pool memory
  iommu/dma: Support decrypted PISA pages
  iommu/dma: Add background page scrubber for PISA allocators
  mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
  net: Allocate small skb heads from PISA
  iommu/dma: Migrate user pages into PISA for TX zerocopy
  iommu/dma: Stop recycling old PISA pages when allocator waste is high
  net/libeth: Use GFP_KERNEL when pre-filling RX rings
  net/gve: Use GFP_KERNEL when pre-filling RX rings
  net/gve: Use PISA memory for RX buffers
  block, mm, iommu/dma: Support PISA for disk I/O and scrub migration
  Documentation: Add PISA architecture documentation
  iommu/dma: Add PISA statistics and debugfs

 .../admin-guide/kernel-parameters.txt         |    6 +
 Documentation/core-api/index.rst              |    1 +
 Documentation/core-api/pisa.rst               |  426 ++++
 block/bio.c                                   |   46 +
 drivers/base/core.c                           |   78 +
 drivers/iommu/Kconfig                         |   34 +
 drivers/iommu/Makefile                        |    5 +
 drivers/iommu/dma-iommu.c                     |   48 +-
 drivers/iommu/dma-iommu.h                     |   16 +
 drivers/iommu/iommu.c                         |   10 +-
 drivers/iommu/pisa-alloc.c                    | 2063 +++++++++++++++++
 drivers/iommu/pisa-kunit.c                    |  669 ++++++
 drivers/iommu/pisa-map.c                      |  553 +++++
 drivers/iommu/pisa-meta.c                     |  300 +++
 drivers/iommu/pisa-node.c                     |  378 +++
 drivers/iommu/pisa-priv.h                     |  511 ++++
 drivers/iommu/pisa-user.c                     |  662 ++++++
 .../ethernet/google/gve/gve_buffer_mgmt_dqo.c |    7 +-
 drivers/net/ethernet/google/gve/gve_rx.c      |   40 +-
 include/linux/device.h                        |   27 +
 include/linux/gfp_types.h                     |   17 +-
 include/linux/netdevice.h                     |    4 +
 include/linux/pisa.h                          |  367 +++
 include/linux/sched.h                         |    4 +
 include/linux/skbuff.h                        |    2 +
 include/linux/slab.h                          |   10 +
 include/linux/swap.h                          |    1 +
 include/net/libeth/rx.h                       |   11 +-
 include/net/sock.h                            |  143 ++
 include/trace/events/mmflags.h                |   11 +-
 include/uapi/asm-generic/mman-common.h        |    2 +
 kernel/dma/direct.c                           |   14 +-
 kernel/exit.c                                 |    2 +
 kernel/fork.c                                 |    2 +
 mm/filemap.c                                  |   15 +-
 mm/madvise.c                                  |   11 +
 mm/page-writeback.c                           |    8 +
 mm/page_alloc.c                               |   59 +
 mm/readahead.c                                |    5 +-
 mm/slab.h                                     |    3 +-
 mm/slab_common.c                              |    5 +-
 mm/slub.c                                     |   14 +-
 net/core/datagram.c                           |   85 +-
 net/core/page_pool.c                          |    3 +
 net/core/skbuff.c                             |   42 +-
 net/core/sock.c                               |   62 +-
 net/ipv4/esp4.c                               |    4 +-
 net/ipv4/ip_output.c                          |   19 +-
 net/ipv4/tcp.c                                |   17 +-
 net/ipv4/tcp_output.c                         |   12 +-
 net/ipv6/esp6.c                               |    4 +-
 net/ipv6/ip6_output.c                         |   19 +-
 52 files changed, 6784 insertions(+), 73 deletions(-)
 create mode 100644 Documentation/core-api/pisa.rst
 create mode 100644 drivers/iommu/pisa-alloc.c
 create mode 100644 drivers/iommu/pisa-kunit.c
 create mode 100644 drivers/iommu/pisa-map.c
 create mode 100644 drivers/iommu/pisa-meta.c
 create mode 100644 drivers/iommu/pisa-node.c
 create mode 100644 drivers/iommu/pisa-priv.h
 create mode 100644 drivers/iommu/pisa-user.c
 create mode 100644 include/linux/pisa.h

-- 
2.56.0.542.g3ef89b7020-goog