From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.208.72 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794248740; b=I4M9OwKAhBjIHFlTzizDUHO7WBxUSVX7o+iWClQ5OUqDGZl+/Bc7Y59HZt9BNIeBeOOS xQ0g9mPmptp8DfkOd3JOo1bkc0fnkxSQpPKHHz0yFjaf4clpWwJrwjJXjH3ud59WmTrU5 mrL/mOb0I84l1l+q5edI+61vIk/oGfmASjAz00eB/qYpwi5DZNYhnlm/2AvuyAWqXZKtd 42rXI11LlufPmXuDYFQsZA8fZ1GfUUTDYXfar1pdaLjDDv+nasw9+rJARFpdGKnk8TzTl k45tC+tFZPf6j1GhbPKNZjoidbZ+9q1AE/xPY0o7tRLgsCmbX2yP6FA+8zP/EvK2+5g== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794248740; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=yXbpKNTutM+q8EpPl46Tft0Cw/JJ6ViVyA1KiMY2C84=; b=tW0sf05BBsh/1kWglM59xNtymQ8pK3Dr014MJ6qdIY7dTTN9wLge/Fe5lrO0V9DlV4Xs P9rsYuaUBzHRe+FLUSDvxHL5IDk6fdu5je8xDiuNr0DHGxrPP/4l5OwmVZoN1Ze+4Ya7f PZ0xzw/Hfi3UYvRIp/s74urg6JQNYf1O0WbjsFeaRZljnYJMKSaMFAV4ipECh4vSh8ykO nuYKq6h957JbRTlJ1JoaHINFuW423ZwY1sJa6ldu4qWZbnSUrVhcA3SBW4CokT3mEJ3yE x9XYoypdqkGkA7UP36Zp87g3vsu/y4Oh5f7bWjX/Q4mh2oHsT/VhwMqHFWcjALO2HAQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=Ej1oO1S0; arc=none smtp.remote-ip=209.85.208.72 Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2273606A332 for ; Mon, 9 Nov 2026 18:25:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org D2273606A332 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=Ej1oO1S0 Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-6b42827bb4bso820908a12.1 for ; Mon, 09 Nov 2026 10:25:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1794248737; x=1794853537; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=yXbpKNTutM+q8EpPl46Tft0Cw/JJ6ViVyA1KiMY2C84=; b=Ej1oO1S0w1d5G5L/0Q1NFkUNsUmb+jkTFezTc5catOvTSjkW5Ano6ghByl4/xx2aqY aVYiNADDKhdZsTzF6c3jOVInPyZXnqpieqfIENSO7bnQNjMYc4olx3xg1SY08d5Fb6J9 ORePJpQrJKzEx1KoCoAXv8gL093OaCdzmbFp8pHFDhsJNGLuL4D3LHpK8sWIqJnjGlwe zIiXFSFKXI0dqUX+JhP1yNn1vZHMxCm2T6ETw++SfFyri8JWpGE0Ve+e18w6mqwUk4dz DUctTq6HHXUAnse22kC4ouZKKJjZWS6GBp4KF7Z30xgTs4XqWbEeIOdwe5Tl6RoSI+/q fQ1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794248737; x=1794853537; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=yXbpKNTutM+q8EpPl46Tft0Cw/JJ6ViVyA1KiMY2C84=; b=01HOLsEVUAo/jP1Jl//DqMnhMUYj2zGCBa2hbOwc0fXG8GPdmnzg42GViwVsSaJwXh 3+G78ur/CJ6SjMV+ZmSC7/Z6wI22ZCDUdBt6IyEBTCEMPGBtQALhzhHmxsd2+YmAWStS qLvapowuYtzY3aDRa+WcKKIeEyTNapIEolUzSCQFbr9yiRxndRM/I5dwn7AUd3B8Zwut JSwPzSH1gLf82gygXAS39c9NxahtWedZ5rCWaWK4HMLelm2jiuOLuhfrvu3k13dP+eeg SSNcMNbXmp1CFSoEzXj5jrnJ1z1lzDQeKenrfs2hZYUcFVcKWE9H6j0D1IB/edDu103I OKrA== X-Forwarded-Encrypted: i=1; AL+hErEP1G1A5epvq3JL98zXle9xa5rSiYl7e4dSEaIaXLHx8XKNcTfe93gtJ/EAnGggJOT0xzQM5IvSpQ8WG30=@vger.kernel.org X-Gm-Message-State: APuU4Z/SpP9d4k9W4x2nUeygLvqsiV8Vo+OXFunoLooRAPvTmtHyjQ1r DkPA3HLS/hH98yOrPC5W8auE/PJHLolR2COkRvXlPO5KlkX7LfmELFqLDEX2KyvBPRBS2808Rn+ xLUoS/Q== X-Received: from edsu12.prod.google.com ([2002:aa7:d98c:0:b0:6ac:c4fc:4feb]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:234a:b0:6ac:8ef4:177d with SMTP id 4fb4d7f45d1cf-6b756ad59a4mr348040a12.20.1794248736768; Mon, 09 Nov 2026 10:25:36 -0800 (PST) Date: Mon, 9 Nov 2026 18:25:08 +0000 In-Reply-To: <20261003212241.3432303-1-lrizzo@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261003212241.3432303-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.0.542.g3ef89b7020-goog Message-ID: <20261109182533.3594984-1-lrizzo@google.com> Subject: [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo From: Luigi Rizzo To: Luigi Rizzo , Joerg Roedel , Will Deacon , Robin Murphy , Christoph Hellwig , Marek Szyprowski , Andrew Morton , Vlastimil Babka , David Hildenbrand , Jens Axboe , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Matthew Wilcox , Lorenzo Stoakes , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Alexander Lobakin , iommu@lists.linux.dev, netdev@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, driver-core@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" This is v2 of the RFC previously posted as DMA_PMD [0] (renamed to avoid confusion with CPU page table PMD levels). PISA stands for "Pre-mapped IOVA Superpage Allocator" and as the name says, it is a specialized memory allocator. Invoked via __GFP_PISA* flag to alloc_pages(), it returns pages ("PISA memory") that make DMA-related operations on that memory a lot more efficient, both in software and in hardware. Specifically, PISA memory has the following properties: - it is backed by 2MB contiguous physical pages, so the IOMMU can use large leaf mappings and significantly reduce IOTLB pressure see [1] - IOMMU mappings are recycled while PISA memory is used for IO buffers, thus greatly reducing IOVA allocation and dma map/unmap costs. This includes cases (e.g. overcommitted VMs or Confidential Compute) where they would trigger expensive VM exits or bounce buffer copies. - PISA memory is returned to the buddy allocator in background, and only after all existing IOMMU and IOTLB mappings have been flushed. This removes a significant security problem in DMA-FQ, which instead returns pages before flushing the IOTLB, without the performance penalty of strict iommu (DMA) where the flushes are synchronous. The pitch line remains: PISA has the performance of identity mapping, guarantees that only active I/O buffers have IOMMU/IOTLB mappings (unlike DMA-FQ), removes bounce-buffer copies in Confidential Computing VMs, and integrates transparently with existing kernel subsystems and drivers. === KEY IDEA: A SPECIALIZED MEMORY ALLOCATOR For avoidance of doubt, PISA is NOT a new buffer cache, page_pool replacement, slab allocator, or custom netdev feature. The kernel already has plenty of specialized caches and pools (page_pool, socket page_frags, SLUB, page cache) built on top of the buddy allocator. Instead, PISA sits *below* those existing subsystems as a specialized memory provider modeled after alloc_pages() and __free_pages_prepare(). It enforces a few properties on the memory it dispenses so that IOMMU mapping, unmapping, and CoCo page decryption become nearly free: 1. PISA allocates order-N blocks out of physically contiguous 2MB pages that are mapped once per IOMMU domain using 2MB IOMMU leaf entries in a reserved per-domain IOVA window (and decrypted once upfront in CoCo guests). Subsequent dma_map_*() calls resolve the IOVA with a simple window offset calculation, and dma_unmap_*() is a no-op. 2. Freed blocks are intercepted in __free_pages_prepare() and recycled back into their PISA allocator while their 2MB IOMMU mappings remain active. A 2MB page is returned to the buddy allocator only when all of its blocks are idle, its IOMMU mappings have been removed from all domains, a synchronous IOTLB flush has completed, and an RCU grace period has elapsed. 3. Callers allocate from per-NUMA-node PISA allocators using standard page allocator calls with __GFP_PISA_RW (DMA_BIDIRECTIONAL) or __GFP_PISA_RO (DMA_TO_DEVICE, mapped IOMMU_READ-only so devices cannot overwrite shared/adjacent data), falling back to the buddy allocator on failure. Because PISA operates at the page allocator and DMA mapping boundary, existing clients require minimal changes. For example, the entire patch integrating page_pool with PISA (patch 13/25) is just 3 lines: --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -16,6 +16,7 @@ #include #include #include +#include @@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(...) if ((gfp & GFP_ATOMIC) == GFP_ATOMIC) gfp |= __GFP_NOWARN; + if (dma_map) + gfp |= pisa_rx_gfp(pool->p.dev); Similarly: - Coherent rings/queues (dma_alloc_attrs()) are served from the per-node order-0 PISA allocator when dev->pisa_coherent is enabled; no driver changes needed. - Socket TX page fragments (skb_page_frag_refill()) use a per-task page_frag fed by __GFP_PISA_RO when the egress device has pisa_tx_bufs enabled; no driver changes needed. - Small TX skb heads (kmalloc_reserve()) use a parallel skbuff_small_head_pisa SLUB cache backed by zeroed, read-only PISA folios (SLAB_CACHE_PISA_RO) when pisa_tx_skb_heads is enabled. This eliminates the per-driver TX header bounce-buffer patches from v1. - TX MSG_ZEROCOPY, O_DIRECT, and buffered disk I/O migrates or allocates order-0 user and page-cache folios from per-node PISA allocators so zero-copy network sends and storage I/O also use 2MB IOMMU mappings. All features are strictly opt-in per device via /sys/devices/.../pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads} (and pisa.user_max_4K_pages for user/page-cache folios). Until a device opts in, PISA is gated off by a static key and allocates no metadata. === CHANGES SINCE v1 [0] I have make use and applied several comments and suggestions received on the previous version. The changes are massive so perhaps it makes sense to look at the new code in isolation, unless you have already revised in detail the previous version. A high level summary is the following: - Renamed DMA_PMD to PISA (CONFIG_PISA, include/linux/pisa.h, drivers/iommu/pisa-*.c, /sys/devices/.../pisa_*). - Hooked __GFP_PISA_RO and __GFP_PISA_RW into __alloc_frozen_pages_noprof() and alloc_pages_bulk_noprof(), replacing custom pool calls in page_pool and socket TX refill. - Added lockless per-CPU block caches (pisa_enable_pcp()) in front of the per-node order-0 RW and SKB_FRAG_PAGE_ORDER RO allocators. - Added SLAB_CACHE_PISA_RO in SLUB and skbuff_small_head_pisa for small TX skb heads, dropping the per-driver TX header bounce-buffer patches. - Added PISA support for MSG_ZEROCOPY user pages (madvise(MADV_PISA) and automatic upgrade on send) and block/page-cache I/O (O_DIRECT and buffered reads/writes), plus age/waste-based compaction and scrub migration (pisa.wasted_pct, pisa.old_ms, pisa.scrub_migrate). - Made metadata table initialization and static key activation lazy on first device opt-in. === PERFORMANCE BENEFITS Your mileage may vary. Enabling the IOMMU may have no throughput impact until system components (bus, IOTLB, CPU) become overloaded. Aside from throughput reduction, one key metric is IOTLB effectiveness. Below are SMMU performance counters on a large ARM64 system with 2x200G NICs running bidirectional traffic: === DMA-FQ MODE (total throughput ~440Gbps) 25,757,488 smmuv3_pmcg_*/event=0x80/ IOTLB lookups 16,634,941 smmuv3_pmcg_*/event=0x81/ IOTLB misses === PISA on top of strict DMA (total throughput ~745Gbps) 22,693,218 smmuv3_pmcg_*/event=0x80/ IOTLB lookups 30,536 smmuv3_pmcg_*/event=0x81/ IOTLB misses The 2MB mappings make IOTLB misses almost non-existent because the working set of IOTLB entries is reduced by a factor of ~512. === NOTES [0] v1 (DMA_PMD): https://lore.kernel.org/all/20261003212241.3432303-1-lrizzo@google.com/ [1] IOMMU IMPACT AND IOTLB THRASHING Documented in detail in Documentation/core-api/pisa.rst. In short, there are three main costs with the IOMMU: - CPU cost for dma_map/unmap - CPU cost and latency for synchronous IOTLB flush (required for security) - IOTLB thrashing when the working set of 4KB pages exceeds IOTLB capacity, causing IOMMU page walks that backpressure PCIe/bus traffic (30..60% throughput drop on high-speed NICs). See also: https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/ [2] WHY NOT USE PAGE_POOL FOR EVERYTHING page_pool is a consumer of PISA (via __GFP_PISA_RW), not a replacement: - page_pool does not split 2MB pages into order-N blocks or manage shared 2MB IOMMU mappings across domains; - page_pool pages cannot be managed by arbitrary get_page()/put_page() callers (socket TX fragments, SLUB slabs, user/page-cache folios); - page_pool requires a single NAPI consumer with BH disabled and a known device at allocation time, whereas socket TX, SLUB, and dma_alloc_attrs() run in process/preemptible context across arbitrary devices. Luigi Rizzo (25): iommu/dma: Introduce CONFIG_PISA and metadata table iommu/dma: Add PISA allocator lifecycle and page recycle hook iommu/dma: Add per-NUMA-node PISA page reservoir iommu/dma: Add PISA block allocation iommu/dma: Add global cap and shrinker for PISA pages iommu/dma: Reserve a per-domain IOVA window for PISA pages iommu/dma: Release PISA domain mappings on domain teardown iommu/dma: Use per-domain IOVA window to map PISA memory driver core: Add per-device pisa_* sysfs attributes for PISA optin iommu/dma: Use PISA for coherent DMA allocations iommu/dma: Add per-CPU block caches to PISA allocators net/core: Use per-node PISA allocators for skb_page_frag_refill() net/core: Use PISA for page_pool memory iommu/dma: Support decrypted PISA pages iommu/dma: Add background page scrubber for PISA allocators mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches net: Allocate small skb heads from PISA iommu/dma: Migrate user pages into PISA for TX zerocopy iommu/dma: Stop recycling old PISA pages when allocator waste is high net/libeth: Use GFP_KERNEL when pre-filling RX rings net/gve: Use GFP_KERNEL when pre-filling RX rings net/gve: Use PISA memory for RX buffers block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Documentation: Add PISA architecture documentation iommu/dma: Add PISA statistics and debugfs .../admin-guide/kernel-parameters.txt | 6 + Documentation/core-api/index.rst | 1 + Documentation/core-api/pisa.rst | 426 ++++ block/bio.c | 46 + drivers/base/core.c | 78 + drivers/iommu/Kconfig | 34 + drivers/iommu/Makefile | 5 + drivers/iommu/dma-iommu.c | 48 +- drivers/iommu/dma-iommu.h | 16 + drivers/iommu/iommu.c | 10 +- drivers/iommu/pisa-alloc.c | 2063 +++++++++++++++++ drivers/iommu/pisa-kunit.c | 669 ++++++ drivers/iommu/pisa-map.c | 553 +++++ drivers/iommu/pisa-meta.c | 300 +++ drivers/iommu/pisa-node.c | 378 +++ drivers/iommu/pisa-priv.h | 511 ++++ drivers/iommu/pisa-user.c | 662 ++++++ .../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 7 +- drivers/net/ethernet/google/gve/gve_rx.c | 40 +- include/linux/device.h | 27 + include/linux/gfp_types.h | 17 +- include/linux/netdevice.h | 4 + include/linux/pisa.h | 367 +++ include/linux/sched.h | 4 + include/linux/skbuff.h | 2 + include/linux/slab.h | 10 + include/linux/swap.h | 1 + include/net/libeth/rx.h | 11 +- include/net/sock.h | 143 ++ include/trace/events/mmflags.h | 11 +- include/uapi/asm-generic/mman-common.h | 2 + kernel/dma/direct.c | 14 +- kernel/exit.c | 2 + kernel/fork.c | 2 + mm/filemap.c | 15 +- mm/madvise.c | 11 + mm/page-writeback.c | 8 + mm/page_alloc.c | 59 + mm/readahead.c | 5 +- mm/slab.h | 3 +- mm/slab_common.c | 5 +- mm/slub.c | 14 +- net/core/datagram.c | 85 +- net/core/page_pool.c | 3 + net/core/skbuff.c | 42 +- net/core/sock.c | 62 +- net/ipv4/esp4.c | 4 +- net/ipv4/ip_output.c | 19 +- net/ipv4/tcp.c | 17 +- net/ipv4/tcp_output.c | 12 +- net/ipv6/esp6.c | 4 +- net/ipv6/ip6_output.c | 19 +- 52 files changed, 6784 insertions(+), 73 deletions(-) create mode 100644 Documentation/core-api/pisa.rst create mode 100644 drivers/iommu/pisa-alloc.c create mode 100644 drivers/iommu/pisa-kunit.c create mode 100644 drivers/iommu/pisa-map.c create mode 100644 drivers/iommu/pisa-meta.c create mode 100644 drivers/iommu/pisa-node.c create mode 100644 drivers/iommu/pisa-priv.h create mode 100644 drivers/iommu/pisa-user.c create mode 100644 include/linux/pisa.h -- 2.56.0.542.g3ef89b7020-goog