* [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:44 ` Randy Dunlap
2026-10-03 21:22 ` [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook Luigi Rizzo
` (49 subsequent siblings)
50 siblings, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
DMA_PMD is a new subsystem to back IO buffers with physically contiguous
PMD_SIZE pages, allowing the use of larger leaves in the IOMMU. This
significantly reduces IOTLB pressure, with great performance benefits. It
support very common cases (2MB PMD, !PREEMPT_RT). The 2MB backing pages
are dynamically allocated and released. DMA_PMD overhead is 128B of
metadata per physical page (64KB per 1GB, or 0.01% of physical RAM).
Add a CONFIG_DMA_PMD option to enable it only when applicable, and
introduce a sparse per-PMD metadata table to store the necessary side
information efficiently (similar in principle to pageblock_flags).
The table is initialized on first use, reserving KVA across the physical
address space up to iomem_resource.end (capped at MAX_PHYSMEM_BITS,
64KB of KVA per 1GB of address space) along with a chunk-populated
bitmap (1 bit per 64MB chunk) for fast lockless lookups. Only chunks
covering online RAM are backed by physical pages at init; memory
hotplugged later within the reserved range is populated on demand when
allocated by a DMA_PMD pool, and safely declined otherwise.
Also add a built-in KUnit test suite (CONFIG_DMA_PMD_META_KUNIT_TEST)
exercising initialization idempotency, PFN/physical round-trip lookups,
out-of-bounds physical address handling, and DMA_PMD membership toggling.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
Documentation/core-api/dma-pmd.rst | 320 +++++++++++++++++++++++++
Documentation/core-api/index.rst | 1 +
drivers/iommu/Kconfig | 32 +++
drivers/iommu/Makefile | 2 +
drivers/iommu/dma-pmd-kunit.c | 102 ++++++++
drivers/iommu/dma-pmd-meta.c | 365 +++++++++++++++++++++++++++++
drivers/iommu/dma-pmd-priv.h | 56 +++++
include/linux/dma-pmd.h | 36 +++
8 files changed, 914 insertions(+)
create mode 100644 Documentation/core-api/dma-pmd.rst
create mode 100644 drivers/iommu/dma-pmd-kunit.c
create mode 100644 drivers/iommu/dma-pmd-meta.c
create mode 100644 drivers/iommu/dma-pmd-priv.h
create mode 100644 include/linux/dma-pmd.h
diff --git a/Documentation/core-api/dma-pmd.rst b/Documentation/core-api/dma-pmd.rst
new file mode 100644
index 0000000000000..1d113b8c15876
--- /dev/null
+++ b/Documentation/core-api/dma-pmd.rst
@@ -0,0 +1,320 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=========================================================
+DMA_PMD: PMD_SIZE IOMMU mappings for DMA-coherent devices
+=========================================================
+
+Overview
+========
+
+DMA_PMD is a transparent enhancement of the current IOMMU modes, that
+gives the performance of identity mode, and like strict IOMMU mode
+(DMA), guarantees that memory will go back to the buddy allocator only
+when all IOMMU mappings have been removed and IOTLB flushed.
+
+The original design aimed at removing IOTLB thrashing, but its mechanism
+also give security guarantees comparable to strict IOMMU, and remove the
+bounce buffer overhead from Confidential Computing and preemptible VMs.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+ (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+ access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+ exceeds the IOTLB size, backpressuring the bus via flow control.
+ This is one of the biggest bottleneck for high speed devices:
+ typical network devices have very little IOVA locality, and that makes the IOTLB
+ almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA map/unmap
+ costs. This is common practice for command, completion and buffer
+ queues, as well as disk I/O and network receive buffers. Network
+ TX buffers are a noticeable exception. They normally come from
+ ``alloc_pages()`` calls unaware of the leaf device, and require
+ mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+ very expensive operation (up to several microseconds to wait for
+ completion, necessary for security), but the way it is implemented
+ in DMA-FQ compromises on security: IOTLB flush requests are run
+ periodically in batches, but the underlying memory is recycled without
+ waiting. During that window, the physical memory is still accessible,
+ opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+ blocks, e.g. 2MB (called PMD) instead of the usual 4KB (PTE). This is not
+ currently pursued by the kernel.
+
+DMA_PMD addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map, unmap, and free
+ device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB pages (PMD) and 2MB IOMMU
+ mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused DMA_PMD memory
+ to the system pool
+
+Execution is what makes DMA_PMD practical: the implementation intercepts
+a small set of kernel APIs so that device will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since DMA_PMD intercepts allocations of device-accessible
+memory, it gives two important benefits:
+- transparent pinning/unpining, required for preemptible VMs
+- transparent configuration of unencrypted mode, required for confidential computing
+
+Limitations
+===========
+
+DMA_PMD is currently limited to systems with ``PMD_SIZE`` equal to 2MB, which
+is the vast majority of existing platforms. It could be extended to larger
+pages without much effort. This is enforced at compile time.
+
+The implementation assumes non-preemptible spinlocks, so DMA_PMD is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+DMA_PMD can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers are always ``DMA_BIDIRECTIONAL``. This
+simplifies the handling of e.g. forwarding between different network
+interfaces.
+
+IOMMU mappings for queues are also ``DMA_BIDIRECTIONAL``, though stricter modes
+can be implemented trivially.
+
+Implementation details
+======================
+
+Most of the code size and design complexity relates to the handling
+of exceptional events (device teardown, memory hotplug) and to
+the safe release of pages through deferred tasks, RCU and grace
+periods.
+
+The implementation lives in ``drivers/iommu/dma-pmd-*.c`` and
+``include/linux/dma-pmd.h``, and relies on the components described below.
+We use the name "DMA_PMD" to refer to the second-level physically contiguous
+pages (typically 2MB) used for I/O memory.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, ``skb_page_frag_refill()``, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+
+and most devices will automatically inherit the benefits.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of DMA_PMD pages, based on either their Physical
+ Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to DMA_PMD page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+- support for preemptible VMs (pinned DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **DMA_PMD metadata** (``struct dma_pmd_meta``):
+
+ Identification of DMA_PMD pages could be done in principle with a flag in
+ ``struct page``, but that would not solve the management of per-page
+ metadata.
+
+ DMA_PMD addresses both requirements without depending on ``struct page``
+ using an idea similar to ``pageblock_flags``. Each 2MB page used as a
+ buffer requires only 256B of metadata, or 0.01% overhead. For metadata,
+ we reserve a sparse virtual memory region covering of approximate
+ size ``max_pfn * 256 / PMD_SIZE``, so the metadata can be accessed
+ directly using the 2MB page index. Metadata pages are populated on
+ demand, with a compact bitmap (1 bit per 32MB chunk, or 4KB per 1TB
+ of address space) tracking which metadata pages are backed, allowing
+ fast lockless validation and direct array indexing. Memory hotplugged
+ later is also supported.
+
+ ``struct dma_pmd_meta`` has flags to mark the page status (used as
+ DMA_PMD buffer, decrypted for confidential computing, or pinned against
+ memory compaction), and tracks which subpages are free, which domains
+ have an active IOMMU mapping for the page, and holds list/RCU linkage
+ to manage its lifetime.
+
+- **Per-domain reserved DMA_PMD IOVA ranges**:
+
+ Another key requirement is to quickly resolve the PA-IOVA mapping for a given domain. Since
+ a DMA_PMD page can be mapped in multiple domains (e.g. a network buffer used
+ to route packets between different NICs), it would be too expensive to manage random
+ PA-IOVA mappings created on the fly.
+ DMA_PMD reserves on each domain an IOVA range
+ covering the physical memory range plus an extra region, allowing a
+ fixed PA-IOVA offset, possibly different for each domain.
+ The IOVA allocation happens the first time a domain is used, hence a
+ IOVA within the reserved range will uniquely identify a DMA_PMD page.
+
+- **alloc_pages() compatible dma_pmd_pool allocator for I/O buffers**:
+
+ DMA_PMD implements a dma_pmd_pool object for fixed-order page allocations
+ that is a natural fit for the allocation of NIC receive buffers and tx
+ socket buffers. The API is similar to ``alloc_pages()`` (in fact, it is an
+ almost direct replacement) and dma_pmd_pools are instantiated as follow:
+ - per NIC-receive-queue (or page_pool) to provide order-0 pages to each queue
+ - per-CPU to provide order-0 and order-3 pages to refill tx socket buffers.
+
+- **dma_pmd_arena allocator to back dma_alloc_attrs()**
+
+ Devices use ``dma_alloc_attrs()`` or variants for long-lived, variable size
+ allocations to back device queues and header buffers. dma_pmd_arena
+ has the same functions: it both allocates and maps memory in the
+ reserved IOVA, and is used exclusively within dma_alloc_attrs() to
+ handle suitable requests (4K or larger, GFP_KERNEL, for devices that
+ specifically enable DMA_PMD)
+
+
+- **Fine grained control**
+
+ Both for experimentation and production use, it is useful to have
+ some form of control on which devices want to use DMA_PMD and for
+ what. DMA_PMD exposes the following controls. Keep in mind that full
+ performance can only be achieved if all regions are mapped using DMA_PMD.
+
+ - /sys/devices/*/*/dma_pmd_{rings,rxbuf,tx_hdrs} per-device entries
+ that can be used within device drivers to decide whether to use DMA_PMD for each of its regions
+
+ - /proc/sys/net/core/tx_enable_dma_pmd controls whether tx socket buffers should use DMA_PMD
+
+- **Safe release of memory to the system**
+
+ A key feature of DMA_PMD is that memory is not released to the system
+ while there are active IOMMU mappings. Release of buffers is managed
+ as follows
+
+ 1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to DMA_PMD,
+ IOMMU mappings are still active, and they can be recycled
+
+ 2. when all blocks in a DMA_PMD page are unused, the page is moved to an
+ "idle" list, still with active mappings and available for reuse
+
+ 3. upon memory pressure or when above some global threshold, a shrinker
+ thread collects idle pages from all pools and starts removing the iommu
+ mappings and issues a synchronous IOTLB flush. The pages are not usable for allocations
+ but not returned to the pool yet
+
+ 4. once the previous step is complete, the system schedules ``call_rcu()``
+ and waits for an RCU grace period (and re-encrypts the page if it was
+ decrypted) before returning the 2MB page to the buddy allocator.
+
+- **Handling domain destruction**
+
+ When a device/domain is destroyed, all I/O issued by the device must have
+ been quiesced and memory released via ``__free_pages()`` or ``dma_free*()``.
+ DMA_PMD hooks into the domain destructor and removes all existing mappings
+ for the disappearing domain and synchronously flushes the IOTLB, similarly
+ to steps #3 and #4 shown before.
+
+- **Allocation and compound splitting**:
+
+ A pool that needs memory calls ``dma_pmd_add_page()`` to allocate a
+ physically contiguous 2MB page on the calling CPU's NUMA node, and
+ ``split_page_compound()`` to split it into independent blocks of
+ ``pool->order``. These blocks can be independently passed through the
+ networking stack.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+DMA_PMD integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of DMA_PMD memory is opt-in via per-device sysfs
+attributes or sysctls, while mapping, unmapping, and freeing automatically
+detect whether a buffer or IOVA belongs to DMA_PMD:
+
+1. **Streaming DMA map and unmap** (``dma_map_page_attrs()``,
+ ``dma_map_single_attrs()``, ``dma_map_sg_attrs()`` and
+ ``dma_unmap_page_attrs()``, ``dma_unmap_single_attrs()``,
+ ``dma_unmap_sg_attrs()`` via ``drivers/iommu/dma-iommu.c`` and
+ ``kernel/dma/direct.c``):
+
+ - On map, any buffer belonging to a DMA_PMD page (``dma_is_pmd_phys(phys)``)
+ is mapped via the domain's reserved ``DMA_PMD`` IOVA window with a simple
+ addition, typically reusing existing mappings.
+ Under ``dma-direct``, decrypted or pinned ``DMA_PMD`` pages
+ (``dma_is_pmd_direct(phys)``) can be mapped directly without ``swiotlb`` bounce.
+
+ On unmap, IOVAs can be identified as DMA_PMD buffers with a simple range check
+ and that results in a no-operation.
+
+2. **Coherent DMA allocation and free** (``dma_alloc_attrs()`` /
+ ``dma_alloc_coherent()`` and ``dma_free_attrs()`` / ``dma_free_coherent()``
+ in ``kernel/dma/mapping.c``):
+
+ - DMA_PMD is enabled per device via ``/sys/devices/.../dma_pmd_rings``
+ (``dev->dma_pmd_rings``). When set, sleepable ``GFP_KERNEL`` coherent
+ allocations are rounded up to 4KB and carved out of DMA_PMD buffers
+ ``dma_pmd_arena`` and mapped accordingly.
+ ``dma_free_attrs()`` automatically detects arena IOVAs via
+ ``dma_pmd_free()`` and returns the sub-blocks to the arena.
+
+3. **Page allocator release** (``__free_pages()`` / ``put_page()`` via
+ ``__free_pages_prepare()`` in ``mm/page_alloc.c``):
+
+ - ``dma_pmd_free_page()`` checks
+ ``dma_is_pmd_page(page_to_pfn(page))`` and recycles DMA_PMD subpages back to
+ their owning ``dma_pmd_pool`` instead of releasing them to the buddy
+ allocator while their 2MB IOMMU mappings remain active.
+
+4. **IOMMU domain teardown** (``iommu_put_dma_cookie()`` in
+ ``drivers/iommu/dma-iommu.c``):
+
+ - A call to ``dma_pmd_domain_release()`` before freeing a
+ domain's IOVA cookie and page tables clears cached domain state across
+ all pools. Existing IOMMU mappings and IOTLB flush has already happened.
+
+5. **Networking ``page_pool``**
+
+ - DMA_PMD is enabled per device via ``/sys/devices/.../dma_pmd_rxbuf``
+ (``dev->dma_pmd_rxbuf``). Any NIC driver using ``page_pool`` with this
+ flag set will back its RX buffers with a per-``page_pool``
+ ``dma_pmd_pool``, and opportunistically replaces plain 4KB pages when pooled 2MB
+ blocks are available.
+
+6. **Socket TX page fragments** (``skb_page_frag_refill()`` in
+ ``net/core/sock.c``):
+
+ - When enabled globally via the ``net.core.tx_enable_dma_pmd`` sysctl
+ socket TX fragments are allocated from per-CPU ``dma_pmd_pool`` instances.
+
+7. **NIC driver RX pools and TX header bounce buffers**
+
+ - **RX buffers:** Controlled per device by ``/sys/devices/.../dma_pmd_rxbuf``
+ (``dev->dma_pmd_rxbuf``) for queue modes managing their own page rings
+ (``idpf``, ``gve`` GQI-RDA, ``gq``).
+ - **TX headers:** Controlled per device by
+ ``/sys/devices/.../dma_pmd_tx_hdrs`` (``dev->dma_pmd_tx_hdrs``),
+ copying linear packet headers into pre-allocated coherent bounce buffers
+ (backed by ``dma_pmd_arena`` when ``dma_pmd_rings`` is also enabled).
+
+8. **Global pool limit and observability:**
+
+ - ``/sys/module/kernel/parameters/dma_pmd_max_pages``: Global cap on 2MB
+ pages held across all pools (defaults to 1/8th of physical RAM).
+ - ``/sys/kernel/debug/dma_pmd/pools``: Debugfs summary of global and per-pool
+ allocation, mapping, and fallback counters.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..49608d2b7509b 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
dma-attributes
dma-isa-lpc
swiotlb
+ dma-pmd
mm-api
cgroup
genalloc
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 6e07bd69467a3..1bb347fd9da4a 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -158,6 +158,38 @@ config IOMMU_DMA
select NEED_SG_DMA_LENGTH
select NEED_SG_DMA_FLAGS if SWIOTLB
+# PMD_SIZE IOMMU backing for DMA-coherent devices
+config DMA_PMD
+ bool "PMD_SIZE IOMMU backing for DMA-coherent devices"
+ depends on IOMMU_DMA
+ depends on X86_64 || (ARM64 && ARM64_4K_PAGES)
+ depends on !PREEMPT_RT
+ default y
+ help
+ Hand out DMA buffers carved out of PMD_SIZE physically contiguous
+ blocks that the IOMMU maps with a single leaf PTE. Mapping,
+ unmapping and IOTLB flushing are done lazily to reduce CPU overhead.
+ The large mappings greatly reduce IOTLB usage. As in strict iommu
+ mode, pages are returned to the buddy allocator only when any existing
+ mappings have been removed and IOTLB flushed.
+ Costs 128KB of metadata per GB of DMA_PMD buffers, allocated on demand.
+
+ The config option enforces requirements (2MB PMD and !PREEMPT_RT) at compile
+ time. Other constraints (e.g. dma-coherent devices) are verified at runtime.
+
+ If unsure, say Y.
+
+config DMA_PMD_META_KUNIT_TEST
+ bool "KUnit test for DMA_PMD metadata table (built-in)" if !KUNIT_ALL_TESTS
+ depends on DMA_PMD && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ Builds KUnit unit tests for the sparse, per-PMD-frame metadata table,
+ PMD page pools, and coherent DMA arenas in drivers/iommu/dma-pmd*.c,
+ verifying metadata lookup, pool recycling, and arena block allocation.
+
+ If unsure, say N.
+
# Shared Virtual Addressing
config IOMMU_SVA
select IOMMU_MM_DATA
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 2f05725eaab18..2ad9b2eefd741 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,6 +11,8 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o
+obj-$(CONFIG_DMA_PMD_META_KUNIT_TEST) += dma-pmd-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
new file mode 100644
index 0000000000000..f80527eaf2e84
--- /dev/null
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -0,0 +1,102 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * KUnit tests for the opaque struct dma_pmd_meta table API.
+ */
+#include <kunit/test.h>
+#include <linux/gfp.h>
+#include <linux/dma-pmd.h>
+#include <linux/mm.h>
+
+#include "dma-pmd-priv.h"
+
+static void test_meta_init_and_roundtrip(struct kunit *test)
+{
+ struct dma_pmd_meta *m_pfn, *m_phys;
+ unsigned long pfn, base_pfn;
+ phys_addr_t pa, base_pa;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
+ /* Second call must be idempotent. */
+ KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
+
+ page = alloc_page(GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PMD_ORDER);
+ pa = page_to_phys(page);
+ base_pa = ALIGN_DOWN(pa, PMD_SIZE);
+
+ m_pfn = dma_pmd_meta_of_pfn(pfn);
+ m_phys = dma_pmd_meta_from_phys(pa);
+
+ KUNIT_EXPECT_PTR_EQ(test, m_pfn, m_phys);
+ KUNIT_EXPECT_EQ(test, dma_pmd_meta_to_pfn(m_pfn), base_pfn);
+ KUNIT_EXPECT_EQ(test, dma_pmd_meta_to_phys(m_phys), base_pa);
+
+ __free_page(page);
+}
+
+static void test_meta_invalid_phys(struct kunit *test)
+{
+ struct dma_pmd_meta *m;
+
+ KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
+
+ m = dma_pmd_meta_from_phys(PHYS_ADDR_MAX);
+ KUNIT_ASSERT_NOT_NULL(test, m);
+ KUNIT_EXPECT_EQ(test, dma_pmd_meta_to_phys(m), PHYS_ADDR_MAX);
+ KUNIT_EXPECT_FALSE(test, dma_is_pmd_page(ULONG_MAX >> PAGE_SHIFT));
+}
+
+static void dma_pmd_meta_set_pooled(struct dma_pmd_meta *m, bool pooled)
+{
+ WRITE_ONCE(m->pooled, pooled);
+}
+
+static void test_meta_pooled_toggle(struct kunit *test)
+{
+ unsigned long pfn, base_pfn;
+ struct dma_pmd_meta *m;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
+
+ page = alloc_pages(GFP_KERNEL, PMD_ORDER);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PMD_ORDER);
+ /* The buddy allocator returns naturally aligned blocks. */
+ KUNIT_EXPECT_EQ(test, pfn, base_pfn);
+ m = dma_pmd_meta_of_pfn(pfn);
+
+ KUNIT_EXPECT_FALSE(test, dma_is_pmd_page(pfn));
+
+ dma_pmd_meta_set_pooled(m, true);
+ KUNIT_EXPECT_TRUE(test, dma_is_pmd_page(base_pfn));
+ KUNIT_EXPECT_TRUE(test, dma_is_pmd_page(pfn));
+ KUNIT_EXPECT_TRUE(test, dma_is_pmd_page(base_pfn + (1UL << PMD_ORDER) - 1));
+
+ dma_pmd_meta_set_pooled(m, false);
+ KUNIT_EXPECT_FALSE(test, dma_is_pmd_page(pfn));
+
+ __free_pages(page, PMD_ORDER);
+}
+
+static struct kunit_case dma_pmd_meta_test_cases[] = {
+ KUNIT_CASE(test_meta_init_and_roundtrip),
+ KUNIT_CASE(test_meta_invalid_phys),
+ KUNIT_CASE(test_meta_pooled_toggle),
+ {}
+};
+
+static struct kunit_suite dma_pmd_meta_test_suite = {
+ .name = "dma_pmd_meta",
+ .test_cases = dma_pmd_meta_test_cases,
+};
+
+kunit_test_suite(dma_pmd_meta_test_suite);
+MODULE_DESCRIPTION("KUnit tests for struct dma_pmd_meta table");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/dma-pmd-meta.c b/drivers/iommu/dma-pmd-meta.c
new file mode 100644
index 0000000000000..fcaf5b121e66a
--- /dev/null
+++ b/drivers/iommu/dma-pmd-meta.c
@@ -0,0 +1,365 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * DMA_PMD sparse per-PMD-frame metadata table.
+ *
+ * See Documentation/core-api/dma-pmd.rst for the architecture overview.
+ */
+
+#include <linux/bitmap.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/dma-pmd.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/ioport.h>
+#include <linux/list.h>
+#include <linux/memblock.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/pgtable.h>
+#include <linux/spinlock.h>
+#include <linux/vmalloc.h>
+
+#include "dma-pmd-priv.h"
+
+/*
+ * Each PAGE_SIZE (4KB) metadata page holds (PAGE_SIZE >> DMA_PMD_META_SHIFT)
+ * struct dma_pmd_meta entries (32 entries of 128B, or 16 entries of 256B with
+ * spinlock debugging), each covering one PMD_SIZE (2MB) physical frame.
+ * One metadata page therefore covers a chunk of DMA_PMD_CHUNK_PAGES 4KB pages
+ * (64MB normally, or 32MB with spinlock debugging), i.e. order
+ * DMA_PMD_CHUNK_ORDER.
+ */
+#define DMA_PMD_CHUNK_ORDER (PMD_ORDER + PAGE_SHIFT - DMA_PMD_META_SHIFT)
+#define DMA_PMD_CHUNK_PAGES BIT(DMA_PMD_CHUNK_ORDER)
+
+static_assert(PAGE_SIZE >= DMA_PMD_META_SIZE);
+static_assert(PMD_SIZE == SZ_2M);
+static_assert(PMD_ORDER <= MAX_PAGE_ORDER);
+static_assert(PMD_SHIFT <= SUBSECTION_SHIFT);
+
+void *dma_pmd_meta_array __read_mostly;
+EXPORT_SYMBOL(dma_pmd_meta_array);
+unsigned long dma_pmd_meta_nframes __read_mostly;
+static unsigned long *dma_pmd_chunk_bitmap __read_mostly;
+
+static struct dma_pmd_meta dma_pmd_meta_nil;
+
+static unsigned long dma_pmd_meta_pages;
+static DEFINE_MUTEX(dma_pmd_meta_mutex);
+
+static __always_inline struct dma_pmd_meta *__dma_pmd_meta_of_pfn(unsigned long pfn)
+{
+ struct dma_pmd_meta *array = dma_pmd_meta_base();
+ unsigned long frame = pfn >> PMD_ORDER;
+
+ if (unlikely(!array || frame >= dma_pmd_meta_nframes ||
+ !test_bit_acquire(pfn >> DMA_PMD_CHUNK_ORDER, dma_pmd_chunk_bitmap)))
+ return &dma_pmd_meta_nil;
+
+ return &array[frame];
+}
+
+bool __dma_is_pmd_page(unsigned long pfn)
+{
+ return READ_ONCE(__dma_pmd_meta_of_pfn(pfn)->pooled);
+}
+EXPORT_SYMBOL(__dma_is_pmd_page);
+
+/**
+ * dma_pmd_meta_of_pfn - Metadata for a PFN known to be used by DMA_PMD.
+ * @pfn: PFN the caller already holds a struct page for
+ */
+struct dma_pmd_meta *dma_pmd_meta_of_pfn(unsigned long pfn)
+{
+ return __dma_pmd_meta_of_pfn(pfn);
+}
+EXPORT_SYMBOL(dma_pmd_meta_of_pfn);
+
+/**
+ * dma_pmd_meta_from_phys - Metadata for the PMD frame containing @pa
+ * @pa: Any physical address
+ *
+ * Safe against MMIO above max_pfn and PFNs inside physical holes: those
+ * resolve to the shared sink entry.
+ *
+ * Return: Pointer to metadata entry (never NULL).
+ */
+struct dma_pmd_meta *dma_pmd_meta_from_phys(phys_addr_t pa)
+{
+ return __dma_pmd_meta_of_pfn(PHYS_PFN(pa));
+}
+EXPORT_SYMBOL(dma_pmd_meta_from_phys);
+
+/**
+ * dma_pmd_meta_to_pfn - Base PFN of @m's PMD frame
+ * @m: Entry in @dma_pmd_meta_array
+ *
+ * Return: PMD-aligned base PFN, or ULONG_MAX for the sink.
+ */
+unsigned long dma_pmd_meta_to_pfn(const struct dma_pmd_meta *m)
+{
+ if (unlikely(m == &dma_pmd_meta_nil))
+ return ULONG_MAX;
+
+ return (unsigned long)(m - dma_pmd_meta_base()) << PMD_ORDER;
+}
+EXPORT_SYMBOL(dma_pmd_meta_to_pfn);
+
+/**
+ * dma_pmd_meta_to_phys - Base physical address of @m's PMD frame
+ * @m: Entry returned by dma_pmd_meta_of_pfn() or dma_pmd_meta_from_phys()
+ *
+ * Return: PMD-aligned physical address, or PHYS_ADDR_MAX for the sink.
+ */
+phys_addr_t dma_pmd_meta_to_phys(const struct dma_pmd_meta *m)
+{
+ if (unlikely(m == &dma_pmd_meta_nil))
+ return PHYS_ADDR_MAX;
+
+ return (phys_addr_t)(m - dma_pmd_meta_base()) << PMD_SHIFT;
+}
+EXPORT_SYMBOL(dma_pmd_meta_to_phys);
+
+/*
+ * Install one preallocated page. The page is allocated by the caller rather
+ * than here because apply_to_page_range() runs this callback under lazy-MMU
+ * mode with the pte level pinned, which is not a context to allocate from.
+ *
+ * @data points at the caller's page pointer and is cleared once the page has
+ * been consumed, so the caller can free it if it was not needed.
+ */
+static int dma_pmd_meta_set_pte(pte_t *ptep, unsigned long addr, void *data)
+{
+ struct page **pagep = data;
+ pte_t pte;
+
+ if (!pte_none(ptep_get(ptep)))
+ return 0;
+
+ pte = pfn_pte(page_to_pfn(*pagep), PAGE_KERNEL);
+
+ spin_lock(&init_mm.page_table_lock);
+ if (likely(pte_none(ptep_get(ptep)))) {
+ set_pte_at(&init_mm, addr, ptep, pte);
+ *pagep = NULL;
+ }
+ spin_unlock(&init_mm.page_table_lock);
+
+ return 0;
+}
+
+/* True iff @pfn is backed by online system RAM (excludes holes and ZONE_DEVICE). */
+static inline bool dma_pmd_pfn_online(unsigned long pfn)
+{
+ struct mem_section *ms;
+
+ if (unlikely(!pfn_valid(pfn)))
+ return false;
+
+ ms = __pfn_to_section(pfn);
+ if (unlikely(!online_section(ms)))
+ return false;
+
+ if (unlikely(online_device_section(ms) && is_zone_device_page(pfn_to_page(pfn))))
+ return false;
+
+ return true;
+}
+
+/**
+ * dma_pmd_meta_populate - back the array for [@start_pfn, @end_pfn)
+ * @array: base virtual address of the reservation
+ * @nframes: number of valid PMD frames in the reservation
+ * @start_pfn: first PFN of the range
+ * @end_pfn: one past the last PFN of the range
+ * @force: if true, populate every chunk unconditionally (hotplug / arena)
+ *
+ * Maps and zeroes one page of the array per chunk of the range that contains
+ * at least one online RAM PFN (or unconditionally when @force is set), and
+ * skips chunks that are already mapped. A chunk
+ * is DMA_PMD_CHUNK_PAGES of physical address space: 64MB normally, 32MB when
+ * spinlock debugging doubles the entry size.
+ * Locking: caller must hold @dma_pmd_meta_mutex.
+ *
+ * Return: 0, or -ENOMEM with the range partially backed.
+ */
+static int dma_pmd_meta_populate(struct dma_pmd_meta *array, unsigned long nframes,
+ unsigned long start_pfn, unsigned long end_pfn, bool force)
+{
+ unsigned long chunk, last, pfn;
+
+ lockdep_assert_held(&dma_pmd_meta_mutex);
+
+ end_pfn = min(end_pfn, nframes << PMD_ORDER);
+ if (start_pfn >= end_pfn)
+ return 0;
+
+ chunk = start_pfn >> DMA_PMD_CHUNK_ORDER;
+ last = (end_pfn - 1) >> DMA_PMD_CHUNK_ORDER;
+
+ for (; chunk <= last; chunk++) {
+ unsigned long addr, base = chunk << DMA_PMD_CHUNK_ORDER;
+ int ret, nid = NUMA_NO_NODE;
+ bool has_valid = false;
+ struct page *page;
+
+ addr = (unsigned long)array + (chunk << PAGE_SHIFT);
+ if (vmalloc_to_page((void *)addr)) {
+ set_bit(chunk, dma_pmd_chunk_bitmap);
+ continue; /* already backed */
+ }
+
+ if (force) {
+ has_valid = true;
+ } else {
+ for (pfn = base; pfn < base + DMA_PMD_CHUNK_PAGES;
+ pfn += 1UL << PMD_ORDER) {
+ if (dma_pmd_pfn_online(pfn)) {
+ has_valid = true;
+ nid = page_to_nid(pfn_to_page(pfn));
+ if (nid != NUMA_NO_NODE && node_online(nid))
+ break;
+ nid = NUMA_NO_NODE;
+ }
+ }
+ }
+ if (!has_valid)
+ continue; /* pure hole, leave it unmapped */
+
+ page = alloc_pages_node(nid, GFP_KERNEL | __GFP_ZERO, 0);
+ if (!page)
+ return -ENOMEM;
+
+ ret = apply_to_page_range(&init_mm, addr, PAGE_SIZE,
+ dma_pmd_meta_set_pte, &page);
+ if (ret) {
+ __free_page(page);
+ return ret;
+ }
+ if (page) {
+ /*
+ * Unreachable while dma_pmd_meta_mutex serialises
+ * every install, but kept so that the defensive
+ * pte_none() re-test in dma_pmd_meta_set_pte() can
+ * never leak the page it declined to consume.
+ */
+ __free_page(page);
+ set_bit(chunk, dma_pmd_chunk_bitmap);
+ continue;
+ }
+
+ flush_cache_vmap(addr, addr + PAGE_SIZE);
+ /* Pair with test_bit_acquire() in readers. */
+ smp_mb__before_atomic();
+ set_bit(chunk, dma_pmd_chunk_bitmap);
+ dma_pmd_meta_pages++;
+ }
+
+ return 0;
+}
+
+bool dma_pmd_meta_ensure_pfn(unsigned long pfn, bool can_block)
+{
+ int ret;
+
+ if (likely(__dma_pmd_meta_of_pfn(pfn) != &dma_pmd_meta_nil))
+ return true;
+
+ if (!can_block || !dma_pmd_meta_base() ||
+ (pfn >> PMD_ORDER) >= dma_pmd_meta_nframes)
+ return false;
+
+ mutex_lock(&dma_pmd_meta_mutex);
+ ret = dma_pmd_meta_populate(dma_pmd_meta_base(), dma_pmd_meta_nframes,
+ pfn, pfn + (1UL << PMD_ORDER), true);
+ mutex_unlock(&dma_pmd_meta_mutex);
+
+ return !ret;
+}
+EXPORT_SYMBOL(dma_pmd_meta_ensure_pfn);
+
+/**
+ * dma_pmd_meta_init - Reserve and populate the sparse per-PMD metadata array
+ *
+ * Populates all present RAM pages before publishing @dma_pmd_meta_array so
+ * no concurrent reader ever sees an unmapped entry.
+ *
+ * Return: 0 on success, or negative errno on failure.
+ */
+int dma_pmd_meta_init(void)
+{
+ unsigned long nframes, nchunks, size;
+ struct dma_pmd_meta *array;
+ struct vm_struct *vm;
+ int ret = 0;
+
+ if (likely(dma_pmd_meta_base()))
+ return 0;
+
+ mutex_lock(&dma_pmd_meta_mutex);
+ if (dma_pmd_meta_array)
+ goto out_unlock;
+
+ nframes = DIV_ROUND_UP(max3((unsigned long)max_pfn,
+ (unsigned long)max_possible_pfn,
+ (unsigned long)min_t(u64, iomem_resource.end >> PAGE_SHIFT,
+ 1ULL << (MAX_PHYSMEM_BITS - PAGE_SHIFT))),
+ 1UL << PMD_ORDER);
+ size = PAGE_ALIGN(nframes << DMA_PMD_META_SHIFT);
+ nchunks = size >> PAGE_SHIFT;
+
+ dma_pmd_chunk_bitmap = bitmap_zalloc(nchunks, GFP_KERNEL);
+ if (!dma_pmd_chunk_bitmap) {
+ ret = -ENOMEM;
+ goto out_unlock;
+ }
+
+ vm = get_vm_area(size, VM_MAP);
+ if (!vm) {
+ bitmap_free(dma_pmd_chunk_bitmap);
+ dma_pmd_chunk_bitmap = NULL;
+ ret = -ENOMEM;
+ goto out_unlock;
+ }
+ array = vm->addr;
+
+ ret = dma_pmd_meta_populate(array, nframes, 0, max_pfn, false);
+ if (ret) {
+ struct page *p, *next;
+ unsigned long chunk;
+ LIST_HEAD(pages);
+
+ for_each_set_bit(chunk, dma_pmd_chunk_bitmap, nchunks) {
+ p = vmalloc_to_page((void *)array + (chunk << PAGE_SHIFT));
+ if (p)
+ list_add(&p->lru, &pages);
+ }
+ dma_pmd_meta_pages = 0;
+ free_vm_area(vm);
+ bitmap_free(dma_pmd_chunk_bitmap);
+ dma_pmd_chunk_bitmap = NULL;
+ list_for_each_entry_safe(p, next, &pages, lru) {
+ list_del_init(&p->lru);
+ __free_page(p);
+ }
+ goto out_unlock;
+ }
+
+ /*
+ * Publish @nframes and the populated pages before the array pointer:
+ * readers pair with smp_load_acquire(&dma_pmd_meta_array).
+ */
+ dma_pmd_meta_nframes = nframes;
+ /* Pairs with smp_load_acquire() in dma_is_pmd_page(). */
+ smp_store_release(&dma_pmd_meta_array, array);
+
+ pr_info("dma_pmd: %lu frames, %lu KB of KVA, %lu pages backed (%lu KB)\n",
+ nframes, size / 1024, dma_pmd_meta_pages,
+ dma_pmd_meta_pages * (PAGE_SIZE / 1024));
+
+out_unlock:
+ mutex_unlock(&dma_pmd_meta_mutex);
+ return ret;
+}
+EXPORT_SYMBOL(dma_pmd_meta_init);
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
new file mode 100644
index 0000000000000..a5ebede11dac2
--- /dev/null
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+#ifndef _DRIVERS_IOMMU_DMA_PMD_PRIV_H
+#define _DRIVERS_IOMMU_DMA_PMD_PRIV_H
+
+#include <linux/dma-pmd.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_DMA_PMD
+
+/*
+ * Normally 128 B per entry (64 KB per GB of RAM), or 256 B when spinlock
+ * debugging enlarges struct dma_pmd_meta. Verified by static_assert().
+ */
+#if defined(CONFIG_DEBUG_SPINLOCK) || defined(CONFIG_DEBUG_LOCK_ALLOC)
+#define DMA_PMD_META_SHIFT 8
+#else
+#define DMA_PMD_META_SHIFT 7
+#endif
+#define DMA_PMD_META_SIZE BIT(DMA_PMD_META_SHIFT)
+
+/**
+ * struct dma_pmd_meta - Metadata for a single DMA_PMD page
+ * @pooled: True while this page is owned by an dma_pmd_pool (at offset 0)
+ *
+ * Lives in the sparse per-PMD-frame array @dma_pmd_meta_array indexed by
+ * (pfn >> PMD_ORDER). Only chunks covering valid RAM are backed by physical
+ * pages (64 KB per GB of RAM).
+ */
+struct dma_pmd_meta {
+ bool pooled;
+} __aligned(DMA_PMD_META_SIZE);
+
+static_assert(offsetof(struct dma_pmd_meta, pooled) == 0);
+static_assert(sizeof(struct dma_pmd_meta) == DMA_PMD_META_SIZE);
+
+extern unsigned long dma_pmd_meta_nframes;
+
+static inline struct dma_pmd_meta *dma_pmd_meta_base(void)
+{
+ /*
+ * Acquire pairs with smp_store_release() in dma_pmd_meta_init():
+ * orders both @dma_pmd_meta_nframes and the populated backing pages.
+ */
+ return smp_load_acquire(&dma_pmd_meta_array);
+}
+
+int dma_pmd_meta_init(void);
+bool dma_pmd_meta_ensure_pfn(unsigned long pfn, bool can_block);
+struct dma_pmd_meta *dma_pmd_meta_of_pfn(unsigned long pfn);
+struct dma_pmd_meta *dma_pmd_meta_from_phys(phys_addr_t pa);
+unsigned long dma_pmd_meta_to_pfn(const struct dma_pmd_meta *m);
+phys_addr_t dma_pmd_meta_to_phys(const struct dma_pmd_meta *m);
+
+#endif /* CONFIG_DMA_PMD */
+#endif /* _DRIVERS_IOMMU_DMA_PMD_PRIV_H */
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
new file mode 100644
index 0000000000000..07df815945257
--- /dev/null
+++ b/include/linux/dma-pmd.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* See Documentation/core-api/dma-pmd.rst for the architecture overview. */
+#ifndef _LINUX_DMA_PMD_H
+#define _LINUX_DMA_PMD_H
+
+#include <linux/compiler.h>
+#include <linux/mm_types.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_DMA_PMD
+
+extern void *dma_pmd_meta_array;
+
+bool __dma_is_pmd_page(unsigned long pfn);
+
+static inline bool dma_is_pmd_page(unsigned long pfn)
+{
+ /*
+ * Acquire pairs with smp_store_release() in dma_pmd_meta_init():
+ * orders both @dma_pmd_meta_nframes and the populated backing pages.
+ */
+ if (likely(!smp_load_acquire(&dma_pmd_meta_array)))
+ return false;
+
+ return __dma_is_pmd_page(pfn);
+}
+
+#else /* !CONFIG_DMA_PMD */
+
+static inline bool dma_is_pmd_page(unsigned long pfn)
+{
+ return false;
+}
+
+#endif /* CONFIG_DMA_PMD */
+#endif /* _LINUX_DMA_PMD_H */
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table
2026-10-03 21:22 ` [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table Luigi Rizzo
@ 2026-10-03 21:44 ` Randy Dunlap
2026-10-04 9:14 ` Luigi Rizzo
0 siblings, 1 reply; 86+ messages in thread
From: Randy Dunlap @ 2026-10-03 21:44 UTC (permalink / raw)
To: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 10/3/26 2:22 PM, Luigi Rizzo wrote:
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index 6e07bd69467a3..1bb347fd9da4a 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -158,6 +158,38 @@ config IOMMU_DMA
> select NEED_SG_DMA_LENGTH
> select NEED_SG_DMA_FLAGS if SWIOTLB
>
> +# PMD_SIZE IOMMU backing for DMA-coherent devices
> +config DMA_PMD
> + bool "PMD_SIZE IOMMU backing for DMA-coherent devices"
> + depends on IOMMU_DMA
> + depends on X86_64 || (ARM64 && ARM64_4K_PAGES)
> + depends on !PREEMPT_RT
> + default y
> + help
> + Hand out DMA buffers carved out of PMD_SIZE physically contiguous
> + blocks that the IOMMU maps with a single leaf PTE. Mapping,
> + unmapping and IOTLB flushing are done lazily to reduce CPU overhead.
> + The large mappings greatly reduce IOTLB usage. As in strict iommu
> + mode, pages are returned to the buddy allocator only when any existing
> + mappings have been removed and IOTLB flushed.
> + Costs 128KB of metadata per GB of DMA_PMD buffers, allocated on demand.
Paragraph 1 of the patch description says 64KB per 1GB, not 128KB.
Or am I mixing things up?
--
~Randy
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table
2026-10-03 21:44 ` Randy Dunlap
@ 2026-10-04 9:14 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-04 9:14 UTC (permalink / raw)
To: Randy Dunlap
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni, Greg Kroah-Hartman,
Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Saeed Mahameed, Tariq Toukan, Tony Nguyen, Przemek Kitszel,
Alexander Lobakin, Michael Chan, Pavan Chebbi, iommu, netdev,
linux-mm, driver-core, linux-doc, linux-kernel
On Sat, Oct 3, 2026 at 11:44 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>
>
>
> On 10/3/26 2:22 PM, Luigi Rizzo wrote:
> > diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> > index 6e07bd69467a3..1bb347fd9da4a 100644
> > --- a/drivers/iommu/Kconfig
> > +++ b/drivers/iommu/Kconfig
> > @@ -158,6 +158,38 @@ config IOMMU_DMA
> > select NEED_SG_DMA_LENGTH
> > select NEED_SG_DMA_FLAGS if SWIOTLB
> >
> > +# PMD_SIZE IOMMU backing for DMA-coherent devices
> > +config DMA_PMD
...
> > + Costs 128KB of metadata per GB of DMA_PMD buffers, allocated on demand.
>
> Paragraph 1 of the patch description says 64KB per 1GB, not 128KB.
> Or am I mixing things up?
256B per 2MB page, 128KB per 1GB is the correct version.
The number in the patch description is a leftover from an earlier
version, I will update.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-10 2:59 ` kernel test robot
2026-10-03 21:22 ` [RFC: DMA_PMD 03/22] mm: Add split_page_compound() Luigi Rizzo
` (48 subsequent siblings)
50 siblings, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
DMA_PMD manages pools of PMD_SIZE physically contiguous memory from which
IO buffers can be allocated. Recycling memory to the pool amortizes the
cost of mapping, unmapping and IOTLB flush, but must be handled so that
memory with active IOMMU mappings cannot be used for other data or code.
Add the pool lifecycle (struct dma_pmd_pool, dma_pmd_pool_create(),
dma_pmd_pool_destroy()) and the page release side. In particular,
__dma_pmd_free_page() is hooked into __free_pages_prepare() so that
DMA_PMD pages are returned to the pool.
Also implement asynchronous DMA_PMD page retirement and release to
the buddy allocator via a WQ_MEM_RECLAIM worker. Retirement removes
DMA_PMD pages from pools and destroys the mappings flushing the IOTLB,
but defers the buddy release by an RCU grace period, so a reader that
already passed dma_is_pmd_page() can safely finish reading p2m.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/dma-pmd-meta.c | 4 +-
drivers/iommu/dma-pmd-pool.c | 379 +++++++++++++++++++++++++++++++++++
drivers/iommu/dma-pmd-priv.h | 129 +++++++++++-
include/linux/dma-pmd.h | 73 ++++++-
mm/page_alloc.c | 4 +
6 files changed, 579 insertions(+), 12 deletions(-)
create mode 100644 drivers/iommu/dma-pmd-pool.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 2ad9b2eefd741..de2c3bad9aa3e 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o
+obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o dma-pmd-pool.o
obj-$(CONFIG_DMA_PMD_META_KUNIT_TEST) += dma-pmd-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/dma-pmd-meta.c b/drivers/iommu/dma-pmd-meta.c
index fcaf5b121e66a..27b8c2568c749 100644
--- a/drivers/iommu/dma-pmd-meta.c
+++ b/drivers/iommu/dma-pmd-meta.c
@@ -43,7 +43,9 @@ EXPORT_SYMBOL(dma_pmd_meta_array);
unsigned long dma_pmd_meta_nframes __read_mostly;
static unsigned long *dma_pmd_chunk_bitmap __read_mostly;
-static struct dma_pmd_meta dma_pmd_meta_nil;
+static struct dma_pmd_meta dma_pmd_meta_nil = {
+ .map_lock = __SPIN_LOCK_UNLOCKED(dma_pmd_meta_nil.map_lock),
+};
static unsigned long dma_pmd_meta_pages;
static DEFINE_MUTEX(dma_pmd_meta_mutex);
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
new file mode 100644
index 0000000000000..5020265d52cd5
--- /dev/null
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -0,0 +1,379 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * DMA_PMD streaming page pool, async page reclaim, shrinker, and debugfs.
+ *
+ * See Documentation/core-api/dma-pmd.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/debugfs.h>
+#include <linux/dma-mapping.h>
+#include <linux/dma-pmd.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/rcupdate.h>
+#include <linux/seq_file.h>
+#include <linux/shrinker.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/srcu.h>
+#include <linux/workqueue.h>
+
+#include "dma-pmd-priv.h"
+
+/*
+ * DMA_PMD pages whose last block has just been freed and which are over the
+ * pool's idle watermark are pushed here locklessly for later release.
+ */
+static LLIST_HEAD(dma_pmd_free_list);
+
+/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
+static struct workqueue_struct *dma_pmd_wq __ro_after_init;
+
+/* All live pools. */
+static LIST_HEAD(dma_pmd_pools);
+static DEFINE_MUTEX(dma_pmd_pools_lock);
+static LLIST_HEAD(dma_pmd_dead_pools);
+
+static void dma_pmd_schedule_reclaim(void);
+
+static void dma_pmd_pool_free_kref(struct kref *kref)
+{
+ struct dma_pmd_pool *pool = container_of(kref, struct dma_pmd_pool, refcount);
+
+ llist_add(&pool->dead_node, &dma_pmd_dead_pools);
+ dma_pmd_schedule_reclaim();
+}
+
+/*
+ * Releasing an idle PMD page proceeds in three stages:
+ *
+ * 1. Detach from pool->idle (or when the last block of a destroyed/over-watermark
+ * PMD page frees in __dma_pmd_free_page()). Because __free_pages_prepare()
+ * may run in hardirq/softirq or with allocator locks held, the PMD page is
+ * pushed locklessly onto dma_pmd_free_list and dma_pmd_reclaim_work is
+ * scheduled on dma_pmd_wq.
+ * 2. In process context, dma_pmd_release_page() unmaps all IOMMU domains
+ * (dma_pmd_unmap_all(), added in a later commit), clears meta->pooled so
+ * new lockless readers stop entering @meta, and queues
+ * dma_pmd_release_page_rcu() via call_rcu().
+ * 3. After an RCU grace period (once no concurrent dma_pmd_free_page() reader
+ * can still be dereferencing @meta), dma_pmd_release_page_rcu() unfreezes
+ * all order-pool->order blocks (already split by split_page_compound()),
+ * returns them to the buddy allocator, and drops pool->refcount.
+ */
+
+/**
+ * dma_pmd_release_page_rcu - Return a retired PMD page to the buddy allocator
+ * @head: @rcu member of the PMD page being released
+ *
+ * Runs once no reader can still be resolving a PFN in this PMD page. All blocks
+ * are unfrozen and handed back to the buddy allocator, and the pool reference
+ * is dropped last.
+ */
+static void dma_pmd_release_page_rcu(struct rcu_head *head)
+{
+ /*
+ * Read @meta->pool before returning the blocks to buddy: once the last
+ * block is freed, the PMD frame can be reallocated and @meta overwritten.
+ */
+ struct dma_pmd_meta *meta = container_of(head, struct dma_pmd_meta, rcu);
+ struct page *page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
+ struct dma_pmd_pool *pool = READ_ONCE(meta->pool);
+ unsigned int nr = DMA_PMD_BLOCKS(pool->order);
+ unsigned int order = pool->order;
+ unsigned long flags;
+ unsigned int i;
+
+ for (i = 0; i < nr; i++) {
+ struct page *block = page + (i << order);
+
+ page_ref_unfreeze(block, 1);
+ __free_pages(block, order);
+ }
+
+ spin_lock_irqsave(&pool->lock, flags);
+ pool->pmd_free_cnt++;
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ kref_put(&pool->refcount, dma_pmd_pool_free_kref);
+}
+
+/**
+ * dma_pmd_release_page - Retire a PMD page and schedule its buddy release
+ * @meta: PMD page metadata structure (must have all usable blocks idle)
+ *
+ * Cost: Slow path; clears the membership bit. The blocks themselves are
+ * returned after an RCU grace period.
+ * Locking: Must not hold @pool->lock.
+ * Frequency: Rare (background reclaim workqueue, pool destruction).
+ */
+static void dma_pmd_release_page(struct dma_pmd_meta *meta)
+{
+ /* dma_pmd_unmap_all(meta) will be called here once IOMMU mappings are added. */
+
+ /*
+ * Clear membership before the grace period. A reader that passed
+ * dma_is_pmd_page() just before this may still be reading @meta, so
+ * the PMD page and its metadata entry must outlive the grace period.
+ */
+ WRITE_ONCE(meta->pooled, false);
+ call_rcu(&meta->rcu, dma_pmd_release_page_rcu);
+}
+
+static void dma_pmd_reclaim_work_fn(struct work_struct *work)
+{
+ struct llist_node *node = llist_del_all(&dma_pmd_free_list);
+ struct dma_pmd_pool *pool, *ptmp;
+ struct dma_pmd_meta *meta, *tmp;
+
+ llist_for_each_entry_safe(meta, tmp, node, llnode)
+ dma_pmd_release_page(meta);
+
+ node = llist_del_all(&dma_pmd_dead_pools);
+ if (node) {
+ mutex_lock(&dma_pmd_pools_lock);
+ llist_for_each_entry_safe(pool, ptmp, node, dead_node)
+ list_del(&pool->node);
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ llist_for_each_entry_safe(pool, ptmp, node, dead_node)
+ kfree(pool);
+ }
+}
+
+static DECLARE_WORK(dma_pmd_reclaim_work, dma_pmd_reclaim_work_fn);
+
+static void dma_pmd_schedule_reclaim(void)
+{
+ struct workqueue_struct *wq = READ_ONCE(dma_pmd_wq);
+
+ if (likely(wq))
+ queue_work(wq, &dma_pmd_reclaim_work);
+ else
+ schedule_work(&dma_pmd_reclaim_work);
+}
+
+/**
+ * dma_pmd_pool_create - Create a DMA_PMD page pool
+ * @order: Block order to dispense, must be <= PMD_ORDER.
+ * @max_idle_pages: Maximum number of completely idle PMD pages to keep cached in
+ * the pool before asynchronously releasing excess PMD pages to buddy
+ * (0 uses default of 16 = 32 MB).
+ *
+ * New PMD physical pages are allocated on demand on the NUMA node of the
+ * calling CPU (the NAPI CPU for RX queue refill, or the application CPU for TX).
+ *
+ * Cost: Process-context control plane; allocates pool metadata structure, and
+ * on first call the global membership bitmap.
+ * Locking: May sleep (GFP_KERNEL).
+ * Frequency: Rare (driver queue initialization or subsystem init).
+ *
+ * Return: Pointer to the created pool, or NULL on failure.
+ */
+struct dma_pmd_pool *dma_pmd_pool_create(unsigned int order, unsigned int max_idle_pages)
+{
+ struct dma_pmd_pool *pool;
+
+ if (order > PMD_ORDER)
+ return NULL;
+
+ pool = kzalloc(sizeof(*pool), GFP_KERNEL);
+ if (!pool)
+ return NULL;
+
+ kref_init(&pool->refcount);
+ spin_lock_init(&pool->lock);
+ pool->order = order;
+ pool->max_idle_pages = max_idle_pages ? : 16;
+ pool->next_alloc_attempt = jiffies;
+ INIT_LIST_HEAD(&pool->partial);
+ INIT_LIST_HEAD(&pool->idle);
+ INIT_LIST_HEAD(&pool->full);
+
+ mutex_lock(&dma_pmd_pools_lock);
+ if (dma_pmd_meta_init()) {
+ mutex_unlock(&dma_pmd_pools_lock);
+ kfree(pool);
+ return NULL;
+ }
+ list_add(&pool->node, &dma_pmd_pools);
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ return pool;
+}
+EXPORT_SYMBOL(dma_pmd_pool_create);
+
+/**
+ * dma_pmd_pool_destroy - Destroy a DMA_PMD page pool and unmap cached PMD pages
+ * @pool: Pool to destroy
+ *
+ * Frees completely idle 2M pages immediately. Device DMA must be quiesced by
+ * the caller first, but blocks already handed to the networking stack (e.g.,
+ * RX skbs waiting in socket queues or TX skbs in TCP retransmit queues) may
+ * still be in flight; those 2M pages stay on @pool->partial / @pool->full and
+ * keep @pool alive on @dma_pmd_pools via @pool->refcount until their last
+ * blocks free.
+ *
+ * Cost: Control plane teardown; flushes async reclaim workqueue.
+ * Locking: Process context (may sleep in flush_work). Acquires @pool->lock.
+ * Frequency: Rare (driver queue teardown or module unload).
+ *
+ * Return: Always NULL, so callers can write `pool = dma_pmd_pool_destroy(pool)`.
+ */
+struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
+{
+ struct dma_pmd_meta *meta, *tmp;
+ LIST_HEAD(release_list);
+ unsigned long flags;
+
+ if (!pool)
+ return NULL;
+
+ spin_lock_irqsave(&pool->lock, flags);
+ pool->destroyed = true;
+
+ /* Take out the idle PMD pages; the unmap happens outside the lock. */
+ list_splice_init(&pool->idle, &release_list);
+ pool->num_idle_pages = 0;
+
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ list_for_each_entry_safe(meta, tmp, &release_list, list) {
+ list_del(&meta->list);
+ dma_pmd_release_page(meta);
+ }
+
+ kref_put(&pool->refcount, dma_pmd_pool_free_kref);
+ flush_work(&dma_pmd_reclaim_work);
+ return NULL;
+}
+EXPORT_SYMBOL(dma_pmd_pool_destroy);
+
+/**
+ * __dma_pmd_free_page - Recycle a freed block back into its DMA_PMD pool
+ * @page: Block being freed
+ *
+ * Called from __free_pages_prepare() when a block's refcount reaches 0.
+ * Marks the block index available in @meta->dirty_bitmap[] (or @meta->free_bitmap[]
+ * when init_on_free zeroes it). If the PMD page becomes completely idle and the
+ * pool exceeds @max_idle_pages, queues the PMD page to dma_pmd_free_list for
+ * asynchronous unmapping and buddy release.
+ *
+ * Cost: O(1) bit set under spinlock (~15-20 cycles).
+ * Locking: Runs inside the rcu_read_lock() section opened by
+ * dma_pmd_free_page(), and acquires @pool->lock (irqsave).
+ * Safe from any context (IRQ, softirq, process). Never sleeps.
+ * Frequency: Very high (called on every kfree_skb / napi_consume_skb / put_page
+ * for buffers allocated from an dma_pmd_pool).
+ *
+ * Return: true if the page belonged to a DMA_PMD pool and was recycled, false
+ * if it is a normal system page that should be freed by the buddy
+ * allocator.
+ */
+bool __dma_pmd_free_page(struct page *page)
+{
+ struct dma_pmd_meta *meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
+ unsigned int nr = DMA_PMD_BLOCKS(meta->pool->order), idx;
+ struct dma_pmd_pool *pool = meta->pool;
+ unsigned long pfn = page_to_pfn(page);
+ bool should_release = false;
+ bool zeroed_on_free = false;
+ unsigned int avail;
+ unsigned long flags;
+
+ idx = (pfn - dma_pmd_meta_to_pfn(meta)) >> pool->order;
+
+ /*
+ * Never reuse or return a hwpoisoned block to buddy: leave its bit
+ * clear in free_bitmap/dirty_bitmap so the PMD page stays pinned and isolated.
+ */
+ if (unlikely(folio_contain_hwpoisoned_page(page_folio(page)))) {
+ pr_err_once("dma_pmd: poisoned block at pfn %lu, leaking it to keep pool %p intact\n",
+ pfn, pool);
+ return true;
+ }
+
+ /*
+ * Reject double frees (bit already set in free_bitmap or dirty_bitmap),
+ * out-of-range indices, or mismatched orders: leak the block rather than
+ * corrupt the pool or hand a live PMD page's block to the buddy allocator.
+ */
+ if (WARN_ON_ONCE(idx >= nr || compound_order(page) != pool->order ||
+ test_bit(idx, meta->free_bitmap) ||
+ test_bit(idx, meta->dirty_bitmap)))
+ return true;
+
+ /*
+ * Pooled blocks stay allocated from buddy until dma_pmd_release_page_rcu()
+ * (which runs KASAN/KMSAN/pgalloc_tag_sub via __free_pages()), and never
+ * come from HIGHMEM (CONFIG_DMA_PMD is 64-bit only).
+ */
+ if (want_init_on_free()) {
+ memset(page_address(page), 0, PAGE_SIZE << pool->order);
+ zeroed_on_free = true;
+ }
+
+ spin_lock_irqsave(&pool->lock, flags);
+
+ if (WARN_ON_ONCE(test_bit(idx, meta->free_bitmap) ||
+ test_bit(idx, meta->dirty_bitmap))) {
+ spin_unlock_irqrestore(&pool->lock, flags);
+ return true;
+ }
+
+ if (zeroed_on_free) {
+ __set_bit(idx, meta->free_bitmap);
+ meta->nr_free++;
+ } else {
+ __set_bit(idx, meta->dirty_bitmap);
+ meta->nr_dirty++;
+ }
+ avail = dma_pmd_meta_avail(meta);
+ pool->block_free_cnt++;
+
+ if (avail == 1 && !pool->destroyed)
+ list_move(&meta->list, &pool->partial);
+
+ if (avail == nr) {
+ if (pool->destroyed || pool->num_idle_pages >= pool->max_idle_pages) {
+ list_del_init(&meta->list);
+ llist_add(&meta->llnode, &dma_pmd_free_list);
+ should_release = true;
+ } else {
+ list_move(&meta->list, &pool->idle);
+ pool->num_idle_pages++;
+ }
+ }
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ if (should_release)
+ dma_pmd_schedule_reclaim();
+
+ return true;
+}
+EXPORT_SYMBOL(__dma_pmd_free_page);
+
+static int __init dma_pmd_init(void)
+{
+ /*
+ * Reclaim frees memory, so it must not queue behind arbitrary work on
+ * system_wq when the machine is already short of it. Failure is not
+ * fatal: dma_pmd_schedule_reclaim() falls back to system_wq.
+ */
+ dma_pmd_wq = alloc_workqueue("dma_pmd", WQ_MEM_RECLAIM, 0);
+ if (!dma_pmd_wq)
+ pr_warn("dma_pmd: no reclaim workqueue, falling back to system_wq\n");
+
+ return 0;
+}
+subsys_initcall(dma_pmd_init);
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index a5ebede11dac2..99bd7d77fd4ef 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -3,37 +3,150 @@
#define _DRIVERS_IOMMU_DMA_PMD_PRIV_H
#include <linux/dma-pmd.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/llist.h>
#include <linux/spinlock.h>
#include <linux/types.h>
#ifdef CONFIG_DMA_PMD
+#define DMA_PMD_BLOCKS(order) (1U << (PMD_ORDER - (order)))
+
+/*
+ * Locking
+ * -------
+ * dma_pmd_meta_mutex Serialises population of dma_pmd_meta_array. Taken
+ * from init and on-demand from dma_pmd_meta_ensure_pfn()
+ * when can_block is true. It is not nested with pool->lock
+ * or meta->map_lock, and sits inside dma_pmd_pools_lock
+ * when dma_pmd_pool_create() calls dma_pmd_meta_init().
+ *
+ * dma_pmd_pools_lock Mutex over the list of live pools. Outermost of the
+ * three, and it must not be taken from a context that
+ * cannot sleep.
+ *
+ * pool->lock IRQ-safe spinlock over one pool's partial/idle/full
+ * lists and its counters. The alloc and free fast paths
+ * take it on its own.
+ *
+ * meta->map_lock IRQ-safe spinlock over one PMD frame's domains_mapped
+ * bitmap. Innermost, and never held across anything that
+ * sleeps.
+ */
+
/*
- * Normally 128 B per entry (64 KB per GB of RAM), or 256 B when spinlock
- * debugging enlarges struct dma_pmd_meta. Verified by static_assert().
+ * 256 B per entry (128 KB per GB of RAM). Verified by static_assert().
*/
-#if defined(CONFIG_DEBUG_SPINLOCK) || defined(CONFIG_DEBUG_LOCK_ALLOC)
#define DMA_PMD_META_SHIFT 8
-#else
-#define DMA_PMD_META_SHIFT 7
-#endif
#define DMA_PMD_META_SIZE BIT(DMA_PMD_META_SHIFT)
/**
* struct dma_pmd_meta - Metadata for a single DMA_PMD page
- * @pooled: True while this page is owned by an dma_pmd_pool (at offset 0)
+ * @pooled: True while this page is owned by an dma_pmd_pool (offset 0)
+ * read locklessly by dma_is_pmd_page().
+ * @nr_free: Number of zeroed available blocks in @free_bitmap
+ * @nr_dirty: Number of dirty available blocks in @dirty_bitmap
+ * @map_lock: Spinlock protecting slow-path updates to @domains_mapped
+ *
+ * Map fast path:
+ * @domains_mapped: One bit per domain index: set once this PMD page's leaf
+ * PTE is installed in that domain.
+ *
+ * Alloc/free fast path, all under pool->lock:
+ * @pool: Owning dma_pmd_pool (holds a kref on the pool while pooled)
+ * @list: Node in pool->partial, pool->idle or pool->full
+ *
+ * Slow path only (disjoint lifetimes):
+ * @llnode: Node in lockless dma_pmd_free_list for async buddy release
+ * @rcu: RCU head used to defer buddy release past lockless readers
+ *
+ * @free_bitmap: Bitmap of zeroed available block indices (up to 1 << PMD_ORDER)
+ * @dirty_bitmap: Bitmap of dirty available block indices
*
* Lives in the sparse per-PMD-frame array @dma_pmd_meta_array indexed by
* (pfn >> PMD_ORDER). Only chunks covering valid RAM are backed by physical
- * pages (64 KB per GB of RAM).
+ * pages (128 KB per GB of RAM).
*/
struct dma_pmd_meta {
bool pooled;
+ u16 nr_free;
+ u16 nr_dirty;
+ spinlock_t map_lock;
+
+ unsigned long domains_mapped;
+
+ struct dma_pmd_pool *pool;
+ struct list_head list;
+
+ union {
+ struct llist_node llnode;
+ struct rcu_head rcu;
+ };
+
+ DECLARE_BITMAP(free_bitmap, 1U << PMD_ORDER) ____cacheline_aligned;
+ DECLARE_BITMAP(dirty_bitmap, 1U << PMD_ORDER) ____cacheline_aligned;
} __aligned(DMA_PMD_META_SIZE);
static_assert(offsetof(struct dma_pmd_meta, pooled) == 0);
static_assert(sizeof(struct dma_pmd_meta) == DMA_PMD_META_SIZE);
+static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
+{
+ return m->nr_free + m->nr_dirty;
+}
+
+/**
+ * struct dma_pmd_pool - Pool managing PMD-backed blocks of a fixed order
+ *
+ * @lock: Spinlock protecting @partial, @idle, @full, @num_idle_pages,
+ * block bitmaps and the statistics counters
+ * @order: Block order managed by this pool (<= PMD_ORDER)
+ * @destroyed: Set when dma_pmd_pool_destroy() has been called
+ * @partial: List of PMD pages with at least 1 free block, but not all
+ * of them (MRU ordered). Candidates for allocation.
+ * @idle: List of PMD pages whose every usable block is free. Held
+ * separately because they can be released under pressure.
+ * @full: List of PMD pages with 0 free blocks
+ * @num_idle_pages: Length of @idle
+ * @max_idle_pages: High watermark of completely idle PMD pages retained in
+ * @idle before triggering asynchronous release to buddy
+ * @pmd_alloc_cnt: Statistics counter of PMD pages allocated from buddy
+ * @pmd_free_cnt: Statistics counter of PMD pages released back to buddy
+ * @block_alloc_cnt: Statistics counter of block allocations satisfied
+ * @block_free_cnt: Statistics counter of block frees recycled into pool
+ * @next_alloc_attempt: Do not attempt a new order-9 allocation before this time.
+ * Damps repeated high-order GFP_ATOMIC failures under
+ * fragmentation, which would otherwise be retried on every
+ * pool miss from softirq context.
+ * @refcount: Reference count; held by pool creator and each active PMD page
+ * @node: Node in the global dma_pmd_pools list
+ * @dead_node: Node in dma_pmd_dead_pools once @refcount reaches 0
+ */
+struct dma_pmd_pool {
+ /* First cacheline: hot fields touched on block alloc and free. */
+ spinlock_t lock;
+ u8 order;
+ bool destroyed;
+ struct list_head partial;
+ struct list_head idle;
+ struct list_head full;
+ unsigned int num_idle_pages;
+ unsigned int max_idle_pages;
+
+ /* Statistics, all updated under @lock. */
+ u64 pmd_alloc_cnt;
+ u64 pmd_free_cnt;
+ u64 block_alloc_cnt;
+ u64 block_free_cnt;
+
+ /* Cold. */
+ unsigned long next_alloc_attempt;
+ struct kref refcount;
+ struct list_head node;
+ struct llist_node dead_node;
+} ____cacheline_aligned;
+
extern unsigned long dma_pmd_meta_nframes;
static inline struct dma_pmd_meta *dma_pmd_meta_base(void)
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
index 07df815945257..236be50b7349f 100644
--- a/include/linux/dma-pmd.h
+++ b/include/linux/dma-pmd.h
@@ -3,9 +3,12 @@
#ifndef _LINUX_DMA_PMD_H
#define _LINUX_DMA_PMD_H
-#include <linux/compiler.h>
-#include <linux/mm_types.h>
#include <linux/types.h>
+#include <linux/mm.h>
+#include <linux/rcupdate.h>
+
+struct device;
+struct dma_pmd_pool;
#ifdef CONFIG_DMA_PMD
@@ -25,6 +28,56 @@ static inline bool dma_is_pmd_page(unsigned long pfn)
return __dma_is_pmd_page(pfn);
}
+bool __dma_pmd_free_page(struct page *page);
+
+static inline bool dma_pmd_free_page(struct page *page)
+{
+ bool ret;
+
+ rcu_read_lock();
+ ret = dma_is_pmd_page(page_to_pfn(page)) && __dma_pmd_free_page(page);
+ rcu_read_unlock();
+
+ return ret;
+}
+
+/*
+ * External API: dma_pmd_pool (streaming DMA page pool)
+ * -----------------------------------------------------
+ * Opportunistic drop-in for alloc_pages_node() that carves fixed-order
+ * (order <= PMD_ORDER) pages out of 2MB physically contiguous buddy pages.
+ *
+ * - Creation (dma_pmd_pool_create):
+ * Records @order and @max_idle_pages; no 2MB pages or IOMMU mappings
+ * are allocated yet.
+ *
+ * - Allocation (dma_pmd_pool_alloc_node / dma_pmd_pool_alloc):
+ * Dispenses an order-@order compound page with refcount 1 (or NULL on
+ * unsupported GFP flags / OOM, so callers fall back to alloc_pages_node()).
+ * On a pool miss, allocates a 2MB page on @nid and splits it into subpages.
+ *
+ * - Mapping (dma_map_page / dma_map_single / dma_map_phys / dma_map_sg):
+ * Intercepts single-buffer and scatterlist maps. On first use of a 2MB page
+ * in an IOMMU domain, lazily installs a 2MB bidirectional leaf PTE in the
+ * domain's DMA_PMD IOVA window; subsequent maps are O(1) arithmetic.
+ *
+ * - Unmapping & Recycling (dma_unmap_* / put_page):
+ * dma_unmap_page/single/phys/sg() is a no-op for IOVAs in the DMA_PMD window,
+ * keeping the 2MB PTE resident across buffer reuse. When the page's last
+ * reference drops (put_page() / __free_pages()), __free_pages_prepare()
+ * intercepts it via dma_pmd_free_page() and recycles the subpage back into
+ * @pool.
+ *
+ * - Unmap & Buddy Release (reclaim / shrinker / dma_pmd_pool_destroy):
+ * A 2MB page is released back to the buddy allocator only when all of its
+ * subpages are free in @pool and either the pool exceeds @max_idle_pages, the
+ * system shrinker reclaims idle pages, or dma_pmd_pool_destroy() is called.
+ * Before __free_pages(PMD_ORDER) is called, the 2MB leaf PTE is unmapped from
+ * every domain that mapped it and the IOTLB is synchronously flushed.
+ */
+struct dma_pmd_pool *dma_pmd_pool_create(unsigned int order, unsigned int max_idle_pages);
+struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool);
+
#else /* !CONFIG_DMA_PMD */
static inline bool dma_is_pmd_page(unsigned long pfn)
@@ -32,5 +85,21 @@ static inline bool dma_is_pmd_page(unsigned long pfn)
return false;
}
+static inline bool dma_pmd_free_page(struct page *page)
+{
+ return false;
+}
+
+static inline struct dma_pmd_pool *
+dma_pmd_pool_create(unsigned int order, unsigned int max_idle_pages)
+{
+ return NULL;
+}
+
+static inline struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
+{
+ return NULL;
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _LINUX_DMA_PMD_H */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c483..e8e9905cdea5b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -17,6 +17,7 @@
#include <linux/stddef.h>
#include <linux/mm.h>
#include <linux/highmem.h>
+#include <linux/dma-pmd.h>
#include <linux/interrupt.h>
#include <linux/jiffies.h>
#include <linux/compiler.h>
@@ -1327,6 +1328,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
VM_BUG_ON_PAGE(PageTail(page), page);
+ if (unlikely(dma_pmd_free_page(page)))
+ return false;
+
trace_mm_page_free(page, order);
kmsan_free_page(page, order);
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook
2026-10-03 21:22 ` [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook Luigi Rizzo
@ 2026-10-10 2:59 ` kernel test robot
0 siblings, 0 replies; 86+ messages in thread
From: kernel test robot @ 2026-10-10 2:59 UTC (permalink / raw)
To: Luigi Rizzo
Cc: oe-lkp, lkp, iommu, linux-kernel, linux-mm, Luigi Rizzo,
Joerg Roedel, Will Deacon, Robin Murphy, Christoph Hellwig,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David Hildenbrand, David S . Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Greg Kroah-Hartman,
Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Saeed Mahameed, Tariq Toukan, Tony Nguyen, Przemek Kitszel,
Alexander Lobakin, Michael Chan, Pavan Chebbi, netdev,
driver-core, linux-doc, Luigi Rizzo, yi1.lai
Hello,
kernel test robot noticed "WARNING:at_kernel/workqueue.c:#alloc_workqueue_va" on:
commit: 3311715530328ddf9249e312fa817c89b767fde9 ("[RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook")
url: https://github.com/intel-lab-lkp/linux/commits/Luigi-Rizzo/iommu-dma-introduce-CONFIG_DMA_PMD-and-metadata-table/20261003-212220
base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/all/20261003212241.3432303-3-lrizzo@google.com/
patch subject: [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook
in testcase: boot
config: x86_64-rhel-9.4-rust
compiler: clang-22
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 4 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <yi1.lai@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202610101013.706708d0-lkp@intel.com
[ 0.966875][ T1] ------------[ cut here ]------------
[ 0.972831][ T1] workqueue: dma_pmd is using neither WQ_PERCPU or WQ_UNBOUND. Setting WQ_PERCPU.
[ 0.978823][ T1] WARNING: kernel/workqueue.c:5939 at alloc_workqueue_va+0x75a/0x7c0, CPU#3: swapper/0/1
[ 0.983827][ T1] Modules linked in:
[ 0.985221][ T1] CPU: 3 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.3.0-rc5+ #1 PREEMPT(lazy)
[ 0.988734][ T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 0.991819][ T1] RIP: 0010:alloc_workqueue_va (workqueue.c:5938)
[ 0.993522][ T1] Code: 00 08 00 00 e8 40 08 f0 ff e9 cb f9 ff ff 0f 0b e9 cd fe ff ff 0f 0b e9 c6 fe ff ff 48 8d 3d 66 07 18 02 49 8d b6 b8 00 00 00 <67> 48 0f b9 3a 41 81 cc 00 01 00 00 e9 75 f9 ff ff 48 8d 3d 57 07
All code
========
0: 00 08 add %cl,(%rax)
2: 00 00 add %al,(%rax)
4: e8 40 08 f0 ff call 0xfffffffffff00849
9: e9 cb f9 ff ff jmp 0xfffffffffffff9d9
e: 0f 0b ud2
10: e9 cd fe ff ff jmp 0xfffffffffffffee2
15: 0f 0b ud2
17: e9 c6 fe ff ff jmp 0xfffffffffffffee2
1c: 48 8d 3d 66 07 18 02 lea 0x2180766(%rip),%rdi # 0x2180789
23: 49 8d b6 b8 00 00 00 lea 0xb8(%r14),%rsi
2a:* 67 48 0f b9 3a ud1 (%edx),%rdi <-- trapping instruction
2f: 41 81 cc 00 01 00 00 or $0x100,%r12d
36: e9 75 f9 ff ff jmp 0xfffffffffffff9b0
3b: 48 rex.W
3c: 8d .byte 0x8d
3d: 3d .byte 0x3d
3e: 57 push %rdi
3f: 07 (bad)
Code starting with the faulting instruction ===========================================
0: 67 48 0f b9 3a ud1 (%edx),%rdi
5: 41 81 cc 00 01 00 00 or $0x100,%r12d
c: e9 75 f9 ff ff jmp 0xfffffffffffff986
11: 48 rex.W
12: 8d .byte 0x8d
13: 3d .byte 0x3d
14: 57 push %rdi
15: 07 (bad)
[ 0.999820][ T1] RSP: 0000:ffffcb9d00013a90 EFLAGS: 00010246
[ 1.001542][ T1] RAX: 0000000000000000 RBX: ffff8b7380b46400 RCX: 33ebeeb162fc7d00
[ 1.004550][ T1] RDX: 0000000000000007 RSI: ffff8b7380b464b8 RDI: ffffffff8e484980
[ 1.006818][ T1] RBP: 0000000000000000 R08: 00000000646d705f R09: 000000000000000f
[ 1.008790][ T1] R10: ffff8b7381cec680 R11: ffffffff8ea6ca80 R12: 0000000000000008
[ 1.011630][ T1] R13: ffffffff8da42c29 R14: ffff8b7380b46400 R15: ffffcb9d00013b10
[ 1.013818][ T1] FS: 0000000000000000(0000) GS:ffff8b7721089000(0000) knlGS:0000000000000000
[ 1.016639][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.018818][ T1] CR2: 0000000000000000 CR3: 00000003e4e20000 CR4: 00000000000406f0
[ 1.021819][ T1] Call Trace:
[ 1.022829][ T1] <TASK>
[ 1.023829][ T1] ? iommu_dma_forcedac_setup (dma-iommu.c:?)
[ 1.025515][ T1] alloc_workqueue_noprof (workqueue.c:6049)
[ 1.026818][ T1] ? blake2s_update (crypto/blake2s.c:131)
[ 1.028234][ T1] ? intel_cleanup_irq_remapping (irq_remapping.c:?)
[ 1.029818][ T1] dma_pmd_init (iommu/dma-pmd-pool.c:373)
[ 1.031195][ T1] do_one_initcall (main.c:1358)
[ 1.032833][ T1] ? get_random_u32 (char/random.c:396 char/random.c:541)
[ 1.033826][ T1] ? __get_random_u32_below (char/random.c:555)
[ 1.036270][ T1] ? alloc_from_new_slab (linux/spinlock.h:425 slub.c:4519)
[ 1.037829][ T1] ? refill_objects (slub.c:7424)
[ 1.038818][ T1] ? irqentry_exit (linux/irq-entry-common.h:539)
[ 1.040264][ T1] ? asm_sysvec_apic_timer_interrupt (x86/include/asm/idtentry.h:674)
[ 1.045448][ T1] ? parameq (params.c:91 params.c:99)
[ 1.046722][ T1] ? parameq (linux/fortify-string.h:218 params.c:99)
[ 1.051786][ T1] ? do_initcall_level (linux/srcutree.h:?)
[ 1.054571][ T1] ? parse_args (params.c:153)
[ 1.056827][ T1] do_initcall_level (main.c:1420)
[ 1.058834][ T1] do_initcalls (main.c:1436)
[ 1.060827][ T1] kernel_init_freeable (main.c:1671)
[ 1.063560][ T1] ? rest_init (main.c:?)
[ 1.065827][ T1] kernel_init (main.c:1561)
[ 1.067786][ T1] ret_from_fork (x86/kernel/process.c:158)
[ 1.069828][ T1] ? rest_init (main.c:?)
[ 1.072821][ T1] ret_from_fork_asm (x86/entry/entry_64.S:245)
[ 1.074826][ T1] </TASK>
[ 1.076251][ T1] ---[ end trace 0000000000000000 ]---
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20261010/202610101013.706708d0-lkp@intel.com
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 86+ messages in thread
* [RFC: DMA_PMD 03/22] mm: Add split_page_compound()
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 04/22] iommu/dma: add DMA_PMD pool block allocation Luigi Rizzo
` (47 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add split_page_compound(), which splits an order-@old_order page into
independently refcounted order-@new_order compound pages (or order-0
pages when @new_order == 0). This is the high-order counterpart of
split_page(), used by DMA_PMD pools to carve PMD pages into smaller
compound blocks.
All resulting pieces are returned frozen (refcount 0) so the caller can
park them in a pool and publish each piece with page_ref_unfreeze(piece, 1)
when handed out.
Also add a KUnit test suite (CONFIG_SPLIT_PAGE_COMPOUND_KUNIT_TEST) in
mm/split_page_compound_kunit.c.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
include/linux/mm.h | 2 +
mm/Kconfig.debug | 11 ++++
mm/Makefile | 1 +
mm/page_alloc.c | 85 +++++++++++++++++++++++++
mm/split_page_compound_kunit.c | 113 +++++++++++++++++++++++++++++++++
5 files changed, 212 insertions(+)
create mode 100644 mm/split_page_compound_kunit.c
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd09c438fa23e..799a42005627f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1985,6 +1985,8 @@ static inline struct folio *virt_to_folio(const void *x)
void __folio_put(struct folio *folio);
void split_page(struct page *page, unsigned int order);
+int split_page_compound(struct page *page, unsigned int old_order,
+ unsigned int new_order);
void folio_copy(struct folio *dst, struct folio *src);
int folio_mc_copy(struct folio *dst, struct folio *src);
diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug
index 15dca19dd07da..e1a04671c3b0d 100644
--- a/mm/Kconfig.debug
+++ b/mm/Kconfig.debug
@@ -347,3 +347,14 @@ config MEM_ALLOC_PROFILING_DEBUG
help
Adds warnings with helpful error messages for memory allocation
profiling.
+
+config SPLIT_PAGE_COMPOUND_KUNIT_TEST
+ bool "KUnit test for split_page_compound() (built-in)" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ Builds KUnit unit tests for split_page_compound() in mm/page_alloc.c,
+ verifying compound splitting, sub-block freezing, and speculative
+ reference handling.
+
+ If unsure, say N.
diff --git a/mm/Makefile b/mm/Makefile
index e7245cb88c665..448c0f5f783f3 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -148,3 +148,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o
obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o
+obj-$(CONFIG_SPLIT_PAGE_COMPOUND_KUNIT_TEST) += split_page_compound_kunit.o
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e8e9905cdea5b..a663ec81a588b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3133,6 +3133,91 @@ void split_page(struct page *page, unsigned int order)
}
EXPORT_SYMBOL_GPL(split_page);
+/*
+ * split_page_compound() - split an order-@old_order page into compound pages
+ * of order @new_order
+ * @page: the page to split
+ * @old_order: the current order of @page
+ * @new_order: the order of the resulting pages
+ *
+ * Unlike split_page(), which produces order-0 pages, this produces
+ * 1 << (@old_order - @new_order) compound pages, each with its own head.
+ * @page must not be compound, and the caller must hold the only reference to
+ * it: the block is frozen while the new heads are built.
+ *
+ * All pieces (including @page at index 0) are returned frozen, with a zero
+ * refcount. Publish each piece with
+ * page_ref_unfreeze(piece, 1) before handing it out: its release store orders
+ * the compound layout built here against anyone who then observes the
+ * refcount. A relaxed set_page_count() would not, and on a weakly ordered
+ * machine a PFN walker could see a live refcount on a head whose layout is
+ * not visible yet.
+ *
+ * Memcg-charged pages are rejected: the accounting helpers assume a split
+ * produces order-0 pages and would mis-attribute the new compound heads.
+ *
+ * Return: 0 on success, -EINVAL if @page cannot be split or if @new_order is
+ * larger than @old_order, -EBUSY if anyone but the caller holds a reference
+ * to it.
+ */
+int split_page_compound(struct page *page, unsigned int old_order,
+ unsigned int new_order)
+{
+ unsigned int i, step = 1U << new_order, nr = 1U << old_order;
+
+ if (WARN_ON_ONCE(PageCompound(page) || !page_count(page)))
+ return -EINVAL;
+
+ if (WARN_ON_ONCE(new_order > old_order))
+ return -EINVAL;
+
+ if (WARN_ON_ONCE(memcg_kmem_online() && PageMemcgKmem(page)))
+ return -EINVAL;
+
+ /*
+ * Shape the new compound pages while the block is frozen, which is
+ * the order __alloc_pages_noprof() itself uses: prep_new_page()
+ * builds the page with a zero refcount and the set_page_refcounted()
+ * that follows is what makes it visible.
+ *
+ * Freezing stops a speculative PFN walker from resolving a compound
+ * page that is only half built: get_page_unless_zero() fails for as
+ * long as the layout below is being written. It does not order
+ * against memory_failure(), which sets PG_hwpoison before taking any
+ * reference, so the non-atomic __SetPageHead() below can still lose a
+ * concurrent poison bit - exactly as it can for every other
+ * prep_compound_page() caller, the page allocator included.
+ *
+ * A failed freeze is not a bug, it means someone holds a speculative
+ * reference. Every PFN walker in the tree gates
+ * get_page_unless_zero() on PageLRU, which a freshly allocated block
+ * is not, so in practice only the hwpoison machinery gets here. Let
+ * the caller fall back rather than warn: a machine running
+ * panic_on_warn should not die of a condition the caller handles.
+ */
+ if (!page_ref_freeze(page, 1))
+ return -EBUSY;
+
+ if (!new_order) {
+ /*
+ * The subpages of a non-compound block are already
+ * independent frozen pages, so only the bookkeeping applies.
+ */
+ split_page_owner(page, old_order, 0);
+ pgalloc_tag_split(page_folio(page), old_order, 0);
+ split_page_memcg(page, old_order);
+ return 0;
+ }
+
+ split_page_owner(page, old_order, new_order);
+ pgalloc_tag_split(page_folio(page), old_order, new_order);
+
+ for (i = 0; i < nr; i += step)
+ prep_compound_page(page + i, new_order);
+
+ return 0;
+}
+
int __isolate_free_page(struct page *page, unsigned int order)
{
struct zone *zone = page_zone(page);
diff --git a/mm/split_page_compound_kunit.c b/mm/split_page_compound_kunit.c
new file mode 100644
index 0000000000000..bb0ac908bbc1e
--- /dev/null
+++ b/mm/split_page_compound_kunit.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * KUnit tests for split_page_compound().
+ */
+#include <kunit/test.h>
+#include <linux/gfp.h>
+#include <linux/mm.h>
+
+static void test_split_compound_pieces(struct kunit *test)
+{
+ const unsigned int old_order = 5, new_order = 2;
+ const unsigned int step = 1U << new_order;
+ const unsigned int nr = 1U << old_order;
+ struct page *page;
+ unsigned int i, j;
+ int ret;
+
+ page = alloc_pages(GFP_KERNEL, old_order);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+ KUNIT_EXPECT_FALSE(test, PageCompound(page));
+
+ ret = split_page_compound(page, old_order, new_order);
+ if (ret)
+ __free_pages(page, old_order);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+
+ for (i = 0; i < nr; i += step) {
+ struct page *sub = page + i;
+
+ /* All heads 0..N-1 are returned frozen (refcount 0). */
+ KUNIT_EXPECT_EQ(test, page_count(sub), 0);
+ KUNIT_EXPECT_TRUE(test, PageHead(sub));
+ KUNIT_EXPECT_EQ(test, compound_order(sub), new_order);
+
+ for (j = 1; j < step; j++) {
+ KUNIT_EXPECT_TRUE(test, PageTail(sub + j));
+ KUNIT_EXPECT_PTR_EQ(test, compound_head(sub + j), sub);
+ }
+
+ page_ref_unfreeze(sub, 1);
+
+ /* Tail get_page()/put_page() must operate on this piece's head. */
+ get_page(sub + 1);
+ KUNIT_EXPECT_EQ(test, page_count(sub), 2);
+ put_page(sub + 1);
+ KUNIT_EXPECT_EQ(test, page_count(sub), 1);
+ }
+
+ /* Each compound piece frees independently at new_order. */
+ for (i = 0; i < nr; i += step)
+ __free_pages(page + i, new_order);
+}
+
+static void test_split_order0_pieces(struct kunit *test)
+{
+ const unsigned int old_order = 3, nr = 1U << old_order;
+ struct page *page;
+ unsigned int i;
+ int ret;
+
+ page = alloc_pages(GFP_KERNEL, old_order);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ ret = split_page_compound(page, old_order, 0);
+ if (ret)
+ __free_pages(page, old_order);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+
+ for (i = 0; i < nr; i++) {
+ struct page *sub = page + i;
+
+ KUNIT_EXPECT_FALSE(test, PageCompound(sub));
+ KUNIT_EXPECT_EQ(test, page_count(sub), 0);
+ page_ref_unfreeze(sub, 1);
+ __free_pages(sub, 0);
+ }
+}
+
+static void test_split_ebusy_extra_ref(struct kunit *test)
+{
+ const unsigned int old_order = 4;
+ const unsigned int new_order = 2;
+ struct page *page;
+ int ret;
+
+ page = alloc_pages(GFP_KERNEL, old_order);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ /* Simulate a concurrent speculative reference. */
+ get_page(page);
+ ret = split_page_compound(page, old_order, new_order);
+ KUNIT_EXPECT_EQ(test, ret, -EBUSY);
+ KUNIT_EXPECT_FALSE(test, PageCompound(page));
+
+ put_page(page);
+ __free_pages(page, old_order);
+}
+
+static struct kunit_case split_page_compound_test_cases[] = {
+ KUNIT_CASE(test_split_compound_pieces),
+ KUNIT_CASE(test_split_order0_pieces),
+ KUNIT_CASE(test_split_ebusy_extra_ref),
+ {}
+};
+
+static struct kunit_suite split_page_compound_test_suite = {
+ .name = "split_page_compound",
+ .test_cases = split_page_compound_test_cases,
+};
+
+kunit_test_suite(split_page_compound_test_suite);
+MODULE_DESCRIPTION("KUnit tests for split_page_compound()");
+MODULE_LICENSE("Dual BSD/GPL");
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 04/22] iommu/dma: add DMA_PMD pool block allocation
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (2 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 03/22] mm: Add split_page_compound() Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 05/22] iommu/dma: Global cap and shrinker for DMA_PMD pool memory Luigi Rizzo
` (46 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add dma_pmd_add_page(), dma_pmd_pool_alloc_bulk_node() (with single-page
inline wrappers dma_pmd_pool_alloc_node() and dma_pmd_pool_alloc()), and
dma_pmd_pool_has_free() to carve PMD_SIZE physical blocks into subpages
of a pool's order with split_page_compound() and hand them out to callers.
A parked subpage is kept frozen (refcount 0) and published with
page_ref_unfreeze() when it is handed out, mirroring the way the page
allocator shapes a page before publishing it, so a stray put_page() on a
parked subpage trips a refcount check instead of freeing it twice.
GFP flags asking for a placement a pool cannot choose - __GFP_THISNODE,
__GFP_DMA, __GFP_DMA32 - are ignored, as documented at
dma_pmd_pool_alloc_bulk_node(); __GFP_ZERO is honoured by zeroing at
dispense.
Also add a KUnit test case (test_pool_alloc_and_recycle) exercising pool
allocation, recycling via __free_page(), and deferred teardown with
in-flight blocks.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-kunit.c | 41 ++++-
drivers/iommu/dma-pmd-pool.c | 279 ++++++++++++++++++++++++++++++++++
include/linux/dma-pmd.h | 40 +++++
3 files changed, 359 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index f80527eaf2e84..24978a5123f54 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -70,6 +70,7 @@ static void test_meta_pooled_toggle(struct kunit *test)
base_pfn = ALIGN_DOWN(pfn, 1UL << PMD_ORDER);
/* The buddy allocator returns naturally aligned blocks. */
KUNIT_EXPECT_EQ(test, pfn, base_pfn);
+ KUNIT_ASSERT_TRUE(test, dma_pmd_meta_ensure_pfn(pfn, true));
m = dma_pmd_meta_of_pfn(pfn);
KUNIT_EXPECT_FALSE(test, dma_is_pmd_page(pfn));
@@ -85,10 +86,48 @@ static void test_meta_pooled_toggle(struct kunit *test)
__free_pages(page, PMD_ORDER);
}
+static void test_pool_alloc_and_recycle(struct kunit *test)
+{
+ struct dma_pmd_pool *pool;
+ struct page *p1, *p2;
+ unsigned int order;
+
+ KUNIT_EXPECT_NULL(test, dma_pmd_pool_destroy(NULL));
+
+ for (order = 0; order <= PMD_ORDER + 1; order++) {
+ pool = dma_pmd_pool_create(order, 2);
+ if (order > PMD_ORDER) {
+ KUNIT_EXPECT_NULL(test, pool);
+ continue;
+ }
+ KUNIT_ASSERT_NOT_NULL(test, pool);
+ KUNIT_EXPECT_FALSE(test, dma_pmd_pool_has_free(pool));
+
+ p1 = dma_pmd_pool_alloc(pool, GFP_KERNEL | __GFP_ZERO);
+ if (!p1)
+ dma_pmd_pool_destroy(pool);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ KUNIT_EXPECT_TRUE(test, dma_is_pmd_page(page_to_pfn(p1)));
+ KUNIT_EXPECT_EQ(test, dma_pmd_pool_has_free(pool), order < PMD_ORDER);
+
+ /* Freeing p1 returns it to the pool; next allocation reuses index 0. */
+ __free_pages(p1, order);
+ KUNIT_EXPECT_TRUE(test, dma_pmd_pool_has_free(pool));
+ p2 = dma_pmd_pool_alloc(pool, GFP_KERNEL);
+ KUNIT_EXPECT_PTR_EQ(test, p1, p2);
+
+ /* Destroy with p2 still in flight; freeing p2 releases the 2M page. */
+ dma_pmd_pool_destroy(pool);
+ if (p2)
+ __free_pages(p2, order);
+ }
+}
+
static struct kunit_case dma_pmd_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_pooled_toggle),
+ KUNIT_CASE(test_pool_alloc_and_recycle),
{}
};
@@ -98,5 +137,5 @@ static struct kunit_suite dma_pmd_meta_test_suite = {
};
kunit_test_suite(dma_pmd_meta_test_suite);
-MODULE_DESCRIPTION("KUnit tests for struct dma_pmd_meta table");
+MODULE_DESCRIPTION("KUnit tests for struct dma_pmd_meta table and DMA_PMD pool");
MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index 5020265d52cd5..e6e5ba7c65f27 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -259,6 +259,285 @@ struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
}
EXPORT_SYMBOL(dma_pmd_pool_destroy);
+/**
+ * dma_pmd_add_page - Allocate, split, and register a new PMD page
+ * @pool: Owning pool
+ * @gfp: GFP allocation flags
+ * @nid: Target NUMA node (or NUMA_NO_NODE for local node)
+ *
+ * Cost: Slow path (pool miss); allocates PMD page from buddy on @nid,
+ * splits into compound blocks, and sets the PMD page's membership bit.
+ * Locking: Lockless with respect to @pool->lock (caller inserts into list).
+ * Frequency: Rare after warmup (only when the pool holds no PMD page with a
+ * free block).
+ *
+ * Return: Pointer to new dma_pmd_meta with all usable blocks free, or NULL
+ * on failure.
+ */
+static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
+ gfp_t gfp, int nid)
+{
+ /*
+ * Require full GFP_KERNEL (not just gfpflags_allow_blocking()):
+ * dma_pmd_meta_ensure_pfn() and dma_pmd_page_prepare() use GFP_KERNEL.
+ */
+ bool can_block = (gfp & GFP_KERNEL) == GFP_KERNEL;
+ unsigned int nr = DMA_PMD_BLOCKS(pool->order);
+ struct dma_pmd_meta *meta;
+ gfp_t alloc_gfp, base_gfp;
+ struct page *page;
+
+ /*
+ * A high-order allocation is expensive when it fails, and under fragmentation
+ * it fails on every pool miss. For GFP_ATOMIC, back off briefly.
+ */
+ if (!can_block && time_before(jiffies, READ_ONCE(pool->next_alloc_attempt)))
+ return NULL;
+
+ /*
+ * The following flags are discarded:
+ *
+ * __GFP_MOVABLE would let the PMD page come from ZONE_MOVABLE or a CMA
+ * pageblock. A pooled PMD page is pinned for the life of the pool and
+ * has no movable_operations, so memory hot-remove and CMA allocation
+ * would fail on it forever.
+ *
+ * __GFP_HIGHMEM goes because pooled blocks are handed out as
+ * directly addressable DMA buffers, and __GFP_COMP because the PMD page
+ * is shaped into compound pieces by hand. What is left is also free
+ * of everything slab treats as a bug in GFP_SLAB_BUG_MASK, so the
+ * metadata allocation below can use it as it stands.
+ * __GFP_ZERO is handled per-block in dma_pmd_pool_alloc_bulk_node().
+ */
+ base_gfp = gfp & ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE | __GFP_ZERO);
+
+ /*
+ * A caller that can block gets __GFP_RETRY_MAYFAIL so the allocation
+ * is allowed to compact for an order-9 page, which is the difference
+ * between a pool that is populated before traffic starts and one that
+ * never populates at all on a fragmented machine. It still fails
+ * rather than triggering the OOM killer: this is an optimisation, and
+ * every caller has a working fallback.
+ */
+ alloc_gfp = base_gfp | __GFP_NOWARN;
+ alloc_gfp |= can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY;
+
+ page = alloc_pages_node(nid, alloc_gfp, PMD_ORDER);
+ if (!page) {
+ /* Retry backoff; could be made shorter or tunable in future. */
+ if (!can_block)
+ WRITE_ONCE(pool->next_alloc_attempt,
+ jiffies + DIV_ROUND_UP(HZ, 100));
+ return NULL;
+ }
+
+ /*
+ * Ensure the metadata chunk for this PFN is populated. Fails if the PFN
+ * is past the reservation, or if it belongs to hotplugged memory and the
+ * caller cannot block to populate its metadata page.
+ */
+ if (unlikely(!dma_pmd_meta_ensure_pfn(page_to_pfn(page), can_block))) {
+ __free_pages(page, PMD_ORDER);
+ return NULL;
+ }
+
+ /* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
+ if (split_page_compound(page, PMD_ORDER, pool->order)) {
+ __free_pages(page, PMD_ORDER);
+ return NULL;
+ }
+
+ meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
+ memset(meta, 0, sizeof(*meta));
+ meta->pool = pool;
+ spin_lock_init(&meta->map_lock);
+
+ bitmap_set(meta->dirty_bitmap, 0, nr);
+ meta->nr_dirty = nr;
+
+ kref_get(&pool->refcount);
+
+ /*
+ * Publish last. Once the bit is set, __free_pages_prepare() on any CPU
+ * will route this PMD page's blocks back into the pool, so every @meta
+ * field a reader consults has to be in place first.
+ *
+ * No explicit barrier: a block cannot reach any reader until this
+ * function returns and the caller publishes the PMD page under
+ * @pool->lock, whose release orders both stores against every consumer.
+ */
+ WRITE_ONCE(meta->pooled, true);
+
+ return meta;
+}
+
+static unsigned int dma_pmd_scan_bitmap(struct dma_pmd_pool *pool,
+ unsigned long *bitmap, u16 *countp,
+ unsigned long base_pfn, unsigned int nr,
+ unsigned long want, struct page **out,
+ bool unfreeze)
+{
+ unsigned int idx, got = 0, used = 0;
+
+ if (!*countp || !want)
+ return 0;
+
+ for_each_set_bit(idx, bitmap, nr) {
+ struct page *block = pfn_to_page(base_pfn + (idx << pool->order));
+
+ __clear_bit(idx, bitmap);
+ used++;
+ if (unlikely(folio_contain_hwpoisoned_page(page_folio(block)))) {
+ pr_err_once("dma_pmd: poisoned block at pfn %lu, leaking it to keep pool %p intact\n",
+ page_to_pfn(block), pool);
+ } else {
+ if (unfreeze)
+ page_ref_unfreeze(block, 1);
+ out[got++] = block;
+ }
+
+ if (used == *countp || got == want)
+ break;
+ }
+ *countp -= used;
+ pool->block_alloc_cnt += used;
+ return got;
+}
+
+/* Extract up to @want available blocks from @meta under @pool->lock. */
+static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
+ struct dma_pmd_meta *meta,
+ unsigned long want,
+ struct page **out, bool zero)
+{
+ unsigned long base_pfn = dma_pmd_meta_to_pfn(meta);
+ unsigned int avail_before = dma_pmd_meta_avail(meta);
+ unsigned int nr = DMA_PMD_BLOCKS(pool->order);
+ unsigned int got = 0, used;
+
+ if (zero) {
+ got += dma_pmd_scan_bitmap(pool, meta->free_bitmap, &meta->nr_free,
+ base_pfn, nr, want, out, true);
+ got += dma_pmd_scan_bitmap(pool, meta->dirty_bitmap, &meta->nr_dirty,
+ base_pfn, nr, want - got, out + got, false);
+ } else {
+ got += dma_pmd_scan_bitmap(pool, meta->dirty_bitmap, &meta->nr_dirty,
+ base_pfn, nr, want, out, false);
+ got += dma_pmd_scan_bitmap(pool, meta->free_bitmap, &meta->nr_free,
+ base_pfn, nr, want - got, out + got, true);
+ }
+ used = avail_before - dma_pmd_meta_avail(meta);
+ if (!dma_pmd_meta_avail(meta) || WARN_ON_ONCE(!used)) {
+ meta->nr_free = 0;
+ meta->nr_dirty = 0;
+ list_move(&meta->list, &pool->full);
+ }
+
+ return got;
+}
+
+/**
+ * dma_pmd_pool_alloc_bulk_node - Allocate up to @nr_pages blocks from @pool
+ * @pool: Pool to allocate from
+ * @gfp: GFP flags used if a new PMD page must be allocated from buddy
+ * @nid: Target NUMA node if a new PMD page is allocated (or NUMA_NO_NODE)
+ * @nr_pages: Number of blocks requested
+ * @page_array: Output array of at least @nr_pages entries
+ *
+ * Cost: Fast path is O(nr_pages) bitmap scan from @pool->partial or @pool->idle
+ * under a single lock acquisition. Slow path (both empty) calls
+ * dma_pmd_add_page().
+ * Locking: Acquires @pool->lock (irqsave) for the bitmap scan. Drops lock
+ * during slow-path PMD buddy allocation and per-block initialization.
+ * Frequency: High (called per RX buffer refill or per SKB TX page frag refill).
+ *
+ * A pool hands out a block of a PMD page it already owns, on the node that
+ * PMD page came from, so it cannot satisfy a hard placement constraint:
+ * requests with __GFP_THISNODE, __GFP_DMA or __GFP_DMA32 are declined, as is
+ * __GFP_ACCOUNT (pooled blocks are shared across callers and cannot be charged
+ * to a single memcg), so the caller falls back to the page allocator. Flags
+ * that only widen what is acceptable, such as __GFP_HIGHMEM, are ignored
+ * harmlessly.
+ *
+ * Return: Number of blocks placed in @page_array (0 on failure).
+ */
+unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
+ int nid, unsigned long nr_pages,
+ struct page **page_array)
+{
+ unsigned long allocated = 0, flags, i;
+ bool zero;
+
+ if (unlikely(!pool || pool->destroyed || !nr_pages ||
+ (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE |
+ __GFP_ACCOUNT))))
+ return 0;
+
+ zero = want_init_on_alloc(gfp);
+
+ spin_lock_irqsave(&pool->lock, flags);
+ while (allocated < nr_pages) {
+ struct dma_pmd_meta *meta;
+
+ /* Partially used pages first, idle ones next. */
+ meta = list_first_entry_or_null(&pool->partial, struct dma_pmd_meta, list);
+ if (!meta && !list_empty(&pool->idle)) {
+ meta = list_first_entry(&pool->idle, struct dma_pmd_meta, list);
+ list_move(&meta->list, &pool->partial);
+ pool->num_idle_pages--;
+ }
+ if (!meta) {
+ /* Fallback to a new allocation */
+ spin_unlock_irqrestore(&pool->lock, flags);
+ meta = dma_pmd_add_page(pool, gfp, nid);
+ if (!meta)
+ goto out_prep;
+ spin_lock_irqsave(&pool->lock, flags);
+ pool->pmd_alloc_cnt++;
+ list_add(&meta->list, &pool->partial);
+ }
+
+ allocated += dma_pmd_take_blocks(pool, meta,
+ nr_pages - allocated,
+ &page_array[allocated], zero);
+ }
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+out_prep:
+ for (i = 0; i < allocated; i++) {
+ if (!page_count(page_array[i])) {
+ page_ref_unfreeze(page_array[i], 1);
+ if (zero)
+ memset(page_address(page_array[i]), 0, PAGE_SIZE << pool->order);
+ }
+ }
+ return allocated;
+}
+EXPORT_SYMBOL(dma_pmd_pool_alloc_bulk_node);
+
+/**
+ * dma_pmd_pool_has_free - Can @pool hand out a block without a new PMD page?
+ * @pool: Pool to query
+ *
+ * Cost: O(1) lockless list check.
+ * Locking: None. The answer is advisory and can be stale the moment it is
+ * returned.
+ * Frequency: High; intended for per-buffer decisions on the RX path.
+ *
+ * For callers choosing between an existing non-pooled buffer and a pooled
+ * replacement. Answering "no" when the pool is empty keeps them from trading a
+ * perfectly reusable page for a failed allocation and another singly-mapped
+ * page, which is strictly worse than leaving it alone.
+ *
+ * Return: true if a PMD page with a free block is present.
+ */
+bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool)
+{
+ return pool && !READ_ONCE(pool->destroyed) &&
+ (!list_empty_careful(&pool->partial) || !list_empty_careful(&pool->idle));
+}
+EXPORT_SYMBOL(dma_pmd_pool_has_free);
+
/**
* __dma_pmd_free_page - Recycle a freed block back into its DMA_PMD pool
* @page: Block being freed
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
index 236be50b7349f..085d9a730c278 100644
--- a/include/linux/dma-pmd.h
+++ b/include/linux/dma-pmd.h
@@ -5,6 +5,7 @@
#include <linux/types.h>
#include <linux/mm.h>
+#include <linux/numa.h>
#include <linux/rcupdate.h>
struct device;
@@ -77,6 +78,22 @@ static inline bool dma_pmd_free_page(struct page *page)
*/
struct dma_pmd_pool *dma_pmd_pool_create(unsigned int order, unsigned int max_idle_pages);
struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool);
+unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
+ int nid, unsigned long nr_pages,
+ struct page **page_array);
+static inline struct page *dma_pmd_pool_alloc_node(struct dma_pmd_pool *pool,
+ gfp_t gfp, int nid)
+{
+ struct page *page = NULL;
+
+ dma_pmd_pool_alloc_bulk_node(pool, gfp, nid, 1, &page);
+ return page;
+}
+static inline struct page *dma_pmd_pool_alloc(struct dma_pmd_pool *pool, gfp_t gfp)
+{
+ return dma_pmd_pool_alloc_node(pool, gfp, NUMA_NO_NODE);
+}
+bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool);
#else /* !CONFIG_DMA_PMD */
@@ -101,5 +118,28 @@ static inline struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *poo
return NULL;
}
+static inline unsigned long
+dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp, int nid,
+ unsigned long nr_pages, struct page **page_array)
+{
+ return 0;
+}
+
+static inline struct page *
+dma_pmd_pool_alloc_node(struct dma_pmd_pool *pool, gfp_t gfp, int nid)
+{
+ return NULL;
+}
+
+static inline struct page *dma_pmd_pool_alloc(struct dma_pmd_pool *pool, gfp_t gfp)
+{
+ return NULL;
+}
+
+static inline bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool)
+{
+ return false;
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _LINUX_DMA_PMD_H */
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 05/22] iommu/dma: Global cap and shrinker for DMA_PMD pool memory
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (3 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 04/22] iommu/dma: add DMA_PMD pool block allocation Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 06/22] iommu/dma: reserve a per-domain IOVA window for DMA_PMD pages Luigi Rizzo
` (45 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Each pool holds some idle pages, for faster future allocations, but on
large systems there is one pool per CPU and the number may become large.
Add two mechanism to bound the number of unused DMA_PMD pages:
- a global ceiling on the number of DMA_PMD pages pinned across all
pools, defaulting to one eighth of RAM with a floor for small machines
if not configured via /sys/module/kernel/parameters/dma_pmd_max_pages
It is checked before each DMA_PMD page allocation and is meant to bound
a pathological configuration, not to be reached in normal operation.
- a shrinker that, under reclaim, offers every completely idle page in
every pool and retires as many as the reclaim path asks for, stopping
there. Idle pages are detached under the pool lock and retired outside
it, since retiring one unmaps it from the IOMMU.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-kunit.c | 21 ++++-
drivers/iommu/dma-pmd-pool.c | 146 +++++++++++++++++++++++++++++++++-
drivers/iommu/dma-pmd-priv.h | 2 +
3 files changed, 165 insertions(+), 4 deletions(-)
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index 24978a5123f54..4e38cc5970433 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -88,8 +88,8 @@ static void test_meta_pooled_toggle(struct kunit *test)
static void test_pool_alloc_and_recycle(struct kunit *test)
{
- struct dma_pmd_pool *pool;
- struct page *p1, *p2;
+ struct dma_pmd_pool *pool, *pool_wm;
+ struct page *p1, *p2, *b1, *b2, *b3;
unsigned int order;
KUNIT_EXPECT_NULL(test, dma_pmd_pool_destroy(NULL));
@@ -121,6 +121,23 @@ static void test_pool_alloc_and_recycle(struct kunit *test)
if (p2)
__free_pages(p2, order);
}
+
+ /*
+ * Exercise max_idle_2m = 1 watermark release: an order-(PMD_ORDER - 1)
+ * pool has 2 blocks per 2M page, so 3 allocations span two 2M pages.
+ */
+ pool_wm = dma_pmd_pool_create(PMD_ORDER - 1, 1);
+ KUNIT_ASSERT_NOT_NULL(test, pool_wm);
+ b1 = dma_pmd_pool_alloc(pool_wm, GFP_KERNEL);
+ b2 = dma_pmd_pool_alloc(pool_wm, GFP_KERNEL);
+ b3 = dma_pmd_pool_alloc(pool_wm, GFP_KERNEL);
+ if (b1)
+ __free_pages(b1, PMD_ORDER - 1);
+ if (b2)
+ __free_pages(b2, PMD_ORDER - 1);
+ if (b3)
+ __free_pages(b3, PMD_ORDER - 1);
+ dma_pmd_pool_destroy(pool_wm);
}
static struct kunit_case dma_pmd_meta_test_cases[] = {
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index e6e5ba7c65f27..78f2537772d1d 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -39,6 +39,17 @@ static LLIST_HEAD(dma_pmd_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *dma_pmd_wq __ro_after_init;
+/*
+ * Running total and global ceiling on DMA_PMD pages pinned across all pools.
+ * If zero, it is set at boot at 1/8 of total RAM.
+ * Without this the bound is per-pool, so the footprint scales with the number
+ * of pools (one per RX queue, two per CPU for sockets) with nothing watching
+ * the aggregate.
+ */
+static atomic_long_t dma_pmd_nr_pages __cacheline_aligned_in_smp;
+static unsigned long dma_pmd_max_pages __read_mostly;
+core_param(dma_pmd_max_pages, dma_pmd_max_pages, ulong, 0644);
+
/* All live pools. */
static LIST_HEAD(dma_pmd_pools);
static DEFINE_MUTEX(dma_pmd_pools_lock);
@@ -101,6 +112,8 @@ static void dma_pmd_release_page_rcu(struct rcu_head *head)
__free_pages(block, order);
}
+ atomic_long_dec(&dma_pmd_nr_pages);
+
spin_lock_irqsave(&pool->lock, flags);
pool->pmd_free_cnt++;
spin_unlock_irqrestore(&pool->lock, flags);
@@ -115,7 +128,7 @@ static void dma_pmd_release_page_rcu(struct rcu_head *head)
* Cost: Slow path; clears the membership bit. The blocks themselves are
* returned after an RCU grace period.
* Locking: Must not hold @pool->lock.
- * Frequency: Rare (background reclaim workqueue, pool destruction).
+ * Frequency: Rare (background reclaim workqueue, shrinker, pool destruction).
*/
static void dma_pmd_release_page(struct dma_pmd_meta *meta)
{
@@ -168,7 +181,9 @@ static void dma_pmd_schedule_reclaim(void)
* @order: Block order to dispense, must be <= PMD_ORDER.
* @max_idle_pages: Maximum number of completely idle PMD pages to keep cached in
* the pool before asynchronously releasing excess PMD pages to buddy
- * (0 uses default of 16 = 32 MB).
+ * (0 uses default of 16 = 32 MB). This is a per-pool bound; the
+ * aggregate across all pools is additionally capped by
+ * dma_pmd_max_pages and trimmed by the shrinker.
*
* New PMD physical pages are allocated on demand on the NUMA node of the
* calling CPU (the NAPI CPU for RX queue refill, or the application CPU for TX).
@@ -294,6 +309,15 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
if (!can_block && time_before(jiffies, READ_ONCE(pool->next_alloc_attempt)))
return NULL;
+ /* Aggregate ceiling across every pool, independent of max_idle_pages. */
+ if (atomic_long_inc_return(&dma_pmd_nr_pages) > READ_ONCE(dma_pmd_max_pages)) {
+ atomic_long_dec(&dma_pmd_nr_pages);
+ if (!can_block)
+ WRITE_ONCE(pool->next_alloc_attempt,
+ jiffies + DIV_ROUND_UP(HZ, 100));
+ return NULL;
+ }
+
/*
* The following flags are discarded:
*
@@ -324,6 +348,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
page = alloc_pages_node(nid, alloc_gfp, PMD_ORDER);
if (!page) {
+ atomic_long_dec(&dma_pmd_nr_pages);
/* Retry backoff; could be made shorter or tunable in future. */
if (!can_block)
WRITE_ONCE(pool->next_alloc_attempt,
@@ -338,12 +363,14 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
*/
if (unlikely(!dma_pmd_meta_ensure_pfn(page_to_pfn(page), can_block))) {
__free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
return NULL;
}
/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
if (split_page_compound(page, PMD_ORDER, pool->order)) {
__free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
return NULL;
}
@@ -642,8 +669,112 @@ bool __dma_pmd_free_page(struct page *page)
}
EXPORT_SYMBOL(__dma_pmd_free_page);
+/*
+ * Idle PMD pages are retained per pool up to max_idle_pages, which is a throughput
+ * knob and deliberately does not react to memory pressure. The shrinker is
+ * what makes that safe: under reclaim every completely idle PMD page in every
+ * pool becomes available, so the pools give the memory back instead of
+ * pinning their high-water mark for the lifetime of the machine.
+ *
+ * Both callbacks use mutex_trylock() rather than mutex_lock(): a thread that
+ * already holds dma_pmd_pools_lock can enter direct reclaim from an
+ * allocation it makes under that lock, and reclaim calls straight back in
+ * here. Giving up is always correct, since there is nothing this shrinker
+ * could free that the holder is not already about to account for.
+ */
+static unsigned long dma_pmd_shrink_count(struct shrinker *shrink,
+ struct shrink_control *sc)
+{
+ struct dma_pmd_pool *pool;
+ unsigned long nr = 0;
+
+ if (!mutex_trylock(&dma_pmd_pools_lock))
+ return 0;
+
+ list_for_each_entry(pool, &dma_pmd_pools, node)
+ nr += READ_ONCE(pool->num_idle_pages);
+
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ return nr << PMD_ORDER;
+}
+
+static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
+ struct shrink_control *sc)
+{
+ struct dma_pmd_meta *meta, *tmp;
+ struct dma_pmd_pool *pool;
+ unsigned long freed = 0;
+ LIST_HEAD(release_list);
+ unsigned long flags;
+
+ if (!mutex_trylock(&dma_pmd_pools_lock))
+ return SHRINK_STOP;
+
+ list_for_each_entry(pool, &dma_pmd_pools, node) {
+ /*
+ * Every PMD page on @idle is a candidate, so this detaches one
+ * per iteration and stops as soon as the reclaim path has
+ * what it asked for: the IRQs-off section is bounded by the
+ * work done, not by how many PMD pages the pool holds.
+ */
+ spin_lock_irqsave(&pool->lock, flags);
+ list_for_each_entry_safe(meta, tmp, &pool->idle, list) {
+ if (freed >= sc->nr_to_scan)
+ break;
+ list_move(&meta->list, &release_list);
+ pool->num_idle_pages--;
+ freed += 1UL << PMD_ORDER;
+ }
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ if (freed >= sc->nr_to_scan)
+ break;
+ }
+
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ /* Retiring must not run under @pool->lock. */
+ list_for_each_entry_safe(meta, tmp, &release_list, list) {
+ list_del(&meta->list);
+ dma_pmd_release_page(meta);
+ }
+
+ /*
+ * Both counts are in pages, matching dma_pmd_shrink_count(). A
+ * 2M page is 512 of them and cannot be freed in parts, so a
+ * SHRINK_BATCH sized request always overshoots. Report what was
+ * really reclaimed so that do_shrink_slab() charges its budget for
+ * the whole PMD page instead of the 128 pages it asked for, and does
+ * not come back four times over for the same pressure.
+ *
+ * Zero scanned pages would leave that budget untouched and spin,
+ * so a scan that reclaimed nothing has to say SHRINK_STOP.
+ */
+ sc->nr_scanned = freed;
+
+ return freed ?: SHRINK_STOP;
+}
+
static int __init dma_pmd_init(void)
{
+ struct shrinker *shrink;
+
+ /*
+ * Hard ceiling on memory diverted from the buddy allocator into 2MB
+ * pools, in PMD pages. One eighth of RAM is far above what the per-pool
+ * watermarks should ever reach; it exists to bound a pathological
+ * configuration (many queues, many CPUs, all pools at their high
+ * water mark) rather than to be hit in normal operation. The floor
+ * keeps small machines usable. Zero here means the dma_pmd_max_pages=
+ * boot parameter did not override it.
+ *
+ * Lowering the cap later only stops pools from growing; the PMD pages
+ * already held are returned by the shrinker as they fall idle.
+ */
+ if (!dma_pmd_max_pages)
+ dma_pmd_max_pages = max(totalram_pages() >> (PMD_ORDER + 3), 16UL);
+
/*
* Reclaim frees memory, so it must not queue behind arbitrary work on
* system_wq when the machine is already short of it. Failure is not
@@ -653,6 +784,17 @@ static int __init dma_pmd_init(void)
if (!dma_pmd_wq)
pr_warn("dma_pmd: no reclaim workqueue, falling back to system_wq\n");
+ shrink = shrinker_alloc(0, "dma_pmd");
+ if (!shrink) {
+ pr_warn("dma_pmd: shrinker registration failed\n");
+ return 0;
+ }
+
+ shrink->count_objects = dma_pmd_shrink_count;
+ shrink->scan_objects = dma_pmd_shrink_scan;
+ shrink->seeks = DEFAULT_SEEKS;
+ shrinker_register(shrink);
+
return 0;
}
subsys_initcall(dma_pmd_init);
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 99bd7d77fd4ef..39295d0add51d 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -21,6 +21,8 @@
* when can_block is true. It is not nested with pool->lock
* or meta->map_lock, and sits inside dma_pmd_pools_lock
* when dma_pmd_pool_create() calls dma_pmd_meta_init().
+ * The only reverse edge on dma_pmd_pools_lock is the
+ * shrinker's mutex_trylock(), which cannot block.
*
* dma_pmd_pools_lock Mutex over the list of live pools. Outermost of the
* three, and it must not be taken from a context that
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 06/22] iommu/dma: reserve a per-domain IOVA window for DMA_PMD pages
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (4 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 05/22] iommu/dma: Global cap and shrinker for DMA_PMD pool memory Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 07/22] iommu/dma: release DMA_PMD domain mappings on domain teardown Luigi Rizzo
` (44 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Embed struct dma_pmd_window in struct iommu_dma_cookie to record the
per-domain IOVA range reserved for DMA_PMD mappings, and expose the
cookie accessors (dma_pmd_dma_window(), dma_pmd_dma_iovad(),
dma_pmd_window_owns()) and dma_info_to_prot() needed by the DMA_PMD
mapping layer in subsequent patches.
No functional change.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-iommu.c | 39 ++++++++++++++++++-
drivers/iommu/dma-iommu.h | 8 ++++
drivers/iommu/dma-pmd-kunit.c | 19 +++++++++
drivers/iommu/dma-pmd-priv.h | 72 +++++++++++++++++++++++++++++++++++
4 files changed, 137 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 58c624513cd43..70aee7a3020c8 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -18,6 +18,7 @@
#include <linux/gfp.h>
#include <linux/huge_mm.h>
#include <linux/iommu.h>
+#include <linux/dma-pmd.h>
#include <linux/iommu-dma.h>
#include <linux/iova.h>
#include <linux/irq.h>
@@ -36,6 +37,7 @@
#include <trace/events/swiotlb.h>
#include "dma-iommu.h"
+#include "dma-pmd-priv.h"
#include "iommu-pages.h"
struct iommu_dma_msi_page {
@@ -75,6 +77,8 @@ struct iommu_dma_cookie {
struct iommu_domain *fq_domain;
/* Options for dma-iommu use */
struct iommu_dma_options options;
+ /* IOVA window reserved for DMA_PMD pages. Nothing else can go here. */
+ struct dma_pmd_window win_dma_pmd;
};
struct iommu_dma_msi_cookie {
@@ -732,7 +736,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev
*
* Return: corresponding IOMMU API page protection flags
*/
-static int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
+int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
unsigned long attrs)
{
int prot;
@@ -1214,6 +1218,39 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,
return iova_offset(iovad, phys | size);
}
+/**
+ * dma_pmd_dma_window - This domain's IOVA window for DMA_PMD
+ * @domain: Domain to look at
+ *
+ * For the DMA_PMD code's slow paths, which need a window but do not have the
+ * cookie layout. The DMA map path is handed the pointer by its caller instead,
+ * so this is never called at map frequency.
+ *
+ * Return: the window, or NULL if @domain is NULL or does not carry a DMA-IOVA
+ * cookie. The cookie shares a union with the MSI, iommufd and fault-handler
+ * ones, so the type has to be tested rather than the pointer. An identity,
+ * passthrough, or MSI-only domain has no window, and a device can be moved to
+ * one after its pool was created (e.g. via sysfs domain type changes or VFIO
+ * attachment).
+ */
+struct dma_pmd_window *dma_pmd_dma_window(struct iommu_domain *domain)
+{
+ /*
+ * Also decline in kdump kernels (iommu_deferred_attach_enabled);
+ * dma_pmd_dma_iovad() uses this check so win->size stays 0.
+ */
+ if (static_branch_unlikely(&iommu_deferred_attach_enabled) ||
+ !domain || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
+ return NULL;
+
+ return &domain->iova_cookie->win_dma_pmd;
+}
+
+struct iova_domain *dma_pmd_dma_iovad(struct iommu_domain *domain)
+{
+ return dma_pmd_dma_window(domain) ? &domain->iova_cookie->iovad : NULL;
+}
+
dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
enum dma_data_direction dir, unsigned long attrs)
{
diff --git a/drivers/iommu/dma-iommu.h b/drivers/iommu/dma-iommu.h
index 040d002525632..241f8de002e50 100644
--- a/drivers/iommu/dma-iommu.h
+++ b/drivers/iommu/dma-iommu.h
@@ -7,6 +7,9 @@
#include <linux/iommu.h>
+struct dma_pmd_window;
+struct iova_domain;
+
#ifdef CONFIG_IOMMU_DMA
void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
@@ -22,6 +25,11 @@ void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
phys_addr_t msi_addr);
+int dma_info_to_prot(enum dma_data_direction dir, bool coherent, unsigned long attrs);
+
+struct iova_domain *dma_pmd_dma_iovad(struct iommu_domain *domain);
+struct dma_pmd_window *dma_pmd_dma_window(struct iommu_domain *domain);
+
extern bool iommu_dma_forcedac;
#else /* CONFIG_IOMMU_DMA */
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index 4e38cc5970433..16312014b8976 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -140,11 +140,30 @@ static void test_pool_alloc_and_recycle(struct kunit *test)
dma_pmd_pool_destroy(pool_wm);
}
+static void test_window_helpers(struct kunit *test)
+{
+ struct dma_pmd_window win = {};
+
+ KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
+
+ /* Empty window (size == 0) never owns any IOVA. */
+ KUNIT_EXPECT_FALSE(test, dma_pmd_window_owns(&win, 0));
+ KUNIT_EXPECT_FALSE(test, dma_pmd_window_owns(&win, SZ_4G));
+
+ win.base = SZ_4G;
+ win.size = SZ_2G;
+ KUNIT_EXPECT_FALSE(test, dma_pmd_window_owns(&win, SZ_4G - 1));
+ KUNIT_EXPECT_TRUE(test, dma_pmd_window_owns(&win, SZ_4G));
+ KUNIT_EXPECT_TRUE(test, dma_pmd_window_owns(&win, SZ_4G + SZ_2G - 1));
+ KUNIT_EXPECT_FALSE(test, dma_pmd_window_owns(&win, SZ_4G + SZ_2G));
+}
+
static struct kunit_case dma_pmd_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_pooled_toggle),
KUNIT_CASE(test_pool_alloc_and_recycle),
+ KUNIT_CASE(test_window_helpers),
{}
};
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 39295d0add51d..8104897572921 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -9,10 +9,46 @@
#include <linux/spinlock.h>
#include <linux/types.h>
+/**
+ * struct dma_pmd_window - A domain's IOVA window reserved for DMA_PMD pages
+ * @base: First IOVA of the window. A page at @phys is mapped, in every domain
+ * that has a window, at @base + @phys.
+ * @size: Window size in bytes, or 0 if this domain has no window, either
+ * because nothing has pooled through it yet, or because it could not
+ * find a free range that large. 0 must make both the map and the unmap
+ * side decline, see dma_pmd_window_owns().
+ * @domain_idx: Dense domain index + DMA_PMD_IDX_FIRST, used for per-PMD-page
+ * presence (bit @domain_idx - DMA_PMD_IDX_FIRST), or one of
+ * DMA_PMD_IDX_NONE, DMA_PMD_IDX_NOHUGE or DMA_PMD_IDX_NOSPACE while
+ * @size is 0. Of the three sentinels, only DMA_PMD_IDX_NONE is retried.
+ *
+ * Lives in the domain's iommu_dma_cookie. The window is a real allocation out
+ * of the domain's iova_domain, so no ordinary IOVA can ever fall inside it,
+ * which is what lets the unmap path recognise a pooled IOVA by range alone.
+ */
+struct dma_pmd_window {
+ dma_addr_t base;
+ u64 size;
+ u16 domain_idx;
+};
+
#ifdef CONFIG_DMA_PMD
#define DMA_PMD_BLOCKS(order) (1U << (PMD_ORDER - (order)))
+/*
+ * Sentinel values of dma_pmd_window.domain_idx below DMA_PMD_IDX_FIRST:
+ * never attempted (0, matching kzalloc), or permanently declined. A real
+ * domain index is >= DMA_PMD_IDX_FIRST (subtract DMA_PMD_IDX_FIRST for
+ * the bitmap bit position).
+ */
+enum {
+ DMA_PMD_IDX_NONE, /* not attempted yet */
+ DMA_PMD_IDX_NOHUGE, /* @domain can never pool */
+ DMA_PMD_IDX_NOSPACE, /* no index or no IOVA range */
+ DMA_PMD_IDX_FIRST, /* first valid domain index */
+};
+
/*
* Locking
* -------
@@ -167,5 +203,41 @@ struct dma_pmd_meta *dma_pmd_meta_from_phys(phys_addr_t pa);
unsigned long dma_pmd_meta_to_pfn(const struct dma_pmd_meta *m);
phys_addr_t dma_pmd_meta_to_phys(const struct dma_pmd_meta *m);
+static inline bool dma_is_pmd_phys(phys_addr_t phys)
+{
+ return dma_is_pmd_page(phys >> PAGE_SHIFT);
+}
+
+/**
+ * dma_pmd_window_owns - Was @dma handed out by a DMA_PMD mapping?
+ * @win: The unmapping domain's window
+ * @dma: IOVA being unmapped
+ *
+ * Only DMA_PMD pages can be mapped within the reserved IOVA window
+ * so a range check suffices.
+ * A domain that has no window has @size 0, so the unsigned subtraction
+ * underflows to a huge value and the test is always false. That is the same
+ * check that stops the map path from using a window the domain does not own.
+ */
+static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_addr_t dma)
+{
+ /* Acquire pairs with the release of @size in dma_pmd_window_assign(). */
+ u64 size = smp_load_acquire(&win->size);
+
+ return size && dma - win->base < size;
+}
+
+#else /* !CONFIG_DMA_PMD */
+
+static inline bool dma_is_pmd_phys(phys_addr_t phys)
+{
+ return false;
+}
+
+static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_addr_t dma)
+{
+ return false;
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _DRIVERS_IOMMU_DMA_PMD_PRIV_H */
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 07/22] iommu/dma: release DMA_PMD domain mappings on domain teardown
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (5 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 06/22] iommu/dma: reserve a per-domain IOVA window for DMA_PMD pages Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 08/22] iommu/dma: use per-domain IOVA window to map DMA_PMD memory Luigi Rizzo
` (43 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Track per-domain PMD_SIZE leaf PTE presence in a 64-entry domain registry
(dma_pmd_domains[]) and a per-PMD-page bitmask (p2m->domains_mapped),
unmapping active domains in dma_pmd_unmap_all() when a DMA_PMD page is
retired.
When an IOMMU DMA domain is torn down at runtime (for example across
VFIO bind/unbind or Dynamic Memory Security domain switches), its page
tables and IOVA domain are freed by the caller, but DMA_PMD pages that were
mapped into it may outlive the domain in per-queue or per-CPU pools.
Add dma_pmd_domain_release() and call it from iommu_put_dma_cookie()
before freeing the cookie:
- Clear the domain's slot in dma_pmd_domains[] and mark it draining so
a racing dma_pmd_unmap_all() skips the dying domain instead of
calling iommu_unmap() on freed page tables.
- Walk all live pools (including destroyed pools that still hold
in-flight DMA_PMD pages via pool->refcount) and clear the domain's bit in
every p2m->domains_mapped bitmap via dma_pmd_forget_domain().
- Flush dma_pmd_reclaim_work and wait on dma_pmd_srcu so any in-flight
unmapper finishes before the domain index is released for reuse.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/dma-iommu.c | 6 +
drivers/iommu/dma-pmd-kunit.c | 8 ++
drivers/iommu/dma-pmd-map.c | 220 ++++++++++++++++++++++++++++++++++
drivers/iommu/dma-pmd-pool.c | 59 ++++++---
drivers/iommu/dma-pmd-priv.h | 34 ++++++
6 files changed, 314 insertions(+), 15 deletions(-)
create mode 100644 drivers/iommu/dma-pmd-map.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index de2c3bad9aa3e..e701b36b9b605 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o dma-pmd-pool.o
+obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o dma-pmd-pool.o dma-pmd-map.o
obj-$(CONFIG_DMA_PMD_META_KUNIT_TEST) += dma-pmd-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 70aee7a3020c8..ee14175b79bc2 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -430,6 +430,12 @@ void iommu_put_dma_cookie(struct iommu_domain *domain)
struct iommu_dma_cookie *cookie = domain->iova_cookie;
struct iommu_dma_msi_page *msi, *tmp;
+ /*
+ * Drop any DMA_PMD mappings cached against this domain before its
+ * page tables and IOVA domain go away.
+ */
+ dma_pmd_domain_release(domain);
+
if (cookie->iovad.granule) {
iommu_dma_free_fq(cookie);
put_iova_domain(&cookie->iovad);
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index 16312014b8976..7f4251845ba49 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -5,6 +5,7 @@
#include <kunit/test.h>
#include <linux/gfp.h>
#include <linux/dma-pmd.h>
+#include <linux/iommu.h>
#include <linux/mm.h>
#include "dma-pmd-priv.h"
@@ -142,7 +143,9 @@ static void test_pool_alloc_and_recycle(struct kunit *test)
static void test_window_helpers(struct kunit *test)
{
+ struct iommu_domain dummy_domain = {};
struct dma_pmd_window win = {};
+ struct dma_pmd_pool *pool;
KUNIT_ASSERT_EQ(test, dma_pmd_meta_init(), 0);
@@ -156,6 +159,11 @@ static void test_window_helpers(struct kunit *test)
KUNIT_EXPECT_TRUE(test, dma_pmd_window_owns(&win, SZ_4G));
KUNIT_EXPECT_TRUE(test, dma_pmd_window_owns(&win, SZ_4G + SZ_2G - 1));
KUNIT_EXPECT_FALSE(test, dma_pmd_window_owns(&win, SZ_4G + SZ_2G));
+ /* Releasing an unregistered domain with an active pool is a safe no-op. */
+ pool = dma_pmd_pool_create(0, 1);
+ KUNIT_ASSERT_NOT_NULL(test, pool);
+ dma_pmd_domain_release(&dummy_domain);
+ dma_pmd_pool_destroy(pool);
}
static struct kunit_case dma_pmd_meta_test_cases[] = {
diff --git a/drivers/iommu/dma-pmd-map.c b/drivers/iommu/dma-pmd-map.c
new file mode 100644
index 0000000000000..cffa94a7c7c84
--- /dev/null
+++ b/drivers/iommu/dma-pmd-map.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * DMA_PMD per-domain IOVA window management and IOMMU mapping.
+ *
+ * See Documentation/core-api/dma-pmd.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-map-ops.h>
+#include <linux/dma-mapping.h>
+#include <linux/dma-pmd.h>
+#include <linux/export.h>
+#include <linux/iommu.h>
+#include <linux/mm.h>
+#include <linux/spinlock.h>
+#include <linux/srcu.h>
+
+#include "dma-iommu.h"
+#include "dma-pmd-priv.h"
+
+/*
+ * Index -> domain, for PMD pages that have to unmap themselves from every domain
+ * that installed them. A slot is cleared before the matching bits are cleared
+ * in dma_pmd_domain_release(), so a PMD page racing with a dying domain sees
+ * NULL and correctly leaves the doomed page tables alone.
+ *
+ * @base is cached here rather than read back from the domain's DMA cookie, so
+ * that teardown never has to dereference a cookie that may be about to be
+ * freed.
+ *
+ * The lock also serialises index allocation, which happens from the DMA map
+ * path and so cannot sleep.
+ */
+static struct {
+ struct iommu_domain *domain;
+ dma_addr_t base;
+ bool draining;
+} dma_pmd_domains[DMA_PMD_MAX_DOMAINS] __read_mostly;
+static DEFINE_SPINLOCK(dma_pmd_domains_lock);
+
+/*
+ * Held by dma_pmd_unmap_all() across the registry read and the iommu_unmap()
+ * that follows, and waited on by dma_pmd_domain_release() once it has cleared
+ * the slot. Without it an unmapper that sampled a domain a moment before the
+ * slot was cleared would walk into page tables their owner is already freeing.
+ * SRCU rather than RCU to avoid holding RCU across IOTLB flushes.
+ */
+DEFINE_SRCU(dma_pmd_srcu);
+
+/**
+ * dma_pmd_unmap_all - Remove a PMD page's PTE from every domain holding it
+ * @meta: PMD page metadata structure
+ *
+ * The window itself is a permanent allocation out of each domain's
+ * iova_domain and is deliberately left alone; only the leaf PTE goes away.
+ *
+ * The bits are cleared before the unmaps, not after. A mapper that observes a
+ * cleared bit takes the slow path and blocks on @meta->map_lock, so it either
+ * re-installs the PTE or waits; the reverse order would leave a window in
+ * which the bit still claims a translation that has already been torn down.
+ *
+ * A domain that has since been released has a NULL registry slot and is
+ * skipped: its page tables are being freed by their owner and must not be
+ * touched here.
+ *
+ * Cost: Slow path (teardown / shrinker only); one iommu_unmap() per domain.
+ * Locking: Acquires @meta->map_lock and @dma_pmd_domains_lock (irqsave), and
+ * drops both before unmapping. Holds @dma_pmd_srcu across the
+ * registry read and the unmap, which is what keeps the domain alive
+ * in between.
+ * Frequency: Rare (only when a PMD page is released back to the buddy
+ * allocator or when a pool is destroyed).
+ */
+void dma_pmd_unmap_all(struct dma_pmd_meta *meta)
+{
+ phys_addr_t phys = dma_pmd_meta_to_phys(meta);
+ unsigned long mapped, flags;
+ int idx, srcu_idx;
+
+ srcu_idx = srcu_read_lock(&dma_pmd_srcu);
+ spin_lock_irqsave(&meta->map_lock, flags);
+ mapped = meta->domains_mapped;
+ meta->domains_mapped = 0;
+ spin_unlock_irqrestore(&meta->map_lock, flags);
+
+ if (!mapped) {
+ srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
+ return;
+ }
+
+ for_each_set_bit(idx, &mapped, DMA_PMD_MAX_DOMAINS) {
+ struct iommu_domain *domain;
+ dma_addr_t base;
+
+ spin_lock_irqsave(&dma_pmd_domains_lock, flags);
+ domain = dma_pmd_domains[idx].domain;
+ base = dma_pmd_domains[idx].base;
+ spin_unlock_irqrestore(&dma_pmd_domains_lock, flags);
+
+ if (!domain)
+ continue;
+
+ iommu_unmap(domain, base + phys, PMD_SIZE);
+ }
+ srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
+}
+
+/**
+ * dma_pmd_forget_domain - Drop a PMD page's PTE record for a dying domain
+ * @meta: PMD page metadata structure
+ * @idx: Index of the domain being torn down
+ *
+ * Deliberately does not unmap: @domain is being destroyed by its owner, which
+ * frees the page tables and the IOVA domain wholesale. Touching it here is
+ * exactly the use-after-free this is meant to prevent, so the bit is only
+ * forgotten.
+ *
+ * Locking: Acquires @meta->map_lock (irqsave).
+ *
+ * Return: 1 if this PMD page had a mapping in @idx, 0 otherwise.
+ */
+unsigned int dma_pmd_forget_domain(struct dma_pmd_meta *meta, int idx)
+{
+ unsigned int dropped = 0;
+ unsigned long flags;
+
+ if (!(READ_ONCE(meta->domains_mapped) & BIT(idx)))
+ return 0;
+
+ spin_lock_irqsave(&meta->map_lock, flags);
+ if (meta->domains_mapped & BIT(idx)) {
+ meta->domains_mapped &= ~BIT(idx);
+ dropped = 1;
+ }
+ spin_unlock_irqrestore(&meta->map_lock, flags);
+
+ return dropped;
+}
+
+/**
+ * dma_pmd_domain_release - Forget every PMD page mapping for @domain
+ * @domain: DMA domain being destroyed
+ *
+ * A PMD page records, as a bit per domain index, which domains hold its PMD leaf
+ * PTE. Those bits are otherwise only cleared when the PMD page itself is
+ * released, so a domain that comes and goes - which this tree does at runtime,
+ * see the DMS comments in __iommu_dma_map_phys() - would permanently consume
+ * one index per generation, and every PMD page that ever mapped through it would
+ * keep claiming a mapping in page tables that no longer exist.
+ *
+ * The registry slot is cleared first and the per-PMD-page bits afterwards.
+ * dma_pmd_unmap_all() reads the bit and then the slot, so that order means it
+ * either sees no bit, or sees a NULL slot and skips: it can never unmap
+ * through a domain whose page tables its owner is freeing. An unmapper that
+ * sampled the slot just before it was cleared is waited out on @dma_pmd_srcu.
+ *
+ * The slot stays reserved until the walk and drain are done, so the index
+ * cannot be handed to a new domain while PMD pages still carry its bit.
+ *
+ * Cost: O(PMD pages) across all pools, but only on domain teardown.
+ * Locking: Takes @dma_pmd_domains_lock, then @dma_pmd_pools_lock, then each
+ * @pool->lock, then each @meta->map_lock. Must be called from process
+ * context.
+ */
+void dma_pmd_domain_release(struct iommu_domain *domain)
+{
+ unsigned int dropped = 0;
+ unsigned long flags;
+ int idx;
+
+ /* Nothing has ever been pooled, so nothing can reference @domain. */
+ if (likely(!dma_pmd_meta_base()))
+ return;
+
+ spin_lock_irqsave(&dma_pmd_domains_lock, flags);
+ for (idx = 0; idx < DMA_PMD_MAX_DOMAINS; idx++) {
+ if (dma_pmd_domains[idx].domain == domain) {
+ dma_pmd_domains[idx].domain = NULL;
+ /*
+ * Keep the slot reserved until every PMD page below has
+ * forgotten it. A new domain handed this index now
+ * would inherit the stale bits, and its first map of
+ * such a PMD page would skip iommu_map() and return an
+ * IOVA with no PTE behind it.
+ */
+ dma_pmd_domains[idx].draining = true;
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&dma_pmd_domains_lock, flags);
+
+ /* @domain never pooled, so no PMD page can be holding a bit for it. */
+ if (idx == DMA_PMD_MAX_DOMAINS)
+ return;
+
+ dropped += dma_pmd_pools_forget_domain(idx);
+
+ /*
+ * Any PMD page removed from a pool list before the walk above is either
+ * already on @dma_pmd_free_list (placed there under @pool->lock in
+ * __dma_pmd_free_page()) or being unmapped under @dma_pmd_srcu (in
+ * the shrinker or dma_pmd_pool_destroy()). Drain the reclaim list and
+ * wait out @dma_pmd_srcu before allowing @idx to be reused.
+ *
+ * synchronize_srcu() also waits out any dma_pmd_unmap_all() that
+ * sampled @domain before the slot was cleared above.
+ */
+ synchronize_srcu(&dma_pmd_srcu);
+
+ /* No PMD page claims @idx any more, so it can be reused. */
+ spin_lock_irqsave(&dma_pmd_domains_lock, flags);
+ dma_pmd_domains[idx].draining = false;
+ spin_unlock_irqrestore(&dma_pmd_domains_lock, flags);
+
+ if (dropped)
+ pr_debug("dma_pmd: released %u PMD page mappings for domain %p (index %d)\n",
+ dropped, domain, idx);
+}
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index 78f2537772d1d..1d4244ba422a3 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -74,9 +74,8 @@ static void dma_pmd_pool_free_kref(struct kref *kref)
* pushed locklessly onto dma_pmd_free_list and dma_pmd_reclaim_work is
* scheduled on dma_pmd_wq.
* 2. In process context, dma_pmd_release_page() unmaps all IOMMU domains
- * (dma_pmd_unmap_all(), added in a later commit), clears meta->pooled so
- * new lockless readers stop entering @meta, and queues
- * dma_pmd_release_page_rcu() via call_rcu().
+ * (dma_pmd_unmap_all()), clears meta->pooled so new lockless readers stop
+ * entering @meta, and queues dma_pmd_release_page_rcu() via call_rcu().
* 3. After an RCU grace period (once no concurrent dma_pmd_free_page() reader
* can still be dereferencing @meta), dma_pmd_release_page_rcu() unfreezes
* all order-pool->order blocks (already split by split_page_compound()),
@@ -125,14 +124,14 @@ static void dma_pmd_release_page_rcu(struct rcu_head *head)
* dma_pmd_release_page - Retire a PMD page and schedule its buddy release
* @meta: PMD page metadata structure (must have all usable blocks idle)
*
- * Cost: Slow path; clears the membership bit. The blocks themselves are
- * returned after an RCU grace period.
- * Locking: Must not hold @pool->lock.
+ * Cost: Slow path; unmaps the IOMMU domains and clears the membership bit.
+ * The blocks themselves are returned after an RCU grace period.
+ * Locking: Process context (calls iommu_unmap). Must not hold @pool->lock.
* Frequency: Rare (background reclaim workqueue, shrinker, pool destruction).
*/
static void dma_pmd_release_page(struct dma_pmd_meta *meta)
{
- /* dma_pmd_unmap_all(meta) will be called here once IOMMU mappings are added. */
+ dma_pmd_unmap_all(meta);
/*
* Clear membership before the grace period. A reader that passed
@@ -176,6 +175,31 @@ static void dma_pmd_schedule_reclaim(void)
schedule_work(&dma_pmd_reclaim_work);
}
+unsigned int dma_pmd_pools_forget_domain(int idx)
+{
+ struct dma_pmd_pool *pool;
+ struct dma_pmd_meta *meta;
+ unsigned int dropped = 0;
+ unsigned long flags;
+
+ mutex_lock(&dma_pmd_pools_lock);
+ list_for_each_entry(pool, &dma_pmd_pools, node) {
+ spin_lock_irqsave(&pool->lock, flags);
+ list_for_each_entry(meta, &pool->partial, list)
+ dropped += dma_pmd_forget_domain(meta, idx);
+ list_for_each_entry(meta, &pool->idle, list)
+ dropped += dma_pmd_forget_domain(meta, idx);
+ list_for_each_entry(meta, &pool->full, list)
+ dropped += dma_pmd_forget_domain(meta, idx);
+ spin_unlock_irqrestore(&pool->lock, flags);
+ }
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ dma_pmd_schedule_reclaim();
+ flush_work(&dma_pmd_reclaim_work);
+ return dropped;
+}
+
/**
* dma_pmd_pool_create - Create a DMA_PMD page pool
* @order: Block order to dispense, must be <= PMD_ORDER.
@@ -232,12 +256,10 @@ EXPORT_SYMBOL(dma_pmd_pool_create);
* dma_pmd_pool_destroy - Destroy a DMA_PMD page pool and unmap cached PMD pages
* @pool: Pool to destroy
*
- * Frees completely idle 2M pages immediately. Device DMA must be quiesced by
- * the caller first, but blocks already handed to the networking stack (e.g.,
- * RX skbs waiting in socket queues or TX skbs in TCP retransmit queues) may
- * still be in flight; those 2M pages stay on @pool->partial / @pool->full and
- * keep @pool alive on @dma_pmd_pools via @pool->refcount until their last
- * blocks free.
+ * Unmaps and frees completely idle PMD pages. If any blocks are still in flight
+ * (e.g., held by socket queues or SKBs), the pool structure and active pages
+ * remain on @dma_pmd_pools via @pool->refcount (visible to dma_pmd_domain_release())
+ * and are automatically unmapped and reclaimed as their last blocks free.
*
* Cost: Control plane teardown; flushes async reclaim workqueue.
* Locking: Process context (may sleep in flush_work). Acquires @pool->lock.
@@ -250,10 +272,12 @@ struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
struct dma_pmd_meta *meta, *tmp;
LIST_HEAD(release_list);
unsigned long flags;
+ int srcu_idx;
if (!pool)
return NULL;
+ srcu_idx = srcu_read_lock(&dma_pmd_srcu);
spin_lock_irqsave(&pool->lock, flags);
pool->destroyed = true;
@@ -267,6 +291,7 @@ struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
list_del(&meta->list);
dma_pmd_release_page(meta);
}
+ srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
kref_put(&pool->refcount, dma_pmd_pool_free_kref);
flush_work(&dma_pmd_reclaim_work);
@@ -707,10 +732,12 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
unsigned long freed = 0;
LIST_HEAD(release_list);
unsigned long flags;
+ int srcu_idx;
if (!mutex_trylock(&dma_pmd_pools_lock))
return SHRINK_STOP;
+ srcu_idx = srcu_read_lock(&dma_pmd_srcu);
list_for_each_entry(pool, &dma_pmd_pools, node) {
/*
* Every PMD page on @idle is a candidate, so this detaches one
@@ -734,11 +761,15 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
mutex_unlock(&dma_pmd_pools_lock);
- /* Retiring must not run under @pool->lock. */
+ /*
+ * Retiring is done outside both locks: dma_pmd_release_page() unmaps
+ * from the IOMMU, which is far too long to hold pool->lock for.
+ */
list_for_each_entry_safe(meta, tmp, &release_list, list) {
list_del(&meta->list);
dma_pmd_release_page(meta);
}
+ srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
/*
* Both counts are in pages, matching dma_pmd_shrink_count(). A
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 8104897572921..8477a8edfac63 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -7,8 +7,11 @@
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/spinlock.h>
+#include <linux/srcu.h>
#include <linux/types.h>
+struct iommu_domain;
+
/**
* struct dma_pmd_window - A domain's IOVA window reserved for DMA_PMD pages
* @base: First IOVA of the window. A page at @phys is mapped, in every domain
@@ -36,6 +39,15 @@ struct dma_pmd_window {
#define DMA_PMD_BLOCKS(order) (1U << (PMD_ORDER - (order)))
+/*
+ * Number of IOMMU domains that may use DMA_PMD at once.
+ *
+ * Each such domain is given a dense index, which is the bit position it uses
+ * in dma_pmd_meta.domains_mapped. One word per PMD page records every domain
+ * that has mapped this PMD page. Used to speed up map and unmap.
+ */
+#define DMA_PMD_MAX_DOMAINS BITS_PER_LONG
+
/*
* Sentinel values of dma_pmd_window.domain_idx below DMA_PMD_IDX_FIRST:
* never attempted (0, matching kzalloc), or permanently declined. A real
@@ -71,6 +83,17 @@ enum {
* meta->map_lock IRQ-safe spinlock over one PMD frame's domains_mapped
* bitmap. Innermost, and never held across anything that
* sleeps.
+ *
+ * The full order is dma_pmd_pools_lock -> pool->lock -> meta->map_lock, as
+ * taken by dma_pmd_domain_release().
+ *
+ * dma_pmd_domains_lock IRQ-safe spinlock over the domain index table and the
+ * per-domain windows. Taken on its own; it is never
+ * nested with dma_pmd_pools_lock in either direction.
+ *
+ * dma_pmd_srcu Read side lets the unmap path dereference a domain out
+ * of dma_pmd_domains[] without blocking a concurrent
+ * dma_pmd_domain_release().
*/
/*
@@ -186,6 +209,7 @@ struct dma_pmd_pool {
} ____cacheline_aligned;
extern unsigned long dma_pmd_meta_nframes;
+extern struct srcu_struct dma_pmd_srcu;
static inline struct dma_pmd_meta *dma_pmd_meta_base(void)
{
@@ -203,6 +227,10 @@ struct dma_pmd_meta *dma_pmd_meta_from_phys(phys_addr_t pa);
unsigned long dma_pmd_meta_to_pfn(const struct dma_pmd_meta *m);
phys_addr_t dma_pmd_meta_to_phys(const struct dma_pmd_meta *m);
+unsigned int dma_pmd_pools_forget_domain(int idx);
+void dma_pmd_unmap_all(struct dma_pmd_meta *meta);
+unsigned int dma_pmd_forget_domain(struct dma_pmd_meta *meta, int idx);
+
static inline bool dma_is_pmd_phys(phys_addr_t phys)
{
return dma_is_pmd_page(phys >> PAGE_SHIFT);
@@ -227,6 +255,8 @@ static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_add
return size && dma - win->base < size;
}
+void dma_pmd_domain_release(struct iommu_domain *domain);
+
#else /* !CONFIG_DMA_PMD */
static inline bool dma_is_pmd_phys(phys_addr_t phys)
@@ -239,5 +269,9 @@ static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_add
return false;
}
+static inline void dma_pmd_domain_release(struct iommu_domain *domain)
+{
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _DRIVERS_IOMMU_DMA_PMD_PRIV_H */
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 08/22] iommu/dma: use per-domain IOVA window to map DMA_PMD memory
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (6 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 07/22] iommu/dma: release DMA_PMD domain mappings on domain teardown Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 09/22] iommu/dma: Add DMA_PMD arena allocator Luigi Rizzo
` (42 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Map each DMA_PMD page at win->base + phys inside the domain's reserved
IOVA window and track per-domain leaf PDE residency in
p2m->domains_mapped:
- dma_pmd_window_assign() assigns a dense domain index in
dma_pmd_domains[] and allocates the per-domain IOVA window via
dma_pmd_dma_window_alloc() on first use.
- dma_pmd_dma_window_alloc() reserves a PMD_SIZE-aligned range via
alloc_iova() from the domain's iova_domain so no ordinary IOVA can
land inside it, allowing unmap to identify pooled IOVAs by range check.
- dma_pmd_dma_map_phys() returns win->base + phys in O(1) when the
domain's bit is set in p2m->domains_mapped, installing the PDE_ORDER leaf
PTE on the first map in a domain.
- iommu_dma_map_phys() and iommu_dma_unmap_phys() route DMA_PMD
buffers through the per-domain window, making dma_unmap_*() an O(1)
range check with no page-table walk or IOTLB flush.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Kconfig | 1 +
drivers/iommu/dma-iommu.c | 27 +++-
drivers/iommu/dma-pmd-map.c | 275 +++++++++++++++++++++++++++++++++++
drivers/iommu/dma-pmd-priv.h | 33 +++++
drivers/iommu/iommu.c | 2 +-
5 files changed, 336 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 1bb347fd9da4a..7cb5b036e0f3b 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -164,6 +164,7 @@ config DMA_PMD
depends on IOMMU_DMA
depends on X86_64 || (ARM64 && ARM64_4K_PAGES)
depends on !PREEMPT_RT
+ select NEED_SG_DMA_FLAGS
default y
help
Hand out DMA buffers carved out of PMD_SIZE physically contiguous
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index ee14175b79bc2..6d33e245530ec 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1286,6 +1286,14 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
arch_sync_dma_flush();
}
+ if (dma_is_pmd_phys(phys)) {
+ iova = dma_pmd_dma_map_phys(dev, domain,
+ &domain->iova_cookie->win_dma_pmd,
+ phys, size, prot, dma_mask);
+ if (likely(iova != DMA_MAPPING_ERROR))
+ return iova;
+ }
+
iova = __iommu_dma_map(dev, phys, size, prot, dma_mask);
if (iova == DMA_MAPPING_ERROR &&
!(attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)))
@@ -1296,14 +1304,19 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
size_t size, enum dma_data_direction dir, unsigned long attrs)
{
+ struct iommu_domain *domain = iommu_get_dma_domain(dev);
phys_addr_t phys;
+ /* DMA_PMD mapped buffer: nothing to unmap or release. */
+ if (dma_pmd_window_owns(&domain->iova_cookie->win_dma_pmd, dma_handle))
+ return;
+
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) {
__iommu_dma_unmap(dev, dma_handle, size);
return;
}
- phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle);
+ phys = iommu_iova_to_phys(domain, dma_handle);
if (WARN_ON(!phys))
return;
@@ -1499,6 +1512,18 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
*/
break;
case PCI_P2PDMA_MAP_NONE:
+ if (dma_is_pmd_phys(sg_phys(s))) {
+ iova = dma_pmd_dma_map_phys(dev, domain,
+ &cookie->win_dma_pmd,
+ sg_phys(s), s_length,
+ prot, dma_get_mask(dev));
+ if (likely(iova != DMA_MAPPING_ERROR)) {
+ s->dma_address = iova;
+ sg_dma_len(s) = s_length;
+ sg_dma_mark_bus_address(s);
+ continue;
+ }
+ }
break;
case PCI_P2PDMA_MAP_BUS_ADDR:
/*
diff --git a/drivers/iommu/dma-pmd-map.c b/drivers/iommu/dma-pmd-map.c
index cffa94a7c7c84..565e10dd1a2e8 100644
--- a/drivers/iommu/dma-pmd-map.c
+++ b/drivers/iommu/dma-pmd-map.c
@@ -13,6 +13,8 @@
#include <linux/dma-pmd.h>
#include <linux/export.h>
#include <linux/iommu.h>
+#include <linux/iova.h>
+#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/spinlock.h>
#include <linux/srcu.h>
@@ -218,3 +220,276 @@ void dma_pmd_domain_release(struct iommu_domain *domain)
pr_debug("dma_pmd: released %u PMD page mappings for domain %p (index %d)\n",
dropped, domain, idx);
}
+
+/**
+ * dma_pmd_dma_window_alloc - Allocate an IOVA window for DMA_PMD mappings
+ * @dev: Device whose addressing limits the window must respect
+ * @domain: Domain to take the window from
+ * @size: Window size in bytes
+ *
+ * Allocates a contiguous IOVA range out of @domain's iova_domain on first use,
+ * aligned to PMD_SIZE matching the mapping. Callers on the map path still
+ * check each device's own dma_mask/bus_dma_limit against the resulting IOVA and
+ * fall back to per-buffer mapping if a narrower device shares @domain.
+ *
+ * Return: base IOVA, or 0 if no range that large is available.
+ */
+static dma_addr_t dma_pmd_dma_window_alloc(struct device *dev,
+ struct iommu_domain *domain, u64 *sizep)
+{
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ struct iova_domain *iovad = dma_pmd_dma_iovad(domain);
+ u64 min_size = ALIGN(PFN_PHYS(max_pfn), PMD_SIZE);
+ unsigned long shift, iova_len;
+ struct iova *new_iova;
+
+ if (!iovad)
+ return 0;
+
+ if (domain->geometry.force_aperture)
+ limit = min_t(u64, limit, domain->geometry.aperture_end);
+
+ if (*sizep > limit / 2 && limit / 2 >= min_size)
+ *sizep = ALIGN_DOWN(limit / 2, PMD_SIZE);
+
+ /* Ignore devices with too many constraints. */
+ if (limit <= DMA_BIT_MASK(32) || *sizep > limit - PMD_SIZE)
+ return 0;
+
+ shift = iova_shift(iovad);
+ iova_len = (*sizep + PMD_SIZE) >> shift;
+
+ /*
+ * Note: dev->iommu->pci_32bit_workaround is intentionally not consulted
+ * here. It is an advisory preference (enabled by default on all PCI
+ * devices).
+ */
+ new_iova = alloc_iova(iovad, iova_len, limit >> shift, false);
+ if (!new_iova)
+ return 0;
+
+ return ALIGN((dma_addr_t)new_iova->pfn_lo << shift, PMD_SIZE);
+}
+
+/**
+ * dma_pmd_window_assign - Give @domain an IOVA window and a domain index
+ * @dev: Device whose addressing limits the window must respect
+ * @domain: Domain to set up
+ * @win: @domain's window, from its DMA cookie
+ *
+ * Called the first time anything tries to pool through @domain. Failure is
+ * recorded in @win->domain_idx rather than retried: the allocation is a
+ * multi-terabyte search of the IOVA tree, and a domain that has no room for it
+ * once will not have room on the next buffer either. The caller tests that
+ * record without the lock, so the reason has to survive in the sentinel.
+ *
+ * A domain that cannot install a PMD leaf is declined. The whole benefit
+ * is the single large PTE; without it a 2M page would cost 512 4KB PTEs, which
+ * is worse than not pooling.
+ *
+ * Locking: Acquires @dma_pmd_domains_lock (irqsave). Callable from the DMA
+ * map path, so it never sleeps.
+ *
+ * Return: 0 if @win is usable on return, negative otherwise.
+ */
+int dma_pmd_window_assign(struct device *dev, struct iommu_domain *domain,
+ struct dma_pmd_window *win)
+{
+ u64 size = ALIGN(PFN_PHYS(dma_pmd_top_pfn()), PMD_SIZE);
+ unsigned long flags;
+ int idx, ret = 0;
+ dma_addr_t base;
+
+ if (!dev_is_dma_coherent(dev))
+ return -EOPNOTSUPP;
+
+ if (min_not_zero(dma_get_mask(dev), dev->bus_dma_limit) <= DMA_BIT_MASK(32))
+ return -ENOSPC;
+
+ spin_lock_irqsave(&dma_pmd_domains_lock, flags);
+
+ if (win->size) /* another CPU got here first */
+ goto out;
+
+ /* Declined earlier; only DMA_PMD_IDX_NONE means "not tried yet". */
+ if (win->domain_idx != DMA_PMD_IDX_NONE) {
+ ret = win->domain_idx == DMA_PMD_IDX_NOHUGE ? -EOPNOTSUPP : -ENOSPC;
+ goto out;
+ }
+
+ if (!(domain->pgsize_bitmap & PMD_SIZE)) {
+ dev_warn_ratelimited(dev,
+ "dma_pmd: domain lacks a %luMB page size, not pooling\n",
+ (unsigned long)(PMD_SIZE >> 20));
+ ret = -EOPNOTSUPP;
+ goto decline_nohuge;
+ }
+
+ for (idx = 0; idx < DMA_PMD_MAX_DOMAINS; idx++) {
+ if (!dma_pmd_domains[idx].domain && !dma_pmd_domains[idx].draining)
+ break;
+ }
+ if (idx == DMA_PMD_MAX_DOMAINS) {
+ pr_warn_ratelimited("dma_pmd: all %d domain indices in use, not pooling for domain %p\n",
+ DMA_PMD_MAX_DOMAINS, domain);
+ ret = -ENOSPC;
+ goto decline;
+ }
+
+ base = dma_pmd_dma_window_alloc(dev, domain, &size);
+ if (!base) {
+ dev_warn_ratelimited(dev,
+ "dma_pmd: no free %llu GB IOVA range for a DMA_PMD window\n",
+ size >> 30);
+ ret = -ENOSPC;
+ goto decline;
+ }
+
+ dma_pmd_domains[idx].domain = domain;
+ dma_pmd_domains[idx].base = base;
+ win->base = base;
+ win->domain_idx = idx + DMA_PMD_IDX_FIRST;
+ /*
+ * Publish .base and .domain_idx before .size. A non-zero .size is what
+ * tells both the map and the unmap side that the other two are valid;
+ * see dma_pmd_window_owns().
+ */
+ smp_store_release(&win->size, size);
+
+ dev_info(dev, "dma_pmd: domain %p index %d window %pad + %llu GB\n",
+ domain, idx, &base, size >> 30);
+ goto out;
+
+decline_nohuge:
+ win->domain_idx = DMA_PMD_IDX_NOHUGE;
+ goto out;
+
+decline:
+ win->domain_idx = DMA_PMD_IDX_NOSPACE;
+out:
+ spin_unlock_irqrestore(&dma_pmd_domains_lock, flags);
+ return ret;
+}
+
+/**
+ * dma_pmd_dma_map_phys - Derive the IOVA of a pool address, mapping if needed
+ * @dev: Device performing DMA
+ * @domain: @dev's DMA domain
+ * @win: @domain's DMA_PMD window, from its DMA cookie
+ * @phys: Physical address within a DMA_PMD page
+ * @size: Mapping size requested
+ * @prot: IOMMU protection flags requested by the caller
+ * @dma_mask: Device DMA mask
+ *
+ * The IOVA is not allocated or cached: it is @win->base plus the PMD page's
+ * physical address, so every block of every PMD page has a fixed
+ * address in this domain that both this function and the unmap path can
+ * compute. All the PMD page has to remember is whether the PMD leaf PTE has
+ * actually been installed here, which is one bit in @meta->domains_mapped.
+ *
+ * The PMD page is mapped with IOMMU_CACHE | IOMMU_READ | IOMMU_WRITE so that
+ * blocks can be used in any DMA direction. Callers requesting additional
+ * protection flags in @prot fall back to per-buffer mapping.
+ *
+ * Cost:
+ * - Fast path: O(1) lockless - one bit test and an add. No IOVA tree, no
+ * page table, and no metadata load beyond the bitmask word.
+ * - First use of a PMD page in a domain: one iommu_map() of the whole PMD
+ * page, once, under @meta->map_lock.
+ * Locking: Fast path is lockless. Slow path acquires @meta->map_lock
+ * (irqsave), and @dma_pmd_domains_lock on the very first PMD page
+ * mapped through @domain.
+ * Frequency: Very high (called on every dma_map_page / dma_map_single for
+ * buffers allocated from an dma_pmd_pool).
+ *
+ * Return: Mapped IOVA, or DMA_MAPPING_ERROR on failure.
+ */
+dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
+ struct dma_pmd_window *win, phys_addr_t phys,
+ size_t size, int prot, u64 dma_mask)
+{
+ int block_prot = IOMMU_CACHE | IOMMU_READ | IOMMU_WRITE;
+ unsigned long pfn = phys >> PAGE_SHIFT;
+ struct dma_pmd_meta *meta;
+ unsigned int domain_idx;
+ unsigned long flags;
+ dma_addr_t iova;
+ u64 win_size;
+ int ret;
+
+ if (unlikely((prot & ~block_prot) || !dev_is_dma_coherent(dev) ||
+ !dma_is_pmd_page(pfn) || (phys & (PMD_SIZE - 1)) + size > PMD_SIZE))
+ return DMA_MAPPING_ERROR;
+
+ meta = dma_pmd_meta_of_pfn(pfn);
+
+ /* Acquire pairs with the .size release in dma_pmd_window_assign(). */
+ win_size = smp_load_acquire(&win->size);
+
+ if (unlikely(!win_size)) {
+ u16 win_idx = READ_ONCE(win->domain_idx);
+
+ /*
+ * A declined domain stays declined, so the answer is cached in
+ * @domain_idx and read without @dma_pmd_domains_lock. Otherwise
+ * every map through such a domain would queue on a global
+ * spinlock only to be told no.
+ */
+ if (unlikely(win_idx == DMA_PMD_IDX_NOHUGE))
+ ret = -EOPNOTSUPP;
+ else if (unlikely(win_idx == DMA_PMD_IDX_NOSPACE))
+ ret = -ENOSPC;
+ else
+ ret = dma_pmd_window_assign(dev, domain, win);
+
+ if (ret)
+ return DMA_MAPPING_ERROR;
+ win_size = READ_ONCE(win->size);
+ }
+
+ iova = dma_pmd_window_iova(win, phys);
+
+ /*
+ * The window was sized against the first device to pool through this
+ * domain (and may have been clamped to the domain aperture). Mirror
+ * __iommu_dma_map() and bound by the window size, bus limit, and
+ * device mask.
+ */
+ if (unlikely(iova - win->base + size > win_size ||
+ iova + size - 1 > min_not_zero(dma_mask, dev->bus_dma_limit)))
+ return DMA_MAPPING_ERROR;
+
+ domain_idx = win->domain_idx - DMA_PMD_IDX_FIRST;
+ if (likely(test_bit(domain_idx, &meta->domains_mapped)))
+ return iova;
+
+ /* First use of this PMD page in this domain: install the leaf PTE. */
+ spin_lock_irqsave(&meta->map_lock, flags);
+ if (test_bit(domain_idx, &meta->domains_mapped)) {
+ /* raced, someone else mapped */
+ ret = 0;
+ } else {
+ ret = iommu_map(domain,
+ dma_pmd_window_iova(win, dma_pmd_meta_to_phys(meta)),
+ dma_pmd_meta_to_phys(meta), PMD_SIZE,
+ block_prot, GFP_ATOMIC);
+ if (!ret) {
+ /*
+ * A lockless reader that sees this bit skips the
+ * iommu_map() above and hands the IOVA straight to the
+ * device, so the leaf PTE has to be visible first.
+ * set_bit() carries no ordering of its own on a weakly
+ * ordered machine; the barrier is a no-op on x86,
+ * where the LOCK prefix already provides it.
+ */
+ smp_mb__before_atomic();
+ set_bit(domain_idx, &meta->domains_mapped);
+ }
+ }
+ spin_unlock_irqrestore(&meta->map_lock, flags);
+
+ if (unlikely(ret))
+ return DMA_MAPPING_ERROR;
+
+ return iova;
+}
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 8477a8edfac63..355f865f44057 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -2,6 +2,7 @@
#ifndef _DRIVERS_IOMMU_DMA_PMD_PRIV_H
#define _DRIVERS_IOMMU_DMA_PMD_PRIV_H
+#include <linux/dma-mapping.h>
#include <linux/dma-pmd.h>
#include <linux/kref.h>
#include <linux/list.h>
@@ -10,6 +11,7 @@
#include <linux/srcu.h>
#include <linux/types.h>
+struct device;
struct iommu_domain;
/**
@@ -220,6 +222,22 @@ static inline struct dma_pmd_meta *dma_pmd_meta_base(void)
return smp_load_acquire(&dma_pmd_meta_array);
}
+/*
+ * Highest PFN covered by the allocated @dma_pmd_meta_array reservation.
+ * Every domain's IOVA window is sized to match this bound.
+ */
+static inline unsigned long dma_pmd_top_pfn(void)
+{
+ return dma_pmd_meta_nframes << PMD_ORDER;
+}
+
+/* IOVA of @phys in @win. Only valid once the PMD page's PTE is installed. */
+static inline dma_addr_t dma_pmd_window_iova(const struct dma_pmd_window *win,
+ phys_addr_t phys)
+{
+ return win->base + phys;
+}
+
int dma_pmd_meta_init(void);
bool dma_pmd_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct dma_pmd_meta *dma_pmd_meta_of_pfn(unsigned long pfn);
@@ -230,6 +248,8 @@ phys_addr_t dma_pmd_meta_to_phys(const struct dma_pmd_meta *m);
unsigned int dma_pmd_pools_forget_domain(int idx);
void dma_pmd_unmap_all(struct dma_pmd_meta *meta);
unsigned int dma_pmd_forget_domain(struct dma_pmd_meta *meta, int idx);
+int dma_pmd_window_assign(struct device *dev, struct iommu_domain *domain,
+ struct dma_pmd_window *win);
static inline bool dma_is_pmd_phys(phys_addr_t phys)
{
@@ -255,6 +275,10 @@ static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_add
return size && dma - win->base < size;
}
+dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
+ struct dma_pmd_window *win, phys_addr_t phys,
+ size_t size, int prot, u64 dma_mask);
+
void dma_pmd_domain_release(struct iommu_domain *domain);
#else /* !CONFIG_DMA_PMD */
@@ -269,6 +293,15 @@ static inline bool dma_pmd_window_owns(const struct dma_pmd_window *win, dma_add
return false;
}
+static inline dma_addr_t dma_pmd_dma_map_phys(struct device *dev,
+ struct iommu_domain *domain,
+ struct dma_pmd_window *win,
+ phys_addr_t phys, size_t size,
+ int prot, u64 dma_mask)
+{
+ return DMA_MAPPING_ERROR;
+}
+
static inline void dma_pmd_domain_release(struct iommu_domain *domain)
{
}
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index cd1bca7ede9af..d918d1294eb0d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2867,7 +2867,7 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
while (i <= nents) {
phys_addr_t s_phys = sg_phys(sg);
- if (len && s_phys != start + len) {
+ if (len && (sg_dma_is_bus_address(sg) || s_phys != start + len)) {
ret = iommu_map_nosync(domain, iova + mapped, start,
len, prot, gfp);
if (ret)
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 09/22] iommu/dma: Add DMA_PMD arena allocator
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (7 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 08/22] iommu/dma: use per-domain IOVA window to map DMA_PMD memory Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 10/22] driver core: Add per-device dma_pmd_* sysfs attributes Luigi Rizzo
` (41 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Device driver use dma_alloc_attrs() for several long lived
device-accessible regions, such as queues and header buffers.
Add a compatible dma_pmd_arena allocator for DMA_PMD pages, which will be
later used by dma_alloc_attrs() for eligible allocations. Reserve a IOVA
region right after the one used to map physical ram, used specifically for
dma_pmd_arena allocations so they can be easily identifyed from their IOVA.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 3 +-
drivers/iommu/dma-pmd-arena.c | 385 ++++++++++++++++++++++++++++++++++
drivers/iommu/dma-pmd-kunit.c | 35 ++++
drivers/iommu/dma-pmd-map.c | 19 +-
drivers/iommu/dma-pmd-meta.c | 61 ++++--
drivers/iommu/dma-pmd-priv.h | 41 +++-
include/linux/dma-pmd.h | 9 +
7 files changed, 519 insertions(+), 34 deletions(-)
create mode 100644 drivers/iommu/dma-pmd-arena.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index e701b36b9b605..e96fcf309a0b2 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,8 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o dma-pmd-pool.o dma-pmd-map.o
+obj-$(CONFIG_DMA_PMD) += dma-pmd-meta.o dma-pmd-pool.o dma-pmd-map.o \
+ dma-pmd-arena.o
obj-$(CONFIG_DMA_PMD_META_KUNIT_TEST) += dma-pmd-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/dma-pmd-arena.c b/drivers/iommu/dma-pmd-arena.c
new file mode 100644
index 0000000000000..578df5be86a86
--- /dev/null
+++ b/drivers/iommu/dma-pmd-arena.c
@@ -0,0 +1,385 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * DMA_PMD coherent arena allocator (ARENA_REGION).
+ *
+ * See Documentation/core-api/dma-pmd.rst for the architecture overview.
+ */
+
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-map-ops.h>
+#include <linux/dma-mapping.h>
+#include <linux/dma-pmd.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/iommu.h>
+#include <linux/mm.h>
+#include <linux/sched/mm.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/vmalloc.h>
+
+#include "dma-iommu.h"
+#include "dma-pmd-priv.h"
+/*
+ * dma_pmd_arena (ARENA_REGION allocator for coherent buffers)
+ * -----------------------------------------------------------
+ * Packs long-lived coherent DMA allocations into PMD physical pages mapped
+ * via PMD IOMMU leaf PTEs in the 16GB ARENA_REGION at the bottom of each
+ * domain's DMA_PMD IOVA window.
+ *
+ * - Allocation & Mapping (dma_pmd_arena_alloc / dma_pmd_dma_alloc):
+ * Allocates a PAGE_SIZE-aligned, zeroed region of @size bytes in
+ * ARENA_REGION, returning the CPU virtual address and storing the IOVA in
+ * *@dma. Each PMD arena page is mapped into a single IOMMU domain and is
+ * never shared across domains. Allocations <= 1MB first try partially used
+ * PMD arena pages already mapped in the caller's domain before allocating a
+ * fresh PMD page. Allocations > 1MB allocate contiguous empty PMD slots in
+ * ARENA_REGION and vmap() them when spanning multiple PMD pages; if the
+ * trailing PMD page has leftover 4KB blocks, it becomes available for
+ * subsequent <= 1MB allocations in the same domain.
+ *
+ * - Unmapping & Release (dma_free_coherent / dma_pmd_free):
+ * dma_pmd_free() checks whether @dma_handle falls in @dev's ARENA_REGION and,
+ * if so, unreserves the 4KB blocks via dma_pmd_arena_free(). Once all 512 4KB
+ * blocks of a PMD slot are free, its PMD PTE is unmapped from its domain and
+ * the backing PMD page is returned to the buddy allocator.
+ */
+
+struct dma_pmd_arena {
+ spinlock_t lock; /* protects bitmaps and arena metadata */
+ DECLARE_BITMAP(fully_empty, DMA_PMD_ARENA_PAGES);
+ DECLARE_BITMAP(partial_empty, DMA_PMD_ARENA_PAGES);
+};
+
+/* Currently one instance for all allocations */
+static struct dma_pmd_arena dma_pmd_arena __cacheline_aligned_in_smp = {
+ .lock = __SPIN_LOCK_UNLOCKED(dma_pmd_arena.lock),
+ .fully_empty = { [0 ... BITS_TO_LONGS(DMA_PMD_ARENA_PAGES) - 1] = ~0UL },
+};
+
+/* Provide a contiguous VA range for multi-page allocations */
+static void *dma_pmd_arena_vmap(unsigned int slot, unsigned int npages)
+{
+ unsigned int nr_subpages = npages * DMA_PMD_BLOCKS(0);
+ unsigned int subpages_per_2m = DMA_PMD_BLOCKS(0);
+ struct page *page, **pages;
+ unsigned int i, j;
+ void *va;
+
+ pages = kvmalloc_array(nr_subpages, sizeof(*pages), GFP_KERNEL);
+ if (!pages)
+ return NULL;
+
+ for (i = 0; i < npages; i++) {
+ page = dma_pmd_arena_meta(slot + i)->arena_page;
+ for (j = 0; j < subpages_per_2m; j++)
+ pages[i * subpages_per_2m + j] = page + j;
+ }
+
+ va = dma_common_pages_remap(pages, nr_subpages << PAGE_SHIFT,
+ PAGE_KERNEL, __builtin_return_address(0));
+ if (!va)
+ kvfree(pages);
+ return va;
+}
+
+/* Provide a contiguous IOVA range for allocations */
+static int dma_pmd_arena_map_domain(struct device *dev, struct iommu_domain *domain,
+ struct dma_pmd_window *win, struct dma_pmd_meta *m)
+{
+ unsigned int domain_idx;
+ unsigned long flags;
+ dma_addr_t iova;
+ int prot, ret;
+ u64 limit;
+
+ if (!domain)
+ return 0;
+
+ iova = win->base + dma_pmd_meta_win_offset(m);
+ limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ if (iova + PMD_SIZE - 1 > limit)
+ return -ENOSPC;
+
+ domain_idx = win->domain_idx - DMA_PMD_IDX_FIRST;
+ if (test_bit(domain_idx, &m->domains_mapped))
+ return 0;
+
+ prot = dma_info_to_prot(DMA_BIDIRECTIONAL, true, 0);
+ spin_lock_irqsave(&m->map_lock, flags);
+ if (test_bit(domain_idx, &m->domains_mapped)) {
+ ret = 0;
+ } else if (unlikely(!dma_pmd_domain_active(domain_idx, domain))) {
+ ret = -ENODEV;
+ } else {
+ ret = iommu_map(domain, iova, page_to_phys(m->arena_page),
+ PMD_SIZE, prot, GFP_ATOMIC);
+ if (!ret) {
+ /* Ensure the leaf PTE is visible before setting the bit. */
+ smp_mb__before_atomic();
+ set_bit(domain_idx, &m->domains_mapped);
+ }
+ }
+ spin_unlock_irqrestore(&m->map_lock, flags);
+ return ret;
+}
+
+/**
+ * dma_pmd_arena_free - Unreserve blocks in an ARENA_REGION allocation
+ * @dev: Device passed to dma_free_coherent()
+ * @size: Size of the allocation being freed
+ * @cpu_addr: CPU virtual address passed to dma_free_coherent()
+ * @dma: DMA address passed to dma_free_coherent()
+ *
+ * Return: true if @dma belonged to ARENA_REGION and was freed.
+ */
+bool dma_pmd_arena_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma)
+{
+ unsigned int slot, off, nblocks, bpp = DMA_PMD_BLOCKS(0);
+ struct iommu_domain *domain;
+ struct dma_pmd_window *win;
+ dma_addr_t arena_base;
+ unsigned long flags;
+
+ if (!dev || !cpu_addr)
+ return false;
+
+ if (!dma_pmd_meta_base())
+ return false;
+
+ if (dev->iommu_group) {
+ domain = iommu_get_dma_domain(dev);
+ win = domain ? dma_pmd_dma_window(domain) : NULL;
+ /* Pairs with smp_store_release() in dma_pmd_window_assign(). */
+ if (!win || !smp_load_acquire(&win->size))
+ return false;
+ arena_base = win->base;
+ } else if (IS_ENABLED(CONFIG_DMA_PMD_META_KUNIT_TEST)) {
+ /* Reachable only from KUnit tests with a dummy device. */
+ arena_base = 0;
+ } else {
+ return false;
+ }
+
+ if (dma < arena_base || dma - arena_base >= ARENA_REGION_SIZE)
+ return false;
+
+ if (is_vmalloc_addr(cpu_addr)) {
+ kvfree(dma_common_find_pages(cpu_addr));
+ dma_common_free_remap(cpu_addr, size);
+ }
+
+ slot = (dma - arena_base) >> PMD_SHIFT;
+ off = ((dma - arena_base) & (PMD_SIZE - 1)) >> PAGE_SHIFT;
+ nblocks = ALIGN(size, PAGE_SIZE) >> PAGE_SHIFT;
+
+ while (nblocks > 0 && slot < DMA_PMD_ARENA_PAGES) {
+ struct dma_pmd_meta *m = dma_pmd_arena_meta(slot);
+ unsigned int n = min(nblocks, bpp - off);
+ struct page *free_page = NULL;
+
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ if (!WARN_ON_ONCE(!m->arena_page)) {
+ bitmap_clear(m->free_bitmap, off, n);
+ m->nr_free += n;
+ if (m->nr_free == bpp) {
+ free_page = m->arena_page;
+ m->arena_page = NULL;
+ m->nr_free = 0;
+ __clear_bit(slot, dma_pmd_arena.partial_empty);
+ } else {
+ __set_bit(slot, dma_pmd_arena.partial_empty);
+ }
+ }
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+
+ if (free_page) {
+ dma_pmd_unmap_all(m);
+ __free_pages(free_page, PMD_ORDER);
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ __set_bit(slot, dma_pmd_arena.fully_empty);
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+ }
+
+ nblocks -= n;
+ off = 0;
+ slot++;
+ }
+
+ return true;
+}
+EXPORT_SYMBOL(dma_pmd_arena_free);
+
+/**
+ * dma_pmd_arena_alloc - Carve a DMA-mapped buffer out of ARENA_REGION
+ * @dev: Device the buffer is allocated and mapped for
+ * @size: Requested size, rounded up to 4KB
+ * @dma: Out parameter, DMA address of the returned buffer in ARENA_REGION
+ * @node: Preferred NUMA node, or NUMA_NO_NODE for the device's own node
+ *
+ * Locking: Acquires @dma_pmd_arena.lock (irqsave) when searching or updating
+ * slot bitmaps.
+ *
+ * Return: Kernel virtual address of the zeroed buffer, or NULL if the caller
+ * should fall back to its own allocation (e.g. dma_alloc_coherent()).
+ */
+void *dma_pmd_arena_alloc(struct device *dev, size_t size, dma_addr_t *dma, int node)
+{
+ unsigned int npages, nblocks, slot, i, bpp = DMA_PMD_BLOCKS(0);
+ unsigned long *bitmap = dma_pmd_arena.fully_empty;
+ struct iommu_domain *domain;
+ struct dma_pmd_window *win;
+ dma_addr_t arena_base;
+ unsigned long flags;
+ void *va;
+
+ if (!dev || (!dev->iommu_group && !IS_ENABLED(CONFIG_DMA_PMD_META_KUNIT_TEST)))
+ return NULL;
+
+ if (node == NUMA_NO_NODE)
+ node = dev_to_node(dev);
+
+ size = ALIGN(size, PAGE_SIZE);
+ if (!size || size > ARENA_REGION_SIZE)
+ return NULL;
+
+ domain = dev->iommu_group ? iommu_get_dma_domain(dev) : NULL;
+ win = domain ? dma_pmd_dma_window(domain) : NULL;
+ if (dev->iommu_group && (!win || !(domain->pgsize_bitmap & PMD_SIZE)))
+ goto err_fallback;
+
+ if (unlikely(dma_pmd_meta_init()))
+ goto err_fallback;
+
+ /* Pairs with smp_store_release() in dma_pmd_window_assign(). */
+ if (win && !smp_load_acquire(&win->size) &&
+ (READ_ONCE(win->domain_idx) != DMA_PMD_IDX_NONE ||
+ dma_pmd_window_assign(dev, domain, win)))
+ goto err_fallback;
+
+ arena_base = win ? win->base : 0;
+ nblocks = size >> PAGE_SHIFT;
+
+ /*
+ * Allocations <= 1MB first try partially empty pages already mapped in
+ * @domain (arena PMD pages are single-domain and never shared across
+ * IOMMU domains).
+ */
+ if (size <= SZ_1M) {
+ unsigned long domain_mask = win ? BIT(win->domain_idx - DMA_PMD_IDX_FIRST) : 0;
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ for_each_set_bit(slot, dma_pmd_arena.partial_empty, DMA_PMD_ARENA_PAGES) {
+ struct dma_pmd_meta *m = dma_pmd_arena_meta(slot);
+ unsigned long off;
+
+ if (READ_ONCE(m->domains_mapped) != domain_mask ||
+ (domain &&
+ arena_base + ((dma_addr_t)(slot + 1) << PMD_SHIFT) - 1 > limit))
+ continue;
+ if (m->nr_free < nblocks)
+ continue;
+
+ off = bitmap_find_next_zero_area(m->free_bitmap, bpp, 0, nblocks,
+ (1UL << get_order(size)) - 1);
+ if (off >= bpp)
+ continue;
+
+ bitmap_set(m->free_bitmap, off, nblocks);
+ m->nr_free -= nblocks;
+ if (m->nr_free == 0)
+ __clear_bit(slot, dma_pmd_arena.partial_empty);
+ va = page_address(m->arena_page) + (off << PAGE_SHIFT);
+ *dma = arena_base + ((dma_addr_t)slot << PMD_SHIFT) +
+ (off << PAGE_SHIFT);
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+
+ memset(va, 0, size);
+ return va;
+ }
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+ }
+
+ /*
+ * Allocations > 1MB (or <= 1MB when no partial page fits) look for
+ * @npages contiguous fully empty pages.
+ */
+ npages = DIV_ROUND_UP(size, PMD_SIZE);
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ slot = 0;
+ while (slot + npages <= DMA_PMD_ARENA_PAGES) {
+ unsigned int next_zero;
+
+ slot = find_next_bit(bitmap, DMA_PMD_ARENA_PAGES, slot);
+ if (slot + npages > DMA_PMD_ARENA_PAGES)
+ break;
+ next_zero = find_next_zero_bit(bitmap, slot + npages, slot);
+ if (next_zero >= slot + npages) {
+ bitmap_clear(bitmap, slot, npages);
+ break;
+ }
+ slot = next_zero + 1;
+ }
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+
+ if (slot + npages > DMA_PMD_ARENA_PAGES)
+ goto err_fallback;
+
+ for (i = 0; i < npages; i++) {
+ struct dma_pmd_meta *m = dma_pmd_arena_meta(slot + i);
+ gfp_t gfp = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
+ struct page *page;
+
+ page = (node == NUMA_NO_NODE) ? alloc_pages(gfp, PMD_ORDER) :
+ alloc_pages_node(node, gfp, PMD_ORDER);
+ if (!page)
+ goto err_free_pages;
+
+ m->arena_page = page;
+ if (dma_pmd_arena_map_domain(dev, domain, win, m)) {
+ m->arena_page = NULL;
+ __free_pages(page, PMD_ORDER);
+ goto err_free_pages;
+ }
+ }
+
+ va = (npages == 1) ? page_address(dma_pmd_arena_meta(slot)->arena_page) :
+ dma_pmd_arena_vmap(slot, npages);
+ if (!va)
+ goto err_free_pages;
+
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ for (i = 0; i < npages; i++) {
+ struct dma_pmd_meta *m = dma_pmd_arena_meta(slot + i);
+ unsigned int n = min(nblocks - i * bpp, bpp);
+
+ bitmap_zero(m->free_bitmap, bpp);
+ bitmap_set(m->free_bitmap, 0, n);
+ m->nr_free = bpp - n;
+ if (m->nr_free > 0)
+ __set_bit(slot + i, dma_pmd_arena.partial_empty);
+ }
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+
+ *dma = arena_base + ((dma_addr_t)slot << PMD_SHIFT);
+ return va;
+
+err_free_pages:
+ while (i--) {
+ struct dma_pmd_meta *m = dma_pmd_arena_meta(slot + i);
+ struct page *page = m->arena_page;
+
+ m->arena_page = NULL;
+ dma_pmd_unmap_all(m);
+ __free_pages(page, PMD_ORDER);
+ }
+ spin_lock_irqsave(&dma_pmd_arena.lock, flags);
+ bitmap_set(bitmap, slot, npages);
+ spin_unlock_irqrestore(&dma_pmd_arena.lock, flags);
+err_fallback:
+ return NULL;
+}
+EXPORT_SYMBOL(dma_pmd_arena_alloc);
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index 7f4251845ba49..b91dbcb468879 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -166,12 +166,47 @@ static void test_window_helpers(struct kunit *test)
dma_pmd_pool_destroy(pool);
}
+static void test_arena_alloc_and_free(struct kunit *test)
+{
+ struct device dev = { .numa_node = NUMA_NO_NODE };
+ void *v1, *v2, *v3, *vl1;
+ dma_addr_t d1, d2, d3, dl1;
+
+ KUNIT_EXPECT_NULL(test,
+ dma_pmd_arena_alloc(NULL, SZ_4K, &d1, NUMA_NO_NODE));
+
+ v1 = dma_pmd_arena_alloc(&dev, SZ_64K, &d1, NUMA_NO_NODE);
+ KUNIT_ASSERT_NOT_NULL(test, v1);
+ v2 = dma_pmd_arena_alloc(&dev, SZ_64K, &d2, NUMA_NO_NODE);
+ KUNIT_ASSERT_NOT_NULL(test, v2);
+ KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+ KUNIT_EXPECT_EQ(test, d2, d1 + SZ_64K);
+
+ /* Freeing v1 unreserves [0, 64K); next 64K alloc reuses it. */
+ KUNIT_EXPECT_TRUE(test, dma_pmd_arena_free(&dev, SZ_64K, v1, d1));
+ v3 = dma_pmd_arena_alloc(&dev, SZ_64K, &d3, NUMA_NO_NODE);
+ KUNIT_EXPECT_PTR_EQ(test, v3, v1);
+ KUNIT_EXPECT_EQ(test, d3, d1);
+
+ /*
+ * Allocate 3MB (spans 2 contiguous fully_empty slots in ARENA_REGION);
+ * the trailing 1MB in the second slot is marked partial_empty and can
+ * satisfy a <= 1MB allocation before both are freed.
+ */
+ vl1 = dma_pmd_arena_alloc(&dev, SZ_2M + SZ_1M, &dl1, NUMA_NO_NODE);
+ KUNIT_ASSERT_NOT_NULL(test, vl1);
+ KUNIT_EXPECT_TRUE(test, dma_pmd_arena_free(&dev, SZ_2M + SZ_1M, vl1, dl1));
+ KUNIT_EXPECT_TRUE(test, dma_pmd_arena_free(&dev, SZ_64K, v2, d2));
+ KUNIT_EXPECT_TRUE(test, dma_pmd_arena_free(&dev, SZ_64K, v3, d3));
+}
+
static struct kunit_case dma_pmd_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_pooled_toggle),
KUNIT_CASE(test_pool_alloc_and_recycle),
KUNIT_CASE(test_window_helpers),
+ KUNIT_CASE(test_arena_alloc_and_free),
{}
};
diff --git a/drivers/iommu/dma-pmd-map.c b/drivers/iommu/dma-pmd-map.c
index 565e10dd1a2e8..1147874ac8068 100644
--- a/drivers/iommu/dma-pmd-map.c
+++ b/drivers/iommu/dma-pmd-map.c
@@ -51,6 +51,11 @@ static DEFINE_SPINLOCK(dma_pmd_domains_lock);
*/
DEFINE_SRCU(dma_pmd_srcu);
+bool dma_pmd_domain_active(unsigned int idx, const struct iommu_domain *domain)
+{
+ return READ_ONCE(dma_pmd_domains[idx].domain) == domain;
+}
+
/**
* dma_pmd_unmap_all - Remove a PMD page's PTE from every domain holding it
* @meta: PMD page metadata structure
@@ -77,7 +82,7 @@ DEFINE_SRCU(dma_pmd_srcu);
*/
void dma_pmd_unmap_all(struct dma_pmd_meta *meta)
{
- phys_addr_t phys = dma_pmd_meta_to_phys(meta);
+ dma_addr_t offset = dma_pmd_meta_win_offset(meta);
unsigned long mapped, flags;
int idx, srcu_idx;
@@ -104,7 +109,7 @@ void dma_pmd_unmap_all(struct dma_pmd_meta *meta)
if (!domain)
continue;
- iommu_unmap(domain, base + phys, PMD_SIZE);
+ iommu_unmap(domain, base + offset, PMD_SIZE);
}
srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
}
@@ -170,6 +175,7 @@ void dma_pmd_domain_release(struct iommu_domain *domain)
{
unsigned int dropped = 0;
unsigned long flags;
+ unsigned int i;
int idx;
/* Nothing has ever been pooled, so nothing can reference @domain. */
@@ -199,6 +205,9 @@ void dma_pmd_domain_release(struct iommu_domain *domain)
dropped += dma_pmd_pools_forget_domain(idx);
+ for (i = 0; i < DMA_PMD_ARENA_PAGES; i++)
+ dropped += dma_pmd_forget_domain(dma_pmd_arena_meta(i), idx);
+
/*
* Any PMD page removed from a pool list before the walk above is either
* already on @dma_pmd_free_list (placed there under @pool->lock in
@@ -225,7 +234,7 @@ void dma_pmd_domain_release(struct iommu_domain *domain)
* dma_pmd_dma_window_alloc - Allocate an IOVA window for DMA_PMD mappings
* @dev: Device whose addressing limits the window must respect
* @domain: Domain to take the window from
- * @size: Window size in bytes
+ * @sizep: In/out window size in bytes
*
* Allocates a contiguous IOVA range out of @domain's iova_domain on first use,
* aligned to PMD_SIZE matching the mapping. Callers on the map path still
@@ -237,9 +246,9 @@ void dma_pmd_domain_release(struct iommu_domain *domain)
static dma_addr_t dma_pmd_dma_window_alloc(struct device *dev,
struct iommu_domain *domain, u64 *sizep)
{
+ u64 min_size = ARENA_REGION_SIZE + ALIGN(PFN_PHYS(max_pfn), PMD_SIZE);
u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
struct iova_domain *iovad = dma_pmd_dma_iovad(domain);
- u64 min_size = ALIGN(PFN_PHYS(max_pfn), PMD_SIZE);
unsigned long shift, iova_len;
struct iova *new_iova;
@@ -295,7 +304,7 @@ static dma_addr_t dma_pmd_dma_window_alloc(struct device *dev,
int dma_pmd_window_assign(struct device *dev, struct iommu_domain *domain,
struct dma_pmd_window *win)
{
- u64 size = ALIGN(PFN_PHYS(dma_pmd_top_pfn()), PMD_SIZE);
+ u64 size = ALIGN(PFN_PHYS(dma_pmd_top_pfn()), PMD_SIZE) + ARENA_REGION_SIZE;
unsigned long flags;
int idx, ret = 0;
dma_addr_t base;
diff --git a/drivers/iommu/dma-pmd-meta.c b/drivers/iommu/dma-pmd-meta.c
index 27b8c2568c749..8f2bc7d77e5bd 100644
--- a/drivers/iommu/dma-pmd-meta.c
+++ b/drivers/iommu/dma-pmd-meta.c
@@ -186,8 +186,9 @@ static inline bool dma_pmd_pfn_online(unsigned long pfn)
*
* Return: 0, or -ENOMEM with the range partially backed.
*/
-static int dma_pmd_meta_populate(struct dma_pmd_meta *array, unsigned long nframes,
- unsigned long start_pfn, unsigned long end_pfn, bool force)
+static int dma_pmd_meta_populate(struct dma_pmd_meta *array, unsigned long *bitmap,
+ unsigned long nframes, unsigned long start_pfn,
+ unsigned long end_pfn, bool force)
{
unsigned long chunk, last, pfn;
@@ -208,7 +209,8 @@ static int dma_pmd_meta_populate(struct dma_pmd_meta *array, unsigned long nfram
addr = (unsigned long)array + (chunk << PAGE_SHIFT);
if (vmalloc_to_page((void *)addr)) {
- set_bit(chunk, dma_pmd_chunk_bitmap);
+ if (bitmap)
+ set_bit(chunk, bitmap);
continue; /* already backed */
}
@@ -247,14 +249,17 @@ static int dma_pmd_meta_populate(struct dma_pmd_meta *array, unsigned long nfram
* never leak the page it declined to consume.
*/
__free_page(page);
- set_bit(chunk, dma_pmd_chunk_bitmap);
+ if (bitmap)
+ set_bit(chunk, bitmap);
continue;
}
flush_cache_vmap(addr, addr + PAGE_SIZE);
- /* Pair with test_bit_acquire() in readers. */
- smp_mb__before_atomic();
- set_bit(chunk, dma_pmd_chunk_bitmap);
+ if (bitmap) {
+ /* Pair with test_bit_acquire() in readers. */
+ smp_mb__before_atomic();
+ set_bit(chunk, bitmap);
+ }
dma_pmd_meta_pages++;
}
@@ -273,8 +278,9 @@ bool dma_pmd_meta_ensure_pfn(unsigned long pfn, bool can_block)
return false;
mutex_lock(&dma_pmd_meta_mutex);
- ret = dma_pmd_meta_populate(dma_pmd_meta_base(), dma_pmd_meta_nframes,
- pfn, pfn + (1UL << PMD_ORDER), true);
+ ret = dma_pmd_meta_populate(dma_pmd_meta_base(), dma_pmd_chunk_bitmap,
+ dma_pmd_meta_nframes, pfn,
+ pfn + (1UL << PMD_ORDER), true);
mutex_unlock(&dma_pmd_meta_mutex);
return !ret;
@@ -284,18 +290,20 @@ EXPORT_SYMBOL(dma_pmd_meta_ensure_pfn);
/**
* dma_pmd_meta_init - Reserve and populate the sparse per-PMD metadata array
*
- * Populates all present RAM pages before publishing @dma_pmd_meta_array so
- * no concurrent reader ever sees an unmapped entry.
+ * Populates all present RAM pages and ARENA_REGION entries before publishing
+ * @dma_pmd_meta_array so no concurrent reader ever sees an unmapped entry.
*
* Return: 0 on success, or negative errno on failure.
*/
int dma_pmd_meta_init(void)
{
- unsigned long nframes, nchunks, size;
- struct dma_pmd_meta *array;
+ unsigned long nframes, total_frames, nchunks, size, i;
+ struct dma_pmd_meta *raw, *array;
struct vm_struct *vm;
int ret = 0;
+ static_assert(IS_ALIGNED(DMA_PMD_ARENA_PAGES << DMA_PMD_META_SHIFT, PAGE_SIZE));
+
if (likely(dma_pmd_meta_base()))
return 0;
@@ -308,8 +316,9 @@ int dma_pmd_meta_init(void)
(unsigned long)min_t(u64, iomem_resource.end >> PAGE_SHIFT,
1ULL << (MAX_PHYSMEM_BITS - PAGE_SHIFT))),
1UL << PMD_ORDER);
- size = PAGE_ALIGN(nframes << DMA_PMD_META_SHIFT);
- nchunks = size >> PAGE_SHIFT;
+ total_frames = DMA_PMD_ARENA_PAGES + nframes;
+ size = PAGE_ALIGN(total_frames << DMA_PMD_META_SHIFT);
+ nchunks = PAGE_ALIGN(nframes << DMA_PMD_META_SHIFT) >> PAGE_SHIFT;
dma_pmd_chunk_bitmap = bitmap_zalloc(nchunks, GFP_KERNEL);
if (!dma_pmd_chunk_bitmap) {
@@ -324,14 +333,25 @@ int dma_pmd_meta_init(void)
ret = -ENOMEM;
goto out_unlock;
}
- array = vm->addr;
-
- ret = dma_pmd_meta_populate(array, nframes, 0, max_pfn, false);
+ raw = vm->addr;
+ array = raw + DMA_PMD_ARENA_PAGES;
+
+ ret = dma_pmd_meta_populate(raw, NULL, DMA_PMD_ARENA_PAGES,
+ 0, DMA_PMD_ARENA_PAGES << PMD_ORDER, true);
+ if (!ret)
+ ret = dma_pmd_meta_populate(array, dma_pmd_chunk_bitmap,
+ nframes, 0, max_pfn, false);
if (ret) {
+ unsigned long off, chunk;
struct page *p, *next;
- unsigned long chunk;
LIST_HEAD(pages);
+ for (off = 0; off < (DMA_PMD_ARENA_PAGES << DMA_PMD_META_SHIFT);
+ off += PAGE_SIZE) {
+ p = vmalloc_to_page((void *)raw + off);
+ if (p)
+ list_add(&p->lru, &pages);
+ }
for_each_set_bit(chunk, dma_pmd_chunk_bitmap, nchunks) {
p = vmalloc_to_page((void *)array + (chunk << PAGE_SHIFT));
if (p)
@@ -348,6 +368,9 @@ int dma_pmd_meta_init(void)
goto out_unlock;
}
+ for (i = 0; i < DMA_PMD_ARENA_PAGES; i++)
+ spin_lock_init(&raw[i].map_lock);
+
/*
* Publish @nframes and the populated pages before the array pointer:
* readers pair with smp_load_acquire(&dma_pmd_meta_array).
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 355f865f44057..b411ecb1fdafc 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -16,8 +16,9 @@ struct iommu_domain;
/**
* struct dma_pmd_window - A domain's IOVA window reserved for DMA_PMD pages
- * @base: First IOVA of the window. A page at @phys is mapped, in every domain
- * that has a window, at @base + @phys.
+ * @base: First IOVA of the window. The leading ARENA_REGION_SIZE bytes
+ * [base, base + ARENA_REGION_SIZE) form ARENA_REGION; a pooled RAM
+ * page at @phys is mapped at @base + ARENA_REGION_SIZE + @phys.
* @size: Window size in bytes, or 0 if this domain has no window, either
* because nothing has pooled through it yet, or because it could not
* find a free range that large. 0 must make both the map and the unmap
@@ -40,6 +41,8 @@ struct dma_pmd_window {
#ifdef CONFIG_DMA_PMD
#define DMA_PMD_BLOCKS(order) (1U << (PMD_ORDER - (order)))
+#define DMA_PMD_ARENA_PAGES 8192U
+#define ARENA_REGION_SIZE ((u64)DMA_PMD_ARENA_PAGES * PMD_SIZE)
/*
* Number of IOMMU domains that may use DMA_PMD at once.
@@ -116,8 +119,9 @@ enum {
* @domains_mapped: One bit per domain index: set once this PMD page's leaf
* PTE is installed in that domain.
*
- * Alloc/free fast path, all under pool->lock:
+ * Alloc/free fast path, all under pool->lock (or dma_pmd_arena.lock):
* @pool: Owning dma_pmd_pool (holds a kref on the pool while pooled)
+ * @arena_page: Allocated 2MB struct page for an ARENA_REGION entry
* @list: Node in pool->partial, pool->idle or pool->full
*
* Slow path only (disjoint lifetimes):
@@ -125,11 +129,13 @@ enum {
* @rcu: RCU head used to defer buddy release past lockless readers
*
* @free_bitmap: Bitmap of zeroed available block indices (up to 1 << PMD_ORDER)
- * @dirty_bitmap: Bitmap of dirty available block indices
+ * for pools, or allocated 4KB blocks for ARENA_REGION entries
+ * @dirty_bitmap: Bitmap of dirty available block indices for pools
*
* Lives in the sparse per-PMD-frame array @dma_pmd_meta_array indexed by
- * (pfn >> PMD_ORDER). Only chunks covering valid RAM are backed by physical
- * pages (128 KB per GB of RAM).
+ * (pfn >> PMD_ORDER), preceded by DMA_PMD_ARENA_PAGES entries for
+ * ARENA_REGION. Only chunks covering valid RAM and ARENA_REGION are backed by
+ * physical pages (128 KB per GB of RAM).
*/
struct dma_pmd_meta {
bool pooled;
@@ -139,7 +145,10 @@ struct dma_pmd_meta {
unsigned long domains_mapped;
- struct dma_pmd_pool *pool;
+ union {
+ struct dma_pmd_pool *pool;
+ struct page *arena_page;
+ };
struct list_head list;
union {
@@ -224,18 +233,29 @@ static inline struct dma_pmd_meta *dma_pmd_meta_base(void)
/*
* Highest PFN covered by the allocated @dma_pmd_meta_array reservation.
- * Every domain's IOVA window is sized to match this bound.
+ * Every domain's IOVA window is sized up to this bound.
*/
static inline unsigned long dma_pmd_top_pfn(void)
{
return dma_pmd_meta_nframes << PMD_ORDER;
}
+static inline struct dma_pmd_meta *dma_pmd_arena_meta(unsigned int slot)
+{
+ return (dma_pmd_meta_base() - DMA_PMD_ARENA_PAGES) + slot;
+}
+
/* IOVA of @phys in @win. Only valid once the PMD page's PTE is installed. */
static inline dma_addr_t dma_pmd_window_iova(const struct dma_pmd_window *win,
phys_addr_t phys)
{
- return win->base + phys;
+ return win->base + ARENA_REGION_SIZE + phys;
+}
+
+/* Offset of @m (either arena or RAM) from the start of a domain's window. */
+static inline dma_addr_t dma_pmd_meta_win_offset(const struct dma_pmd_meta *m)
+{
+ return (dma_addr_t)(m - dma_pmd_arena_meta(0)) << PMD_SHIFT;
}
int dma_pmd_meta_init(void);
@@ -250,6 +270,7 @@ void dma_pmd_unmap_all(struct dma_pmd_meta *meta);
unsigned int dma_pmd_forget_domain(struct dma_pmd_meta *meta, int idx);
int dma_pmd_window_assign(struct device *dev, struct iommu_domain *domain,
struct dma_pmd_window *win);
+bool dma_pmd_domain_active(unsigned int idx, const struct iommu_domain *domain);
static inline bool dma_is_pmd_phys(phys_addr_t phys)
{
@@ -281,6 +302,8 @@ dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
void dma_pmd_domain_release(struct iommu_domain *domain);
+void *dma_pmd_arena_alloc(struct device *dev, size_t size, dma_addr_t *dma, int node);
+
#else /* !CONFIG_DMA_PMD */
static inline bool dma_is_pmd_phys(phys_addr_t phys)
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
index 085d9a730c278..55c770fe232b0 100644
--- a/include/linux/dma-pmd.h
+++ b/include/linux/dma-pmd.h
@@ -95,6 +95,9 @@ static inline struct page *dma_pmd_pool_alloc(struct dma_pmd_pool *pool, gfp_t g
}
bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool);
+/* Hooks for kernel/dma/mapping.c */
+bool dma_pmd_arena_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma);
+
#else /* !CONFIG_DMA_PMD */
static inline bool dma_is_pmd_page(unsigned long pfn)
@@ -141,5 +144,11 @@ static inline bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool)
return false;
}
+static inline bool dma_pmd_arena_free(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma)
+{
+ return false;
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _LINUX_DMA_PMD_H */
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 10/22] driver core: Add per-device dma_pmd_* sysfs attributes
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (8 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 09/22] iommu/dma: Add DMA_PMD arena allocator Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 11/22] dma-mapping: Use DMA_PMD arena for dma_alloc_attrs() Luigi Rizzo
` (40 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add four boolean attributes to struct device exposed under
/sys/devices/*/*/dma_pmd_*:
- dma_pmd_rxbuf: enable DMA_PMD pools for RX buffers / page_pool
- dma_pmd_tx_hdrs: enable DMA_PMD pools for TX header bounce buffers
- dma_pmd_rings: enable DMA_PMD arena for coherent rings
- dma_pmd_debug: log coherent DMA alloc timing and arena usage
These can be used by device drivers to selectively enable DMA_PMD for
their allocations.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 51 ++++++++++++++++++++++++++++++++++++++++++
include/linux/device.h | 5 +++++
2 files changed, 56 insertions(+)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 4c0c373998a19..f8c3c42d84b8d 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2905,6 +2905,43 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
}
static const DEVICE_ATTR_RO(removable);
+/* Per-device configuration to selectively enable DMA_PMD */
+#define DMA_PMD_BOOL_ATTR(name) \
+static ssize_t name##_show(struct device *dev, \
+ struct device_attribute *attr, char *buf) \
+{ \
+ return sysfs_emit(buf, "%u\n", READ_ONCE(dev->name)); \
+} \
+static ssize_t name##_store(struct device *dev, \
+ struct device_attribute *attr, \
+ const char *buf, size_t count) \
+{ \
+ bool val; \
+ int ret = kstrtobool(buf, &val); \
+ if (ret < 0) \
+ return ret; \
+ WRITE_ONCE(dev->name, val); \
+ return count; \
+} \
+static DEVICE_ATTR_RW(name)
+
+DMA_PMD_BOOL_ATTR(dma_pmd_rxbuf);
+DMA_PMD_BOOL_ATTR(dma_pmd_tx_hdrs);
+DMA_PMD_BOOL_ATTR(dma_pmd_rings);
+DMA_PMD_BOOL_ATTR(dma_pmd_debug);
+
+static struct attribute *dev_attr_dma_pmd[] = {
+ &dev_attr_dma_pmd_rxbuf.attr,
+ &dev_attr_dma_pmd_tx_hdrs.attr,
+ &dev_attr_dma_pmd_rings.attr,
+ &dev_attr_dma_pmd_debug.attr,
+ NULL,
+};
+
+static const struct attribute_group dev_attr_dma_pmd_group = {
+ .attrs = dev_attr_dma_pmd,
+};
+
int device_add_groups(struct device *dev,
const struct attribute_group *const *groups)
{
@@ -3012,8 +3049,20 @@ static int device_add_attrs(struct device *dev)
goto err_remove_dev_removable;
}
+ if (dev->dma_mask) {
+ error = device_add_group(dev, &dev_attr_dma_pmd_group);
+ if (error)
+ goto err_remove_dev_physical_location;
+ }
+
return 0;
+ err_remove_dev_physical_location:
+ if (dev->physical_location) {
+ device_remove_group(dev, &dev_attr_physical_location_group);
+ kfree(dev->physical_location);
+ dev->physical_location = NULL;
+ }
err_remove_dev_removable:
device_remove_file(dev, &dev_attr_removable);
err_remove_dev_waiting_for_supplier:
@@ -3037,6 +3086,8 @@ static void device_remove_attrs(struct device *dev)
const struct class *class = dev->class;
const struct device_type *type = dev->type;
+ device_remove_group(dev, &dev_attr_dma_pmd_group);
+
if (dev->physical_location) {
device_remove_group(dev, &dev_attr_physical_location_group);
kfree(dev->physical_location);
diff --git a/include/linux/device.h b/include/linux/device.h
index aee79fd6b32b4..52aa4a7e1cc45 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -794,6 +794,11 @@ struct device {
enum device_removable removable;
+ bool dma_pmd_rxbuf;
+ bool dma_pmd_tx_hdrs;
+ bool dma_pmd_rings;
+ bool dma_pmd_debug;
+
DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
};
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 11/22] dma-mapping: Use DMA_PMD arena for dma_alloc_attrs()
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (9 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 10/22] driver core: Add per-device dma_pmd_* sysfs attributes Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill() Luigi Rizzo
` (39 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Wire dma_alloc_attrs() and dma_free_attrs() into the DMA_PMD arena
when dev->dma_pmd_rings is enabled on a device:
- Add dma_pmd_dma_alloc() and dma_is_pmd_dma() backed by
dma_pmd_arena.
- Route GFP_KERNEL dma_alloc_attrs() requests through dma_pmd_dma_alloc()
when dev->dma_pmd_rings is set, logging latency when
dev->dma_pmd_debug is enabled.
- Reclaim sub-allocations back to the arena in dma_free_attrs() via
dma_pmd_arena_free() while keeping the PMD_SIZE IOMMU mapping active.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-arena.c | 27 ++++++++++++++++++++++++
drivers/iommu/dma-pmd-kunit.c | 11 +++++++---
drivers/iommu/dma-pmd-map.c | 17 +++++++++++++++
include/linux/dma-pmd.h | 29 ++++++++++++++++++++++++++
kernel/dma/mapping.c | 39 +++++++++++++++++++++++++++++------
5 files changed, 114 insertions(+), 9 deletions(-)
diff --git a/drivers/iommu/dma-pmd-arena.c b/drivers/iommu/dma-pmd-arena.c
index 578df5be86a86..e1a7d9d6f0f60 100644
--- a/drivers/iommu/dma-pmd-arena.c
+++ b/drivers/iommu/dma-pmd-arena.c
@@ -383,3 +383,30 @@ void *dma_pmd_arena_alloc(struct device *dev, size_t size, dma_addr_t *dma, int
return NULL;
}
EXPORT_SYMBOL(dma_pmd_arena_alloc);
+
+void *dma_pmd_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma,
+ gfp_t gfp, unsigned long attrs)
+{
+ void *va;
+
+ if (!dev || !dev->iommu_group || !dev_is_dma_coherent(dev))
+ return NULL;
+ /*
+ * The arena allocates PMD pages, IOMMU page tables, and vmap() areas
+ * with GFP_KERNEL, and shares PMD pages across allocations on the same
+ * device. Decline non-GFP_KERNEL requests (including GFP_NOFS/GFP_NOIO
+ * or scoped memalloc_nofs/noio contexts) and per-allocation modifiers
+ * such as __GFP_ACCOUNT or __GFP_NORETRY so the caller falls back to
+ * the standard DMA allocator.
+ */
+ if ((current_gfp_context(gfp) & ~(__GFP_ZERO | __GFP_NOWARN)) != GFP_KERNEL ||
+ (attrs & ~DMA_ATTR_NO_WARN))
+ return NULL;
+
+ va = dma_pmd_arena_alloc(dev, size, dma, dev_to_node(dev));
+ if (!va && !(gfp & __GFP_NOWARN) && !(attrs & DMA_ATTR_NO_WARN))
+ dev_warn_ratelimited(dev, "DMA_PMD arena alloc failed (size %zu)\n",
+ size);
+ return va;
+}
+EXPORT_SYMBOL(dma_pmd_dma_alloc);
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index b91dbcb468879..c2363277e2ecd 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -169,11 +169,16 @@ static void test_window_helpers(struct kunit *test)
static void test_arena_alloc_and_free(struct kunit *test)
{
struct device dev = { .numa_node = NUMA_NO_NODE };
- void *v1, *v2, *v3, *vl1;
dma_addr_t d1, d2, d3, dl1;
+ void *v1, *v2, *v3, *vl1;
- KUNIT_EXPECT_NULL(test,
- dma_pmd_arena_alloc(NULL, SZ_4K, &d1, NUMA_NO_NODE));
+ KUNIT_EXPECT_NULL(test, dma_pmd_arena_alloc(NULL, SZ_4K, &d1, NUMA_NO_NODE));
+ KUNIT_EXPECT_NULL(test, dma_pmd_dma_alloc(NULL, SZ_4K, &d1, GFP_KERNEL, 0));
+ KUNIT_EXPECT_NULL(test, dma_pmd_dma_alloc(&dev, SZ_4K, &d1, GFP_KERNEL, 0));
+ KUNIT_EXPECT_NULL(test, dma_pmd_dma_alloc(&dev, SZ_4K, &d1,
+ GFP_KERNEL | __GFP_ACCOUNT, 0));
+ KUNIT_EXPECT_FALSE(test, dma_pmd_free(&dev, SZ_4K, NULL, 0));
+ KUNIT_EXPECT_FALSE(test, dma_is_pmd_dma(&dev, 0));
v1 = dma_pmd_arena_alloc(&dev, SZ_64K, &d1, NUMA_NO_NODE);
KUNIT_ASSERT_NOT_NULL(test, v1);
diff --git a/drivers/iommu/dma-pmd-map.c b/drivers/iommu/dma-pmd-map.c
index 1147874ac8068..76cb5d01cb1e8 100644
--- a/drivers/iommu/dma-pmd-map.c
+++ b/drivers/iommu/dma-pmd-map.c
@@ -380,6 +380,23 @@ int dma_pmd_window_assign(struct device *dev, struct iommu_domain *domain,
return ret;
}
+/**
+ * dma_is_pmd_dma - Test whether @dma lies in @dev's DMA_PMD IOVA window
+ * @dev: Device performing DMA
+ * @dma: DMA address to test
+ */
+bool dma_is_pmd_dma(struct device *dev, dma_addr_t dma)
+{
+ struct dma_pmd_window *win;
+
+ if (!dev || !dev->iommu_group)
+ return false;
+
+ win = dma_pmd_dma_window(iommu_get_dma_domain(dev));
+ return win && dma_pmd_window_owns(win, dma);
+}
+EXPORT_SYMBOL(dma_is_pmd_dma);
+
/**
* dma_pmd_dma_map_phys - Derive the IOVA of a pool address, mapping if needed
* @dev: Device performing DMA
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
index 55c770fe232b0..7b7e9477d44e1 100644
--- a/include/linux/dma-pmd.h
+++ b/include/linux/dma-pmd.h
@@ -96,7 +96,19 @@ static inline struct page *dma_pmd_pool_alloc(struct dma_pmd_pool *pool, gfp_t g
bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool);
/* Hooks for kernel/dma/mapping.c */
+void *dma_pmd_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma,
+ gfp_t gfp, unsigned long attrs);
bool dma_pmd_arena_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma);
+bool dma_is_pmd_dma(struct device *dev, dma_addr_t dma);
+
+static inline bool dma_pmd_free(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma_handle)
+{
+ /* Pairs with smp_store_release() in dma_pmd_meta_init(). */
+ if (likely(!smp_load_acquire(&dma_pmd_meta_array)))
+ return false;
+ return dma_pmd_arena_free(dev, size, cpu_addr, dma_handle);
+}
#else /* !CONFIG_DMA_PMD */
@@ -144,11 +156,28 @@ static inline bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool)
return false;
}
+static inline void *dma_pmd_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma,
+ gfp_t gfp, unsigned long attrs)
+{
+ return NULL;
+}
+
static inline bool dma_pmd_arena_free(struct device *dev, size_t size,
void *cpu_addr, dma_addr_t dma)
{
return false;
}
+static inline bool dma_is_pmd_dma(struct device *dev, dma_addr_t dma)
+{
+ return false;
+}
+
+static inline bool dma_pmd_free(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma_handle)
+{
+ return false;
+}
+
#endif /* CONFIG_DMA_PMD */
#endif /* _LINUX_DMA_PMD_H */
diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
index bf2651a70b7c2..cc18896017e35 100644
--- a/kernel/dma/mapping.c
+++ b/kernel/dma/mapping.c
@@ -10,6 +10,8 @@
#include <linux/dma-map-ops.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/ktime.h>
+#include <linux/dma-pmd.h>
#include <linux/iommu-dma.h>
#include <linux/kmsan.h>
#include <linux/of_device.h>
@@ -180,7 +182,8 @@ dma_addr_t dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
if (!is_mmio)
kmsan_handle_dma(phys, size, dir);
trace_dma_map_phys(dev, phys, addr, size, dir, attrs);
- debug_dma_map_phys(dev, phys, size, dir, addr, attrs);
+ if (IS_ENABLED(CONFIG_DMA_API_DEBUG) && !dma_is_pmd_dma(dev, addr))
+ debug_dma_map_phys(dev, phys, size, dir, addr, attrs);
return addr;
}
@@ -223,7 +226,8 @@ void dma_unmap_phys(struct device *dev, dma_addr_t addr, size_t size,
else if (ops->unmap_phys)
ops->unmap_phys(dev, addr, size, dir, attrs);
trace_dma_unmap_phys(dev, addr, size, dir, attrs);
- debug_dma_unmap_phys(dev, addr, size, dir, attrs);
+ if (IS_ENABLED(CONFIG_DMA_API_DEBUG) && !dma_is_pmd_dma(dev, addr))
+ debug_dma_unmap_phys(dev, addr, size, dir, attrs);
}
EXPORT_SYMBOL_GPL(dma_unmap_phys);
@@ -633,7 +637,11 @@ EXPORT_SYMBOL_GPL(dma_get_required_mask);
void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
gfp_t flag, unsigned long attrs)
{
+ bool debug = unlikely(READ_ONCE(dev->dma_pmd_debug)) && gfpflags_allow_blocking(flag);
const struct dma_map_ops *ops = get_dma_ops(dev);
+ bool can_block = gfpflags_allow_blocking(flag);
+ bool from_arena = false;
+ u64 start_ns = 0;
void *cpu_addr;
WARN_ON_ONCE(!dev->coherent_dma_mask);
@@ -655,10 +663,14 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
if (force_dma_unencrypted(dev))
attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
+ if (debug) {
+ start_ns = ktime_get_ns();
+ *dma_handle = 0;
+ }
if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) {
trace_dma_alloc(dev, cpu_addr, *dma_handle, size,
DMA_BIDIRECTIONAL, flag, attrs);
- return cpu_addr;
+ goto out_debug;
}
/* let the implementation decide on the zone to allocate from: */
@@ -667,18 +679,31 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
if (dma_alloc_direct(dev, ops) || arch_dma_alloc_direct(dev)) {
cpu_addr = dma_direct_alloc(dev, size, dma_handle, flag, attrs);
} else if (use_dma_iommu(dev)) {
- cpu_addr = iommu_dma_alloc(dev, size, dma_handle, flag, attrs);
+ if (READ_ONCE(dev->dma_pmd_rings) && can_block) {
+ cpu_addr = dma_pmd_dma_alloc(dev, size, dma_handle, flag, attrs);
+ from_arena = !!cpu_addr;
+ }
+ if (!from_arena)
+ cpu_addr = iommu_dma_alloc(dev, size, dma_handle, flag, attrs);
} else if (ops->alloc) {
cpu_addr = ops->alloc(dev, size, dma_handle, flag, attrs);
} else {
+ cpu_addr = NULL;
trace_dma_alloc(dev, NULL, 0, size, DMA_BIDIRECTIONAL, flag,
attrs);
- return NULL;
+ goto out_debug;
}
trace_dma_alloc(dev, cpu_addr, *dma_handle, size, DMA_BIDIRECTIONAL,
flag, attrs);
- debug_dma_alloc_coherent(dev, size, *dma_handle, cpu_addr, attrs);
+ if (!from_arena)
+ debug_dma_alloc_coherent(dev, size, *dma_handle, cpu_addr, attrs);
+out_debug:
+ if (debug)
+ dev_info(dev,
+ "dma_alloc_coherent(size=%zu, gfp=%pGg, attrs=%#lx) -> va=%p, dma=%pad, arena=%d in %llu ns\n",
+ size, &flag, attrs, cpu_addr, dma_handle, from_arena,
+ ktime_get_ns() - start_ns);
return cpu_addr;
}
EXPORT_SYMBOL(dma_alloc_attrs);
@@ -703,6 +728,8 @@ void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr,
attrs);
if (!cpu_addr)
return;
+ if (use_dma_iommu(dev) && dma_pmd_free(dev, size, cpu_addr, dma_handle))
+ return;
debug_dma_free_coherent(dev, size, cpu_addr, dma_handle, attrs);
if (dma_alloc_direct(dev, ops) || arch_dma_free_direct(dev, dma_handle))
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill()
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (10 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 11/22] dma-mapping: Use DMA_PMD arena for dma_alloc_attrs() Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 13/22] net/core: Use DMA_PMD for page_pool memory Luigi Rizzo
` (38 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add sysctl net.core.tx_enable_dma_pmd to back skb_page_frag_refill() with
per-CPU order-3 (32KB block) and order-0 (4KB block) DMA_PMD page pools.
When enabled, TX socket buffer page fragment allocations draw fragments
from DMA_PMD physical pages registered with per-CPU dma_pmd_pool. The
first allocation maps the entire DMA_PMD page, subsequent allocations
reuse the cached PMD_SIZE IOVA mapping locklessly without per-packet unmap
or IOTLB flush.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
include/net/sock.h | 3 ++
net/core/sock.c | 86 +++++++++++++++++++++++++++++++++++++-
net/core/sysctl_net_core.c | 7 ++++
3 files changed, 94 insertions(+), 2 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index 60ea55dc18854..a06b53e9ff225 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -3092,6 +3092,9 @@ extern __u32 sysctl_rmem_default;
#define SKB_FRAG_PAGE_ORDER get_order(32768)
DECLARE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key);
+DECLARE_STATIC_KEY_FALSE(net_tx_enable_dma_pmd_key);
+int net_tx_dma_pmd_sysctl(const struct ctl_table *table, int write,
+ void *buffer, size_t *lenp, loff_t *ppos);
static inline int sk_get_wmem0(const struct sock *sk, const struct proto *proto)
{
diff --git a/net/core/sock.c b/net/core/sock.c
index e8551df8330ff..1fd338a547ce1 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -115,6 +115,7 @@
#include <linux/memcontrol.h>
#include <linux/prefetch.h>
#include <linux/compat.h>
+#include <linux/dma-pmd.h>
#include <linux/mroute.h>
#include <linux/mroute6.h>
#include <linux/icmpv6.h>
@@ -3173,6 +3174,87 @@ static void sk_leave_memory_pressure(struct sock *sk)
}
DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key);
+DEFINE_STATIC_KEY_FALSE(net_tx_enable_dma_pmd_key);
+
+static DEFINE_PER_CPU(struct dma_pmd_pool *, tx_pmd_pool_high);
+static DEFINE_PER_CPU(struct dma_pmd_pool *, tx_pmd_pool_order0);
+
+/*
+ * Lazily initialize the per-CPU DMA_PMD page pools on the first write to sysctl
+ * net.core.tx_enable_dma_pmd.
+ *
+ * Once initialized, the struct dma_pmd_pool descriptors remain allocated for
+ * the lifetime of the kernel so that lockless raw_cpu_read() in
+ * __alloc_pmd() is always safe against concurrent sysctl
+ * toggles. If pool creation fails partway through, all pools created so far are
+ * destroyed and per-CPU pointers are reset to NULL.
+ */
+static int net_tx_dma_pmd_init(void)
+{
+ static DEFINE_MUTEX(mutex);
+ struct dma_pmd_pool *pool;
+ static bool initialized;
+ int cpu;
+
+ if (!IS_ENABLED(CONFIG_DMA_PMD))
+ return -EOPNOTSUPP;
+
+ mutex_lock(&mutex);
+ if (initialized) {
+ mutex_unlock(&mutex);
+ return 0;
+ }
+
+ for_each_possible_cpu(cpu) {
+ if (SKB_FRAG_PAGE_ORDER) {
+ pool = dma_pmd_pool_create(SKB_FRAG_PAGE_ORDER, 16);
+ if (!pool)
+ goto err_cleanup;
+ per_cpu(tx_pmd_pool_high, cpu) = pool;
+ }
+
+ pool = dma_pmd_pool_create(0, 16);
+ if (!pool)
+ goto err_cleanup;
+ per_cpu(tx_pmd_pool_order0, cpu) = pool;
+ }
+
+ initialized = true;
+ mutex_unlock(&mutex);
+ return 0;
+
+err_cleanup:
+ for_each_possible_cpu(cpu) {
+ per_cpu(tx_pmd_pool_high, cpu) =
+ dma_pmd_pool_destroy(per_cpu(tx_pmd_pool_high, cpu));
+ per_cpu(tx_pmd_pool_order0, cpu) =
+ dma_pmd_pool_destroy(per_cpu(tx_pmd_pool_order0, cpu));
+ }
+ mutex_unlock(&mutex);
+ return -ENOMEM;
+}
+
+int net_tx_dma_pmd_sysctl(const struct ctl_table *table, int write,
+ void *buffer, size_t *lenp, loff_t *ppos)
+{
+ if (write) {
+ int ret = net_tx_dma_pmd_init();
+
+ if (ret)
+ return ret;
+ }
+
+ return proc_do_static_key(table, write, buffer, lenp, ppos);
+}
+
+static struct page *__alloc_pmd(gfp_t gfp, unsigned int order)
+{
+ if (!static_branch_unlikely(&net_tx_enable_dma_pmd_key))
+ return alloc_pages(gfp, order);
+ if (order == SKB_FRAG_PAGE_ORDER)
+ return dma_pmd_pool_alloc(raw_cpu_read(tx_pmd_pool_high), gfp);
+ return dma_pmd_pool_alloc(raw_cpu_read(tx_pmd_pool_order0), gfp) ?: alloc_page(gfp);
+}
/**
* skb_page_frag_refill - check that a page_frag contains enough room
@@ -3200,7 +3282,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
if (SKB_FRAG_PAGE_ORDER &&
!static_branch_unlikely(&net_high_order_alloc_disable_key)) {
/* Avoid direct reclaim but allow kswapd to wake */
- pfrag->page = alloc_pages((gfp & ~__GFP_DIRECT_RECLAIM) |
+ pfrag->page = __alloc_pmd((gfp & ~__GFP_DIRECT_RECLAIM) |
__GFP_COMP | __GFP_NOWARN |
__GFP_NORETRY,
SKB_FRAG_PAGE_ORDER);
@@ -3209,7 +3291,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
return true;
}
}
- pfrag->page = alloc_page(gfp);
+ pfrag->page = __alloc_pmd(gfp, 0);
if (likely(pfrag->page)) {
pfrag->size = PAGE_SIZE;
return true;
diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c
index eb35da3556f4a..d9a7d9cb349e0 100644
--- a/net/core/sysctl_net_core.c
+++ b/net/core/sysctl_net_core.c
@@ -651,6 +651,13 @@ static struct ctl_table net_core_table[] = {
.mode = 0644,
.proc_handler = proc_do_static_key,
},
+ {
+ .procname = "tx_enable_dma_pmd",
+ .data = &net_tx_enable_dma_pmd_key.key,
+ .maxlen = sizeof(net_tx_enable_dma_pmd_key),
+ .mode = 0644,
+ .proc_handler = net_tx_dma_pmd_sysctl,
+ },
{
.procname = "gro_normal_batch",
.data = &net_hotdata.gro_normal_batch,
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 13/22] net/core: Use DMA_PMD for page_pool memory
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (11 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill() Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 14/22] iommu/dma: Support decrypted and pinned DMA_PMD pages Luigi Rizzo
` (37 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_rxbuf is enabled on the pool device,
use a per-page_pool DMA_PMD pool for page_pool allocations
(__page_pool_alloc_netmems_slow() and __page_pool_alloc_page_order()).
When enabled, pages leaving the page_pool cache (e.g. held by socket
buffers beyond NAPI) return to the DMA_PMD pool on final put_page(),
retaining their cached IOVA mapping, avoiding per-page 4KB IOMMU
PTE updates and IOTLB flushes on replenishment. Plain 4KB pages are
opportunistically replaced in __page_pool_page_can_be_recycled() when
free pooled blocks are available.
Also skip inserting and erasing DMA_PMD pages in the pool->dma_mapped
XArray (used for DMA_ATTR_SKIP_CPU_SYNC tracking), as dma_unmap_page_attrs()
is a no-op for DMA_PMD pages and skipping the XArray avoids xa_lock
contention on refill and release.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
include/net/page_pool/types.h | 3 ++
net/core/page_pool.c | 61 ++++++++++++++++++++++++++++-------
2 files changed, 52 insertions(+), 12 deletions(-)
diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h
index 03da138722f58..62a5c50dda43f 100644
--- a/include/net/page_pool/types.h
+++ b/include/net/page_pool/types.h
@@ -34,6 +34,8 @@
#define PP_FLAG_ALL (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \
PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM)
+struct dma_pmd_pool;
+
/* Index limit to stay within PP_DMA_INDEX_BITS for DMA indices */
#define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1)
@@ -234,6 +236,7 @@ struct page_pool {
void *mp_priv;
const struct memory_provider_ops *mp_ops;
+ struct dma_pmd_pool *dma_pmd_pool;
struct xarray dma_mapped;
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index d7c88c0b67e53..5774247a06291 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -22,6 +22,7 @@
#include <linux/page-flags.h>
#include <linux/mm.h> /* for put_page() */
#include <linux/poison.h>
+#include <linux/dma-pmd.h>
#include <linux/ethtool.h>
#include <linux/netdevice.h>
@@ -304,6 +305,10 @@ static int page_pool_init(struct page_pool *pool,
goto free_ptr_ring;
}
+ if (pool->dma_map && pool->p.dev && !pool->mp_ops &&
+ READ_ONCE(pool->p.dev->dma_pmd_rxbuf))
+ pool->dma_pmd_pool = dma_pmd_pool_create(pool->p.order, 8);
+
return 0;
free_ptr_ring:
@@ -318,6 +323,7 @@ static int page_pool_init(struct page_pool *pool,
static void page_pool_uninit(struct page_pool *pool)
{
+ pool->dma_pmd_pool = dma_pmd_pool_destroy(pool->dma_pmd_pool);
ptr_ring_cleanup(&pool->ring, NULL);
xa_destroy(&pool->dma_mapped);
@@ -406,7 +412,9 @@ static noinline netmem_ref page_pool_refill_alloc_cache(struct page_pool *pool)
if (unlikely(!netmem))
break;
- if (likely(netmem_is_pref_nid(netmem, pref_nid))) {
+ if (likely(netmem_is_pref_nid(netmem, pref_nid)) ||
+ (pool->dma_pmd_pool &&
+ dma_is_pmd_page(page_to_pfn(netmem_to_page(netmem))))) {
pool->alloc.cache[pool->alloc.count++] = netmem;
} else {
/* NUMA mismatch;
@@ -565,9 +573,20 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem, gfp_t g
goto unmap_failed;
}
- err = page_pool_register_dma_index(pool, netmem, gfp);
- if (err)
- goto unset_failed;
+ /*
+ * DMA_PMD mappings are owned by dma_pmd_pool (and torn down on pool or
+ * IOMMU domain release without pool->p.dev), not unmapped per-page in
+ * page_pool_scrub(). Leave dma_index == 0 so page_pool_release_dma_index()
+ * skips dma_unmap_page_attrs() without dereferencing pool->p.dev on
+ * late returns after device removal.
+ */
+ if (pool->dma_pmd_pool && dma_is_pmd_dma(pool->p.dev, dma)) {
+ netmem_set_dma_index(netmem, 0);
+ } else {
+ err = page_pool_register_dma_index(pool, netmem, gfp);
+ if (err)
+ goto unset_failed;
+ }
page_pool_dma_sync_for_device(pool, netmem, pool->p.max_len);
@@ -585,10 +604,13 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem, gfp_t g
static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
gfp_t gfp)
{
- struct page *page;
+ struct page *page = NULL;
gfp |= __GFP_COMP;
- page = alloc_pages_node(pool->p.nid, gfp, pool->p.order);
+ if (pool->dma_pmd_pool)
+ page = dma_pmd_pool_alloc_node(pool->dma_pmd_pool, gfp, pool->p.nid);
+ if (!page)
+ page = alloc_pages_node(pool->p.nid, gfp, pool->p.order);
if (unlikely(!page))
return NULL;
@@ -634,8 +656,14 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(struct page_pool *pool
/* Mark empty alloc.cache slots "empty" for alloc_pages_bulk */
memset(&pool->alloc.cache, 0, sizeof(void *) * bulk);
- nr_pages = alloc_pages_bulk_node(gfp, pool->p.nid, bulk,
- (struct page **)pool->alloc.cache);
+ nr_pages = 0;
+ if (pool->dma_pmd_pool)
+ nr_pages = dma_pmd_pool_alloc_bulk_node(pool->dma_pmd_pool, gfp,
+ pool->p.nid, bulk,
+ (struct page **)pool->alloc.cache);
+ if (!nr_pages)
+ nr_pages = alloc_pages_bulk_node(gfp, pool->p.nid, bulk,
+ (struct page **)pool->alloc.cache);
if (unlikely(!nr_pages))
return 0;
@@ -825,9 +853,18 @@ static bool page_pool_recycle_in_cache(netmem_ref netmem,
static bool __page_pool_page_can_be_recycled(netmem_ref netmem)
{
- return netmem_is_net_iov(netmem) ||
- (page_ref_count(netmem_to_page(netmem)) == 1 &&
- !page_is_pfmemalloc(netmem_to_page(netmem)));
+ struct page *page;
+
+ if (netmem_is_net_iov(netmem))
+ return true;
+
+ page = netmem_to_page(netmem);
+ if (page_ref_count(page) != 1 || page_is_pfmemalloc(page))
+ return false;
+ if (page->pp->dma_pmd_pool && !dma_is_pmd_page(page_to_pfn(page)) &&
+ dma_pmd_pool_has_free(page->pp->dma_pmd_pool))
+ return false;
+ return true;
}
/* If the page refcnt == 1, this will try to recycle the page.
@@ -1178,7 +1215,7 @@ static void page_pool_scrub(struct page_pool *pool)
* if there are no outstanding mapped pages.
*/
if (dma_dev_need_sync(pool->p.dev) &&
- !xa_empty(&pool->dma_mapped))
+ (pool->dma_pmd_pool || !xa_empty(&pool->dma_mapped)))
synchronize_net();
}
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 14/22] iommu/dma: Support decrypted and pinned DMA_PMD pages
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (12 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 13/22] net/core: Use DMA_PMD for page_pool memory Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 15/22] iommu/dma: Add background page scrubber for DMA_PMD pools Luigi Rizzo
` (36 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
In Confidential Computing (CoCo) guests (AMD SEV-SNP, Intel TDX, ARM CCA),
guest memory is private/encrypted by default and DMA buffers must be
converted to shared/decrypted in the host page tables before a device
can access them. Similarly, in preemptible VMs, IO-accessible memory may
need to be pinned in the host on allocation and unpinned on release.
Doing these conversions on a per-packet 4K basis is prohibitive. With
DMA_PMD pools, a 2MB page is allocated once from the buddy allocator,
handed out in sub-blocks, and only returned when completely idle.
Add two module parameters (dma_pmd.decrypt and dma_pmd.pin, with
dma_pmd.decrypt auto-enabled when CC_ATTR_GUEST_MEM_ENCRYPT is active)
and per-PMD-page state flags in meta->flags (DMA_PMD_DECRYPTED,
DMA_PMD_PINNED):
- On 2MB page allocation (dma_pmd_add_page), call dma_pmd_page_prepare()
before splitting the compound page to invoke set_memory_decrypted()
and/or the host pin hook. Skip conversion if the caller cannot sleep
(!gfpflags_allow_blocking), since set_memory_decrypted() acquires
cpa_lock and allocates PTE pages.
- On 2MB page release (dma_pmd_release_page / dma_pmd_release_page_rcu),
call dma_pmd_page_unprepare() before RCU free to restore
set_memory_encrypted() and/or unpin before returning the 2MB page to
the buddy allocator. If restoration fails, leak the 2MB page rather
than returning a page with inconsistent encryption/pin state to buddy.
- In dma_pmd_shrink_scan(), defer decrypted/pinned pages to the async
reclaim workqueue (dma_pmd_free_list) instead of calling
dma_pmd_release_page() inline under fs_reclaim.
- Add dma_is_pmd_direct(phys) and use it in dma_direct_map_phys() to
bypass swiotlb_map() bounce buffering for decrypted/pinned DMA_PMD
pages, using phys_to_dma_direct() so force_dma_unencrypted() devices
receive unencrypted DMA addresses.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-kunit.c | 2 +-
drivers/iommu/dma-pmd-meta.c | 15 +++-
drivers/iommu/dma-pmd-pool.c | 147 ++++++++++++++++++++++++++++------
drivers/iommu/dma-pmd-priv.h | 14 +++-
include/linux/dma-pmd.h | 15 ++++
kernel/dma/direct.c | 6 +-
6 files changed, 168 insertions(+), 31 deletions(-)
diff --git a/drivers/iommu/dma-pmd-kunit.c b/drivers/iommu/dma-pmd-kunit.c
index c2363277e2ecd..7fcc2bcb315bc 100644
--- a/drivers/iommu/dma-pmd-kunit.c
+++ b/drivers/iommu/dma-pmd-kunit.c
@@ -53,7 +53,7 @@ static void test_meta_invalid_phys(struct kunit *test)
static void dma_pmd_meta_set_pooled(struct dma_pmd_meta *m, bool pooled)
{
- WRITE_ONCE(m->pooled, pooled);
+ WRITE_ONCE(m->flags, pooled ? DMA_PMD_POOLED : 0);
}
static void test_meta_pooled_toggle(struct kunit *test)
diff --git a/drivers/iommu/dma-pmd-meta.c b/drivers/iommu/dma-pmd-meta.c
index 8f2bc7d77e5bd..744964d2c7974 100644
--- a/drivers/iommu/dma-pmd-meta.c
+++ b/drivers/iommu/dma-pmd-meta.c
@@ -8,6 +8,7 @@
#include <linux/bitmap.h>
#include <linux/cache.h>
#include <linux/cacheflush.h>
+#include <linux/cc_platform.h>
#include <linux/dma-pmd.h>
#include <linux/export.h>
#include <linux/gfp.h>
@@ -64,10 +65,22 @@ static __always_inline struct dma_pmd_meta *__dma_pmd_meta_of_pfn(unsigned long
bool __dma_is_pmd_page(unsigned long pfn)
{
- return READ_ONCE(__dma_pmd_meta_of_pfn(pfn)->pooled);
+ return READ_ONCE(__dma_pmd_meta_of_pfn(pfn)->flags) & DMA_PMD_POOLED;
}
EXPORT_SYMBOL(__dma_is_pmd_page);
+bool __dma_is_pmd_direct(unsigned long pfn)
+{
+ u8 flags = READ_ONCE(__dma_pmd_meta_of_pfn(pfn)->flags);
+
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+ !(flags & DMA_PMD_DECRYPTED))
+ return false;
+ return (flags & DMA_PMD_POOLED) &&
+ (flags & (DMA_PMD_DECRYPTED | DMA_PMD_PINNED));
+}
+EXPORT_SYMBOL(__dma_is_pmd_direct);
+
/**
* dma_pmd_meta_of_pfn - Metadata for a PFN known to be used by DMA_PMD.
* @pfn: PFN the caller already holds a struct page for
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index 1d4244ba422a3..deacb6f7d3610 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/cc_platform.h>
#include <linux/debugfs.h>
#include <linux/dma-mapping.h>
#include <linux/dma-pmd.h>
@@ -22,6 +23,7 @@
#include <linux/mutex.h>
#include <linux/rcupdate.h>
#include <linux/seq_file.h>
+#include <linux/set_memory.h>
#include <linux/shrinker.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
@@ -39,6 +41,14 @@ static LLIST_HEAD(dma_pmd_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *dma_pmd_wq __ro_after_init;
+/* Mark pooled PMD pages decrypted for DMA (auto-enabled on CoCo guests). */
+static bool dma_pmd_decrypt __read_mostly;
+module_param_named(decrypt, dma_pmd_decrypt, bool, 0644);
+
+/* Pin pooled PMD pages in the host hypervisor on allocation and unpin on release. */
+static bool dma_pmd_pin __read_mostly;
+module_param_named(pin, dma_pmd_pin, bool, 0644);
+
/*
* Running total and global ceiling on DMA_PMD pages pinned across all pools.
* If zero, it is set at boot at 1/8 of total RAM.
@@ -65,6 +75,70 @@ static void dma_pmd_pool_free_kref(struct kref *kref)
dma_pmd_schedule_reclaim();
}
+static int dma_pmd_page_prepare(struct page *page, struct dma_pmd_meta *meta,
+ bool can_block)
+{
+ bool want_decrypt = READ_ONCE(dma_pmd_decrypt);
+ bool want_pin = READ_ONCE(dma_pmd_pin);
+ void *vaddr = page_address(page);
+ int ret;
+
+ if (!want_decrypt && !want_pin)
+ return 0;
+
+ /*
+ * Decryption and host-pinning hypercalls may sleep. In atomic context,
+ * leave DMA_PMD_DECRYPTED/PINNED unset so the page is still pooled but
+ * uses normal swiotlb bounce buffering when required.
+ */
+ if (!can_block) {
+ pr_warn_ratelimited("%s: cannot block but want %s %s\n", __func__,
+ want_decrypt ? "decrypt" : "", want_pin ? "pin" : "");
+ return 0;
+ }
+
+ /*
+ * Explicit page decryption is for CoCo guests (where force_dma_unencrypted()
+ * is true), not bare-metal SME hosts where 64-bit devices and the IOMMU
+ * set the encryption bit in DMA addresses and IOMMU PTEs.
+ */
+ if (want_decrypt && !WARN_ON_ONCE(cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT))) {
+ ret = set_memory_decrypted((unsigned long)vaddr, 1 << PMD_ORDER);
+ if (ret)
+ return ret;
+ /* Initialize cleartext cachelines after flipping encryption state. */
+ memset(vaddr, 0, PMD_SIZE);
+ meta->flags |= DMA_PMD_DECRYPTED;
+ }
+
+ if (want_pin) {
+ /* Hook point for hypervisor GPA range pinning. */
+ meta->flags |= DMA_PMD_PINNED;
+ }
+
+ return 0;
+}
+
+static int dma_pmd_page_unprepare(struct dma_pmd_meta *meta)
+{
+ struct page *page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
+
+ if (meta->flags & DMA_PMD_PINNED) {
+ /* Hook point for hypervisor GPA range unpinning. */
+ WRITE_ONCE(meta->flags, meta->flags & ~DMA_PMD_PINNED);
+ }
+
+ if (meta->flags & DMA_PMD_DECRYPTED) {
+ if (set_memory_encrypted((unsigned long)page_address(page),
+ 1 << PMD_ORDER)) {
+ pr_warn_ratelimited("dma_pmd: leaking PMD page that could not be re-encrypted\n");
+ return -EIO;
+ }
+ WRITE_ONCE(meta->flags, meta->flags & ~DMA_PMD_DECRYPTED);
+ }
+ return 0;
+}
+
/*
* Releasing an idle PMD page proceeds in three stages:
*
@@ -74,7 +148,7 @@ static void dma_pmd_pool_free_kref(struct kref *kref)
* pushed locklessly onto dma_pmd_free_list and dma_pmd_reclaim_work is
* scheduled on dma_pmd_wq.
* 2. In process context, dma_pmd_release_page() unmaps all IOMMU domains
- * (dma_pmd_unmap_all()), clears meta->pooled so new lockless readers stop
+ * (dma_pmd_unmap_all()), clears DMA_PMD_POOLED so new lockless readers stop
* entering @meta, and queues dma_pmd_release_page_rcu() via call_rcu().
* 3. After an RCU grace period (once no concurrent dma_pmd_free_page() reader
* can still be dereferencing @meta), dma_pmd_release_page_rcu() unfreezes
@@ -97,22 +171,21 @@ static void dma_pmd_release_page_rcu(struct rcu_head *head)
* block is freed, the PMD frame can be reallocated and @meta overwritten.
*/
struct dma_pmd_meta *meta = container_of(head, struct dma_pmd_meta, rcu);
- struct page *page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
+ struct page *block, *page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
struct dma_pmd_pool *pool = READ_ONCE(meta->pool);
- unsigned int nr = DMA_PMD_BLOCKS(pool->order);
- unsigned int order = pool->order;
+ unsigned int order = pool->order, i, nr = DMA_PMD_BLOCKS(order);
+ bool leak = READ_ONCE(meta->flags) & DMA_PMD_DECRYPTED;
unsigned long flags;
- unsigned int i;
-
- for (i = 0; i < nr; i++) {
- struct page *block = page + (i << order);
- page_ref_unfreeze(block, 1);
- __free_pages(block, order);
+ if (!leak) {
+ for (i = 0; i < nr; i++) {
+ block = page + (i << order);
+ page_ref_unfreeze(block, 1);
+ __free_pages(block, order);
+ }
+ atomic_long_dec(&dma_pmd_nr_pages);
}
- atomic_long_dec(&dma_pmd_nr_pages);
-
spin_lock_irqsave(&pool->lock, flags);
pool->pmd_free_cnt++;
spin_unlock_irqrestore(&pool->lock, flags);
@@ -138,7 +211,9 @@ static void dma_pmd_release_page(struct dma_pmd_meta *meta)
* dma_is_pmd_page() just before this may still be reading @meta, so
* the PMD page and its metadata entry must outlive the grace period.
*/
- WRITE_ONCE(meta->pooled, false);
+ WRITE_ONCE(meta->flags, meta->flags & ~DMA_PMD_POOLED);
+ dma_pmd_page_unprepare(meta);
+
call_rcu(&meta->rcu, dma_pmd_release_page_rcu);
}
@@ -392,20 +467,33 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
return NULL;
}
- /* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
- if (split_page_compound(page, PMD_ORDER, pool->order)) {
+ meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
+ memset(meta, 0, sizeof(*meta));
+ meta->pool = pool;
+ spin_lock_init(&meta->map_lock);
+
+ if (dma_pmd_page_prepare(page, meta, can_block)) {
__free_pages(page, PMD_ORDER);
atomic_long_dec(&dma_pmd_nr_pages);
return NULL;
}
- meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
- memset(meta, 0, sizeof(*meta));
- meta->pool = pool;
- spin_lock_init(&meta->map_lock);
+ /* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
+ if (split_page_compound(page, PMD_ORDER, pool->order)) {
+ if (!dma_pmd_page_unprepare(meta)) {
+ __free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
+ }
+ return NULL;
+ }
- bitmap_set(meta->dirty_bitmap, 0, nr);
- meta->nr_dirty = nr;
+ if (meta->flags & DMA_PMD_DECRYPTED) {
+ bitmap_set(meta->free_bitmap, 0, nr);
+ meta->nr_free = nr;
+ } else {
+ bitmap_set(meta->dirty_bitmap, 0, nr);
+ meta->nr_dirty = nr;
+ }
kref_get(&pool->refcount);
@@ -418,7 +506,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
* function returns and the caller publishes the PMD page under
* @pool->lock, whose release orders both stores against every consumer.
*/
- WRITE_ONCE(meta->pooled, true);
+ WRITE_ONCE(meta->flags, meta->flags | DMA_PMD_POOLED);
return meta;
}
@@ -764,10 +852,18 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
/*
* Retiring is done outside both locks: dma_pmd_release_page() unmaps
* from the IOMMU, which is far too long to hold pool->lock for.
+ * Pages that need set_memory_encrypted() cannot be released under
+ * fs_reclaim (CPA allocates PTE pages on x86); hand them to the
+ * reclaim workqueue instead.
*/
list_for_each_entry_safe(meta, tmp, &release_list, list) {
- list_del(&meta->list);
- dma_pmd_release_page(meta);
+ list_del_init(&meta->list);
+ if (meta->flags & (DMA_PMD_DECRYPTED | DMA_PMD_PINNED)) {
+ llist_add(&meta->llnode, &dma_pmd_free_list);
+ dma_pmd_schedule_reclaim();
+ } else {
+ dma_pmd_release_page(meta);
+ }
}
srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
@@ -806,6 +902,9 @@ static int __init dma_pmd_init(void)
if (!dma_pmd_max_pages)
dma_pmd_max_pages = max(totalram_pages() >> (PMD_ORDER + 3), 16UL);
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ dma_pmd_decrypt = true;
+
/*
* Reclaim frees memory, so it must not queue behind arbitrary work on
* system_wq when the machine is already short of it. Failure is not
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index b411ecb1fdafc..8c7a50d5b7220 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -107,10 +107,16 @@ enum {
#define DMA_PMD_META_SHIFT 8
#define DMA_PMD_META_SIZE BIT(DMA_PMD_META_SHIFT)
+enum {
+ DMA_PMD_POOLED = BIT(0),
+ DMA_PMD_DECRYPTED = BIT(1),
+ DMA_PMD_PINNED = BIT(2),
+};
+
/**
* struct dma_pmd_meta - Metadata for a single DMA_PMD page
- * @pooled: True while this page is owned by an dma_pmd_pool (offset 0)
- * read locklessly by dma_is_pmd_page().
+ * @flags: Bitmask of DMA_PMD_POOLED, DMA_PMD_DECRYPTED, DMA_PMD_PINNED
+ * (offset 0), read locklessly by dma_is_pmd_page() and dma_is_pmd_direct().
* @nr_free: Number of zeroed available blocks in @free_bitmap
* @nr_dirty: Number of dirty available blocks in @dirty_bitmap
* @map_lock: Spinlock protecting slow-path updates to @domains_mapped
@@ -138,7 +144,7 @@ enum {
* physical pages (128 KB per GB of RAM).
*/
struct dma_pmd_meta {
- bool pooled;
+ u8 flags;
u16 nr_free;
u16 nr_dirty;
spinlock_t map_lock;
@@ -160,7 +166,7 @@ struct dma_pmd_meta {
DECLARE_BITMAP(dirty_bitmap, 1U << PMD_ORDER) ____cacheline_aligned;
} __aligned(DMA_PMD_META_SIZE);
-static_assert(offsetof(struct dma_pmd_meta, pooled) == 0);
+static_assert(offsetof(struct dma_pmd_meta, flags) == 0);
static_assert(sizeof(struct dma_pmd_meta) == DMA_PMD_META_SIZE);
static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
diff --git a/include/linux/dma-pmd.h b/include/linux/dma-pmd.h
index 7b7e9477d44e1..c9775e86979f9 100644
--- a/include/linux/dma-pmd.h
+++ b/include/linux/dma-pmd.h
@@ -16,6 +16,7 @@ struct dma_pmd_pool;
extern void *dma_pmd_meta_array;
bool __dma_is_pmd_page(unsigned long pfn);
+bool __dma_is_pmd_direct(unsigned long pfn);
static inline bool dma_is_pmd_page(unsigned long pfn)
{
@@ -29,6 +30,15 @@ static inline bool dma_is_pmd_page(unsigned long pfn)
return __dma_is_pmd_page(pfn);
}
+static inline bool dma_is_pmd_direct(phys_addr_t phys)
+{
+ /* Pairs with smp_store_release() in dma_pmd_meta_init(). */
+ if (likely(!smp_load_acquire(&dma_pmd_meta_array)))
+ return false;
+
+ return __dma_is_pmd_direct(phys >> PAGE_SHIFT);
+}
+
bool __dma_pmd_free_page(struct page *page);
static inline bool dma_pmd_free_page(struct page *page)
@@ -117,6 +127,11 @@ static inline bool dma_is_pmd_page(unsigned long pfn)
return false;
}
+static inline bool dma_is_pmd_direct(phys_addr_t phys)
+{
+ return false;
+}
+
static inline bool dma_pmd_free_page(struct page *page)
{
return false;
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index da665ca22d5c0..a786c2869b869 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -7,6 +7,7 @@
#include <linux/memblock.h> /* for max_pfn */
#include <linux/export.h>
#include <linux/mm.h>
+#include <linux/dma-pmd.h>
#include <linux/dma-map-ops.h>
#include <linux/scatterlist.h>
#include <linux/pfn.h>
@@ -679,7 +680,10 @@ dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
attrs |= DMA_ATTR_CC_SHARED;
}
- if (is_swiotlb_force_bounce(dev)) {
+ if (dma_is_pmd_direct(phys)) {
+ if (force_dma_unencrypted(dev))
+ attrs |= DMA_ATTR_CC_SHARED;
+ } else if (is_swiotlb_force_bounce(dev)) {
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT))
return DMA_MAPPING_ERROR;
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 15/22] iommu/dma: Add background page scrubber for DMA_PMD pools
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (13 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 14/22] iommu/dma: Support decrypted and pinned DMA_PMD pages Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 16/22] iommu/dma: Add per-NUMA-node PMD page reservoir Luigi Rizzo
` (35 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add a background scrubber worker (dma_pmd_scrub_work) that zeroes dirty
blocks in meta->dirty_bitmap[] and moves them to meta->free_bitmap[],
and change __GFP_ZERO (want_init_on_alloc) allocations to take blocks
exclusively from free_bitmap[]:
- Remove the per-block memset() and out_prep: loop from
dma_pmd_pool_alloc_bulk_node(). When dma_pmd_add_page() allocates a
new PMD page for a zeroed request, zero the 2MB page once and place
all blocks in free_bitmap[].
- Maintain pool->partial (nr_free > 0, with clean-only pages at head and
mixed pages at tail), pool->partial_dirty (nr_free == 0, nr_dirty > 0),
and pool->idle (clean pages at head, dirty pages at tail) so both
allocation and scrubbing find candidate PMD pages in O(1) time.
- Trigger dma_pmd_scrub_work only when meta->nr_dirty transitions from 0
to 1 on block free (or when a non-zeroed PMD page is added).
- Detach up to DMA_PMD_SCRUB_BATCH (64) dirty blocks under pool->lock,
zero them outside the lock, and re-insert them into free_bitmap[].
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-pool.c | 336 ++++++++++++++++++++++++++++++-----
drivers/iommu/dma-pmd-priv.h | 9 +-
2 files changed, 296 insertions(+), 49 deletions(-)
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index deacb6f7d3610..d774cd05347b8 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -32,6 +32,8 @@
#include "dma-pmd-priv.h"
+#define DMA_PMD_SCRUB_BATCH 64U
+
/*
* DMA_PMD pages whose last block has just been freed and which are over the
* pool's idle watermark are pushed here locklessly for later release.
@@ -66,6 +68,7 @@ static DEFINE_MUTEX(dma_pmd_pools_lock);
static LLIST_HEAD(dma_pmd_dead_pools);
static void dma_pmd_schedule_reclaim(void);
+static void dma_pmd_schedule_scrub(void);
static void dma_pmd_pool_free_kref(struct kref *kref)
{
@@ -250,6 +253,216 @@ static void dma_pmd_schedule_reclaim(void)
schedule_work(&dma_pmd_reclaim_work);
}
+/*
+ * Background page scrubber
+ * ------------------------
+ * Freed blocks are returned to @meta->dirty_bitmap so __dma_pmd_free_page()
+ * never pays a memset() on the free hot path. Callers that do not require
+ * zeroed memory take dirty blocks first, while callers with __GFP_ZERO (or
+ * init_on_alloc) take pre-zeroed blocks from @meta->free_bitmap.
+ *
+ * The scrubber runs on system_wq to asynchronously convert dirty blocks into
+ * zeroed blocks in @meta->free_bitmap. To avoid holding @pool->lock across
+ * memset(), it detaches up to DMA_PMD_SCRUB_BATCH (64) dirty bits under
+ * @pool->lock, zeroes the pages unlocked (while their bits are clear in both
+ * bitmaps so no allocator or double-free check can race on them), and
+ * republishes them in @meta->free_bitmap under @pool->lock.
+ *
+ * Triggered lazily (without touching any list on every __dma_pmd_free_page()):
+ * when a PMD page transitions from 0 to 1 dirty block, when a page with dirty
+ * blocks becomes completely idle, or when a __GFP_ZERO allocation misses on
+ * clean pages while dirty pages are present.
+ */
+
+/*
+ * Place @meta on the appropriate pool list according to its available clean
+ * (@nr_free) and dirty (@nr_dirty) block counts. Caller must hold @pool->lock
+ * and update @pool->num_idle_pages when moving into or out of @pool->idle.
+ *
+ * List invariants:
+ * - @pool->full: avail == 0
+ * - @pool->partial: 0 < avail < nr, nr_free > 0
+ * (nr_dirty == 0 at head, nr_dirty > 0 at tail)
+ * - @pool->partial_dirty: 0 < avail < nr, nr_free == 0, nr_dirty > 0
+ * - @pool->idle: avail == nr
+ * (nr_dirty == 0 at head, nr_dirty > 0 at tail)
+ */
+static void dma_pmd_place_meta(struct dma_pmd_pool *pool,
+ struct dma_pmd_meta *meta)
+{
+ unsigned int avail = dma_pmd_meta_avail(meta);
+ unsigned int nr = DMA_PMD_BLOCKS(pool->order);
+
+ if (!avail) {
+ list_move(&meta->list, &pool->full);
+ } else if (avail == nr) {
+ if (!meta->nr_dirty)
+ list_move(&meta->list, &pool->idle);
+ else
+ list_move_tail(&meta->list, &pool->idle);
+ } else if (!meta->nr_free) {
+ list_move_tail(&meta->list, &pool->partial_dirty);
+ } else if (!meta->nr_dirty) {
+ list_move(&meta->list, &pool->partial);
+ } else {
+ list_move_tail(&meta->list, &pool->partial);
+ }
+}
+
+static struct dma_pmd_meta *dma_pmd_pick_dirty_meta(struct dma_pmd_pool *pool)
+{
+ struct dma_pmd_meta *meta;
+
+ meta = list_first_entry_or_null(&pool->partial_dirty,
+ struct dma_pmd_meta, list);
+ if (meta)
+ return meta;
+
+ if (!list_empty(&pool->partial)) {
+ meta = list_last_entry(&pool->partial, struct dma_pmd_meta, list);
+ if (meta->nr_dirty)
+ return meta;
+ }
+
+ if (!list_empty(&pool->idle)) {
+ meta = list_last_entry(&pool->idle, struct dma_pmd_meta, list);
+ if (meta->nr_dirty) {
+ pool->num_idle_pages--;
+ return meta;
+ }
+ }
+
+ return NULL;
+}
+
+/**
+ * dma_pmd_scrub_pool - Scrub one batch of dirty blocks in @pool
+ * @pool: Pool to scrub
+ *
+ * Cost: Zeroes up to DMA_PMD_SCRUB_BATCH blocks (e.g. up to 256 KB for
+ * order-0) outside @pool->lock; two brief O(batch) bitmap passes under
+ * @pool->lock.
+ * Locking: Process context. Acquires @pool->lock (irqsave) to detach and
+ * republish block bits; drops lock during memset().
+ * Frequency: Background worker only (on system_wq).
+ *
+ * Return: true if a batch was scrubbed (more dirty blocks may remain),
+ * false if @pool has no dirty blocks left or is destroyed.
+ */
+static bool dma_pmd_scrub_pool(struct dma_pmd_pool *pool)
+{
+ unsigned int nr, order, idx, count = 0, used = 0;
+ u16 idxs[DMA_PMD_SCRUB_BATCH];
+ struct dma_pmd_meta *meta;
+ bool should_release = false;
+ unsigned long base_pfn;
+ unsigned long flags;
+
+ spin_lock_irqsave(&pool->lock, flags);
+ if (pool->destroyed) {
+ spin_unlock_irqrestore(&pool->lock, flags);
+ return false;
+ }
+
+ meta = dma_pmd_pick_dirty_meta(pool);
+ if (!meta) {
+ spin_unlock_irqrestore(&pool->lock, flags);
+ return false;
+ }
+
+ order = pool->order;
+ nr = DMA_PMD_BLOCKS(order);
+ base_pfn = dma_pmd_meta_to_pfn(meta);
+
+ for_each_set_bit(idx, meta->dirty_bitmap, nr) {
+ struct page *block = pfn_to_page(base_pfn + (idx << order));
+
+ __clear_bit(idx, meta->dirty_bitmap);
+ used++;
+ if (unlikely(folio_contain_hwpoisoned_page(page_folio(block)))) {
+ pr_err_once("dma_pmd: poisoned block at pfn %lu, leaking it to keep pool %p intact\n",
+ page_to_pfn(block), pool);
+ } else {
+ idxs[count++] = idx;
+ }
+ if (used == meta->nr_dirty || count == DMA_PMD_SCRUB_BATCH)
+ break;
+ }
+ meta->nr_dirty -= used;
+ dma_pmd_place_meta(pool, meta);
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ if (unlikely(!count))
+ return used > 0;
+
+ for (idx = 0; idx < count; idx++) {
+ struct page *block = pfn_to_page(base_pfn + (idxs[idx] << order));
+
+ memset(page_address(block), 0, PAGE_SIZE << order);
+ }
+
+ spin_lock_irqsave(&pool->lock, flags);
+ for (idx = 0; idx < count; idx++)
+ __set_bit(idxs[idx], meta->free_bitmap);
+ meta->nr_free += count;
+ pool->block_scrub_cnt += count;
+
+ if (dma_pmd_meta_avail(meta) == nr) {
+ if (pool->destroyed || pool->num_idle_pages >= pool->max_idle_pages) {
+ list_del_init(&meta->list);
+ llist_add(&meta->llnode, &dma_pmd_free_list);
+ should_release = true;
+ } else {
+ pool->num_idle_pages++;
+ dma_pmd_place_meta(pool, meta);
+ }
+ } else if (!pool->destroyed) {
+ dma_pmd_place_meta(pool, meta);
+ }
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ if (should_release)
+ dma_pmd_schedule_reclaim();
+
+ return true;
+}
+
+/*
+ * Walk all live pools and scrub dirty blocks until every pool is clean.
+ *
+ * Cost: Proportional to total dirty blocks across all pools; yields via
+ * cond_resched() between batches of DMA_PMD_SCRUB_BATCH blocks.
+ * Locking: Process context (system_wq). Holds @dma_pmd_pools_lock across the
+ * pool walk.
+ * Frequency: Scheduled on demand via dma_pmd_schedule_scrub(); coalesced by
+ * work_pending().
+ */
+static void dma_pmd_scrub_work_fn(struct work_struct *work)
+{
+ struct dma_pmd_pool *pool;
+ bool progress;
+
+ do {
+ progress = false;
+ mutex_lock(&dma_pmd_pools_lock);
+ list_for_each_entry(pool, &dma_pmd_pools, node) {
+ while (dma_pmd_scrub_pool(pool)) {
+ progress = true;
+ cond_resched();
+ }
+ }
+ mutex_unlock(&dma_pmd_pools_lock);
+ } while (progress);
+}
+
+static DECLARE_WORK(dma_pmd_scrub_work, dma_pmd_scrub_work_fn);
+
+static void dma_pmd_schedule_scrub(void)
+{
+ if (!work_pending(&dma_pmd_scrub_work))
+ schedule_work(&dma_pmd_scrub_work);
+}
+
unsigned int dma_pmd_pools_forget_domain(int idx)
{
struct dma_pmd_pool *pool;
@@ -262,6 +475,8 @@ unsigned int dma_pmd_pools_forget_domain(int idx)
spin_lock_irqsave(&pool->lock, flags);
list_for_each_entry(meta, &pool->partial, list)
dropped += dma_pmd_forget_domain(meta, idx);
+ list_for_each_entry(meta, &pool->partial_dirty, list)
+ dropped += dma_pmd_forget_domain(meta, idx);
list_for_each_entry(meta, &pool->idle, list)
dropped += dma_pmd_forget_domain(meta, idx);
list_for_each_entry(meta, &pool->full, list)
@@ -311,6 +526,7 @@ struct dma_pmd_pool *dma_pmd_pool_create(unsigned int order, unsigned int max_id
pool->max_idle_pages = max_idle_pages ? : 16;
pool->next_alloc_attempt = jiffies;
INIT_LIST_HEAD(&pool->partial);
+ INIT_LIST_HEAD(&pool->partial_dirty);
INIT_LIST_HEAD(&pool->idle);
INIT_LIST_HEAD(&pool->full);
@@ -368,6 +584,7 @@ struct dma_pmd_pool *dma_pmd_pool_destroy(struct dma_pmd_pool *pool)
}
srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
+ flush_work(&dma_pmd_scrub_work);
kref_put(&pool->refcount, dma_pmd_pool_free_kref);
flush_work(&dma_pmd_reclaim_work);
return NULL;
@@ -379,6 +596,7 @@ EXPORT_SYMBOL(dma_pmd_pool_destroy);
* @pool: Owning pool
* @gfp: GFP allocation flags
* @nid: Target NUMA node (or NUMA_NO_NODE for local node)
+ * @zero: True if caller requires zeroed blocks in @meta->free_bitmap
*
* Cost: Slow path (pool miss); allocates PMD page from buddy on @nid,
* splits into compound blocks, and sets the PMD page's membership bit.
@@ -390,7 +608,7 @@ EXPORT_SYMBOL(dma_pmd_pool_destroy);
* on failure.
*/
static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
- gfp_t gfp, int nid)
+ gfp_t gfp, int nid, bool zero)
{
/*
* Require full GFP_KERNEL (not just gfpflags_allow_blocking()):
@@ -431,7 +649,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
* is shaped into compound pieces by hand. What is left is also free
* of everything slab treats as a bug in GFP_SLAB_BUG_MASK, so the
* metadata allocation below can use it as it stands.
- * __GFP_ZERO is handled per-block in dma_pmd_pool_alloc_bulk_node().
+ * __GFP_ZERO is handled explicitly below after dma_pmd_page_prepare().
*/
base_gfp = gfp & ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE | __GFP_ZERO);
@@ -487,7 +705,9 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
return NULL;
}
- if (meta->flags & DMA_PMD_DECRYPTED) {
+ if (zero || (meta->flags & DMA_PMD_DECRYPTED)) {
+ if (!(meta->flags & DMA_PMD_DECRYPTED))
+ memset(page_address(page), 0, PMD_SIZE);
bitmap_set(meta->free_bitmap, 0, nr);
meta->nr_free = nr;
} else {
@@ -514,8 +734,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
static unsigned int dma_pmd_scan_bitmap(struct dma_pmd_pool *pool,
unsigned long *bitmap, u16 *countp,
unsigned long base_pfn, unsigned int nr,
- unsigned long want, struct page **out,
- bool unfreeze)
+ unsigned long want, struct page **out)
{
unsigned int idx, got = 0, used = 0;
@@ -531,8 +750,7 @@ static unsigned int dma_pmd_scan_bitmap(struct dma_pmd_pool *pool,
pr_err_once("dma_pmd: poisoned block at pfn %lu, leaking it to keep pool %p intact\n",
page_to_pfn(block), pool);
} else {
- if (unfreeze)
- page_ref_unfreeze(block, 1);
+ page_ref_unfreeze(block, 1);
out[got++] = block;
}
@@ -544,6 +762,41 @@ static unsigned int dma_pmd_scan_bitmap(struct dma_pmd_pool *pool,
return got;
}
+static struct dma_pmd_meta *dma_pmd_pick_meta(struct dma_pmd_pool *pool, bool zero)
+{
+ struct dma_pmd_meta *meta;
+
+ if (!zero) {
+ meta = list_first_entry_or_null(&pool->partial_dirty,
+ struct dma_pmd_meta, list);
+ if (meta)
+ return meta;
+ if (!list_empty(&pool->partial))
+ return list_last_entry(&pool->partial,
+ struct dma_pmd_meta, list);
+ if (!list_empty(&pool->idle)) {
+ meta = list_last_entry(&pool->idle,
+ struct dma_pmd_meta, list);
+ pool->num_idle_pages--;
+ return meta;
+ }
+ return NULL;
+ }
+
+ meta = list_first_entry_or_null(&pool->partial, struct dma_pmd_meta, list);
+ if (meta)
+ return meta;
+
+ list_for_each_entry(meta, &pool->idle, list) {
+ if (meta->nr_free) {
+ pool->num_idle_pages--;
+ return meta;
+ }
+ }
+
+ return NULL;
+}
+
/* Extract up to @want available blocks from @meta under @pool->lock. */
static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
struct dma_pmd_meta *meta,
@@ -555,23 +808,18 @@ static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
unsigned int nr = DMA_PMD_BLOCKS(pool->order);
unsigned int got = 0, used;
- if (zero) {
- got += dma_pmd_scan_bitmap(pool, meta->free_bitmap, &meta->nr_free,
- base_pfn, nr, want, out, true);
+ if (!zero)
got += dma_pmd_scan_bitmap(pool, meta->dirty_bitmap, &meta->nr_dirty,
- base_pfn, nr, want - got, out + got, false);
- } else {
- got += dma_pmd_scan_bitmap(pool, meta->dirty_bitmap, &meta->nr_dirty,
- base_pfn, nr, want, out, false);
- got += dma_pmd_scan_bitmap(pool, meta->free_bitmap, &meta->nr_free,
- base_pfn, nr, want - got, out + got, true);
- }
+ base_pfn, nr, want, out);
+ got += dma_pmd_scan_bitmap(pool, meta->free_bitmap, &meta->nr_free,
+ base_pfn, nr, want - got, out + got);
+
used = avail_before - dma_pmd_meta_avail(meta);
- if (!dma_pmd_meta_avail(meta) || WARN_ON_ONCE(!used)) {
+ if (WARN_ON_ONCE(!used)) {
meta->nr_free = 0;
meta->nr_dirty = 0;
- list_move(&meta->list, &pool->full);
}
+ dma_pmd_place_meta(pool, meta);
return got;
}
@@ -588,7 +836,7 @@ static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
* under a single lock acquisition. Slow path (both empty) calls
* dma_pmd_add_page().
* Locking: Acquires @pool->lock (irqsave) for the bitmap scan. Drops lock
- * during slow-path PMD buddy allocation and per-block initialization.
+ * during slow-path PMD buddy allocation.
* Frequency: High (called per RX buffer refill or per SKB TX page frag refill).
*
* A pool hands out a block of a PMD page it already owns, on the node that
@@ -605,7 +853,8 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
int nid, unsigned long nr_pages,
struct page **page_array)
{
- unsigned long allocated = 0, flags, i;
+ unsigned long allocated = 0, flags;
+ bool should_scrub = false;
bool zero;
if (unlikely(!pool || pool->destroyed || !nr_pages ||
@@ -619,22 +868,18 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
while (allocated < nr_pages) {
struct dma_pmd_meta *meta;
- /* Partially used pages first, idle ones next. */
- meta = list_first_entry_or_null(&pool->partial, struct dma_pmd_meta, list);
- if (!meta && !list_empty(&pool->idle)) {
- meta = list_first_entry(&pool->idle, struct dma_pmd_meta, list);
- list_move(&meta->list, &pool->partial);
- pool->num_idle_pages--;
- }
+ meta = dma_pmd_pick_meta(pool, zero);
if (!meta) {
/* Fallback to a new allocation */
spin_unlock_irqrestore(&pool->lock, flags);
- meta = dma_pmd_add_page(pool, gfp, nid);
+ meta = dma_pmd_add_page(pool, gfp, nid, zero);
if (!meta)
- goto out_prep;
+ goto out;
spin_lock_irqsave(&pool->lock, flags);
pool->pmd_alloc_cnt++;
list_add(&meta->list, &pool->partial);
+ if (meta->nr_dirty)
+ should_scrub = true;
}
allocated += dma_pmd_take_blocks(pool, meta,
@@ -643,14 +888,9 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
}
spin_unlock_irqrestore(&pool->lock, flags);
-out_prep:
- for (i = 0; i < allocated; i++) {
- if (!page_count(page_array[i])) {
- page_ref_unfreeze(page_array[i], 1);
- if (zero)
- memset(page_address(page_array[i]), 0, PAGE_SIZE << pool->order);
- }
- }
+out:
+ if (should_scrub)
+ dma_pmd_schedule_scrub();
return allocated;
}
EXPORT_SYMBOL(dma_pmd_pool_alloc_bulk_node);
@@ -674,7 +914,9 @@ EXPORT_SYMBOL(dma_pmd_pool_alloc_bulk_node);
bool dma_pmd_pool_has_free(struct dma_pmd_pool *pool)
{
return pool && !READ_ONCE(pool->destroyed) &&
- (!list_empty_careful(&pool->partial) || !list_empty_careful(&pool->idle));
+ (!list_empty_careful(&pool->partial) ||
+ !list_empty_careful(&pool->partial_dirty) ||
+ !list_empty_careful(&pool->idle));
}
EXPORT_SYMBOL(dma_pmd_pool_has_free);
@@ -706,8 +948,8 @@ bool __dma_pmd_free_page(struct page *page)
struct dma_pmd_pool *pool = meta->pool;
unsigned long pfn = page_to_pfn(page);
bool should_release = false;
+ bool should_scrub = false;
bool zeroed_on_free = false;
- unsigned int avail;
unsigned long flags;
idx = (pfn - dma_pmd_meta_to_pfn(meta)) >> pool->order;
@@ -755,28 +997,28 @@ bool __dma_pmd_free_page(struct page *page)
meta->nr_free++;
} else {
__set_bit(idx, meta->dirty_bitmap);
- meta->nr_dirty++;
+ should_scrub = !meta->nr_dirty++;
}
- avail = dma_pmd_meta_avail(meta);
pool->block_free_cnt++;
- if (avail == 1 && !pool->destroyed)
- list_move(&meta->list, &pool->partial);
-
- if (avail == nr) {
+ if (dma_pmd_meta_avail(meta) == nr) {
if (pool->destroyed || pool->num_idle_pages >= pool->max_idle_pages) {
list_del_init(&meta->list);
llist_add(&meta->llnode, &dma_pmd_free_list);
should_release = true;
} else {
- list_move(&meta->list, &pool->idle);
pool->num_idle_pages++;
+ dma_pmd_place_meta(pool, meta);
}
+ } else if (!pool->destroyed) {
+ dma_pmd_place_meta(pool, meta);
}
spin_unlock_irqrestore(&pool->lock, flags);
if (should_release)
dma_pmd_schedule_reclaim();
+ else if (should_scrub)
+ dma_pmd_schedule_scrub();
return true;
}
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 8c7a50d5b7220..82a5c869aa216 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -181,8 +181,10 @@ static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
* block bitmaps and the statistics counters
* @order: Block order managed by this pool (<= PMD_ORDER)
* @destroyed: Set when dma_pmd_pool_destroy() has been called
- * @partial: List of PMD pages with at least 1 free block, but not all
- * of them (MRU ordered). Candidates for allocation.
+ * @partial: List of partially used PMD pages with @nr_free > 0 (clean-only
+ * at head, mixed clean/dirty at tail)
+ * @partial_dirty: List of partially used PMD pages with @nr_free == 0 and
+ * @nr_dirty > 0
* @idle: List of PMD pages whose every usable block is free. Held
* separately because they can be released under pressure.
* @full: List of PMD pages with 0 free blocks
@@ -193,6 +195,7 @@ static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
* @pmd_free_cnt: Statistics counter of PMD pages released back to buddy
* @block_alloc_cnt: Statistics counter of block allocations satisfied
* @block_free_cnt: Statistics counter of block frees recycled into pool
+ * @block_scrub_cnt: Statistics counter of dirty blocks zeroed by scrubber
* @next_alloc_attempt: Do not attempt a new order-9 allocation before this time.
* Damps repeated high-order GFP_ATOMIC failures under
* fragmentation, which would otherwise be retried on every
@@ -207,6 +210,7 @@ struct dma_pmd_pool {
u8 order;
bool destroyed;
struct list_head partial;
+ struct list_head partial_dirty;
struct list_head idle;
struct list_head full;
unsigned int num_idle_pages;
@@ -217,6 +221,7 @@ struct dma_pmd_pool {
u64 pmd_free_cnt;
u64 block_alloc_cnt;
u64 block_free_cnt;
+ u64 block_scrub_cnt;
/* Cold. */
unsigned long next_alloc_attempt;
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 16/22] iommu/dma: Add per-NUMA-node PMD page reservoir
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (14 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 15/22] iommu/dma: Add background page scrubber for DMA_PMD pools Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 17/22] net/gve: Use DMA_PMD memory for RX buffers Luigi Rizzo
` (34 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
Add a per-NUMA-node reservoir of pre-allocated, pre-zeroed, and
pre-prepared 2MB pages (dma_pmd_reservoirs[MAX_NUMNODES]) refilled in
process context by a background worker (dma_pmd_reservoir_work):
- Controlled by dma_pmd.reservoir_max (default 16 PMD pages = 32 MB per
N_MEMORY node, subject to the global dma_pmd_max_pages ceiling).
- dma_pmd_add_page() first tries dma_pmd_reservoir_get(nid) (preferring
the target NUMA node and falling back to other N_MEMORY nodes),
avoiding blocking or failing order-9 buddy allocations on pool misses.
- When a node's reservoir drops to <= reservoir_max / 2 (or on pool
creation), dma_pmd_reservoir_work is scheduled to refill in the
background.
- Integrate the reservoir with dma_pmd_shrink_count() and
dma_pmd_shrink_scan() so reservoir pages are reclaimed first under
memory pressure.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-pool.c | 307 +++++++++++++++++++++++++++++++----
1 file changed, 277 insertions(+), 30 deletions(-)
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index d774cd05347b8..d48c8e19741ab 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -21,6 +21,7 @@
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/mutex.h>
+#include <linux/nodemask.h>
#include <linux/rcupdate.h>
#include <linux/seq_file.h>
#include <linux/set_memory.h>
@@ -62,6 +63,18 @@ static atomic_long_t dma_pmd_nr_pages __cacheline_aligned_in_smp;
static unsigned long dma_pmd_max_pages __read_mostly;
core_param(dma_pmd_max_pages, dma_pmd_max_pages, ulong, 0644);
+/* Target number of pre-allocated PMD pages per NUMA node in reservoir. */
+static unsigned int dma_pmd_reservoir_max __read_mostly = 16;
+module_param_named(reservoir_max, dma_pmd_reservoir_max, uint, 0644);
+
+struct dma_pmd_reservoir {
+ spinlock_t lock; /* protects @pages and @nr_pages */
+ struct list_head pages;
+ unsigned int nr_pages;
+};
+
+static struct dma_pmd_reservoir dma_pmd_reservoirs[MAX_NUMNODES];
+
/* All live pools. */
static LIST_HEAD(dma_pmd_pools);
static DEFINE_MUTEX(dma_pmd_pools_lock);
@@ -69,6 +82,7 @@ static LLIST_HEAD(dma_pmd_dead_pools);
static void dma_pmd_schedule_reclaim(void);
static void dma_pmd_schedule_scrub(void);
+static void dma_pmd_schedule_reservoir(void);
static void dma_pmd_pool_free_kref(struct kref *kref)
{
@@ -81,8 +95,10 @@ static void dma_pmd_pool_free_kref(struct kref *kref)
static int dma_pmd_page_prepare(struct page *page, struct dma_pmd_meta *meta,
bool can_block)
{
- bool want_decrypt = READ_ONCE(dma_pmd_decrypt);
- bool want_pin = READ_ONCE(dma_pmd_pin);
+ bool want_decrypt = READ_ONCE(dma_pmd_decrypt) &&
+ !(meta->flags & DMA_PMD_DECRYPTED);
+ bool want_pin = READ_ONCE(dma_pmd_pin) &&
+ !(meta->flags & DMA_PMD_PINNED);
void *vaddr = page_address(page);
int ret;
@@ -142,6 +158,16 @@ static int dma_pmd_page_unprepare(struct dma_pmd_meta *meta)
return 0;
}
+static void dma_pmd_free_reservoir_page(struct dma_pmd_meta *meta)
+{
+ struct page *page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
+
+ if (!dma_pmd_page_unprepare(meta)) {
+ __free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
+ }
+}
+
/*
* Releasing an idle PMD page proceeds in three stages:
*
@@ -226,8 +252,12 @@ static void dma_pmd_reclaim_work_fn(struct work_struct *work)
struct dma_pmd_pool *pool, *ptmp;
struct dma_pmd_meta *meta, *tmp;
- llist_for_each_entry_safe(meta, tmp, node, llnode)
- dma_pmd_release_page(meta);
+ llist_for_each_entry_safe(meta, tmp, node, llnode) {
+ if (meta->pool)
+ dma_pmd_release_page(meta);
+ else
+ dma_pmd_free_reservoir_page(meta);
+ }
node = llist_del_all(&dma_pmd_dead_pools);
if (node) {
@@ -463,6 +493,186 @@ static void dma_pmd_schedule_scrub(void)
schedule_work(&dma_pmd_scrub_work);
}
+/*
+ * Per-NUMA-node PMD page reservoir
+ * --------------------------------
+ * Order-9 (2MB) buddy allocations can compact/reclaim and fail or stall in
+ * atomic context (such as NAPI RX refill or softirq TX), and CoCo decryption /
+ * hypervisor pinning cannot run in atomic context at all.
+ *
+ * Each NUMA node maintains a small reservoir of up to @dma_pmd_reservoir_max
+ * (default 8 = 16 MB/node) pre-allocated, pre-zeroed, and pre-decrypted/pinned
+ * PMD pages. On a pool miss, dma_pmd_add_page() pops a ready PMD page from the
+ * target node's reservoir in O(1) under a spinlock (falling back to other
+ * nodes before calling alloc_pages_node() directly).
+ *
+ * A background worker on system_wq refills nodes whenever a node's count drops
+ * to or below half of @dma_pmd_reservoir_max, on a reservoir miss, or when a
+ * new pool is created. Pages held in the reservoir count toward the global
+ * @dma_pmd_nr_pages budget and are reclaimable by the shrinker under memory
+ * pressure.
+ */
+
+/**
+ * dma_pmd_reservoir_refill_node - Refill node @nid's reservoir up to @max
+ * @nid: NUMA node ID
+ * @max: Target number of PMD pages to hold in @nid's reservoir
+ *
+ * Cost: Slow path; allocates order-9 pages with GFP_KERNEL | __GFP_RETRY_MAYFAIL,
+ * populates metadata, decrypts/pins if configured, and zeroes 2MB per page.
+ * Locking: Process context (system_wq); sleeps during buddy allocation, CPA
+ * decryption, and cond_resched(). Takes @res->lock briefly to enqueue.
+ * Frequency: Rare (when reservoir drops to <= @max / 2 or on pool creation).
+ */
+static void dma_pmd_reservoir_refill_node(int nid, unsigned int max)
+{
+ gfp_t gfp = GFP_KERNEL | __GFP_THISNODE | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
+ struct dma_pmd_reservoir *res = &dma_pmd_reservoirs[nid];
+
+ while (READ_ONCE(res->nr_pages) < max) {
+ struct dma_pmd_meta *meta;
+ unsigned long flags;
+ struct page *page;
+
+ if (atomic_long_inc_return(&dma_pmd_nr_pages) >
+ READ_ONCE(dma_pmd_max_pages)) {
+ atomic_long_dec(&dma_pmd_nr_pages);
+ break;
+ }
+
+ page = alloc_pages_node(nid, gfp, PMD_ORDER);
+ if (!page) {
+ atomic_long_dec(&dma_pmd_nr_pages);
+ break;
+ }
+
+ if (unlikely(!dma_pmd_meta_ensure_pfn(page_to_pfn(page), true))) {
+ __free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
+ break;
+ }
+
+ meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
+ memset(meta, 0, sizeof(*meta));
+ spin_lock_init(&meta->map_lock);
+
+ if (dma_pmd_page_prepare(page, meta, true)) {
+ __free_pages(page, PMD_ORDER);
+ atomic_long_dec(&dma_pmd_nr_pages);
+ break;
+ }
+
+ if (!(meta->flags & DMA_PMD_DECRYPTED))
+ memset(page_address(page), 0, PMD_SIZE);
+
+ spin_lock_irqsave(&res->lock, flags);
+ list_add(&meta->list, &res->pages);
+ res->nr_pages++;
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ cond_resched();
+ }
+}
+
+static void dma_pmd_reservoir_work_fn(struct work_struct *work)
+{
+ unsigned int max = READ_ONCE(dma_pmd_reservoir_max);
+ int nid;
+
+ if (!max || !dma_pmd_meta_base())
+ return;
+
+ for_each_node_state(nid, N_MEMORY)
+ dma_pmd_reservoir_refill_node(nid, max);
+}
+
+static DECLARE_WORK(dma_pmd_reservoir_work, dma_pmd_reservoir_work_fn);
+
+static void dma_pmd_schedule_reservoir(void)
+{
+ if (READ_ONCE(dma_pmd_reservoir_max) &&
+ !work_pending(&dma_pmd_reservoir_work))
+ schedule_work(&dma_pmd_reservoir_work);
+}
+
+/*
+ * Pop one pre-prepared PMD page from node @nid's reservoir.
+ *
+ * Cost: O(1) list pop under @res->lock.
+ * Locking: Acquires @res->lock (irqsave). Safe from any context (IRQ, softirq,
+ * process). Never sleeps.
+ * Frequency: Only on pool miss (dma_pmd_add_page()).
+ */
+static struct dma_pmd_meta *dma_pmd_reservoir_pop_node(int nid, unsigned int max)
+{
+ struct dma_pmd_reservoir *res = &dma_pmd_reservoirs[nid];
+ struct dma_pmd_meta *meta;
+ unsigned long flags;
+ bool refill = false;
+
+ if (!READ_ONCE(res->nr_pages))
+ return NULL;
+
+ spin_lock_irqsave(&res->lock, flags);
+ meta = list_first_entry_or_null(&res->pages, struct dma_pmd_meta, list);
+ if (meta) {
+ list_del_init(&meta->list);
+ res->nr_pages--;
+ refill = res->nr_pages <= max / 2;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ if (refill)
+ dma_pmd_schedule_reservoir();
+
+ return meta;
+}
+
+/**
+ * dma_pmd_reservoir_get - Obtain a pre-prepared PMD page, preferring @nid
+ * @nid: Preferred NUMA node (or NUMA_NO_NODE for local node)
+ *
+ * Tries @nid's reservoir first, then falls back to other memory nodes and
+ * schedules an async refill if a fallback or empty reservoir is encountered.
+ *
+ * Cost: O(1) on local node hit; O(MAX_NUMNODES) lockless checks on miss.
+ * Locking: Acquires @res->lock (irqsave) on non-empty node. Safe from any
+ * context. Never sleeps.
+ * Frequency: Only on pool miss (dma_pmd_add_page()).
+ *
+ * Return: Metadata pointer of a ready PMD page (not yet compound-split or
+ * bound to a pool), or NULL if all reservoirs are empty.
+ */
+static struct dma_pmd_meta *dma_pmd_reservoir_get(int nid)
+{
+ unsigned int max = READ_ONCE(dma_pmd_reservoir_max);
+ struct dma_pmd_meta *meta;
+ int target_nid, n;
+
+ if (!max)
+ return NULL;
+
+ target_nid = (nid == NUMA_NO_NODE) ? numa_mem_id() : nid;
+ if (target_nid >= 0 && target_nid < MAX_NUMNODES) {
+ meta = dma_pmd_reservoir_pop_node(target_nid, max);
+ if (meta)
+ return meta;
+ }
+
+ for_each_node_state(n, N_MEMORY) {
+ if (n == target_nid)
+ continue;
+ meta = dma_pmd_reservoir_pop_node(n, max);
+ if (meta) {
+ dma_pmd_schedule_reservoir();
+ return meta;
+ }
+ }
+
+ dma_pmd_schedule_reservoir();
+ return NULL;
+}
+
unsigned int dma_pmd_pools_forget_domain(int idx)
{
struct dma_pmd_pool *pool;
@@ -538,6 +748,7 @@ struct dma_pmd_pool *dma_pmd_pool_create(unsigned int order, unsigned int max_id
}
list_add(&pool->node, &dma_pmd_pools);
mutex_unlock(&dma_pmd_pools_lock);
+ dma_pmd_schedule_reservoir();
return pool;
}
@@ -616,10 +827,18 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
*/
bool can_block = (gfp & GFP_KERNEL) == GFP_KERNEL;
unsigned int nr = DMA_PMD_BLOCKS(pool->order);
+ bool from_reservoir = false;
struct dma_pmd_meta *meta;
gfp_t alloc_gfp, base_gfp;
struct page *page;
+ meta = dma_pmd_reservoir_get(nid);
+ if (meta) {
+ page = pfn_to_page(dma_pmd_meta_to_pfn(meta));
+ from_reservoir = true;
+ goto prepare;
+ }
+
/*
* A high-order allocation is expensive when it fails, and under fragmentation
* it fails on every pool miss. For GFP_ATOMIC, back off briefly.
@@ -687,9 +906,10 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
meta = dma_pmd_meta_of_pfn(page_to_pfn(page));
memset(meta, 0, sizeof(*meta));
- meta->pool = pool;
spin_lock_init(&meta->map_lock);
+prepare:
+ meta->pool = pool;
if (dma_pmd_page_prepare(page, meta, can_block)) {
__free_pages(page, PMD_ORDER);
atomic_long_dec(&dma_pmd_nr_pages);
@@ -705,8 +925,8 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
return NULL;
}
- if (zero || (meta->flags & DMA_PMD_DECRYPTED)) {
- if (!(meta->flags & DMA_PMD_DECRYPTED))
+ if (from_reservoir || zero || (meta->flags & DMA_PMD_DECRYPTED)) {
+ if (!from_reservoir && !(meta->flags & DMA_PMD_DECRYPTED))
memset(page_address(page), 0, PMD_SIZE);
bitmap_set(meta->free_bitmap, 0, nr);
meta->nr_free = nr;
@@ -1042,14 +1262,17 @@ static unsigned long dma_pmd_shrink_count(struct shrinker *shrink,
{
struct dma_pmd_pool *pool;
unsigned long nr = 0;
+ int nid;
- if (!mutex_trylock(&dma_pmd_pools_lock))
- return 0;
+ for_each_node_state(nid, N_MEMORY)
+ nr += READ_ONCE(dma_pmd_reservoirs[nid].nr_pages);
- list_for_each_entry(pool, &dma_pmd_pools, node)
- nr += READ_ONCE(pool->num_idle_pages);
+ if (mutex_trylock(&dma_pmd_pools_lock)) {
+ list_for_each_entry(pool, &dma_pmd_pools, node)
+ nr += READ_ONCE(pool->num_idle_pages);
- mutex_unlock(&dma_pmd_pools_lock);
+ mutex_unlock(&dma_pmd_pools_lock);
+ }
return nr << PMD_ORDER;
}
@@ -1062,34 +1285,50 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
unsigned long freed = 0;
LIST_HEAD(release_list);
unsigned long flags;
- int srcu_idx;
+ int srcu_idx, nid;
- if (!mutex_trylock(&dma_pmd_pools_lock))
- return SHRINK_STOP;
+ for_each_node_state(nid, N_MEMORY) {
+ struct dma_pmd_reservoir *res = &dma_pmd_reservoirs[nid];
- srcu_idx = srcu_read_lock(&dma_pmd_srcu);
- list_for_each_entry(pool, &dma_pmd_pools, node) {
- /*
- * Every PMD page on @idle is a candidate, so this detaches one
- * per iteration and stops as soon as the reclaim path has
- * what it asked for: the IRQs-off section is bounded by the
- * work done, not by how many PMD pages the pool holds.
- */
- spin_lock_irqsave(&pool->lock, flags);
- list_for_each_entry_safe(meta, tmp, &pool->idle, list) {
+ if (!READ_ONCE(res->nr_pages))
+ continue;
+ spin_lock_irqsave(&res->lock, flags);
+ list_for_each_entry_safe(meta, tmp, &res->pages, list) {
if (freed >= sc->nr_to_scan)
break;
list_move(&meta->list, &release_list);
- pool->num_idle_pages--;
+ res->nr_pages--;
freed += 1UL << PMD_ORDER;
}
- spin_unlock_irqrestore(&pool->lock, flags);
-
+ spin_unlock_irqrestore(&res->lock, flags);
if (freed >= sc->nr_to_scan)
break;
}
- mutex_unlock(&dma_pmd_pools_lock);
+ srcu_idx = srcu_read_lock(&dma_pmd_srcu);
+ if (freed < sc->nr_to_scan && mutex_trylock(&dma_pmd_pools_lock)) {
+ list_for_each_entry(pool, &dma_pmd_pools, node) {
+ /*
+ * Every PMD page on @idle is a candidate, so this detaches one
+ * per iteration and stops as soon as the reclaim path has
+ * what it asked for: the IRQs-off section is bounded by the
+ * work done, not by how many PMD pages the pool holds.
+ */
+ spin_lock_irqsave(&pool->lock, flags);
+ list_for_each_entry_safe(meta, tmp, &pool->idle, list) {
+ if (freed >= sc->nr_to_scan)
+ break;
+ list_move(&meta->list, &release_list);
+ pool->num_idle_pages--;
+ freed += 1UL << PMD_ORDER;
+ }
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ if (freed >= sc->nr_to_scan)
+ break;
+ }
+ mutex_unlock(&dma_pmd_pools_lock);
+ }
/*
* Retiring is done outside both locks: dma_pmd_release_page() unmaps
@@ -1103,8 +1342,10 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
if (meta->flags & (DMA_PMD_DECRYPTED | DMA_PMD_PINNED)) {
llist_add(&meta->llnode, &dma_pmd_free_list);
dma_pmd_schedule_reclaim();
- } else {
+ } else if (meta->pool) {
dma_pmd_release_page(meta);
+ } else {
+ dma_pmd_free_reservoir_page(meta);
}
}
srcu_read_unlock(&dma_pmd_srcu, srcu_idx);
@@ -1128,6 +1369,12 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
static int __init dma_pmd_init(void)
{
struct shrinker *shrink;
+ int nid;
+
+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
+ spin_lock_init(&dma_pmd_reservoirs[nid].lock);
+ INIT_LIST_HEAD(&dma_pmd_reservoirs[nid].pages);
+ }
/*
* Hard ceiling on memory diverted from the buddy allocator into 2MB
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 17/22] net/gve: Use DMA_PMD memory for RX buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (15 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 16/22] iommu/dma: Add per-NUMA-node PMD page reservoir Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 18/22] net/gve: Use DMA_PMD memory for tx header bounce buffers Luigi Rizzo
` (33 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_rxbuf is enabled, back GQI-RDA RX packet buffers with
a per-queue DMA_PMD page pool (rx->dma_pmd_pool); DQO-RDA uses page_pool,
which checks dev->dma_pmd_rxbuf directly.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve.h | 2 +
drivers/net/ethernet/google/gve/gve_main.c | 5 ++
drivers/net/ethernet/google/gve/gve_rx.c | 61 ++++++++++++++++------
3 files changed, 53 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h
index c280ff35ee771..11330404f681b 100644
--- a/drivers/net/ethernet/google/gve/gve.h
+++ b/drivers/net/ethernet/google/gve/gve.h
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/dmapool.h>
#include <linux/ethtool_netlink.h>
+#include <linux/dma-pmd.h>
#include <linux/netdevice.h>
#include <linux/net_tstamp.h>
#include <linux/pci.h>
@@ -258,6 +259,7 @@ struct gve_rx_ring {
u32 qpl_copy_pool_mask;
u32 qpl_copy_pool_head;
struct gve_rx_slot_page_info *qpl_copy_pool;
+ struct dma_pmd_pool *dma_pmd_pool;
};
/* DQO fields. */
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 9cc343a162712..e5631987b050f 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1055,6 +1055,11 @@ static int gve_queues_mem_alloc(struct gve_priv *priv,
if (err)
goto free_tx;
+ if (rx_alloc_cfg->raw_addressing && priv->pdev->dev.dma_pmd_rxbuf)
+ dev_info(&priv->pdev->dev,
+ "DMA_PMD rx_bufs pool: enabled on %u RX queue(s)\n",
+ rx_alloc_cfg->qcfg_rx->num_queues);
+
return 0;
free_tx:
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 81ea800e66e94..45b5e841efcd3 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -121,6 +121,7 @@ void gve_rx_free_ring_gqi(struct gve_priv *priv, struct gve_rx_ring *rx,
}
gve_rx_unfill_pages(priv, rx, cfg);
+ rx->dma_pmd_pool = dma_pmd_pool_destroy(rx->dma_pmd_pool);
if (rx->data.data_ring) {
bytes = sizeof(*rx->data.data_ring) * slots;
@@ -159,14 +160,29 @@ static void gve_setup_rx_buffer(struct gve_rx_ring *rx,
static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
struct gve_rx_slot_page_info *page_info,
union gve_rx_data_slot *data_slot,
- struct gve_rx_ring *rx)
+ struct gve_rx_ring *rx, gfp_t gfp)
{
- struct page *page;
+ struct page *page = NULL;
dma_addr_t dma;
- int err;
+ int err = 0;
- err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE,
- GFP_ATOMIC);
+ if (rx->dma_pmd_pool) {
+ page = dma_pmd_pool_alloc_node(rx->dma_pmd_pool,
+ gfp | __GFP_NOWARN, priv->numa_node);
+ if (page) {
+ dma = dma_map_page(dev, page, 0, PAGE_SIZE,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev, dma)) {
+ priv->dma_mapping_error++;
+ put_page(page);
+ page = NULL;
+ err = -ENOMEM;
+ }
+ }
+ }
+ if (!page && !err)
+ err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE,
+ gfp);
if (err) {
u64_stats_update_begin(&rx->statss);
rx->rx_buf_alloc_fail++;
@@ -209,7 +225,8 @@ static int gve_rx_prefill_pages(struct gve_rx_ring *rx,
}
err = gve_rx_alloc_buffer(priv, &priv->pdev->dev,
&rx->data.page_info[i],
- &rx->data.data_ring[i], rx);
+ &rx->data.data_ring[i], rx,
+ GFP_KERNEL);
if (err)
goto alloc_err_rda;
}
@@ -319,6 +336,8 @@ int gve_rx_alloc_ring_gqi(struct gve_priv *priv,
err = -ENOMEM;
goto abort_with_copy_pool;
}
+ } else if (hdev->dma_pmd_rxbuf) {
+ rx->dma_pmd_pool = dma_pmd_pool_create(0, 8);
}
filled_pages = gve_rx_prefill_pages(rx, cfg);
@@ -365,6 +384,7 @@ int gve_rx_alloc_ring_gqi(struct gve_priv *priv,
abort_filled:
gve_rx_unfill_pages(priv, rx, cfg);
abort_with_qpl:
+ rx->dma_pmd_pool = dma_pmd_pool_destroy(rx->dma_pmd_pool);
if (!rx->data.raw_addressing) {
gve_free_queue_page_list(priv, rx->data.qpl, qpl_id);
rx->data.qpl = NULL;
@@ -495,13 +515,19 @@ static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *sl
*(slot_addr) ^= offset;
}
-static int gve_rx_can_recycle_buffer(struct gve_rx_slot_page_info *page_info)
+static int gve_rx_can_recycle_buffer(struct gve_rx_ring *rx,
+ struct gve_rx_slot_page_info *page_info)
{
int pagecount = page_count(page_info->page);
/* This page is not being used by any SKBs - reuse */
- if (pagecount == page_info->pagecnt_bias)
+ if (pagecount == page_info->pagecnt_bias) {
+ if (rx && rx->dma_pmd_pool &&
+ !dma_is_pmd_page(page_to_pfn(page_info->page)) &&
+ dma_pmd_pool_has_free(rx->dma_pmd_pool))
+ return 0;
return 1;
+ }
/* This page is still being used by an SKB - we can't reuse */
else if (pagecount > page_info->pagecnt_bias)
return 0;
@@ -545,7 +571,7 @@ static struct sk_buff *gve_rx_copy_to_pool(struct gve_rx_ring *rx,
copy_page_info = &rx->qpl_copy_pool[pool_idx];
if (!copy_page_info->can_flip) {
- int recycle = gve_rx_can_recycle_buffer(copy_page_info);
+ int recycle = gve_rx_can_recycle_buffer(NULL, copy_page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(rx->gve);
@@ -665,7 +691,7 @@ static struct sk_buff *gve_rx_skb(struct gve_priv *priv, struct gve_rx_ring *rx,
u64_stats_update_end(&rx->statss);
}
} else {
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(priv);
@@ -971,7 +997,7 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
* owns half the page it is impossible to tell which half. Either
* the whole page is free or it needs to be replaced.
*/
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (recycle < 0) {
if (!rx->data.raw_addressing)
@@ -983,11 +1009,16 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
union gve_rx_data_slot *data_slot =
&rx->data.data_ring[idx];
struct device *dev = &priv->pdev->dev;
- gve_rx_free_buffer(dev, page_info, data_slot);
- page_info->page = NULL;
+ struct gve_rx_slot_page_info old_info = *page_info;
+ union gve_rx_data_slot old_slot = *data_slot;
+
if (gve_rx_alloc_buffer(priv, dev, page_info,
- data_slot, rx)) {
- break;
+ data_slot, rx, GFP_ATOMIC)) {
+ if (page_count(page_info->page) >
+ page_info->pagecnt_bias)
+ break;
+ } else {
+ gve_rx_free_buffer(dev, &old_info, &old_slot);
}
}
}
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 18/22] net/gve: Use DMA_PMD memory for tx header bounce buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (16 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 17/22] net/gve: Use DMA_PMD memory for RX buffers Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 19/22] net/mlx5e: " Luigi Rizzo
` (32 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
bounce buffer per TX slot/packet out of the DMA_PMD arena in raw-addressing
mode (both GQI and DQO). When skb_headlen(skb) <= MAX_TCP_HEADER, the
linear portion is copied into the pre-mapped bounce buffer instead of
calling dma_map_single()/dma_unmap_single() on every packet.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve.h | 3 ++
drivers/net/ethernet/google/gve/gve_tx.c | 38 ++++++++++++++---
drivers/net/ethernet/google/gve/gve_tx_dqo.c | 45 +++++++++++++++-----
3 files changed, 69 insertions(+), 17 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h
index 11330404f681b..00e438fe05a50 100644
--- a/drivers/net/ethernet/google/gve/gve.h
+++ b/drivers/net/ethernet/google/gve/gve.h
@@ -18,6 +18,7 @@
#include <linux/ptp_clock_kernel.h>
#include <linux/u64_stats_sync.h>
#include <net/page_pool/helpers.h>
+#include <net/tcp.h>
#include <net/xdp.h>
#include "gve_desc.h"
@@ -642,6 +643,8 @@ struct gve_tx_ring {
};
} dqo;
} ____cacheline_aligned;
+ void *tx_hdr_bufs;
+ dma_addr_t tx_hdr_bufs_dma;
struct netdev_queue *netdev_txq;
struct gve_queue_resources *q_resources; /* head and tail pointer idx */
struct device *dev;
diff --git a/drivers/net/ethernet/google/gve/gve_tx.c b/drivers/net/ethernet/google/gve/gve_tx.c
index 79338fd9bf66e..9610b54ef4792 100644
--- a/drivers/net/ethernet/google/gve/gve_tx.c
+++ b/drivers/net/ethernet/google/gve/gve_tx.c
@@ -224,6 +224,11 @@ static void gve_tx_free_ring_gqi(struct gve_priv *priv, struct gve_tx_ring *tx,
u32 slots;
slots = tx->mask + 1;
+ if (tx->tx_hdr_bufs) {
+ dma_free_coherent(hdev, slots * MAX_TCP_HEADER,
+ tx->tx_hdr_bufs, tx->tx_hdr_bufs_dma);
+ tx->tx_hdr_bufs = NULL;
+ }
dma_free_coherent(hdev, sizeof(*tx->q_resources),
tx->q_resources, tx->q_resources_bus);
tx->q_resources = NULL;
@@ -298,6 +303,11 @@ static int gve_tx_alloc_ring_gqi(struct gve_priv *priv,
/* map Tx FIFO */
if (gve_tx_fifo_init(priv, &tx->tx_fifo))
goto abort_with_qpl;
+ } else if (hdev->dma_pmd_tx_hdrs) {
+ tx->tx_hdr_bufs = dma_alloc_coherent(hdev,
+ cfg->ring_size * MAX_TCP_HEADER,
+ &tx->tx_hdr_bufs_dma,
+ GFP_KERNEL);
}
tx->q_resources =
@@ -311,6 +321,12 @@ static int gve_tx_alloc_ring_gqi(struct gve_priv *priv,
return 0;
abort_with_fifo:
+ if (tx->tx_hdr_bufs) {
+ dma_free_coherent(hdev,
+ cfg->ring_size * MAX_TCP_HEADER,
+ tx->tx_hdr_bufs, tx->tx_hdr_bufs_dma);
+ tx->tx_hdr_bufs = NULL;
+ }
if (!tx->raw_addressing)
gve_tx_fifo_release(priv, &tx->tx_fifo);
abort_with_qpl:
@@ -423,6 +439,8 @@ static inline int gve_skb_fifo_bytes_required(struct gve_tx_ring *tx,
#define MAX_TX_DESC_NEEDED (MAX_SKB_FRAGS + 4)
static void gve_tx_unmap_buf(struct device *dev, struct gve_tx_buffer_state *info)
{
+ if (!dma_unmap_len(info, len))
+ return;
if (info->skb) {
dma_unmap_single(dev, dma_unmap_addr(info, dma),
dma_unmap_len(info, len),
@@ -657,13 +675,21 @@ static int gve_tx_add_skb_no_copy(struct gve_priv *priv, struct gve_tx_ring *tx,
info->skb = skb;
- addr = dma_map_single(tx->dev, skb->data, len, DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(tx->dev, addr))) {
- tx->dma_mapping_error++;
- goto drop;
+ if (tx->tx_hdr_bufs && len <= MAX_TCP_HEADER) {
+ u32 hdr_off = idx * MAX_TCP_HEADER;
+
+ memcpy(tx->tx_hdr_bufs + hdr_off, skb->data, len);
+ addr = tx->tx_hdr_bufs_dma + hdr_off;
+ dma_unmap_len_set(info, len, 0);
+ } else {
+ addr = dma_map_single(tx->dev, skb->data, len, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(tx->dev, addr))) {
+ tx->dma_mapping_error++;
+ goto drop;
+ }
+ dma_unmap_len_set(info, len, len);
+ dma_unmap_addr_set(info, dma, addr);
}
- dma_unmap_len_set(info, len, len);
- dma_unmap_addr_set(info, dma, addr);
num_descriptors = 1 + shinfo->nr_frags;
if (hlen < len)
diff --git a/drivers/net/ethernet/google/gve/gve_tx_dqo.c b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
index ad99cbb77e291..ac69793492c9f 100644
--- a/drivers/net/ethernet/google/gve/gve_tx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_tx_dqo.c
@@ -176,8 +176,9 @@ static void gve_unmap_packet(struct device *dev,
return;
/* SKB linear portion is guaranteed to be mapped */
- dma_unmap_single(dev, dma_unmap_addr(pkt, dma[0]),
- dma_unmap_len(pkt, len[0]), DMA_TO_DEVICE);
+ if (dma_unmap_len(pkt, len[0]))
+ dma_unmap_single(dev, dma_unmap_addr(pkt, dma[0]),
+ dma_unmap_len(pkt, len[0]), DMA_TO_DEVICE);
for (i = 1; i < pkt->num_bufs; i++) {
netmem_dma_unmap_page_attrs(dev, dma_unmap_addr(pkt, dma[i]),
dma_unmap_len(pkt, len[i]),
@@ -232,6 +233,13 @@ static void gve_tx_free_ring_dqo(struct gve_priv *priv, struct gve_tx_ring *tx,
size_t bytes;
u32 qpl_id;
+ if (tx->tx_hdr_bufs) {
+ dma_free_coherent(hdev,
+ tx->dqo.num_pending_packets * MAX_TCP_HEADER,
+ tx->tx_hdr_bufs, tx->tx_hdr_bufs_dma);
+ tx->tx_hdr_bufs = NULL;
+ }
+
if (tx->q_resources) {
dma_free_coherent(hdev, sizeof(*tx->q_resources),
tx->q_resources, tx->q_resources_bus);
@@ -399,6 +407,12 @@ static int gve_tx_alloc_ring_dqo(struct gve_priv *priv,
if (gve_tx_qpl_buf_init(tx))
goto err;
+ } else if (hdev->dma_pmd_tx_hdrs) {
+ tx->tx_hdr_bufs =
+ dma_alloc_coherent(hdev,
+ tx->dqo.num_pending_packets * MAX_TCP_HEADER,
+ &tx->tx_hdr_bufs_dma,
+ GFP_KERNEL);
}
return 0;
@@ -712,12 +726,20 @@ static int gve_tx_add_skb_no_copy_dqo(struct gve_tx_ring *tx,
u32 len = skb_headlen(skb);
dma_addr_t addr;
- addr = dma_map_single(tx->dev, skb->data, len, DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(tx->dev, addr)))
- goto err;
+ if (tx->tx_hdr_bufs && len <= MAX_TCP_HEADER) {
+ u32 hdr_off = (u32)completion_tag * MAX_TCP_HEADER;
- dma_unmap_len_set(pkt, len[pkt->num_bufs], len);
- dma_unmap_addr_set(pkt, dma[pkt->num_bufs], addr);
+ memcpy(tx->tx_hdr_bufs + hdr_off, skb->data, len);
+ addr = tx->tx_hdr_bufs_dma + hdr_off;
+ dma_unmap_len_set(pkt, len[pkt->num_bufs], 0);
+ } else {
+ addr = dma_map_single(tx->dev, skb->data, len,
+ DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(tx->dev, addr)))
+ goto err;
+ dma_unmap_len_set(pkt, len[pkt->num_bufs], len);
+ dma_unmap_addr_set(pkt, dma[pkt->num_bufs], addr);
+ }
++pkt->num_bufs;
gve_tx_fill_pkt_desc_dqo(tx, desc_idx, enable_csum, len, addr,
@@ -748,10 +770,11 @@ static int gve_tx_add_skb_no_copy_dqo(struct gve_tx_ring *tx,
err:
for (i = 0; i < pkt->num_bufs; i++) {
if (i == 0) {
- dma_unmap_single(tx->dev,
- dma_unmap_addr(pkt, dma[i]),
- dma_unmap_len(pkt, len[i]),
- DMA_TO_DEVICE);
+ if (dma_unmap_len(pkt, len[i]))
+ dma_unmap_single(tx->dev,
+ dma_unmap_addr(pkt, dma[i]),
+ dma_unmap_len(pkt, len[i]),
+ DMA_TO_DEVICE);
} else {
dma_unmap_page(tx->dev,
dma_unmap_addr(pkt, dma[i]),
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 19/22] net/mlx5e: Use DMA_PMD memory for tx header bounce buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (17 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 18/22] net/gve: Use DMA_PMD memory for tx header bounce buffers Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-14 17:22 ` Dragos Tatulea
2026-10-03 21:22 ` [RFC: DMA_PMD 20/22] net/idpf: " Luigi Rizzo
` (31 subsequent siblings)
50 siblings, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
bounce buffer per TX DMA FIFO slot out of the DMA_PMD arena. When the
linear portion of a transmitted packet is at most MAX_TCP_HEADER bytes,
copy it into the pre-mapped bounce buffer instead of calling
dma_map_single()/dma_unmap_single() on every packet.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 ++
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 14 ++++++++++
.../net/ethernet/mellanox/mlx5/core/en_tx.c | 28 +++++++++++++++----
4 files changed, 42 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index a3945195f454e..d762ad1c1ffaf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -442,6 +442,8 @@ struct mlx5e_txqsq {
struct mlx5e_sq_dma *dma_fifo;
struct mlx5e_skb_fifo skb_fifo;
struct mlx5e_tx_wqe_info *wqe_info;
+ void *tx_hdr_bufs;
+ dma_addr_t tx_hdr_bufs_dma;
} db;
void __iomem *uar_map;
struct netdev_queue *txq;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index f2a8453d8dce0..6de7a2a430d72 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -365,7 +365,9 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)
{
switch (dma->type) {
case MLX5E_DMA_MAP_SINGLE:
- dma_unmap_single(pdev, dma->addr, dma->size, DMA_TO_DEVICE);
+ if (dma->size)
+ dma_unmap_single(pdev, dma->addr, dma->size,
+ DMA_TO_DEVICE);
break;
case MLX5E_DMA_MAP_PAGE:
netmem_dma_unmap_page_attrs(pdev, dma->addr, dma->size,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 4c6060d54fbed..efd766aa43a02 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1666,6 +1666,13 @@ static void mlx5e_free_icosq(struct mlx5e_icosq *sq)
void mlx5e_free_txqsq_db(struct mlx5e_txqsq *sq)
{
+ if (sq->db.tx_hdr_bufs) {
+ int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
+ int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
+
+ dma_free_coherent(sq->pdev, df_sz * MAX_TCP_HEADER,
+ sq->db.tx_hdr_bufs, sq->db.tx_hdr_bufs_dma);
+ }
kvfree(sq->db.wqe_info);
kvfree(sq->db.skb_fifo.fifo);
kvfree(sq->db.dma_fifo);
@@ -1690,6 +1697,13 @@ int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa)
return -ENOMEM;
}
+ if (sq->pdev->dma_pmd_tx_hdrs)
+ sq->db.tx_hdr_bufs =
+ dma_alloc_coherent(sq->pdev,
+ df_sz * MAX_TCP_HEADER,
+ &sq->db.tx_hdr_bufs_dma,
+ GFP_KERNEL | __GFP_NOWARN);
+
sq->dma_fifo_mask = df_sz - 1;
sq->db.skb_fifo.pc = &sq->skb_fifo_pc;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 0b5e600e4a6a9..3b6a7a8b0128e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -177,6 +177,27 @@ mlx5e_tx_get_gso_ihs(struct mlx5e_txqsq *sq, struct sk_buff *skb)
return ihs;
}
+static inline dma_addr_t
+mlx5e_tx_map_hdr(struct mlx5e_txqsq *sq, unsigned char *data, u32 len)
+{
+ dma_addr_t dma_addr;
+
+ if (sq->db.tx_hdr_bufs && len <= MAX_TCP_HEADER) {
+ u32 off = (sq->dma_fifo_pc & sq->dma_fifo_mask) *
+ MAX_TCP_HEADER;
+
+ memcpy(sq->db.tx_hdr_bufs + off, data, len);
+ dma_addr = sq->db.tx_hdr_bufs_dma + off;
+ mlx5e_dma_push_single(sq, dma_addr, 0);
+ return dma_addr;
+ }
+
+ dma_addr = dma_map_single(sq->pdev, data, len, DMA_TO_DEVICE);
+ if (likely(!dma_mapping_error(sq->pdev, dma_addr)))
+ mlx5e_dma_push_single(sq, dma_addr, len);
+ return dma_addr;
+}
+
static inline int
mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
unsigned char *skb_data, u16 headlen,
@@ -187,8 +208,7 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
int i;
if (headlen) {
- dma_addr = dma_map_single(sq->pdev, skb_data, headlen,
- DMA_TO_DEVICE);
+ dma_addr = mlx5e_tx_map_hdr(sq, skb_data, headlen);
if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
goto dma_unmap_wqe_err;
@@ -196,7 +216,6 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
dseg->lkey = sq->mkey_be;
dseg->byte_count = cpu_to_be32(headlen);
- mlx5e_dma_push_single(sq, dma_addr, headlen);
num_dma++;
dseg++;
}
@@ -578,7 +597,7 @@ mlx5e_sq_xmit_mpwqe(struct mlx5e_txqsq *sq, struct sk_buff *skb,
txd.data = skb->data;
txd.len = skb->len;
- txd.dma_addr = dma_map_single(sq->pdev, txd.data, txd.len, DMA_TO_DEVICE);
+ txd.dma_addr = mlx5e_tx_map_hdr(sq, txd.data, txd.len);
if (unlikely(dma_mapping_error(sq->pdev, txd.dma_addr)))
goto err_unmap;
@@ -591,7 +610,6 @@ mlx5e_sq_xmit_mpwqe(struct mlx5e_txqsq *sq, struct sk_buff *skb,
sq->stats->xmit_more += xmit_more;
- mlx5e_dma_push_single(sq, txd.dma_addr, txd.len);
mlx5e_skb_fifo_push(&sq->db.skb_fifo, skb);
mlx5e_tx_mpwqe_add_dseg(sq, &txd);
mlx5e_tx_skb_update_ts_flags(skb);
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 19/22] net/mlx5e: Use DMA_PMD memory for tx header bounce buffers
2026-10-03 21:22 ` [RFC: DMA_PMD 19/22] net/mlx5e: " Luigi Rizzo
@ 2026-10-14 17:22 ` Dragos Tatulea
2026-10-14 17:50 ` Dragos Tatulea
0 siblings, 1 reply; 86+ messages in thread
From: Dragos Tatulea @ 2026-10-14 17:22 UTC (permalink / raw)
To: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 03.10.26 23:22, Luigi Rizzo wrote:
> When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
> bounce buffer per TX DMA FIFO slot out of the DMA_PMD arena. When the
> linear portion of a transmitted packet is at most MAX_TCP_HEADER bytes,
> copy it into the pre-mapped bounce buffer instead of calling
> dma_map_single()/dma_unmap_single() on every packet.
>
What is the difference between using the DMA_PMD pool or one (or a few)
coherent DMA (2MB) buffers allocated by the driver?
> Signed-off-by: Luigi Rizzo <lrizzo@google.com>
> ---
> drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 ++
> .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++-
> .../net/ethernet/mellanox/mlx5/core/en_main.c | 14 ++++++++++
> .../net/ethernet/mellanox/mlx5/core/en_tx.c | 28 +++++++++++++++----
> 4 files changed, 42 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> index a3945195f454e..d762ad1c1ffaf 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> @@ -442,6 +442,8 @@ struct mlx5e_txqsq {
> struct mlx5e_sq_dma *dma_fifo;
> struct mlx5e_skb_fifo skb_fifo;
> struct mlx5e_tx_wqe_info *wqe_info;
> + void *tx_hdr_bufs;
> + dma_addr_t tx_hdr_bufs_dma;
Why are only headers considered for bouncing? Because they have the
highest mapping overhead?
Note that TSO/GSO packets will copy the header inline in the WQE
already. But not for small packets.
> } db;
> void __iomem *uar_map;
> struct netdev_queue *txq;
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> index f2a8453d8dce0..6de7a2a430d72 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> @@ -365,7 +365,9 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)
> {
> switch (dma->type) {
> case MLX5E_DMA_MAP_SINGLE:
> - dma_unmap_single(pdev, dma->addr, dma->size, DMA_TO_DEVICE);
> + if (dma->size)
> + dma_unmap_single(pdev, dma->addr, dma->size,
> + DMA_TO_DEVICE);
> break;
> case MLX5E_DMA_MAP_PAGE:
> netmem_dma_unmap_page_attrs(pdev, dma->addr, dma->size,
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> index 4c6060d54fbed..efd766aa43a02 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> @@ -1666,6 +1666,13 @@ static void mlx5e_free_icosq(struct mlx5e_icosq *sq)
>
> void mlx5e_free_txqsq_db(struct mlx5e_txqsq *sq)
> {
> + if (sq->db.tx_hdr_bufs) {
> + int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
> + int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
> +
> + dma_free_coherent(sq->pdev, df_sz * MAX_TCP_HEADER,
> + sq->db.tx_hdr_bufs, sq->db.tx_hdr_bufs_dma);
> + }
> kvfree(sq->db.wqe_info);
> kvfree(sq->db.skb_fifo.fifo);
> kvfree(sq->db.dma_fifo);
> @@ -1690,6 +1697,13 @@ int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa)
> return -ENOMEM;
> }
>
> + if (sq->pdev->dma_pmd_tx_hdrs)
> + sq->db.tx_hdr_bufs =
> + dma_alloc_coherent(sq->pdev,
> + df_sz * MAX_TCP_HEADER,
> + &sq->db.tx_hdr_bufs_dma,
> + GFP_KERNEL | __GFP_NOWARN);
> +
> sq->dma_fifo_mask = df_sz - 1;
>
> sq->db.skb_fifo.pc = &sq->skb_fifo_pc;
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> index 0b5e600e4a6a9..3b6a7a8b0128e 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> @@ -177,6 +177,27 @@ mlx5e_tx_get_gso_ihs(struct mlx5e_txqsq *sq, struct sk_buff *skb)
> return ihs;
> }
>
> +static inline dma_addr_t
> +mlx5e_tx_map_hdr(struct mlx5e_txqsq *sq, unsigned char *data, u32 len)
> +{
> + dma_addr_t dma_addr;
> +
> + if (sq->db.tx_hdr_bufs && len <= MAX_TCP_HEADER) {
> + u32 off = (sq->dma_fifo_pc & sq->dma_fifo_mask) *
> + MAX_TCP_HEADER;
> +
> + memcpy(sq->db.tx_hdr_bufs + off, data, len);
> + dma_addr = sq->db.tx_hdr_bufs_dma + off;
> + mlx5e_dma_push_single(sq, dma_addr, 0);
> + return dma_addr;
> + }
> +
> + dma_addr = dma_map_single(sq->pdev, data, len, DMA_TO_DEVICE);
> + if (likely(!dma_mapping_error(sq->pdev, dma_addr)))
> + mlx5e_dma_push_single(sq, dma_addr, len);
> + return dma_addr;
> +}
> +
This looks like a form of tx-copybreak.
Thanks,
Dragos
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 19/22] net/mlx5e: Use DMA_PMD memory for tx header bounce buffers
2026-10-14 17:22 ` Dragos Tatulea
@ 2026-10-14 17:50 ` Dragos Tatulea
2026-10-14 18:28 ` Luigi Rizzo
0 siblings, 1 reply; 86+ messages in thread
From: Dragos Tatulea @ 2026-10-14 17:50 UTC (permalink / raw)
To: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 14.10.26 19:22, Dragos Tatulea wrote:
>
>
> On 03.10.26 23:22, Luigi Rizzo wrote:
>> When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
>> bounce buffer per TX DMA FIFO slot out of the DMA_PMD arena. When the
>> linear portion of a transmitted packet is at most MAX_TCP_HEADER bytes,
>> copy it into the pre-mapped bounce buffer instead of calling
>> dma_map_single()/dma_unmap_single() on every packet.
>>
> What is the difference between using the DMA_PMD pool or one (or a few)
> coherent DMA (2MB) buffers allocated by the driver?
>
The patch was actually using the coherent buffer. My bad, sorry.
>> Signed-off-by: Luigi Rizzo <lrizzo@google.com>
>> ---
>> drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 ++
>> .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++-
>> .../net/ethernet/mellanox/mlx5/core/en_main.c | 14 ++++++++++
>> .../net/ethernet/mellanox/mlx5/core/en_tx.c | 28 +++++++++++++++----
>> 4 files changed, 42 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>> index a3945195f454e..d762ad1c1ffaf 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>> @@ -442,6 +442,8 @@ struct mlx5e_txqsq {
>> struct mlx5e_sq_dma *dma_fifo;
>> struct mlx5e_skb_fifo skb_fifo;
>> struct mlx5e_tx_wqe_info *wqe_info;
>> + void *tx_hdr_bufs;
>> + dma_addr_t tx_hdr_bufs_dma;
> Why are only headers considered for bouncing? Because they have the
> highest mapping overhead?
I meant to say headers and small packets.
>
> Note that TSO/GSO packets will copy the header inline in the WQE
> already. But not for small packets.
>
>> } db;
>> void __iomem *uar_map;
>> struct netdev_queue *txq;
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
>> index f2a8453d8dce0..6de7a2a430d72 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
>> @@ -365,7 +365,9 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)
>> {
>> switch (dma->type) {
>> case MLX5E_DMA_MAP_SINGLE:
>> - dma_unmap_single(pdev, dma->addr, dma->size, DMA_TO_DEVICE);
>> + if (dma->size)
>> + dma_unmap_single(pdev, dma->addr, dma->size,
>> + DMA_TO_DEVICE);
>> break;
>> case MLX5E_DMA_MAP_PAGE:
>> netmem_dma_unmap_page_attrs(pdev, dma->addr, dma->size,
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> index 4c6060d54fbed..efd766aa43a02 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> @@ -1666,6 +1666,13 @@ static void mlx5e_free_icosq(struct mlx5e_icosq *sq)
>>
>> void mlx5e_free_txqsq_db(struct mlx5e_txqsq *sq)
>> {
>> + if (sq->db.tx_hdr_bufs) {
>> + int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
>> + int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
>> +
>> + dma_free_coherent(sq->pdev, df_sz * MAX_TCP_HEADER,
>> + sq->db.tx_hdr_bufs, sq->db.tx_hdr_bufs_dma);
>> + }
>> kvfree(sq->db.wqe_info);
>> kvfree(sq->db.skb_fifo.fifo);
>> kvfree(sq->db.dma_fifo);
>> @@ -1690,6 +1697,13 @@ int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa)
>> return -ENOMEM;
>> }
>>
>> + if (sq->pdev->dma_pmd_tx_hdrs)
>> + sq->db.tx_hdr_bufs =
>> + dma_alloc_coherent(sq->pdev,
>> + df_sz * MAX_TCP_HEADER,
>> + &sq->db.tx_hdr_bufs_dma,
>> + GFP_KERNEL | __GFP_NOWARN);
>> +
>> sq->dma_fifo_mask = df_sz - 1;
>>
>> sq->db.skb_fifo.pc = &sq->skb_fifo_pc;
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
>> index 0b5e600e4a6a9..3b6a7a8b0128e 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
>> @@ -177,6 +177,27 @@ mlx5e_tx_get_gso_ihs(struct mlx5e_txqsq *sq, struct sk_buff *skb)
>> return ihs;
>> }
>>
>> +static inline dma_addr_t
>> +mlx5e_tx_map_hdr(struct mlx5e_txqsq *sq, unsigned char *data, u32 len)
>> +{
>> + dma_addr_t dma_addr;
>> +
>> + if (sq->db.tx_hdr_bufs && len <= MAX_TCP_HEADER) {
>> + u32 off = (sq->dma_fifo_pc & sq->dma_fifo_mask) *
>> + MAX_TCP_HEADER;
>> +
>> + memcpy(sq->db.tx_hdr_bufs + off, data, len);
>> + dma_addr = sq->db.tx_hdr_bufs_dma + off;
>> + mlx5e_dma_push_single(sq, dma_addr, 0);
>> + return dma_addr;
>> + }
>> +
>> + dma_addr = dma_map_single(sq->pdev, data, len, DMA_TO_DEVICE);
>> + if (likely(!dma_mapping_error(sq->pdev, dma_addr)))
>> + mlx5e_dma_push_single(sq, dma_addr, len);
>> + return dma_addr;
>> +}
>> +
> This looks like a form of tx-copybreak.
>
> Thanks,
> Dragos
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 19/22] net/mlx5e: Use DMA_PMD memory for tx header bounce buffers
2026-10-14 17:50 ` Dragos Tatulea
@ 2026-10-14 18:28 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-14 18:28 UTC (permalink / raw)
To: Dragos Tatulea
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni, Greg Kroah-Hartman,
Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Saeed Mahameed, Tariq Toukan, Tony Nguyen, Przemek Kitszel,
Alexander Lobakin, Michael Chan, Pavan Chebbi, iommu, netdev,
linux-mm, driver-core, linux-doc, linux-kernel
On Wed, Oct 14, 2026 at 7:51 PM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>
> > On 03.10.26 23:22, Luigi Rizzo wrote:
> > Why are only headers considered for bouncing? Because they have the
> > highest mapping overhead?
>
> I meant to say headers and small packets.
Headers have a high cost for the map/unmap (especially in strict iommu mode),
and also cause significant IOTLB thrashing, because they come from skb_head
that is not pre-mapped. The impact of IOTLB thrashing depends on
the hardware and load but they consume one IOTLB entry per packet with
almost non-existent recycling.
In the following version I have modified the slub and skb allocator to
feed skb head from DMA_PMD memory, which removes the need
for header bounce buffers. It is a compromise (the skb head includes
skb_shinfo, which we may not always want to expose to the device),
but gives an option to those who want it, and in that case drivers
that use pagepool can get all the benefits of DMA_PMD with zero
modifications.
> > Note that TSO/GSO packets will copy the header inline in the WQE
> > already. But not for small packets.
Yes additional bounce buffers are not needed on hardware that are
essentially using the WQE as a bounce buffer. I'll double check the patch
(mechanically generated).
Small packets should not be too problematic, as long as they come from
fragments allocated during the copy_from_user.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* [RFC: DMA_PMD 20/22] net/idpf: Use DMA_PMD memory for tx header bounce buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (18 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 19/22] net/mlx5e: " Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 21/22] net/bnxt: " Luigi Rizzo
` (30 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
bounce buffer per TX buffer slot out of the DMA_PMD arena. When the
linear portion of a transmitted packet is at most MAX_TCP_HEADER bytes,
copy it into the pre-mapped bounce buffer and set the SQE unmap length
to 0 instead of calling dma_map_single()/dma_unmap_page() on every
packet.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
.../ethernet/intel/idpf/idpf_singleq_txrx.c | 6 ++---
drivers/net/ethernet/intel/idpf/idpf_txrx.c | 21 ++++++++++++---
drivers/net/ethernet/intel/idpf/idpf_txrx.h | 27 ++++++++++++++++++-
include/net/libeth/tx.h | 5 ++--
4 files changed, 50 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c b/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c
index e3ddf18dcbf51..24a2dfec6212d 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c
@@ -261,7 +261,7 @@ static void idpf_tx_singleq_map(struct idpf_tx_queue *tx_q,
tx_desc = &tx_q->base_tx[i];
- dma = dma_map_single(tx_q->dev, skb->data, size, DMA_TO_DEVICE);
+ dma = idpf_tx_map_hdr(tx_q, skb, first, size);
/* write each descriptor with CRC bit */
if (idpf_queue_has(CRC_EN, tx_q))
@@ -274,8 +274,7 @@ static void idpf_tx_singleq_map(struct idpf_tx_queue *tx_q,
return idpf_tx_singleq_dma_map_error(tx_q, skb,
first, i);
- /* record length, and DMA address */
- dma_unmap_len_set(tx_buf, len, size);
+ /* record DMA address */
dma_unmap_addr_set(tx_buf, dma, dma);
tx_buf->type = LIBETH_SQE_FRAG;
@@ -326,6 +325,7 @@ static void idpf_tx_singleq_map(struct idpf_tx_queue *tx_q,
size = skb_frag_size(frag);
data_len -= size;
+ dma_unmap_len_set(tx_buf, len, size);
dma = skb_frag_dma_map(tx_q->dev, frag, 0, size,
DMA_TO_DEVICE);
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
index 4311ffa30bb18..4bdec468ea2ce 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
@@ -90,6 +90,14 @@ static void idpf_tx_buf_rel_all(struct idpf_tx_queue *txq)
else
idpf_tx_buf_clean(txq);
+ if (txq->hdr_buf_va) {
+ dma_free_coherent(txq->dev,
+ txq->buf_pool_size * MAX_TCP_HEADER,
+ txq->hdr_buf_va, txq->hdr_buf_dma);
+ txq->hdr_buf_va = NULL;
+ txq->hdr_buf_dma = 0;
+ }
+
kfree(txq->tx_buf);
txq->tx_buf = NULL;
}
@@ -187,6 +195,13 @@ static int idpf_tx_buf_alloc_all(struct idpf_tx_queue *tx_q)
if (!tx_q->tx_buf)
return -ENOMEM;
+ if (tx_q->dev->dma_pmd_tx_hdrs)
+ tx_q->hdr_buf_va =
+ dma_alloc_coherent(tx_q->dev,
+ tx_q->buf_pool_size * MAX_TCP_HEADER,
+ &tx_q->hdr_buf_dma,
+ GFP_KERNEL | __GFP_NOWARN);
+
return 0;
}
@@ -2663,7 +2678,7 @@ static void idpf_tx_splitq_map(struct idpf_tx_queue *tx_q,
tx_desc = &tx_q->flex_tx[i];
- dma = dma_map_single(tx_q->dev, skb->data, size, DMA_TO_DEVICE);
+ dma = idpf_tx_map_hdr(tx_q, skb, first, size);
tx_buf = first;
first->nr_frags = 0;
@@ -2680,8 +2695,7 @@ static void idpf_tx_splitq_map(struct idpf_tx_queue *tx_q,
first->nr_frags++;
tx_buf->type = LIBETH_SQE_FRAG;
- /* record length, and DMA address */
- dma_unmap_len_set(tx_buf, len, size);
+ /* record DMA address */
dma_unmap_addr_set(tx_buf, dma, dma);
/* buf_addr is in same location for both desc types */
@@ -2784,6 +2798,7 @@ static void idpf_tx_splitq_map(struct idpf_tx_queue *tx_q,
size = skb_frag_size(frag);
data_len -= size;
+ dma_unmap_len_set(tx_buf, len, size);
dma = skb_frag_dma_map(tx_q->dev, frag, 0, size,
DMA_TO_DEVICE);
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
index 93547597efd2a..39458a8e5baa0 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
@@ -8,6 +8,7 @@
#include <linux/net/intel/virtchnl2_lan_desc.h>
#include <net/libeth/cache.h>
+#include <net/libeth/tx.h>
#include <net/libeth/types.h>
#include <net/netdev_queues.h>
#include <net/tcp.h>
@@ -639,6 +640,8 @@ libeth_cacheline_set_assert(struct idpf_rx_queue,
* @q_vector: Backreference to associated vector
* @buf_pool_size: Total number of idpf_tx_buf
* @rel_q_id: relative virtchnl queue index
+ * @hdr_buf_va: Base of the TX header bounce buffer array, or NULL
+ * @hdr_buf_dma: DMA address matching @hdr_buf_va
*/
struct idpf_tx_queue {
__cacheline_group_begin_aligned(read_mostly);
@@ -715,6 +718,8 @@ struct idpf_tx_queue {
u32 buf_pool_size;
u32 rel_q_id;
+ void *hdr_buf_va;
+ dma_addr_t hdr_buf_dma;
__cacheline_group_end_aligned(cold);
};
libeth_cacheline_set_assert(struct idpf_tx_queue, 64,
@@ -723,7 +728,7 @@ libeth_cacheline_set_assert(struct idpf_tx_queue, 64,
offsetofend(struct idpf_tx_queue, timer) +
offsetof(struct idpf_tx_queue, q_stats) -
offsetofend(struct idpf_tx_queue, tstamp_task),
- 32);
+ 48);
/**
* struct idpf_buf_queue - software structure representing a buffer queue
@@ -1131,4 +1136,24 @@ int idpf_tso(struct sk_buff *skb, struct idpf_tx_offload_params *off);
void idpf_wait_for_sw_marker_completion(const struct idpf_tx_queue *txq);
+static inline dma_addr_t idpf_tx_map_hdr(struct idpf_tx_queue *tx_q,
+ const struct sk_buff *skb,
+ struct libeth_sqe *first,
+ unsigned int size)
+{
+ u32 buf_idx = first - tx_q->tx_buf;
+
+ if (tx_q->hdr_buf_va && buf_idx < tx_q->buf_pool_size &&
+ size <= MAX_TCP_HEADER) {
+ u32 offset = buf_idx * MAX_TCP_HEADER;
+
+ memcpy(tx_q->hdr_buf_va + offset, skb->data, size);
+ dma_unmap_len_set(first, len, 0);
+ return tx_q->hdr_buf_dma + offset;
+ }
+
+ dma_unmap_len_set(first, len, size);
+ return dma_map_single(tx_q->dev, skb->data, size, DMA_TO_DEVICE);
+}
+
#endif /* !_IDPF_TXRX_H_ */
diff --git a/include/net/libeth/tx.h b/include/net/libeth/tx.h
index c3db5c6f16410..2720ab9393e90 100644
--- a/include/net/libeth/tx.h
+++ b/include/net/libeth/tx.h
@@ -130,8 +130,9 @@ static inline void libeth_tx_complete(struct libeth_sqe *sqe,
case LIBETH_SQE_SKB:
case LIBETH_SQE_FRAG:
case LIBETH_SQE_SLAB:
- dma_unmap_page(cp->dev, dma_unmap_addr(sqe, dma),
- dma_unmap_len(sqe, len), DMA_TO_DEVICE);
+ if (dma_unmap_len(sqe, len))
+ dma_unmap_page(cp->dev, dma_unmap_addr(sqe, dma),
+ dma_unmap_len(sqe, len), DMA_TO_DEVICE);
break;
default:
break;
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 21/22] net/bnxt: Use DMA_PMD memory for tx header bounce buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (19 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 20/22] net/idpf: " Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 22/22] iommu/dma: Add DMA_PMD statistics and debugfs Luigi Rizzo
` (29 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
When dev->dma_pmd_tx_hdrs is enabled, allocate a MAX_TCP_HEADER
bounce buffer per TX ring slot out of the DMA_PMD arena. When the
linear portion of a transmitted packet is at most MAX_TCP_HEADER bytes,
copy it into the pre-mapped bounce buffer instead of calling
dma_map_single()/dma_unmap_single() on every packet.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 45 ++++++++++++++++++-----
drivers/net/ethernet/broadcom/bnxt/bnxt.h | 2 +
2 files changed, 37 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index d7728d0c5b6e6..b9c91eb3aaddd 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -680,13 +680,22 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
length = BNXT_MIN_PKT_SIZE;
}
- mapping = dma_map_single(&pdev->dev, skb->data, len, DMA_TO_DEVICE);
-
- if (unlikely(dma_mapping_error(&pdev->dev, mapping)))
- goto tx_free;
-
- dma_unmap_addr_set(tx_buf, mapping, mapping);
- dma_unmap_len_set(tx_buf, len, len);
+ if (txr->tx_hdr_bufs && len + pad <= MAX_TCP_HEADER) {
+ u32 off = RING_TX(bp, prod) * MAX_TCP_HEADER;
+
+ memcpy(txr->tx_hdr_bufs + off, skb->data, len);
+ if (pad && !last_frag)
+ memset(txr->tx_hdr_bufs + off + len, 0, pad);
+ mapping = txr->tx_hdr_bufs_dma + off;
+ dma_unmap_len_set(tx_buf, len, 0);
+ } else {
+ mapping = dma_map_single(&pdev->dev, skb->data, len,
+ DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(&pdev->dev, mapping)))
+ goto tx_free;
+ dma_unmap_addr_set(tx_buf, mapping, mapping);
+ dma_unmap_len_set(tx_buf, len, len);
+ }
flags = (len << TX_BD_LEN_SHIFT) | TX_BD_TYPE_LONG_TX_BD |
TX_BD_CNT(last_frag + 2);
@@ -800,8 +809,9 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
/* start back at beginning and unmap skb */
prod = txr->tx_prod;
tx_buf = &txr->tx_buf_ring[RING_TX(bp, prod)];
- dma_unmap_single(&pdev->dev, dma_unmap_addr(tx_buf, mapping),
- skb_headlen(skb), DMA_TO_DEVICE);
+ if (dma_unmap_len(tx_buf, len))
+ dma_unmap_single(&pdev->dev, dma_unmap_addr(tx_buf, mapping),
+ skb_headlen(skb), DMA_TO_DEVICE);
prod = NEXT_TX(prod);
/* unmap remaining mapped pages */
@@ -885,7 +895,6 @@ static bool __bnxt_tx_int(struct bnxt *bp, struct bnxt_tx_ring_info *txr,
dma_unmap_single(&pdev->dev, dma_addr, dma_len,
DMA_TO_DEVICE);
}
-
last = tx_buf->nr_frags;
for (j = 0; j < last; j++) {
@@ -4111,6 +4120,15 @@ static void bnxt_free_tx_rings(struct bnxt *bp)
ring = &txr->tx_ring_struct;
+ if (txr->tx_hdr_bufs) {
+ dma_free_coherent(&pdev->dev,
+ ring->ring_mem.nr_pages *
+ TX_DESC_CNT * MAX_TCP_HEADER,
+ txr->tx_hdr_bufs,
+ txr->tx_hdr_bufs_dma);
+ txr->tx_hdr_bufs = NULL;
+ }
+
bnxt_free_ring(bp, &ring->ring_mem);
}
}
@@ -4181,6 +4199,13 @@ static int bnxt_alloc_tx_rings(struct bnxt *bp)
if (rc)
return rc;
}
+ if (pdev->dev.dma_pmd_tx_hdrs && i >= bp->tx_nr_rings_xdp)
+ txr->tx_hdr_bufs =
+ dma_alloc_coherent(&pdev->dev,
+ ring->ring_mem.nr_pages *
+ TX_DESC_CNT * MAX_TCP_HEADER,
+ &txr->tx_hdr_bufs_dma,
+ GFP_KERNEL | __GFP_NOWARN);
qidx = bp->tc_to_qidx[j];
ring->queue_id = bp->q_info[qidx].queue_id;
spin_lock_init(&txr->xdp_tx_lock);
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index c673b2ce4a0d2..b42bbabc0eb98 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -1003,6 +1003,8 @@ struct bnxt_tx_ring_info {
struct tx_push_buffer *tx_push;
dma_addr_t tx_push_mapping;
__le64 data_mapping;
+ void *tx_hdr_bufs;
+ dma_addr_t tx_hdr_bufs_dma;
void *tx_inline_buf;
dma_addr_t tx_inline_dma;
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC: DMA_PMD 22/22] iommu/dma: Add DMA_PMD statistics and debugfs
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (20 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 21/22] net/bnxt: " Luigi Rizzo
@ 2026-10-03 21:22 ` Luigi Rizzo
2026-10-11 9:02 ` [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Pranjal Shrivastava
` (28 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-03 21:22 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel,
Luigi Rizzo
DMA_PMD pools fail silently. dma_pmd_pool_alloc() returns NULL for four
distinct reasons - allocation backoff, the global DMA_PMD page ceiling, an
PMD_ORDER allocation failure, and a split_page_compound() rejection - and
every caller quietly falls back to a plain page, so a pool that has
stopped pooling looks exactly like one that is working. The map path is no
better: a buffer can be mapped individually because the domain has no
usable IOVA window, because the domain has no 2MB page size, or because
the PMD_ORDER page mapping itself failed, and none of that was observable
either. The only symptom is a throughput loss with no counter to point at.
Add a counter for each of those cases plus the number of 2M page mappings
established and dropped, and export them, along with the existing per-pool
statistics, through /sys/kernel/debug/dma_pmd/pools.
The new counters are atomic64_t rather than being folded into the existing
lock-protected statistics because their update sites cannot take
pool->lock: dma_pmd_add_page() runs outside it, and the map slow path
holds p2m->map_lock, so taking pool->lock there would invert the order
against dma_pmd_domain_release().
Also emit a one-shot warning on the first PMD_ORDER allocation failure.
That request carries __GFP_NOWARN, so the page allocator says nothing, and
this is the one failure mode that is worth a line in dmesg rather than a
counter somebody has to know to look at.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-pmd-map.c | 16 +++-
drivers/iommu/dma-pmd-pool.c | 143 ++++++++++++++++++++++++++++++++---
drivers/iommu/dma-pmd-priv.h | 43 ++++++++++-
3 files changed, 186 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/dma-pmd-map.c b/drivers/iommu/dma-pmd-map.c
index 76cb5d01cb1e8..46c6ab5c1b7de 100644
--- a/drivers/iommu/dma-pmd-map.c
+++ b/drivers/iommu/dma-pmd-map.c
@@ -468,8 +468,13 @@ dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
else
ret = dma_pmd_window_assign(dev, domain, win);
- if (ret)
+ if (ret) {
+ if (ret == -EOPNOTSUPP)
+ atomic64_inc(&meta->pool->fallback_nopool);
+ else
+ atomic64_inc(&meta->pool->fallback_nowindow);
return DMA_MAPPING_ERROR;
+ }
win_size = READ_ONCE(win->size);
}
@@ -482,8 +487,10 @@ dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
* device mask.
*/
if (unlikely(iova - win->base + size > win_size ||
- iova + size - 1 > min_not_zero(dma_mask, dev->bus_dma_limit)))
+ iova + size - 1 > min_not_zero(dma_mask, dev->bus_dma_limit))) {
+ atomic64_inc(&meta->pool->fallback_nowindow);
return DMA_MAPPING_ERROR;
+ }
domain_idx = win->domain_idx - DMA_PMD_IDX_FIRST;
if (likely(test_bit(domain_idx, &meta->domains_mapped)))
@@ -510,12 +517,15 @@ dma_addr_t dma_pmd_dma_map_phys(struct device *dev, struct iommu_domain *domain,
*/
smp_mb__before_atomic();
set_bit(domain_idx, &meta->domains_mapped);
+ atomic64_inc(&meta->pool->pmd_map_cnt);
}
}
spin_unlock_irqrestore(&meta->map_lock, flags);
- if (unlikely(ret))
+ if (unlikely(ret)) {
+ atomic64_inc(&meta->pool->fallback_maperr);
return DMA_MAPPING_ERROR;
+ }
return iova;
}
diff --git a/drivers/iommu/dma-pmd-pool.c b/drivers/iommu/dma-pmd-pool.c
index d48c8e19741ab..a85aa550b500c 100644
--- a/drivers/iommu/dma-pmd-pool.c
+++ b/drivers/iommu/dma-pmd-pool.c
@@ -682,16 +682,21 @@ unsigned int dma_pmd_pools_forget_domain(int idx)
mutex_lock(&dma_pmd_pools_lock);
list_for_each_entry(pool, &dma_pmd_pools, node) {
+ unsigned int n = 0;
+
spin_lock_irqsave(&pool->lock, flags);
list_for_each_entry(meta, &pool->partial, list)
- dropped += dma_pmd_forget_domain(meta, idx);
+ n += dma_pmd_forget_domain(meta, idx);
list_for_each_entry(meta, &pool->partial_dirty, list)
- dropped += dma_pmd_forget_domain(meta, idx);
+ n += dma_pmd_forget_domain(meta, idx);
list_for_each_entry(meta, &pool->idle, list)
- dropped += dma_pmd_forget_domain(meta, idx);
+ n += dma_pmd_forget_domain(meta, idx);
list_for_each_entry(meta, &pool->full, list)
- dropped += dma_pmd_forget_domain(meta, idx);
+ n += dma_pmd_forget_domain(meta, idx);
spin_unlock_irqrestore(&pool->lock, flags);
+
+ atomic64_add(n, &pool->domain_forget_cnt);
+ dropped += n;
}
mutex_unlock(&dma_pmd_pools_lock);
@@ -843,8 +848,10 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
* A high-order allocation is expensive when it fails, and under fragmentation
* it fails on every pool miss. For GFP_ATOMIC, back off briefly.
*/
- if (!can_block && time_before(jiffies, READ_ONCE(pool->next_alloc_attempt)))
+ if (!can_block && time_before(jiffies, READ_ONCE(pool->next_alloc_attempt))) {
+ atomic64_inc(&pool->fail_backoff);
return NULL;
+ }
/* Aggregate ceiling across every pool, independent of max_idle_pages. */
if (atomic_long_inc_return(&dma_pmd_nr_pages) > READ_ONCE(dma_pmd_max_pages)) {
@@ -852,6 +859,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
if (!can_block)
WRITE_ONCE(pool->next_alloc_attempt,
jiffies + DIV_ROUND_UP(HZ, 100));
+ atomic64_inc(&pool->fail_budget);
return NULL;
}
@@ -890,6 +898,14 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
if (!can_block)
WRITE_ONCE(pool->next_alloc_attempt,
jiffies + DIV_ROUND_UP(HZ, 100));
+ atomic64_inc(&pool->fail_nomem);
+ /*
+ * __GFP_NOWARN suppresses the page allocator's own report, so
+ * without this the pool can stop pooling entirely and look
+ * exactly like one that is working.
+ */
+ pr_warn_once("dma_pmd: order-%u allocation failed for pool %p; further failures are counted in debugfs only\n",
+ PMD_ORDER, pool);
return NULL;
}
@@ -922,6 +938,7 @@ static struct dma_pmd_meta *dma_pmd_add_page(struct dma_pmd_pool *pool,
__free_pages(page, PMD_ORDER);
atomic_long_dec(&dma_pmd_nr_pages);
}
+ atomic64_inc(&pool->fail_split);
return NULL;
}
@@ -1020,7 +1037,7 @@ static struct dma_pmd_meta *dma_pmd_pick_meta(struct dma_pmd_pool *pool, bool ze
/* Extract up to @want available blocks from @meta under @pool->lock. */
static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
struct dma_pmd_meta *meta,
- unsigned long want,
+ int target_nid, unsigned long want,
struct page **out, bool zero)
{
unsigned long base_pfn = dma_pmd_meta_to_pfn(meta);
@@ -1040,6 +1057,8 @@ static unsigned long dma_pmd_take_blocks(struct dma_pmd_pool *pool,
meta->nr_dirty = 0;
}
dma_pmd_place_meta(pool, meta);
+ if (page_to_nid(pfn_to_page(base_pfn)) != target_nid)
+ pool->numa_mismatch_cnt += used;
return got;
}
@@ -1075,13 +1094,18 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
{
unsigned long allocated = 0, flags;
bool should_scrub = false;
+ int target_nid;
bool zero;
- if (unlikely(!pool || pool->destroyed || !nr_pages ||
- (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE |
- __GFP_ACCOUNT))))
+ if (unlikely(!pool || pool->destroyed || !nr_pages))
+ return 0;
+
+ if (unlikely(gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))) {
+ atomic64_inc(&pool->fail_gfp);
return 0;
+ }
+ target_nid = (nid == NUMA_NO_NODE) ? numa_mem_id() : nid;
zero = want_init_on_alloc(gfp);
spin_lock_irqsave(&pool->lock, flags);
@@ -1093,8 +1117,11 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
/* Fallback to a new allocation */
spin_unlock_irqrestore(&pool->lock, flags);
meta = dma_pmd_add_page(pool, gfp, nid, zero);
- if (!meta)
+ if (!meta) {
+ if (!allocated)
+ atomic64_inc(&pool->block_alloc_fail);
goto out;
+ }
spin_lock_irqsave(&pool->lock, flags);
pool->pmd_alloc_cnt++;
list_add(&meta->list, &pool->partial);
@@ -1102,7 +1129,7 @@ unsigned long dma_pmd_pool_alloc_bulk_node(struct dma_pmd_pool *pool, gfp_t gfp,
should_scrub = true;
}
- allocated += dma_pmd_take_blocks(pool, meta,
+ allocated += dma_pmd_take_blocks(pool, meta, target_nid,
nr_pages - allocated,
&page_array[allocated], zero);
}
@@ -1366,6 +1393,91 @@ static unsigned long dma_pmd_shrink_scan(struct shrinker *shrink,
return freed ?: SHRINK_STOP;
}
+/*
+ * One block of lines per pool, in /sys/kernel/debug/dma_pmd/pools.
+ *
+ * Pools that have seen no activity at all are skipped: two are created per
+ * possible CPU and one per RX queue, so on a large machine the overwhelming
+ * majority are idle and would bury the few that carry traffic. A pool that
+ * tried and failed is never skipped - that is the case this file exists for.
+ *
+ * The u64 statistics are snapshotted under @pool->lock and printed after it is
+ * dropped; the atomic64 ones are read locklessly, so a line is not a
+ * consistent instant. That is fine for what this is used for - watching which
+ * way a counter moves - and keeps the file off the locking hot path.
+ *
+ * The partial/full split is not reported: counting it means walking both lists
+ * with @pool->lock held and interrupts off, which is unbounded in the number of
+ * PMD pages a pool holds. "held" is derived from the got/put counters instead.
+ */
+static int dma_pmd_pools_show(struct seq_file *s, void *unused)
+{
+ unsigned int reservoir_nr = 0;
+ struct dma_pmd_pool *pool;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ reservoir_nr += READ_ONCE(dma_pmd_reservoirs[nid].nr_pages);
+
+ seq_printf(s, "2m_pages %ld/%lu reservoir %u; pools with no activity are omitted\n",
+ atomic_long_read(&dma_pmd_nr_pages), dma_pmd_max_pages, reservoir_nr);
+
+ mutex_lock(&dma_pmd_pools_lock);
+ list_for_each_entry(pool, &dma_pmd_pools, node) {
+ u64 alloc_2m, free_2m, block_alloc, block_free, block_scrub;
+ u64 block_fail, fail_gfp, numa_mismatch;
+ unsigned int num_idle, max_idle;
+ unsigned long flags;
+
+ block_fail = atomic64_read(&pool->block_alloc_fail);
+ fail_gfp = atomic64_read(&pool->fail_gfp);
+
+ spin_lock_irqsave(&pool->lock, flags);
+ alloc_2m = pool->pmd_alloc_cnt;
+ free_2m = pool->pmd_free_cnt;
+ block_alloc = pool->block_alloc_cnt;
+ block_free = pool->block_free_cnt;
+ block_scrub = pool->block_scrub_cnt;
+ numa_mismatch = pool->numa_mismatch_cnt;
+ num_idle = pool->num_idle_pages;
+ max_idle = pool->max_idle_pages;
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ /*
+ * A pool that acquired no PMD page is the one worth looking at,
+ * not the one to hide: it is indistinguishable from a healthy
+ * pool everywhere else, because its callers just fall back.
+ * Omit a pool only when nothing has ever happened to it, which
+ * is the idle-per-CPU case this filter exists for.
+ */
+ if (!alloc_2m && !block_alloc && !block_fail && !fail_gfp)
+ continue;
+
+ seq_printf(s, "pool %p order %u\n", pool, pool->order);
+ seq_printf(s, " 2m_pages got %llu put %llu held %llu idle %u/%u\n",
+ alloc_2m, free_2m, alloc_2m - free_2m, num_idle, max_idle);
+ seq_printf(s, " blocks alloc %llu free %llu scrub %llu numa_mismatch %llu failed %llu gfp %llu\n",
+ block_alloc, block_free, block_scrub, numa_mismatch,
+ block_fail, fail_gfp);
+ seq_printf(s, " no_2m backoff %lld budget %lld nomem %lld split %lld\n",
+ atomic64_read(&pool->fail_backoff),
+ atomic64_read(&pool->fail_budget),
+ atomic64_read(&pool->fail_nomem),
+ atomic64_read(&pool->fail_split));
+ seq_printf(s, " domains mapped %lld forgotten %lld\n",
+ atomic64_read(&pool->pmd_map_cnt),
+ atomic64_read(&pool->domain_forget_cnt));
+ seq_printf(s, " per_buf no_window %lld no_pool %lld map_err %lld\n",
+ atomic64_read(&pool->fallback_nowindow),
+ atomic64_read(&pool->fallback_nopool),
+ atomic64_read(&pool->fallback_maperr));
+ }
+ mutex_unlock(&dma_pmd_pools_lock);
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_pmd_pools);
+
static int __init dma_pmd_init(void)
{
struct shrinker *shrink;
@@ -1377,7 +1489,7 @@ static int __init dma_pmd_init(void)
}
/*
- * Hard ceiling on memory diverted from the buddy allocator into 2MB
+ * Hard ceiling on memory diverted from the buddy allocator into DMA_PMD
* pools, in PMD pages. One eighth of RAM is far above what the per-pool
* watermarks should ever reach; it exists to bound a pathological
* configuration (many queues, many CPUs, all pools at their high
@@ -1403,6 +1515,13 @@ static int __init dma_pmd_init(void)
if (!dma_pmd_wq)
pr_warn("dma_pmd: no reclaim workqueue, falling back to system_wq\n");
+ /*
+ * No error handling and no dentry kept: this is built in and never
+ * unloaded, and the stubs are no-ops when debugfs is disabled.
+ */
+ debugfs_create_file("pools", 0444, debugfs_create_dir("dma_pmd", NULL),
+ NULL, &dma_pmd_pools_fops);
+
shrink = shrinker_alloc(0, "dma_pmd");
if (!shrink) {
pr_warn("dma_pmd: shrinker registration failed\n");
diff --git a/drivers/iommu/dma-pmd-priv.h b/drivers/iommu/dma-pmd-priv.h
index 82a5c869aa216..b9bf48eb7f0f7 100644
--- a/drivers/iommu/dma-pmd-priv.h
+++ b/drivers/iommu/dma-pmd-priv.h
@@ -178,7 +178,8 @@ static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
* struct dma_pmd_pool - Pool managing PMD-backed blocks of a fixed order
*
* @lock: Spinlock protecting @partial, @idle, @full, @num_idle_pages,
- * block bitmaps and the statistics counters
+ * the block bitmaps and the u64 statistics counters. The
+ * atomic64_t counters below are deliberately outside it.
* @order: Block order managed by this pool (<= PMD_ORDER)
* @destroyed: Set when dma_pmd_pool_destroy() has been called
* @partial: List of partially used PMD pages with @nr_free > 0 (clean-only
@@ -196,6 +197,7 @@ static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
* @block_alloc_cnt: Statistics counter of block allocations satisfied
* @block_free_cnt: Statistics counter of block frees recycled into pool
* @block_scrub_cnt: Statistics counter of dirty blocks zeroed by scrubber
+ * @numa_mismatch_cnt: Block allocations satisfied from a non-target NUMA node
* @next_alloc_attempt: Do not attempt a new order-9 allocation before this time.
* Damps repeated high-order GFP_ATOMIC failures under
* fragmentation, which would otherwise be retried on every
@@ -203,6 +205,26 @@ static inline unsigned int dma_pmd_meta_avail(const struct dma_pmd_meta *m)
* @refcount: Reference count; held by pool creator and each active PMD page
* @node: Node in the global dma_pmd_pools list
* @dead_node: Node in dma_pmd_dead_pools once @refcount reaches 0
+ * @fail_backoff: PMD page acquisitions skipped because @next_alloc_attempt was set
+ * @fail_budget: PMD page acquisitions refused by global dma_pmd_max_pages
+ * @fail_nomem: PMD page acquisitions that ran out of memory for the order-9
+ * allocation. Silent otherwise: the request carries __GFP_NOWARN
+ * so the page allocator says nothing.
+ * @fail_split: split_page_compound() rejections
+ * @block_alloc_fail: PMD page acquisitions that failed, making
+ * dma_pmd_pool_alloc() return NULL so the caller had to fall
+ * back to a plain page
+ * @fail_gfp: dma_pmd_pool_alloc() calls declined due to unsupported GFP
+ * flags (__GFP_DMA, __GFP_DMA32, __GFP_THISNODE, __GFP_ACCOUNT)
+ * @pmd_map_cnt: PMD page mappings established across all domains
+ * @fallback_nowindow: Buffers mapped individually because the domain has no
+ * usable IOVA window - no free range large enough, no free
+ * domain index, or a window the device cannot address
+ * @fallback_nopool: Buffers mapped individually because the domain can never
+ * pool: direct isolation, or no PMD page size
+ * @fallback_maperr: Buffers mapped individually because the PMD page mapping
+ * itself failed
+ * @domain_forget_cnt: Cached mappings dropped by dma_pmd_domain_release()
*/
struct dma_pmd_pool {
/* First cacheline: hot fields touched on block alloc and free. */
@@ -222,12 +244,31 @@ struct dma_pmd_pool {
u64 block_alloc_cnt;
u64 block_free_cnt;
u64 block_scrub_cnt;
+ u64 numa_mismatch_cnt;
/* Cold. */
unsigned long next_alloc_attempt;
struct kref refcount;
struct list_head node;
struct llist_node dead_node;
+
+ /*
+ * Statistics updated without @lock. dma_pmd_add_page() runs outside
+ * it, and the map slow path holds @meta->map_lock - taking @lock there
+ * would invert the order against dma_pmd_domain_release(), which
+ * walks @lock then map_lock.
+ */
+ atomic64_t fail_backoff;
+ atomic64_t fail_budget;
+ atomic64_t fail_nomem;
+ atomic64_t fail_split;
+ atomic64_t block_alloc_fail;
+ atomic64_t fail_gfp;
+ atomic64_t pmd_map_cnt;
+ atomic64_t fallback_nowindow;
+ atomic64_t fallback_nopool;
+ atomic64_t fallback_maperr;
+ atomic64_t domain_forget_cnt;
} ____cacheline_aligned;
extern unsigned long dma_pmd_meta_nframes;
--
2.56.0.rc1.315.gc6ed9934b7-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (21 preceding siblings ...)
2026-10-03 21:22 ` [RFC: DMA_PMD 22/22] iommu/dma: Add DMA_PMD statistics and debugfs Luigi Rizzo
@ 2026-10-11 9:02 ` Pranjal Shrivastava
2026-10-12 13:51 ` David Hildenbrand (Arm)
` (27 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Pranjal Shrivastava @ 2026-10-11 9:02 UTC (permalink / raw)
To: Luigi Rizzo
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni, Greg Kroah-Hartman,
Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Saeed Mahameed, Tariq Toukan, Tony Nguyen, Przemek Kitszel,
Alexander Lobakin, Michael Chan, Pavan Chebbi, iommu, netdev,
linux-mm, driver-core, linux-doc, linux-kernel
On Sat, Oct 03, 2026 at 09:22:19PM +0000, Luigi Rizzo wrote:
Hi Luigi,
> Here is a subsystem called DMA_PMD on which I would like feedback on
> architecture, possible enhancements, or kernel components that could be
> reused to avoid duplication.
>
Thanks for posting this series, it solves more than one problems.
I went through the whole series and the following is my take on the
overall design (I might wait before replying to individual patches
hence this might be verbose).
> I think it can be extremely useful for those affected by the HW/SW overhead
> of IOMMU (especially IOTLB thrashing, see [1]), or confidential computing,
> or preemptable VMs.
>
> The (not too exaggerated) pitch line is
>
> DMA_PMD has the performance of identity, guarantees that only IO buffers
> can ever have active IOMMU or IOTLB mappings, removes the bounce buffer
> overhead in confidential computing and preemptable VMs, and integrates
> smoothly with existing kernel APIs.
>
> === ARCHITECTURE
>
> DMA_PMD was initially designed to address IOTLB thrashing (details in [1]),
> but turned out to also resolve nicely the strict IOMMU overhead, and avoid
> the bounce buffer overhead in preemptible VMs and Confidential Computing.
>
I'm thinking if we can re-work the dma pools here to fit the needs of
DMA PMD. Apart from that my main concern is the isolation model.
As implemented:
- dma_unmap_*() of a pooled buffer is a no-op, i.e. unmap doesn't happen
till the page is retired.
- the whole 2M page is mapped IOMMU_READ | IOMMU_WRITE, regardless of
the direction of the buffer that triggered the mapping
- the mapping stays in every domain that ever mapped any block of the
page until the page is retired, which under steady traffic may be
never
- the TX pools are per-CPU and shared by all sockets and devices.
So, a NIC ends up with long-lived RW access to other sockets' data in
the same 2M pages, including loopback payloads and kTLS SW plaintext
(both come from sk_page_frag), and a DMA after unmap no longer faults
but silently hits whoever owns the block next. For pooled memory this
is identity-like isolation and weaker than DMA-FQ, whose exposure is
bounded by the flush timeout. Thus, I'm not fully sure of the claim:
"security guarantees comparable to strict IOMMU".
It also isn't opt-in: dma_is_pmd_phys() is checked for every device in
iommu_dma_map_phys(), thus, any device that is handed a pooled page gets
a window, including in domains where the admin asked for strict mode.
I think DMA_PMD has to be an explicit per-group default-domain policy,
like DMA-FQ: selectable through iommu_groups/N/type and/or a boot
option, off by default, decided when the default domain is set up and
recorded in the cookie and the map path can check it before doing any
pool lookup.
That is also the natural place to refuse untrusted devices. Today the
window path runs after the swiotlb decision in iommu_dma_map_phys(),
which for untrusted devices only bounces buffers that aren't
granule-aligned, so a page-aligned pooled block gives an
external-facing device a permanent RW 2M window over unrelated data.
iommu_get_default_domain_type() already forces strict DMA for such groups
DMA_PMD must not undo that.
I'm thinking If pooled memory is going to be shared across opt-in
devices anyway, would an explicit shared domain for opted-in devices
be cleaner?
It removes the per-domain machinery, but those devices then lose
isolation from each other for all DMA, and the core would need to
support a default domain shared across groups
Additionally, we should handle edge cases where the dma unmap (or it's
TLB flush) fails for some reason but the page is freed to buddy..
> It works by combining several known techniques:
>
> - transparently feed allocators of device memory (skb_page_frag_refill(),
> dma_alloc_attrs(), pagepool, ...) with DMA_PMD pages, i.e. physically
> contiguous PMD_SIZE (2MB) pages mapped via PDE_SIZE IOMMU entries
>
> - heavy recycling of DMA_PMD pages (like pagepool) and lazy IOMMU unmapping
> (like DMA-FQ), BUT:
>
> - like strict IOMMU (DMA), safely release memory back to the kernel only
> after destroying all of its IOMMU mappings and a synchronous IOTLB flush
>
That's one of the strict-mode properties that this series keeps and two
things undermine it today, dma_pmd_unmap_all() ignores the iommu_unmap()
retval after it has already cleared the per-domain bits (for fellow
reviewers this is in patch 7). And window PTEs outlive driver unbind and
DMA ownership trasnfer because dma_pmd_domain_release() only runs when
the cookie is freed, if a device is handed back from VFIO, it'll regain
access to live host pages and patch 13 additionally gives up page_pool's
destroy unmap of in-flight pafes.. The mappings need to go when the last
DMA API user of the domain goes away not only at domain free..
> - on allocation, DMA_PMD pages can be configured to be pinned in the host
> (hence suitable for preemptible VMs) and/or unencrypted (hence suitable
> for Confidential Computing), removing the need for bounce buffers
>
In CoCo guests, dma_pmd.decrypt is swtiched on silently and applied to
every pool. inclusing the per-CPU skb_page_frag_refill pools that serve
every socket. Enabling tx_enable_dma_pmd thenmakes loopback payloads and
kTLS plaintext host readable & writable before encyrption. Similarly
decrypted Rx pages lose the private copythat swiotlb had making it
possible for the host to change packets after the stack has looked at
them.. Can we make it much more explicit to the Guest/ Guest user that
they're opting in to sharing their data with host for performance?
> - there are global and per-device optin /sys/device/.../dma_pmd_*
> and /proc/sys/net/core/tx_enable_dma_pmd
>
Bools with sysfs files on every device having a dma_mask doesn't seem
too nice.. the iommu policy belongs to the group's default domain..
> NIC drivers can typically use dma_pmd with no modifications for rings,
> tx buffers, rx buffers (if they use pagepool), and rx headers. For tx
> headers, almost all drivers need changes (see later patches in the series)
> to implement cheap bounce buffers and avoid individual 4K mappings.
>
> The series has the following main components:
>
> - a sparse array (similar to pageblock_flags) to quickly attach metadata
> to a 2MB page without fiddling with the struct page. Cost is 128B per 2MB
> page used as an IO buffer, totally negligible.
>
IIUC, the entries are 256B and the array is sized from theoretical
phys_addr space rather that from memory: up to 8G of KVA on x86 4-level,
32G on arm64 with 48-bit PA and 512G 52-bit PA, where the chunk bitmap no
longer fits in kmalloc, failing the init. Init also populates metadata
for the entire RAM.. it'd be nice to size it from real memory ranges or
use an xarray or per-section array with lazy population..
> - dma_pmd_pool, a replacement for alloc_pages() that can be instantiated
> per-CPU or per receive queue. It handles the split of PMD_SIZE pages
> into order-N blocks, handles dma_map and unmap, and aggressively recycles
> entries. It is used to feed skb_page_frag_refill(), pagepool, and
> receive buffers for drivers that do not use pagepool.
> See [2] for "WHY NOT PAGEPOOL FOR TX AND EVERYTHING"
>
I am seeing that the recycles page blocks bypass KASAN/KMSAN poisoning,
page owner, alloc tagging and bad_page checks.. which might make a UAF
go undetected. We might have to use it or a similar technique to ensure
the UAF of a polled buffer remains safe in such a case.. IIRC,
kasan_mempool_(un)poison_pages are used for such cache (I'd let the mm
folks chime in on this).
> - dma_pmd_arena, is another allocator backed by DMA_PMD pages and
> is used exclusively as the backend for dma_alloc_attrs().
> Used for longer-lived allocations (descriptor/completion rings,
> rx and tx header buffers)
>
> - glue code to hook DMA_PMD into pagepool [2], skb_page_frag_refill(),
> dma_alloc_attrs(), dma_map/unmap...
>
On the IOMMU side there are also a few structural limits:
- the domain registry has BITS_PER_LONG entries, and a decline is
cached in the cookie forever. Default domains are per IOMMU group,
often one per function (VFs included) on ACS-capable machines, and TX
pages reach every NIC, so a limit of 64 is reachable, and which
devices get to pool would depend on probe order.
- the per-domain window is a multi-TB alloc_iova() carved on the first
map, under a global IRQ-off lock, sized from iomem_resource.end and
from whichever device in the group maps firstF
- I'm wondering it iova - base == PA (plus a fixed 16G offset), is safe?
As devices learn host physical addresses.
Can we consider giving each PMD page a dense slot number when it
enters a pool, and using iova = base + slot * 2M? That keeps the O(1)
map and the range-check unmap, shrinks the window to cap * 2M, works
with narrower masks and stops exposing PAs, and per-domain tracking of
mapped slots would lift the 64-domain limit as well. The window could
then be reserved in iommu_dma_init_domain() through a dma-iommu helper,
instead of handing the iova_domain to dma-pmd-map.c.
Thanks,
Praan
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (22 preceding siblings ...)
2026-10-11 9:02 ` [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Pranjal Shrivastava
@ 2026-10-12 13:51 ` David Hildenbrand (Arm)
2026-10-12 14:45 ` Luigi Rizzo
2026-10-12 14:49 ` Mostafa Saleh
2026-10-14 17:28 ` Dragos Tatulea
` (26 subsequent siblings)
50 siblings, 2 replies; 86+ messages in thread
From: David Hildenbrand (Arm) @ 2026-10-12 13:51 UTC (permalink / raw)
To: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 10/3/26 23:22, Luigi Rizzo wrote:
> Here is a subsystem called DMA_PMD on which I would like feedback on
> architecture, possible enhancements, or kernel components that could be
> reused to avoid duplication.
Here is a feedback to a subsystem called DMA_PMD ;)
Why is basing this on PMDs (and embedding it in the name) the right approach?
On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
possibly also have TLB benefits (cont-pte)?
--
Cheers,
David
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 13:51 ` David Hildenbrand (Arm)
@ 2026-10-12 14:45 ` Luigi Rizzo
2026-10-12 19:43 ` David Hildenbrand (Arm)
2026-10-12 14:49 ` Mostafa Saleh
1 sibling, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-12 14:45 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 10/3/26 23:22, Luigi Rizzo wrote:
> > Here is a subsystem called DMA_PMD on which I would like feedback on
> > architecture, possible enhancements, or kernel components that could be
> > reused to avoid duplication.
>
> Here is a feedback to a subsystem called DMA_PMD ;)
>
> Why is basing this on PMDs (and embedding it in the name) the right approach?
>
> On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
> want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
> possibly also have TLB benefits (cont-pte)?
I had to call it something... an earlier prototype used the name IOMMU_2M,
but I didn't want to suggest that the design was limited to 2M (it is not).
Aside from the name, the _current_ prototype restricts to 2M allocations,
exactly because too large units are expensive and wasteful.
In the code, decoupling the allocation size from the PMD size is trivial.
To enjoy the reduced IOTLB pressure the architecture needs to support
IOMMU leaves with the correct size. Looks like ARM supports a
"Contiguous" hint, but I am not sure how to tell if the hint is honored
(ie the IOTLB actually uses one entry for the full contiguous range)
other than measuring.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 14:45 ` Luigi Rizzo
@ 2026-10-12 19:43 ` David Hildenbrand (Arm)
2026-10-13 10:42 ` Robin Murphy
2026-10-17 17:11 ` Luigi Rizzo
0 siblings, 2 replies; 86+ messages in thread
From: David Hildenbrand (Arm) @ 2026-10-12 19:43 UTC (permalink / raw)
To: Luigi Rizzo
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On 10/12/26 16:45, Luigi Rizzo wrote:
> On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 10/3/26 23:22, Luigi Rizzo wrote:
>>> Here is a subsystem called DMA_PMD on which I would like feedback on
>>> architecture, possible enhancements, or kernel components that could be
>>> reused to avoid duplication.
>>
>> Here is a feedback to a subsystem called DMA_PMD ;)
>>
>> Why is basing this on PMDs (and embedding it in the name) the right approach?
>>
>> On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
>> want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
>> possibly also have TLB benefits (cont-pte)?
>
> I had to call it something... an earlier prototype used the name IOMMU_2M,
> but I didn't want to suggest that the design was limited to 2M (it is not).
>
> Aside from the name, the _current_ prototype restricts to 2M allocations,
> exactly because too large units are expensive and wasteful.
>
> In the code, decoupling the allocation size from the PMD size is trivial.
Yeah, we should do that.
We should probably just find a way to describe that this is a POOL that focuses
on physical contiguity in some configured granularity.
--
Cheers,
David
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 19:43 ` David Hildenbrand (Arm)
@ 2026-10-13 10:42 ` Robin Murphy
2026-10-14 10:47 ` David Hildenbrand (Arm)
2026-10-17 17:11 ` Luigi Rizzo
1 sibling, 1 reply; 86+ messages in thread
From: Robin Murphy @ 2026-10-13 10:42 UTC (permalink / raw)
To: David Hildenbrand (Arm), Luigi Rizzo
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Christoph Hellwig,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 12/10/2026 8:43 pm, David Hildenbrand (Arm) wrote:
> On 10/12/26 16:45, Luigi Rizzo wrote:
>> On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
>> <david@kernel.org> wrote:
>>>
>>> On 10/3/26 23:22, Luigi Rizzo wrote:
>>>> Here is a subsystem called DMA_PMD on which I would like feedback on
>>>> architecture, possible enhancements, or kernel components that could be
>>>> reused to avoid duplication.
>>>
>>> Here is a feedback to a subsystem called DMA_PMD ;)
>>>
>>> Why is basing this on PMDs (and embedding it in the name) the right approach?
>>>
>>> On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
>>> want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
>>> possibly also have TLB benefits (cont-pte)?
>>
>> I had to call it something... an earlier prototype used the name IOMMU_2M,
>> but I didn't want to suggest that the design was limited to 2M (it is not).
>>
>> Aside from the name, the _current_ prototype restricts to 2M allocations,
>> exactly because too large units are expensive and wasteful.
>>
>> In the code, decoupling the allocation size from the PMD size is trivial.
>
> Yeah, we should do that.
>
> We should probably just find a way to describe that this is a POOL that focuses
> on physical contiguity in some configured granularity.
And since it's intended for DMA we could call it dma_pool and oh look it
already exists...
Sure the current implementation is capped at PAGE_SIZE for allocation
(and thus mapping too), but I don't see why that couldn't just be made
more flexible.
Thanks,
Robin.
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-13 10:42 ` Robin Murphy
@ 2026-10-14 10:47 ` David Hildenbrand (Arm)
2026-10-14 11:05 ` Luigi Rizzo
0 siblings, 1 reply; 86+ messages in thread
From: David Hildenbrand (Arm) @ 2026-10-14 10:47 UTC (permalink / raw)
To: Robin Murphy, Luigi Rizzo
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Christoph Hellwig,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On 10/13/26 12:42, Robin Murphy wrote:
> On 12/10/2026 8:43 pm, David Hildenbrand (Arm) wrote:
>> On 10/12/26 16:45, Luigi Rizzo wrote:
>>> On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
>>> <david@kernel.org> wrote:
>>>
>>> I had to call it something... an earlier prototype used the name IOMMU_2M,
>>> but I didn't want to suggest that the design was limited to 2M (it is not).
>>>
>>> Aside from the name, the _current_ prototype restricts to 2M allocations,
>>> exactly because too large units are expensive and wasteful.
>>>
>>> In the code, decoupling the allocation size from the PMD size is trivial.
>>
>> Yeah, we should do that.
>>
>> We should probably just find a way to describe that this is a POOL that focuses
>> on physical contiguity in some configured granularity.
> And since it's intended for DMA we could call it dma_pool and oh look it already
> exists...
Heh, I did have "DMA_POOL" in mind when writing my reply.
>
> Sure the current implementation is capped at PAGE_SIZE for allocation (and thus
> mapping too), but I don't see why that couldn't just be made more flexible.
That intuitively sounds like the right direction.
--
Cheers,
David
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-14 10:47 ` David Hildenbrand (Arm)
@ 2026-10-14 11:05 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-14 11:05 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: Robin Murphy, Luigi Rizzo, Joerg Roedel, Will Deacon,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Wed, Oct 14, 2026 at 12:48 PM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 10/13/26 12:42, Robin Murphy wrote:
> > On 12/10/2026 8:43 pm, David Hildenbrand (Arm) wrote:
> >> On 10/12/26 16:45, Luigi Rizzo wrote:
> >>> On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
> >>> <david@kernel.org> wrote:
> >>>
> >>> I had to call it something... an earlier prototype used the name IOMMU_2M,
> >>> but I didn't want to suggest that the design was limited to 2M (it is not).
> >>>
> >>> Aside from the name, the _current_ prototype restricts to 2M allocations,
> >>> exactly because too large units are expensive and wasteful.
> >>>
> >>> In the code, decoupling the allocation size from the PMD size is trivial.
> >>
> >> Yeah, we should do that.
> >>
> >> We should probably just find a way to describe that this is a POOL that focuses
> >> on physical contiguity in some configured granularity.
> > And since it's intended for DMA we could call it dma_pool and oh look it already
> > exists...
>
> Heh, I did have "DMA_POOL" in mind when writing my reply.
which confirms that naming is one of the hardest problems in CS...
Let's try to figure out a good name before renaming everything.
> > Sure the current implementation is capped at PAGE_SIZE for allocation (and thus
> > mapping too), but I don't see why that couldn't just be made more flexible.
>
> That intuitively sounds like the right direction.
Point taken, I will decouple the allocaiton size from PMD_SIZE this in
the next iteration.
I have also followed up on most of Pranjal's feedback (stricter error checking,
compact IOVA ranges, opt-in per destination on the tx path...) .
I am also looking at reducing some inefficiencies (e.g. pagepool already caches,
and allocations of new pages are relatively rare, so there is not much
of a point
of having a dma_pmd_pool per queue; and on the tx side, something similar to the
percpu page cache to reduce locking costs when a page is allocated on one CPU
and freed on a different one)
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 19:43 ` David Hildenbrand (Arm)
2026-10-13 10:42 ` Robin Murphy
@ 2026-10-17 17:11 ` Luigi Rizzo
2026-10-22 20:12 ` Jason Gunthorpe
1 sibling, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-17 17:11 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Mon, Oct 12, 2026 at 9:44 PM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 10/12/26 16:45, Luigi Rizzo wrote:
> > On Mon, Oct 12, 2026 at 3:51 PM David Hildenbrand (Arm)
> > <david@kernel.org> wrote:
> >>
> >> On 10/3/26 23:22, Luigi Rizzo wrote:
> >>> Here is a subsystem called DMA_PMD on which I would like feedback on
> >>> architecture, possible enhancements, or kernel components that could be
> >>> reused to avoid duplication.
> >>
> >> Here is a feedback to a subsystem called DMA_PMD ;)
> >>
> >> Why is basing this on PMDs (and embedding it in the name) the right approach?
> >>
> >> On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
> >> want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
> >> possibly also have TLB benefits (cont-pte)?
> >
> > I had to call it something... an earlier prototype used the name IOMMU_2M,
> > but I didn't want to suggest that the design was limited to 2M (it is not).
> >
> > Aside from the name, the _current_ prototype restricts to 2M allocations,
> > exactly because too large units are expensive and wasteful.
> >
> > In the code, decoupling the allocation size from the PMD size is trivial.
>
> Yeah, we should do that.
>
> We should probably just find a way to describe that this is a POOL that focuses
> on physical contiguity in some configured granularity.
Before embarking in a global renaming, I would like to get opinions on
acceptable alternative names (possibly with a short prefix usable for
identifiers in the code). I could think of the following
DMA_PMAP (for PreMapped or Persistent Mapped)
or, ahemm...
DMA_PISA (Pooled IOMMU Superpage Allocator)
both using dmap_ for internal name prefixes
Other suggestions ?
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-17 17:11 ` Luigi Rizzo
@ 2026-10-22 20:12 ` Jason Gunthorpe
2026-10-22 21:51 ` Luigi Rizzo
0 siblings, 1 reply; 86+ messages in thread
From: Jason Gunthorpe @ 2026-10-22 20:12 UTC (permalink / raw)
To: Luigi Rizzo
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Sat, Oct 17, 2026 at 07:11:28PM +0200, Luigi Rizzo wrote:
> > We should probably just find a way to describe that this is a POOL that focuses
> > on physical contiguity in some configured granularity.
>
> Before embarking in a global renaming, I would like to get opinions on
> acceptable alternative names (possibly with a short prefix usable for
> identifiers in the code). I could think of the following
>
> DMA_PMAP (for PreMapped or Persistent Mapped)
> or, ahemm...
> DMA_PISA (Pooled IOMMU Superpage Allocator)
> both using dmap_ for internal name prefixes
Can someone explain what outside netdev would use this? And why you
couldn't do this in the netdev page pool?
It is not a common workload to have the DMA allocator be the one that
allocates the memory. Usually we are DMA'ing from memory that was
allocated someplace else.
I think Robin is right, this is just a special case of the dma pool,
and it certainly shouldn't be something as grandiose as a subsystem..
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-22 20:12 ` Jason Gunthorpe
@ 2026-10-22 21:51 ` Luigi Rizzo
2026-10-23 14:01 ` Jason Gunthorpe
0 siblings, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-22 21:51 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Thu, Oct 22, 2026 at 10:12 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Sat, Oct 17, 2026 at 07:11:28PM +0200, Luigi Rizzo wrote:
> > > We should probably just find a way to describe that this is a POOL that focuses
> > > on physical contiguity in some configured granularity.
> >
> > Before embarking in a global renaming, I would like to get opinions on
> > acceptable alternative names (possibly with a short prefix usable for
> > identifiers in the code). I could think of the following
> >
> > DMA_PMAP (for PreMapped or Persistent Mapped)
> > or, ahemm...
> > DMA_PISA (Pooled IOMMU Superpage Allocator)
> > both using dmap_ for internal name prefixes
>
> Can someone explain what outside netdev would use this? And why you
> couldn't do this in the netdev page pool?
I can try.
Regarding users, the next target is disks (not included in this series)
Regarding why not just pagepool, see below.
FWIW when I started this (after the previous work on swiotlb copy elision
https://lore.kernel.org/all/20260824152932.1583506-1-lrizzo@google.com/ )
I thought this would take only some simple tweaks to pagepool and
skb_page_frag_refill, a couple of days with some agents tops.
But the agents did point out that there were a ton of corner
cases to handle (device detach, reconfiguration, memory
reclaiming, reducing locking contention...).
As a reminder my goals are the following:
- reduce DMA map/unmap costs in strict iommu mode without the DMA-FQ
compromise on security
- address the IOTLB thrashing problem when using 4K mappings
(very serious even with DMA-FQ)
- help reduce bounce buffer usage in confidential computing and
overcommitted VMs.
With that in mind:
> It is not a common workload to have the DMA allocator be the one that
> allocates the memory. Usually we are DMA'ing from memory that was
> allocated someplace else.
Aside from the fact that dma_alloc_attr()/dma_alloc_coherent()
does memory allocation + iova allocation/mapping,
this is not what happens here.
Maybe the confusion comes from one function name
(dma_map_alloc_pages()), but it doesn't map any thing.
My code (in this version, and more in the upcoming release)
mainly implements a specialized memory allocator with an
API equivalent to alloc_pages() and variants, but ensures that
the order-N blocks come from contiguous 2MB blocks reserved on the fly.
The physical addresses of these blocks are recognized by the
"dma allocator" (I suppose you meant the entity that allocates the IOVA range)
so that it can apply specialized IOMMU mappings backed by 2MB IOMMU leaves.
And finally, it hooks into existing kernel subsystem (pagepool,
the skb transmit buffer allocation, dma_alloc_*(), dma_map_*()
so that devices can opt-in to use it without any driver code change.
Regarding pagepool:
For the receive path, I do use my allocator to feed pagepool
(in the next version I have wrapped it into a proper memory provider
interface. Non-pagepool NICs can either use a few trivial changes
to call dma_map_alloc_pages() when filling their rings,
or convert to pagepool (slightly bigger change; anyways
there are options).
Note that just feeding the pagepool with 2MB pages is not efficient because
the fragmentation in pagepool is done with a bump
allocator, so recycling blocks from a 2MB page is difficult
(it has to wait until all 512 blocks to become idle). My code
has a bitmap to track usage, so it naturally recycles at the block level.
Then there is the transmit path.
For skb heads, one simple solution is to use tx head bounce buffers
some of the patches in the series show that option (the next version
implements a proper tx-copybreak).
Otherwise we need to populate skb heads with pre-mapped
memory. In the next version I have a simple modification to
the slub code to accept an opt-in alternative to alloc_pages().
I considered using the pagepool, but the changes would be
similar and the layering seemed really odd, with a cache on top
of a cache.
For transmit, there is a bit of impedance mismatch as well:
skb_page_frag_refill() does not know which device
we want to use (in fact, it is a per-task so there may be
many netdevs used by a task). We could tweak the pagepool
to be device-agnostic, but that would be a bit more
intrusive (more architecturally than in terms of code);
also we need to make sure dma_map() on those pages does the right thing
without trying to map/unmap the individual 4K fragments,
and we need to make sure the skb_free() etc does
not leak those pages into free_pages().
>
> I think Robin is right, this is just a special case of the dma pool,
> and it certainly shouldn't be something as grandiose as a subsystem..
>
dma_pool() could be tweaked to allocate 2MB and return order-N fragments,
so maybe that could be a provider for pagepool, but it has some caveats:
- when fragmenting, the freelist is constructed in-band, ie using
memory exposed to the IO device. That is really a no-go if we use the IOMMU
because we don't trust the device, so we would also need to
allocate extra memory for the freelist (no big deal, but
that is basically what my dma_pmd_meta supports)
- memory is never released until the pool is destroyed,
so there is a problem of stranded idle memory.
Trying to use dma_pool for the transmit path has all the problems
listed above: lack of device, inefficient release of memory,
coarse grained locking...
In summary: of course we could tweak existing subsystems, trying
to reuse them in ways that were probably not anticipated
(see the above partial list of missing features).
I think that once one addresses the functional and performance
problems, the changes would become quite intrusive and the risk
of penalizing the currently supported code path might become significant.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-22 21:51 ` Luigi Rizzo
@ 2026-10-23 14:01 ` Jason Gunthorpe
2026-10-23 14:30 ` Luigi Rizzo
0 siblings, 1 reply; 86+ messages in thread
From: Jason Gunthorpe @ 2026-10-23 14:01 UTC (permalink / raw)
To: Luigi Rizzo
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Thu, Oct 22, 2026 at 11:51:40PM +0200, Luigi Rizzo wrote:
> On Thu, Oct 22, 2026 at 10:12 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Sat, Oct 17, 2026 at 07:11:28PM +0200, Luigi Rizzo wrote:
> > > > We should probably just find a way to describe that this is a POOL that focuses
> > > > on physical contiguity in some configured granularity.
> > >
> > > Before embarking in a global renaming, I would like to get opinions on
> > > acceptable alternative names (possibly with a short prefix usable for
> > > identifiers in the code). I could think of the following
> > >
> > > DMA_PMAP (for PreMapped or Persistent Mapped)
> > > or, ahemm...
> > > DMA_PISA (Pooled IOMMU Superpage Allocator)
> > > both using dmap_ for internal name prefixes
> >
> > Can someone explain what outside netdev would use this? And why you
> > couldn't do this in the netdev page pool?
>
> I can try.
>
> Regarding users, the next target is disks (not included in this series)
Disks don't use a memory pool, they do IO to folios allocated by the
page cache.
The only other candiates I know of besides page pool would be the
existing dma pool users. I don't think they are so performance
sensitive as netdev.
IOW I think this entire series is only useful to netdev.
> > It is not a common workload to have the DMA allocator be the one that
> > allocates the memory. Usually we are DMA'ing from memory that was
> > allocated someplace else.
>
> Aside from the fact that dma_alloc_attr()/dma_alloc_coherent()
> does memory allocation + iova allocation/mapping,
> this is not what happens here.
> Maybe the confusion comes from one function name
> (dma_map_alloc_pages()), but it doesn't map any thing.
> My code (in this version, and more in the upcoming release)
> mainly implements a specialized memory allocator with an
> API equivalent to alloc_pages() and variants, but ensures that
> the order-N blocks come from contiguous 2MB blocks reserved on the fly.
>
> The physical addresses of these blocks are recognized by the
> "dma allocator" (I suppose you meant the entity that allocates the IOVA range)
> so that it can apply specialized IOMMU mappings backed by 2MB IOMMU leaves.
Why? This seems completely divergent from how anything else works in
DMA mapping, and I don't see a reason for it to be anything like
this. Other than it makes page pool work a bit easier
Frankly this looks like you had AI to transpose a bunch netdev page
pool into iommu.
> Note that just feeding the pagepool with 2MB pages is not efficient because
> the fragmentation in pagepool is done with a bump
> allocator, so recycling blocks from a 2MB page is difficult
> (it has to wait until all 512 blocks to become idle). My code
> has a bitmap to track usage, so it naturally recycles at the block level.
It doesn't seem like this little issue should motivate all this
> Then there is the transmit path.
>
> For skb heads, one simple solution is to use tx head bounce buffers
> some of the patches in the series show that option (the next version
> implements a proper tx-copybreak).
> Otherwise we need to populate skb heads with pre-mapped
> memory. In the next version I have a simple modification to
> the slub code to accept an opt-in alternative to alloc_pages().
> I considered using the pagepool, but the changes would be
> similar and the layering seemed really odd, with a cache on top
> of a cache.
With AI there is no limit to the layering violations one can make :(
> For transmit, there is a bit of impedance mismatch as well:
> skb_page_frag_refill() does not know which device
> we want to use (in fact, it is a per-task so there may be
> many netdevs used by a task). We could tweak the pagepool
> to be device-agnostic, but that would be a bit more
> intrusive (more architecturally than in terms of code);
> also we need to make sure dma_map() on those pages does the right thing
> without trying to map/unmap the individual 4K fragments,
> and we need to make sure the skb_free() etc does
> not leak those pages into free_pages().
This is all a page pool problem, has nothing to do with iommu or the
common dma layer.
You can already get optimally mapped 2M IOVA and memory out of the
APIs we have. I can see no justification for the intrustive mess under
this beyond it makes netdev page pool easier to hack on top.
Every thing you listed boils down to 'netdev page pool work the way I
want'
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 14:01 ` Jason Gunthorpe
@ 2026-10-23 14:30 ` Luigi Rizzo
2026-10-23 14:50 ` Jason Gunthorpe
0 siblings, 1 reply; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-23 14:30 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Fri, Oct 23, 2026 at 4:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Thu, Oct 22, 2026 at 11:51:40PM +0200, Luigi Rizzo wrote:
> > On Thu, Oct 22, 2026 at 10:12 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
...
> > > Can someone explain what outside netdev would use this? And why you
> > > couldn't do this in the netdev page pool?
> >
> > I can try.
> >
> > Regarding users, the next target is disks (not included in this series)
>
> Disks don't use a memory pool, they do IO to folios allocated by the
> page cache.
I know. As I said I have a separate patch to cover those IO buffers
block/bio.c | 21 +++++
block/blk-mq-dma.c | 4 +-
drivers/iommu/dma-pmd-node.c | 3 +-
drivers/iommu/dma-pmd-pool.c | 14 ++-
drivers/iommu/dma-pmd-priv.h | 7 +-
drivers/iommu/dma-pmd-user.c | 390
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
include/linux/dma-pmd.h | 53 +++++++++++
mm/filemap.c | 27 +++++-
mm/page-writeback.c | 13 +++
mm/readahead.c | 9 +-
Disk I/O is still potentially affected by the same problems
(cost if io mapping/unmapping, iotlb thrashing, bounce buffers
in confidential computing...)
>
> The only other candiates I know of besides page pool would be the
> existing dma pool users. I don't think they are so performance
> sensitive as netdev.
>
> IOW I think this entire series is only useful to netdev.
Well, the series provide mechanisms to address disk I/O as well.
>
> > > It is not a common workload to have the DMA allocator be the one that
> > > allocates the memory. Usually we are DMA'ing from memory that was
> > > allocated someplace else.
> >
> > Aside from the fact that dma_alloc_attr()/dma_alloc_coherent()
> > does memory allocation + iova allocation/mapping,
> > this is not what happens here.
> > Maybe the confusion comes from one function name
> > (dma_map_alloc_pages()), but it doesn't map any thing.
>
> > My code (in this version, and more in the upcoming release)
> > mainly implements a specialized memory allocator with an
> > API equivalent to alloc_pages() and variants, but ensures that
> > the order-N blocks come from contiguous 2MB blocks reserved on the fly.
> >
> > The physical addresses of these blocks are recognized by the
> > "dma allocator" (I suppose you meant the entity that allocates the IOVA range)
> > so that it can apply specialized IOMMU mappings backed by 2MB IOMMU leaves.
>
> Why? This seems completely divergent from how anything else works in
> DMA mapping, and I don't see a reason for it to be anything like
> this. Other than it makes page pool work a bit easier
>
> Frankly this looks like you had AI to transpose a bunch netdev page
> pool into iommu.
(yes, AI assisted, heavily human-guided).
Frankly, it's very orthogonal to pagepool, given that it uses pagepool
for what it does well, and it provides functionality that pagepool
does not have.
Anyways, reading your next comments, are you suggesting that all
these functionalities should be implemented as pagepool extensions?
(2MB mappings and iommu backing, proper allocator of fragments out of larger
pages, device-less pagepools, multi-domain mappings, pagepool
as memory provider for skb heads, pagepool as memory provider for descriptor
rings ?)
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 14:30 ` Luigi Rizzo
@ 2026-10-23 14:50 ` Jason Gunthorpe
2026-10-23 15:12 ` Luigi Rizzo
` (2 more replies)
0 siblings, 3 replies; 86+ messages in thread
From: Jason Gunthorpe @ 2026-10-23 14:50 UTC (permalink / raw)
To: Luigi Rizzo
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Fri, Oct 23, 2026 at 04:30:25PM +0200, Luigi Rizzo wrote:
> On Fri, Oct 23, 2026 at 4:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Thu, Oct 22, 2026 at 11:51:40PM +0200, Luigi Rizzo wrote:
> > > On Thu, Oct 22, 2026 at 10:12 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> ...
> > > > Can someone explain what outside netdev would use this? And why you
> > > > couldn't do this in the netdev page pool?
> > >
> > > I can try.
> > >
> > > Regarding users, the next target is disks (not included in this series)
> >
> > Disks don't use a memory pool, they do IO to folios allocated by the
> > page cache.
>
> I know. As I said I have a separate patch to cover those IO buffers
>
> block/bio.c | 21 +++++
> block/blk-mq-dma.c | 4 +-
> drivers/iommu/dma-pmd-node.c | 3 +-
> drivers/iommu/dma-pmd-pool.c | 14 ++-
> drivers/iommu/dma-pmd-priv.h | 7 +-
> drivers/iommu/dma-pmd-user.c | 390
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
> include/linux/dma-pmd.h | 53 +++++++++++
> mm/filemap.c | 27 +++++-
> mm/page-writeback.c | 13 +++
> mm/readahead.c | 9 +-
>
>
> Disk I/O is still potentially affected by the same problems
> (cost if io mapping/unmapping, iotlb thrashing, bounce buffers
> in confidential computing...)
The disk folks already have a proposal to address all of this using
io_uring and preallocated buffers. I really doubt yet another version
is going to be interesting there.
> Anyways, reading your next comments, are you suggesting that all
> these functionalities should be implemented as pagepool extensions?
> (2MB mappings and iommu backing, proper allocator of fragments out of larger
> pages, device-less pagepools, multi-domain mappings, pagepool
> as memory provider for skb heads, pagepool as memory provider for descriptor
> rings ?)
netdev shouldn't be touching iommu, it can get a 2M page and map it
using the normal DMA API. AFAIK there is not such a big reason to need
any of these layering violations. They just make it easier on page
pool..
"multi-domain mappings" is that just for page pool because it can't
easially keep track of IOVAs for all the devices mapping the same PMD?
I admit hacking in a special IOVA range so every netdev has the same
IOVA is certainly a creative way to ignore all layering and solve
it. :\
But you could just figure out a way to keep track of the per-device
per-PMD iova mapping..
That said, if you came with some much narrower series to request some
kind of 'globally shared iova' during dma allocation that might be
something dma-api folks could agree with.. It seems like the only
actual DMA API request here anyhow?
That's a bit tricky since it is impossible as a general concept, but
maybe you can come up with some way to narrow it..
I'm still not convinced this is even right for netdev, especially
compared to what Dragos has been working on. The CC stuff seems
actively dangerous. Unconditionally always using CC shared memory for
skbs is completely security wrong. What I have traffic flowing inside
my VM between containers or something? Does this end up putting those
skbs in shared memory even if they never hit a physical NIC? Same
with a TDISP capable NIC, does this unconditionally degrade its
security?
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 14:50 ` Jason Gunthorpe
@ 2026-10-23 15:12 ` Luigi Rizzo
2026-10-23 17:27 ` Jakub Kicinski
2026-11-03 8:09 ` Christoph Hellwig
2 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-23 15:12 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Fri, Oct 23, 2026 at 4:50 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Oct 23, 2026 at 04:30:25PM +0200, Luigi Rizzo wrote:
Just addressing the last part:
> ...The CC stuff seems
> actively dangerous. Unconditionally always using CC shared memory for
> skbs is completely security wrong. What I have traffic flowing inside
> my VM between containers or something? Does this end up putting those
> skbs in shared memory even if they never hit a physical NIC? Same
> with a TDISP capable NIC, does this unconditionally degrade its
> security?
the current series was RFC and "unconditional" just to show the concept.
My next version (a trivial change, really) is fully opt-in on a
per-device and per direction basis.
The unencrypted (or pre-mapped) buffers will only be used for traffic
that hits a device that opted in, and only for the directions that were
opted in. So we can finely tune the configuration based on hw capabilities,
threat model and performance requirements.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 14:50 ` Jason Gunthorpe
2026-10-23 15:12 ` Luigi Rizzo
@ 2026-10-23 17:27 ` Jakub Kicinski
2026-10-23 20:22 ` Luigi Rizzo
2026-11-03 8:09 ` Christoph Hellwig
2 siblings, 1 reply; 86+ messages in thread
From: Jakub Kicinski @ 2026-10-23 17:27 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, 23 Oct 2026 11:50:23 -0300 Jason Gunthorpe wrote:
> > Anyways, reading your next comments, are you suggesting that all
> > these functionalities should be implemented as pagepool extensions?
> > (2MB mappings and iommu backing, proper allocator of fragments out of larger
> > pages, device-less pagepools, multi-domain mappings, pagepool
> > as memory provider for skb heads, pagepool as memory provider for descriptor
> > rings ?)
>
> netdev shouldn't be touching iommu, it can get a 2M page and map it
> using the normal DMA API. AFAIK there is not such a big reason to need
> any of these layering violations. They just make it easier on page
> pool..
FWIW we're entirely happy on the netdev side to add this to page pool.
In fact the initial RFC I posted for "page pool memory providers"
was addressing this exact use case. I didn't push it forward due
to low time and suspicion that IOMMU folks would prefer a more generic
solution. Since then memory providers were pushed forward and merged
for dmabuf, and we have netlink APIs to configure things. So adding
large mappings should be SMOC.
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 17:27 ` Jakub Kicinski
@ 2026-10-23 20:22 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-23 20:22 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jason Gunthorpe, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Oct 23, 2026 at 7:27 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 23 Oct 2026 11:50:23 -0300 Jason Gunthorpe wrote:
> > > Anyways, reading your next comments, are you suggesting that all
> > > these functionalities should be implemented as pagepool extensions?
> > > (2MB mappings and iommu backing, proper allocator of fragments out of larger
> > > pages, device-less pagepools, multi-domain mappings, pagepool
> > > as memory provider for skb heads, pagepool as memory provider for descriptor
> > > rings ?)
> >
> > netdev shouldn't be touching iommu, it can get a 2M page and map it
> > using the normal DMA API. AFAIK there is not such a big reason to need
> > any of these layering violations. They just make it easier on page
> > pool..
>
> FWIW we're entirely happy on the netdev side to add this to page pool.
> In fact the initial RFC I posted for "page pool memory providers"
> was addressing this exact use case. I didn't push it forward due
> to low time and suspicion that IOMMU folks would prefer a more generic
> solution. Since then memory providers were pushed forward and merged
> for dmabuf, and we have netlink APIs to configure things. So adding
> large mappings should be SMOC.
Maybe technicalities, maybe not...
Getting a 2MB contiguous and aligned page is only the first step, and
the easy one.
Next we need to split it into 4K fragments that can be recycled
without waiting for the entire 512 of them to be returned.
This is not rocket science, but requires much more state
that what fits in the struct page, so that's another layer that needs
to be added to quickly access the state from the pfn.
Essentially my dma_pmd_meta (because I could not/did not want to/ use
fields in struct page to access ).
Doing it in pagepool could reuse some fields in
struct page to point to the metadata and simplify the implementation,
but maybe leave people unhappy because now something may
require an extra indirection.
Then there is a question of resource management: when 512 fragments
come out of a much larger page, deciding which one to recycle matters a lot.
Without a strategy, there is a significant risk of stranding a
lot of lightly used 2M pages that cannot be returned because some
fragments are in use.
Again no rocket science but that takes some compromises (sorting
based on usage ? periodically drain old pages ? etc) between
effectiveness, code complexity and runtime cost.
And then of course adapting to zero or many devices...
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-23 14:50 ` Jason Gunthorpe
2026-10-23 15:12 ` Luigi Rizzo
2026-10-23 17:27 ` Jakub Kicinski
@ 2026-11-03 8:09 ` Christoph Hellwig
2026-11-03 9:13 ` Luigi Rizzo
2026-11-03 15:10 ` Jason Gunthorpe
2 siblings, 2 replies; 86+ messages in thread
From: Christoph Hellwig @ 2026-11-03 8:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Fri, Oct 23, 2026 at 11:50:23AM -0300, Jason Gunthorpe wrote:
> > Disk I/O is still potentially affected by the same problems
> > (cost if io mapping/unmapping, iotlb thrashing, bounce buffers
> > in confidential computing...)
>
> The disk folks already have a proposal to address all of this using
> io_uring and preallocated buffers. I really doubt yet another version
> is going to be interesting there.
Huh? If you mean the dma-buf I/O it doesn't really have anything
to do with disks, except that this is the first user. The dma-buf
iovec can work for every ->read_iter/->write_iter consumer, and
should be adopted more broadly. Including by networking instead of
their own totally weird different world.
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-03 8:09 ` Christoph Hellwig
@ 2026-11-03 9:13 ` Luigi Rizzo
2026-11-03 15:10 ` Jason Gunthorpe
1 sibling, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-03 9:13 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jason Gunthorpe, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Tue, Nov 3, 2026 at 9:09 AM Christoph Hellwig <hch@lst.de> wrote:
>
> On Fri, Oct 23, 2026 at 11:50:23AM -0300, Jason Gunthorpe wrote:
> > > Disk I/O is still potentially affected by the same problems
> > > (cost if io mapping/unmapping, iotlb thrashing, bounce buffers
> > > in confidential computing...)
> >
> > The disk folks already have a proposal to address all of this using
> > io_uring and preallocated buffers. I really doubt yet another version
> > is going to be interesting there.
>
> Huh? If you mean the dma-buf I/O it doesn't really have anything
> to do with disks, except that this is the first user. The dma-buf
> iovec can work for every ->read_iter/->write_iter consumer, and
> should be adopted more broadly. Including by networking instead of
> their own totally weird different world.
Christoph, is this the dma-buf series you are referring to ?
https://lore.kernel.org/all/cover.1791975048.git.asml.silence@gmail.com/
I am not claiming that my proposal is a good fit for disk or accelerator I/O,
but every subsystem has different constraints, so it is not unreasonable
that a proposal for (N)etwork and a proposal for (D)isk focus on the
different bottlenecks (still there can be learning from looking at the
other solution).
As an example, in networking we have to deal with the following:
- typically at least two buffers per packet (header+payload)
- often, very small fragments (1400-ish MTU plus crumbles from
data in different buffers
- extremely high I/O rate with almost no spatial locality, which is an IOTLB
killer and the main reason behind my proposal and some of its complexity.
- on tx no good way for the application to pre-reserve buffers
(buffers are sliced from a 32K per-task block,
going to possibly different destinations)
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-03 8:09 ` Christoph Hellwig
2026-11-03 9:13 ` Luigi Rizzo
@ 2026-11-03 15:10 ` Jason Gunthorpe
2026-11-06 17:21 ` Naman Gulati
1 sibling, 1 reply; 86+ messages in thread
From: Jason Gunthorpe @ 2026-11-03 15:10 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Tue, Nov 03, 2026 at 09:09:33AM +0100, Christoph Hellwig wrote:
> On Fri, Oct 23, 2026 at 11:50:23AM -0300, Jason Gunthorpe wrote:
> > > Disk I/O is still potentially affected by the same problems
> > > (cost if io mapping/unmapping, iotlb thrashing, bounce buffers
> > > in confidential computing...)
> >
> > The disk folks already have a proposal to address all of this using
> > io_uring and preallocated buffers. I really doubt yet another version
> > is going to be interesting there.
>
> Huh? If you mean the dma-buf I/O it doesn't really have anything
> to do with disks, except that this is the first user.
Yeah
> The dma-buf iovec can work for every ->read_iter/->write_iter
> consumer, and should be adopted more broadly. Including by
> networking instead of their own totally weird different world.
I'm not holding my breath on that one..
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-03 15:10 ` Jason Gunthorpe
@ 2026-11-06 17:21 ` Naman Gulati
2026-11-06 17:30 ` Jason Gunthorpe
0 siblings, 1 reply; 86+ messages in thread
From: Naman Gulati @ 2026-11-06 17:21 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
I drafted an implementation [1] for premapped TX and RX buffers on top
of the existing networking page_pool. It is implemented entirely in
the networking layer and extends page_pool as the default RX
allocator, to the TX path. There are no hooks in the IOMMU, DMA or
page allocator code. I wanted to understand if this implementation
correctly captured the feedback on this thread. Key components of this
design:
- Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
chunk is mapped once per opted-in device through the DMA API and
records its IOVA for each device, rather than using a shared IOVA
window.
- TX: page_pool accepts a memory provider for pools that are not bound
to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
of their NUMA node; TCP copies sendmsg() data into them through the
task page_frag and allocates its skb heads from them.
- RX: an instance of the same memory provider, sharing the chunk code,
backs RX queue page_pools. RX chunks of order-0 pages belong to one
device and NUMA node, and are shared by that device's queues on the
node.
[1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
On Tue, Nov 3, 2026 at 8:24 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Nov 03, 2026 at 09:09:33AM +0100, Christoph Hellwig wrote:
> > On Fri, Oct 23, 2026 at 11:50:23AM -0300, Jason Gunthorpe wrote:
> > > > Disk I/O is still potentially affected by the same problems
> > > > (cost if io mapping/unmapping, iotlb thrashing, bounce buffers
> > > > in confidential computing...)
> > >
> > > The disk folks already have a proposal to address all of this using
> > > io_uring and preallocated buffers. I really doubt yet another version
> > > is going to be interesting there.
> >
> > Huh? If you mean the dma-buf I/O it doesn't really have anything
> > to do with disks, except that this is the first user.
>
> Yeah
>
> > The dma-buf iovec can work for every ->read_iter/->write_iter
> > consumer, and should be adopted more broadly. Including by
> > networking instead of their own totally weird different world.
>
> I'm not holding my breath on that one..
>
> Jason
>
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 17:21 ` Naman Gulati
@ 2026-11-06 17:30 ` Jason Gunthorpe
2026-11-06 18:00 ` Naman Gulati
0 siblings, 1 reply; 86+ messages in thread
From: Jason Gunthorpe @ 2026-11-06 17:30 UTC (permalink / raw)
To: Naman Gulati
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> I drafted an implementation [1] for premapped TX and RX buffers on top
> of the existing networking page_pool. It is implemented entirely in
> the networking layer and extends page_pool as the default RX
> allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> page allocator code. I wanted to understand if this implementation
> correctly captured the feedback on this thread. Key components of this
> design:
>
> - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> chunk is mapped once per opted-in device through the DMA API and
> records its IOVA for each device, rather than using a shared IOVA
> window.
>
> - TX: page_pool accepts a memory provider for pools that are not bound
> to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> of their NUMA node; TCP copies sendmsg() data into them through the
> task page_frag and allocates its skb heads from them.
>
> - RX: an instance of the same memory provider, sharing the chunk code,
> backs RX queue page_pools. RX chunks of order-0 pages belong to one
> device and NUMA node, and are shared by that device's queues on the
> node.
>
> [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
IMHO the netstack should never read from shared/decrypted memory. If
you put it in a SKB you have to be sure the only touch after that
point will be a DMA read from an untrusted device on the TX side
(sending it to a trusted device would be a fail) and the final copy to
userspace on the RX side.
Everything else is very security questionable.
It is unclear to me how any of these page pool proposals meet that
requirement since page pool typically provides memory that ends up
widely used in the netstack.
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 17:30 ` Jason Gunthorpe
@ 2026-11-06 18:00 ` Naman Gulati
2026-11-06 18:07 ` Jason Gunthorpe
0 siblings, 1 reply; 86+ messages in thread
From: Naman Gulati @ 2026-11-06 18:00 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 6, 2026 at 9:31 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> > I drafted an implementation [1] for premapped TX and RX buffers on top
> > of the existing networking page_pool. It is implemented entirely in
> > the networking layer and extends page_pool as the default RX
> > allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> > page allocator code. I wanted to understand if this implementation
> > correctly captured the feedback on this thread. Key components of this
> > design:
> >
> > - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> > chunk is mapped once per opted-in device through the DMA API and
> > records its IOVA for each device, rather than using a shared IOVA
> > window.
> >
> > - TX: page_pool accepts a memory provider for pools that are not bound
> > to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> > of their NUMA node; TCP copies sendmsg() data into them through the
> > task page_frag and allocates its skb heads from them.
> >
> > - RX: an instance of the same memory provider, sharing the chunk code,
> > backs RX queue page_pools. RX chunks of order-0 pages belong to one
> > device and NUMA node, and are shared by that device's queues on the
> > node.
> >
> > [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
>
> IMHO the netstack should never read from shared/decrypted memory. If
> you put it in a SKB you have to be sure the only touch after that
> point will be a DMA read from an untrusted device on the TX side
> (sending it to a trusted device would be a fail) and the final copy to
> userspace on the RX side.
That's likely a fair callout. TBH I'm not very familiar with CoCo and
will spend some time understanding the constraints. I kept CoCo out of
my implementation as I approached this solution from the overcommitted
VM problem where we are trying to overcommit memory in VMs and use
these pre-mapped pages to avoid using bounce buffers.
>
> Everything else is very security questionable.
>
> It is unclear to me how any of these page pool proposals meet that
> requirement since page pool typically provides memory that ends up
> widely used in the netstack.
>
> Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 18:00 ` Naman Gulati
@ 2026-11-06 18:07 ` Jason Gunthorpe
2026-11-06 18:17 ` Naman Gulati
0 siblings, 1 reply; 86+ messages in thread
From: Jason Gunthorpe @ 2026-11-06 18:07 UTC (permalink / raw)
To: Naman Gulati
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 06, 2026 at 10:00:35AM -0800, Naman Gulati wrote:
> On Fri, Nov 6, 2026 at 9:31 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> > > I drafted an implementation [1] for premapped TX and RX buffers on top
> > > of the existing networking page_pool. It is implemented entirely in
> > > the networking layer and extends page_pool as the default RX
> > > allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> > > page allocator code. I wanted to understand if this implementation
> > > correctly captured the feedback on this thread. Key components of this
> > > design:
> > >
> > > - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> > > chunk is mapped once per opted-in device through the DMA API and
> > > records its IOVA for each device, rather than using a shared IOVA
> > > window.
> > >
> > > - TX: page_pool accepts a memory provider for pools that are not bound
> > > to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> > > of their NUMA node; TCP copies sendmsg() data into them through the
> > > task page_frag and allocates its skb heads from them.
> > >
> > > - RX: an instance of the same memory provider, sharing the chunk code,
> > > backs RX queue page_pools. RX chunks of order-0 pages belong to one
> > > device and NUMA node, and are shared by that device's queues on the
> > > node.
> > >
> > > [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
> >
> > IMHO the netstack should never read from shared/decrypted memory. If
> > you put it in a SKB you have to be sure the only touch after that
> > point will be a DMA read from an untrusted device on the TX side
> > (sending it to a trusted device would be a fail) and the final copy to
> > userspace on the RX side.
>
> That's likely a fair callout. TBH I'm not very familiar with CoCo and
> will spend some time understanding the constraints. I kept CoCo out of
> my implementation as I approached this solution from the overcommitted
> VM problem where we are trying to overcommit memory in VMs and use
> these pre-mapped pages to avoid using bounce buffers.
Why does overcomitted VM use bounce buffers? Do you mean this is
about getting better memory clustering in page pool since the VM
paging granule is 2M?
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 18:07 ` Jason Gunthorpe
@ 2026-11-06 18:17 ` Naman Gulati
2026-11-06 18:43 ` Jason Gunthorpe
0 siblings, 1 reply; 86+ messages in thread
From: Naman Gulati @ 2026-11-06 18:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 6, 2026 at 10:07 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Nov 06, 2026 at 10:00:35AM -0800, Naman Gulati wrote:
> > On Fri, Nov 6, 2026 at 9:31 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> > > > I drafted an implementation [1] for premapped TX and RX buffers on top
> > > > of the existing networking page_pool. It is implemented entirely in
> > > > the networking layer and extends page_pool as the default RX
> > > > allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> > > > page allocator code. I wanted to understand if this implementation
> > > > correctly captured the feedback on this thread. Key components of this
> > > > design:
> > > >
> > > > - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> > > > chunk is mapped once per opted-in device through the DMA API and
> > > > records its IOVA for each device, rather than using a shared IOVA
> > > > window.
> > > >
> > > > - TX: page_pool accepts a memory provider for pools that are not bound
> > > > to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> > > > of their NUMA node; TCP copies sendmsg() data into them through the
> > > > task page_frag and allocates its skb heads from them.
> > > >
> > > > - RX: an instance of the same memory provider, sharing the chunk code,
> > > > backs RX queue page_pools. RX chunks of order-0 pages belong to one
> > > > device and NUMA node, and are shared by that device's queues on the
> > > > node.
> > > >
> > > > [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
> > >
> > > IMHO the netstack should never read from shared/decrypted memory. If
> > > you put it in a SKB you have to be sure the only touch after that
> > > point will be a DMA read from an untrusted device on the TX side
> > > (sending it to a trusted device would be a fail) and the final copy to
> > > userspace on the RX side.
> >
> > That's likely a fair callout. TBH I'm not very familiar with CoCo and
> > will spend some time understanding the constraints. I kept CoCo out of
> > my implementation as I approached this solution from the overcommitted
> > VM problem where we are trying to overcommit memory in VMs and use
> > these pre-mapped pages to avoid using bounce buffers.
>
> Why does overcomitted VM use bounce buffers? Do you mean this is
> about getting better memory clustering in page pool since the VM
> paging granule is 2M?
Not exactly, we are trying to use VFIO passthrough NICs in
overcommitted VMs. Since the guest memory is overcommitted we can't
dma-map the entire guest address space (as non-overcommitted VMs do).
So overcommitted VMs can use bounce buffers to have a set of
pre-mapped buffers to bounce traffic in and out. We're trying to use
this solution to map pages dynamically with virtio-iommu and recycle
them via page-pool to avoid traps on each TX/RX and avoid the copy
cost of bounce buffers.
This problem extends to generally reducing IOMMU overhead in
networking - taking the dma map/unmap cost out of the hot-path and
avoiding IOTLB thrashing.
>
> Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 18:17 ` Naman Gulati
@ 2026-11-06 18:43 ` Jason Gunthorpe
2026-11-06 19:01 ` Luigi Rizzo
2026-11-06 19:11 ` Naman Gulati
0 siblings, 2 replies; 86+ messages in thread
From: Jason Gunthorpe @ 2026-11-06 18:43 UTC (permalink / raw)
To: Naman Gulati
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 06, 2026 at 10:17:38AM -0800, Naman Gulati wrote:
> On Fri, Nov 6, 2026 at 10:07 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Fri, Nov 06, 2026 at 10:00:35AM -0800, Naman Gulati wrote:
> > > On Fri, Nov 6, 2026 at 9:31 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > >
> > > > On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> > > > > I drafted an implementation [1] for premapped TX and RX buffers on top
> > > > > of the existing networking page_pool. It is implemented entirely in
> > > > > the networking layer and extends page_pool as the default RX
> > > > > allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> > > > > page allocator code. I wanted to understand if this implementation
> > > > > correctly captured the feedback on this thread. Key components of this
> > > > > design:
> > > > >
> > > > > - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> > > > > chunk is mapped once per opted-in device through the DMA API and
> > > > > records its IOVA for each device, rather than using a shared IOVA
> > > > > window.
> > > > >
> > > > > - TX: page_pool accepts a memory provider for pools that are not bound
> > > > > to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> > > > > of their NUMA node; TCP copies sendmsg() data into them through the
> > > > > task page_frag and allocates its skb heads from them.
> > > > >
> > > > > - RX: an instance of the same memory provider, sharing the chunk code,
> > > > > backs RX queue page_pools. RX chunks of order-0 pages belong to one
> > > > > device and NUMA node, and are shared by that device's queues on the
> > > > > node.
> > > > >
> > > > > [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
> > > >
> > > > IMHO the netstack should never read from shared/decrypted memory. If
> > > > you put it in a SKB you have to be sure the only touch after that
> > > > point will be a DMA read from an untrusted device on the TX side
> > > > (sending it to a trusted device would be a fail) and the final copy to
> > > > userspace on the RX side.
> > >
> > > That's likely a fair callout. TBH I'm not very familiar with CoCo and
> > > will spend some time understanding the constraints. I kept CoCo out of
> > > my implementation as I approached this solution from the overcommitted
> > > VM problem where we are trying to overcommit memory in VMs and use
> > > these pre-mapped pages to avoid using bounce buffers.
> >
> > Why does overcomitted VM use bounce buffers? Do you mean this is
> > about getting better memory clustering in page pool since the VM
> > paging granule is 2M?
>
> Not exactly, we are trying to use VFIO passthrough NICs in
> overcommitted VMs. Since the guest memory is overcommitted we can't
> dma-map the entire guest address space (as non-overcommitted VMs do).
?? I understood you could and this was implemented under the
virtio-iommu by pinning the memory under active DMA?
Where in upstream does the DMA API force bounce buffering for this
case?
> So overcommitted VMs can use bounce buffers to have a set of
> pre-mapped buffers to bounce traffic in and out. We're trying to use
> this solution to map pages dynamically with virtio-iommu and recycle
> them via page-pool to avoid traps on each TX/RX and avoid the copy
> cost of bounce buffers.
>
> This problem extends to generally reducing IOMMU overhead in
> networking - taking the dma map/unmap cost out of the hot-path and
> avoiding IOTLB thrashing.
Sure, that makes sense, but just having a pool of entirely normal
pre-dma-mapped memory is quite a different problem from what launched
this thread...
Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 18:43 ` Jason Gunthorpe
@ 2026-11-06 19:01 ` Luigi Rizzo
2026-11-06 19:11 ` Naman Gulati
1 sibling, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-06 19:01 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Naman Gulati, Christoph Hellwig, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 6, 2026 at 7:43 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Nov 06, 2026 at 10:17:38AM -0800, Naman Gulati wrote:
...
> > This problem extends to generally reducing IOMMU overhead in
> > networking - taking the dma map/unmap cost out of the hot-path and
> > avoiding IOTLB thrashing.
>
> Sure, that makes sense, but just having a pool of entirely normal
> pre-dma-mapped memory is quite a different problem from what launched
> this thread...
Since I launched the thread, I guess I should comment...
I wanted to address multiple problems, in random order:
- IOTLB thrashing
- better security than DMA-FQ
- reduce dma map/unmap costs (both on the host and guest)
- reduce/avoid bounce buffer overhead in Confidential Computing and
overcommitted VMs
- not just for networking, also for disks and accelerators etc.
Now it turns out that one way to address all of them was to
- create a special memory allocator, which distributed ordinary memory but
with additional metadata that make it easier to implement large
iommu mappings,
track lifetime etc.
- hook this allocator into relevant kernel subsystems so that consumers
could use it with little or no changes.
As it happens, the first version I posted last month was not terribly clean,
and possibly not terribly well presented, and had some redundancy that
was unnecessary.
The next version (hopefully next week) is hopefully a lot cleaner,
and i think the best witness is how small is the change to pagepool,
basically "if the device has opted in, ask for that special memory"
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index b775b6305fb78..e12ba47a20e43 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
@@ -602,6 +603,8 @@ static noinline netmem_ref
__page_pool_alloc_netmems_slow(struct page_pool *pool
*/
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
/* Don't support bulk alloc for high-order pages */
if (unlikely(pp_order))
The network tx side is more complicated because, as discussed
earlier, we need to be sure that the leaf device is one that needs
those special dma-able buffers, and there isn't (or I have not found)
one central place to hook into for tcp, udp, zerocopy,
security things etc., leading to a plethora of small hooks
net/core/skbuff.c | 42 +++-
net/core/sock.c | 62 +++++-
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 19 +-
net/ipv4/tcp.c | 23 ++-
net/ipv4/tcp_output.c | 12 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 19 +-
net/tls/tls_device.c | 4 +-
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-11-06 18:43 ` Jason Gunthorpe
2026-11-06 19:01 ` Luigi Rizzo
@ 2026-11-06 19:11 ` Naman Gulati
1 sibling, 0 replies; 86+ messages in thread
From: Naman Gulati @ 2026-11-06 19:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Christoph Hellwig, Luigi Rizzo, David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Marek Szyprowski, Andrew Morton, Vlastimil Babka,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
On Fri, Nov 6, 2026 at 10:43 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Nov 06, 2026 at 10:17:38AM -0800, Naman Gulati wrote:
> > On Fri, Nov 6, 2026 at 10:07 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Fri, Nov 06, 2026 at 10:00:35AM -0800, Naman Gulati wrote:
> > > > On Fri, Nov 6, 2026 at 9:31 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > > >
> > > > > On Fri, Nov 06, 2026 at 09:21:08AM -0800, Naman Gulati wrote:
> > > > > > I drafted an implementation [1] for premapped TX and RX buffers on top
> > > > > > of the existing networking page_pool. It is implemented entirely in
> > > > > > the networking layer and extends page_pool as the default RX
> > > > > > allocator, to the TX path. There are no hooks in the IOMMU, DMA or
> > > > > > page allocator code. I wanted to understand if this implementation
> > > > > > correctly captured the feedback on this thread. Key components of this
> > > > > > design:
> > > > > >
> > > > > > - Chunks: as in Luigi's series, memory comes in 2 MiB chunks, but each
> > > > > > chunk is mapped once per opted-in device through the DMA API and
> > > > > > records its IOVA for each device, rather than using a shared IOVA
> > > > > > window.
> > > > > >
> > > > > > - TX: page_pool accepts a memory provider for pools that are not bound
> > > > > > to an RX queue. Per-CPU page_pools take 32 KiB pages from the chunks
> > > > > > of their NUMA node; TCP copies sendmsg() data into them through the
> > > > > > task page_frag and allocates its skb heads from them.
> > > > > >
> > > > > > - RX: an instance of the same memory provider, sharing the chunk code,
> > > > > > backs RX queue page_pools. RX chunks of order-0 pages belong to one
> > > > > > device and NUMA node, and are shared by that device's queues on the
> > > > > > node.
> > > > > >
> > > > > > [1] https://github.com/torvalds/linux/compare/master...NamanGulati:linux:netdev_dma_pool
> > > > >
> > > > > IMHO the netstack should never read from shared/decrypted memory. If
> > > > > you put it in a SKB you have to be sure the only touch after that
> > > > > point will be a DMA read from an untrusted device on the TX side
> > > > > (sending it to a trusted device would be a fail) and the final copy to
> > > > > userspace on the RX side.
> > > >
> > > > That's likely a fair callout. TBH I'm not very familiar with CoCo and
> > > > will spend some time understanding the constraints. I kept CoCo out of
> > > > my implementation as I approached this solution from the overcommitted
> > > > VM problem where we are trying to overcommit memory in VMs and use
> > > > these pre-mapped pages to avoid using bounce buffers.
> > >
> > > Why does overcomitted VM use bounce buffers? Do you mean this is
> > > about getting better memory clustering in page pool since the VM
> > > paging granule is 2M?
> >
> > Not exactly, we are trying to use VFIO passthrough NICs in
> > overcommitted VMs. Since the guest memory is overcommitted we can't
> > dma-map the entire guest address space (as non-overcommitted VMs do).
>
> ?? I understood you could and this was implemented under the
> virtio-iommu by pinning the memory under active DMA?
>
> Where in upstream does the DMA API force bounce buffering for this
> case?
Sorry, should have been clearer: the DMA API does not enforce this,
bounce buffering is typically implemented by the drivers in this case.
Such as QPL mode in the GVE driver.
>
> > So overcommitted VMs can use bounce buffers to have a set of
> > pre-mapped buffers to bounce traffic in and out. We're trying to use
> > this solution to map pages dynamically with virtio-iommu and recycle
> > them via page-pool to avoid traps on each TX/RX and avoid the copy
> > cost of bounce buffers.
> >
> > This problem extends to generally reducing IOMMU overhead in
> > networking - taking the dma map/unmap cost out of the hot-path and
> > avoiding IOTLB thrashing.
>
> Sure, that makes sense, but just having a pool of entirely normal
> pre-dma-mapped memory is quite a different problem from what launched
> this thread...
>
> Jason
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 13:51 ` David Hildenbrand (Arm)
2026-10-12 14:45 ` Luigi Rizzo
@ 2026-10-12 14:49 ` Mostafa Saleh
2026-10-12 15:02 ` Luigi Rizzo
1 sibling, 1 reply; 86+ messages in thread
From: Mostafa Saleh @ 2026-10-12 14:49 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Mon, Oct 12, 2026 at 03:51:39PM +0200, David Hildenbrand (Arm) wrote:
> On 10/3/26 23:22, Luigi Rizzo wrote:
> > Here is a subsystem called DMA_PMD on which I would like feedback on
> > architecture, possible enhancements, or kernel components that could be
> > reused to avoid duplication.
>
> Here is a feedback to a subsystem called DMA_PMD ;)
>
> Why is basing this on PMDs (and embedding it in the name) the right approach?
>
> On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
> want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
> possibly also have TLB benefits (cont-pte)?
+1 I raised the same point in the LPC talk.
AFAIK, this series tries to solve both IOTLB pressure and CoCo bouncing
at the same time which might be too strict; as for CoCo there is no
actual IOMMU, so there is no sizing constraint.
Also to reduce bouncing, I think it is sufficient to use the SWIOTLB
instead of having a new pool + recycling, the tricky part is how to
abstract that to client drivers as this would be needed beyond SKB.
Thanks,
Mostafa
>
> --
> Cheers,
>
> David
>
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-12 14:49 ` Mostafa Saleh
@ 2026-10-12 15:02 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-12 15:02 UTC (permalink / raw)
To: Mostafa Saleh
Cc: David Hildenbrand (Arm),
Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Greg Kroah-Hartman, Rafael J . Wysocki,
Danilo Krummrich, Jonathan Corbet, Jesper Dangaard Brouer,
Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
Joshua Washington, Harshitha Ramamurthy, Saeed Mahameed,
Tariq Toukan, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Michael Chan, Pavan Chebbi, iommu, netdev, linux-mm, driver-core,
linux-doc, linux-kernel
On Mon, Oct 12, 2026 at 4:49 PM Mostafa Saleh <smostafa@google.com> wrote:
>
> On Mon, Oct 12, 2026 at 03:51:39PM +0200, David Hildenbrand (Arm) wrote:
> > On 10/3/26 23:22, Luigi Rizzo wrote:
> > > Here is a subsystem called DMA_PMD on which I would like feedback on
> > > architecture, possible enhancements, or kernel components that could be
> > > reused to avoid duplication.
> >
> > Here is a feedback to a subsystem called DMA_PMD ;)
> >
> > Why is basing this on PMDs (and embedding it in the name) the right approach?
> >
> > On architctures with large PMDs (512 MiB on arm64 with 64k), would we really
> > want to use 512 MiB pages, or rather something smaller (e.g., 2 MiB) that might
> > possibly also have TLB benefits (cont-pte)?
>
> +1 I raised the same point in the LPC talk.
See my other reply, he code makes it clear that the target is
reasonably-sized chunks (2M being the obvious choice with 4K PTE/2M PMD,
for other cases we need hw support, and it is relatively simple to
decouple page size and chunk size.
> AFAIK, this series tries to solve both IOTLB pressure and CoCo bouncing
> at the same time which might be too strict; as for CoCo there is no
> actual IOMMU, so there is no sizing constraint.
>
> Also to reduce bouncing, I think it is sufficient to use the SWIOTLB
> instead of having a new pool + recycling, the tricky part is how to
> abstract that to client drivers as this would be needed beyond SKB.
I did have another series to build buffers out of the swiotlb pool
https://lore.kernel.org/all/20260824152932.1583506-1-lrizzo@google.com/
but I found it does not scale well under pressure: the existing swiotlb
pools use linear search to find an empty region of suitable size,
linear scanning when one pool is full, which quickly degrades to
O(N) on each allocation.
I had some other (not posted) changes to build a buddy allocator
on top of the pool (so searching for a slot becomes O(1)) and
to spread the search order when a pool is full (avoiding O(N)
locking conflicts), but there is a bit of extra overhead in the
light-load case. Also it felt like reimplementing the wheel.
Overall I think DMA_[NOT_JUST_]PMD is cleaner and superior.
cheers
luigi
> Thanks,
> Mostafa
>
> >
> > --
> > Cheers,
> >
> > David
> >
--
-----------------------------------------
Luigi RIZZO, rizzo.unipi@gmail.com
http://www.iet.unipi.it/~luigi/
Mobile +39-338-6809875
-----------------------------------------
^ permalink raw reply [flat|nested] 86+ messages in thread
* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (23 preceding siblings ...)
2026-10-12 13:51 ` David Hildenbrand (Arm)
@ 2026-10-14 17:28 ` Dragos Tatulea
2026-10-14 18:14 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
` (25 subsequent siblings)
50 siblings, 1 reply; 86+ messages in thread
From: Dragos Tatulea @ 2026-10-14 17:28 UTC (permalink / raw)
To: Luigi Rizzo, Luigi Rizzo, Joerg Roedel, Will Deacon,
Robin Murphy, Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Jesper Dangaard Brouer, Ilias Apalodimas,
Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
Harshitha Ramamurthy, Saeed Mahameed, Tariq Toukan, Tony Nguyen,
Przemek Kitszel, Alexander Lobakin, Michael Chan, Pavan Chebbi,
iommu, netdev, linux-mm, driver-core, linux-doc, linux-kernel
Hi Luigi,
On 03.10.26 23:22, Luigi Rizzo wrote:
> Here is a subsystem called DMA_PMD on which I would like feedback on
> architecture, possible enhancements, or kernel components that could be
> reused to avoid duplication.
>
> I think it can be extremely useful for those affected by the HW/SW overhead
> of IOMMU (especially IOTLB thrashing, see [1]), or confidential computing,
> or preemptable VMs.
>
> The (not too exaggerated) pitch line is
>
> DMA_PMD has the performance of identity, guarantees that only IO buffers
> can ever have active IOMMU or IOTLB mappings, removes the bounce buffer
> overhead in confidential computing and preemptable VMs, and integrates
> smoothly with existing kernel APIs.
>
> === ARCHITECTURE
>
> DMA_PMD was initially designed to address IOTLB thrashing (details in [1]),
> but turned out to also resolve nicely the strict IOMMU overhead, and avoid
> the bounce buffer overhead in preemptible VMs and Confidential Computing.
>
> It works by combining several known techniques:
>
> - transparently feed allocators of device memory (skb_page_frag_refill(),
> dma_alloc_attrs(), pagepool, ...) with DMA_PMD pages, i.e. physically
> contiguous PMD_SIZE (2MB) pages mapped via PDE_SIZE IOMMU entries
>
> - heavy recycling of DMA_PMD pages (like pagepool) and lazy IOMMU unmapping
> (like DMA-FQ), BUT:
>
> - like strict IOMMU (DMA), safely release memory back to the kernel only
> after destroying all of its IOMMU mappings and a synchronous IOTLB flush
>
> - on allocation, DMA_PMD pages can be configured to be pinned in the host
> (hence suitable for preemptible VMs) and/or unencrypted (hence suitable
> for Confidential Computing), removing the need for bounce buffers
>
> - there are global and per-device optin /sys/device/.../dma_pmd_*
> and /proc/sys/net/core/tx_enable_dma_pmd
>
> NIC drivers can typically use dma_pmd with no modifications for rings,
> tx buffers, rx buffers (if they use pagepool), and rx headers. For tx
> headers, almost all drivers need changes (see later patches in the series)
> to implement cheap bounce buffers and avoid individual 4K mappings.
>
> The series has the following main components:
>
> - a sparse array (similar to pageblock_flags) to quickly attach metadata
> to a 2MB page without fiddling with the struct page. Cost is 128B per 2MB
> page used as an IO buffer, totally negligible.
>
> - dma_pmd_pool, a replacement for alloc_pages() that can be instantiated
> per-CPU or per receive queue. It handles the split of PMD_SIZE pages
> into order-N blocks, handles dma_map and unmap, and aggressively recycles
> entries. It is used to feed skb_page_frag_refill(), pagepool, and
> receive buffers for drivers that do not use pagepool.
> See [2] for "WHY NOT PAGEPOOL FOR TX AND EVERYTHING"
>
> - dma_pmd_arena, is another allocator backed by DMA_PMD pages and
> is used exclusively as the backend for dma_alloc_attrs().
> Used for longer-lived allocations (descriptor/completion rings,
> rx and tx header buffers)
>
> - glue code to hook DMA_PMD into pagepool [2], skb_page_frag_refill(),
> dma_alloc_attrs(), dma_map/unmap...
>
> - per-driver patches, where necessary (e.g. tx header buffers [3])
>
> === PERFORMANCE BENEFITS
>
> Your mileage may vary. Enabling the IOMMU may have no throughput
> impact, until it does when some system components (bus, IOTLB, CPU)
> become overloaded. Aside from throughput reduction, one interesting
> parameter is the effectiveness of the IOTLB. Here is a sample of SMMU
> performance counters for a large ARM system with 2x200G NICs doing
> bidirectional traffic:
>
> === DMA-FQ MODE (total throughput ~440Gbps)
> 25,757,488 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
> 16,634,941 smmuv3_pmcg_*/event=0x81/ IOTLB misses
>
> === DMA_PMD on top of strict DMA (total throughput ~745Gbps)
> 22,693,218 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
> 30,536 smmuv3_pmcg_*/event=0x81/ IOTLB misses
> (not a mistake, also lookups went down despite the higher rate
> because the tx side can use larger segments)
>
> The 2MB mappings made IOTLB misses almost non existent, because
> the working set is reduced by a factor of ~512.
>
> Note, the code has more verbose comments than I would like.
> Several of them are there to avoid Sashiko getting confused and
> flagging false positives.
>
> === NOTES
>
> [1] IOMMU IMPACT AND IOTLB THRASHING
> This is documented in more detail in Documentation/core-api/dma-pmd.rst
> but the compact version is below.
>
> There are three main costs involved with using the IOMMU:
> - CPU cost for dma map/unmap
> - CPU cost and latency for synchronous IOTLB flush (required for strong security)
> - IOTLB thrashing, shows up dramatically when the IO access pattern
> exceeds the IOTLB size, and IOMMU page walks slow down bus activity
> up to a point where we see over 30..60% throughput reduction just for
> this reason. Some relevant references
>
> https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/
> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/
>
> Specifically, NICs access 4 or more pages per packet (descriptor queue,
> completion queue, rx or tx header, one or more buffers), with very little
> locality especially for data buffers and tx headers. Each queue has 1-4K
> entries, a fast NIC uses 16 or more queues per direction, so just the
> data buffers cover 32..128K pages. Using 4KB mappings makes the IOTLB
> ineffective, as shown by the performance counters shown earlier.
>
> [2] WHY NOT PAGEPOOL FOR TX AND EVERYTHING
> There is some overlap between pagepool and dma_pmd_pool in that both
> implement a cache, but pagepool is missing some important features:
>
> - pagepool does not handle splitting 2M pages in smaller chunks, or
> the mappings. That would need to be implemented (and is what dma_pmd_pool does)
>
> - pagepool's provided pages cannot be cleanly managed by
> get_page()/put_page(), which is what all the consumers of
> skb_page_frag_refill(). Fixing that would require dozen of changes
> with high risk of missing some paths.
>
> - even for the receive path, a provider for pagepool has more restrictions
> than alloc_pages-supplied memory, so if I wanted to wrap dma_pmd_pool
> as a provider I would have needed changes to the provider interface.
>
> - pagepool requires a device at allocation time, which is not known when
> skb_page_frag_refill() runs, hence the IOMMU mappings cannot be handled
> at allocation time.
>
> - page_pool_alloc_pages assumes a single NAPI consumer with BH disabled,
> whereas skb_page_frag_refill() runs from a preemptive context.
>
> - and last not least, we need to handle the dma_alloc_attrs() allocations
> which don't match pagepool API
>
> Of course all of the above could be modified, but the complexity
> would be similar to that of implementing dma_pmd_pool, and with a
> huge risk of breaking existing functionality or missing some path.
>
> [3] DRIVER CHANGES
>
> Most drivers need no change for rings, transmit buffers, or receive buffers
> (if they use pagepool). Tx headers are generally mapped on the fly and
> that is enough to trigger IOTLB thrashing, so most drivers need a small
> change to implement very cheap tx bounce buffers backed by DMA_PMD pages.
>
> We could avoid the bounce buffers using DMA_PMD for the skb->head region,
> but that memory is contiguous to skb_shinfo so it would be exposed to
> IO device access, which may be undesirable for security.
>
Thanks for the RFC. Here are my notes (some points rephrase what was already
said):
- On CoCo VMs it seems that the only way to opt in the IOMMU optimization is to
also opt in the copy avoidance which is not always desirable.
- Having the pool generic opens up more visibility from the untrusted host
side. Having device based pools at least isolates one device's memory from
the rest of the system.
- Also, device driver could help with provisioning just the right amount of
memory that is needed. And per device control also means that probably
you don't need to map everything RW to the IOMMU.
Thanks,
Dragos
^ permalink raw reply [flat|nested] 86+ messages in thread* Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
2026-10-14 17:28 ` Dragos Tatulea
@ 2026-10-14 18:14 ` Luigi Rizzo
0 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-10-14 18:14 UTC (permalink / raw)
To: Dragos Tatulea
Cc: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni, Greg Kroah-Hartman,
Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Saeed Mahameed, Tariq Toukan, Tony Nguyen, Przemek Kitszel,
Alexander Lobakin, Michael Chan, Pavan Chebbi, iommu, netdev,
linux-mm, driver-core, linux-doc, linux-kernel
On Wed, Oct 14, 2026 at 7:28 PM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>
> Hi Luigi,
>
> On 03.10.26 23:22, Luigi Rizzo wrote:
> > Here is a subsystem called DMA_PMD on which I would like feedback on
...
>
> Thanks for the RFC. Here are my notes (some points rephrase what was already
> said):
>
> - On CoCo VMs it seems that the only way to opt in the IOMMU optimization is to
> also opt in the copy avoidance which is not always desirable.
>
> - Having the pool generic opens up more visibility from the untrusted host
> side. Having device based pools at least isolates one device's memory from
> the rest of the system.
>
> - Also, device driver could help with provisioning just the right amount of
> memory that is needed. And per device control also means that probably
> you don't need to map everything RW to the IOMMU.
Yes on all three. I am looking into providing more knobs for different levels
of opt-in and isolation, so users may adjust them according to the
threat models and performance goals.
I am not especially worried about provisioning because the system is
already reasonably adaptive, and leveraging existing caches, the waste
the waste due to allocating 2MB units can be kept relatively small.
As an example, the update I am working on has the following:
- a lean per-cpu cache that can be attached on top of a dma_pmd_pool,
similar to the one used by the buddy allocator. The dma_pmd_pool will hold
the 2MB pages, and the caches can buffer smaller sets, reducing the waste.
Useful on the tx side.
- on the receive side, one dma_pmd_pool per device will feed all
pagepools for that device (typically one per queue).
The pagepool is already a pretty good cache that does not
access the allocator often, so once again the waste due to
partially used pages is minimized.
cheers
luigi
^ permalink raw reply [flat|nested] 86+ messages in thread
* [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (24 preceding siblings ...)
2026-10-14 17:28 ` Dragos Tatulea
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
` (24 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
This is v2 of the RFC previously posted as DMA_PMD [0] (renamed to avoid
confusion with CPU page table PMD levels).
PISA stands for "Pre-mapped IOVA Superpage Allocator" and as the name
says, it is a specialized memory allocator. Invoked via __GFP_PISA* flag
to alloc_pages(), it returns pages ("PISA memory") that make DMA-related
operations on that memory a lot more efficient, both in software and
in hardware.
Specifically, PISA memory has the following properties:
- it is backed by 2MB contiguous physical pages, so the IOMMU can
use large leaf mappings and significantly reduce IOTLB pressure see [1]
- IOMMU mappings are recycled while PISA memory is used for IO buffers,
thus greatly reducing IOVA allocation and dma map/unmap costs.
This includes cases (e.g. overcommitted VMs or Confidential Compute)
where they would trigger expensive VM exits or bounce buffer copies.
- PISA memory is returned to the buddy allocator in background, and
only after all existing IOMMU and IOTLB mappings have been flushed.
This removes a significant security problem in DMA-FQ, which instead returns
pages before flushing the IOTLB, without the performance penalty of
strict iommu (DMA) where the flushes are synchronous.
The pitch line remains:
PISA has the performance of identity mapping, guarantees that only active
I/O buffers have IOMMU/IOTLB mappings (unlike DMA-FQ), removes bounce-buffer
copies in Confidential Computing VMs, and integrates transparently with
existing kernel subsystems and drivers.
=== KEY IDEA: A SPECIALIZED MEMORY ALLOCATOR
For avoidance of doubt, PISA is NOT a new buffer cache, page_pool
replacement, slab allocator, or custom netdev feature. The kernel already
has plenty of specialized caches and pools (page_pool, socket page_frags,
SLUB, page cache) built on top of the buddy allocator.
Instead, PISA sits *below* those existing subsystems as a specialized
memory provider modeled after alloc_pages() and __free_pages_prepare().
It enforces a few properties on the memory it dispenses so that IOMMU
mapping, unmapping, and CoCo page decryption become nearly free:
1. PISA allocates order-N blocks out of physically contiguous 2MB pages that
are mapped once per IOMMU domain using 2MB IOMMU leaf entries in a
reserved per-domain IOVA window (and decrypted once upfront in CoCo
guests). Subsequent dma_map_*() calls resolve the IOVA with a simple
window offset calculation, and dma_unmap_*() is a no-op.
2. Freed blocks are intercepted in __free_pages_prepare() and recycled
back into their PISA allocator while their 2MB IOMMU mappings remain
active. A 2MB page is returned to the buddy allocator only when all
of its blocks are idle, its IOMMU mappings have been removed from all
domains, a synchronous IOTLB flush has completed, and an RCU grace
period has elapsed.
3. Callers allocate from per-NUMA-node PISA allocators using standard page
allocator calls with __GFP_PISA_RW (DMA_BIDIRECTIONAL) or __GFP_PISA_RO
(DMA_TO_DEVICE, mapped IOMMU_READ-only so devices cannot overwrite
shared/adjacent data), falling back to the buddy allocator on failure.
Because PISA operates at the page allocator and DMA mapping boundary,
existing clients require minimal changes. For example, the entire patch
integrating page_pool with PISA (patch 13/25) is just 3 lines:
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
@@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(...)
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
Similarly:
- Coherent rings/queues (dma_alloc_attrs()) are served from the per-node
order-0 PISA allocator when dev->pisa_coherent is enabled; no driver
changes needed.
- Socket TX page fragments (skb_page_frag_refill()) use a per-task
page_frag fed by __GFP_PISA_RO when the egress device has pisa_tx_bufs
enabled; no driver changes needed.
- Small TX skb heads (kmalloc_reserve()) use a parallel
skbuff_small_head_pisa SLUB cache backed by zeroed, read-only PISA
folios (SLAB_CACHE_PISA_RO) when pisa_tx_skb_heads is enabled. This
eliminates the per-driver TX header bounce-buffer patches from v1.
- TX MSG_ZEROCOPY, O_DIRECT, and buffered disk I/O migrates or allocates
order-0 user and page-cache folios from per-node PISA allocators so
zero-copy network sends and storage I/O also use 2MB IOMMU mappings.
All features are strictly opt-in per device via
/sys/devices/.../pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads} (and
pisa.user_max_4K_pages for user/page-cache folios). Until a device opts in,
PISA is gated off by a static key and allocates no metadata.
=== CHANGES SINCE v1 [0]
I have make use and applied several comments and suggestions received on
the previous version. The changes are massive so perhaps it makes sense
to look at the new code in isolation, unless you have already revised
in detail the previous version.
A high level summary is the following:
- Renamed DMA_PMD to PISA (CONFIG_PISA, include/linux/pisa.h,
drivers/iommu/pisa-*.c, /sys/devices/.../pisa_*).
- Hooked __GFP_PISA_RO and __GFP_PISA_RW into __alloc_frozen_pages_noprof()
and alloc_pages_bulk_noprof(), replacing custom pool calls in page_pool
and socket TX refill.
- Added lockless per-CPU block caches (pisa_enable_pcp()) in front of the
per-node order-0 RW and SKB_FRAG_PAGE_ORDER RO allocators.
- Added SLAB_CACHE_PISA_RO in SLUB and skbuff_small_head_pisa for small
TX skb heads, dropping the per-driver TX header bounce-buffer patches.
- Added PISA support for MSG_ZEROCOPY user pages (madvise(MADV_PISA) and
automatic upgrade on send) and block/page-cache I/O (O_DIRECT and
buffered reads/writes), plus age/waste-based compaction and scrub
migration (pisa.wasted_pct, pisa.old_ms, pisa.scrub_migrate).
- Made metadata table initialization and static key activation lazy on
first device opt-in.
=== PERFORMANCE BENEFITS
Your mileage may vary. Enabling the IOMMU may have no throughput impact
until system components (bus, IOTLB, CPU) become overloaded. Aside from
throughput reduction, one key metric is IOTLB effectiveness. Below are SMMU
performance counters on a large ARM64 system with 2x200G NICs running
bidirectional traffic:
=== DMA-FQ MODE (total throughput ~440Gbps)
25,757,488 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
16,634,941 smmuv3_pmcg_*/event=0x81/ IOTLB misses
=== PISA on top of strict DMA (total throughput ~745Gbps)
22,693,218 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
30,536 smmuv3_pmcg_*/event=0x81/ IOTLB misses
The 2MB mappings make IOTLB misses almost non-existent because the working
set of IOTLB entries is reduced by a factor of ~512.
=== NOTES
[0] v1 (DMA_PMD):
https://lore.kernel.org/all/20261003212241.3432303-1-lrizzo@google.com/
[1] IOMMU IMPACT AND IOTLB THRASHING
Documented in detail in Documentation/core-api/pisa.rst. In short, there
are three main costs with the IOMMU:
- CPU cost for dma_map/unmap
- CPU cost and latency for synchronous IOTLB flush (required for security)
- IOTLB thrashing when the working set of 4KB pages exceeds IOTLB capacity,
causing IOMMU page walks that backpressure PCIe/bus traffic (30..60%
throughput drop on high-speed NICs). See also:
https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/
[2] WHY NOT USE PAGE_POOL FOR EVERYTHING
page_pool is a consumer of PISA (via __GFP_PISA_RW), not a replacement:
- page_pool does not split 2MB pages into order-N blocks or manage shared
2MB IOMMU mappings across domains;
- page_pool pages cannot be managed by arbitrary get_page()/put_page()
callers (socket TX fragments, SLUB slabs, user/page-cache folios);
- page_pool requires a single NAPI consumer with BH disabled and a known
device at allocation time, whereas socket TX, SLUB, and dma_alloc_attrs()
run in process/preemptible context across arbitrary devices.
Luigi Rizzo (25):
iommu/dma: Introduce CONFIG_PISA and metadata table
iommu/dma: Add PISA allocator lifecycle and page recycle hook
iommu/dma: Add per-NUMA-node PISA page reservoir
iommu/dma: Add PISA block allocation
iommu/dma: Add global cap and shrinker for PISA pages
iommu/dma: Reserve a per-domain IOVA window for PISA pages
iommu/dma: Release PISA domain mappings on domain teardown
iommu/dma: Use per-domain IOVA window to map PISA memory
driver core: Add per-device pisa_* sysfs attributes for PISA optin
iommu/dma: Use PISA for coherent DMA allocations
iommu/dma: Add per-CPU block caches to PISA allocators
net/core: Use per-node PISA allocators for skb_page_frag_refill()
net/core: Use PISA for page_pool memory
iommu/dma: Support decrypted PISA pages
iommu/dma: Add background page scrubber for PISA allocators
mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
net: Allocate small skb heads from PISA
iommu/dma: Migrate user pages into PISA for TX zerocopy
iommu/dma: Stop recycling old PISA pages when allocator waste is high
net/libeth: Use GFP_KERNEL when pre-filling RX rings
net/gve: Use GFP_KERNEL when pre-filling RX rings
net/gve: Use PISA memory for RX buffers
block, mm, iommu/dma: Support PISA for disk I/O and scrub migration
Documentation: Add PISA architecture documentation
iommu/dma: Add PISA statistics and debugfs
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/pisa.rst | 426 ++++
block/bio.c | 46 +
drivers/base/core.c | 78 +
drivers/iommu/Kconfig | 34 +
drivers/iommu/Makefile | 5 +
drivers/iommu/dma-iommu.c | 48 +-
drivers/iommu/dma-iommu.h | 16 +
drivers/iommu/iommu.c | 10 +-
drivers/iommu/pisa-alloc.c | 2063 +++++++++++++++++
drivers/iommu/pisa-kunit.c | 669 ++++++
drivers/iommu/pisa-map.c | 553 +++++
drivers/iommu/pisa-meta.c | 300 +++
drivers/iommu/pisa-node.c | 378 +++
drivers/iommu/pisa-priv.h | 511 ++++
drivers/iommu/pisa-user.c | 662 ++++++
.../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 7 +-
drivers/net/ethernet/google/gve/gve_rx.c | 40 +-
include/linux/device.h | 27 +
include/linux/gfp_types.h | 17 +-
include/linux/netdevice.h | 4 +
include/linux/pisa.h | 367 +++
include/linux/sched.h | 4 +
include/linux/skbuff.h | 2 +
include/linux/slab.h | 10 +
include/linux/swap.h | 1 +
include/net/libeth/rx.h | 11 +-
include/net/sock.h | 143 ++
include/trace/events/mmflags.h | 11 +-
include/uapi/asm-generic/mman-common.h | 2 +
kernel/dma/direct.c | 14 +-
kernel/exit.c | 2 +
kernel/fork.c | 2 +
mm/filemap.c | 15 +-
mm/madvise.c | 11 +
mm/page-writeback.c | 8 +
mm/page_alloc.c | 59 +
mm/readahead.c | 5 +-
mm/slab.h | 3 +-
mm/slab_common.c | 5 +-
mm/slub.c | 14 +-
net/core/datagram.c | 85 +-
net/core/page_pool.c | 3 +
net/core/skbuff.c | 42 +-
net/core/sock.c | 62 +-
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 19 +-
net/ipv4/tcp.c | 17 +-
net/ipv4/tcp_output.c | 12 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 19 +-
52 files changed, 6784 insertions(+), 73 deletions(-)
create mode 100644 Documentation/core-api/pisa.rst
create mode 100644 drivers/iommu/pisa-alloc.c
create mode 100644 drivers/iommu/pisa-kunit.c
create mode 100644 drivers/iommu/pisa-map.c
create mode 100644 drivers/iommu/pisa-meta.c
create mode 100644 drivers/iommu/pisa-node.c
create mode 100644 drivers/iommu/pisa-priv.h
create mode 100644 drivers/iommu/pisa-user.c
create mode 100644 include/linux/pisa.h
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (25 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
` (23 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA ("Pre-mapped IOVA Superpage Allocator") is a memory allocator
that provides devices which opt in with DMA-able memory backed by large
physically contiguous pages (typically 2MB, but fully configurable),
and mapped by IOMMU leaves of the same size.
PISA has the following features:
- greatly reduces IOTLB pressure by using fewer, much larger IOMMU
entries.
- greatly reduces DMA map/unmap cost. Mapping only happens when the
first block from a 2MB page is used. Unmapping is a no-op for the
caller, and happens in a separate task when no blocks from the 2MB
page are in use and the page should be returned to the buddy
allocator.
- much better security than DMA-FQ, because buffers are not returned to
the buddy allocator until the mappings have been removed and the IOTLB
flushed.
- removes the cost of copying into bounce buffers, e.g. for
overcommitted or Confidential Computing VMs.
- fine-grained opt-in selection (each device can opt in selectively for
descriptors, transmit and/or receive data buffers and headers) permits
choosing the correct security and performance tradeoffs.
- default is to use the base IOMMU mode (DMA or DMA-FQ) of the device.
PISA is integrated with the main IO APIs (dma_alloc_attrs,
dma_map/unmap, page_pool, TX socket buffers and disk IO buffers) so that
no device driver modifications should be necessary.
Add a CONFIG_PISA option, enabled only on supported kernels (64bit,
!PREEMPT_RT) and introduce a sparse metadata table to hold per-2MB buffer
state (similar to pageblock_flags).
The metadata table is initialized on first use, consumes 4KB of RAM
and 2MB of virtual address space for each 16GB of physical RAM.
Also add a built-in KUnit test suite (CONFIG_PISA_META_KUNIT_TEST)
exercising initialization idempotency, PFN/physical round-trip lookups,
out-of-bounds physical address handling, and PISA membership toggling.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Kconfig | 33 +++++
drivers/iommu/Makefile | 2 +
drivers/iommu/pisa-kunit.c | 102 +++++++++++++
drivers/iommu/pisa-meta.c | 286 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 31 ++++
include/linux/pisa.h | 79 ++++++++++
6 files changed, 533 insertions(+)
create mode 100644 drivers/iommu/pisa-kunit.c
create mode 100644 drivers/iommu/pisa-meta.c
create mode 100644 drivers/iommu/pisa-priv.h
create mode 100644 include/linux/pisa.h
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 85a8d7fdfa30b..15e06eaed0924 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -423,4 +423,37 @@ config IOMMU_DEBUG_PAGEALLOC
If unsure, say N.
endif # IOMMU_SUPPORT
+# Pre-mapped IOVA Superpage Allocator
+config PISA
+ bool "Pre-mapped IOVA Superpage Allocator"
+ depends on X86_64 || ARM64
+ depends on !PREEMPT_RT
+ default y
+ help
+ PISA is a specialized memory allocator with an API similar to
+ alloc_pages(). For devices that opt-in, it provides DMA-able
+ pages backed by 2MB physically contiguous pages, and mapped in
+ the IOMMU maps with a single leaf PTE of the same size. Mapping,
+ unmapping and IOTLB flushing are done lazily to reduce CPU overhead.
+ The large mappings greatly reduce IOTLB usage. As in strict iommu
+ mode, pages are returned to the buddy allocator only when any existing
+ mappings have been removed and IOTLB flushed.
+ Uses 4KB of metadata for each 32MB of PISA buffers, allocated on demand.
+
+ The config option enforces requirements (2MB large pages and !PREEMPT_RT) at compile
+ time. Other constraints (e.g. dma-coherent devices) are verified at runtime.
+
+ If unsure, say Y.
+
+config PISA_META_KUNIT_TEST
+ bool "KUnit test for PISA metadata table (built-in)" if !KUNIT_ALL_TESTS
+ depends on PISA && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ Builds KUnit unit tests for the sparse, per-PISA-frame metadata table,
+ and PISA page allocators in drivers/iommu/pisa-*.c, verifying metadata
+ lookup, allocator recycling, and coherent block allocation.
+
+ If unsure, say N.
+
source "drivers/iommu/generic_pt/Kconfig"
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 29a26d2a3af12..b5d3be1df3eda 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,6 +11,8 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_PISA) += pisa-meta.o
+obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
new file mode 100644
index 0000000000000..0633f3238e012
--- /dev/null
+++ b/drivers/iommu/pisa-kunit.c
@@ -0,0 +1,102 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * KUnit tests for the opaque struct pisa_meta table API.
+ */
+#include <kunit/test.h>
+#include <linux/gfp.h>
+#include <linux/pisa.h>
+#include <linux/mm.h>
+
+#include "pisa-priv.h"
+
+static void test_meta_init_and_roundtrip(struct kunit *test)
+{
+ unsigned long pfn, base_pfn;
+ struct pisa_meta *m_pfn;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+ /* Second call must be idempotent. */
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ page = alloc_page(GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PISA_ORDER);
+ __free_page(page);
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_ensure_pfn(pfn, true), 0);
+ m_pfn = pisa_meta_of_pfn(pfn);
+ KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
+}
+
+static void test_meta_invalid_phys(struct kunit *test)
+{
+ struct pisa_meta *meta;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ meta = pisa_meta_of_pfn(PHYS_PFN(PHYS_ADDR_MAX));
+ KUNIT_ASSERT_NOT_NULL(test, meta);
+ KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(meta), ULONG_MAX);
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(ULONG_MAX >> PAGE_SHIFT));
+ KUNIT_EXPECT_NE(test, pisa_meta_ensure_pfn(ULONG_MAX >> PAGE_SHIFT, false), 0);
+}
+
+static void pisa_meta_set_active(struct pisa_meta *meta, bool active)
+{
+ WRITE_ONCE(meta->flags, active ? PISA_ACTIVE : 0);
+}
+
+static void test_meta_active_toggle(struct kunit *test)
+{
+ unsigned long pfn, base_pfn;
+ struct pisa_meta *meta;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ page = alloc_pages(GFP_KERNEL, PISA_ORDER);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PISA_ORDER);
+ /* The buddy allocator returns naturally aligned blocks. */
+ KUNIT_EXPECT_EQ(test, pfn, base_pfn);
+ if (pisa_meta_ensure_pfn(pfn, true)) {
+ __free_pages(page, PISA_ORDER);
+ KUNIT_FAIL(test, "pisa_meta_ensure_pfn(%lu) failed", pfn);
+ return;
+ }
+ KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
+ meta = pisa_meta_of_pfn(pfn);
+
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+ pisa_meta_set_active(meta, true);
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+
+ pisa_meta_set_active(meta, false);
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+ __free_pages(page, PISA_ORDER);
+}
+
+static struct kunit_case pisa_meta_test_cases[] = {
+ KUNIT_CASE(test_meta_init_and_roundtrip),
+ KUNIT_CASE(test_meta_invalid_phys),
+ KUNIT_CASE(test_meta_active_toggle),
+ {}
+};
+
+static struct kunit_suite pisa_meta_test_suite = {
+ .name = "pisa_meta",
+ .test_cases = pisa_meta_test_cases,
+};
+
+kunit_test_suite(pisa_meta_test_suite);
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
new file mode 100644
index 0000000000000..5ceea91e4d5e8
--- /dev/null
+++ b/drivers/iommu/pisa-meta.c
@@ -0,0 +1,286 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA sparse per-PISA-frame metadata table.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/bitmap.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/memblock.h>
+#include <linux/memory.h>
+#include <linux/moduleparam.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/pgtable.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/vmalloc.h>
+
+#include "pisa-priv.h"
+
+static_assert(PISA_META_SIZE <= PAGE_SIZE);
+static_assert(PISA_SIZE <= SZ_2M);
+static_assert(PISA_BLOCKS(0) <= PISA_MAX_BLOCKS);
+static_assert(PISA_ORDER <= MAX_PAGE_ORDER);
+static_assert(PISA_SHIFT <= SUBSECTION_SHIFT);
+
+/* Enabled by pisa_meta_init() to activate fast-path hooks. */
+DEFINE_STATIC_KEY_FALSE(pisa_key);
+EXPORT_SYMBOL(pisa_key);
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Physical address bits covered by @pisa_meta_array (0 = max_possible_pfn). */
+static unsigned int pisa_max_bits __read_mostly;
+module_param_named(max_bits, pisa_max_bits, uint, 0444);
+
+/*
+ * Metadata is in a sparse @pisa_meta_array[pfn >> PISA_ORDER] in vmalloc space
+ * (256B per 2MB frame). Each 4KB backing page covers a chunk of 16 entries (32MB)
+ * and is tracked in @pisa_chunk_bitmap, 1 bit per chunk, 4KB per 1TB, 1MB for 48bit
+ * of address space. The total address space is @pisa_meta_nframes, all the variables
+ * are set and memory allocated before enabling @pisa_key.
+ */
+
+/* Sparse per-PISA-frame metadata table, indexed by pfn >> PISA_ORDER */
+struct pisa_meta *pisa_meta_array __read_mostly;
+EXPORT_SYMBOL(pisa_meta_array);
+
+/* Number of PISA frames covered by @pisa_meta_array. */
+unsigned long pisa_meta_nframes __read_mostly;
+EXPORT_SYMBOL(pisa_meta_nframes);
+
+/* One bit per 32MB chunk, set once its metadata page is backed. */
+unsigned long *pisa_chunk_bitmap __read_mostly;
+EXPORT_SYMBOL(pisa_chunk_bitmap);
+
+/* Shared sink entry returned for unbacked or out-of-range PFNs */
+static struct pisa_meta pisa_meta_nil __read_mostly;
+
+/* Protects init and page-table allocation */
+static DEFINE_MUTEX(pisa_meta_mutex);
+
+/* Return metadata for @pfn, or shared sink @pisa_meta_nil (flags == 0) if unbacked. */
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
+{
+ unsigned long frame = pfn >> PISA_ORDER;
+
+ if (unlikely(!static_branch_unlikely(&pisa_key) || frame >= pisa_meta_nframes ||
+ !test_bit_acquire(pfn >> PISA_CHUNK_ORDER, pisa_chunk_bitmap)))
+ return &pisa_meta_nil;
+
+ return &pisa_meta_array[frame];
+}
+
+/* Base PFN of @meta's PISA frame, or ULONG_MAX if out-of-range/sink. */
+unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
+{
+ unsigned long frame = meta - pisa_meta_array;
+
+ if (unlikely(frame >= pisa_meta_nframes))
+ return ULONG_MAX;
+
+ return frame << PISA_ORDER;
+}
+
+/* PTE for @addr in the array, or NULL if its page table was not allocated. */
+static pte_t *pisa_meta_pte(unsigned long addr)
+{
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ if (pgd_none(pgdp_get(pgd)))
+ return NULL;
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(p4dp_get(p4d)))
+ return NULL;
+ pud = pud_offset(p4d, addr);
+ if (pud_none(pudp_get(pud)))
+ return NULL;
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(pmdp_get(pmd)))
+ return NULL;
+ return pte_offset_kernel(pmd, addr);
+}
+
+/* Back chunk of @pfn in @pisa_meta_array with a zeroed page and set its bitmap bit. */
+static int pisa_meta_back(unsigned long pfn, gfp_t gfp)
+{
+ /* Protects lazy installation of backing pages into preallocated page tables. */
+ static DEFINE_SPINLOCK(pisa_meta_pte_lock);
+ unsigned long idx = pfn >> PISA_CHUNK_ORDER;
+ unsigned long addr = (unsigned long)pisa_meta_array + (idx << PAGE_SHIFT), flags;
+ struct page *online = pfn_to_online_page(pfn);
+ int nid = online ? page_to_nid(online) : NUMA_NO_NODE;
+ pte_t *ptep = pisa_meta_pte(addr);
+ struct page *page;
+
+ if (!ptep)
+ return -EINVAL;
+
+ page = alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_NOWARN, 0);
+ if (!page)
+ return -ENOMEM;
+
+ spin_lock_irqsave(&pisa_meta_pte_lock, flags);
+ if (pte_none(ptep_get(ptep))) {
+ set_pte_at(&init_mm, addr, ptep, pfn_pte(page_to_pfn(page), PAGE_KERNEL));
+ flush_cache_vmap(addr, addr + PAGE_SIZE);
+ page = NULL;
+ /* Pair with test_bit_acquire() in readers. */
+ smp_mb__before_atomic();
+ set_bit(idx, pisa_chunk_bitmap);
+ }
+ spin_unlock_irqrestore(&pisa_meta_pte_lock, flags);
+
+ if (page)
+ __free_page(page);
+ return 0;
+}
+
+/* Allocate the page tables for the entries of [@start_pfn, @end_pfn). */
+static int pisa_meta_prealloc(struct pisa_meta *array, unsigned long nframes,
+ unsigned long start_pfn, unsigned long end_pfn)
+{
+ unsigned long start, end;
+
+ end_pfn = min(end_pfn, nframes << PISA_ORDER);
+ if (start_pfn >= end_pfn)
+ return 0;
+
+ start = round_down((unsigned long)&array[start_pfn >> PISA_ORDER], PAGE_SIZE);
+ end = round_up((unsigned long)&array[((end_pfn - 1) >> PISA_ORDER) + 1], PAGE_SIZE);
+ return apply_to_page_range(&init_mm, start, end - start, NULL, NULL);
+}
+
+struct pisa_meta_range {
+ struct pisa_meta *array;
+ unsigned long nframes;
+};
+
+static int pisa_meta_prealloc_ram(unsigned long start_pfn, unsigned long nr_pages, void *arg)
+{
+ struct pisa_meta_range *r = arg;
+
+ return pisa_meta_prealloc(r->array, r->nframes, start_pfn, start_pfn + nr_pages);
+}
+
+/* Page tables for memory about to go online; failure keeps it out of the allocators. */
+static int pisa_meta_memory_cb(struct notifier_block *nb, unsigned long action, void *arg)
+{
+ struct memory_notify *mn = arg;
+
+ if (action != MEM_GOING_ONLINE)
+ return NOTIFY_OK;
+
+ mutex_lock(&pisa_meta_mutex);
+ if (pisa_meta_array)
+ pisa_meta_prealloc(pisa_meta_array, pisa_meta_nframes, mn->start_pfn,
+ mn->start_pfn + mn->nr_pages);
+ mutex_unlock(&pisa_meta_mutex);
+ return NOTIFY_OK;
+}
+
+/* Memory hotplug notifier for preallocating metadata page tables. */
+static struct notifier_block pisa_meta_memory_nb = {
+ .notifier_call = pisa_meta_memory_cb,
+};
+
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block)
+{
+ unsigned long idx = pfn >> PISA_CHUNK_ORDER;
+
+ if (!static_branch_unlikely(&pisa_key) || (pfn >> PISA_ORDER) >= pisa_meta_nframes)
+ return -ERANGE;
+
+ if (likely(test_bit_acquire(idx, pisa_chunk_bitmap)))
+ return 0;
+
+ /* Memory onlined before the notifier was registered has no page table. */
+ if (can_block && !pisa_meta_pte((unsigned long)pisa_meta_array + (idx << PAGE_SHIFT))) {
+ int err;
+
+ mutex_lock(&pisa_meta_mutex);
+ err = pisa_meta_prealloc(pisa_meta_array, pisa_meta_nframes, pfn, pfn + 1);
+ mutex_unlock(&pisa_meta_mutex);
+ if (err)
+ return err;
+ }
+
+ return pisa_meta_back(pfn, can_block ? GFP_KERNEL : GFP_ATOMIC);
+}
+
+/* Reserve the sparse @pisa_meta_array and preallocate page tables for present RAM. */
+int pisa_meta_init(void)
+{
+ unsigned long top_pfn, nframes, nchunks, size;
+ struct pisa_meta_range range;
+ struct pisa_meta *array;
+ struct vm_struct *vm;
+ int ret = 0;
+
+ if (static_branch_unlikely(&pisa_key))
+ return 0;
+
+ mutex_lock(&pisa_meta_mutex);
+ if (pisa_meta_array)
+ goto out_unlock;
+
+ if (pisa_max_bits)
+ top_pfn = 1UL << (clamp_t(unsigned int, pisa_max_bits, PISA_SHIFT,
+ MAX_PHYSMEM_BITS) - PAGE_SHIFT);
+ else
+ top_pfn = max_t(unsigned long, max_pfn, max_possible_pfn);
+ nframes = DIV_ROUND_UP(top_pfn, 1UL << PISA_ORDER);
+ size = PAGE_ALIGN(nframes << PISA_META_SHIFT);
+ nchunks = size >> PAGE_SHIFT;
+
+ /* kvalloc() to avoid kmalloc limitations on large layouts */
+ pisa_chunk_bitmap = kvcalloc(BITS_TO_LONGS(nchunks), sizeof(unsigned long), GFP_KERNEL);
+ if (!pisa_chunk_bitmap) {
+ ret = -ENOMEM;
+ goto out_unlock;
+ }
+
+ vm = get_vm_area(size, VM_MAP);
+ if (!vm) {
+ ret = -ENOMEM;
+ goto out_free_bitmap;
+ }
+ array = vm->addr;
+
+ range.array = array;
+ range.nframes = nframes;
+ ret = walk_system_ram_range(0, nframes << PISA_ORDER, &range, pisa_meta_prealloc_ram);
+ if (ret)
+ goto out_free_vm;
+
+ pisa_meta_nframes = nframes;
+ pisa_meta_array = array;
+ static_branch_enable(&pisa_key);
+
+ pr_info("pisa: %lu frames, %lu KB of KVA\n", nframes, size / 1024);
+ mutex_unlock(&pisa_meta_mutex);
+
+ /* Register outside pisa_meta_mutex (taken by the notifier under chain lock). */
+ register_memory_notifier(&pisa_meta_memory_nb);
+ return 0;
+
+out_free_vm:
+ free_vm_area(vm);
+out_free_bitmap:
+ kvfree(pisa_chunk_bitmap);
+ pisa_chunk_bitmap = NULL;
+out_unlock:
+ mutex_unlock(&pisa_meta_mutex);
+ return ret;
+}
+EXPORT_SYMBOL(pisa_meta_init);
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
new file mode 100644
index 0000000000000..c873970f6a670
--- /dev/null
+++ b/drivers/iommu/pisa-priv.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
+#define _DRIVERS_IOMMU_PISA_PRIV_H
+
+#include <linux/pisa.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_PISA
+
+#define PISA_MAX_BLOCKS 512U
+
+#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+
+/**
+ * struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
+ * @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ */
+struct pisa_meta {
+ u8 flags;
+} __aligned(PISA_META_SIZE);
+
+static_assert(offsetof(struct pisa_meta, flags) == 0);
+static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
+unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+
+#endif /* CONFIG_PISA */
+#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
new file mode 100644
index 0000000000000..acf92ced245d8
--- /dev/null
+++ b/include/linux/pisa.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* See Documentation/core-api/pisa.rst for the architecture overview. */
+#ifndef _LINUX_PISA_H
+#define _LINUX_PISA_H
+
+#include <linux/bitops.h>
+#include <linux/jump_label.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_PISA
+
+#define PISA_SHIFT 21U
+#define PISA_SIZE (1UL << PISA_SHIFT)
+#define PISA_MASK (~(PISA_SIZE - 1))
+#define PISA_ORDER (PISA_SHIFT - PAGE_SHIFT)
+
+/*
+ * Each struct pisa_meta entry in pisa_meta_array has size PISA_META_SIZE
+ * (256 B) and starts with a u8 flags byte.
+ */
+#define PISA_META_SHIFT 8
+#define PISA_META_SIZE (1U << PISA_META_SHIFT)
+#define PISA_CHUNK_ORDER (PISA_ORDER + PAGE_SHIFT - PISA_META_SHIFT)
+
+enum {
+ PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
+};
+
+/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
+DECLARE_STATIC_KEY_FALSE(pisa_key);
+
+struct pisa_meta;
+extern struct pisa_meta *pisa_meta_array;
+extern unsigned long pisa_meta_nframes;
+extern unsigned long *pisa_chunk_bitmap;
+
+int pisa_meta_init(void);
+
+/* fast membership check for pisa pfn/pages */
+static inline bool is_pisa_pfn(unsigned long pfn)
+{
+ unsigned long frame = pfn >> PISA_ORDER;
+ const u8 *flags;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+ if (unlikely(frame >= pisa_meta_nframes ||
+ !test_bit_acquire(pfn >> PISA_CHUNK_ORDER, pisa_chunk_bitmap)))
+ return false;
+
+ flags = (const u8 *)pisa_meta_array + (frame << PISA_META_SHIFT);
+ return READ_ONCE(*flags) & PISA_ACTIVE;
+}
+
+#else /* !CONFIG_PISA */
+
+static inline int pisa_meta_init(void)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline bool is_pisa_pfn(unsigned long pfn)
+{
+ return false;
+}
+
+#endif /* CONFIG_PISA */
+
+static inline bool is_pisa_phys(phys_addr_t phys)
+{
+ return is_pisa_pfn(phys >> PAGE_SHIFT);
+}
+
+static inline bool is_pisa_page(const struct page *page)
+{
+ return is_pisa_pfn(page_to_pfn(page));
+}
+#endif /* _LINUX_PISA_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (26 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
` (22 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA allocators use 2MB physically contiguous memory pages from which IO
buffers are allocated. Recycling memory to the PISA allocator amortizes the
cost of mapping, unmapping and IOTLB flush. Memory is returned to the buddy
allocator only when all active IOMMU/IOTLB mappings have been flushed,
so sensitive data or code will not ever be accessible to IO devices.
Add the allocator creation (struct pisa_alloc, pisa_create()) and the
page release side, hooking into __free_pages_prepare() so that PISA
pages are returned to the allocator.
Also implement asynchronous PISA page retirement and release to the
buddy allocator via a WQ_MEM_RECLAIM worker. Retirement removes PISA
pages from allocators and destroys the mappings flushing the IOTLB, but
defers the buddy release by an RCU grace period.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 369 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 10 +-
drivers/iommu/pisa-priv.h | 85 +++++++++
include/linux/pisa.h | 28 ++-
mm/page_alloc.c | 21 +++
6 files changed, 512 insertions(+), 3 deletions(-)
create mode 100644 drivers/iommu/pisa-alloc.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index b5d3be1df3eda..bda0b102dd3ba 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
new file mode 100644
index 0000000000000..68f5d333ae15c
--- /dev/null
+++ b/drivers/iommu/pisa-alloc.c
@@ -0,0 +1,369 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA streaming page allocator, async page reclaim, shrinker, and debugfs.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-mapping.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#include "pisa-priv.h"
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Idle PISA pages over the allocator watermark queued locklessly for release. */
+static LLIST_HEAD(pisa_free_list);
+
+/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
+static struct workqueue_struct *pisa_wq __ro_after_init;
+
+/* Max completely idle PISA pages kept per allocator before async release. */
+static unsigned int pisa_max_idle __read_mostly = 1;
+module_param_named(max_idle, pisa_max_idle, uint, 0644);
+
+/* Grace period in ms before releasing idle PISA pages above @pisa_max_idle (0 = immediate). */
+static unsigned int pisa_idle_ms __read_mostly = 200;
+module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
+
+/* All live allocators, protected by pisa_allocs_lock. */
+static LIST_HEAD(pisa_allocs);
+static DEFINE_RWLOCK(pisa_allocs_lock);
+
+static void pisa_schedule_reclaim(void);
+
+/*
+ * Idle PISA page release proceeds in three stages:
+ * 1. Detach onto lockless pisa_free_list and schedule pisa_reclaim_work.
+ * 2. In process context (pisa_release_page()), unmap from all IOMMU domains,
+ * clear PISA_ACTIVE, and queue pisa_release_page_rcu() via call_rcu().
+ * 3. After an RCU grace period, pisa_release_page_rcu() unfreezes all blocks
+ * and returns them to the buddy allocator.
+ */
+
+/* RCU callback: unfreeze and return a retired PISA page's blocks to the buddy allocator. */
+static void pisa_release_page_rcu(struct rcu_head *head)
+{
+ /* Read @meta before freeing blocks; the frame can be reused immediately. */
+ struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
+ struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
+ unsigned int i, order = meta->pa->order;
+
+ WRITE_ONCE(meta->pa, NULL);
+
+ for (i = 0; i < PISA_BLOCKS(order); i++) {
+ block = page + (i << order);
+ page_ref_unfreeze(block, 1);
+ __free_pages(block, order);
+ }
+}
+
+/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
+static void pisa_release_page(struct pisa_meta *meta)
+{
+ /* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
+
+ /* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
+ WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
+ call_rcu(&meta->rcu, pisa_release_page_rcu);
+}
+
+/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
+enum pisa_meta_list {
+ PISA_LIST_NONE,
+ PISA_LIST_FULL,
+ PISA_LIST_IDLE_CLEAN, /* head of @idle */
+ PISA_LIST_IDLE_DIRTY, /* tail of @idle */
+ PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
+ PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
+ PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
+ PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
+};
+
+static void pisa_reclaim_work_fn(struct work_struct *work);
+
+/* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
+static DECLARE_DELAYED_WORK(pisa_reclaim_work, pisa_reclaim_work_fn);
+
+static void pisa_reclaim_work_fn(struct work_struct *work)
+{
+ unsigned long grace = msecs_to_jiffies(READ_ONCE(pisa_idle_ms));
+ unsigned long next_delay = MAX_JIFFY_OFFSET;
+ struct pisa_meta *meta, *tmp;
+ struct llist_node *node;
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ unsigned long expire = READ_ONCE(pa->idle_jiffies) + grace;
+ unsigned long now = jiffies, flags;
+
+ if (READ_ONCE(pa->num_idle_pages) <= pa->max_idle_pages)
+ continue;
+ if (grace && time_before(now, expire)) {
+ next_delay = min(next_delay, expire - now);
+ continue;
+ }
+ spin_lock_irqsave(&pa->lock, flags);
+ while (pa->num_idle_pages > pa->max_idle_pages && !list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ list_del_init(&meta->list);
+ WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+ llist_add(&meta->llnode, &pisa_free_list);
+ pa->num_idle_pages--;
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ node = llist_del_all(&pisa_free_list);
+ llist_for_each_entry_safe(meta, tmp, node, llnode)
+ pisa_release_page(meta);
+
+ if (next_delay != MAX_JIFFY_OFFSET)
+ queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
+}
+
+static void pisa_schedule_reclaim(void)
+{
+ struct workqueue_struct *wq = pisa_wq ?: system_wq;
+ unsigned int idle_ms = READ_ONCE(pisa_idle_ms);
+
+ if (!llist_empty(&pisa_free_list) || !idle_ms)
+ mod_delayed_work(wq, &pisa_reclaim_work, 0);
+ else
+ queue_delayed_work(wq, &pisa_reclaim_work, msecs_to_jiffies(idle_ms));
+}
+/*
+ * Update @meta's position on @pa's lists under @pa->lock. When all blocks
+ * become available (avail == bpp), either queues @meta to @pisa_free_list (and
+ * returns true so the caller schedules reclaim) or moves it to @pa->idle.
+ */
+static bool pisa_place_meta(struct pisa_meta *meta)
+{
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int avail = pisa_meta_avail(meta), bpp = PISA_BLOCKS(pa->order);
+ u8 on_list;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (avail == bpp) {
+ if (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms)) {
+ list_del_init(&meta->list);
+ WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+ llist_add(&meta->llnode, &pisa_free_list);
+ return true;
+ }
+ if (pa->num_idle_pages >= pa->max_idle_pages)
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ pa->num_idle_pages++;
+ on_list = meta->nr_dirty ? PISA_LIST_IDLE_DIRTY : PISA_LIST_IDLE_CLEAN;
+ } else if (!avail) {
+ on_list = PISA_LIST_FULL;
+ } else {
+ bool sparse;
+
+ if (avail >= bpp - (bpp >> 2))
+ sparse = true;
+ else if (avail <= (bpp >> 1))
+ sparse = false;
+ else
+ sparse = meta->on_list == PISA_LIST_PARTIAL_CLEAN_SPARSE ||
+ meta->on_list == PISA_LIST_PARTIAL_DIRTY_SPARSE;
+
+ if (meta->nr_clean)
+ on_list = sparse ? PISA_LIST_PARTIAL_CLEAN_SPARSE :
+ PISA_LIST_PARTIAL_CLEAN_DENSE;
+ else
+ on_list = sparse ? PISA_LIST_PARTIAL_DIRTY_SPARSE :
+ PISA_LIST_PARTIAL_DIRTY_DENSE;
+ }
+
+ if (on_list == meta->on_list)
+ return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
+ WRITE_ONCE(meta->on_list, on_list);
+
+ switch (on_list) {
+ case PISA_LIST_FULL:
+ list_move(&meta->list, &pa->full);
+ break;
+ case PISA_LIST_IDLE_CLEAN:
+ list_move(&meta->list, &pa->idle);
+ break;
+ case PISA_LIST_IDLE_DIRTY:
+ list_move_tail(&meta->list, &pa->idle);
+ break;
+ case PISA_LIST_PARTIAL_CLEAN_DENSE:
+ list_move(&meta->list, &pa->partial);
+ break;
+ case PISA_LIST_PARTIAL_CLEAN_SPARSE:
+ list_move_tail(&meta->list, &pa->partial);
+ break;
+ case PISA_LIST_PARTIAL_DIRTY_DENSE:
+ list_move(&meta->list, &pa->partial_dirty);
+ break;
+ default:
+ list_move_tail(&meta->list, &pa->partial_dirty);
+ }
+ return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
+}
+
+/* Check for hwpoisoned block; if poisoned, warn once and leak it to keep @pa intact. */
+static bool pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
+{
+ unsigned int i, nr = 1U << pa->order;
+
+ for (i = 0; i < nr; i++) {
+ if (unlikely(PageHWPoison(block + i))) {
+ pr_err_once("pisa: poisoned block at pfn %lu, leaking it to keep allocator %p intact\n",
+ page_to_pfn(block), pa);
+ return true;
+ }
+ }
+ return false;
+}
+
+/**
+ * pisa_create - Create a PISA page allocator
+ * @nid: Home NUMA node for this allocator
+ * @order: Block order to dispense (<= PISA_ORDER)
+ * @dir: Supported DMA direction (DMA_BIDIRECTIONAL, DMA_TO_DEVICE, or
+ * DMA_FROM_DEVICE); other mappings fall back to per-buffer IOMMU mappings
+ *
+ * Return: Pointer to the created allocator, or NULL on failure.
+ */
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir)
+{
+ struct pisa_alloc *pa;
+
+ if (order > PISA_ORDER || !valid_dma_direction(dir))
+ return NULL;
+
+ pa = kzalloc(sizeof(*pa), GFP_KERNEL);
+ if (!pa)
+ return NULL;
+
+ spin_lock_init(&pa->lock);
+ pa->order = order;
+ pa->nid = pisa_sanitize_nid(nid);
+ pa->max_idle_pages = READ_ONCE(pisa_max_idle) ? : 1;
+ pisa_set_name(pa, "%ps", (void *)_RET_IP_);
+ INIT_LIST_HEAD(&pa->partial);
+ INIT_LIST_HEAD(&pa->partial_dirty);
+ INIT_LIST_HEAD(&pa->idle);
+ INIT_LIST_HEAD(&pa->full);
+
+ write_lock(&pisa_allocs_lock);
+ list_add_tail(&pa->node, &pisa_allocs);
+ write_unlock(&pisa_allocs_lock);
+
+ return pa;
+}
+
+/**
+ * pisa_set_name - Name a PISA allocator for debugfs
+ * @pa: Allocator to name (may be NULL)
+ * @fmt: printf-style format, truncated to fit @pa->name
+ */
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...)
+{
+ va_list args;
+
+ if (!pa)
+ return;
+ va_start(args, fmt);
+ vsnprintf(pa->name, sizeof(pa->name), fmt, args);
+ va_end(args);
+}
+
+/* Return freed @block (at refcount 0) to @pa's bitmaps under @pa->lock. */
+static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block, bool *release)
+{
+ unsigned long pfn = page_to_pfn(block);
+ struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+ unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (WARN_ON_ONCE(test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+ return;
+
+ /* __pisa_free_page() zeroed it if init_on_free is set. */
+ if (want_init_on_free()) {
+ __set_bit(idx, meta->clean_bitmap);
+ meta->nr_clean++;
+ } else {
+ __set_bit(idx, meta->dirty_bitmap);
+ meta->nr_dirty++;
+ }
+ *release |= pisa_place_meta(meta);
+}
+
+/**
+ * __pisa_free_page - Recycle a freed block back into its PISA allocator
+ * @page: Block being freed (called from __free_pages_prepare() under RCU)
+ *
+ * Return: true if the page belonged to a PISA allocator and was recycled.
+ */
+bool __pisa_free_page(struct page *page)
+{
+ bool should_release = false;
+ unsigned long pfn = page_to_pfn(page), flags;
+ struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+ /* Reject double frees, mismatched orders, or unaligned blocks. */
+ if (WARN_ON_ONCE(!IS_ALIGNED(pfn, 1UL << pa->order) ||
+ compound_order(page) != pa->order ||
+ test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+ return true;
+
+ /* Never reuse or return a hwpoisoned block to the buddy allocator. */
+ if (pisa_block_poisoned(pa, page))
+ return true;
+
+ /* Checks skipped by the early return in __free_pages_prepare(). */
+ if (unlikely(!free_page_prepare_pisa(page, pa->order)))
+ return true;
+
+ /* KASAN/KMSAN/pgalloc_tag_sub run when pisa_release_page_rcu() frees to buddy. */
+ if (want_init_on_free())
+ memset(page_address(page), 0, PAGE_SIZE << pa->order);
+
+ spin_lock_irqsave(&pa->lock, flags);
+ pisa_return_block_locked(pa, page, &should_release);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return true;
+}
+
+static int __init pisa_init(void)
+{
+ /* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
+ pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
+ if (!pisa_wq)
+ pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+
+ return 0;
+}
+subsys_initcall(pisa_init);
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 0633f3238e012..12edc73abc9c6 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -85,10 +85,18 @@ static void test_meta_active_toggle(struct kunit *test)
__free_pages(page, PISA_ORDER);
}
+static void test_alloc_and_recycle(struct kunit *test)
+{
+ KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, 0, DMA_NONE));
+ KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, PISA_ORDER + 1, DMA_BIDIRECTIONAL));
+ pisa_set_name(NULL, "ignored");
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
+ KUNIT_CASE(test_alloc_and_recycle),
{}
};
@@ -98,5 +106,5 @@ static struct kunit_suite pisa_meta_test_suite = {
};
kunit_test_suite(pisa_meta_test_suite);
-MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table and PISA allocator");
MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index c873970f6a670..769b0ef071705 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -2,8 +2,13 @@
#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
#define _DRIVERS_IOMMU_PISA_PRIV_H
+#include <linux/dma-direction.h>
#include <linux/pisa.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/nodemask.h>
#include <linux/spinlock.h>
+#include <linux/topology.h>
#include <linux/types.h>
#ifdef CONFIG_PISA
@@ -12,20 +17,100 @@
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/*
+ * Lock hierarchy:
+ * - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
+ * - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
+ * - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ */
+
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ * @nr_clean: Zeroed available blocks in @clean_bitmap
+ * @nr_dirty: Dirty available blocks in @dirty_bitmap
+ * @on_list: Current placement in @pa (enum pisa_meta_list)
+ * @pa: Owning pisa_alloc (NULL for reservoir pages)
+ * @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
+ * @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
+ * @rcu: RCU head used to defer release to the buddy allocator
+ * @clean_bitmap: Bitmap of zeroed available block indices
+ * @dirty_bitmap: Bitmap of dirty available block indices
*/
struct pisa_meta {
u8 flags;
+ u16 nr_clean;
+ u16 nr_dirty;
+ u8 on_list;
+ struct pisa_alloc *pa;
+ struct list_head list;
+
+ union {
+ struct llist_node llnode;
+ struct rcu_head rcu;
+ };
+
+ DECLARE_BITMAP(clean_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
+ DECLARE_BITMAP(dirty_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
} __aligned(PISA_META_SIZE);
static_assert(offsetof(struct pisa_meta, flags) == 0);
static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
+{
+ return meta->nr_clean + meta->nr_dirty;
+}
+
+/**
+ * struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
+ * @lock: Spinlock protecting lists and block bitmaps
+ * @order: Block order managed by this allocator (<= PISA_ORDER)
+ * @partial: Partially used PISA pages with @nr_clean > 0
+ * @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
+ * @idle: Completely free PISA pages (reclaimable under pressure)
+ * @full: List of PISA pages with 0 free blocks
+ * @num_idle_pages: Length of @idle
+ * @max_idle_pages: High watermark of @idle before async release to the buddy allocator
+ * @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @nid: NUMA node id
+ * @node: Node in the global pisa_allocs list
+ * @name: Debug name (defaults to caller of pisa_create())
+ */
+struct pisa_alloc {
+ /* First cacheline: hot fields touched on block alloc and free. */
+ spinlock_t lock;
+ u8 order;
+ struct list_head partial;
+ struct list_head partial_dirty;
+ struct list_head idle;
+ struct list_head full;
+ unsigned int num_idle_pages;
+ unsigned int max_idle_pages;
+ unsigned long idle_jiffies;
+
+ /* Cold. */
+ int nid;
+ struct list_head node;
+ char name[48];
+} ____cacheline_aligned;
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
+__printf(2, 3)
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+
+static inline int pisa_sanitize_nid(int nid)
+{
+ if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
+ nid = numa_mem_id();
+ if (!node_state(nid, N_MEMORY))
+ nid = first_memory_node;
+ return nid;
+}
+
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index acf92ced245d8..dddcedb8c93f6 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -3,10 +3,11 @@
#ifndef _LINUX_PISA_H
#define _LINUX_PISA_H
+#include <linux/types.h>
#include <linux/bitops.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
-#include <linux/types.h>
+#include <linux/rcupdate.h>
#ifdef CONFIG_PISA
@@ -53,6 +54,24 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return READ_ONCE(*flags) & PISA_ACTIVE;
}
+static inline bool is_pisa_page(const struct page *page);
+
+bool __pisa_free_page(struct page *page);
+bool free_page_prepare_pisa(struct page *page, unsigned int order);
+
+static inline bool pisa_free_page(struct page *page)
+{
+ bool ret;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+ rcu_read_lock();
+ ret = is_pisa_page(page) && __pisa_free_page(page);
+ rcu_read_unlock();
+
+ return ret;
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -60,11 +79,18 @@ static inline int pisa_meta_init(void)
return -EOPNOTSUPP;
}
+bool free_page_prepare_pisa(struct page *page, unsigned int order);
+
static inline bool is_pisa_pfn(unsigned long pfn)
{
return false;
}
+static inline bool pisa_free_page(struct page *page)
+{
+ return false;
+}
+
#endif /* CONFIG_PISA */
static inline bool is_pisa_phys(phys_addr_t phys)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index eb4f02617f53c..48df87a19d312 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -17,6 +17,7 @@
#include <linux/stddef.h>
#include <linux/mm.h>
#include <linux/highmem.h>
+#include <linux/pisa.h>
#include <linux/interrupt.h>
#include <linux/jiffies.h>
#include <linux/compiler.h>
@@ -1298,6 +1299,23 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
#endif /* CONFIG_MEM_ALLOC_PROFILING */
+/*
+ * pisa_free_page() in __free_pages_prepare() recycles a PISA block into
+ * its allocator before the rest of that function runs. __pisa_free_page() calls
+ * this to apply the buddy allocator's checks.
+ *
+ * Return: false if the block is in a bad state (already reported): leak it.
+ */
+bool free_page_prepare_pisa(struct page *page, unsigned int order)
+{
+ if (!IS_ENABLED(CONFIG_PISA))
+ return true;
+ if (is_check_pages_enabled() && free_page_is_bad(page))
+ return false;
+ page_table_check_free(page, order);
+ return true;
+}
+
static __always_inline bool __free_pages_prepare(struct page *page,
unsigned int order, fpi_t fpi_flags)
{
@@ -1312,6 +1330,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
VM_BUG_ON_PAGE(PageTail(page), page);
+ if (unlikely(pisa_free_page(page)))
+ return false;
+
trace_mm_page_free(page, order);
kmsan_free_page(page, order);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (27 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
` (21 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Allocating a large page may involve compacting and blocking, so it cannot
be issued in GFP_ATOMIC context.
Add a per-NUMA-node reservoir of pre-allocated and set up PISA pages,
that can serve allocations without blocking. The reservoir is refilled in
process context by a per-node work item, queued with queue_work_node()
on system_dfl_wq so each node is refilled from its own CPUs, not bound
to the CPU that caused the miss.
pisa.reservoir_max, writable at runtime, sets the number of PISA pages
per node (default 16 = 32 MB per node; 0 = disabled).
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 17 ++-
drivers/iommu/pisa-kunit.c | 7 +
drivers/iommu/pisa-meta.c | 1 +
drivers/iommu/pisa-node.c | 268 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 30 +++++
6 files changed, 323 insertions(+), 2 deletions(-)
create mode 100644 drivers/iommu/pisa-node.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index bda0b102dd3ba..bc58166f9303a 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 68f5d333ae15c..cf559ee86d6f9 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -46,6 +46,14 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+struct pisa_node pisa_nodes[MAX_NUMNODES] = {
+ [0 ... MAX_NUMNODES - 1] = {
+ .next_alloc = INITIAL_JIFFIES,
+ .lock = __SPIN_LOCK_UNLOCKED(pisa_nodes.lock),
+ },
+};
+
static void pisa_schedule_reclaim(void);
/*
@@ -75,8 +83,13 @@ static void pisa_release_page_rcu(struct rcu_head *head)
}
/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
-static void pisa_release_page(struct pisa_meta *meta)
+void pisa_release_page(struct pisa_meta *meta)
{
+ if (!meta->pa) {
+ __free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
+ return;
+ }
+
/* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
@@ -359,6 +372,8 @@ bool __pisa_free_page(struct page *page)
static int __init pisa_init(void)
{
+ pisa_node_init();
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 12edc73abc9c6..079f6220e742a 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -92,11 +92,18 @@ static void test_alloc_and_recycle(struct kunit *test)
pisa_set_name(NULL, "ignored");
}
+static void test_reservoir_and_scrubber(struct kunit *test)
+{
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+ KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
KUNIT_CASE(test_alloc_and_recycle),
+ KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
index 5ceea91e4d5e8..00939c366ba6d 100644
--- a/drivers/iommu/pisa-meta.c
+++ b/drivers/iommu/pisa-meta.c
@@ -272,6 +272,7 @@ int pisa_meta_init(void)
/* Register outside pisa_meta_mutex (taken by the notifier under chain lock). */
register_memory_notifier(&pisa_meta_memory_nb);
+ pisa_schedule_reservoir();
return 0;
out_free_vm:
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
new file mode 100644
index 0000000000000..524f0d4457ad7
--- /dev/null
+++ b/drivers/iommu/pisa-node.c
@@ -0,0 +1,268 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-NUMA-node state: PISA page reservoir and background work.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/pisa.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/nodemask.h>
+#include <linux/sched.h>
+#include <linux/sched/mm.h>
+#include <linux/spinlock.h>
+#include <linux/topology.h>
+#include <linux/workqueue.h>
+
+#include "pisa-priv.h"
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/*
+ * Per-NUMA-node reservoir of pre-allocated, pre-zeroed, and pre-decrypted
+ * PISA pages (pisa.reservoir_max, default 16, 0 = disabled). Refilled and
+ * trimmed by each node's work item on system_dfl_wq, which also scrubs
+ * allocators on that node.
+ */
+unsigned int pisa_reservoir_max __read_mostly = 16;
+
+static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ /* Schedule per-node workers to refill or trim once pisa_meta_init() has run. */
+ if (!ret && static_branch_unlikely(&pisa_key))
+ pisa_schedule_reservoir();
+ return ret;
+}
+
+static const struct kernel_param_ops pisa_reservoir_max_ops = {
+ .set = pisa_reservoir_max_set,
+ .get = param_get_uint,
+};
+module_param_cb(reservoir_max, &pisa_reservoir_max_ops, &pisa_reservoir_max, 0644);
+
+/* Apply GFP_NOFS/GFP_NOIO scope around internal GFP_KERNEL allocations. */
+static unsigned int pisa_memalloc_save(gfp_t gfp)
+{
+ unsigned int flags = 0;
+
+ if (!(gfp & __GFP_FS))
+ flags |= PF_MEMALLOC_NOFS;
+ if (!(gfp & __GFP_IO))
+ flags |= PF_MEMALLOC_NOIO;
+ return memalloc_flags_save(flags);
+}
+
+/* Allocate a PISA page from the buddy allocator and initialize its metadata. */
+static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
+ bool *clean)
+{
+ struct pisa_meta *meta;
+ unsigned int memflags;
+ struct page *page;
+
+ if (unlikely(!static_branch_unlikely(&pisa_key))) {
+ if (!can_block || (gfp & (__GFP_FS | __GFP_IO)) != (__GFP_FS | __GFP_IO) ||
+ pisa_meta_init())
+ return ERR_PTR(-EAGAIN);
+ }
+
+ /*
+ * Long-lived active pages must be unmovable, lowmem, non-compound, and
+ * must not dip into emergency reserves or trigger OOM/warnings.
+ */
+ gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+ gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
+
+ if (can_block)
+ memflags = pisa_memalloc_save(gfp);
+ page = alloc_pages_node(nid, gfp, PISA_ORDER);
+ if (!page) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ /* Ensure the metadata chunk for this PFN is populated. */
+ if (unlikely(pisa_meta_ensure_pfn(page_to_pfn(page), can_block))) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ __free_pages(page, PISA_ORDER);
+ return ERR_PTR(-ERANGE);
+ }
+ if (can_block)
+ memalloc_flags_restore(memflags);
+
+ meta = pisa_meta_of_pfn(page_to_pfn(page));
+ memset(meta, 0, sizeof(*meta));
+ *clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+
+ return meta;
+}
+
+
+/* Refill node @nid's reservoir up to pisa_reservoir_max pre-zeroed PISA pages. */
+static void pisa_reservoir_refill_node(int nid)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ while (res->nr_pages < READ_ONCE(pisa_reservoir_max)) {
+ struct pisa_meta *meta;
+ unsigned long flags;
+ bool clean;
+
+ meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+ if (IS_ERR(meta))
+ break;
+
+ /* init_on_alloc/init_on_free pages are already zeroed. */
+ if (!clean)
+ memset(page_address(pfn_to_page(pisa_meta_to_pfn(meta))), 0, PISA_SIZE);
+
+ spin_lock_irqsave(&res->lock, flags);
+ list_add(&meta->list, &res->pages);
+ res->nr_pages++;
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ cond_resched();
+ }
+}
+
+/* Release PISA pages in node @nid's reservoir above @max. */
+static void pisa_reservoir_trim_node(int nid, unsigned int max)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+ struct pisa_meta *meta, *tmp;
+ LIST_HEAD(free_list);
+ unsigned long flags;
+
+ spin_lock_irqsave(&res->lock, flags);
+ while (res->nr_pages > max && !list_empty(&res->pages)) {
+ list_move(res->pages.prev, &free_list);
+ res->nr_pages--;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ list_for_each_entry_safe(meta, tmp, &free_list, list) {
+ list_del_init(&meta->list);
+ pisa_release_page(meta);
+ cond_resched();
+ }
+}
+
+static void pisa_node_work_fn(struct work_struct *work)
+{
+ struct pisa_node *res = container_of(work, struct pisa_node, work);
+ int nid = res - pisa_nodes;
+ unsigned int max = READ_ONCE(pisa_reservoir_max);
+
+ if (unlikely(pisa_meta_init()))
+ return;
+
+ if (READ_ONCE(res->nr_pages) < max)
+ pisa_reservoir_refill_node(nid);
+ else if (READ_ONCE(res->nr_pages) > max)
+ pisa_reservoir_trim_node(nid, max);
+}
+
+/* Queue @nid's reservoir worker on a CPU belonging to @nid. */
+static void pisa_schedule_reservoir_node(int nid)
+{
+ if (nid >= 0 && nid < MAX_NUMNODES && node_state(nid, N_MEMORY))
+ queue_work_node(nid, system_dfl_wq, &pisa_nodes[nid].work);
+}
+
+/* Queue reservoir refill/trim workers on all memory nodes. */
+void pisa_schedule_reservoir(void)
+{
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ pisa_schedule_reservoir_node(nid);
+}
+
+/* Pop one pre-prepared PISA page from node @nid's reservoir under @res->lock. */
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+ struct pisa_meta *meta;
+ unsigned long flags;
+ bool refill = false;
+
+ if (!READ_ONCE(res->nr_pages))
+ return NULL;
+
+ spin_lock_irqsave(&res->lock, flags);
+ meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
+ if (meta) {
+ list_del_init(&meta->list);
+ res->nr_pages--;
+ refill = res->nr_pages <= max / 2;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ if (refill)
+ pisa_schedule_reservoir_node(nid);
+
+ return meta;
+}
+
+/*
+ * Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
+ * refill when <= half full or empty), or allocate one inline on miss.
+ */
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+{
+ bool can_block = gfpflags_allow_blocking(gfp);
+ unsigned int max = READ_ONCE(pisa_reservoir_max);
+ struct pisa_meta *meta;
+
+ nid = pisa_sanitize_nid(nid);
+ if (max) {
+ meta = pisa_reservoir_pop_node(nid, max);
+ if (meta) {
+ *clean = true;
+ return meta;
+ }
+ /* Empty: pop_node() only schedules after a successful pop. */
+ pisa_schedule_reservoir_node(nid);
+ }
+
+ /* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
+ if (!can_block && time_before(jiffies, READ_ONCE(pisa_nodes[nid].next_alloc)))
+ return ERR_PTR(-EAGAIN);
+
+ meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+ if (IS_ERR(meta))
+ pisa_alloc_backoff(nid, can_block);
+ return meta;
+}
+
+/* Number of PISA pages held in all the reservoirs. */
+unsigned long pisa_reservoir_count(void)
+{
+ unsigned long nr = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ nr += READ_ONCE(pisa_nodes[nid].nr_pages);
+ return nr;
+}
+
+void __init pisa_node_init(void)
+{
+ int nid;
+
+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
+ spin_lock_init(&pisa_nodes[nid].lock);
+ INIT_LIST_HEAD(&pisa_nodes[nid].pages);
+ INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
+ }
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 769b0ef071705..0d097c665c790 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -4,12 +4,14 @@
#include <linux/dma-direction.h>
#include <linux/pisa.h>
+#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/nodemask.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
#include <linux/types.h>
+#include <linux/workqueue.h>
#ifdef CONFIG_PISA
@@ -95,6 +97,19 @@ struct pisa_alloc {
char name[48];
} ____cacheline_aligned;
+/* Per-NUMA-node PISA state. */
+struct pisa_node {
+ unsigned long next_alloc; /* atomic alloc backoff jiffies */
+
+ spinlock_t lock; /* protects @pages and @nr_pages */
+ struct list_head pages; /* reservoir */
+ unsigned int nr_pages;
+ struct work_struct work; /* refill/trim, on this node */
+} ____cacheline_aligned_in_smp;
+
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
@@ -102,6 +117,7 @@ unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
__printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+void pisa_release_page(struct pisa_meta *meta);
static inline int pisa_sanitize_nid(int nid)
{
@@ -112,5 +128,19 @@ static inline int pisa_sanitize_nid(int nid)
return nid;
}
+static inline void pisa_alloc_backoff(int nid, bool can_block)
+{
+ nid = pisa_sanitize_nid(nid);
+ if (!can_block)
+ WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
+}
+
+/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
+extern unsigned int pisa_reservoir_max;
+void pisa_node_init(void);
+void pisa_schedule_reservoir(void);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+unsigned long pisa_reservoir_count(void);
+
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (28 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
` (20 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Add block allocation (pisa_alloc_bulk(), pisa_alloc())
and split_page_compound(), which splits a newly acquired 2MB PISA page
into blocks of the allocator's order, frozen at refcount 0.
Introduce __GFP_PISA_RO and __GFP_PISA_RW GFP flags and hook
__alloc_frozen_pages_noprof() and alloc_pages_bulk_noprof() so callers
can allocate from per-node PISA allocators (pisa_nodes[nid].pa[ro][order])
using standard page allocation APIs.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 345 +++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 78 +++++++-
drivers/iommu/pisa-priv.h | 24 ++-
include/linux/gfp_types.h | 17 +-
include/linux/pisa.h | 25 +++
include/trace/events/mmflags.h | 11 +-
mm/page_alloc.c | 40 +++-
7 files changed, 527 insertions(+), 13 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index cf559ee86d6f9..1f1b99023094c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -14,13 +14,19 @@
#include <linux/gfp.h>
#include <linux/list.h>
#include <linux/llist.h>
+#include <linux/memcontrol.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/mutex.h>
+#include <linux/page_owner.h>
+#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
+#include <linux/sched/mm.h>
+#include <linux/skbuff.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
+#include <net/sock.h>
#include "pisa-priv.h"
@@ -328,6 +334,318 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
*release |= pisa_place_meta(meta);
}
+/*
+ * Return the predefined per-node allocator for (@nid, @order, @ro), or NULL.
+ */
+struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro)
+{
+ if (order > PISA_ORDER)
+ return NULL;
+ nid = pisa_sanitize_nid(nid);
+
+ return pisa_nodes[nid].pa[ro][order];
+}
+
+/**
+ * pisa_alloc_pages_bulk - Allocate pages from a per-node __GFP_PISA_* allocator
+ * @gfp: GFP flags (must have __GFP_PISA_RW or __GFP_PISA_RO set)
+ * @order: Page order
+ * @preferred_nid: Preferred NUMA node (or NUMA_NO_NODE)
+ * @nodemask: Allowed NUMA nodemask (declined if non-NULL)
+ * @nr_pages: Total entries in @page_array
+ * @page_array: Output array of page pointers (populated entries are skipped)
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array, bool frozen)
+{
+ struct pisa_alloc *pa = NULL;
+
+ if (unlikely(nr_pages <= 0))
+ return 0;
+ if (likely(!nodemask) && (!order || (gfp & __GFP_COMP)))
+ pa = pisa_node_pa(preferred_nid, order, !(gfp & __GFP_WRITE));
+ return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
+}
+
+/*
+ * Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
+ * pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
+ * Fails with -EBUSY if a speculative PFN walker holds a reference to @page.
+ */
+static int split_page_compound(struct page *page, unsigned int old_order, unsigned int new_order)
+{
+ unsigned int i;
+
+ if (WARN_ON_ONCE(PageCompound(page) || !page_count(page) ||
+ new_order > old_order ||
+ folio_memcg_charged(page_folio(page))))
+ return -EINVAL;
+
+ /* Freeze @page[0] (1 -> 0) so all resulting pieces start at refcount 0. */
+ if (!page_ref_freeze(page, 1))
+ return -EBUSY;
+
+ if (new_order < old_order) {
+ split_page_owner(page, old_order, new_order);
+ pgalloc_tag_split(page_folio(page), old_order, new_order);
+ }
+
+ /* Subpages of non-compound pages are already initialized as order 0. */
+ if (new_order == 0)
+ return 0;
+
+ for (i = 0; i < (1U << old_order); i += (1U << new_order))
+ prep_compound_page(page + i, new_order);
+
+ return 0;
+}
+
+/* Obtain and split a new PISA page on @nid into blocks of @order. */
+static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, int nid)
+{
+ bool can_block = gfpflags_allow_blocking(gfp), clean;
+ unsigned int bpp = PISA_BLOCKS(order);
+ struct pisa_meta *meta;
+ struct page *page;
+
+ /* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
+ meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+ if (IS_ERR(meta))
+ return meta;
+
+ page = pfn_to_page(pisa_meta_to_pfn(meta));
+
+ /* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
+ if (split_page_compound(page, PISA_ORDER, order)) {
+ pisa_release_page(meta);
+ pisa_alloc_backoff(nid, can_block);
+ return ERR_PTR(-EBUSY);
+ }
+
+ INIT_LIST_HEAD(&meta->list);
+
+ if (clean) {
+ bitmap_set(meta->clean_bitmap, 0, bpp);
+ meta->nr_clean = bpp;
+ } else {
+ bitmap_set(meta->dirty_bitmap, 0, bpp);
+ meta->nr_dirty = bpp;
+ }
+
+ return meta;
+}
+
+/* Reset fields normally cleared by prep_new_page()/post_alloc_hook(). */
+static void pisa_block_prep(struct page *block)
+{
+ set_page_private(block, 0);
+ clear_page_pfmemalloc(block);
+}
+
+/*
+ * Collect up to @want blocks from @meta's dirty or clean bitmap. With
+ * @unfreeze they get refcount 1; otherwise they stay at refcount 0.
+ */
+static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
+ unsigned int want, struct page **out, bool unfreeze)
+{
+ unsigned long *bitmap = dirty ? meta->dirty_bitmap : meta->clean_bitmap;
+ u16 *countp = dirty ? &meta->nr_dirty : &meta->nr_clean;
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int idx, got = 0, used = 0;
+ unsigned long base_pfn;
+
+ if (!*countp || !want)
+ return 0;
+
+ base_pfn = pisa_meta_to_pfn(meta);
+ for_each_set_bit(idx, bitmap, PISA_BLOCKS(pa->order)) {
+ struct page *block = pfn_to_page(base_pfn + (idx << pa->order));
+
+ __clear_bit(idx, bitmap);
+ used++;
+ if (!pisa_block_poisoned(pa, block)) {
+ pisa_block_prep(block);
+ if (unfreeze)
+ page_ref_unfreeze(block, 1);
+ out[got++] = block;
+ }
+
+ if (used == *countp || got == want)
+ break;
+ }
+ *countp -= used;
+ return got;
+}
+
+/* @zero: prefer PISA pages with blocks in @meta->clean_bitmap. */
+static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
+{
+ struct pisa_meta *m_dirty, *m_clean, *meta;
+
+ if (zero) {
+ /* Clean blocks first: the head of @partial, then of @idle. */
+ if (!list_empty(&pa->partial))
+ return list_first_entry(&pa->partial, struct pisa_meta, list);
+ meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
+ if (meta && meta->nr_clean) {
+ pa->num_idle_pages--;
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ return meta;
+ }
+ }
+
+ m_dirty = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+ m_clean = list_first_entry_or_null(&pa->partial, struct pisa_meta, list);
+ if (m_dirty && m_clean) {
+ bool dirty_dense = m_dirty->on_list == PISA_LIST_PARTIAL_DIRTY_DENSE;
+ bool clean_dense = m_clean->on_list == PISA_LIST_PARTIAL_CLEAN_DENSE;
+
+ if (dirty_dense != clean_dense)
+ return dirty_dense ? m_dirty : m_clean;
+ return pisa_meta_avail(m_dirty) <= pisa_meta_avail(m_clean) ?
+ m_dirty : m_clean;
+ }
+ if (m_dirty)
+ return m_dirty;
+ if (m_clean)
+ return m_clean;
+ if (!list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ pa->num_idle_pages--;
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ return meta;
+ }
+ return NULL;
+}
+
+/* Extract up to @want blocks from @meta under @pa->lock. */
+static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
+ struct page **out, bool zero, bool frozen)
+{
+ unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
+ struct pisa_alloc *pa = meta->pa;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (zero) {
+ unsigned int dirty, i;
+
+ /* First choice are clean blocks, followed by dirty */
+ got += pisa_scan_bitmap(meta, false, want, out, !frozen);
+ dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
+ for (i = got; i < got + dirty; i++)
+ set_page_private(out[i], 1);
+ got += dirty;
+ } else {
+ /* First choice are dirty blocks, followed by clean */
+ got += pisa_scan_bitmap(meta, true, want, out, !frozen);
+ got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+ }
+
+ used = avail_before - pisa_meta_avail(meta);
+ /* If counts disagreed with bitmaps, zero them so @meta moves to @pa->full. */
+ if (WARN_ON_ONCE(!used)) {
+ meta->nr_clean = 0;
+ meta->nr_dirty = 0;
+ }
+ pisa_place_meta(meta);
+
+ return got;
+}
+
+/* Allocate @nr_pages blocks from @pa. */
+static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+ struct page **pages, bool zero, bool frozen)
+{
+ unsigned long n, flags, i;
+ struct pisa_meta *meta;
+ unsigned int want;
+
+ /* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
+ spin_lock_irqsave(&pa->lock, flags);
+ for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+ want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
+ meta = pisa_pick_meta(pa, zero);
+ if (meta)
+ continue;
+
+ /* Fallback to a new allocation */
+ spin_unlock_irqrestore(&pa->lock, flags);
+ /* With @zero, fresh dirty blocks are zeroed below. */
+ meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
+ if (IS_ERR(meta))
+ goto out;
+ spin_lock_irqsave(&pa->lock, flags);
+ meta->pa = pa;
+ /* Set PISA_ACTIVE after @meta is fully set up. */
+ WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+out:
+ for (i = 0; zero && i < n; i++) {
+ /* Dirty blocks for a zeroing caller were marked with page_private = 1. */
+ if (!page_private(pages[i]))
+ continue;
+ set_page_private(pages[i], 0);
+ memset(page_address(pages[i]), 0, PAGE_SIZE << pa->order);
+ if (frozen) /* Order zeroing before caller publishes the page. */
+ smp_wmb();
+ else
+ page_ref_unfreeze(pages[i], 1);
+ }
+ return n;
+}
+
+/**
+ * pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
+ * @pa: Allocator to allocate from
+ * @gfp: GFP flags (requests with __GFP_THISNODE, __GFP_DMA, __GFP_DMA32, or
+ * __GFP_ACCOUNT are declined)
+ * @nr_pages: Number of entries in @page_array (populated entries are skipped)
+ * @page_array: Output array of page pointers
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+ unsigned long nr_pages, struct page **page_array, bool frozen)
+{
+ unsigned long i = 0, n, got;
+
+ if (unlikely(!pa ||
+ (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+ goto skip;
+
+ while (i < nr_pages) {
+ if (page_array[i]) {
+ i++;
+ continue;
+ }
+ /* Fill the run of @n NULL entries at @i. */
+ for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
+ ;
+ got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+ want_init_on_alloc(gfp), frozen);
+ i += got;
+ if (got < n)
+ break;
+ }
+ return i;
+
+skip:
+ /* Declined: report the populated prefix, as alloc_pages_bulk_node() does. */
+ while (i < nr_pages && page_array[i])
+ i++;
+ return i;
+}
+
+
/**
* __pisa_free_page - Recycle a freed block back into its PISA allocator
* @page: Block being freed (called from __free_pages_prepare() under RCU)
@@ -372,8 +690,35 @@ bool __pisa_free_page(struct page *page)
static int __init pisa_init(void)
{
+ const struct {
+ u8 order;
+ bool ro;
+ } cfg[] = {
+ { 0, false },
+ { 2, true },
+ { SKB_FRAG_PAGE_ORDER, true },
+ };
+ int nid, i;
+
pisa_node_init();
+ /* Preallocate the supported per-node allocators. */
+ for_each_node_state(nid, N_MEMORY) {
+ for (i = 0; i < ARRAY_SIZE(cfg); i++) {
+ unsigned int order = cfg[i].order;
+ bool ro = cfg[i].ro;
+ struct pisa_alloc *pa;
+
+ if (order > PISA_ORDER || pisa_nodes[nid].pa[ro][order])
+ continue;
+ pa = pisa_create(nid, order, ro ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL);
+ if (!pa)
+ continue;
+ pisa_set_name(pa, "pisa%s_o%u_n%d", ro ? "_ro" : "", order, nid);
+ pisa_nodes[nid].pa[ro][order] = pa;
+ }
+ }
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 079f6220e742a..bdc52e6497b05 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,8 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/mm.h>
+#include <linux/skbuff.h>
+#include <net/sock.h>
#include "pisa-priv.h"
@@ -28,6 +30,7 @@ static void test_meta_init_and_roundtrip(struct kunit *test)
KUNIT_ASSERT_EQ(test, pisa_meta_ensure_pfn(pfn, true), 0);
m_pfn = pisa_meta_of_pfn(pfn);
+ KUNIT_EXPECT_PTR_EQ(test, pisa_meta_of_pfn(base_pfn), m_pfn);
KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
}
@@ -37,7 +40,7 @@ static void test_meta_invalid_phys(struct kunit *test)
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
- meta = pisa_meta_of_pfn(PHYS_PFN(PHYS_ADDR_MAX));
+ meta = pisa_meta_of_pfn(ULONG_MAX >> PAGE_SHIFT);
KUNIT_ASSERT_NOT_NULL(test, meta);
KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(meta), ULONG_MAX);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(ULONG_MAX >> PAGE_SHIFT));
@@ -87,9 +90,78 @@ static void test_meta_active_toggle(struct kunit *test)
static void test_alloc_and_recycle(struct kunit *test)
{
- KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, 0, DMA_NONE));
- KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, PISA_ORDER + 1, DMA_BIDIRECTIONAL));
+ const struct {
+ unsigned int order;
+ bool ro;
+ } cases[] = {
+ { 0, false },
+ { 2, true },
+ { SKB_FRAG_PAGE_ORDER, true },
+ };
+ struct page *p1, *p2, *bulk[3];
+ struct pisa_alloc *pa;
+ unsigned int i;
+
+ KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, 0, DMA_NONE));
+ KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, PISA_ORDER + 1,
+ DMA_BIDIRECTIONAL));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 1, false));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 0, true));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, PISA_ORDER + 1, false));
pisa_set_name(NULL, "ignored");
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ unsigned int order = cases[i].order;
+
+ pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
+
+ p1 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p1));
+
+ /* Freeing p1 returns it to the allocator; next allocation reuses index 0. */
+ __free_pages(p1, order);
+ p2 = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p2);
+ if (!want_init_on_alloc(GFP_KERNEL) || want_init_on_free() || order == PISA_ORDER)
+ KUNIT_EXPECT_PTR_EQ(test, p1, p2);
+
+ if (order == 0) {
+ /* Bulk alloc skips already populated entries. */
+ bulk[0] = NULL;
+ bulk[1] = p2;
+ bulk[2] = NULL;
+ KUNIT_EXPECT_EQ(test,
+ pisa_alloc_bulk(pa, GFP_KERNEL, 3, bulk, false),
+ 3UL);
+ KUNIT_EXPECT_PTR_EQ(test, bulk[1], p2);
+ KUNIT_EXPECT_NOT_NULL(test, bulk[0]);
+ KUNIT_EXPECT_NOT_NULL(test, bulk[2]);
+ /* Declined GFP reports populated prefix length. */
+ KUNIT_EXPECT_EQ(test,
+ pisa_alloc_bulk(pa,
+ GFP_KERNEL | __GFP_ACCOUNT,
+ 3, bulk, false),
+ 3UL);
+ if (bulk[0])
+ __free_pages(bulk[0], 0);
+ if (bulk[2])
+ __free_pages(bulk[2], 0);
+ }
+
+ if (p2)
+ __free_pages(p2, order);
+ }
+
+ /* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
+ p1 = alloc_page(GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
+ __free_pages(p1, 0);
}
static void test_reservoir_and_scrubber(struct kunit *test)
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 0d097c665c790..f15934ccdeea4 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -100,6 +100,7 @@ struct pisa_alloc {
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
+ struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -114,11 +115,6 @@ int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
-struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
-__printf(2, 3)
-void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
-void pisa_release_page(struct pisa_meta *meta);
-
static inline int pisa_sanitize_nid(int nid)
{
if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
@@ -135,6 +131,24 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
+/* pisa-alloc.c, also used by the per-node code in pisa-node.c */
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
+struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro);
+__printf(2, 3)
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+void pisa_release_page(struct pisa_meta *meta);
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+ unsigned long nr_pages, struct page **page_array, bool frozen);
+
+static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
+{
+ struct page *page = NULL;
+
+ if (pa)
+ pisa_alloc_bulk(pa, gfp, 1, &page, false);
+ return page;
+}
+
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h
index c1331bf0b2398..65fbfca3e08d1 100644
--- a/include/linux/gfp_types.h
+++ b/include/linux/gfp_types.h
@@ -33,7 +33,7 @@ enum {
___GFP_IO_BIT,
___GFP_FS_BIT,
___GFP_ZERO_BIT,
- ___GFP_UNUSED_BIT, /* 0x200u unused */
+ ___GFP_PISA_RO_BIT,
___GFP_DIRECT_RECLAIM_BIT,
___GFP_KSWAPD_RECLAIM_BIT,
___GFP_WRITE_BIT,
@@ -68,7 +68,11 @@ enum {
#define ___GFP_IO BIT(___GFP_IO_BIT)
#define ___GFP_FS BIT(___GFP_FS_BIT)
#define ___GFP_ZERO BIT(___GFP_ZERO_BIT)
-/* 0x200u unused */
+#ifdef CONFIG_PISA
+#define ___GFP_PISA_RO BIT(___GFP_PISA_RO_BIT)
+#else
+#define ___GFP_PISA_RO 0
+#endif
#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT)
#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT)
#define ___GFP_WRITE BIT(___GFP_WRITE_BIT)
@@ -95,6 +99,11 @@ enum {
#else
#define ___GFP_NOLOCKDEP 0
#endif
+#ifdef CONFIG_PISA
+#define ___GFP_PISA_RW (___GFP_PISA_RO | ___GFP_WRITE)
+#else
+#define ___GFP_PISA_RW 0
+#endif
/*
* Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -289,6 +298,10 @@ enum {
/* Disable lockdep for GFP context tracking */
#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
+/* Allocate from per-node PISA allocators (fall back to buddy on failure) */
+#define __GFP_PISA_RO ((__force gfp_t)___GFP_PISA_RO)
+#define __GFP_PISA_RW ((__force gfp_t)___GFP_PISA_RW)
+
/* Room for N __GFP_FOO bits */
#define __GFP_BITS_SHIFT ___GFP_LAST_BIT
#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index dddcedb8c93f6..18ba1c83bd6d3 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -7,6 +7,7 @@
#include <linux/bitops.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
+#include <linux/numa.h>
#include <linux/rcupdate.h>
#ifdef CONFIG_PISA
@@ -72,6 +73,11 @@ static inline bool pisa_free_page(struct page *page)
return ret;
}
+unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
+ int preferred_nid, nodemask_t *nodemask,
+ int nr_pages, struct page **page_array,
+ bool frozen);
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -91,8 +97,26 @@ static inline bool pisa_free_page(struct page *page)
return false;
}
+static inline unsigned long
+pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array, bool frozen)
+{
+ return 0;
+}
+
#endif /* CONFIG_PISA */
+static inline struct page *
+pisa_alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask)
+{
+ struct page *page = NULL;
+
+ pisa_alloc_pages_bulk(gfp, order, preferred_nid, nodemask, 1, &page, true);
+ return page;
+}
+
static inline bool is_pisa_phys(phys_addr_t phys)
{
return is_pisa_pfn(phys >> PAGE_SHIFT);
@@ -102,4 +126,5 @@ static inline bool is_pisa_page(const struct page *page)
{
return is_pisa_pfn(page_to_pfn(page));
}
+
#endif /* _LINUX_PISA_H */
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 3c153b3ad8450..65331bd8f0315 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -54,10 +54,18 @@
# define TRACE_GFP_FLAGS_LOCKDEP
#endif
+#ifdef CONFIG_PISA
+# define TRACE_GFP_FLAGS_PISA \
+ TRACE_GFP_EM(PISA_RO)
+#else
+# define TRACE_GFP_FLAGS_PISA
+#endif
+
#define TRACE_GFP_FLAGS \
TRACE_GFP_FLAGS_GENERAL \
TRACE_GFP_FLAGS_KASAN \
- TRACE_GFP_FLAGS_LOCKDEP
+ TRACE_GFP_FLAGS_LOCKDEP \
+ TRACE_GFP_FLAGS_PISA
#undef TRACE_GFP_EM
#define TRACE_GFP_EM(a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
@@ -65,7 +73,6 @@
TRACE_GFP_FLAGS
/* Just in case these are ever used */
-TRACE_DEFINE_ENUM(___GFP_UNUSED_BIT);
TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
#define gfpflag_string(flag) {(__force unsigned long)flag, #flag}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 48df87a19d312..b8e1f0fcb3f41 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1302,17 +1302,40 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
/*
* pisa_free_page() in __free_pages_prepare() recycles a PISA block into
* its allocator before the rest of that function runs. __pisa_free_page() calls
- * this to apply the buddy allocator's checks.
+ * this to apply the buddy allocator's checks, and to clear the order-0 state of
+ * blocks that were user pages (see drivers/iommu/pisa-user.c).
*
* Return: false if the block is in a bad state (already reported): leak it.
*/
bool free_page_prepare_pisa(struct page *page, unsigned int order)
{
+ struct folio *folio;
+ unsigned int i;
+
if (!IS_ENABLED(CONFIG_PISA))
return true;
+
+ folio = page_folio(page);
+
+ if (!order) {
+ if (unlikely(folio_test_mlocked(folio))) {
+ __folio_clear_mlocked(folio);
+ zone_stat_mod_folio(folio, NR_MLOCK, -1);
+ count_vm_events(UNEVICTABLE_PGCLEARED, 1);
+ }
+ folio->mapping = NULL;
+ }
if (is_check_pages_enabled() && free_page_is_bad(page))
return false;
page_table_check_free(page, order);
+ for (i = 0; i < (1U << order); i++)
+ page_kasan_tag_reset(page + i);
+ kasan_unpoison_range(page_address(page), PAGE_SIZE << order);
+ if (!order) {
+ page_cpupid_reset_last(page);
+ page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ page->private = 0;
+ }
return true;
}
@@ -5223,6 +5246,14 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
while (nr_populated < nr_pages && page_array[nr_populated])
nr_populated++;
+ if (unlikely(gfp & __GFP_PISA_RO)) {
+ nr_populated = pisa_alloc_pages_bulk(gfp, 0, preferred_nid, nodemask,
+ nr_pages, page_array, false);
+ gfp &= ~__GFP_PISA_RW;
+ if (nr_populated == nr_pages)
+ goto out;
+ }
+
/* No pages requested? */
if (unlikely(nr_pages <= 0))
goto out;
@@ -5457,6 +5488,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
fastpath_alloc_flags |= ALLOC_WMARK_LOW;
}
+ if (unlikely(gfp & __GFP_PISA_RO)) {
+ page = pisa_alloc_pages(gfp, order, preferred_nid, nodemask);
+ if (page)
+ return page;
+ gfp &= ~__GFP_PISA_RW;
+ }
+
gfp &= gfp_allowed_mask;
/*
* Apply scoped allocation constraints. This is mainly about GFP_NOFS
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (29 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
` (19 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Add two mechanisms to bound the number of unused PISA pages:
- a global ceiling, pisa.max_pages, on the number of PISA pages pinned
across all allocators and reservoirs, defaulting to one eighth of RAM,
with a floor for small machines.
- a shrinker that, under reclaim, drains the per-node reservoirs first
and then completely idle pages across all allocators.
Idle pages are detached under the allocator lock but retired outside
it, to avoid holding the lock during the expensive unmaps/iotlb flush.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 91 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 15 +++++++
drivers/iommu/pisa-node.c | 25 +++++++++++
drivers/iommu/pisa-priv.h | 15 +++++++
4 files changed, 146 insertions(+)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 1f1b99023094c..de8009fd6fff3 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -4,6 +4,7 @@
* See Documentation/core-api/pisa.rst for the architecture overview.
*/
+#include <kunit/visibility.h>
#include <linux/atomic.h>
#include <linux/bitmap.h>
#include <linux/bitops.h>
@@ -22,6 +23,7 @@
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
+#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
@@ -40,6 +42,11 @@ static LLIST_HEAD(pisa_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *pisa_wq __ro_after_init;
+/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */
+atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp;
+unsigned long pisa_max_pages __read_mostly;
+module_param_named(max_pages, pisa_max_pages, ulong, 0644);
+
/* Max completely idle PISA pages kept per allocator before async release. */
static unsigned int pisa_max_idle __read_mostly = 1;
module_param_named(max_idle, pisa_max_idle, uint, 0644);
@@ -81,6 +88,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
WRITE_ONCE(meta->pa, NULL);
+ atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
page_ref_unfreeze(block, 1);
@@ -93,6 +101,7 @@ void pisa_release_page(struct pisa_meta *meta)
{
if (!meta->pa) {
__free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
+ atomic_long_dec(&pisa_nr_pages);
return;
}
@@ -688,6 +697,72 @@ bool __pisa_free_page(struct page *page)
return true;
}
+/* Shrinker: releases reservoir pages first, then idle allocator pages. */
+static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+ unsigned long nr = pisa_reservoir_count();
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node)
+ nr += READ_ONCE(pa->num_idle_pages);
+ read_unlock(&pisa_allocs_lock);
+
+ return nr << PISA_ORDER;
+}
+
+/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+ unsigned int *nr, struct list_head *release_list,
+ unsigned long nr_to_scan)
+{
+ unsigned long freed = 0, flags;
+
+ spin_lock_irqsave(lock, flags);
+ while (freed < nr_to_scan && !list_empty(list)) {
+ list_move(list->next, release_list);
+ (*nr)--;
+ freed += 1UL << PISA_ORDER;
+ }
+ spin_unlock_irqrestore(lock, flags);
+
+ return freed;
+}
+
+VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+ struct pisa_meta *meta, *tmp;
+ struct pisa_alloc *pa;
+ unsigned long freed;
+ LIST_HEAD(release_list);
+
+ freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+
+ if (freed < sc->nr_to_scan) {
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ freed += pisa_shrink_list(&pa->lock, &pa->idle,
+ &pa->num_idle_pages, &release_list,
+ sc->nr_to_scan - freed);
+
+ if (freed >= sc->nr_to_scan)
+ break;
+ }
+ read_unlock(&pisa_allocs_lock);
+ }
+
+ /* Retiring must not run under @pa->lock. */
+ list_for_each_entry_safe(meta, tmp, &release_list, list) {
+ list_del_init(&meta->list);
+ pisa_release_page(meta);
+ }
+
+ /* Charge do_shrink_slab() for the full PISA pages reclaimed. */
+ sc->nr_scanned = freed;
+
+ return freed ?: SHRINK_STOP;
+}
+
static int __init pisa_init(void)
{
const struct {
@@ -698,10 +773,15 @@ static int __init pisa_init(void)
{ 2, true },
{ SKB_FRAG_PAGE_ORDER, true },
};
+ struct shrinker *shrink;
int nid, i;
pisa_node_init();
+ /* Default ceiling is 1/8th of RAM (floor of 16 PISA pages). */
+ if (!pisa_max_pages)
+ pisa_max_pages = max(totalram_pages() >> (PISA_ORDER + 3), 16UL);
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
@@ -724,6 +804,17 @@ static int __init pisa_init(void)
if (!pisa_wq)
pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+ shrink = shrinker_alloc(0, "pisa");
+ if (!shrink) {
+ pr_warn("pisa: shrinker registration failed\n");
+ return 0;
+ }
+
+ shrink->count_objects = pisa_shrink_count;
+ shrink->scan_objects = pisa_shrink_scan;
+ shrink->seeks = DEFAULT_SEEKS;
+ shrinker_register(shrink);
+
return 0;
}
subsys_initcall(pisa_init);
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index bdc52e6497b05..96bc8dbc38fb4 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,7 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/mm.h>
+#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <net/sock.h>
@@ -98,8 +99,10 @@ static void test_alloc_and_recycle(struct kunit *test)
{ 2, true },
{ SKB_FRAG_PAGE_ORDER, true },
};
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct page *p1, *p2, *bulk[3];
struct pisa_alloc *pa;
+ unsigned long old_max;
unsigned int i;
KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, 0, DMA_NONE));
@@ -115,6 +118,7 @@ static void test_alloc_and_recycle(struct kunit *test)
pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
KUNIT_ASSERT_NOT_NULL(test, pa);
+ pisa_shrink_scan(NULL, &sc);
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
@@ -155,6 +159,7 @@ static void test_alloc_and_recycle(struct kunit *test)
if (p2)
__free_pages(p2, order);
+ pisa_shrink_scan(NULL, &sc);
}
/* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
@@ -162,6 +167,16 @@ static void test_alloc_and_recycle(struct kunit *test)
KUNIT_ASSERT_NOT_NULL(test, p1);
KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
__free_pages(p1, 0);
+
+ /* Global budget cap (pisa_max_pages = 0) refuses new PISA pages. */
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ old_max = READ_ONCE(pisa_max_pages);
+ WRITE_ONCE(pisa_max_pages, 0);
+ WRITE_ONCE(pisa_reservoir_max, 0);
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL));
+ WRITE_ONCE(pisa_reservoir_max, 16);
+ WRITE_ONCE(pisa_max_pages, old_max);
}
static void test_reservoir_and_scrubber(struct kunit *test)
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 524f0d4457ad7..0f23b94a7a6f8 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -74,6 +74,10 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
return ERR_PTR(-EAGAIN);
}
+ /* Racy check against global ceiling across allocators and reservoirs. */
+ if (atomic_long_read(&pisa_nr_pages) >= READ_ONCE(pisa_max_pages))
+ return ERR_PTR(-ENOSPC);
+
/*
* Long-lived active pages must be unmovable, lowmem, non-compound, and
* must not dip into emergency reserves or trigger OOM/warnings.
@@ -102,6 +106,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
meta = pisa_meta_of_pfn(page_to_pfn(page));
memset(meta, 0, sizeof(*meta));
+
+ atomic_long_inc(&pisa_nr_pages);
*clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
return meta;
@@ -256,6 +262,25 @@ unsigned long pisa_reservoir_count(void)
return nr;
}
+/* Detach up to @nr_to_scan base pages from reservoirs onto @release_list. */
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan)
+{
+ unsigned long freed = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ if (!READ_ONCE(res->nr_pages))
+ continue;
+ freed += pisa_shrink_list(&res->lock, &res->pages, &res->nr_pages,
+ release_list, nr_to_scan - freed);
+ if (freed >= nr_to_scan)
+ break;
+ }
+ return freed;
+}
+
void __init pisa_node_init(void)
{
int nid;
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f15934ccdeea4..d5ea83745ba47 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -115,6 +115,16 @@ int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+struct shrink_control;
+struct shrinker;
+#if IS_ENABLED(CONFIG_KUNIT)
+unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+#endif
+
+/* Global PISA page budget across all allocators and per-node reservoirs. */
+extern atomic_long_t pisa_nr_pages;
+extern unsigned long pisa_max_pages;
+
static inline int pisa_sanitize_nid(int nid)
{
if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
@@ -149,12 +159,17 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
return page;
}
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+ unsigned int *nr, struct list_head *release_list,
+ unsigned long nr_to_scan);
+
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
unsigned long pisa_reservoir_count(void);
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window for PISA pages
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (30 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
` (18 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA buffers need to be quickly identified from their iova, and may need
to be mapped in multiple domains. We can achieve both by reserving an IOVA
range for PISA pages in each domain.
Add per-domain PISA IOVA window descriptors in struct iommu_dma_cookie
and a global slot allocator (pisa_slot_alloc() / pisa_slot_free()).
The window is sized to twice the maximum allowed PISA buffer space,
defaulting to max(64MB, 1/4 of physical RAM).
A buffer is mapped at the same slot offset in all windows for the domains
that need to access it. IOVA allocation is done with a bitmap (1 bit
per 2MB buffer, hence a 4KB bitmap supports 64GB of buffer space) at
negligible cost.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 3 ++
drivers/iommu/dma-iommu.c | 18 ++++++++
drivers/iommu/dma-iommu.h | 16 +++++++
drivers/iommu/pisa-alloc.c | 12 ++++-
drivers/iommu/pisa-kunit.c | 24 ++++++++++
drivers/iommu/pisa-map.c | 90 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 58 ++++++++++++++++++++++++
7 files changed, 220 insertions(+), 1 deletion(-)
create mode 100644 drivers/iommu/pisa-map.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index bc58166f9303a..125cf7e813cdc 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -12,6 +12,9 @@ obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
+ifeq ($(CONFIG_IOMMU_DMA),y)
+obj-$(CONFIG_PISA) += pisa-map.o
+endif
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 78742aeba932d..fba6084feac81 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -36,6 +36,7 @@
#include <trace/events/swiotlb.h>
#include "dma-iommu.h"
+#include "pisa-priv.h"
#include "iommu-pages.h"
#include "iommu-priv.h"
@@ -76,6 +77,8 @@ struct iommu_dma_cookie {
struct iommu_domain *fq_domain;
/* Options for dma-iommu use */
struct iommu_dma_options options;
+ /* IOVA window reserved for PISA pages. Nothing else can go here. */
+ struct pisa_window win_pisa;
};
struct iommu_dma_msi_cookie {
@@ -1215,6 +1218,21 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,
return iova_offset(iovad, phys | size);
}
+/* Return @domain's PISA window, or NULL if @domain cannot use PISA. */
+struct pisa_window *pisa_dma_window(struct iommu_domain *domain)
+{
+ if (static_branch_unlikely(&iommu_deferred_attach_enabled) ||
+ !domain || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
+ return NULL;
+
+ return &domain->iova_cookie->win_pisa;
+}
+
+struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain)
+{
+ return pisa_dma_window(domain) ? &domain->iova_cookie->iovad : NULL;
+}
+
dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
enum dma_data_direction dir, unsigned long attrs)
{
diff --git a/drivers/iommu/dma-iommu.h b/drivers/iommu/dma-iommu.h
index 040d002525632..f112b47396984 100644
--- a/drivers/iommu/dma-iommu.h
+++ b/drivers/iommu/dma-iommu.h
@@ -7,6 +7,9 @@
#include <linux/iommu.h>
+struct pisa_window;
+struct iova_domain;
+
#ifdef CONFIG_IOMMU_DMA
void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
@@ -22,6 +25,9 @@ void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
phys_addr_t msi_addr);
+struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain);
+struct pisa_window *pisa_dma_window(struct iommu_domain *domain);
+
extern bool iommu_dma_forcedac;
#else /* CONFIG_IOMMU_DMA */
@@ -59,5 +65,15 @@ static inline int iommu_dma_sw_msi(struct iommu_domain *domain,
return -ENODEV;
}
+static inline struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain)
+{
+ return NULL;
+}
+
+static inline struct pisa_window *pisa_dma_window(struct iommu_domain *domain)
+{
+ return NULL;
+}
+
#endif /* CONFIG_IOMMU_DMA */
#endif /* __DMA_IOMMU_H */
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index de8009fd6fff3..d7b73c3a6c0b6 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -84,10 +84,12 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Read @meta before freeing blocks; the frame can be reused immediately. */
struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
- unsigned int i, order = meta->pa->order;
+ unsigned int i, order = meta->pa->order, slot = meta->slot;
WRITE_ONCE(meta->pa, NULL);
+ /* The page has been unmapped from every domain by pisa_release_page() */
+ pisa_slot_free(slot);
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -426,10 +428,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
if (IS_ERR(meta))
return meta;
+ if (unlikely(pisa_slot_alloc(meta))) {
+ pisa_release_page(meta);
+ return ERR_PTR(-ENOSPC);
+ }
+
page = pfn_to_page(pisa_meta_to_pfn(meta));
/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
if (split_page_compound(page, PISA_ORDER, order)) {
+ pisa_slot_free(meta->slot);
pisa_release_page(meta);
pisa_alloc_backoff(nid, can_block);
return ERR_PTR(-EBUSY);
@@ -782,6 +790,8 @@ static int __init pisa_init(void)
if (!pisa_max_pages)
pisa_max_pages = max(totalram_pages() >> (PISA_ORDER + 3), 16UL);
+ pisa_slots_init();
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 96bc8dbc38fb4..5a2e806e74654 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -10,6 +10,7 @@
#include <linux/skbuff.h>
#include <net/sock.h>
+#include "dma-iommu.h"
#include "pisa-priv.h"
static void test_meta_init_and_roundtrip(struct kunit *test)
@@ -179,6 +180,26 @@ static void test_alloc_and_recycle(struct kunit *test)
WRITE_ONCE(pisa_max_pages, old_max);
}
+#ifdef CONFIG_IOMMU_DMA
+static void test_window_helpers(struct kunit *test)
+{
+ struct pisa_window win = {};
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ /* Empty window (base == 0) never owns any IOVA. */
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, 0));
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G));
+
+ win.base = SZ_4G;
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G - 1));
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G));
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size - 1));
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size));
+}
+
+#endif /* CONFIG_IOMMU_DMA */
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -190,6 +211,9 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
KUNIT_CASE(test_alloc_and_recycle),
+#ifdef CONFIG_IOMMU_DMA
+ KUNIT_CASE(test_window_helpers),
+#endif
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
new file mode 100644
index 0000000000000..2e8d635dec34c
--- /dev/null
+++ b/drivers/iommu/pisa-map.c
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-domain IOVA window management and IOMMU mapping.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/pisa.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include "dma-iommu.h"
+#include "pisa-priv.h"
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Optional module param override for pisa_nr_slots (0 = auto: 2 * max_pages). */
+static unsigned int pisa_cfg_slots __read_mostly;
+module_param_named(slots, pisa_cfg_slots, uint, 0444);
+
+/*
+ * Number of 2MB IOVA slots in each domain's window (computed at init from
+ * pisa_cfg_slots or 2 * max_pages). Each active PISA page holds one slot
+ * in [0, pisa_nr_slots) and maps at slot * PISA_SIZE in every domain's window.
+ */
+unsigned int pisa_nr_slots __read_mostly;
+u64 pisa_window_size __read_mostly;
+/* One bit per IOVA slot, set while an active PISA page holds the slot. */
+static unsigned long *pisa_slot_bitmap __read_mostly;
+/* Metadata of the PISA page backing each active slot, or NULL. */
+static struct pisa_meta **pisa_slot2meta __read_mostly;
+static unsigned int pisa_slot_hint; /* no free slot below this */
+/* Protects pisa_slot_bitmap and pisa_slot_hint. */
+static DEFINE_SPINLOCK(pisa_slot_lock);
+
+/* Cost is typically low though the bitmap can be up to O(100K) bits on large systems. */
+int pisa_slot_alloc(struct pisa_meta *meta)
+{
+ unsigned long flags;
+ unsigned int slot;
+
+ spin_lock_irqsave(&pisa_slot_lock, flags);
+ slot = find_next_zero_bit(pisa_slot_bitmap, pisa_nr_slots, pisa_slot_hint);
+ if (slot < pisa_nr_slots) {
+ __set_bit(slot, pisa_slot_bitmap);
+ pisa_slot_hint = slot + 1;
+ WRITE_ONCE(meta->slot, slot);
+ WRITE_ONCE(pisa_slot2meta[slot], meta);
+ }
+ spin_unlock_irqrestore(&pisa_slot_lock, flags);
+
+ if (slot >= pisa_nr_slots)
+ return -ENOSPC;
+ return 0;
+}
+
+/* Return @slot once no domain can translate it any more. */
+void pisa_slot_free(unsigned int slot)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&pisa_slot_lock, flags);
+ WRITE_ONCE(pisa_slot2meta[slot], NULL);
+ __clear_bit(slot, pisa_slot_bitmap);
+ if (slot < pisa_slot_hint)
+ pisa_slot_hint = slot;
+ spin_unlock_irqrestore(&pisa_slot_lock, flags);
+}
+
+void __init pisa_slots_init(void)
+{
+ /* Fixed at boot: sizes every domain's window (default 2 * max_pages). */
+ pisa_nr_slots = pisa_cfg_slots ?: min_t(unsigned long, 2 * pisa_max_pages, UINT_MAX >> 1);
+ pisa_slot_bitmap = kvcalloc(BITS_TO_LONGS(pisa_nr_slots),
+ sizeof(unsigned long), GFP_KERNEL);
+ pisa_slot2meta = kvcalloc(pisa_nr_slots, sizeof(*pisa_slot2meta), GFP_KERNEL);
+ if (!pisa_slot_bitmap || !pisa_slot2meta) {
+ kvfree(pisa_slot_bitmap);
+ kvfree(pisa_slot2meta);
+ pisa_slot_bitmap = NULL;
+ pisa_slot2meta = NULL;
+ pisa_nr_slots = 0;
+ }
+ /* Size of every domain's window: one 2MB page per slot. */
+ pisa_window_size = (u64)pisa_nr_slots << PISA_SHIFT;
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index d5ea83745ba47..3ef6e51402ceb 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -13,12 +13,35 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+/**
+ * struct pisa_window - A domain's IOVA window reserved for PISA pages
+ * @base: Window base IOVA (PISA_SIZE aligned), or 0 if unassigned
+ * @domain_idx: Dense index + PISA_IDX_FIRST, or PISA_IDX_* sentinel while @base == 0
+ * @idx_gen: Value of pisa_idx_gen when @domain_idx was set to NOIDX
+ *
+ * Lives in iommu_dma_cookie. Active PISA pages map at @base + slot * PISA_SIZE.
+ */
+struct pisa_window {
+ dma_addr_t base;
+ u16 domain_idx;
+ u32 idx_gen;
+};
+
#ifdef CONFIG_PISA
#define PISA_MAX_BLOCKS 512U
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/* Sentinel values of pisa_window.domain_idx below PISA_IDX_FIRST. */
+enum {
+ PISA_IDX_NONE, /* not attempted yet */
+ PISA_IDX_NOHUGE, /* @domain can never use PISA */
+ PISA_IDX_NOWINDOW, /* no IOVA range for the window */
+ PISA_IDX_NOIDX, /* all domain indices were in use */
+ PISA_IDX_FIRST, /* first valid domain index */
+};
+
/*
* Lock hierarchy:
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
@@ -32,6 +55,7 @@
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
+ * @slot: Dense IOVA slot index in every domain's window
* @pa: Owning pisa_alloc (NULL for reservoir pages)
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
@@ -44,6 +68,7 @@ struct pisa_meta {
u16 nr_clean;
u16 nr_dirty;
u8 on_list;
+ u32 slot;
struct pisa_alloc *pa;
struct list_head list;
@@ -172,4 +197,37 @@ unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
#endif /* CONFIG_PISA */
+
+#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
+
+/* Size in bytes of every domain's PISA IOVA window, fixed at boot. */
+extern u64 pisa_window_size;
+extern unsigned int pisa_nr_slots;
+
+void pisa_slots_init(void);
+int pisa_slot_alloc(struct pisa_meta *meta);
+void pisa_slot_free(unsigned int slot);
+
+/* Return true if @dma falls within @win's reserved PISA IOVA window. */
+static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
+{
+ /* Acquire pairs with the release of @base in pisa_window_assign(). */
+ dma_addr_t base = smp_load_acquire(&win->base);
+
+ return base && dma - base < pisa_window_size;
+}
+
+#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
+
+#define pisa_nr_slots 0U
+static inline void pisa_slots_init(void) {}
+static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
+static inline void pisa_slot_free(unsigned int slot) {}
+
+static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
+{
+ return false;
+}
+
+#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (31 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
` (17 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Each PISA page can be mapped by multiple IOMMU domains.
Add pisa_domain_release() (called from iommu_put_dma_cookie()) to drop
a dying domain from the registry, clear its bit across all active PISA
pages, and wait on SRCU before reusing the domain index. If iommu_unmap()
fails on page release, mark the 2MB page PISA_QUARANTINED and leak it
so a device can never access reallocated memory.
When a group's last DMA API user detaches
(iommu_device_unuse_default_domain() with owner_cnt reaching 0), call
pisa_domain_quiesce() to unmap all active PISA pages from the default
domain and flush the IOTLB while keeping the domain's window and index
for future demand mappings.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-iommu.c | 3 +
drivers/iommu/iommu.c | 5 +
drivers/iommu/pisa-alloc.c | 14 ++-
drivers/iommu/pisa-kunit.c | 18 ++++
drivers/iommu/pisa-map.c | 214 ++++++++++++++++++++++++++++++++++++-
drivers/iommu/pisa-priv.h | 56 +++++++++-
include/linux/pisa.h | 1 +
7 files changed, 306 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index fba6084feac81..025bceebb2262 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -430,6 +430,9 @@ void iommu_put_dma_cookie(struct iommu_domain *domain)
struct iommu_dma_cookie *cookie = domain->iova_cookie;
struct iommu_dma_msi_page *msi, *tmp;
+ /* Drop cached PISA mappings before the IOVA domain goes away. */
+ pisa_domain_release(domain);
+
if (cookie->iovad.granule) {
iommu_dma_free_fq(cookie);
put_iova_domain(&cookie->iovad);
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b486b8bbd1fcd..dfba3d5c277c2 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -37,6 +37,7 @@
#include <linux/generic_pt/iommu.h>
#include "dma-iommu.h"
+#include "pisa-priv.h"
#include "iommu-priv.h"
static struct kset *iommu_group_kset;
@@ -3363,6 +3364,10 @@ void iommu_device_unuse_default_domain(struct device *dev)
if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
group->owner_cnt--;
+ /* Unmap persistent PISA windows when the last driver detaches. */
+ if (!group->owner_cnt)
+ pisa_domain_quiesce(group->default_domain);
+
mutex_unlock(&group->mutex);
}
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index d7b73c3a6c0b6..dd90aad296edf 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -13,6 +13,7 @@
#include <linux/pisa.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/iommu.h>
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/memcontrol.h>
@@ -30,6 +31,7 @@
#include <linux/workqueue.h>
#include <net/sock.h>
+#include "dma-iommu.h"
#include "pisa-priv.h"
/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
@@ -88,6 +90,10 @@ static void pisa_release_page_rcu(struct rcu_head *head)
WRITE_ONCE(meta->pa, NULL);
+ /* Possibly still mapped by a device: leak it, still charged. */
+ if (meta->flags & PISA_QUARANTINED)
+ return;
+
/* The page has been unmapped from every domain by pisa_release_page() */
pisa_slot_free(slot);
atomic_long_dec(&pisa_nr_pages);
@@ -107,7 +113,8 @@ void pisa_release_page(struct pisa_meta *meta)
return;
}
- /* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
+ /* Quarantined on unmap failure and leaked in pisa_release_page_rcu(). */
+ pisa_unmap_all(meta);
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
@@ -194,7 +201,8 @@ static bool pisa_place_meta(struct pisa_meta *meta)
lockdep_assert_held(&pa->lock);
if (avail == bpp) {
- if (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms)) {
+ if ((meta->flags & PISA_QUARANTINED) ||
+ (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms))) {
list_del_init(&meta->list);
WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
llist_add(&meta->llnode, &pisa_free_list);
@@ -204,7 +212,7 @@ static bool pisa_place_meta(struct pisa_meta *meta)
WRITE_ONCE(pa->idle_jiffies, jiffies);
pa->num_idle_pages++;
on_list = meta->nr_dirty ? PISA_LIST_IDLE_DIRTY : PISA_LIST_IDLE_CLEAN;
- } else if (!avail) {
+ } else if (!avail || unlikely(meta->flags & PISA_QUARANTINED)) {
on_list = PISA_LIST_FULL;
} else {
bool sparse;
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5a2e806e74654..f0775aa45a058 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -5,6 +5,7 @@
#include <kunit/test.h>
#include <linux/gfp.h>
#include <linux/pisa.h>
+#include <linux/iommu.h>
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -183,6 +184,7 @@ static void test_alloc_and_recycle(struct kunit *test)
#ifdef CONFIG_IOMMU_DMA
static void test_window_helpers(struct kunit *test)
{
+ struct iommu_domain dummy_domain = {};
struct pisa_window win = {};
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -196,6 +198,22 @@ static void test_window_helpers(struct kunit *test)
KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G));
KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size - 1));
KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size));
+
+ /* Sentinel values in pisa_window_declined(). */
+ win.domain_idx = PISA_IDX_NONE;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+ win.domain_idx = PISA_IDX_NOHUGE;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -EOPNOTSUPP);
+ win.domain_idx = PISA_IDX_NOWINDOW;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -ENOSPC);
+ win.domain_idx = PISA_IDX_NOIDX;
+ win.idx_gen = READ_ONCE(pisa_idx_gen);
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -EBUSY);
+ win.idx_gen--;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+
+ /* Releasing an unregistered domain with an active allocator is a safe no-op. */
+ pisa_domain_release(&dummy_domain);
}
#endif /* CONFIG_IOMMU_DMA */
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2e8d635dec34c..2ba977bb68734 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -4,13 +4,19 @@
* See Documentation/core-api/pisa.rst for the architecture overview.
*/
+#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/dma-map-ops.h>
+#include <linux/dma-mapping.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/iommu.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
-#include <linux/pisa.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <linux/srcu.h>
#include "dma-iommu.h"
#include "pisa-priv.h"
@@ -88,3 +94,209 @@ void __init pisa_slots_init(void)
/* Size of every domain's window: one 2MB page per slot. */
pisa_window_size = (u64)pisa_nr_slots << PISA_SHIFT;
}
+
+/*
+ * Registry of PISA IOMMU domains (0 .. PISA_MAX_DOMAINS - 1). Each slot is
+ * FREE (!domain && !base), ACTIVE (domain), or DRAINING (!domain && base) while
+ * pisa_domain_release() clears @meta->domains_mapped bits.
+ */
+static struct {
+ struct iommu_domain *domain;
+ dma_addr_t base;
+} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
+static DEFINE_SPINLOCK(pisa_domains_lock);
+
+/* Protects pisa_unmap_all() readers against concurrent pisa_domain_release(). */
+DEFINE_STATIC_SRCU(pisa_srcu);
+
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+unsigned int pisa_idx_gen __read_mostly;
+
+/* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
+static void pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
+ int idx, dma_addr_t iova, size_t unmapped)
+{
+ pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
+ &iova, domain, idx, unmapped);
+ WRITE_ONCE(meta->flags, meta->flags | PISA_QUARANTINED);
+}
+
+/*
+ * Unmap @meta's leaf PTE from every active domain holding it. Returns 0 on
+ * success, or -EIO if any unmap failed (@meta is then quarantined).
+ */
+int pisa_unmap_all(struct pisa_meta *meta)
+{
+ dma_addr_t offset = pisa_meta_win_offset(meta);
+ int idx, srcu_idx;
+
+ srcu_idx = srcu_read_lock(&pisa_srcu);
+ for_each_set_bit(idx, meta->domains_mapped, PISA_MAX_DOMAINS) {
+ struct iommu_domain *domain;
+ unsigned long flags;
+ size_t unmapped;
+ dma_addr_t base;
+
+ if (!test_and_clear_bit(idx, meta->domains_mapped))
+ continue;
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ domain = pisa_domains[idx].domain;
+ base = pisa_domains[idx].base;
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ /* Free, or draining: pisa_domain_release() handles the bit. */
+ if (!domain)
+ continue;
+
+ unmapped = iommu_unmap(domain, base + offset, PISA_SIZE);
+ if (unlikely(unmapped != PISA_SIZE))
+ pisa_quarantine_meta(meta, domain, idx, base + offset, unmapped);
+ }
+ srcu_read_unlock(&pisa_srcu, srcu_idx);
+
+ return (meta->flags & PISA_QUARANTINED) ? -EIO : 0;
+}
+
+/* Clear domain @idx from all populated slots. */
+static unsigned int pisa_forget_domain(int idx)
+{
+ unsigned int slot, dropped = 0;
+
+ rcu_read_lock();
+ for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+ struct pisa_meta *meta = READ_ONCE(pisa_slot2meta[slot]);
+
+ if (meta && test_bit(idx, meta->domains_mapped) &&
+ test_and_clear_bit(idx, meta->domains_mapped))
+ dropped++;
+ }
+ rcu_read_unlock();
+ return dropped;
+}
+
+/*
+ * Release @domain's registry index and clear its bit in all active
+ * PISA pages, waiting out concurrent unmappers on @pisa_srcu before freeing
+ * the slot.
+ */
+void pisa_domain_release(struct iommu_domain *domain)
+{
+ struct pisa_window *win;
+ unsigned int dropped;
+ unsigned long flags;
+ int idx;
+
+ /* Nothing has ever been active, so nothing can reference @domain. */
+ if (!static_branch_unlikely(&pisa_key))
+ return;
+
+ win = pisa_dma_window(domain);
+ /* @domain never active, so no PISA page can be holding a bit for it. */
+ if (!win || !smp_load_acquire(&win->base))
+ return;
+
+ idx = win->domain_idx - PISA_IDX_FIRST;
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ if (pisa_domains[idx].domain != domain) {
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+ return;
+ }
+ /* Leaving @base set moves the slot to DRAINING state */
+ pisa_domains[idx].domain = NULL;
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ dropped = pisa_forget_domain(idx);
+
+ /* Wait out in-flight pisa_unmap_all() readers before reusing @idx. */
+ synchronize_srcu(&pisa_srcu);
+
+ /* Mark slot FREE and bump @pisa_idx_gen so NOIDX domains retry. */
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ pisa_domains[idx].base = 0;
+ WRITE_ONCE(pisa_idx_gen, pisa_idx_gen + 1);
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ if (dropped)
+ pr_debug("pisa: released %u PISA page mappings for domain %p (index %d)\n",
+ dropped, domain, idx);
+}
+
+/*
+ * Remove @meta's leaf PTE from live @domain (registered at @idx with @base),
+ * batching the IOTLB flush into @gather. Returns 1 if unmapped, 0 otherwise.
+ */
+static unsigned int pisa_unmap_domain_locked(struct pisa_meta *meta,
+ struct iommu_domain *domain,
+ int idx, dma_addr_t base,
+ struct iommu_iotlb_gather *gather)
+{
+ dma_addr_t iova = base + pisa_meta_win_offset(meta);
+ size_t unmapped;
+
+ if (!test_and_clear_bit(idx, meta->domains_mapped))
+ return 0;
+
+ unmapped = iommu_unmap_fast(domain, iova, PISA_SIZE, gather);
+ if (unlikely(unmapped != PISA_SIZE))
+ pisa_quarantine_meta(meta, domain, idx, iova, unmapped);
+ return 1;
+}
+
+/*
+ * Unmap all PISA leaf PTEs from a domain whose last DMA API user detached,
+ * keeping its window and domain index so mappings are re-installed on demand.
+ */
+void pisa_domain_quiesce(struct iommu_domain *domain)
+{
+ struct iommu_iotlb_gather gather;
+ struct pisa_window *win;
+ unsigned int slot, done = 0;
+ bool release = false;
+ dma_addr_t base;
+ int idx;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return;
+ win = pisa_dma_window(domain);
+ /* Pairs with smp_store_release() in pisa_window_assign(). */
+ base = win ? smp_load_acquire(&win->base) : 0;
+ if (!base)
+ return;
+ idx = win->domain_idx - PISA_IDX_FIRST;
+
+ iommu_iotlb_gather_init(&gather);
+ /* Keep call_rcu()-freed pages from reaching the buddy allocator before IOTLB sync. */
+ rcu_read_lock();
+ for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+ struct pisa_meta *meta = READ_ONCE(pisa_slot2meta[slot]);
+ struct pisa_alloc *pa;
+ unsigned long flags;
+
+ if (!meta || !test_bit(idx, meta->domains_mapped))
+ continue;
+ done += pisa_unmap_domain_locked(meta, domain, idx, base, &gather);
+ if (likely(!(meta->flags & PISA_QUARANTINED)))
+ continue;
+ pa = READ_ONCE(meta->pa);
+ if (!pa)
+ continue;
+ spin_lock_irqsave(&pa->lock, flags);
+ if (meta->pa == pa) {
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+ release |= pisa_place_meta(meta);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ }
+
+ if (done)
+ iommu_iotlb_sync(domain, &gather);
+ rcu_read_unlock();
+ if (release)
+ pisa_schedule_reclaim();
+ if (done)
+ pr_debug("pisa: unmapped %u PISA pages from idle domain %p (index %d)\n",
+ done, domain, idx);
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 3ef6e51402ceb..f97f69a13d1a9 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -13,6 +13,9 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+struct iommu_domain;
+struct iommu_iotlb_gather;
+
/**
* struct pisa_window - A domain's IOVA window reserved for PISA pages
* @base: Window base IOVA (PISA_SIZE aligned), or 0 if unassigned
@@ -33,6 +36,9 @@ struct pisa_window {
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/* Max simultaneous PISA IOMMU domains (bit index in pisa_meta.domains_mapped). */
+#define PISA_MAX_DOMAINS 128
+
/* Sentinel values of pisa_window.domain_idx below PISA_IDX_FIRST. */
enum {
PISA_IDX_NONE, /* not attempted yet */
@@ -47,15 +53,18 @@ enum {
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ * - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
+ * - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
- * @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ * @flags: PISA_ACTIVE / PISA_QUARANTINED
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
* @slot: Dense IOVA slot index in every domain's window
+ * @domains_mapped: Bitmap of domain indices holding this page's leaf PTE
* @pa: Owning pisa_alloc (NULL for reservoir pages)
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
@@ -69,6 +78,9 @@ struct pisa_meta {
u16 nr_dirty;
u8 on_list;
u32 slot;
+
+ DECLARE_BITMAP(domains_mapped, PISA_MAX_DOMAINS);
+
struct pisa_alloc *pa;
struct list_head list;
@@ -136,6 +148,12 @@ struct pisa_node {
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+/* Offset of @meta from the start of a domain's window. */
+static inline dma_addr_t pisa_meta_win_offset(const struct pisa_meta *meta)
+{
+ return (dma_addr_t)meta->slot << PISA_SHIFT;
+}
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
@@ -202,11 +220,30 @@ unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned lon
/* Size in bytes of every domain's PISA IOVA window, fixed at boot. */
extern u64 pisa_window_size;
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+extern unsigned int pisa_idx_gen;
extern unsigned int pisa_nr_slots;
void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
+int pisa_unmap_all(struct pisa_meta *meta);
+
+/* Return negative errno if @win's domain was permanently/currently declined, or 0. */
+static inline int pisa_window_declined(const struct pisa_window *win)
+{
+ switch (READ_ONCE(win->domain_idx)) {
+ case PISA_IDX_NOHUGE:
+ return -EOPNOTSUPP;
+ case PISA_IDX_NOWINDOW:
+ return -ENOSPC;
+ case PISA_IDX_NOIDX:
+ return READ_ONCE(win->idx_gen) == READ_ONCE(pisa_idx_gen) ?
+ -EBUSY : 0;
+ default:
+ return 0;
+ }
+}
/* Return true if @dma falls within @win's reserved PISA IOVA window. */
static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
@@ -217,17 +254,34 @@ static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
return base && dma - base < pisa_window_size;
}
+void pisa_domain_release(struct iommu_domain *domain);
+void pisa_domain_quiesce(struct iommu_domain *domain);
+
#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
#define pisa_nr_slots 0U
static inline void pisa_slots_init(void) {}
static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
+static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+
+static inline int pisa_window_declined(const struct pisa_window *win)
+{
+ return -EOPNOTSUPP;
+}
static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
{
return false;
}
+static inline void pisa_domain_release(struct iommu_domain *domain)
+{
+}
+
+static inline void pisa_domain_quiesce(struct iommu_domain *domain)
+{
+}
+
#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 18ba1c83bd6d3..ab1c32e065bbb 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -27,6 +27,7 @@
enum {
PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
+ PISA_QUARANTINED = BIT(1), /* IOMMU unmap failed: never reuse */
};
/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (32 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
` (16 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Intercept the DMA mapping and unmapping functions so that PISA
buffers are mapped within the reserved IOVA range.
This is implemented as follows:
- each domain that enables PISA (via per-device optin) is dynamically
assigned an IOVA window and a compact index the first time it maps a
PISA page.
This way, one bit in pisa_meta suffices to record whether the domain
has already mapped the page.
- The page is mapped in the domain at win->base + slot * 2MB on first
use, and subsequent mappings reuse that translation.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Kconfig | 1 +
drivers/iommu/dma-iommu.c | 15 +++
drivers/iommu/iommu.c | 5 +-
drivers/iommu/pisa-alloc.c | 5 +-
drivers/iommu/pisa-kunit.c | 133 +++++++++++++++++++++++++++
drivers/iommu/pisa-map.c | 184 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 35 ++++++-
include/linux/pisa.h | 3 +
8 files changed, 377 insertions(+), 4 deletions(-)
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 15e06eaed0924..52cce0963525b 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -428,6 +428,7 @@ config PISA
bool "Pre-mapped IOVA Superpage Allocator"
depends on X86_64 || ARM64
depends on !PREEMPT_RT
+ select NEED_SG_DMA_FLAGS
default y
help
PISA is a specialized memory allocator with an API similar to
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 025bceebb2262..eb920ad77c783 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -837,6 +837,8 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
struct iommu_iotlb_gather iotlb_gather;
size_t unmapped;
+ if (is_pisa_iova(&cookie->win_pisa, dma_addr))
+ return;
dma_addr -= iova_off;
size = iova_align(iovad, size + iova_off);
iommu_iotlb_gather_init(&iotlb_gather);
@@ -863,6 +865,8 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
iommu_deferred_attach(dev, domain))
return DMA_MAPPING_ERROR;
+ if (!pisa_map_phys(&iova, dev, domain, phys, size, prot))
+ return iova;
/* If anyone ever wants this we'd need support in the IOVA allocator */
if (dev_WARN_ONCE(dev, dma_get_min_align_mask(dev) > iova_mask(iovad),
"Unsupported alignment constraint\n"))
@@ -1277,6 +1281,10 @@ void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
{
phys_addr_t phys;
+ /* PISA mapped buffer: nothing to unmap or release. */
+ if (is_pisa_iova(&iommu_get_dma_domain(dev)->iova_cookie->win_pisa, dma_handle))
+ return;
+
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) {
__iommu_dma_unmap(dev, dma_handle, size);
return;
@@ -1478,6 +1486,13 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
*/
break;
case PCI_P2PDMA_MAP_NONE:
+ /* Map PISA pages and mark as bus addresses to skip IOVA alloc and unmap. */
+ if (!pisa_map_phys(&s->dma_address, dev, domain,
+ sg_phys(s), s_length, prot)) {
+ sg_dma_len(s) = s_length;
+ sg_dma_mark_bus_address(s);
+ continue;
+ }
break;
case PCI_P2PDMA_MAP_BUS_ADDR:
/*
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index dfba3d5c277c2..068eaf262c7f6 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2867,8 +2867,9 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
while (i <= nents) {
phys_addr_t s_phys = sg_phys(sg);
+ bool is_bus = i < nents && sg_dma_is_bus_address(sg);
- if (len && s_phys != start + len) {
+ if (len && (is_bus || s_phys != start + len)) {
ret = iommu_map_nosync(domain, iova + mapped, start,
len, prot, gfp);
if (ret)
@@ -2878,7 +2879,7 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
len = 0;
}
- if (sg_dma_is_bus_address(sg))
+ if (is_bus)
goto next;
if (len) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index dd90aad296edf..f98142882519c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -299,9 +299,11 @@ struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direct
spin_lock_init(&pa->lock);
pa->order = order;
- pa->nid = pisa_sanitize_nid(nid);
+ pa->prot = (dir != DMA_FROM_DEVICE ? IOMMU_READ : 0) |
+ (dir != DMA_TO_DEVICE ? IOMMU_WRITE : 0);
pa->max_idle_pages = READ_ONCE(pisa_max_idle) ? : 1;
pisa_set_name(pa, "%ps", (void *)_RET_IP_);
+ pa->nid = pisa_sanitize_nid(nid);
INIT_LIST_HEAD(&pa->partial);
INIT_LIST_HEAD(&pa->partial_dirty);
INIT_LIST_HEAD(&pa->idle);
@@ -607,6 +609,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
goto out;
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
+ meta->prot = pa->prot;
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
}
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index f0775aa45a058..e16268d55f3de 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,7 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/iommu.h>
+#include <linux/iova.h>
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -184,8 +185,10 @@ static void test_alloc_and_recycle(struct kunit *test)
#ifdef CONFIG_IOMMU_DMA
static void test_window_helpers(struct kunit *test)
{
+ struct device dev = { .numa_node = NUMA_NO_NODE };
struct iommu_domain dummy_domain = {};
struct pisa_window win = {};
+ dma_addr_t d;
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -212,10 +215,139 @@ static void test_window_helpers(struct kunit *test)
win.idx_gen--;
KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+ /* A device without an IOMMU group has no PISA window. */
+ KUNIT_EXPECT_NE(test, pisa_map_phys(&d, NULL, NULL, 0, PAGE_SIZE, 0), 0);
+ KUNIT_EXPECT_NE(test, pisa_map_phys(&d, &dev, NULL, 0, PAGE_SIZE, 0), 0);
/* Releasing an unregistered domain with an active allocator is a safe no-op. */
pisa_domain_release(&dummy_domain);
}
+/*
+ * Fake DMA domain for the mapping and coherent allocation tests, backed by a
+ * real iommu_dma_cookie and no-op page table ops.
+ */
+static int fake_map_pages(struct iommu_domain *domain, unsigned long iova,
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
+{
+ *mapped = pgsize * pgcount;
+ return 0;
+}
+
+static size_t fake_unmap_pages(struct iommu_domain *domain, unsigned long iova,
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *iotlb_gather)
+{
+ return pgsize * pgcount;
+}
+
+static const struct iommu_domain_ops fake_domain_ops = {
+ .map_pages = fake_map_pages,
+ .unmap_pages = fake_unmap_pages,
+};
+
+static struct iommu_domain fake_domain = {
+ .type = IOMMU_DOMAIN_DMA,
+ .pgsize_bitmap = SZ_4K | PISA_SIZE,
+ .ops = &fake_domain_ops,
+};
+
+static void fake_domain_teardown(void *unused)
+{
+ iommu_put_dma_cookie(&fake_domain);
+ fake_domain.cookie_type = IOMMU_COOKIE_NONE;
+ fake_domain.iova_cookie = NULL;
+ iova_cache_put();
+}
+
+static int fake_domain_setup(struct kunit *test)
+{
+ int ret = iova_cache_get();
+
+ if (ret)
+ return ret;
+ ret = iommu_get_dma_cookie(&fake_domain);
+ if (ret) {
+ iova_cache_put();
+ return ret;
+ }
+ init_iova_domain(pisa_dma_iovad(&fake_domain), SZ_4K, 1);
+ return kunit_add_action_or_reset(test, fake_domain_teardown, NULL);
+}
+
+static void test_map_and_domain_lifecycle(struct kunit *test)
+{
+ struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
+ .coherent_dma_mask = DMA_BIT_MASK(64) };
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct pisa_window *win;
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ unsigned int domain_idx;
+ dma_addr_t d1, d2;
+ struct page *p;
+ phys_addr_t phys;
+
+ dev.dma_mask = &dev.coherent_dma_mask;
+ set_dev_node(&dev, first_memory_node);
+#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
+ dev.dma_coherent = true;
+#endif
+ KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+ win = pisa_dma_window(&fake_domain);
+
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ p = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p);
+
+ phys = page_to_phys(p);
+ meta = pisa_meta_of_pfn(page_to_pfn(p));
+ KUNIT_EXPECT_TRUE(test, is_pisa_phys(phys));
+
+ /* TX-only allocator declines IOMMU_WRITE mappings. */
+ KUNIT_EXPECT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), -EACCES);
+
+ /* First map assigns the window and installs the 2MB PTE. */
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(win, d1));
+ domain_idx = win->domain_idx - PISA_IDX_FIRST;
+ KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Second map at an offset hits the lockless fast path. */
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, phys + 128, 256,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1 + 128);
+
+ /* Quiescing the domain unmaps the PISA page but keeps the window. */
+ pisa_domain_quiesce(&fake_domain);
+ KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1);
+ KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Releasing the domain forgets the mapping bit across allocators. */
+ pisa_domain_release(&fake_domain);
+ KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Re-map into a fresh window so shrinking exercises pisa_unmap_all(). */
+ *win = (struct pisa_window){};
+ KUNIT_EXPECT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+
+ __free_pages(p, 2);
+ pisa_shrink_scan(NULL, &sc);
+}
#endif /* CONFIG_IOMMU_DMA */
static void test_reservoir_and_scrubber(struct kunit *test)
@@ -231,6 +363,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_alloc_and_recycle),
#ifdef CONFIG_IOMMU_DMA
KUNIT_CASE(test_window_helpers),
+ KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
KUNIT_CASE(test_reservoir_and_scrubber),
{}
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2ba977bb68734..2c481e514575c 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -12,6 +12,8 @@
#include <linux/pisa.h>
#include <linux/export.h>
#include <linux/iommu.h>
+#include <linux/iova.h>
+#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
@@ -103,6 +105,7 @@ void __init pisa_slots_init(void)
static struct {
struct iommu_domain *domain;
dma_addr_t base;
+ spinlock_t map_lock; /* serialises iommu_map() in __pisa_map_phys() */
} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
static DEFINE_SPINLOCK(pisa_domains_lock);
@@ -300,3 +303,184 @@ void pisa_domain_quiesce(struct iommu_domain *domain)
pr_debug("pisa: unmapped %u PISA pages from idle domain %p (index %d)\n",
done, domain, idx);
}
+
+/* Allocate a PISA_SIZE-aligned IOVA window of @size bytes from @domain. */
+static dma_addr_t pisa_dma_window_alloc(struct device *dev, struct iommu_domain *domain, u64 size)
+{
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ struct iova_domain *iovad = pisa_dma_iovad(domain);
+ unsigned long shift, iova_len;
+ struct iova *new_iova;
+
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+
+ if (!iovad)
+ return 0;
+
+ if (domain->geometry.force_aperture)
+ limit = min_t(u64, limit, domain->geometry.aperture_end);
+
+ /* Ignore devices with too many constraints. */
+ if (limit <= DMA_BIT_MASK(32) || size > limit - PISA_SIZE)
+ return 0;
+
+ shift = iova_shift(iovad);
+ iova_len = (size + PISA_SIZE) >> shift;
+
+ /* Ignore advisory dev->iommu->pci_32bit_workaround preference. */
+ new_iova = alloc_iova(iovad, iova_len, limit >> shift, false);
+ if (!new_iova)
+ return 0;
+
+ return ALIGN((dma_addr_t)new_iova->pfn_lo << shift, PISA_SIZE);
+}
+
+/*
+ * Assign @domain an IOVA window and dense domain index on first use, recording
+ * permanent or retryable failures in @win->domain_idx.
+ */
+int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win)
+{
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ u64 size = pisa_window_size;
+ unsigned long flags;
+ dma_addr_t base = 0;
+ int idx, ret;
+
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+
+ if (!dev_is_dma_coherent(dev))
+ return -EOPNOTSUPP;
+
+ if (limit <= DMA_BIT_MASK(32) || size > limit - PISA_SIZE)
+ return -ENOSPC;
+
+ if (!(domain->pgsize_bitmap & PISA_SIZE)) {
+ if (cmpxchg(&win->domain_idx, PISA_IDX_NONE, PISA_IDX_NOHUGE) == PISA_IDX_NONE)
+ dev_warn_ratelimited(dev, "pisa: domain lacks a %luKB page size, not PISA\n",
+ (unsigned long)(PISA_SIZE >> 10));
+ return -EOPNOTSUPP;
+ }
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+
+ /* Bail if another CPU got here first, or if declined earlier. */
+ ret = win->base ? 0 : pisa_window_declined(win);
+ if (win->base || ret) {
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+ return ret;
+ }
+
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (!pisa_domains[idx].domain && !pisa_domains[idx].base)
+ break;
+ }
+ if (idx == PISA_MAX_DOMAINS) {
+ win->idx_gen = pisa_idx_gen;
+ WRITE_ONCE(win->domain_idx, PISA_IDX_NOIDX);
+ ret = -EBUSY;
+ } else {
+ base = pisa_dma_window_alloc(dev, domain, size);
+ if (!base) {
+ WRITE_ONCE(win->domain_idx, PISA_IDX_NOWINDOW);
+ ret = -ENOSPC;
+ } else {
+ pisa_domains[idx].domain = domain;
+ pisa_domains[idx].base = base;
+ spin_lock_init(&pisa_domains[idx].map_lock);
+ WRITE_ONCE(win->domain_idx, idx + PISA_IDX_FIRST);
+ /* Publish .domain_idx before .base (paired with is_pisa_iova()). */
+ smp_store_release(&win->base, base);
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ if (ret == -EBUSY)
+ pr_warn_ratelimited("pisa: all %d domain indices in use, not PISA for domain %p until one is released\n",
+ PISA_MAX_DOMAINS, domain);
+ else if (ret == -ENOSPC)
+ dev_warn_ratelimited(dev,
+ "pisa: no free %llu MB IOVA range for a PISA window\n",
+ size >> 20);
+ else
+ dev_info(dev, "pisa: domain %p index %d window %pad + %llu MB\n",
+ domain, idx, &base, size >> 20);
+ return ret;
+}
+
+/*
+ * Compute the window IOVA of active address @phys for @dev and store it in
+ * *@iova. On first use in a domain, reserves @win via alloc_iova(), and on
+ * first use of a PISA page in that domain, installs its 2MB leaf PTE via
+ * iommu_map(..., GFP_ATOMIC) under spinlocks.
+ * Returns 0 on success, or negative errno if @phys/@prot/@dev cannot use the window.
+ */
+int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ struct pisa_window *win = pisa_dma_window(domain);
+ unsigned long pfn = phys >> PAGE_SHIFT, flags;
+ struct pisa_meta *meta;
+ unsigned int domain_idx;
+ int block_prot, ret;
+ dma_addr_t addr;
+ u64 limit;
+
+ if (unlikely(!iova || !dev || !win || !dev_is_dma_coherent(dev) ||
+ (phys & (PISA_SIZE - 1)) + size > PISA_SIZE))
+ return -EINVAL;
+
+ meta = pisa_meta_of_pfn(pfn);
+
+ /* Every leaf PTE of the page carries IOMMU_CACHE | @meta->prot. */
+ block_prot = IOMMU_CACHE | meta->prot;
+ if (unlikely(prot & ~block_prot))
+ return -EACCES;
+
+ /* Acquire pairs with the .base release in pisa_window_assign(). */
+ if (unlikely(!smp_load_acquire(&win->base))) {
+ /* Test cached decline in @domain_idx before taking @pisa_domains_lock. */
+ ret = pisa_window_declined(win);
+ if (likely(!ret))
+ ret = pisa_window_assign(dev, domain, win);
+
+ if (ret)
+ return ret;
+ }
+
+ addr = pisa_phys_to_iova(win, meta, phys);
+
+ /* Check @dev's own DMA mask and bus limit against the window IOVA. */
+ limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+ if (unlikely(addr + size - 1 > limit))
+ return -ENOSPC;
+
+ domain_idx = win->domain_idx - PISA_IDX_FIRST;
+ if (likely(test_bit(domain_idx, meta->domains_mapped))) {
+ *iova = addr;
+ return 0;
+ }
+
+ /* First use of this PISA page in this domain: install the leaf PTE. */
+ ret = 0;
+ spin_lock_irqsave(&pisa_domains[domain_idx].map_lock, flags);
+ if (!test_bit(domain_idx, meta->domains_mapped)) {
+ phys_addr_t pisa_phys = phys & PISA_MASK;
+
+ ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
+ pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+ if (likely(!ret)) {
+ /* Ensure leaf PTE is visible before setting the domain bit. */
+ smp_mb__before_atomic();
+ set_bit(domain_idx, meta->domains_mapped);
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains[domain_idx].map_lock, flags);
+ if (unlikely(ret))
+ return ret;
+
+ *iova = addr;
+ return 0;
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f97f69a13d1a9..8d83ec09c19a9 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -2,7 +2,7 @@
#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
#define _DRIVERS_IOMMU_PISA_PRIV_H
-#include <linux/dma-direction.h>
+#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/jiffies.h>
#include <linux/list.h>
@@ -13,6 +13,7 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+struct device;
struct iommu_domain;
struct iommu_iotlb_gather;
@@ -53,6 +54,7 @@ enum {
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ * - pisa_domains[].map_lock (spinlock, irqsave): first-time iommu_map() in __pisa_map_phys().
* - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
* - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
@@ -60,6 +62,7 @@ enum {
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE / PISA_QUARANTINED
+ * @prot: Owning allocator's IOMMU_READ/WRITE protection bits
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
@@ -74,6 +77,7 @@ enum {
*/
struct pisa_meta {
u8 flags;
+ u8 prot;
u16 nr_clean;
u16 nr_dirty;
u8 on_list;
@@ -105,6 +109,7 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
* struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
* @lock: Spinlock protecting lists and block bitmaps
* @order: Block order managed by this allocator (<= PISA_ORDER)
+ * @prot: IOMMU_READ and/or IOMMU_WRITE from @dir passed to pisa_create()
* @partial: Partially used PISA pages with @nr_clean > 0
* @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
* @idle: Completely free PISA pages (reclaimable under pressure)
@@ -120,6 +125,7 @@ struct pisa_alloc {
/* First cacheline: hot fields touched on block alloc and free. */
spinlock_t lock;
u8 order;
+ u8 prot;
struct list_head partial;
struct list_head partial_dirty;
struct list_head idle;
@@ -148,6 +154,14 @@ struct pisa_node {
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+/* IOVA of @phys inside active RAM page @meta in @win. */
+static inline dma_addr_t pisa_phys_to_iova(const struct pisa_window *win,
+ const struct pisa_meta *meta, phys_addr_t phys)
+{
+ return win->base + ((dma_addr_t)meta->slot << PISA_SHIFT) +
+ (phys & (PISA_SIZE - 1));
+}
+
/* Offset of @meta from the start of a domain's window. */
static inline dma_addr_t pisa_meta_win_offset(const struct pisa_meta *meta)
{
@@ -228,6 +242,18 @@ void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
int pisa_unmap_all(struct pisa_meta *meta);
+int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win);
+int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot);
+
+static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ return is_pisa_phys(phys) ?
+ __pisa_map_phys(iova, dev, domain, phys, size, prot) : -EINVAL;
+}
/* Return negative errno if @win's domain was permanently/currently declined, or 0. */
static inline int pisa_window_declined(const struct pisa_window *win)
@@ -265,6 +291,13 @@ static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ return -EOPNOTSUPP;
+}
+
static inline int pisa_window_declined(const struct pisa_window *win)
{
return -EOPNOTSUPP;
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index ab1c32e065bbb..c64689b81befe 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -5,11 +5,14 @@
#include <linux/types.h>
#include <linux/bitops.h>
+#include <linux/dma-mapping.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
#include <linux/numa.h>
#include <linux/rcupdate.h>
+struct device;
+
#ifdef CONFIG_PISA
#define PISA_SHIFT 21U
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (33 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
` (15 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Because of the performance and security implications, the use of PISA
features is opt-in and configurable per device with a fine granularity.
Add boolean attributes to struct device exposed under
/sys/devices/*/*/pisa_*:
- pisa_coherent: enable PISA for coherent allocations
- pisa_rx_bufs: enable PISA for RX buffers / page_pool
- pisa_tx_bufs: enable PISA for TX socket page frags
- pisa_tx_skb_heads: enable PISA slab cache for small TX skb heads
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 64 ++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-alloc.c | 24 ++++++++++++++
drivers/iommu/pisa-kunit.c | 67 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 1 -
include/linux/device.h | 12 +++++++
include/linux/pisa.h | 19 +++++++++++
6 files changed, 186 insertions(+), 1 deletion(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index cbe05197d4a0b..97cd2d03ff77c 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -25,6 +25,7 @@
#include <linux/notifier.h>
#include <linux/of.h>
#include <linux/of_device.h>
+#include <linux/pisa.h>
#include <linux/pm_runtime.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
@@ -2907,6 +2908,54 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
}
static const DEVICE_ATTR_RO(removable);
+/* Per-device configuration to selectively enable PISA */
+static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
+{
+ bool val;
+ int ret = kstrtobool(buf, &val);
+
+ if (ret < 0)
+ return ret;
+ if (val) {
+ ret = pisa_meta_init();
+ if (ret < 0)
+ return ret;
+ }
+ WRITE_ONCE(*field, val);
+ return count;
+}
+
+#define PISA_BOOL_ATTR(name) \
+static ssize_t name##_show(struct device *dev, \
+ struct device_attribute *attr, char *buf) \
+{ \
+ return sysfs_emit(buf, "%u\n", READ_ONCE(dev->name)); \
+} \
+static ssize_t name##_store(struct device *dev, \
+ struct device_attribute *attr, \
+ const char *buf, size_t count) \
+{ \
+ return pisa_attr_store(buf, count, &dev->name); \
+} \
+static DEVICE_ATTR_RW(name)
+
+PISA_BOOL_ATTR(pisa_coherent);
+PISA_BOOL_ATTR(pisa_rx_bufs);
+PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_BOOL_ATTR(pisa_tx_skb_heads);
+
+static struct attribute *dev_attr_pisa[] = {
+ &dev_attr_pisa_coherent.attr,
+ &dev_attr_pisa_rx_bufs.attr,
+ &dev_attr_pisa_tx_bufs.attr,
+ &dev_attr_pisa_tx_skb_heads.attr,
+ NULL,
+};
+
+static const struct attribute_group dev_attr_pisa_group = {
+ .attrs = dev_attr_pisa,
+};
+
int device_add_groups(struct device *dev,
const struct attribute_group *const *groups)
{
@@ -3014,8 +3063,20 @@ static int device_add_attrs(struct device *dev)
goto err_remove_dev_removable;
}
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
+ error = device_add_group(dev, &dev_attr_pisa_group);
+ if (error)
+ goto err_remove_dev_physical_location;
+ }
+
return 0;
+ err_remove_dev_physical_location:
+ if (dev->physical_location) {
+ device_remove_group(dev, &dev_attr_physical_location_group);
+ kfree(dev->physical_location);
+ dev->physical_location = NULL;
+ }
err_remove_dev_removable:
device_remove_file(dev, &dev_attr_removable);
err_remove_dev_waiting_for_supplier:
@@ -3039,6 +3100,9 @@ static void device_remove_attrs(struct device *dev)
const struct class *class = dev->class;
const struct device_type *type = dev->type;
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+ device_remove_group(dev, &dev_attr_pisa_group);
+
if (dev->physical_location) {
device_remove_group(dev, &dev_attr_physical_location_group);
kfree(dev->physical_location);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index f98142882519c..b0d968aa9181c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/export.h>
@@ -392,6 +393,29 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred
return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
}
+/**
+ * pisa_has_free_blocks_rx - Check if @dev has free PISA RX blocks of @order on @nid
+ * @dev: Device to check
+ * @order: Block order
+ * @nid: Target NUMA node (or NUMA_NO_NODE)
+ *
+ * Return: true if @dev->pisa_rx_bufs is enabled and the per-node allocator
+ * has a free block available.
+ */
+bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid)
+{
+ struct pisa_alloc *pa;
+
+ if (!dev || !READ_ONCE(dev->pisa_rx_bufs))
+ return false;
+ pa = pisa_node_pa(nid, order, false);
+ return pa &&
+ (!list_empty_careful(&pa->partial) ||
+ !list_empty_careful(&pa->partial_dirty) ||
+ !list_empty_careful(&pa->idle));
+}
+EXPORT_SYMBOL(pisa_has_free_blocks_rx);
+
/*
* Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
* pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index e16268d55f3de..ab64657ad5c58 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -5,6 +5,8 @@
#include <kunit/test.h>
#include <linux/gfp.h>
#include <linux/pisa.h>
+#include <linux/kernfs.h>
+#include <linux/sysfs.h>
#include <linux/iommu.h>
#include <linux/iova.h>
#include <linux/mm.h>
@@ -350,6 +352,70 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
}
#endif /* CONFIG_IOMMU_DMA */
+static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
+ const char *name, bool *field)
+{
+ const struct sysfs_ops *ops = dev->kobj.ktype->sysfs_ops;
+ struct kernfs_node *kn;
+ char *buf;
+
+ buf = (char *)__get_free_page(GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, buf);
+ if (!buf)
+ return;
+ kn = kernfs_find_and_get(dev->kobj.sd, name);
+ KUNIT_EXPECT_NOT_NULL(test, kn);
+ if (!kn)
+ goto out_free;
+
+ KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+ KUNIT_EXPECT_STREQ(test, buf, "0\n");
+ KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+ KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "1\n", 2), 2);
+ KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+ KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+ KUNIT_EXPECT_STREQ(test, buf, "1\n");
+
+ KUNIT_EXPECT_LT(test, ops->store(&dev->kobj, kn->priv, "bad", 3), 0);
+ KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+
+ KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "0\n", 2), 2);
+ KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+ kernfs_put(kn);
+out_free:
+ free_page((unsigned long)buf);
+}
+
+static void kunit_attr_dev_release(struct device *dev)
+{
+ kfree(dev);
+}
+
+static void test_dev_pisa_attrs(struct kunit *test)
+{
+ struct device *dev;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, dev);
+ dev->init_name = "pisa_attr_kunit";
+ dev->release = kunit_attr_dev_release;
+ dev->coherent_dma_mask = DMA_BIT_MASK(64);
+ dev->dma_mask = &dev->coherent_dma_mask;
+
+ if (device_register(dev)) {
+ put_device(dev);
+ KUNIT_FAIL(test, "device_register failed");
+ return;
+ }
+ kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
+ kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ device_unregister(dev);
+}
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -365,6 +431,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_window_helpers),
KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
+ KUNIT_CASE(test_dev_pisa_attrs),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 8d83ec09c19a9..e5ca95d2654cd 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -133,7 +133,6 @@ struct pisa_alloc {
unsigned int num_idle_pages;
unsigned int max_idle_pages;
unsigned long idle_jiffies;
-
/* Cold. */
int nid;
struct list_head node;
diff --git a/include/linux/device.h b/include/linux/device.h
index 429e39f647cf9..b3c11e32116ca 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -695,6 +695,13 @@ enum struct_device_flags {
* @removable: Whether the device can be removed from the system. This
* should be set by the subsystem / bus driver that discovered
* the device.
+ * @pisa_coherent: Back coherent dma_alloc_attrs() with PISA
+ * (sysfs pisa_coherent).
+ * @pisa_rx_bufs: Use PISA allocators for RX buffers (sysfs pisa_rx_bufs).
+ * @pisa_tx_bufs: Use PISA allocators for socket TX page fragments
+ * (sysfs pisa_tx_bufs).
+ * @pisa_tx_skb_heads: Use PISA slab cache for small TX skb heads
+ * (sysfs pisa_tx_skb_heads).
* @flags: DEV_FLAG_XXX flags. Use atomic bitfield operations to modify.
*
* At the lowest level, every device in a Linux system is represented by an
@@ -799,6 +806,11 @@ struct device {
enum device_removable removable;
+ bool pisa_coherent;
+ bool pisa_rx_bufs;
+ bool pisa_tx_bufs;
+ bool pisa_tx_skb_heads;
+
DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
};
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index c64689b81befe..92482fce943e9 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -7,6 +7,7 @@
#include <linux/bitops.h>
#include <linux/dma-mapping.h>
#include <linux/jump_label.h>
+#include <linux/device.h>
#include <linux/mm.h>
#include <linux/numa.h>
#include <linux/rcupdate.h>
@@ -82,6 +83,13 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int nr_pages, struct page **page_array,
bool frozen);
+bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
+
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+ return dev && READ_ONCE(dev->pisa_rx_bufs) ? __GFP_PISA_RW : 0;
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -109,6 +117,17 @@ pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
return 0;
}
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+ return 0;
+}
+
+static inline bool pisa_has_free_blocks_rx(const struct device *dev,
+ unsigned int order, int nid)
+{
+ return false;
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (34 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
` (14 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Coherent DMA allocations (e.g. TX/RX descriptor rings allocated via
dma_alloc_attrs() / dma_alloc_coherent()) are long-lived and accessed
on every packet or I/O operation, so they benefit from sharing 2MB IOMMU
leaf mappings.
When dev->pisa_coherent is enabled, serve coherent DMA allocations up
to 2MB from the device's per-NUMA-node order-0 read-write PISA
allocator.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-iommu.c | 10 +++
drivers/iommu/pisa-alloc.c | 146 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 67 +++++++++++++++++
include/linux/pisa.h | 30 ++++++++
kernel/dma/direct.c | 9 +++
5 files changed, 262 insertions(+)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index eb920ad77c783..a0c5ddab7a8b6 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1658,6 +1658,8 @@ static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr)
} else {
/* Lowmem means a coherent atomic or CMA allocation */
page = virt_to_page(cpu_addr);
+ if (pisa_free_coherent(page, alloc_size))
+ return;
}
if (pages)
@@ -1716,6 +1718,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
gfp_t gfp, unsigned long attrs)
{
bool coherent = dev_is_dma_coherent(dev);
+ struct iommu_domain *domain = iommu_get_dma_domain(dev);
int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs);
bool is_alloc_cc_shared = attrs & __DMA_ATTR_ALLOC_CC_SHARED;
struct page *page = NULL;
@@ -1727,6 +1730,12 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
gfp |= __GFP_ZERO;
+ if (domain) {
+ cpu_addr = pisa_alloc_coherent(&page, dev, domain, size, gfp, attrs);
+ if (cpu_addr)
+ goto map;
+ }
+
if (gfpflags_allow_blocking(gfp) &&
!(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) {
return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs);
@@ -1744,6 +1753,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
return NULL;
}
+map:
*handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot,
dev->coherent_dma_mask);
if (*handle == DMA_MAPPING_ERROR) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index b0d968aa9181c..25e4645310678 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -10,6 +10,8 @@
#include <linux/bitops.h>
#include <linux/cache.h>
#include <linux/device.h>
+#include <linux/dma-direct.h>
+#include <linux/dma-map-ops.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/export.h>
@@ -697,6 +699,150 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
return i;
}
+/* Find and reserve @nblocks aligned contiguous blocks in @meta under @pa->lock. */
+static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblocks,
+ unsigned long align_mask)
+{
+ DECLARE_BITMAP(busy, PISA_MAX_BLOCKS);
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int bpp = PISA_BLOCKS(0), i;
+ bool poisoned = false;
+ unsigned long off;
+ struct page *page;
+
+ if (pisa_meta_avail(meta) < nblocks)
+ return NULL;
+
+ bitmap_or(busy, meta->clean_bitmap, meta->dirty_bitmap, bpp);
+ bitmap_complement(busy, busy, bpp);
+
+ off = 0;
+ for (;;) {
+ off = bitmap_find_next_zero_area(busy, bpp, off, nblocks, align_mask);
+ if (off >= bpp) {
+ if (poisoned) {
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+ pisa_place_meta(meta);
+ }
+ return NULL;
+ }
+
+ page = pfn_to_page(pisa_meta_to_pfn(meta) + off);
+ for (i = 0; i < nblocks; i++) {
+ if (!pisa_block_poisoned(pa, page + i))
+ continue;
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap))
+ meta->nr_clean--;
+ else if (__test_and_clear_bit(off + i, meta->dirty_bitmap))
+ meta->nr_dirty--;
+ __set_bit(off + i, busy);
+ poisoned = true;
+ off += i + 1;
+ break;
+ }
+ if (i == nblocks)
+ break;
+ }
+
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+
+ for (i = 0; i < nblocks; i++) {
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
+ meta->nr_clean--;
+ } else {
+ __clear_bit(off + i, meta->dirty_bitmap);
+ meta->nr_dirty--;
+ }
+ pisa_block_prep(page + i);
+ page_ref_unfreeze(page + i, 1);
+ }
+ pisa_place_meta(meta);
+ return page;
+}
+
+/*
+ * Specialized memory provider for iommu_dma_alloc() and dma_direct_alloc():
+ * allocate a zeroed, naturally aligned contiguous range of order-0 pages for
+ * @size bytes (<= PISA_SIZE) from @dev's per-node coherent allocator into
+ * *@pagep and return its kernel virtual address, or NULL.
+ */
+void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
+ struct iommu_domain *domain, size_t size,
+ gfp_t gfp, unsigned long attrs)
+{
+ bool should_release = false;
+ struct list_head *lists[3];
+ unsigned long align_mask, flags;
+ unsigned int nblocks, i;
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ struct page *page = NULL;
+
+ size = PAGE_ALIGN(size);
+ if (!pagep || !dev || !READ_ONCE(dev->pisa_coherent) ||
+ !dev_is_dma_coherent(dev) || !size || size > PISA_SIZE)
+ return NULL;
+ if (min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) <= DMA_BIT_MASK(32))
+ return NULL;
+ if ((current_gfp_context(gfp) & ~(__GFP_ZERO | __GFP_NOWARN)) != GFP_KERNEL ||
+ (attrs & ~DMA_ATTR_NO_WARN))
+ return NULL;
+
+ if (domain) {
+ struct pisa_window *win = pisa_dma_window(domain);
+
+ if (!win || pisa_window_declined(win) || !(domain->pgsize_bitmap & PISA_SIZE))
+ return NULL;
+ } else if (is_swiotlb_for_alloc(dev) ||
+ force_dma_unencrypted(dev) ||
+ !dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
+ return NULL;
+ }
+
+ pa = pisa_node_pa(dev_to_node(dev), 0, false);
+ if (!pa)
+ return NULL;
+
+ nblocks = size >> PAGE_SHIFT;
+ align_mask = (1UL << get_order(size)) - 1;
+ lists[0] = &pa->partial;
+ lists[1] = &pa->partial_dirty;
+ lists[2] = &pa->idle;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < ARRAY_SIZE(lists) && !page; i++) {
+ list_for_each_entry(meta, lists[i], list) {
+ page = pisa_take_contig(meta, nblocks, align_mask);
+ if (page)
+ break;
+ }
+ }
+ if (!page) {
+ spin_unlock_irqrestore(&pa->lock, flags);
+ meta = pisa_alloc_split_page(0, gfp, pa->nid);
+ if (IS_ERR(meta))
+ return NULL;
+ spin_lock_irqsave(&pa->lock, flags);
+ meta->pa = pa;
+ meta->prot = pa->prot;
+ WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ page = pisa_take_contig(meta, nblocks, align_mask);
+ if (unlikely(!page))
+ should_release = pisa_place_meta(meta);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+ if (!page)
+ return NULL;
+ *pagep = page;
+ return memset(page_address(page), 0, size);
+}
/**
* __pisa_free_page - Recycle a freed block back into its PISA allocator
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index ab64657ad5c58..72ee8986b4802 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -352,6 +352,72 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
}
#endif /* CONFIG_IOMMU_DMA */
+static void test_coherent_alloc_and_free(struct kunit *test)
+{
+ struct device dev = { .init_name = "pisa_kunit", .numa_node = NUMA_NO_NODE,
+ .coherent_dma_mask = DMA_BIT_MASK(64),
+ .pisa_coherent = true };
+ struct page *p1, *p2, *p3;
+ void *v1, *v2, *v3;
+#ifdef CONFIG_IOMMU_DMA
+ struct pisa_window *win;
+ dma_addr_t d1, d2;
+#endif
+
+ dev.dma_mask = &dev.coherent_dma_mask;
+ /* NUMA_NO_NODE would follow the node of the CPU the test runs on. */
+ set_dev_node(&dev, first_memory_node);
+#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
+ dev.dma_coherent = true;
+#endif
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, NULL, NULL, SZ_4K, GFP_KERNEL, 0));
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, NULL, SZ_4K,
+ GFP_KERNEL | __GFP_ACCOUNT, 0));
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, NULL, PISA_SIZE + SZ_4K,
+ GFP_KERNEL, 0));
+
+#ifdef CONFIG_IOMMU_DMA
+ /* Uninitialized domain has no DMA cookie/window yet. */
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, &fake_domain, SZ_4K,
+ GFP_KERNEL | __GFP_NOWARN, 0));
+
+ KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+ win = pisa_dma_window(&fake_domain);
+
+ v1 = pisa_alloc_coherent(&p1, &dev, &fake_domain, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v1);
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, page_to_phys(p1), SZ_64K,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(win, d1));
+
+ v2 = pisa_alloc_coherent(&p2, &dev, &fake_domain, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v2);
+ KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, page_to_phys(p2), SZ_64K,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1 + SZ_64K);
+#else
+ v1 = pisa_alloc_coherent(&p1, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v1);
+ v2 = pisa_alloc_coherent(&p2, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v2);
+ KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+#endif
+
+ /* Freeing p1 returns its 16 order-0 pages; next 64K alloc reuses them. */
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p1, SZ_64K));
+ v3 = pisa_alloc_coherent(&p3, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v3);
+ KUNIT_EXPECT_PTR_EQ(test, p3, p1);
+
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p2, SZ_64K));
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p3, SZ_64K));
+}
+
static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
const char *name, bool *field)
{
@@ -431,6 +497,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_window_helpers),
KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
+ KUNIT_CASE(test_coherent_alloc_and_free),
KUNIT_CASE(test_dev_pisa_attrs),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 92482fce943e9..a6556aa403a09 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -13,6 +13,7 @@
#include <linux/rcupdate.h>
struct device;
+struct iommu_domain;
#ifdef CONFIG_PISA
@@ -78,6 +79,21 @@ static inline bool pisa_free_page(struct page *page)
return ret;
}
+void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
+ struct iommu_domain *domain, size_t size,
+ gfp_t gfp, unsigned long attrs);
+
+static inline bool pisa_free_coherent(struct page *page, size_t size)
+{
+ unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+ if (!page || !is_pisa_page(page))
+ return false;
+ while (count--)
+ __free_page(page++);
+ return true;
+}
+
unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int preferred_nid, nodemask_t *nodemask,
int nr_pages, struct page **page_array,
@@ -109,6 +125,20 @@ static inline bool pisa_free_page(struct page *page)
return false;
}
+static inline void *pisa_alloc_coherent(struct page **pagep,
+ struct device *dev,
+ struct iommu_domain *domain,
+ size_t size, gfp_t gfp,
+ unsigned long attrs)
+{
+ return NULL;
+}
+
+static inline bool pisa_free_coherent(struct page *page, size_t size)
+{
+ return false;
+}
+
static inline unsigned long
pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 25e9a86835fd3..11052b65e573b 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -15,6 +15,7 @@
#include <linux/slab.h>
#include <linux/pci-p2pdma.h>
#include <linux/cc_platform.h>
+#include <linux/pisa.h>
#include "direct.h"
@@ -244,6 +245,12 @@ void *dma_direct_alloc(struct device *dev, size_t size,
if (attrs & DMA_ATTR_NO_WARN)
gfp |= __GFP_NOWARN;
+ cpu_addr = pisa_alloc_coherent(&page, dev, NULL, size, gfp, attrs);
+ if (cpu_addr) {
+ *dma_handle = phys_to_dma_direct(dev, page_to_phys(page), false);
+ return cpu_addr;
+ }
+
if (dma_direct_use_no_mapping(dev, attrs))
return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp);
@@ -424,6 +431,8 @@ void dma_direct_free(struct device *dev, size_t size,
if (is_vmalloc_addr(cpu_addr)) {
vunmap(cpu_addr);
+ } else if (pisa_free_coherent(virt_to_page(cpu_addr), size)) {
+ return;
} else {
if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED))
arch_dma_clear_uncached(cpu_addr, size);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (35 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
` (13 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Every block allocation and free takes the allocator's lock. Highly
concurrent users e.g. socket tx fragments, can cause significant lock
contention.
Add pisa_enable_pcp(), an opt-in per-CPU front cache, similar
to the page allocator's per-CPU lists:
- Each per-CPU cache keeps up to 64 free blocks, and refills it from
the main allocator in batches of 32.
- returned blocks belonging to the local NUMA node are kept in the local
cache.
- returned blocks belonging to a remote NUMA node are returned to the
proper per-node allocator.
The cache is fully integrated with the shrinker and CPU offlining to
drain the cache.
Enable per-CPU caches for the per-node order-0 RW and SKB_FRAG_PAGE_ORDER
RO allocators.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 342 +++++++++++++++++++++++++++++++++++--
drivers/iommu/pisa-kunit.c | 78 +++++++++
drivers/iommu/pisa-priv.h | 35 +++-
include/linux/pisa.h | 6 +
4 files changed, 448 insertions(+), 13 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 25e4645310678..913706a503341 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/cpuhotplug.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
#include <linux/dma-map-ops.h>
@@ -24,6 +25,7 @@
#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/page_owner.h>
+#include <linux/percpu.h>
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
@@ -60,6 +62,10 @@ module_param_named(max_idle, pisa_max_idle, uint, 0644);
static unsigned int pisa_idle_ms __read_mostly = 200;
module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
+/* Stop recycling partial-page blocks when free blocks exceed @pisa_compact PISA pages. */
+static unsigned int pisa_compact __read_mostly;
+module_param_named(compact, pisa_compact, uint, 0644);
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -90,7 +96,11 @@ static void pisa_release_page_rcu(struct rcu_head *head)
struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
unsigned int i, order = meta->pa->order, slot = meta->slot;
+ unsigned long flags;
+ spin_lock_irqsave(&meta->pa->lock, flags);
+ meta->pa->stats.page_free_cnt++;
+ spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
/* Possibly still mapped by a device: leak it, still charged. */
@@ -355,9 +365,223 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
__set_bit(idx, meta->dirty_bitmap);
meta->nr_dirty++;
}
+ pa->blocks_in_use--;
*release |= pisa_place_meta(meta);
}
+/**
+ * pisa_page_can_recycle - Check if a block can stay cached in a caller pool
+ * @page: Block head page
+ *
+ * Called on the recycle fast path (per-CPU cache, TX page_frag, page_pool).
+ * Fast-path cost is a static-key branch plus a READ_ONCE() of module parameters
+ * (~1-2 ns when compaction is off); when enabled, it does a lockless RCU +
+ * metadata array lookup (~5-10 ns).
+ *
+ * Return: true if @page can be recycled in place, false if it should be
+ * released back to its allocator so sparse 2MB pages can drain.
+ */
+bool pisa_page_can_recycle(const struct page *page)
+{
+ unsigned long bpp, in_use, free_blocks, pfn;
+ const struct pisa_meta *meta;
+ const struct pisa_alloc *pa;
+ unsigned int compact;
+ bool ret = true;
+ u8 on_list;
+ long held;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return true;
+
+ compact = READ_ONCE(pisa_compact);
+ if (!compact)
+ return true;
+
+ pfn = page_to_pfn(page);
+ rcu_read_lock();
+ if (!is_pisa_pfn(pfn))
+ goto out;
+
+ meta = pisa_meta_of_pfn(pfn);
+ on_list = READ_ONCE(meta->on_list);
+ if (on_list == PISA_LIST_FULL || on_list == PISA_LIST_NONE)
+ goto out;
+
+ pa = meta->pa;
+ if (!pa)
+ goto out;
+
+ /* Do not drain the partial page currently being filled at list head. */
+ if (READ_ONCE(pa->partial_dirty.next) == &meta->list ||
+ READ_ONCE(pa->partial.next) == &meta->list)
+ goto out;
+
+ bpp = PISA_BLOCKS(pa->order);
+ /* Do not drain dense pages that still have more than half their blocks in use. */
+ if (READ_ONCE(meta->nr_clean) + READ_ONCE(meta->nr_dirty) < (bpp >> 1))
+ goto out;
+
+ held = pisa_alloc_held_pages(pa);
+ in_use = READ_ONCE(pa->blocks_in_use);
+ free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
+ (unsigned long)held * bpp - in_use : 0;
+
+ ret = free_blocks <= (unsigned long)compact * bpp;
+out:
+ rcu_read_unlock();
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pisa_page_can_recycle);
+
+/* Return @n freed blocks to @pa under a single acquisition of @pa->lock. */
+static void pisa_free_bulk(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+{
+ bool release = false;
+ unsigned long flags;
+ unsigned int i;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < n; i++) {
+ if (likely(!pisa_block_poisoned(pa, blocks[i])))
+ pisa_return_block_locked(pa, blocks[i], &release);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (release)
+ pisa_schedule_reclaim();
+}
+
+/*
+ * Per-CPU block cache: each CPU caches up to PISA_PCP_HIGH free blocks
+ * (refcount 0, not in bitmaps), refilling and flushing in PISA_PCP_BATCH
+ * batches. Drained by the shrinker and CPU hotplug.
+ */
+
+/* Max blocks moved per per-CPU cache refill or flush (1..PISA_PCP_BATCH). */
+static unsigned int pisa_pcp_batch __read_mostly = PISA_PCP_BATCH;
+
+static int pisa_pcp_batch_set(const char *val, const struct kernel_param *kp)
+{
+ return param_set_uint_minmax(val, kp, 1, PISA_PCP_BATCH);
+}
+
+static const struct kernel_param_ops pisa_pcp_batch_ops = {
+ .set = pisa_pcp_batch_set,
+ .get = param_get_uint,
+};
+module_param_cb(pcp_batch, &pisa_pcp_batch_ops, &pisa_pcp_batch, 0644);
+
+/* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
+static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+{
+ unsigned int i, batch = READ_ONCE(pisa_pcp_batch);
+ struct pisa_pcp *pcp;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ pcp = this_cpu_ptr(pa->pcp);
+ spin_lock(&pcp->lock);
+ for (i = 0; i < n; i++) {
+ if (pcp->count == PISA_PCP_HIGH) {
+ /* Keep the most recently freed, likely still in cache. */
+ pisa_free_bulk(pa, pcp->blocks, batch);
+ pcp->count -= batch;
+ memmove(pcp->blocks, pcp->blocks + batch,
+ pcp->count * sizeof(*pcp->blocks));
+ }
+ pcp->blocks[pcp->count++] = blocks[i];
+ }
+ spin_unlock(&pcp->lock);
+ local_irq_restore(flags);
+}
+
+/* Take up to @nr blocks, at refcount 0, from this CPU's cache. */
+static unsigned int pisa_pcp_pop(struct pisa_alloc *pa, unsigned int nr, struct page **out)
+{
+ struct pisa_pcp *pcp;
+ unsigned int got = 0;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ pcp = this_cpu_ptr(pa->pcp);
+ spin_lock(&pcp->lock);
+ while (got < nr && pcp->count)
+ out[got++] = pcp->blocks[--pcp->count];
+ spin_unlock(&pcp->lock);
+ local_irq_restore(flags);
+
+ return got;
+}
+
+/* Return the blocks @cpu caches for @pa. Process context. */
+static void pisa_pcp_drain_cpu(struct pisa_alloc *pa, int cpu)
+{
+ struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+ unsigned long flags;
+
+ spin_lock_irqsave(&pcp->lock, flags);
+ if (pcp->count) {
+ pisa_free_bulk(pa, pcp->blocks, pcp->count);
+ pcp->count = 0;
+ }
+ spin_unlock_irqrestore(&pcp->lock, flags);
+}
+
+/* Drain every non-empty CPU cache of @pa. */
+void pisa_pcp_drain(struct pisa_alloc *pa)
+{
+ int cpu;
+
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (!smp_load_acquire(&pa->pcp))
+ return;
+ for_each_possible_cpu(cpu) {
+ if (!READ_ONCE(per_cpu_ptr(pa->pcp, cpu)->count))
+ continue;
+ pisa_pcp_drain_cpu(pa, cpu);
+ }
+}
+
+/* Blocks in @pa's caches, for shrinker and debugfs. */
+static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+{
+ unsigned long n = 0;
+ int cpu;
+
+ if (!pa->pcp)
+ return 0;
+ for_each_possible_cpu(cpu) {
+ /* A racy sum is good enough for reclaim and statistics. */
+ n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+ }
+ return n;
+}
+
+/**
+ * pisa_enable_pcp - Put per-CPU block caches in front of an allocator
+ * @pa: Allocator to cache
+ *
+ * Return: 0, or -ENOMEM; @pa works without the cache.
+ */
+int pisa_enable_pcp(struct pisa_alloc *pa)
+{
+ struct pisa_pcp __percpu *pcp;
+ int cpu;
+
+ if (!pa || pa->pcp)
+ return 0;
+ pcp = alloc_percpu(struct pisa_pcp);
+ if (!pcp)
+ return -ENOMEM;
+ for_each_possible_cpu(cpu)
+ spin_lock_init(&per_cpu_ptr(pcp, cpu)->lock);
+ /* Publish initialized locks; readers depend on the pointer. */
+ if (cmpxchg_release(&pa->pcp, NULL, pcp))
+ free_percpu(pcp);
+ return 0;
+}
+
/*
* Return the predefined per-node allocator for (@nid, @order, @ro), or NULL.
*/
@@ -412,7 +636,9 @@ bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int n
return false;
pa = pisa_node_pa(nid, order, false);
return pa &&
- (!list_empty_careful(&pa->partial) ||
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ ((smp_load_acquire(&pa->pcp) && raw_cpu_read(pa->pcp->count)) ||
+ !list_empty_careful(&pa->partial) ||
!list_empty_careful(&pa->partial_dirty) ||
!list_empty_careful(&pa->idle));
}
@@ -532,6 +758,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
break;
}
*countp -= used;
+ pa->blocks_in_use += used;
return got;
}
@@ -638,6 +865,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
meta->prot = pa->prot;
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ pa->stats.page_alloc_cnt++;
}
spin_unlock_irqrestore(&pa->lock, flags);
@@ -656,6 +884,49 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
return n;
}
+/* Allocate @nr_pages blocks through this CPU's cache, refilling in batches on miss. */
+static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+ struct page **page_array, bool frozen)
+{
+ unsigned int pcp_batch = READ_ONCE(pisa_pcp_batch);
+ unsigned long got, n, i, j;
+
+ got = pisa_pcp_pop(pa, nr_pages, page_array);
+ if (got < nr_pages) {
+ if (nr_pages - got >= pcp_batch) {
+ got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
+ page_array + got, false, true);
+ } else {
+ struct page *batch[PISA_PCP_BATCH];
+
+ n = __pisa_alloc_bulk(pa, gfp, pcp_batch, batch, false, true);
+ for (i = 0; i < n && got < nr_pages; i++)
+ page_array[got++] = batch[i];
+ if (i < n)
+ pisa_pcp_push(pa, batch + i, n - i);
+ }
+ }
+
+ for (i = j = 0; i < got; i++) {
+ struct page *block = page_array[i];
+
+ page_array[i] = NULL;
+ /* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
+ if (pisa_block_poisoned(pa, block))
+ continue;
+ pisa_block_prep(block);
+ if (want_init_on_alloc(gfp)) {
+ memset(page_address(block), 0, PAGE_SIZE << pa->order);
+ if (frozen) /* Order zeroing before caller publishes the page. */
+ smp_wmb();
+ }
+ if (!frozen)
+ page_ref_unfreeze(block, 1);
+ page_array[j++] = block;
+ }
+ return j;
+}
+
/**
* pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
* @pa: Allocator to allocate from
@@ -681,11 +952,16 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
i++;
continue;
}
+
/* Fill the run of @n NULL entries at @i. */
for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
;
- got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
- want_init_on_alloc(gfp), frozen);
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp))
+ got = pisa_pcp_alloc(pa, gfp, n, page_array + i, frozen);
+ else
+ got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+ want_init_on_alloc(gfp), frozen);
i += got;
if (got < n)
break;
@@ -733,10 +1009,13 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
for (i = 0; i < nblocks; i++) {
if (!pisa_block_poisoned(pa, page + i))
continue;
- if (__test_and_clear_bit(off + i, meta->clean_bitmap))
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
meta->nr_clean--;
- else if (__test_and_clear_bit(off + i, meta->dirty_bitmap))
+ pa->blocks_in_use++;
+ } else if (__test_and_clear_bit(off + i, meta->dirty_bitmap)) {
meta->nr_dirty--;
+ pa->blocks_in_use++;
+ }
__set_bit(off + i, busy);
poisoned = true;
off += i + 1;
@@ -760,6 +1039,7 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
pisa_block_prep(page + i);
page_ref_unfreeze(page + i, 1);
}
+ pa->blocks_in_use += nblocks;
pisa_place_meta(meta);
return page;
}
@@ -806,6 +1086,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
pa = pisa_node_pa(dev_to_node(dev), 0, false);
if (!pa)
return NULL;
+ pisa_pcp_drain(pa);
nblocks = size >> PAGE_SHIFT;
align_mask = (1UL << get_order(size)) - 1;
@@ -830,6 +1111,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
meta->pa = pa;
meta->prot = pa->prot;
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ pa->stats.page_alloc_cnt++;
page = pisa_take_contig(meta, nblocks, align_mask);
if (unlikely(!page))
should_release = pisa_place_meta(meta);
@@ -876,6 +1158,17 @@ bool __pisa_free_page(struct page *page)
if (want_init_on_free())
memset(page_address(page), 0, PAGE_SIZE << pa->order);
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
+ page_to_nid(page) == numa_mem_id()) {
+ if (pisa_page_can_recycle(page)) {
+ pisa_pcp_push(pa, &page, 1);
+ return true;
+ }
+ if (this_cpu_read(pa->pcp->count))
+ pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
+ }
+
spin_lock_irqsave(&pa->lock, flags);
pisa_return_block_locked(pa, page, &should_release);
spin_unlock_irqrestore(&pa->lock, flags);
@@ -889,15 +1182,17 @@ bool __pisa_free_page(struct page *page)
/* Shrinker: releases reservoir pages first, then idle allocator pages. */
static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
{
- unsigned long nr = pisa_reservoir_count();
+ unsigned long nr = pisa_reservoir_count() << PISA_ORDER;
struct pisa_alloc *pa;
read_lock(&pisa_allocs_lock);
- list_for_each_entry(pa, &pisa_allocs, node)
- nr += READ_ONCE(pa->num_idle_pages);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
+ nr += pisa_pcp_count(pa) << pa->order;
+ }
read_unlock(&pisa_allocs_lock);
- return nr << PISA_ORDER;
+ return nr;
}
/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
@@ -930,6 +1225,8 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
if (freed < sc->nr_to_scan) {
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
+ /* Cached blocks keep PISA pages from going idle. */
+ pisa_pcp_drain(pa);
freed += pisa_shrink_list(&pa->lock, &pa->idle,
&pa->num_idle_pages, &release_list,
sc->nr_to_scan - freed);
@@ -952,15 +1249,30 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
return freed ?: SHRINK_STOP;
}
+/* Return the blocks a dead CPU cached, which nothing else would. */
+static int pisa_cpu_dead(unsigned int cpu)
+{
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node)
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp))
+ pisa_pcp_drain_cpu(pa, cpu);
+ read_unlock(&pisa_allocs_lock);
+ return 0;
+}
+
static int __init pisa_init(void)
{
const struct {
u8 order;
bool ro;
+ bool pcp;
} cfg[] = {
- { 0, false },
- { 2, true },
- { SKB_FRAG_PAGE_ORDER, true },
+ { 0, false, true },
+ { 2, true, false },
+ { SKB_FRAG_PAGE_ORDER, true, true },
};
struct shrinker *shrink;
int nid, i;
@@ -986,6 +1298,8 @@ static int __init pisa_init(void)
if (!pa)
continue;
pisa_set_name(pa, "pisa%s_o%u_n%d", ro ? "_ro" : "", order, nid);
+ if (cfg[i].pcp)
+ pisa_enable_pcp(pa);
pisa_nodes[nid].pa[ro][order] = pa;
}
}
@@ -995,6 +1309,10 @@ static int __init pisa_init(void)
if (!pisa_wq)
pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+ if (cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN, "iommu/pisa:dead", NULL,
+ pisa_cpu_dead) < 0)
+ pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
+
shrink = shrinker_alloc(0, "pisa");
if (!shrink) {
pr_warn("pisa: shrinker registration failed\n");
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 72ee8986b4802..5bbb540eb505e 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -482,6 +482,82 @@ static void test_dev_pisa_attrs(struct kunit *test)
device_unregister(dev);
}
+static void test_pcp_cache(struct kunit *test)
+{
+ struct page *blocks[PISA_PCP_HIGH + 1];
+ struct pisa_alloc *pa;
+ struct page *p1, *p2;
+ unsigned int i, n = 0;
+
+ KUNIT_EXPECT_EQ(test, pisa_enable_pcp(NULL), 0);
+ pa = pisa_node_pa(first_memory_node, 0, false);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ KUNIT_ASSERT_EQ(test, pisa_enable_pcp(pa), 0);
+
+ p1 = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ __free_page(p1);
+ p2 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+ KUNIT_EXPECT_PTR_EQ(test, p2, p1);
+ if (p2)
+ __free_page(p2);
+
+ /* Overflowing PISA_PCP_HIGH flushes a batch back to the allocator. */
+ for (i = 0; i < ARRAY_SIZE(blocks); i++) {
+ blocks[i] = pisa_alloc(pa, GFP_KERNEL);
+ if (!blocks[i])
+ break;
+ n++;
+ }
+ KUNIT_EXPECT_EQ(test, n, (unsigned int)ARRAY_SIZE(blocks));
+ for (i = 0; i < n; i++)
+ __free_page(blocks[i]);
+}
+
+static void test_gfp_pisa(struct kunit *test)
+{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct device dev = { .numa_node = first_memory_node };
+ struct page *p, *p_ro, *bulk[4] = {};
+ unsigned long n;
+ int i;
+
+ pisa_shrink_scan(NULL, &sc);
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(NULL), 0);
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), 0);
+ KUNIT_EXPECT_FALSE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+
+ dev.pisa_rx_bufs = true;
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), __GFP_PISA_RW);
+
+ p = alloc_pages_node(first_memory_node, GFP_KERNEL | pisa_rx_gfp(&dev), 0);
+ KUNIT_ASSERT_NOT_NULL(test, p);
+ if (p) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p));
+ KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p))->prot,
+ IOMMU_READ | IOMMU_WRITE);
+ __free_page(p);
+ KUNIT_EXPECT_TRUE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+ }
+
+ p_ro = alloc_pages(GFP_KERNEL | __GFP_COMP | __GFP_PISA_RO, 3);
+ KUNIT_ASSERT_NOT_NULL(test, p_ro);
+ if (p_ro) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p_ro));
+ KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p_ro))->prot,
+ IOMMU_READ);
+ __free_pages(p_ro, 3);
+ }
+
+ n = alloc_pages_bulk_node(GFP_KERNEL | __GFP_PISA_RW, first_memory_node,
+ ARRAY_SIZE(bulk), bulk);
+ KUNIT_EXPECT_EQ(test, n, ARRAY_SIZE(bulk));
+ for (i = 0; i < n; i++) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(bulk[i]));
+ __free_page(bulk[i]);
+ }
+}
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -499,6 +575,8 @@ static struct kunit_case pisa_meta_test_cases[] = {
#endif
KUNIT_CASE(test_coherent_alloc_and_free),
KUNIT_CASE(test_dev_pisa_attrs),
+ KUNIT_CASE(test_pcp_cache),
+ KUNIT_CASE(test_gfp_pisa),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index e5ca95d2654cd..8f87bdf7d4ee2 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -53,6 +53,7 @@ enum {
* Lock hierarchy:
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
+ * - pcp->lock (spinlock, irqsave): per-CPU block cache.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
* - pisa_domains[].map_lock (spinlock, irqsave): first-time iommu_map() in __pisa_map_phys().
* - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
@@ -105,11 +106,28 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
return meta->nr_clean + meta->nr_dirty;
}
+/* Per-CPU front cache of free blocks at refcount 0, see pisa_enable_pcp(). */
+#define PISA_PCP_HIGH 64U /* blocks cached per CPU at most */
+#define PISA_PCP_BATCH 32U /* blocks moved per refill or flush */
+
+struct pisa_pcp {
+ spinlock_t lock; /* by this CPU, or by a drain from any CPU */
+ unsigned int count;
+ struct page *blocks[PISA_PCP_HIGH];
+};
+
+/* Per-allocator statistics, see pisa_allocs_show(). */
+struct pisa_alloc_stats {
+ u64 page_alloc_cnt;
+ u64 page_free_cnt;
+};
+
/**
* struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
- * @lock: Spinlock protecting lists and block bitmaps
+ * @lock: Spinlock protecting lists, block bitmaps, and u64 @stats
* @order: Block order managed by this allocator (<= PISA_ORDER)
* @prot: IOMMU_READ and/or IOMMU_WRITE from @dir passed to pisa_create()
+ * @pcp: Per-CPU block caches, or NULL (see pisa_enable_pcp())
* @partial: Partially used PISA pages with @nr_clean > 0
* @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
* @idle: Completely free PISA pages (reclaimable under pressure)
@@ -117,15 +135,18 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
* @num_idle_pages: Length of @idle
* @max_idle_pages: High watermark of @idle before async release to the buddy allocator
* @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @blocks_in_use: Blocks handed out and not yet freed
* @nid: NUMA node id
* @node: Node in the global pisa_allocs list
* @name: Debug name (defaults to caller of pisa_create())
+ * @stats: Per-allocator statistics
*/
struct pisa_alloc {
/* First cacheline: hot fields touched on block alloc and free. */
spinlock_t lock;
u8 order;
u8 prot;
+ struct pisa_pcp __percpu *pcp;
struct list_head partial;
struct list_head partial_dirty;
struct list_head idle;
@@ -133,12 +154,22 @@ struct pisa_alloc {
unsigned int num_idle_pages;
unsigned int max_idle_pages;
unsigned long idle_jiffies;
+ unsigned long blocks_in_use;
/* Cold. */
int nid;
struct list_head node;
char name[48];
+ struct pisa_alloc_stats stats;
} ____cacheline_aligned;
+/* Active non-idle PISA pages held by @pa, for soft compaction checks. */
+static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
+{
+ return (long)(READ_ONCE(pa->stats.page_alloc_cnt) -
+ READ_ONCE(pa->stats.page_free_cnt)) -
+ READ_ONCE(pa->num_idle_pages);
+}
+
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
@@ -202,6 +233,8 @@ struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direct
struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro);
__printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+int pisa_enable_pcp(struct pisa_alloc *pa);
+void pisa_pcp_drain(struct pisa_alloc *pa);
void pisa_release_page(struct pisa_meta *meta);
unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
unsigned long nr_pages, struct page **page_array, bool frozen);
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index a6556aa403a09..cdf176464d74a 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -100,6 +100,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
bool frozen);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
+bool pisa_page_can_recycle(const struct page *page);
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
@@ -139,6 +140,11 @@ static inline bool pisa_free_coherent(struct page *page, size_t size)
return false;
}
+static inline bool pisa_page_can_recycle(const struct page *page)
+{
+ return true;
+}
+
static inline unsigned long
pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill()
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (36 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
` (12 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Transmit socket buffers use memory coming from skb_page_frag_refill().
Sockets whose leaf device has enabled pisa_tx_bufs should use PISA
memory.
Add a dedicated per-task page_frag, current->task_frag_pisa, populated
with read-only PISA blocks via __GFP_PISA_RO, and select it on TX when
the socket's (or cork's) route goes out in the clear through a PISA-enabled
device.
Because walking lower devices can be expensive (e.g. bonded devices),
cache the leaf device check on the net_device and refresh it at most once
every ~250ms. Special paths that hold plaintext before encryption (IPsec,
kTLS) or do not use a DMA device keep using regular pages;
ESP output uses skb_page_frag_refill_dev() to opt in when the
egress device has pisa_tx_bufs enabled.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 24 +++++++---
drivers/iommu/pisa-alloc.c | 57 ++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 3 +-
include/linux/device.h | 9 ++++
include/linux/netdevice.h | 4 ++
include/linux/pisa.h | 20 +++++++++
include/linux/sched.h | 4 ++
include/linux/skbuff.h | 2 +
include/net/sock.h | 89 ++++++++++++++++++++++++++++++++++++++
kernel/exit.c | 2 +
kernel/fork.c | 2 +
net/core/sock.c | 41 ++++++++++++++++++
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 11 +++--
net/ipv4/tcp.c | 3 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 11 +++--
17 files changed, 270 insertions(+), 20 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 97cd2d03ff77c..ade0c94b02ae7 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2909,7 +2909,8 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
static const DEVICE_ATTR_RO(removable);
/* Per-device configuration to selectively enable PISA */
-static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
+static ssize_t pisa_attr_store(struct device *dev, const char *buf, size_t count,
+ bool *field, int (*setter)(struct device *, bool))
{
bool val;
int ret = kstrtobool(buf, &val);
@@ -2921,11 +2922,17 @@ static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
if (ret < 0)
return ret;
}
- WRITE_ONCE(*field, val);
+ if (setter) {
+ ret = setter(dev, val);
+ if (ret < 0)
+ return ret;
+ } else {
+ WRITE_ONCE(*field, val);
+ }
return count;
}
-#define PISA_BOOL_ATTR(name) \
+#define PISA_SET_ATTR(name, setter) \
static ssize_t name##_show(struct device *dev, \
struct device_attribute *attr, char *buf) \
{ \
@@ -2935,13 +2942,15 @@ static ssize_t name##_store(struct device *dev, \
struct device_attribute *attr, \
const char *buf, size_t count) \
{ \
- return pisa_attr_store(buf, count, &dev->name); \
+ return pisa_attr_store(dev, buf, count, &dev->name, setter); \
} \
static DEVICE_ATTR_RW(name)
+#define PISA_BOOL_ATTR(name) PISA_SET_ATTR(name, NULL)
+
PISA_BOOL_ATTR(pisa_coherent);
PISA_BOOL_ATTR(pisa_rx_bufs);
-PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
PISA_BOOL_ATTR(pisa_tx_skb_heads);
static struct attribute *dev_attr_pisa[] = {
@@ -3100,8 +3109,11 @@ static void device_remove_attrs(struct device *dev)
const struct class *class = dev->class;
const struct device_type *type = dev->type;
- if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
device_remove_group(dev, &dev_attr_pisa_group);
+ if (dev->pisa_tx_bufs)
+ net_pisa_tx_bufs_set(dev, false);
+ }
if (dev->physical_location) {
device_remove_group(dev, &dev_attr_physical_location_group);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 913706a503341..7b9d0a7acc101 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -644,6 +644,63 @@ bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int n
}
EXPORT_SYMBOL(pisa_has_free_blocks_rx);
+#ifdef CONFIG_NET
+/* Enabled while at least one device has pisa_tx_bufs set. */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+EXPORT_SYMBOL(net_pisa_tx_bufs_key);
+
+int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+ static DEFINE_MUTEX(lock);
+
+ mutex_lock(&lock);
+ if (dev->pisa_tx_bufs != val) {
+ WRITE_ONCE(dev->pisa_tx_bufs, val);
+ if (val)
+ static_branch_inc(&net_pisa_tx_bufs_key);
+ else
+ static_branch_dec(&net_pisa_tx_bufs_key);
+ }
+ mutex_unlock(&lock);
+ return 0;
+}
+
+/*
+ * The sysfs attribute pisa_tx_bufs lives on the physical DMA parent
+ * (dev->dev.parent), whereas a socket route (dst->dev) may point to an upper
+ * virtual net_device (bond, VLAN, ipvlan, etc.). Walking lower devices via
+ * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
+ * expensive, so dev->pisa_tx_stamp caches the walk result on the net_device
+ * and refreshes it at most once every 256 jiffies:
+ * bit 0 (PISA_TX_STAMP_OK): pisa_tx_bufs enabled on lower device(s)
+ * bits 31..8 (PISA_TX_STAMP_MASK): upper 24 bits of (u32)jiffies
+ */
+static int netdev_pisa_tx_bufs_walk(struct net_device *dev, struct netdev_nested_priv *priv)
+{
+ return dev->dev.parent && READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+}
+
+/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
+bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+{
+ bool ok;
+
+ if (dev->dev.parent) {
+ ok = READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ } else {
+ rcu_read_lock();
+ ok = netdev_walk_all_lower_dev_rcu(dev,
+ netdev_pisa_tx_bufs_walk,
+ NULL) != 0;
+ rcu_read_unlock();
+ }
+ now |= ok ? PISA_TX_STAMP_OK : 0;
+ WRITE_ONCE(dev->pisa_tx_stamp, now);
+ return now & PISA_TX_STAMP_OK;
+}
+EXPORT_SYMBOL(netdev_tx_pisa_refresh);
+#endif /* CONFIG_NET */
+
/*
* Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
* pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5bbb540eb505e..5f6b2ebcab1d9 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -477,7 +477,8 @@ static void test_dev_pisa_attrs(struct kunit *test)
}
kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
- kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+ if (IS_ENABLED(CONFIG_NET))
+ kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
device_unregister(dev);
}
diff --git a/include/linux/device.h b/include/linux/device.h
index b3c11e32116ca..08f511b978ded 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -814,6 +814,15 @@ struct device {
DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
};
+#if defined(CONFIG_NET) && defined(CONFIG_PISA)
+int net_pisa_tx_bufs_set(struct device *dev, bool val);
+#else
+static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+ return -EOPNOTSUPP;
+}
+#endif
+
#define __create_dev_flag_accessors(accessor_name, flag_name) \
static inline bool dev_##accessor_name(const struct device *dev) \
{ \
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e5..41c999223a15e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2161,6 +2161,7 @@ enum netdev_reg_state {
* @neighbours: List heads pointing to this device's neighbours'
* dev_list, one per address-family.
* @hwprov: Tracks which PTP performs hardware packet time stamping.
+ * @pisa_tx_stamp: Cached leaf pisa flags and timestamp.
*
* FIXME: cleanup struct net_device such that network protocol info
* moves out.
@@ -2276,6 +2277,9 @@ struct net_device {
} adj_list;
/* Read-mostly cache-line for fast-path access */
+#ifdef CONFIG_PISA
+ u32 pisa_tx_stamp;
+#endif
xdp_features_t xdp_features;
const struct xdp_metadata_ops *xdp_metadata_ops;
const struct xsk_tx_metadata_ops *xsk_tx_metadata_ops;
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index cdf176464d74a..822e1556d78ef 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -107,6 +107,18 @@ static inline gfp_t pisa_rx_gfp(const struct device *dev)
return dev && READ_ONCE(dev->pisa_rx_bufs) ? __GFP_PISA_RW : 0;
}
+/* current->task_frag_pisa lifetime, see task_page_frag(). */
+static inline void pisa_task_frag_init(struct task_struct *tsk)
+{
+ tsk->task_frag_pisa.page = NULL;
+}
+
+static inline void pisa_task_frag_release(struct task_struct *tsk)
+{
+ if (tsk->task_frag_pisa.page)
+ put_page(tsk->task_frag_pisa.page);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -164,6 +176,14 @@ static inline bool pisa_has_free_blocks_rx(const struct device *dev,
return false;
}
+static inline void pisa_task_frag_init(struct task_struct *tsk)
+{
+}
+
+static inline void pisa_task_frag_release(struct task_struct *tsk)
+{
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e3..1c7ef0d90dc35 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1453,6 +1453,10 @@ struct task_struct {
struct pipe_inode_info *splice_pipe;
struct page_frag task_frag;
+#ifdef CONFIG_PISA
+ /* TX frag that may hold PISA pages, see sk_page_frag_dst(). */
+ struct page_frag task_frag_pisa;
+#endif
#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
struct lazy_mmu_state lazy_mmu_state;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 6787fcf6ac3e4..2fce6eafa1cb3 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3808,6 +3808,8 @@ static inline void skb_frag_page_copy(skb_frag_t *fragto,
}
bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio);
+bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
+ const struct sk_buff *skb);
/**
* __skb_frag_dma_map - maps a paged fragment via the DMA API
diff --git a/include/net/sock.h b/include/net/sock.h
index 173b37b2b8bb0..73906351d11db 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -2670,6 +2670,95 @@ static inline struct page_frag *sk_page_frag(struct sock *sk)
return &sk->sk_frag;
}
+/* 256-jiffy cache in dev->pisa_tx_stamp; see netdev_tx_pisa_refresh(). */
+#define PISA_TX_STAMP_OK BIT(0)
+#define PISA_TX_STAMP_MASK (~0xffU)
+
+#ifdef CONFIG_PISA
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+
+static inline bool net_pisa_tx_bufs_enabled(void)
+{
+ return static_branch_unlikely(&net_pisa_tx_bufs_key);
+}
+
+bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+{
+ u32 stamp, now;
+
+ if (!dev)
+ return false;
+ stamp = READ_ONCE(dev->pisa_tx_stamp);
+ now = (u32)jiffies & PISA_TX_STAMP_MASK;
+ if (likely((stamp & PISA_TX_STAMP_MASK) == now))
+ return stamp & PISA_TX_STAMP_OK;
+ return netdev_tx_pisa_refresh((struct net_device *)dev, now);
+}
+#else
+static inline bool net_pisa_tx_bufs_enabled(void)
+{
+ return false;
+}
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+{
+ return false;
+}
+#endif
+
+/*
+ * May TX payload routed through @dst be copied into PISA pages? Those pages
+ * stay readable by every device that mapped them for as long as they are
+ * active, so only data that leaves the host in clear qualifies: IPsec payload
+ * is still plaintext here.
+ */
+static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+{
+ bool ok = false;
+
+ if (!net_pisa_tx_bufs_enabled() || !dst)
+ return false;
+
+ rcu_read_lock();
+ if (!dst_xfrm(dst))
+ ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst));
+ rcu_read_unlock();
+ return ok;
+}
+
+/* Is @pfrag the task's frag reserved for PISA-eligible data? */
+static inline bool is_pisa_pfrag(const struct page_frag *pfrag)
+{
+#ifdef CONFIG_PISA
+ return pfrag == ¤t->task_frag_pisa;
+#else
+ return false;
+#endif
+}
+
+/* current->task_frag, or its PISA twin if @dst qualifies. */
+static inline struct page_frag *task_page_frag(const struct dst_entry *dst)
+{
+#ifdef CONFIG_PISA
+ if (dst_tx_pisa_ok(dst))
+ return ¤t->task_frag_pisa;
+#endif
+ return ¤t->task_frag;
+}
+
+/*
+ * sk_page_frag() for data routed through @dst. PISA pages only ever enter
+ * current->task_frag_pisa, so data of sockets that do not qualify (or that use
+ * sk_page_frag() directly: kTLS, sockmap, KCM, MPTCP) never lands in them.
+ */
+static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
+ const struct dst_entry *dst)
+{
+ return sk->sk_use_task_frag ? task_page_frag(dst) : &sk->sk_frag;
+}
+
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag);
static inline bool __sock_writeable(const struct sock *sk, int wmem_alloc)
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804a..ac7ca1fb180c3 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -6,6 +6,7 @@
*/
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/slab.h>
#include <linux/sched/autogroup.h>
#include <linux/sched/mm.h>
@@ -1009,6 +1010,7 @@ void __noreturn do_exit(long code)
if (tsk->task_frag.page)
put_page(tsk->task_frag.page);
+ pisa_task_frag_release(tsk);
exit_task_stack_account(tsk);
diff --git a/kernel/fork.c b/kernel/fork.c
index 5efceb8d58a78..d235d7ab04687 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -44,6 +44,7 @@
#include <linux/mmu_notifier.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/mm_inline.h>
#include <linux/memblock.h>
#include <linux/nsproxy.h>
@@ -982,6 +983,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
#endif
tsk->splice_pipe = NULL;
tsk->task_frag.page = NULL;
+ pisa_task_frag_init(tsk);
tsk->wake_q.next = NULL;
tsk->worker_private = NULL;
diff --git a/net/core/sock.c b/net/core/sock.c
index e2401d735c5a7..695fefe316925 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -115,6 +115,7 @@
#include <linux/memcontrol.h>
#include <linux/prefetch.h>
#include <linux/compat.h>
+#include <linux/pisa.h>
#include <linux/mroute.h>
#include <linux/mroute6.h>
#include <linux/icmpv6.h>
@@ -3257,13 +3258,20 @@ DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key);
*/
bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
{
+ /* PISA pages only go into current->task_frag_pisa, see sk_page_frag_dst(). */
+ if (is_pisa_pfrag(pfrag))
+ gfp |= __GFP_PISA_RO;
+
if (pfrag->page) {
+ if ((gfp & __GFP_PISA_RO) && !pisa_page_can_recycle(pfrag->page))
+ goto drop_page;
if (page_ref_count(pfrag->page) == 1) {
pfrag->offset = 0;
return true;
}
if (pfrag->offset + sz <= pfrag->size)
return true;
+drop_page:
put_page(pfrag->page);
}
@@ -3289,6 +3297,39 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
}
EXPORT_SYMBOL(skb_page_frag_refill);
+/**
+ * skb_page_frag_refill_dev - GFP_ATOMIC skb_page_frag_refill() for @skb TX
+ * @sz: minimum size of the fragment we want to get
+ * @pfrag: pointer to page_frag, private to the caller
+ * @skb: packet being transmitted
+ *
+ * Pages come from the per-node DMA_TO_DEVICE PISA allocators when they are
+ * useful for @skb's egress device (__netdev_tx_pisa_ok()), otherwise from the
+ * buddy allocator. PISA pages are readable by every device that maps them
+ * while they stay active, so @pfrag must only ever hold data meant for devices,
+ * e.g. ciphertext; it may mix PISA and ordinary pages.
+ */
+bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
+ const struct sk_buff *skb)
+{
+ gfp_t gfp = GFP_ATOMIC;
+
+ /*
+ * Before ip_finish_output2() populates skb->dev (e.g. esp_output),
+ * the egress device is carried by skb_dst(skb).
+ */
+ if (net_pisa_tx_bufs_enabled()) {
+ const struct dst_entry *dst = skb_dst(skb);
+
+ rcu_read_lock();
+ if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL)))
+ gfp |= __GFP_PISA_RO;
+ rcu_read_unlock();
+ }
+ return skb_page_frag_refill(sz, pfrag, gfp);
+}
+EXPORT_SYMBOL(skb_page_frag_refill_dev);
+
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
{
if (likely(skb_page_frag_refill(32U, pfrag, sk->sk_allocation)))
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index ae701561d9f12..feecf094f582c 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -451,7 +451,7 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto cow;
}
@@ -560,7 +560,7 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto error_free;
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 3396f177a18d0..a5ea530bc8b5b 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -986,7 +986,10 @@ static int __ip_append_data(struct sock *sk,
exthdrlen = !skb ? rt->dst.header_len : 0;
mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize;
- paged = !!cork->gso_size;
+ paged = !!cork->gso_size ||
+ (sk->sk_type != SOCK_RAW &&
+ (rt->dst.dev->features & NETIF_F_SG) &&
+ is_pisa_pfrag(pfrag));
hh_len = LL_RESERVED_SPACE(rt->dst.dev);
@@ -1383,8 +1386,8 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4,
}
return __ip_append_data(sk, fl4, &sk->sk_write_queue, &inet->cork.base,
- sk_page_frag(sk), getfrag,
- from, length, transhdrlen, flags);
+ sk_page_frag_dst(sk, inet->cork.base.dst),
+ getfrag, from, length, transhdrlen, flags);
}
static void ip_cork_release(struct inet_cork *cork)
@@ -1592,7 +1595,7 @@ struct sk_buff *ip_make_skb(struct sock *sk,
return ERR_PTR(err);
err = __ip_append_data(sk, fl4, &queue, cork,
- ¤t->task_frag, getfrag,
+ task_page_frag(cork->dst), getfrag,
from, length, transhdrlen, flags);
if (err) {
__ip_flush_pending_frames(sk, &queue, cork);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index b9de7c302ba83..b8d44a0638914 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1290,9 +1290,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
copy = msg_data_left(msg);
if (zc == 0) {
+ const struct dst_entry *dst = __sk_dst_get(sk);
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
- struct page_frag *pfrag = sk_page_frag(sk);
+ struct page_frag *pfrag = sk_page_frag_dst(sk, dst);
if (!sk_page_frag_refill(sk, pfrag))
goto wait_for_space;
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index d654f0f442388..cdedda548749a 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -481,7 +481,7 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto cow;
}
@@ -592,7 +592,7 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto error_free;
}
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 25fe0ba98b1a6..957c38a08680c 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1486,7 +1486,10 @@ static int __ip6_append_data(struct sock *sk,
dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len;
}
- paged = !!cork->gso_size;
+ paged = !!cork->gso_size ||
+ (sk->sk_type != SOCK_RAW &&
+ (rt->dst.dev->features & NETIF_F_SG) &&
+ is_pisa_pfrag(pfrag));
mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize;
orig_mtu = mtu;
@@ -1889,8 +1892,8 @@ int ip6_append_data(struct sock *sk,
}
return __ip6_append_data(sk, &sk->sk_write_queue, &inet->cork,
- sk_page_frag(sk), getfrag,
- from, length, transhdrlen, flags);
+ sk_page_frag_dst(sk, inet->cork.base.dst),
+ getfrag, from, length, transhdrlen, flags);
}
EXPORT_SYMBOL_GPL(ip6_append_data);
@@ -2092,7 +2095,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk,
}
err = __ip6_append_data(sk, &queue, cork,
- ¤t->task_frag, getfrag, from,
+ task_page_frag(cork->base.dst), getfrag, from,
length + exthdrlen, transhdrlen + exthdrlen,
flags);
if (err) {
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (37 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
` (11 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When a page_pool performs DMA mapping (PP_FLAG_DMA_MAP) on a device with
pisa_rx_bufs enabled, pass __GFP_PISA_RW to use PISA memory.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
net/core/page_pool.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index d7c88c0b67e53..91cae5ba78373 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
@@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(struct page_pool *pool
*/
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
/* Don't support bulk alloc for high-order pages */
if (unlikely(pp_order))
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (38 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
` (10 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
In Confidential Computing (CoCo) guests (AMD SEV-SNP, Intel TDX, ARM CCA),
guest memory is private/encrypted by default, and DMA buffers must be
converted to shared/decrypted in the host page tables before a device
can access them.
Doing that hypercall per-packet on 4KB pages is prohibitively expensive,
so guests normally bounce DMA through swiotlb. With PISA, 2MB pages are
decrypted once upfront when allocated and re-encrypted only when retired
back to the buddy allocator, avoiding both per-packet hypercalls and
bounce-buffer copies.
Add pisa.decrypt (auto-enabled when CC_ATTR_GUEST_MEM_ENCRYPT is active)
and the PISA_DECRYPTED flag to manage 2MB page decryption/re-encryption
and bypass swiotlb bouncing for decrypted PISA pages.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 117 ++++++++++++++++++++++++++++++++-----
drivers/iommu/pisa-kunit.c | 11 ++++
drivers/iommu/pisa-map.c | 2 +-
drivers/iommu/pisa-meta.c | 13 +++++
drivers/iommu/pisa-node.c | 86 ++++++++++++++++-----------
drivers/iommu/pisa-priv.h | 16 ++++-
include/linux/pisa.h | 16 +++++
kernel/dma/direct.c | 5 +-
8 files changed, 212 insertions(+), 54 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 7b9d0a7acc101..b4a25c65bf97e 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/cc_platform.h>
#include <linux/cpuhotplug.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
@@ -29,6 +30,7 @@
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
+#include <linux/set_memory.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <linux/slab.h>
@@ -49,6 +51,10 @@ static LLIST_HEAD(pisa_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *pisa_wq __ro_after_init;
+/* Mark active PISA pages decrypted for DMA (auto-enabled on CoCo guests). */
+static bool pisa_decrypt __read_mostly;
+module_param_named(decrypt, pisa_decrypt, bool, 0444);
+
/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */
atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp;
unsigned long pisa_max_pages __read_mostly;
@@ -80,6 +86,51 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
static void pisa_schedule_reclaim(void);
+/* Return true if a new PISA page should be decrypted. */
+bool pisa_want_decrypt(void)
+{
+ if (!pisa_decrypt)
+ return false;
+
+ /* Explicit page decryption is for CoCo guests, not bare-metal SME hosts. */
+ return !WARN_ON_ONCE(cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT));
+}
+
+/* Decrypt and zero a new PISA page. Must be called from sleepable context. */
+int pisa_page_prepare(struct page *page, struct pisa_meta *meta)
+{
+ void *vaddr = page_address(page);
+ int ret;
+
+ ret = set_memory_decrypted((unsigned long)vaddr, 1 << PISA_ORDER);
+ if (ret) {
+ /* Leak partially shared page rather than returning it to the buddy allocator. */
+ pr_warn_ratelimited("pisa: leaking PISA page that could not be decrypted\n");
+ return ret;
+ }
+ /* Initialize cleartext cachelines after flipping encryption state. */
+ memset(vaddr, 0, PISA_SIZE);
+ meta->flags |= PISA_DECRYPTED;
+ return 0;
+}
+
+/* Undo pisa_page_prepare(). */
+int pisa_page_unprepare(struct pisa_meta *meta)
+{
+ struct page *page;
+
+ if (!(meta->flags & PISA_DECRYPTED))
+ return 0;
+ page = pfn_to_page(pisa_meta_to_pfn(meta));
+ if (set_memory_encrypted((unsigned long)page_address(page), 1 << PISA_ORDER)) {
+ pr_warn_ratelimited("pisa: leaking PISA page that could not be re-encrypted\n");
+ return -EIO;
+ }
+ WRITE_ONCE(meta->flags, meta->flags & ~PISA_DECRYPTED);
+ return 0;
+}
+
+
/*
* Idle PISA page release proceeds in three stages:
* 1. Detach onto lockless pisa_free_list and schedule pisa_reclaim_work.
@@ -103,12 +154,13 @@ static void pisa_release_page_rcu(struct rcu_head *head)
spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
- /* Possibly still mapped by a device: leak it, still charged. */
- if (meta->flags & PISA_QUARANTINED)
- return;
-
/* The page has been unmapped from every domain by pisa_release_page() */
- pisa_slot_free(slot);
+ if (!(meta->flags & PISA_QUARANTINED))
+ pisa_slot_free(slot);
+
+ /* Still shared with the host, or possibly still mapped by a device. */
+ if (meta->flags & (PISA_DECRYPTED | PISA_QUARANTINED))
+ return;
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -120,7 +172,12 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
void pisa_release_page(struct pisa_meta *meta)
{
+ u8 flags;
+
if (!meta->pa) {
+ /* Unbound (reservoir) page: leaked if re-encryption fails. */
+ if (pisa_page_unprepare(meta))
+ return;
__free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
atomic_long_dec(&pisa_nr_pages);
return;
@@ -128,9 +185,15 @@ void pisa_release_page(struct pisa_meta *meta)
/* Quarantined on unmap failure and leaked in pisa_release_page_rcu(). */
pisa_unmap_all(meta);
+ flags = meta->flags & ~PISA_ACTIVE;
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
- WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
+ WRITE_ONCE(meta->flags, flags);
+ if ((flags & (PISA_DECRYPTED | PISA_QUARANTINED)) == PISA_DECRYPTED) {
+ synchronize_rcu();
+ pisa_page_unprepare(meta);
+ }
+
call_rcu(&meta->rcu, pisa_release_page_rcu);
}
@@ -741,15 +804,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
unsigned int bpp = PISA_BLOCKS(order);
struct pisa_meta *meta;
struct page *page;
+ int ret;
/* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
- meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+ meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, pisa_want_decrypt(), &clean);
if (IS_ERR(meta))
return meta;
if (unlikely(pisa_slot_alloc(meta))) {
- pisa_release_page(meta);
- return ERR_PTR(-ENOSPC);
+ ret = -ENOSPC;
+ goto err_free;
}
page = pfn_to_page(pisa_meta_to_pfn(meta));
@@ -757,9 +821,9 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
if (split_page_compound(page, PISA_ORDER, order)) {
pisa_slot_free(meta->slot);
- pisa_release_page(meta);
pisa_alloc_backoff(nid, can_block);
- return ERR_PTR(-EBUSY);
+ ret = -EBUSY;
+ goto err_free;
}
INIT_LIST_HEAD(&meta->list);
@@ -773,6 +837,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
}
return meta;
+
+err_free:
+ if (!can_block && (meta->flags & PISA_DECRYPTED)) {
+ /* set_memory_encrypted() may sleep; defer to reclaim worker. */
+ llist_add(&meta->llnode, &pisa_free_list);
+ pisa_schedule_reclaim();
+ } else {
+ pisa_release_page(meta);
+ }
+ return ERR_PTR(ret);
}
/* Reset fields normally cleared by prep_new_page()/post_alloc_hook(). */
@@ -1135,7 +1209,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
if (!win || pisa_window_declined(win) || !(domain->pgsize_bitmap & PISA_SIZE))
return NULL;
} else if (is_swiotlb_for_alloc(dev) ||
- force_dma_unencrypted(dev) ||
+ (force_dma_unencrypted(dev) && !pisa_want_decrypt()) ||
!dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
return NULL;
}
@@ -1294,10 +1368,15 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
read_unlock(&pisa_allocs_lock);
}
- /* Retiring must not run under @pa->lock. */
+ /* Unmap outside locks; defer decrypted pages to workqueue outside fs_reclaim. */
list_for_each_entry_safe(meta, tmp, &release_list, list) {
list_del_init(&meta->list);
- pisa_release_page(meta);
+ if (meta->flags & PISA_DECRYPTED) {
+ llist_add(&meta->llnode, &pisa_free_list);
+ pisa_schedule_reclaim();
+ } else {
+ pisa_release_page(meta);
+ }
}
/* Charge do_shrink_slab() for the full PISA pages reclaimed. */
@@ -1342,6 +1421,16 @@ static int __init pisa_init(void)
pisa_slots_init();
+ /*
+ * CoCo guests require decrypted pages and cannot decrypt in atomic
+ * context, so ensure the per-node reservoir is enabled to serve them.
+ */
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
+ pisa_decrypt = true;
+ if (!pisa_reservoir_max)
+ pisa_reservoir_max = 16;
+ }
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5f6b2ebcab1d9..b63cd6585b673 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -63,6 +63,7 @@ static void test_meta_active_toggle(struct kunit *test)
unsigned long pfn, base_pfn;
struct pisa_meta *meta;
struct page *page;
+ phys_addr_t pa;
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -80,16 +81,26 @@ static void test_meta_active_toggle(struct kunit *test)
}
KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
meta = pisa_meta_of_pfn(pfn);
+ pa = page_to_phys(page);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
pisa_meta_set_active(meta, true);
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
+ KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), pa);
+
+ WRITE_ONCE(meta->flags, PISA_ACTIVE | PISA_DECRYPTED);
+ KUNIT_EXPECT_TRUE(test, is_pisa_direct(pa, PAGE_SIZE));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PISA_SIZE + PAGE_SIZE));
+ KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), dma_addr_unencrypted(pa));
pisa_meta_set_active(meta, false);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
__free_pages(page, PISA_ORDER);
}
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2c481e514575c..fc58c639f3288 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -470,7 +470,7 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
phys_addr_t pisa_phys = phys & PISA_MASK;
ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
- pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+ pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
if (likely(!ret)) {
/* Ensure leaf PTE is visible before setting the domain bit. */
smp_mb__before_atomic();
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
index 00939c366ba6d..3acdec815935f 100644
--- a/drivers/iommu/pisa-meta.c
+++ b/drivers/iommu/pisa-meta.c
@@ -78,6 +78,19 @@ struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
return &pisa_meta_array[frame];
}
+bool __is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ u8 flags = READ_ONCE(pisa_meta_of_pfn(PHYS_PFN(phys))->flags);
+
+ /*
+ * Only decrypted active pages can bypass swiotlb bouncing, and the
+ * transfer must not straddle the end of the PISA frame.
+ */
+ return (flags & (PISA_ACTIVE | PISA_DECRYPTED)) ==
+ (PISA_ACTIVE | PISA_DECRYPTED) &&
+ (phys & (PISA_SIZE - 1)) + size <= PISA_SIZE;
+}
+
/* Base PFN of @meta's PISA frame, or ULONG_MAX if out-of-range/sink. */
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
{
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 0f23b94a7a6f8..dfde9c021fe05 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -62,7 +62,7 @@ static unsigned int pisa_memalloc_save(gfp_t gfp)
/* Allocate a PISA page from the buddy allocator and initialize its metadata. */
static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
- bool *clean)
+ bool decrypt, bool *clean)
{
struct pisa_meta *meta;
unsigned int memflags;
@@ -83,6 +83,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
* must not dip into emergency reserves or trigger OOM/warnings.
*/
gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+ if (decrypt)
+ gfp &= ~__GFP_ZERO;
gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
if (can_block)
@@ -101,14 +103,21 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
__free_pages(page, PISA_ORDER);
return ERR_PTR(-ERANGE);
}
- if (can_block)
- memalloc_flags_restore(memflags);
meta = pisa_meta_of_pfn(page_to_pfn(page));
memset(meta, 0, sizeof(*meta));
atomic_long_inc(&pisa_nr_pages);
- *clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+ /* On decrypt failure the page is leaked and stays charged to budget. */
+ if (decrypt && pisa_page_prepare(page, meta)) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ return ERR_PTR(-EIO);
+ }
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ *clean = (gfp & __GFP_ZERO) || (meta->flags & PISA_DECRYPTED) ||
+ want_init_on_alloc(0) || want_init_on_free();
return meta;
}
@@ -124,11 +133,12 @@ static void pisa_reservoir_refill_node(int nid)
unsigned long flags;
bool clean;
- meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+ meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
+ pisa_want_decrypt(), &clean);
if (IS_ERR(meta))
break;
- /* init_on_alloc/init_on_free pages are already zeroed. */
+ /* Decrypted and init_on_alloc/init_on_free pages are already zeroed. */
if (!clean)
memset(page_address(pfn_to_page(pisa_meta_to_pfn(meta))), 0, PISA_SIZE);
@@ -145,47 +155,48 @@ static void pisa_reservoir_refill_node(int nid)
static void pisa_reservoir_trim_node(int nid, unsigned int max)
{
struct pisa_node *res = &pisa_nodes[nid];
- struct pisa_meta *meta, *tmp;
- LIST_HEAD(free_list);
- unsigned long flags;
- spin_lock_irqsave(&res->lock, flags);
- while (res->nr_pages > max && !list_empty(&res->pages)) {
- list_move(res->pages.prev, &free_list);
- res->nr_pages--;
- }
- spin_unlock_irqrestore(&res->lock, flags);
+ while (res->nr_pages > max && res->nr_pages > READ_ONCE(pisa_reservoir_max)) {
+ struct pisa_meta *meta;
+ unsigned long flags;
+
+ spin_lock_irqsave(&res->lock, flags);
+ meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
+ if (meta) {
+ list_del_init(&meta->list);
+ res->nr_pages--;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+ if (!meta)
+ break;
- list_for_each_entry_safe(meta, tmp, &free_list, list) {
- list_del_init(&meta->list);
pisa_release_page(meta);
cond_resched();
}
}
+/* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
static void pisa_node_work_fn(struct work_struct *work)
{
- struct pisa_node *res = container_of(work, struct pisa_node, work);
- int nid = res - pisa_nodes;
- unsigned int max = READ_ONCE(pisa_reservoir_max);
-
- if (unlikely(pisa_meta_init()))
- return;
+ struct pisa_node *node = container_of(work, struct pisa_node, work);
+ int nid = node - pisa_nodes;
- if (READ_ONCE(res->nr_pages) < max)
+ pisa_reservoir_trim_node(nid, READ_ONCE(pisa_reservoir_max));
+ if (READ_ONCE(pisa_reservoir_max) && static_branch_unlikely(&pisa_key))
pisa_reservoir_refill_node(nid);
- else if (READ_ONCE(res->nr_pages) > max)
- pisa_reservoir_trim_node(nid, max);
}
-/* Queue @nid's reservoir worker on a CPU belonging to @nid. */
+/* Queue node @nid's work if its reservoir is below or above the target. */
static void pisa_schedule_reservoir_node(int nid)
{
- if (nid >= 0 && nid < MAX_NUMNODES && node_state(nid, N_MEMORY))
- queue_work_node(nid, system_dfl_wq, &pisa_nodes[nid].work);
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ if (node_state(nid, N_MEMORY) &&
+ READ_ONCE(res->nr_pages) != READ_ONCE(pisa_reservoir_max) &&
+ !work_pending(&res->work))
+ queue_work_node(nid, system_dfl_wq, &res->work);
}
-/* Queue reservoir refill/trim workers on all memory nodes. */
void pisa_schedule_reservoir(void)
{
int nid;
@@ -195,7 +206,7 @@ void pisa_schedule_reservoir(void)
}
/* Pop one pre-prepared PISA page from node @nid's reservoir under @res->lock. */
-static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool decrypt)
{
struct pisa_node *res = &pisa_nodes[nid];
struct pisa_meta *meta;
@@ -207,10 +218,12 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
spin_lock_irqsave(&res->lock, flags);
meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
- if (meta) {
+ if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
list_del_init(&meta->list);
res->nr_pages--;
refill = res->nr_pages <= max / 2;
+ } else {
+ meta = NULL;
}
spin_unlock_irqrestore(&res->lock, flags);
@@ -224,7 +237,7 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
* Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
* refill when <= half full or empty), or allocate one inline on miss.
*/
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean)
{
bool can_block = gfpflags_allow_blocking(gfp);
unsigned int max = READ_ONCE(pisa_reservoir_max);
@@ -232,7 +245,7 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
nid = pisa_sanitize_nid(nid);
if (max) {
- meta = pisa_reservoir_pop_node(nid, max);
+ meta = pisa_reservoir_pop_node(nid, max, decrypt);
if (meta) {
*clean = true;
return meta;
@@ -244,8 +257,11 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
/* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
if (!can_block && time_before(jiffies, READ_ONCE(pisa_nodes[nid].next_alloc)))
return ERR_PTR(-EAGAIN);
+ /* Page decryption may sleep; atomic callers must rely on the reservoir. */
+ if (decrypt && !can_block)
+ return ERR_PTR(-EAGAIN);
- meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+ meta = __pisa_page_alloc(gfp, nid, can_block, decrypt, clean);
if (IS_ERR(meta))
pisa_alloc_backoff(nid, can_block);
return meta;
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 8f87bdf7d4ee2..f08388f003fad 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -7,6 +7,7 @@
#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/llist.h>
+#include <linux/mem_encrypt.h>
#include <linux/nodemask.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
@@ -62,7 +63,7 @@ enum {
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
- * @flags: PISA_ACTIVE / PISA_QUARANTINED
+ * @flags: PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED
* @prot: Owning allocator's IOMMU_READ/WRITE protection bits
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
@@ -101,6 +102,14 @@ struct pisa_meta {
static_assert(offsetof(struct pisa_meta, flags) == 0);
static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+/* Physical address for IOMMU PTE of @meta at @phys (shared alias if decrypted). */
+static inline phys_addr_t pisa_pte_phys(const struct pisa_meta *meta, phys_addr_t phys)
+{
+ if (meta->flags & PISA_DECRYPTED)
+ return dma_addr_unencrypted(phys);
+ return phys;
+}
+
static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
{
return meta->nr_clean + meta->nr_dirty;
@@ -227,6 +236,9 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
if (!can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
+bool pisa_want_decrypt(void);
+int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
+int pisa_page_unprepare(struct pisa_meta *meta);
/* pisa-alloc.c, also used by the per-node code in pisa-node.c */
struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
@@ -256,7 +268,7 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 822e1556d78ef..89eb0722d90bc 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -33,6 +33,7 @@ struct iommu_domain;
enum {
PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
PISA_QUARANTINED = BIT(1), /* IOMMU unmap failed: never reuse */
+ PISA_DECRYPTED = BIT(2), /* page was set_memory_decrypted() */
};
/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
@@ -45,6 +46,8 @@ extern unsigned long *pisa_chunk_bitmap;
int pisa_meta_init(void);
+bool __is_pisa_direct(phys_addr_t phys, size_t size);
+
/* fast membership check for pisa pfn/pages */
static inline bool is_pisa_pfn(unsigned long pfn)
{
@@ -61,6 +64,14 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return READ_ONCE(*flags) & PISA_ACTIVE;
}
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+
+ return __is_pisa_direct(phys, size);
+}
+
static inline bool is_pisa_page(const struct page *page);
bool __pisa_free_page(struct page *page);
@@ -133,6 +144,11 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return false;
}
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ return false;
+}
+
static inline bool pisa_free_page(struct page *page)
{
return false;
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 11052b65e573b..a7fa1efc0ba09 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -707,8 +707,9 @@ dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
if (is_swiotlb_force_bounce(dev)) {
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT))
return DMA_MAPPING_ERROR;
-
- return swiotlb_map(dev, phys, size, dir, attrs);
+ if (!is_pisa_direct(phys, size))
+ return swiotlb_map(dev, phys, size, dir, attrs);
+ attrs |= DMA_ATTR_CC_SHARED;
}
if (attrs & DMA_ATTR_CC_SHARED)
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (39 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
` (9 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Blocks returned to a pisa_alloc are marked dirty and need to be zeroed
before they can serve __GFP_ZERO requests. Zeroing blocks inline at
allocation time adds CPU cost and latency.
Add an optional background scrubber (enabled via pisa.scrub_max) that
runs on the allocator's home NUMA node work item, zeroes dirty blocks
outside pa->lock in batches, and moves them to meta->clean_bitmap[].
Scrubbing is demand-driven by the number of dirty blocks that __GFP_ZERO
allocations had to zero inline.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 188 ++++++++++++++++++++++++++++++++++++-
drivers/iommu/pisa-kunit.c | 49 +++++++++-
drivers/iommu/pisa-node.c | 31 ++++++
drivers/iommu/pisa-priv.h | 11 ++-
4 files changed, 272 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index b4a25c65bf97e..4b61ec84ed6d1 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -29,6 +29,7 @@
#include <linux/percpu.h>
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
+#include <linux/sched/clock.h>
#include <linux/sched/mm.h>
#include <linux/set_memory.h>
#include <linux/shrinker.h>
@@ -72,6 +73,10 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
+/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
+unsigned int pisa_scrub_max __read_mostly;
+#define PISA_SCRUB_MAX_LIMIT 65536U
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -86,6 +91,24 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
static void pisa_schedule_reclaim(void);
+static int pisa_scrub_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (ret)
+ return ret;
+ WRITE_ONCE(pisa_scrub_max, min(pisa_scrub_max, PISA_SCRUB_MAX_LIMIT));
+ /* Serve demand accumulated while the scrubber was off (once initialized). */
+ if (pisa_scrub_max && static_branch_unlikely(&pisa_key))
+ pisa_schedule_scrub_all();
+ return 0;
+}
+
+static const struct kernel_param_ops pisa_scrub_ops = {
+ .set = pisa_scrub_set,
+ .get = param_get_uint,
+};
+module_param_cb(scrub_max, &pisa_scrub_ops, &pisa_scrub_max, 0644);
/* Return true if a new PISA page should be decrypted. */
bool pisa_want_decrypt(void)
{
@@ -263,6 +286,7 @@ static void pisa_schedule_reclaim(void)
else
queue_delayed_work(wq, &pisa_reclaim_work, msecs_to_jiffies(idle_ms));
}
+
/*
* Update @meta's position on @pa's lists under @pa->lock. When all blocks
* become available (avail == bpp), either queues @meta to @pisa_free_list (and
@@ -353,6 +377,148 @@ static bool pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
return false;
}
+/*
+ * Background page scrubber: pre-zeroes dirty blocks in @pa->nid's work item
+ * to match inline-zeroed demand (@pa->scrub_demand) when scrub_max > 0.
+ */
+static struct pisa_meta *pisa_pick_dirty_meta(struct pisa_alloc *pa)
+{
+ struct pisa_meta *meta;
+
+ meta = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+ if (meta)
+ return meta;
+
+ list_for_each_entry(meta, &pa->partial, list) {
+ if (meta->nr_dirty)
+ return meta;
+ }
+
+ if (!list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ if (meta->nr_dirty) {
+ pa->num_idle_pages--;
+ return meta;
+ }
+ }
+
+ return NULL;
+}
+
+/* Dirty blocks per allocator, inline-zeroed since the last run, that queue a run. */
+#define PISA_SCRUB_TRIGGER 32U
+/* Cap on @pa->scrub_demand, so a long-off scrubber does not owe too much. */
+#define PISA_SCRUB_DEMAND_MAX 4096U
+/* Bytes zeroed per batch: a few microseconds of memset(). */
+/* XXX assumes PAGE_SIZE <= SZ_128K (32 * 4KB pages). */
+#define PISA_SCRUB_BYTES SZ_128K
+#define PISA_SCRUB_BATCH (PISA_SCRUB_BYTES >> PAGE_SHIFT)
+/* Time after which a run stops and requeues. */
+#define PISA_SCRUB_RUN_NS (50 * NSEC_PER_USEC)
+
+/*
+ * Scrub one batch of dirty blocks in @pa outside @pa->lock.
+ * Returns the number of PAGE_SIZE pages taken off the dirty bitmap.
+ */
+static unsigned long pisa_scrub_alloc(struct pisa_alloc *pa)
+{
+ unsigned int bpp, order, idx, count = 0, used = 0, want;
+ u16 idxs[PISA_SCRUB_BATCH];
+ unsigned long base_pfn, flags;
+ bool should_release;
+ struct pisa_meta *meta;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ if (!pa->scrub_demand) {
+ spin_unlock_irqrestore(&pa->lock, flags);
+ return 0;
+ }
+
+ meta = pisa_pick_dirty_meta(pa);
+ if (!meta) {
+ pa->scrub_demand = 0;
+ spin_unlock_irqrestore(&pa->lock, flags);
+ return 0;
+ }
+
+ order = pa->order;
+ bpp = PISA_BLOCKS(order);
+ base_pfn = pisa_meta_to_pfn(meta);
+ want = min(max(PISA_SCRUB_BATCH >> order, 1U), pa->scrub_demand);
+
+ for_each_set_bit(idx, meta->dirty_bitmap, bpp) {
+ struct page *block = pfn_to_page(base_pfn + (idx << order));
+
+ __clear_bit(idx, meta->dirty_bitmap);
+ used++;
+ if (!pisa_block_poisoned(pa, block))
+ idxs[count++] = idx;
+ if (used == meta->nr_dirty || count == want)
+ break;
+ }
+ meta->nr_dirty -= used;
+ pa->blocks_in_use += used - count;
+ pisa_place_meta(meta);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (unlikely(!count))
+ return (unsigned long)used << order;
+
+ for (idx = 0; idx < count; idx++) {
+ struct page *block = pfn_to_page(base_pfn + (idxs[idx] << order));
+
+ memset(page_address(block), 0, PAGE_SIZE << order);
+ }
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (idx = 0; idx < count; idx++)
+ __set_bit(idxs[idx], meta->clean_bitmap);
+ meta->nr_clean += count;
+ pa->scrub_demand -= min(count, pa->scrub_demand);
+ should_release = pisa_place_meta(meta);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return (unsigned long)used << order;
+}
+
+/*
+ * Scrub dirty blocks for allocators homed on @nid up to @pisa_scrub_max pages or
+ * PISA_SCRUB_RUN_NS. Returns true if stopped early with demand remaining.
+ */
+bool pisa_scrub_node(int nid)
+{
+ unsigned long budget = READ_ONCE(pisa_scrub_max), done = 0, n;
+ u64 deadline = local_clock() + PISA_SCRUB_RUN_NS;
+ struct pisa_alloc *pa;
+ bool more = false;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ if (pa->nid != nid || !READ_ONCE(pa->scrub_demand))
+ continue;
+ read_unlock(&pisa_allocs_lock);
+
+ while ((n = pisa_scrub_alloc(pa))) {
+ done += n;
+ if (done >= budget || local_clock() >= deadline) {
+ more = true;
+ break;
+ }
+ cond_resched();
+ }
+
+ read_lock(&pisa_allocs_lock);
+ if (more)
+ break;
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ return more;
+}
+
/**
* pisa_create - Create a PISA page allocator
* @nid: Home NUMA node for this allocator
@@ -952,6 +1118,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
for (i = got; i < got + dirty; i++)
set_page_private(out[i], 1);
got += dirty;
+ pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
} else {
/* First choice are dirty blocks, followed by clean */
got += pisa_scan_bitmap(meta, true, want, out, !frozen);
@@ -973,6 +1140,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
struct page **pages, bool zero, bool frozen)
{
+ bool should_scrub = false;
unsigned long n, flags, i;
struct pisa_meta *meta;
unsigned int want;
@@ -986,6 +1154,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
continue;
/* Fallback to a new allocation */
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
/* With @zero, fresh dirty blocks are zeroed below. */
meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
@@ -998,6 +1167,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
}
+ /* Queue the scrubber when the inline zeroing crosses the trigger. */
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
out:
@@ -1012,6 +1183,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
else
page_ref_unfreeze(pages[i], 1);
}
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
return n;
}
@@ -1112,7 +1285,7 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
{
DECLARE_BITMAP(busy, PISA_MAX_BLOCKS);
struct pisa_alloc *pa = meta->pa;
- unsigned int bpp = PISA_BLOCKS(0), i;
+ unsigned int bpp = PISA_BLOCKS(0), i, dirty = 0;
bool poisoned = false;
unsigned long off;
struct page *page;
@@ -1166,11 +1339,13 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
} else {
__clear_bit(off + i, meta->dirty_bitmap);
meta->nr_dirty--;
+ dirty++;
}
pisa_block_prep(page + i);
page_ref_unfreeze(page + i, 1);
}
pa->blocks_in_use += nblocks;
+ pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
pisa_place_meta(meta);
return page;
}
@@ -1185,7 +1360,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
struct iommu_domain *domain, size_t size,
gfp_t gfp, unsigned long attrs)
{
- bool should_release = false;
+ bool should_scrub = false, should_release = false;
struct list_head *lists[3];
unsigned long align_mask, flags;
unsigned int nblocks, i;
@@ -1234,10 +1409,14 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
}
}
if (!page) {
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
meta = pisa_alloc_split_page(0, gfp, pa->nid);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
return NULL;
+ }
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
@@ -1247,10 +1426,13 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
if (unlikely(!page))
should_release = pisa_place_meta(meta);
}
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
if (should_release)
pisa_schedule_reclaim();
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
if (!page)
return NULL;
*pagep = page;
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index b63cd6585b673..40bf7801183da 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -572,8 +572,53 @@ static void test_gfp_pisa(struct kunit *test)
static void test_reservoir_and_scrubber(struct kunit *test)
{
- KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
- KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+ unsigned int old_res = READ_ONCE(pisa_reservoir_max);
+ unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ struct page *p;
+ int nid;
+
+ pisa_shrink_scan(NULL, &sc);
+
+ /* Refill reservoir to 1 PISA page per memory node, then consume one. */
+ WRITE_ONCE(pisa_reservoir_max, 1);
+ pisa_schedule_reservoir();
+ for_each_node_state(nid, N_MEMORY)
+ flush_work(&pisa_nodes[nid].work);
+ KUNIT_EXPECT_GE(test, pisa_reservoir_count(), 1UL);
+
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ p = pisa_alloc(pa, GFP_KERNEL);
+ if (p) {
+ meta = pisa_meta_of_pfn(page_to_pfn(p));
+ /* Reservoir pages come pre-zeroed (all remaining blocks clean). */
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ __free_pages(p, 2);
+ if (!want_init_on_free()) {
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 1U);
+
+ /* Scrub dirty block back into clean_bitmap. */
+ pa->scrub_demand = 1;
+ WRITE_ONCE(pisa_scrub_max, 512);
+ pisa_scrub_node(pa->nid);
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ }
+ }
+ pisa_shrink_scan(NULL, &sc);
+
+ /* Trim reservoirs back to 0. */
+ WRITE_ONCE(pisa_reservoir_max, 0);
+ pisa_schedule_reservoir();
+ for_each_node_state(nid, N_MEMORY)
+ flush_work(&pisa_nodes[nid].work);
+ KUNIT_EXPECT_EQ(test, pisa_reservoir_count(), 0UL);
+
+ WRITE_ONCE(pisa_reservoir_max, old_res);
+ pisa_schedule_reservoir();
+ WRITE_ONCE(pisa_scrub_max, old_scrub);
}
static struct kunit_case pisa_meta_test_cases[] = {
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index dfde9c021fe05..cd29c146c281e 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -5,6 +5,7 @@
*/
#include <linux/atomic.h>
+#include <linux/bitops.h>
#include <linux/pisa.h>
#include <linux/err.h>
#include <linux/gfp.h>
@@ -32,6 +33,9 @@
*/
unsigned int pisa_reservoir_max __read_mostly = 16;
+/* Scrub the allocators whose home node (@pa->nid) is this one. */
+#define PISA_NODE_SCRUB 0
+
static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
{
int ret = param_set_uint(val, kp);
@@ -175,6 +179,30 @@ static void pisa_reservoir_trim_node(int nid, unsigned int max)
}
}
+/* Schedule node @nid's work item to scrub allocators homed on @nid. */
+void pisa_schedule_scrub(int nid)
+{
+ struct pisa_node *node;
+
+ if (!READ_ONCE(pisa_scrub_max) || nid == NUMA_NO_NODE)
+ return;
+
+ node = &pisa_nodes[nid];
+ set_bit(PISA_NODE_SCRUB, &node->flags);
+ /* Pairs with the clearing of PENDING before the work runs. */
+ smp_mb__after_atomic();
+ if (!work_pending(&node->work))
+ queue_work_node(nid, system_dfl_wq, &node->work);
+}
+
+void pisa_schedule_scrub_all(void)
+{
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ pisa_schedule_scrub(nid);
+}
+
/* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
static void pisa_node_work_fn(struct work_struct *work)
{
@@ -184,6 +212,9 @@ static void pisa_node_work_fn(struct work_struct *work)
pisa_reservoir_trim_node(nid, READ_ONCE(pisa_reservoir_max));
if (READ_ONCE(pisa_reservoir_max) && static_branch_unlikely(&pisa_key))
pisa_reservoir_refill_node(nid);
+
+ if (test_and_clear_bit(PISA_NODE_SCRUB, &node->flags) && pisa_scrub_node(nid))
+ pisa_schedule_scrub(nid);
}
/* Queue node @nid's work if its reservoir is below or above the target. */
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f08388f003fad..ef3abb1db9049 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -144,8 +144,9 @@ struct pisa_alloc_stats {
* @num_idle_pages: Length of @idle
* @max_idle_pages: High watermark of @idle before async release to the buddy allocator
* @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @scrub_demand: Dirty blocks zeroed inline since the scrubber last ran
* @blocks_in_use: Blocks handed out and not yet freed
- * @nid: NUMA node id
+ * @nid: NUMA node whose worker scrubs this allocator
* @node: Node in the global pisa_allocs list
* @name: Debug name (defaults to caller of pisa_create())
* @stats: Per-allocator statistics
@@ -162,6 +163,7 @@ struct pisa_alloc {
struct list_head full;
unsigned int num_idle_pages;
unsigned int max_idle_pages;
+ unsigned int scrub_demand;
unsigned long idle_jiffies;
unsigned long blocks_in_use;
/* Cold. */
@@ -187,7 +189,8 @@ struct pisa_node {
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
unsigned int nr_pages;
- struct work_struct work; /* refill/trim, on this node */
+ unsigned long flags; /* PISA_NODE_* requests */
+ struct work_struct work; /* refill/trim/scrub, on this node */
} ____cacheline_aligned_in_smp;
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
@@ -263,11 +266,15 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
unsigned int *nr, struct list_head *release_list,
unsigned long nr_to_scan);
+extern unsigned int pisa_scrub_max;
+bool pisa_scrub_node(int nid);
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
+void pisa_schedule_scrub(int nid);
+void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (40 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
` (8 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
skb heads are provided by a slab allocator that is fed by alloc_pages().
Add SLAB_CACHE_PISA_RO so that a slab cache can back its slabs with
zeroed, read-only PISA folios (__GFP_PISA_RO | __GFP_ZERO) via
pisa_alloc_folio(), clamping the slab order to
[2, PAGE_ALLOC_COSTLY_ORDER].
Slabs are still released with free_frozen_pages(), so pages from PISA
return to their allocator via __free_pages_prepare(). Caches with
SLAB_CACHE_PISA_RO skip KFENCE and are never merged.
Existing caches are unaffected.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 27 +++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 19 +++++++++++++++++++
include/linux/pisa.h | 7 +++++++
include/linux/slab.h | 10 ++++++++++
mm/slab.h | 3 ++-
mm/slab_common.c | 5 ++++-
mm/slub.c | 14 +++++++++++++-
7 files changed, 82 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 4b61ec84ed6d1..3b5c49d143a7e 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -848,6 +848,33 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred
return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
}
+/**
+ * pisa_alloc_folio - Allocate a frozen folio for SLUB (SLAB_CACHE_PISA_RO)
+ * @gfp: GFP flags (must have __GFP_PISA_RW or __GFP_PISA_RO set)
+ * @order: Folio order
+ * @nid: Target NUMA node (or NUMA_NO_NODE)
+ *
+ * Drops __GFP_THISNODE before calling the per-node allocator and verifies
+ * node placement afterwards.
+ *
+ * Return: Frozen folio of @order, or NULL.
+ */
+struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+ struct page *page;
+
+ page = pisa_alloc_pages(gfp & ~__GFP_THISNODE, order, nid, NULL);
+ if (!page)
+ return NULL;
+ if (unlikely((gfp & __GFP_THISNODE) && nid != NUMA_NO_NODE &&
+ page_to_nid(page) != nid)) {
+ page_ref_unfreeze(page, 1);
+ __free_pages(page, order);
+ return NULL;
+ }
+ return page_folio(page);
+}
+
/**
* pisa_has_free_blocks_rx - Check if @dev has free PISA RX blocks of @order on @nid
* @dev: Device to check
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 40bf7801183da..9cc1328893ace 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
+#include <linux/slab.h>
#include <net/sock.h>
#include "dma-iommu.h"
@@ -621,6 +622,23 @@ static void test_reservoir_and_scrubber(struct kunit *test)
WRITE_ONCE(pisa_scrub_max, old_scrub);
}
+static void test_slub_page_alloc(struct kunit *test)
+{
+ struct kmem_cache *s;
+ void *obj;
+
+ s = kmem_cache_create("pisa_kunit_slab", 1024, NULL,
+ SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+ KUNIT_ASSERT_NOT_NULL(test, s);
+
+ obj = kmem_cache_alloc(s, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, obj);
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(virt_to_page(obj)));
+ kmem_cache_free(s, obj);
+
+ kmem_cache_destroy(s);
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
@@ -635,6 +653,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_pcp_cache),
KUNIT_CASE(test_gfp_pisa),
KUNIT_CASE(test_reservoir_and_scrubber),
+ KUNIT_CASE(test_slub_page_alloc),
{}
};
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 89eb0722d90bc..b9c5197f2d88d 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -110,6 +110,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int nr_pages, struct page **page_array,
bool frozen);
+struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
bool pisa_page_can_recycle(const struct page *page);
@@ -181,6 +182,12 @@ pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
return 0;
}
+static inline struct folio *
+pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+ return NULL;
+}
+
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
return 0;
diff --git a/include/linux/slab.h b/include/linux/slab.h
index cda126def67af..df19c5898e9fe 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -66,6 +66,9 @@ enum _slab_flag_bits {
#endif
#endif
_SLAB_NO_SHEAVES,
+#ifdef CONFIG_PISA
+ _SLAB_CACHE_PISA_RO,
+#endif
_SLAB_FLAGS_LAST_BIT
};
@@ -260,6 +263,13 @@ enum _slab_flag_bits {
#define SLAB_OBJ_EXT_IN_OBJ __SLAB_FLAG_UNUSED
#endif
+/* Back slabs with zeroed read-only PISA folios (__GFP_PISA_RO | __GFP_ZERO) */
+#ifdef CONFIG_PISA
+#define SLAB_CACHE_PISA_RO __SLAB_FLAG_BIT(_SLAB_CACHE_PISA_RO)
+#else
+#define SLAB_CACHE_PISA_RO __SLAB_FLAG_UNUSED
+#endif
+
/*
* ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
*
diff --git a/mm/slab.h b/mm/slab.h
index e94ac63d10dc0..53620e40b216b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -473,7 +473,8 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
SLAB_NOLEAKTRACE | SLAB_RECLAIM_ACCOUNT | \
SLAB_TEMPORARY | SLAB_ACCOUNT | \
- SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE)
+ SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE | \
+ SLAB_CACHE_PISA_RO)
#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
SLAB_TRACE | SLAB_CONSISTENCY_CHECKS)
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 01c64ef661e7f..d6c00047b1623 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -49,7 +49,7 @@ struct kmem_cache *kmem_cache;
*/
#define SLAB_NEVER_MERGE (SLAB_DEBUG_FLAGS | SLAB_TYPESAFE_BY_RCU | \
SLAB_NOLEAKTRACE | SLAB_FAILSLAB | SLAB_NO_MERGE | \
- SLAB_OBJ_EXT_IN_OBJ)
+ SLAB_OBJ_EXT_IN_OBJ | SLAB_CACHE_PISA_RO)
#define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
SLAB_CACHE_DMA32 | SLAB_ACCOUNT | SLAB_MAY_ACCOUNT)
@@ -377,6 +377,9 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
if (s)
goto out_unlock;
+ if (flags & SLAB_CACHE_PISA_RO)
+ flags |= SLAB_SKIP_KFENCE;
+
cache_name = kstrdup_const(name, GFP_KERNEL);
if (!cache_name) {
err = -ENOMEM;
diff --git a/mm/slub.c b/mm/slub.c
index 45e077c7526c2..63609768f4e3f 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -46,6 +46,7 @@
#include <linux/prandom.h>
#include <kunit/test.h>
#include <kunit/test-bug.h>
+#include <linux/pisa.h>
#include <linux/sort.h>
#include <linux/irq_work.h>
#include <linux/kprobes.h>
@@ -3378,6 +3379,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,
if (unlikely(!allow_spin))
page = alloc_frozen_pages_nolock(0/* __GFP_COMP is implied */,
node, order);
+ else if (unlikely(flags & __GFP_PISA_RO))
+ page = (struct page *)pisa_alloc_folio(flags, order, node);
else if (node == NUMA_NO_NODE)
page = alloc_frozen_pages(flags, order);
else
@@ -3484,7 +3487,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
gfp_t alloc_gfp;
void *start;
- flags &= gfp_allowed_mask;
+ flags &= gfp_allowed_mask & ~__GFP_PISA_RW;
flags |= s->allocflags;
@@ -3495,9 +3498,13 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
if ((alloc_gfp & __GFP_DIRECT_RECLAIM) && oo_order(oo) > oo_order(s->min))
alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;
+ if (unlikely(s->flags & SLAB_CACHE_PISA_RO))
+ alloc_gfp = (flags | __GFP_NOWARN) & ~__GFP_NOFAIL;
slab = alloc_slab_page(alloc_gfp, node, oo, allow_spin);
if (unlikely(!slab)) {
+ if (s->flags & SLAB_CACHE_PISA_RO)
+ return NULL;
oo = s->min;
alloc_gfp = flags;
/*
@@ -8182,6 +8189,11 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
if (s->flags & SLAB_RECLAIM_ACCOUNT)
s->allocflags |= __GFP_RECLAIMABLE;
+ if (s->flags & SLAB_CACHE_PISA_RO) {
+ s->allocflags |= __GFP_PISA_RO | __GFP_ZERO;
+ order = clamp_t(unsigned int, order, 2, PAGE_ALLOC_COSTLY_ORDER);
+ }
+
/*
* For kmalloc caches we enable sheaves later by
* bootstrap_kmalloc_sheaves() to avoid recursion.
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (41 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
` (7 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
The linear part of most transmitted skbs is a small head from the
skbuff_small_head cache. To avoid per-packet IOMMU map/unmap operations
or bounce-buffer copies for headers, allow backing small TX skb heads
with PISA memory.
When per-device pisa_tx_skb_heads is enabled, create a parallel
skbuff_small_head_pisa slab cache backed by zeroed, read-only PISA
folios (SLAB_CACHE_PISA_RO). Eligible TX paths passing __GFP_PISA_RO
allocate small heads from this cache and fall back to skbuff_small_head
on failure.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 4 +-
drivers/iommu/pisa-alloc.c | 133 ++++++++++++++++++++++++++++++++-----
drivers/iommu/pisa-kunit.c | 5 +-
include/linux/device.h | 6 ++
include/linux/pisa.h | 6 ++
include/net/sock.h | 74 ++++++++++++++++++---
net/core/skbuff.c | 39 ++++++++---
net/core/sock.c | 23 +++++--
net/ipv4/ip_output.c | 8 ++-
net/ipv4/tcp.c | 2 +-
net/ipv4/tcp_output.c | 12 ++--
net/ipv6/ip6_output.c | 8 ++-
12 files changed, 263 insertions(+), 57 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index ade0c94b02ae7..efc1f9b9d255d 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2951,7 +2951,7 @@ static DEVICE_ATTR_RW(name)
PISA_BOOL_ATTR(pisa_coherent);
PISA_BOOL_ATTR(pisa_rx_bufs);
PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
-PISA_BOOL_ATTR(pisa_tx_skb_heads);
+PISA_SET_ATTR(pisa_tx_skb_heads, net_pisa_tx_skb_heads_set);
static struct attribute *dev_attr_pisa[] = {
&dev_attr_pisa_coherent.attr,
@@ -3113,6 +3113,8 @@ static void device_remove_attrs(struct device *dev)
device_remove_group(dev, &dev_attr_pisa_group);
if (dev->pisa_tx_bufs)
net_pisa_tx_bufs_set(dev, false);
+ if (dev->pisa_tx_skb_heads)
+ net_pisa_tx_skb_heads_set(dev, false);
}
if (dev->physical_location) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 3b5c49d143a7e..7df2ecde32916 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -37,6 +37,7 @@
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
+#include <net/hotdata.h>
#include <net/sock.h>
#include "dma-iommu.h"
@@ -922,37 +923,133 @@ int net_pisa_tx_bufs_set(struct device *dev, bool val)
}
/*
- * The sysfs attribute pisa_tx_bufs lives on the physical DMA parent
- * (dev->dev.parent), whereas a socket route (dst->dev) may point to an upper
- * virtual net_device (bond, VLAN, ipvlan, etc.). Walking lower devices via
- * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
- * expensive, so dev->pisa_tx_stamp caches the walk result on the net_device
- * and refreshes it at most once every 256 jiffies:
- * bit 0 (PISA_TX_STAMP_OK): pisa_tx_bufs enabled on lower device(s)
+ * dev->pisa_tx_skb_heads: when set on an egress device, small TX skb heads
+ * (__GFP_PISA_RO) come from a copy of skbuff_small_head whose slab pages are
+ * PISA blocks. The linear part of most transmitted skbs is then mapped
+ * through the PISA window without per-driver bounce buffers. Several heads
+ * share a slab page and each one ends with its skb_shared_info, so:
+ * - the allocators are TX-only (DMA_TO_DEVICE): their PISA pages are mapped device
+ * read-only, and a DMA mapping that wants device writes gets a per-buffer
+ * IOMMU mapping, as for any kmalloc() buffer;
+ * - slab pages are zeroed when taken from the allocator, so the device can read
+ * skb heads (live or freed) but never older contents of the page.
+ * Heads are freed with kfree() whichever cache they came from, so the key
+ * can be toggled at any time. The cache and allocators are created on the first
+ * write and never destroyed.
+ */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
+EXPORT_SYMBOL(net_pisa_tx_skb_heads_key);
+
+static struct kmem_cache *pisa_skb_small_head_cache __read_mostly;
+
+/* Lazily initialize skbuff_small_head_pisa on first enable. */
+static int pisa_skb_head_init(void)
+{
+ unsigned int size = kmem_cache_size(net_hotdata.skb_small_head_cache);
+ struct kmem_cache_args args = {
+ /* As skbuff_small_head, see skb_init(). */
+ .useroffset = 0,
+ .usersize = SKB_WITH_OVERHEAD(size),
+ .sheaf_capacity = 32,
+ };
+ static DEFINE_MUTEX(mutex);
+ struct kmem_cache *s;
+ int ret = 0;
+
+ mutex_lock(&mutex);
+ if (pisa_skb_small_head_cache)
+ goto out;
+
+ /* No slab is allocated before the cache is published below. */
+ s = kmem_cache_create("skbuff_small_head_pisa", size, &args,
+ SLAB_HWCACHE_ALIGN | SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+ if (!s) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ WRITE_ONCE(pisa_skb_small_head_cache, s);
+out:
+ mutex_unlock(&mutex);
+ return ret;
+}
+
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+ static DEFINE_MUTEX(lock);
+
+ if (val) {
+ int ret = pisa_skb_head_init();
+
+ if (ret)
+ return ret;
+ }
+
+ mutex_lock(&lock);
+ if (dev->pisa_tx_skb_heads != val) {
+ WRITE_ONCE(dev->pisa_tx_skb_heads, val);
+ if (val)
+ static_branch_inc(&net_pisa_tx_skb_heads_key);
+ else
+ static_branch_dec(&net_pisa_tx_skb_heads_key);
+ }
+ mutex_unlock(&lock);
+ return 0;
+}
+
+void *pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+ if (!static_branch_unlikely(&net_pisa_tx_skb_heads_key))
+ return NULL;
+ return kmem_cache_alloc_node(pisa_skb_small_head_cache,
+ flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node);
+}
+
+/*
+ * The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the
+ * physical DMA parent (dev->dev.parent), whereas a socket route (dst->dev)
+ * may point to an upper virtual net_device (bond, VLAN, ipvlan, etc.).
+ * Walking lower devices via netdev_walk_all_lower_dev_rcu() on every TX
+ * allocation would be too expensive, so dev->pisa_tx_stamp caches the walk
+ * result on the net_device and refreshes it at most once every 256 jiffies:
+ * bit 0 (PISA_TX_STAMP_BUFS): pisa_tx_bufs enabled on lower device(s)
+ * bit 1 (PISA_TX_STAMP_HEADS): pisa_tx_skb_heads enabled on lower device(s)
* bits 31..8 (PISA_TX_STAMP_MASK): upper 24 bits of (u32)jiffies
*/
-static int netdev_pisa_tx_bufs_walk(struct net_device *dev, struct netdev_nested_priv *priv)
+static u32 netdev_pisa_tx_flags(const struct net_device *dev)
{
- return dev->dev.parent && READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ struct device *parent = dev->dev.parent;
+ u32 flags = 0;
+
+ if (parent) {
+ if (READ_ONCE(parent->pisa_tx_bufs))
+ flags |= PISA_TX_STAMP_BUFS;
+ if (READ_ONCE(parent->pisa_tx_skb_heads))
+ flags |= PISA_TX_STAMP_HEADS;
+ }
+ return flags;
}
-/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
-bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+static int netdev_pisa_tx_walk(struct net_device *dev, struct netdev_nested_priv *priv)
{
- bool ok;
+ priv->data = (void *)((uintptr_t)priv->data | netdev_pisa_tx_flags(dev));
+ return 0;
+}
+/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
+u32 netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+{
if (dev->dev.parent) {
- ok = READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ now |= netdev_pisa_tx_flags(dev);
} else {
+ struct netdev_nested_priv priv = {};
+
rcu_read_lock();
- ok = netdev_walk_all_lower_dev_rcu(dev,
- netdev_pisa_tx_bufs_walk,
- NULL) != 0;
+ netdev_walk_all_lower_dev_rcu(dev, netdev_pisa_tx_walk, &priv);
rcu_read_unlock();
+ now |= (uintptr_t)priv.data;
}
- now |= ok ? PISA_TX_STAMP_OK : 0;
WRITE_ONCE(dev->pisa_tx_stamp, now);
- return now & PISA_TX_STAMP_OK;
+ return now;
}
EXPORT_SYMBOL(netdev_tx_pisa_refresh);
#endif /* CONFIG_NET */
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 9cc1328893ace..3fc17024bc9d0 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -489,9 +489,10 @@ static void test_dev_pisa_attrs(struct kunit *test)
}
kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
- if (IS_ENABLED(CONFIG_NET))
+ if (IS_ENABLED(CONFIG_NET)) {
kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
- kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ }
device_unregister(dev);
}
diff --git a/include/linux/device.h b/include/linux/device.h
index 08f511b978ded..d588ae27c960c 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -816,11 +816,17 @@ struct device {
#if defined(CONFIG_NET) && defined(CONFIG_PISA)
int net_pisa_tx_bufs_set(struct device *dev, bool val);
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val);
#else
static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
{
return -EOPNOTSUPP;
}
+
+static inline int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+ return -EOPNOTSUPP;
+}
#endif
#define __create_dev_flag_accessors(accessor_name, flag_name) \
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index b9c5197f2d88d..452b8fa20c8d7 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -113,6 +113,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
bool pisa_page_can_recycle(const struct page *page);
+void *pisa_skb_head_alloc_node(gfp_t flags, int node);
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
@@ -169,6 +170,11 @@ static inline bool pisa_free_coherent(struct page *page, size_t size)
return false;
}
+static inline void *pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+ return NULL;
+}
+
static inline bool pisa_page_can_recycle(const struct page *page)
{
return true;
diff --git a/include/net/sock.h b/include/net/sock.h
index 73906351d11db..78f16b76dfc3b 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1900,6 +1900,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname,
sockptr_t optval, sockptr_t optlen);
int sock_gettstamp(struct socket *sock, void __user *userstamp,
bool timeval, bool time32);
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order,
+ gfp_t gfp);
struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
unsigned long data_len, int noblock,
int *errcode, int max_page_order);
@@ -2671,20 +2675,27 @@ static inline struct page_frag *sk_page_frag(struct sock *sk)
}
/* 256-jiffy cache in dev->pisa_tx_stamp; see netdev_tx_pisa_refresh(). */
-#define PISA_TX_STAMP_OK BIT(0)
+#define PISA_TX_STAMP_BUFS BIT(0)
+#define PISA_TX_STAMP_HEADS BIT(1)
#define PISA_TX_STAMP_MASK (~0xffU)
#ifdef CONFIG_PISA
DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
static inline bool net_pisa_tx_bufs_enabled(void)
{
return static_branch_unlikely(&net_pisa_tx_bufs_key);
}
-bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+static inline bool net_pisa_tx_skb_heads_enabled(void)
+{
+ return static_branch_unlikely(&net_pisa_tx_skb_heads_key);
+}
-static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+u32 netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev, u32 flag)
{
u32 stamp, now;
@@ -2692,9 +2703,9 @@ static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
return false;
stamp = READ_ONCE(dev->pisa_tx_stamp);
now = (u32)jiffies & PISA_TX_STAMP_MASK;
- if (likely((stamp & PISA_TX_STAMP_MASK) == now))
- return stamp & PISA_TX_STAMP_OK;
- return netdev_tx_pisa_refresh((struct net_device *)dev, now);
+ if (unlikely((stamp & PISA_TX_STAMP_MASK) != now))
+ stamp = netdev_tx_pisa_refresh((struct net_device *)dev, now);
+ return stamp & flag;
}
#else
static inline bool net_pisa_tx_bufs_enabled(void)
@@ -2702,7 +2713,12 @@ static inline bool net_pisa_tx_bufs_enabled(void)
return false;
}
-static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+static inline bool net_pisa_tx_skb_heads_enabled(void)
+{
+ return false;
+}
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev, u32 flag)
{
return false;
}
@@ -2714,20 +2730,58 @@ static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
* active, so only data that leaves the host in clear qualifies: IPsec payload
* is still plaintext here.
*/
-static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+static inline bool __dst_tx_pisa_ok(const struct dst_entry *dst, u32 flag)
{
bool ok = false;
- if (!net_pisa_tx_bufs_enabled() || !dst)
+ if (!dst)
return false;
rcu_read_lock();
if (!dst_xfrm(dst))
- ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst));
+ ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst), flag);
rcu_read_unlock();
return ok;
}
+static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+{
+ return net_pisa_tx_bufs_enabled() &&
+ __dst_tx_pisa_ok(dst, PISA_TX_STAMP_BUFS);
+}
+
+static inline gfp_t pisa_tx_gfp_dst(const struct dst_entry *dst)
+{
+ if (net_pisa_tx_skb_heads_enabled() &&
+ __dst_tx_pisa_ok(dst, PISA_TX_STAMP_HEADS))
+ return __GFP_PISA_RO;
+ return 0;
+}
+
+static inline gfp_t pisa_tx_gfp_sk(const struct sock *sk)
+{
+ if (net_pisa_tx_skb_heads_enabled() && sk) {
+ bool ok;
+
+ rcu_read_lock();
+ ok = __dst_tx_pisa_ok(rcu_dereference(sk->sk_dst_cache),
+ PISA_TX_STAMP_HEADS);
+ rcu_read_unlock();
+ if (ok)
+ return __GFP_PISA_RO;
+ }
+ return 0;
+}
+
+static inline struct sk_buff *sock_alloc_send_skb_dst(struct sock *sk,
+ unsigned long size,
+ int noblock, int *errcode,
+ const struct dst_entry *dst)
+{
+ return __sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0,
+ pisa_tx_gfp_dst(dst));
+}
+
/* Is @pfrag the task's frag reserved for PISA-eligible data? */
static inline bool is_pisa_pfrag(const struct page_frag *pfrag)
{
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 8da53d5b76fc2..04ae3bb4bd905 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -93,6 +93,7 @@
#include <linux/user_namespace.h>
#include <linux/indirect_call_wrapper.h>
#include <linux/textsearch.h>
+#include <linux/pisa.h>
#include "dev.h"
#include "devmem.h"
@@ -607,16 +608,23 @@ static void *kmalloc_pfmemalloc(size_t obj_size, gfp_t flags, int node)
static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node,
struct sk_buff *skb)
{
+ bool pisa = flags & __GFP_PISA_RO;
size_t obj_size;
void *obj;
+ flags &= ~__GFP_PISA_RW;
obj_size = SKB_HEAD_ALIGN(*size);
if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE &&
!(flags & KMALLOC_NOT_NORMAL_BITS)) {
+ *size = SKB_SMALL_HEAD_CACHE_SIZE;
+ if (unlikely(pisa)) {
+ obj = pisa_skb_head_alloc_node(flags, node);
+ if (likely(obj))
+ goto out;
+ }
obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache,
flags | __GFP_NOMEMALLOC | __GFP_NOWARN,
node);
- *size = SKB_SMALL_HEAD_CACHE_SIZE;
if (likely(obj))
goto out;
/* Try again but now we are using pfmemalloc reserves */
@@ -2144,6 +2152,15 @@ void skb_copy_header(struct sk_buff *new, const struct sk_buff *old)
}
EXPORT_SYMBOL(skb_copy_header);
+static inline gfp_t skb_head_pisa_gfp(const struct sk_buff *skb)
+{
+ if (net_pisa_tx_skb_heads_enabled() &&
+ !skb->head_frag &&
+ is_pisa_page(virt_to_page(skb->head)))
+ return __GFP_PISA_RO;
+ return 0;
+}
+
static inline int skb_alloc_rx_flag(const struct sk_buff *skb)
{
if (skb_pfmemalloc(skb))
@@ -2182,7 +2199,7 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
headerlen = skb_headroom(skb);
size = skb_end_offset(skb) + skb->data_len;
- n = __alloc_skb(size, gfp_mask,
+ n = __alloc_skb(size, gfp_mask | skb_head_pisa_gfp(skb),
skb_alloc_rx_flag(skb), NUMA_NO_NODE);
if (!n)
return NULL;
@@ -2226,7 +2243,8 @@ struct sk_buff *__pskb_copy_fclone(struct sk_buff *skb, int headroom,
{
unsigned int size = skb_headlen(skb) + headroom;
int flags = skb_alloc_rx_flag(skb) | (fclone ? SKB_ALLOC_FCLONE : 0);
- struct sk_buff *n = __alloc_skb(size, gfp_mask, flags, NUMA_NO_NODE);
+ struct sk_buff *n = __alloc_skb(size, gfp_mask | skb_head_pisa_gfp(skb),
+ flags, NUMA_NO_NODE);
if (!n)
goto out;
@@ -2315,7 +2333,8 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
goto nodata;
size = SKB_WITH_OVERHEAD(size);
@@ -2524,7 +2543,8 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
oldheadroom = skb_headroom(skb);
n = __alloc_skb(newheadroom + skb->len + newtailroom,
- gfp_mask, skb_alloc_rx_flag(skb),
+ gfp_mask | skb_head_pisa_gfp(skb),
+ skb_alloc_rx_flag(skb),
NUMA_NO_NODE);
if (!n)
return NULL;
@@ -5025,7 +5045,8 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
hsize = len;
nskb = __alloc_skb(hsize + doffset + headroom,
- GFP_ATOMIC, skb_alloc_rx_flag(head_skb),
+ GFP_ATOMIC | skb_head_pisa_gfp(head_skb),
+ skb_alloc_rx_flag(head_skb),
NUMA_NO_NODE);
if (unlikely(!nskb))
@@ -6974,7 +6995,8 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
return -ENOMEM;
size = SKB_WITH_OVERHEAD(size);
@@ -7096,7 +7118,8 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
return -ENOMEM;
size = SKB_WITH_OVERHEAD(size);
diff --git a/net/core/sock.c b/net/core/sock.c
index 695fefe316925..0ddb5e57f4969 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -3061,9 +3061,10 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
* Generic send/receive buffer handlers
*/
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
- unsigned long data_len, int noblock,
- int *errcode, int max_page_order)
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order,
+ gfp_t gfp)
{
struct sk_buff *skb;
long timeo;
@@ -3092,7 +3093,8 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
timeo = sock_wait_for_wmem(sk, timeo);
}
skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
- errcode, sk->sk_allocation);
+ errcode,
+ sk->sk_allocation | (!data_len ? gfp : 0));
if (skb)
skb_set_owner_w(skb, sk);
return skb;
@@ -3103,6 +3105,16 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
*errcode = err;
return NULL;
}
+EXPORT_SYMBOL(__sock_alloc_send_pskb);
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order)
+{
+ return __sock_alloc_send_pskb(sk, header_len, data_len, noblock,
+ errcode, max_page_order,
+ pisa_tx_gfp_sk(sk));
+}
EXPORT_SYMBOL(sock_alloc_send_pskb);
int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
@@ -3322,7 +3334,8 @@ bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
const struct dst_entry *dst = skb_dst(skb);
rcu_read_lock();
- if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL)))
+ if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL),
+ PISA_TX_STAMP_BUFS))
gfp |= __GFP_PISA_RO;
rcu_read_unlock();
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index a5ea530bc8b5b..c49a33355ea45 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1134,14 +1134,16 @@ static int __ip_append_data(struct sock *sk,
alloclen += alloc_extra;
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk, alloclen,
- (flags & MSG_DONTWAIT), &err);
+ skb = sock_alloc_send_skb_dst(sk, alloclen,
+ flags & MSG_DONTWAIT,
+ &err, &rt->dst);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
skb = alloc_skb(alloclen,
- sk->sk_allocation);
+ sk->sk_allocation |
+ pisa_tx_gfp_dst(&rt->dst));
if (unlikely(!skb))
err = -ENOBUFS;
}
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index b8d44a0638914..038063717951c 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -931,7 +931,7 @@ struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp,
{
struct sk_buff *skb;
- skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp);
+ skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp | pisa_tx_gfp_sk(sk));
if (likely(skb)) {
bool mem_scheduled;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index cf3ddc055d48d..b7f04a9df199a 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -3892,8 +3892,8 @@ void tcp_send_fin(struct sock *sk)
}
} else {
skb = alloc_skb_fclone(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC |
- __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) |
+ pisa_tx_gfp_sk(sk));
if (unlikely(!skb))
return;
@@ -3921,7 +3921,7 @@ void tcp_send_active_reset(struct sock *sk, enum sk_rst_reason reason)
TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS);
/* NOTE: No TCP options attached and we never retransmit this. */
- skb = alloc_skb(MAX_TCP_HEADER, priority);
+ skb = alloc_skb(MAX_TCP_HEADER, priority | pisa_tx_gfp_sk(sk));
if (!skb) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED);
return;
@@ -4011,7 +4011,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
int mss;
u64 now;
- skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC);
+ skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC | pisa_tx_gfp_dst(dst));
if (unlikely(!skb)) {
dst_release(dst);
return NULL;
@@ -4548,7 +4548,7 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt, u16 flags)
* sock.
*/
buff = alloc_skb(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) | pisa_tx_gfp_sk(sk));
if (unlikely(!buff)) {
struct inet_connection_sock *icsk = inet_csk(sk);
unsigned long delay;
@@ -4601,7 +4601,7 @@ static int tcp_xmit_probe_skb(struct sock *sk, int urgent, int mib)
/* We don't queue it, tcp_transmit_skb() sets ownership. */
skb = alloc_skb(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) | pisa_tx_gfp_sk(sk));
if (!skb)
return -1;
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 957c38a08680c..47a6fea3456fe 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1695,14 +1695,16 @@ static int __ip6_append_data(struct sock *sk,
goto error;
}
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk, alloclen,
- (flags & MSG_DONTWAIT), &err);
+ skb = sock_alloc_send_skb_dst(sk, alloclen,
+ flags & MSG_DONTWAIT,
+ &err, &rt->dst);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
skb = alloc_skb(alloclen,
- sk->sk_allocation);
+ sk->sk_allocation |
+ pisa_tx_gfp_dst(&rt->dst));
if (unlikely(!skb))
err = -ENOBUFS;
}
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (42 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
` (6 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
MSG_ZEROCOPY sends map each user page with a per-buffer IOMMU mapping
and IOTLB flush. Moving user TX buffers into PISA blocks once allows
subsequent zerocopy sends to reuse pre-mapped 2MB IOMMU entries.
Add drivers/iommu/pisa-user.c to migrate private anonymous order-0 user
pages into per-node DMA_TO_DEVICE PISA allocators, controlled by
pisa.user_max_4K_pages (0 = disabled):
- Explicit migration via madvise(MADV_PISA) and MADV_NOPISA (also
supported via process_madvise()).
- Automatic best-effort migration on MSG_ZEROCOPY sends in
zerocopy_fill_skb_from_iter() when the egress device has pisa_tx_bufs
enabled. For TCP, pages that cannot be migrated fall back to the
regular copy path into PISA TX page_frags.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 2 +
drivers/iommu/pisa-priv.h | 2 +
drivers/iommu/pisa-user.c | 278 +++++++++++++++++++++++++
include/linux/pisa.h | 28 +++
include/linux/swap.h | 1 +
include/uapi/asm-generic/mman-common.h | 2 +
mm/madvise.c | 11 +
net/core/datagram.c | 82 +++++++-
net/core/skbuff.c | 3 +-
net/ipv4/tcp.c | 12 +-
11 files changed, 416 insertions(+), 7 deletions(-)
create mode 100644 drivers/iommu/pisa-user.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 125cf7e813cdc..cd14f80a594a7 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o pisa-user.o
ifeq ($(CONFIG_IOMMU_DMA),y)
obj-$(CONFIG_PISA) += pisa-map.o
endif
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 7df2ecde32916..1930135880d2b 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -1756,6 +1756,8 @@ static int __init pisa_init(void)
}
}
+ pisa_user_boot_enable();
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index ef3abb1db9049..fe1bff0bba322 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -185,6 +185,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
+ struct pisa_alloc *user_pa; /* user page migration pa */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -278,6 +279,7 @@ void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+void pisa_user_boot_enable(void);
#endif /* CONFIG_PISA */
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
new file mode 100644
index 0000000000000..a16909d429aca
--- /dev/null
+++ b/drivers/iommu/pisa-user.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA user and page-cache folios: allocate or migrate order-0 folios into
+ * per-node PISA allocators so zero-copy TX and block I/O use 2MB IOMMU mappings.
+ * Disabled on CoCo guests (user PTEs map pages encrypted).
+ */
+#include <linux/cc_platform.h>
+#include <linux/pisa.h>
+#include <linux/gfp.h>
+#include <linux/migrate.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/nodemask.h>
+#include <linux/pagewalk.h>
+#include <linux/swap.h>
+
+#include "pisa-priv.h"
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Soft cap on 4KB user pages held in PISA blocks (0 = disabled). */
+static unsigned long pisa_user_max_4k_pages __read_mostly;
+DEFINE_STATIC_KEY_FALSE(pisa_user_key);
+
+static bool pisa_user_init(void);
+
+static int pisa_user_max_set(const char *val, const struct kernel_param *kp)
+{
+ unsigned long max_pages;
+ int ret;
+
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ return -EOPNOTSUPP;
+ ret = kstrtoul(val, 0, &max_pages);
+ if (ret)
+ return ret;
+ WRITE_ONCE(pisa_user_max_4k_pages, max_pages);
+ if (max_pages) {
+ if (!pisa_nodes[first_memory_node].pa[0][0])
+ return 0;
+ if (!pisa_user_init())
+ return -ENOMEM;
+ static_branch_enable(&pisa_user_key);
+ } else {
+ static_branch_disable(&pisa_user_key);
+ }
+ return 0;
+}
+
+static const struct kernel_param_ops pisa_user_max_ops = {
+ .set = pisa_user_max_set,
+ .get = param_get_ulong,
+};
+module_param_cb(user_max_4K_pages, &pisa_user_max_ops, &pisa_user_max_4k_pages, 0644);
+
+/* True once per-node user allocators have been allocated, serialized by pisa_user_lock. */
+static bool pisa_user_ready __read_mostly;
+static DEFINE_MUTEX(pisa_user_lock);
+
+/* Lazily create per-node user allocators on first use (process context). */
+static bool pisa_user_init(void)
+{
+ int nid;
+
+ if (!READ_ONCE(pisa_user_max_4k_pages) || cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ return false;
+ /* Pairs with smp_store_release() below: the allocators are visible. */
+ if (smp_load_acquire(&pisa_user_ready))
+ return true;
+ if (pisa_meta_init())
+ return false;
+ mutex_lock(&pisa_user_lock);
+ for_each_node_state(nid, N_MEMORY) {
+ if (!pisa_nodes[nid].user_pa)
+ pisa_nodes[nid].user_pa = pisa_create(nid, 0, DMA_TO_DEVICE);
+ if (!pisa_nodes[nid].user_pa)
+ goto out; /* retried on the next call */
+ }
+ /* Publish pisa_nodes[].user_pa to lockless readers. */
+ smp_store_release(&pisa_user_ready, true);
+out:
+ mutex_unlock(&pisa_user_lock);
+ return pisa_user_ready;
+}
+
+void pisa_user_boot_enable(void)
+{
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
+ WRITE_ONCE(pisa_user_max_4k_pages, 0);
+ return;
+ }
+ if (READ_ONCE(pisa_user_max_4k_pages) && pisa_user_init())
+ static_branch_enable(&pisa_user_key);
+}
+
+/* User pages currently held in PISA blocks. */
+static unsigned long pisa_user_in_use(void)
+{
+ unsigned long used = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ const struct pisa_alloc *pa = pisa_nodes[nid].user_pa;
+
+ used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ }
+ return used;
+}
+
+/* Pages that can still be moved in before reaching pisa.user_max_4K_pages. */
+static long pisa_user_budget(void)
+{
+ return (long)(READ_ONCE(pisa_user_max_4k_pages) - pisa_user_in_use());
+}
+
+/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
+static bool pisa_user_want(struct folio *folio, bool to_pisa)
+{
+ if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
+ folio_test_ksm(folio) || folio_test_large(folio) ||
+ folio_maybe_mapped_shared(folio))
+ return false;
+ return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+}
+
+/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
+static bool pisa_user_busy(struct folio *folio, int extra)
+{
+ return folio_ref_count(folio) > folio_expected_ref_count(folio) + extra;
+}
+
+struct pisa_user_mtc {
+ gfp_t gfp;
+ long budget;
+};
+
+/* new_folio_t for migration into PISA memory. */
+static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
+{
+ struct pisa_user_mtc *mtc = (void *)private;
+ int nid = folio_nid(src);
+ struct page *page;
+
+ if (mtc->budget <= 0)
+ return NULL;
+ page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+ if (!page)
+ return NULL;
+ if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
+ put_page(page);
+ mtc->budget = 0;
+ return NULL;
+ }
+ mtc->budget--;
+ return page_folio(page);
+}
+
+/* new_folio_t for migration out of PISA memory: allocators never serve these. */
+static struct folio *pisa_user_old_folio(struct folio *src, unsigned long private)
+{
+ return __folio_alloc_node(GFP_HIGHUSER_MOVABLE | __GFP_NOWARN, 0, folio_nid(src));
+}
+
+/* Migrate the isolated folios on @list; failures are put back. */
+static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+{
+ struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+ enum migrate_mode mode = MIGRATE_SYNC;
+
+ if (list_empty(list))
+ return;
+ if (to_pisa) {
+ mtc.budget = pisa_user_budget();
+ if (!gfpflags_allow_blocking(gfp))
+ mode = MIGRATE_ASYNC;
+ }
+ if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
+ NULL, (unsigned long)&mtc, mode, MR_SYSCALL, NULL))
+ putback_movable_pages(list);
+}
+
+/**
+ * pisa_upgrade_user_pages - Try to move user pages about to be sent into PISA
+ * @pages: Pages from iov_iter_get_pages2(), one reference each
+ * @nr: Number of entries in @pages
+ *
+ * Best effort, process context. Pages that cannot be moved (e.g. still
+ * referenced by an in-flight skb, or in a large folio) are skipped; for TCP
+ * with pisa_tx_bufs enabled the caller copies them instead.
+ *
+ * Return: false if nothing was attempted and the references are untouched;
+ * true if all references were dropped and the caller must look the pages up
+ * again.
+ */
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+{
+ bool drained = false;
+ LIST_HEAD(list);
+ unsigned int i, want = 0;
+
+ if (!pisa_user_init())
+ return false;
+ /* Common case: all pages already in PISA, no budget check needed. */
+ for (i = 0; i < nr; i++)
+ want += pisa_user_want(page_folio(pages[i]), true);
+ if (!want)
+ return false;
+ if (pisa_user_budget() <= 0)
+ return false;
+ for (i = 0; i < nr; i++) {
+ struct folio *folio = page_folio(pages[i]);
+
+ if (!pisa_user_want(folio, true))
+ continue;
+ /* A freshly faulted page may still sit in a per-CPU LRU batch. */
+ if (!drained && !folio_test_lru(folio)) {
+ lru_add_drain();
+ drained = true;
+ }
+ /* The reference from iov_iter_get_pages2() is expected. */
+ if (pisa_user_busy(folio, 1))
+ continue;
+ isolate_folio_to_list(folio, &list);
+ }
+ if (list_empty(&list))
+ return false;
+ /* Migration fails on pages with references it does not expect. */
+ for (i = 0; i < nr; i++)
+ put_page(pages[i]);
+ pisa_user_migrate(&list, true,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ return true;
+}
+
+#define PISA_USER_BATCH 512
+
+/**
+ * pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
+ * @vma: VMA, with the mmap lock held for reading
+ * @start: Start address
+ * @end: End address
+ * @to_pisa: true to move present pages into PISA, false to move them out
+ *
+ * Return: 0, -EINVAL for unsupported mappings, or -EOPNOTSUPP if disabled.
+ */
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool to_pisa)
+{
+ gfp_t gfp = GFP_USER | __GFP_NOWARN;
+ struct folio_walk fw;
+ unsigned long addr;
+ unsigned int n = 0;
+ LIST_HEAD(list);
+
+ if (vma->vm_flags & (VM_SHARED | VM_IO | VM_PFNMAP | VM_HUGETLB))
+ return -EINVAL;
+ if (to_pisa && !pisa_user_init())
+ return -EOPNOTSUPP;
+ lru_add_drain();
+ for (addr = start; addr < end; addr += PAGE_SIZE) {
+ struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
+
+ if (folio) {
+ if (pisa_user_want(folio, to_pisa) && !pisa_user_busy(folio, 0) &&
+ isolate_folio_to_list(folio, &list))
+ n++;
+ folio_walk_end(&fw, vma);
+ }
+ if (n == PISA_USER_BATCH) {
+ pisa_user_migrate(&list, to_pisa, gfp);
+ n = 0;
+ }
+ cond_resched();
+ }
+ pisa_user_migrate(&list, to_pisa, gfp);
+ return 0;
+}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 452b8fa20c8d7..d11fd3d663dc1 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -132,6 +132,18 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
put_page(tsk->task_frag_pisa.page);
}
+/* User pages in PISA blocks, see drivers/iommu/pisa-user.c */
+DECLARE_STATIC_KEY_FALSE(pisa_user_key);
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr);
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+ unsigned long end, bool to_pisa);
+
+/* Is pisa.user_max_4K_pages non-zero? */
+static inline bool pisa_user_enabled(void)
+{
+ return static_branch_unlikely(&pisa_user_key);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -213,6 +225,22 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
{
}
+static inline bool pisa_user_enabled(void)
+{
+ return false;
+}
+
+static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+{
+ return false;
+}
+
+static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+ unsigned long end, bool to_pisa)
+{
+ return -EINVAL;
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4f686709d71f0..3736786ccc5a3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -314,6 +314,7 @@ extern unsigned long totalreserve_pages;
/* linux/mm/folio.c */
void folio_add_lru(struct folio *folio);
void folio_mark_accessed(struct folio *folio);
+void lru_add_drain(void);
void lru_add_drain_all(void);
enum lru_cache_drained {
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index ef1c27fa3c570..c4332db8e56ff 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -81,6 +81,8 @@
#define MADV_GUARD_INSTALL 102 /* fatal signal on access to range */
#define MADV_GUARD_REMOVE 103 /* unguard range */
+#define MADV_PISA 210 /* move pages into PISA memory */
+#define MADV_NOPISA 211 /* move pages out of PISA memory */
/* compatibility flags */
#define MAP_FILE 0
diff --git a/mm/madvise.c b/mm/madvise.c
index 0214efd2a014f..0eccb44594b04 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -6,6 +6,7 @@
* Copyright (C) 2002 Christoph Hellwig
*/
+#include <linux/pisa.h>
#include <linux/mman.h>
#include <linux/pagemap.h>
#include <linux/syscalls.h>
@@ -1393,6 +1394,9 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)
return madvise_guard_install(madv_behavior);
case MADV_GUARD_REMOVE:
return madvise_guard_remove(madv_behavior);
+ case MADV_PISA:
+ case MADV_NOPISA:
+ return pisa_madvise(vma, range->start, range->end, behavior == MADV_PISA);
/* The below behaviours update VMAs via madvise_update_vma(). */
@@ -1582,6 +1586,9 @@ madvise_behavior_valid(int behavior)
case MADV_HWPOISON:
#endif
return true;
+ case MADV_PISA:
+ case MADV_NOPISA:
+ return IS_ENABLED(CONFIG_PISA);
default:
return false;
@@ -1596,6 +1603,8 @@ static bool process_madvise_remote_valid(int behavior)
case MADV_PAGEOUT:
case MADV_WILLNEED:
case MADV_COLLAPSE:
+ case MADV_PISA:
+ case MADV_NOPISA:
return true;
default:
return false;
@@ -1782,6 +1791,8 @@ static enum madvise_lock_mode get_lock_mode(struct madvise_behavior *madv_behavi
case MADV_POPULATE_READ:
case MADV_POPULATE_WRITE:
case MADV_COLLAPSE:
+ case MADV_PISA:
+ case MADV_NOPISA:
return MADVISE_MMAP_READ_LOCK;
case MADV_GUARD_INSTALL:
case MADV_GUARD_REMOVE:
diff --git a/net/core/datagram.c b/net/core/datagram.c
index ecc30a6f83023..40f20d0faa859 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -39,6 +39,7 @@
#include <linux/kernel.h>
#include <linux/uaccess.h>
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/interrupt.h>
#include <linux/errno.h>
#include <linux/sched.h>
@@ -632,10 +633,74 @@ int skb_copy_datagram_from_iter_full(struct sk_buff *skb, int offset,
}
EXPORT_SYMBOL(skb_copy_datagram_from_iter_full);
-int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
- struct iov_iter *from, size_t length)
+/*
+ * Upgrade user @pages to PISA memory and, for TCP over a pisa_tx_bufs device,
+ * trim non-PISA trailing pages from the batch.
+ *
+ * Return: false to proceed with attaching *@copied bytes from @pages to @skb,
+ * or true to stop and make the caller return *@ret.
+ */
+static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, size_t *length,
+ const struct sock *sk, struct page **pages, int max_pages,
+ unsigned int orig_len, size_t *start, ssize_t *copied)
+{
+ unsigned int i, nr;
+ ssize_t keep;
+
+ if (!pisa_user_enabled() || !user_backed_iter(from))
+ return false;
+
+ /* Move user pages into PISA memory, then look them up again. */
+ if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE))) {
+ iov_iter_revert(from, *copied);
+ *copied = iov_iter_get_pages2(from, pages, *length, max_pages, start);
+ if (*copied < 0) {
+ *ret = -EFAULT;
+ return true;
+ }
+ }
+
+ /* Only TCP (which sets skb->sk at transmit time) handles short appends/-EXDEV. */
+ if (!sk || skb->sk)
+ return false;
+
+ /* Only plain MSG_ZEROCOPY ubufs (not io_uring/vhost/devmem). */
+ if (skb_zcopy(skb) && skb_zcopy(skb)->ops != &msg_zerocopy_ubuf_ops)
+ return false;
+
+ /* Egress device must have pisa_tx_bufs enabled so the copy uses a PISA page_frag. */
+ if (!dst_tx_pisa_ok(__sk_dst_get(sk)))
+ return false;
+
+ /*
+ * Trim @from to the leading pages that are in PISA memory: drop the
+ * references to the others, give their bytes back to @from, and cap
+ * *@length so that the caller ends after attaching what is left.
+ */
+ nr = DIV_ROUND_UP(*start + *copied, PAGE_SIZE);
+ for (i = 0; i < nr && is_pisa_page(pages[i]); i++)
+ ;
+ if (i == nr)
+ return false;
+
+ keep = i ? i * PAGE_SIZE - *start : 0;
+ while (nr > i)
+ put_page(pages[--nr]);
+ iov_iter_revert(from, *copied - keep);
+ *copied = keep;
+ *length = keep;
+ if (keep)
+ return false;
+
+ *ret = (skb->len != orig_len) ? 0 : -EXDEV;
+ return true;
+}
+
+static int __zerocopy_fill_skb_from_iter(struct sk_buff *skb, struct iov_iter *from,
+ size_t length, const struct sock *sk)
{
int frag = skb_shinfo(skb)->nr_frags;
+ unsigned int orig_len = skb->len;
if (!skb_frags_readable(skb))
return -EFAULT;
@@ -643,7 +708,7 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
while (length && iov_iter_count(from)) {
struct page *head, *last_head = NULL;
struct page *pages[MAX_SKB_FRAGS];
- int refs, order, n = 0;
+ int refs, order, n = 0, ret;
size_t start;
ssize_t copied;
@@ -654,6 +719,9 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
MAX_SKB_FRAGS - frag, &start);
if (copied < 0)
return -EFAULT;
+ if (pisa_zc_iter(&ret, skb, from, &length, sk, pages,
+ MAX_SKB_FRAGS - frag, orig_len, &start, &copied))
+ return ret;
length -= copied;
@@ -703,6 +771,12 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
return 0;
}
+int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
+ struct iov_iter *from, size_t length)
+{
+ return __zerocopy_fill_skb_from_iter(skb, from, length, NULL);
+}
+
static int
zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from,
int length,
@@ -760,7 +834,7 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk,
else if (binding)
ret = zerocopy_fill_skb_from_devmem(skb, from, length, binding);
else
- ret = zerocopy_fill_skb_from_iter(skb, from, length);
+ ret = __zerocopy_fill_skb_from_iter(skb, from, length, sk);
truesize = skb->truesize - orig_size;
if (sk && sk->sk_type == SOCK_STREAM) {
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 04ae3bb4bd905..739d09615638c 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1935,7 +1935,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
err = __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len,
binding);
- if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {
+ if (err == -EFAULT ||
+ ((err == -EMSGSIZE || err == -EXDEV) && skb->len == orig_len)) {
struct sock *save_sk = skb->sk;
/* Streams do not free skb on error. Reset to prev state. */
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 038063717951c..9214d6213cae5 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1128,6 +1128,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
int process_backlog = 0;
int sockc_err = 0;
int zc = 0;
+ bool must_copy = false; /* copy the next chunk of a MSG_ZEROCOPY send */
long timeo;
flags = msg->msg_flags;
@@ -1289,7 +1290,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
if (copy > msg_data_left(msg))
copy = msg_data_left(msg);
- if (zc == 0) {
+ if (zc == 0 || unlikely(must_copy)) {
const struct dst_entry *dst = __sk_dst_get(sk);
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
@@ -1335,6 +1336,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
page_ref_inc(pfrag->page);
}
pfrag->offset += copy;
+ must_copy = false;
} else if (zc == MSG_ZEROCOPY) {
/* First append to a fragless skb builds initial
* pure zerocopy skb
@@ -1354,6 +1356,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
tcp_mark_push(tp, skb);
goto new_segment;
}
+ /*
+ * The next user page is not in PISA memory: copy this
+ * chunk, without reporting SO_EE_CODE_ZEROCOPY_COPIED.
+ */
+ if (unlikely(err == -EXDEV)) {
+ must_copy = true;
+ continue;
+ }
if (err < 0)
goto do_error;
copy = err;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (43 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
` (5 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When a PISA allocator holds many partially used 2MB pages with low
overall utilization, callers that recycle blocks in place (page_pool,
per-task TX page_frags, and per-CPU block caches) can keep old 2MB pages
pinned indefinitely with only a few blocks in use.
Record @alloc_jiffies in struct pisa_meta when a 2MB page is bound to an
allocator or taken from the allocator's idle list, and add two module
parameters:
- pisa.wasted_pct (1..99%, 0 disables): wasted block percentage
threshold above which compaction kicks in.
- pisa.old_ms (1..100000 ms, 0 disables): age threshold after which a
partially used 2MB page is eligible to stop recycling.
When both parameters are set and an allocator's wasted block fraction
exceeds wasted_pct, pisa_page_can_recycle() returns false for partial
2MB pages older than old_ms (except the head page actively being
filled).
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 48 ++++++++++++++++++++++++++++++++------
drivers/iommu/pisa-priv.h | 2 ++
2 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 1930135880d2b..5054a96f0f4e9 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -74,6 +74,14 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
+/* Stop recycling partial pages older than @pisa_old_ms when wasted blocks > @pisa_wasted_pct%. */
+#define PISA_OLD_MS_MAX 100000U
+static unsigned int pisa_wasted_pct __read_mostly;
+module_param_named(wasted_pct, pisa_wasted_pct, uint, 0644);
+
+static unsigned int pisa_old_ms __read_mostly;
+module_param_named(old_ms, pisa_old_ms, uint, 0644);
+
/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
unsigned int pisa_scrub_max __read_mostly;
#define PISA_SCRUB_MAX_LIMIT 65536U
@@ -399,6 +407,7 @@ static struct pisa_meta *pisa_pick_dirty_meta(struct pisa_alloc *pa)
meta = list_last_entry(&pa->idle, struct pisa_meta, list);
if (meta->nr_dirty) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
return meta;
}
}
@@ -603,7 +612,7 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
* pisa_page_can_recycle - Check if a block can stay cached in a caller pool
* @page: Block head page
*
- * Called on the recycle fast path (per-CPU cache, TX page_frag, page_pool).
+ * Called on the recycle fast path (per-CPU cache, TX page_frag).
* Fast-path cost is a static-key branch plus a READ_ONCE() of module parameters
* (~1-2 ns when compaction is off); when enabled, it does a lockless RCU +
* metadata array lookup (~5-10 ns).
@@ -613,10 +622,11 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
*/
bool pisa_page_can_recycle(const struct page *page)
{
- unsigned long bpp, in_use, free_blocks, pfn;
+ unsigned int compact, wasted_pct, old_ms;
+ unsigned long bpp, in_use, free_blocks;
const struct pisa_meta *meta;
const struct pisa_alloc *pa;
- unsigned int compact;
+ unsigned long pfn;
bool ret = true;
u8 on_list;
long held;
@@ -625,7 +635,9 @@ bool pisa_page_can_recycle(const struct page *page)
return true;
compact = READ_ONCE(pisa_compact);
- if (!compact)
+ wasted_pct = READ_ONCE(pisa_wasted_pct);
+ old_ms = READ_ONCE(pisa_old_ms);
+ if (!compact && (!wasted_pct || !old_ms))
return true;
pfn = page_to_pfn(page);
@@ -638,6 +650,13 @@ bool pisa_page_can_recycle(const struct page *page)
if (on_list == PISA_LIST_FULL || on_list == PISA_LIST_NONE)
goto out;
+ if (!compact) {
+ unsigned long max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ if (!time_after(jiffies, READ_ONCE(meta->alloc_jiffies) + max_age))
+ goto out;
+ }
+
pa = meta->pa;
if (!pa)
goto out;
@@ -657,12 +676,21 @@ bool pisa_page_can_recycle(const struct page *page)
free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
(unsigned long)held * bpp - in_use : 0;
- ret = free_blocks <= (unsigned long)compact * bpp;
+ if (compact && free_blocks > (unsigned long)compact * bpp) {
+ ret = false;
+ goto out;
+ }
+ if (wasted_pct && old_ms && held > 0 &&
+ free_blocks * 100 > (unsigned long)held * bpp * wasted_pct) {
+ unsigned long max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ ret = !time_after(jiffies, READ_ONCE(meta->alloc_jiffies) + max_age);
+ }
out:
rcu_read_unlock();
return ret;
}
-EXPORT_SYMBOL_GPL(pisa_page_can_recycle);
+EXPORT_SYMBOL(pisa_page_can_recycle);
/* Return @n freed blocks to @pa under a single acquisition of @pa->lock. */
static void pisa_free_bulk(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
@@ -1195,6 +1223,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
if (meta && meta->nr_clean) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(pa->idle_jiffies, jiffies);
return meta;
}
@@ -1218,6 +1247,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
if (!list_empty(&pa->idle)) {
meta = list_last_entry(&pa->idle, struct pisa_meta, list);
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(pa->idle_jiffies, jiffies);
return meta;
}
@@ -1287,6 +1317,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
@@ -1454,8 +1485,10 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
}
if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
- meta->on_list == PISA_LIST_IDLE_DIRTY)
+ meta->on_list == PISA_LIST_IDLE_DIRTY) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
+ }
for (i = 0; i < nblocks; i++) {
if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
@@ -1544,6 +1577,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
page = pisa_take_contig(meta, nblocks, align_mask);
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index fe1bff0bba322..4d04bb191ecea 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -74,6 +74,7 @@ enum {
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
* @rcu: RCU head used to defer release to the buddy allocator
+ * @alloc_jiffies: Timestamp (jiffies) when the page entered or re-entered use
* @clean_bitmap: Bitmap of zeroed available block indices
* @dirty_bitmap: Bitmap of dirty available block indices
*/
@@ -94,6 +95,7 @@ struct pisa_meta {
struct llist_node llnode;
struct rcu_head rcu;
};
+ unsigned long alloc_jiffies;
DECLARE_BITMAP(clean_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
DECLARE_BITMAP(dirty_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (44 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
` (4 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
libeth_rx_alloc() is called both when pre-filling RX fill queues in
process context (for indices 0 .. fq->count - 2, when buf->truesize is
still 0) and when refilling RX buffers from NAPI softirq.
Use GFP_KERNEL during the initial process-context ring fill so page_pool
and PISA page allocations (including set_memory_decrypted() under CoCo)
are not forced into GFP_ATOMIC context.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
include/net/libeth/rx.h | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h
index 0e736846c5e8a..de1e5fe44aa40 100644
--- a/include/net/libeth/rx.h
+++ b/include/net/libeth/rx.h
@@ -106,10 +106,17 @@ void libeth_rx_fq_destroy(struct libeth_fq *fq);
static inline dma_addr_t libeth_rx_alloc(const struct libeth_fq_fp *fq, u32 i)
{
struct libeth_fqe *buf = &fq->fqes[i];
+ /* fq->fqes is zero-allocated and pre-filled in process context for
+ * indices [0 .. fq->count - 2]; slot fq->count - 1 is first allocated
+ * in NAPI. Use GFP_KERNEL for initial pre-fill and GFP_ATOMIC after.
+ */
+ gfp_t gfp = (buf->truesize || i == fq->count - 1) ?
+ GFP_ATOMIC : GFP_KERNEL;
buf->truesize = fq->truesize;
- buf->netmem = page_pool_dev_alloc_netmem(fq->pp, &buf->offset,
- &buf->truesize);
+ buf->netmem = page_pool_alloc_netmem(fq->pp, &buf->offset,
+ &buf->truesize,
+ gfp | __GFP_NOWARN);
if (unlikely(!buf->netmem))
return DMA_MAPPING_ERROR;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 21/25] net/gve: Use GFP_KERNEL when pre-filling RX rings
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (45 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
` (3 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When pre-filling DQO and GQI RX rings in process context before NAPI is
enabled (e.g. gve_open), use GFP_KERNEL instead of GFP_ATOMIC so page
allocation (and CoCo page decryption) can sleep instead of dipping into
atomic reserves.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c | 7 ++++++-
drivers/net/ethernet/google/gve/gve_rx.c | 8 ++++++--
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
index 6880d1531ba5e..3b13aa4999ae5 100644
--- a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
@@ -223,13 +223,18 @@ void gve_free_to_page_pool(struct gve_rx_ring *rx,
static int gve_alloc_from_page_pool(struct gve_rx_ring *rx,
struct gve_rx_buf_state_dqo *buf_state)
{
+ /*
+ * Use GFP_KERNEL when pre-filling RX rings in process context (e.g.
+ * gve_open) so PISA metadata init and CoCo page decryption can sleep.
+ */
+ gfp_t gfp = gve_get_napi_enabled(rx->gve) ? GFP_ATOMIC : GFP_KERNEL;
netmem_ref netmem;
buf_state->page_info.buf_size = rx->packet_buffer_truesize;
netmem = page_pool_alloc_netmem(rx->dqo.page_pool,
&buf_state->page_info.page_offset,
&buf_state->page_info.buf_size,
- GFP_ATOMIC);
+ gfp);
if (!netmem)
return -ENOMEM;
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 81ea800e66e94..89f287b678bf0 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -161,12 +161,16 @@ static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
union gve_rx_data_slot *data_slot,
struct gve_rx_ring *rx)
{
+ /*
+ * Use GFP_KERNEL when pre-filling RX rings in process context (e.g.
+ * gve_open) so PISA metadata init and CoCo page decryption can sleep.
+ */
+ gfp_t gfp = gve_get_napi_enabled(priv) ? GFP_ATOMIC : GFP_KERNEL;
struct page *page;
dma_addr_t dma;
int err;
- err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE,
- GFP_ATOMIC);
+ err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
if (err) {
u64_stats_update_begin(&rx->statss);
rx->rx_buf_alloc_fail++;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (46 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
` (2 subsequent siblings)
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When dev->pisa_rx_bufs is enabled, allocate GQI-RDA RX packet buffers
with pisa_rx_gfp(dev) so they come from the per-node PISA
allocator, and evict plain pages when a free PISA RX block is available;
DQO-RDA uses page_pool, which checks dev->pisa_rx_bufs directly.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve_rx.c | 32 ++++++++++++++++--------
1 file changed, 22 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 89f287b678bf0..85c243bbdeb33 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -4,6 +4,7 @@
* Copyright (C) 2015-2021 Google, Inc.
*/
+#include <linux/pisa.h>
#include "gve.h"
#include "gve_adminq.h"
#include "gve_utils.h"
@@ -170,6 +171,8 @@ static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
dma_addr_t dma;
int err;
+ if (rx->data.raw_addressing)
+ gfp |= pisa_rx_gfp(dev);
err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
if (err) {
u64_stats_update_begin(&rx->statss);
@@ -499,13 +502,19 @@ static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *sl
*(slot_addr) ^= offset;
}
-static int gve_rx_can_recycle_buffer(struct gve_rx_slot_page_info *page_info)
+static int gve_rx_can_recycle_buffer(struct gve_rx_ring *rx,
+ struct gve_rx_slot_page_info *page_info)
{
int pagecount = page_count(page_info->page);
/* This page is not being used by any SKBs - reuse */
- if (pagecount == page_info->pagecnt_bias)
+ if (pagecount == page_info->pagecnt_bias) {
+ if (rx && rx->data.raw_addressing &&
+ !is_pisa_page(page_info->page) &&
+ pisa_has_free_blocks_rx(&rx->gve->pdev->dev, 0, rx->gve->numa_node))
+ return 0;
return 1;
+ }
/* This page is still being used by an SKB - we can't reuse */
else if (pagecount > page_info->pagecnt_bias)
return 0;
@@ -549,7 +558,7 @@ static struct sk_buff *gve_rx_copy_to_pool(struct gve_rx_ring *rx,
copy_page_info = &rx->qpl_copy_pool[pool_idx];
if (!copy_page_info->can_flip) {
- int recycle = gve_rx_can_recycle_buffer(copy_page_info);
+ int recycle = gve_rx_can_recycle_buffer(NULL, copy_page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(rx->gve);
@@ -669,7 +678,7 @@ static struct sk_buff *gve_rx_skb(struct gve_priv *priv, struct gve_rx_ring *rx,
u64_stats_update_end(&rx->statss);
}
} else {
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(priv);
@@ -975,7 +984,7 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
* owns half the page it is impossible to tell which half. Either
* the whole page is free or it needs to be replaced.
*/
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (recycle < 0) {
if (!rx->data.raw_addressing)
@@ -987,12 +996,15 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
union gve_rx_data_slot *data_slot =
&rx->data.data_ring[idx];
struct device *dev = &priv->pdev->dev;
- gve_rx_free_buffer(dev, page_info, data_slot);
- page_info->page = NULL;
- if (gve_rx_alloc_buffer(priv, dev, page_info,
- data_slot, rx)) {
+ struct gve_rx_slot_page_info new_info;
+ union gve_rx_data_slot new_slot;
+
+ if (gve_rx_alloc_buffer(priv, dev, &new_info,
+ &new_slot, rx))
break;
- }
+ gve_rx_free_buffer(dev, page_info, data_slot);
+ *page_info = new_info;
+ *data_slot = new_slot;
}
}
fill_cnt++;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (47 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Extend PISA user/page-cache allocators to support O_DIRECT and buffered
disk I/O, gated per block/controller device through pisa_rx_bufs (read)
and pisa_tx_bufs (write):
- Split per-node user_pa into DMA_BIDIRECTIONAL, DMA_TO_DEVICE, and
DMA_FROM_DEVICE allocators, promoting pages to DMA_BIDIRECTIONAL if
later accessed in the opposite direction.
- Upgrade pinned O_DIRECT user pages in bio_iov_iter_get_pages(), and
allocate/promote order-0 page-cache folios for buffered I/O when PISA
is enabled on the target block device.
- Bypass IOVA allocation in dma_iova_try_alloc() for PISA pages so
blk-mq uses the pre-mapped 2MB window.
- Add pisa.scrub_migrate to compact or demote lingering folios off old,
sparse user_pa 2MB pages.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
block/bio.c | 46 ++++
drivers/iommu/dma-iommu.c | 2 +-
drivers/iommu/pisa-alloc.c | 36 ++--
drivers/iommu/pisa-node.c | 3 +-
drivers/iommu/pisa-priv.h | 21 +-
drivers/iommu/pisa-user.c | 417 ++++++++++++++++++++++++++++++++-----
include/linux/pisa.h | 100 ++++++++-
mm/filemap.c | 15 +-
mm/page-writeback.c | 8 +
mm/readahead.c | 5 +-
net/core/datagram.c | 3 +-
11 files changed, 581 insertions(+), 75 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index b0fcc69de6535..5b43430b5280c 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -6,6 +6,7 @@
#include <linux/swap.h>
#include <linux/bio-integrity.h>
#include <linux/blkdev.h>
+#include <linux/pisa.h>
#include <linux/uio.h>
#include <linux/iocontext.h>
#include <linux/slab.h>
@@ -1316,6 +1317,43 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
}
#endif
+/*
+ * Migrate pinned order-0 user pages in @bio into PISA memory when enabled on
+ * @bio->bi_bdev; returns true if pages were released and the caller must retry.
+ */
+static bool bio_upgrade_pages_to_pisa(struct bio *bio)
+{
+ enum dma_data_direction dir;
+ struct page *pages[64];
+ unsigned int i, nr = 0;
+
+ if (!IS_ENABLED(CONFIG_PISA) || !bio_flagged(bio, BIO_PAGE_PINNED))
+ return false;
+ dir = op_is_write(bio_op(bio)) ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+ if (!is_bdev_pisa_enabled(bio->bi_bdev, dir))
+ return false;
+
+ for (i = 0; i < bio->bi_vcnt && nr < ARRAY_SIZE(pages); i++) {
+ struct page *page = bio->bi_io_vec[i].bv_page;
+
+ if (bvec_nr_pages(&bio->bi_io_vec[i]) == 1 && !is_pisa_page(page)) {
+ get_page(page);
+ pages[nr++] = page;
+ }
+ }
+ if (!nr)
+ return false;
+
+ bio_release_pages(bio, false);
+ bio->bi_vcnt = 0;
+ bio->bi_iter.bi_size = 0;
+ if (!pisa_upgrade_user_pages(pages, nr, dir, false)) {
+ while (nr)
+ put_page(pages[--nr]);
+ }
+ return true;
+}
+
/**
* bio_iov_iter_get_pages - add user or kernel pages to a bio
* @bio: bio to add pages to
@@ -1346,6 +1384,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
{
iov_iter_extraction_t flags = 0;
struct iov_iter saved_iter;
+ bool tried_pisa = false;
int ret;
if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
@@ -1367,6 +1406,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue))
flags |= ITER_ALLOW_P2PDMA;
+retry:
saved_iter = *iter;
do {
ssize_t size;
@@ -1386,6 +1426,12 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
bio->bi_iter.bi_size += size;
} while (iov_iter_count(iter) && !bio_full(bio, 0));
+ if (!tried_pisa && bio_upgrade_pages_to_pisa(bio)) {
+ tried_pisa = true;
+ *iter = saved_iter;
+ goto retry;
+ }
+
if (is_pci_p2pdma_page(bio->bi_io_vec->bv_page))
bio->bi_opf |= REQ_NOMERGE;
ret = bio_iov_iter_align_down(bio, iter, len_align_mask);
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index a0c5ddab7a8b6..acedfe04bde26 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1868,7 +1868,7 @@ bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state,
dma_addr_t addr;
memset(state, 0, sizeof(*state));
- if (!use_dma_iommu(dev))
+ if (!use_dma_iommu(dev) || is_pisa_phys(phys))
return false;
domain = iommu_get_dma_domain(dev);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 5054a96f0f4e9..cedb1a62d20e2 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -75,11 +75,10 @@ static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
/* Stop recycling partial pages older than @pisa_old_ms when wasted blocks > @pisa_wasted_pct%. */
-#define PISA_OLD_MS_MAX 100000U
-static unsigned int pisa_wasted_pct __read_mostly;
+unsigned int pisa_wasted_pct __read_mostly;
module_param_named(wasted_pct, pisa_wasted_pct, uint, 0644);
-static unsigned int pisa_old_ms __read_mostly;
+unsigned int pisa_old_ms __read_mostly;
module_param_named(old_ms, pisa_old_ms, uint, 0644);
/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
@@ -98,8 +97,6 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
},
};
-static void pisa_schedule_reclaim(void);
-
static int pisa_scrub_set(const char *val, const struct kernel_param *kp)
{
int ret = param_set_uint(val, kp);
@@ -229,18 +226,6 @@ void pisa_release_page(struct pisa_meta *meta)
call_rcu(&meta->rcu, pisa_release_page_rcu);
}
-/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
-enum pisa_meta_list {
- PISA_LIST_NONE,
- PISA_LIST_FULL,
- PISA_LIST_IDLE_CLEAN, /* head of @idle */
- PISA_LIST_IDLE_DIRTY, /* tail of @idle */
- PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
- PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
- PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
- PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
-};
-
static void pisa_reclaim_work_fn(struct work_struct *work);
/* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
@@ -285,7 +270,7 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
}
-static void pisa_schedule_reclaim(void)
+void pisa_schedule_reclaim(void)
{
struct workqueue_struct *wq = pisa_wq ?: system_wq;
unsigned int idle_ms = READ_ONCE(pisa_idle_ms);
@@ -301,7 +286,7 @@ static void pisa_schedule_reclaim(void)
* become available (avail == bpp), either queues @meta to @pisa_free_list (and
* returns true so the caller schedules reclaim) or moves it to @pa->idle.
*/
-static bool pisa_place_meta(struct pisa_meta *meta)
+bool pisa_place_meta(struct pisa_meta *meta)
{
struct pisa_alloc *pa = meta->pa;
unsigned int avail = pisa_meta_avail(meta), bpp = PISA_BLOCKS(pa->order);
@@ -505,6 +490,9 @@ bool pisa_scrub_node(int nid)
struct pisa_alloc *pa;
bool more = false;
+ if (!budget)
+ goto migrate;
+
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
if (pa->nid != nid || !READ_ONCE(pa->scrub_demand))
@@ -526,6 +514,16 @@ bool pisa_scrub_node(int nid)
}
read_unlock(&pisa_allocs_lock);
+migrate:
+ if (READ_ONCE(pisa_scrub_migrate)) {
+ unsigned long rem = !budget ? 512 : (budget > done ? budget - done : 0);
+
+ if (rem && local_clock() < deadline)
+ more |= pisa_user_scrub_node(nid, rem, deadline);
+ else
+ more = true;
+ }
+
return more;
}
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index cd29c146c281e..7469674133620 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -184,7 +184,8 @@ void pisa_schedule_scrub(int nid)
{
struct pisa_node *node;
- if (!READ_ONCE(pisa_scrub_max) || nid == NUMA_NO_NODE)
+ if ((!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate)) ||
+ nid == NUMA_NO_NODE)
return;
node = &pisa_nodes[nid];
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 4d04bb191ecea..1ecfebbaeb381 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -61,6 +61,18 @@ enum {
* - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
+/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
+enum pisa_meta_list {
+ PISA_LIST_NONE,
+ PISA_LIST_FULL,
+ PISA_LIST_IDLE_CLEAN, /* head of @idle */
+ PISA_LIST_IDLE_DIRTY, /* tail of @idle */
+ PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
+ PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
+ PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
+ PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
+};
+
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED
@@ -187,7 +199,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
- struct pisa_alloc *user_pa; /* user page migration pa */
+ struct pisa_alloc *user_pa[3]; /* user/page-cache allocators by dir */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -243,6 +255,8 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
bool pisa_want_decrypt(void);
+bool pisa_place_meta(struct pisa_meta *meta);
+void pisa_schedule_reclaim(void);
int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
int pisa_page_unprepare(struct pisa_meta *meta);
@@ -266,11 +280,16 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
return page;
}
+#define PISA_OLD_MS_MAX 100000U
+extern unsigned int pisa_wasted_pct;
+extern unsigned int pisa_old_ms;
+extern unsigned int pisa_scrub_migrate;
unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
unsigned int *nr, struct list_head *release_list,
unsigned long nr_to_scan);
extern unsigned int pisa_scrub_max;
bool pisa_scrub_node(int nid);
+bool pisa_user_scrub_node(int nid, unsigned long budget, u64 deadline);
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
index a16909d429aca..7dca3bf066f0d 100644
--- a/drivers/iommu/pisa-user.c
+++ b/drivers/iommu/pisa-user.c
@@ -4,15 +4,23 @@
* per-node PISA allocators so zero-copy TX and block I/O use 2MB IOMMU mappings.
* Disabled on CoCo guests (user PTEs map pages encrypted).
*/
+#include <linux/blkdev.h>
#include <linux/cc_platform.h>
+#include <linux/device.h>
#include <linux/pisa.h>
+#include <linux/fs.h>
#include <linux/gfp.h>
+#include <linux/iommu.h>
+#include <linux/jiffies.h>
#include <linux/migrate.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/nodemask.h>
+#include <linux/pagemap.h>
#include <linux/pagewalk.h>
+#include <linux/folio_batch.h>
+#include <linux/sched/clock.h>
#include <linux/swap.h>
#include "pisa-priv.h"
@@ -55,6 +63,27 @@ static const struct kernel_param_ops pisa_user_max_ops = {
};
module_param_cb(user_max_4K_pages, &pisa_user_max_ops, &pisa_user_max_4k_pages, 0644);
+/*
+ * Scrub-time migration of old sparse user_pa PISA pages when waste exceeds
+ * pisa.wasted_pct (0 = off, 1 = compact in allocator, 2 = demote to the buddy allocator).
+ */
+unsigned int pisa_scrub_migrate __read_mostly;
+
+static int pisa_scrub_migrate_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (!ret && pisa_scrub_migrate && static_branch_unlikely(&pisa_key))
+ pisa_schedule_scrub_all();
+ return ret;
+}
+
+static const struct kernel_param_ops pisa_scrub_migrate_ops = {
+ .set = pisa_scrub_migrate_set,
+ .get = param_get_uint,
+};
+module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 0644);
+
/* True once per-node user allocators have been allocated, serialized by pisa_user_lock. */
static bool pisa_user_ready __read_mostly;
static DEFINE_MUTEX(pisa_user_lock);
@@ -62,6 +91,7 @@ static DEFINE_MUTEX(pisa_user_lock);
/* Lazily create per-node user allocators on first use (process context). */
static bool pisa_user_init(void)
{
+ static const char * const dir_names[3] = { "bi", "tx", "rx" };
int nid;
if (!READ_ONCE(pisa_user_max_4k_pages) || cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
@@ -73,10 +103,20 @@ static bool pisa_user_init(void)
return false;
mutex_lock(&pisa_user_lock);
for_each_node_state(nid, N_MEMORY) {
- if (!pisa_nodes[nid].user_pa)
- pisa_nodes[nid].user_pa = pisa_create(nid, 0, DMA_TO_DEVICE);
- if (!pisa_nodes[nid].user_pa)
- goto out; /* retried on the next call */
+ unsigned int dir;
+
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+
+ if (!pa) {
+ pa = pisa_create(nid, 0, dir);
+ if (!pa)
+ goto out; /* retried on the next call */
+ pisa_set_name(pa, "user_%s_n%d", dir_names[dir], nid);
+ pisa_enable_pcp(pa);
+ pisa_nodes[nid].user_pa[dir] = pa;
+ }
+ }
}
/* Publish pisa_nodes[].user_pa to lockless readers. */
smp_store_release(&pisa_user_ready, true);
@@ -102,9 +142,13 @@ static unsigned long pisa_user_in_use(void)
int nid;
for_each_node_state(nid, N_MEMORY) {
- const struct pisa_alloc *pa = pisa_nodes[nid].user_pa;
+ unsigned int dir;
- used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ const struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+
+ used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ }
}
return used;
}
@@ -115,14 +159,43 @@ static long pisa_user_budget(void)
return (long)(READ_ONCE(pisa_user_max_4k_pages) - pisa_user_in_use());
}
-/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
-static bool pisa_user_want(struct folio *folio, bool to_pisa)
+/*
+ * Return true if @folio should be migrated into (@to_pisa) or out of PISA
+ * memory, promoting existing PISA folios to DMA_BIDIRECTIONAL when needed.
+ */
+static bool pisa_user_want_dir(struct folio *folio, bool to_pisa,
+ enum dma_data_direction dir, bool allow_file)
{
- if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
- folio_test_ksm(folio) || folio_test_large(folio) ||
- folio_maybe_mapped_shared(folio))
+ unsigned long pfn = folio_pfn(folio);
+
+ if (is_zero_folio(folio) || folio_is_zone_device(folio) ||
+ folio_test_ksm(folio) || folio_test_large(folio))
return false;
- return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+ if (folio_test_anon(folio)) {
+ if (folio_maybe_mapped_shared(folio))
+ return false;
+ } else if (!allow_file || !folio_mapping(folio) ||
+ folio_test_locked(folio) || folio_test_writeback(folio)) {
+ return false;
+ }
+
+ if (!to_pisa)
+ return is_pisa_pfn(pfn);
+ if (!is_pisa_pfn(pfn))
+ return true;
+
+ if (dir == DMA_TO_DEVICE || dir == DMA_FROM_DEVICE) {
+ int need = (dir == DMA_FROM_DEVICE) ? IOMMU_WRITE : IOMMU_READ;
+ const struct pisa_meta *meta;
+ bool promote;
+
+ rcu_read_lock();
+ meta = is_pisa_pfn(pfn) ? pisa_meta_of_pfn(pfn) : NULL;
+ promote = meta && (meta->prot & need) != need;
+ rcu_read_unlock();
+ return promote;
+ }
+ return false;
}
/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
@@ -132,20 +205,26 @@ static bool pisa_user_busy(struct folio *folio, int extra)
}
struct pisa_user_mtc {
- gfp_t gfp;
- long budget;
+ gfp_t gfp;
+ long budget;
+ enum dma_data_direction dir;
+ bool same_alloc;
};
/* new_folio_t for migration into PISA memory. */
static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
{
struct pisa_user_mtc *mtc = (void *)private;
+ bool already_pisa = is_pisa_pfn(folio_pfn(src));
+ enum dma_data_direction dir = mtc->dir;
int nid = folio_nid(src);
struct page *page;
- if (mtc->budget <= 0)
+ if (!already_pisa && mtc->budget <= 0)
return NULL;
- page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+ if (already_pisa && !mtc->same_alloc)
+ dir = DMA_BIDIRECTIONAL;
+ page = pisa_alloc(pisa_nodes[nid].user_pa[dir], mtc->gfp);
if (!page)
return NULL;
if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
@@ -153,7 +232,8 @@ static struct folio *pisa_user_new_folio(struct folio *src, unsigned long privat
mtc->budget = 0;
return NULL;
}
- mtc->budget--;
+ if (!already_pisa)
+ mtc->budget--;
return page_folio(page);
}
@@ -164,78 +244,319 @@ static struct folio *pisa_user_old_folio(struct folio *src, unsigned long privat
}
/* Migrate the isolated folios on @list; failures are put back. */
-static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
+ enum dma_data_direction dir,
+ bool same_alloc, gfp_t gfp)
{
- struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+ struct pisa_user_mtc mtc = {
+ .gfp = gfp,
+ .budget = LONG_MAX,
+ .dir = dir,
+ .same_alloc = same_alloc,
+ };
enum migrate_mode mode = MIGRATE_SYNC;
+ unsigned int done = 0;
if (list_empty(list))
- return;
+ return 0;
if (to_pisa) {
mtc.budget = pisa_user_budget();
if (!gfpflags_allow_blocking(gfp))
mode = MIGRATE_ASYNC;
}
if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
- NULL, (unsigned long)&mtc, mode, MR_SYSCALL, NULL))
+ NULL, (unsigned long)&mtc, mode, MR_SYSCALL, &done))
putback_movable_pages(list);
+ return done;
}
/**
- * pisa_upgrade_user_pages - Try to move user pages about to be sent into PISA
- * @pages: Pages from iov_iter_get_pages2(), one reference each
+ * pisa_upgrade_user_pages - Try to move user pages into PISA for @dir
+ * @pages: Pages from iov_iter_get_pages2() or iov_iter_extract_pages()
* @nr: Number of entries in @pages
+ * @dir: I/O direction (DMA_TO_DEVICE or DMA_FROM_DEVICE)
+ * @pinned: true if @pages are FOLL_PIN-pinned, false if refcounted (+1)
*
- * Best effort, process context. Pages that cannot be moved (e.g. still
- * referenced by an in-flight skb, or in a large folio) are skipped; for TCP
- * with pisa_tx_bufs enabled the caller copies them instead.
- *
- * Return: false if nothing was attempted and the references are untouched;
- * true if all references were dropped and the caller must look the pages up
- * again.
+ * Return: true if references/pins were dropped and the caller must re-lookup.
*/
-bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir, bool pinned)
{
+ int extra = pinned ? GUP_PIN_COUNTING_BIAS : 1;
bool drained = false;
LIST_HEAD(list);
- unsigned int i, want = 0;
+ unsigned int i, want = 0, need_budget = 0;
if (!pisa_user_init())
return false;
- /* Common case: all pages already in PISA, no budget check needed. */
- for (i = 0; i < nr; i++)
- want += pisa_user_want(page_folio(pages[i]), true);
+ /* Common case: all pages already in PISA with matching prot. */
+ for (i = 0; i < nr; i++) {
+ struct folio *folio = page_folio(pages[i]);
+
+ if (pisa_user_want_dir(folio, true, dir, false)) {
+ want++;
+ need_budget += !is_pisa_pfn(folio_pfn(folio));
+ }
+ }
if (!want)
return false;
- if (pisa_user_budget() <= 0)
+ if (need_budget && pisa_user_budget() <= 0)
return false;
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
- if (!pisa_user_want(folio, true))
+ if (!pisa_user_want_dir(folio, true, dir, false))
continue;
/* A freshly faulted page may still sit in a per-CPU LRU batch. */
if (!drained && !folio_test_lru(folio)) {
lru_add_drain();
drained = true;
}
- /* The reference from iov_iter_get_pages2() is expected. */
- if (pisa_user_busy(folio, 1))
+ if (pisa_user_busy(folio, extra))
continue;
isolate_folio_to_list(folio, &list);
}
if (list_empty(&list))
return false;
- /* Migration fails on pages with references it does not expect. */
- for (i = 0; i < nr; i++)
- put_page(pages[i]);
- pisa_user_migrate(&list, true,
- GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ /* Migration fails on pages with references or pins it does not expect. */
+ if (pinned) {
+ unpin_user_pages(pages, nr);
+ } else {
+ for (i = 0; i < nr; i++)
+ put_page(pages[i]);
+ }
+ pisa_user_migrate_dir(&list, true, dir, false,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ return true;
+}
+
+/**
+ * __pisa_filemap_alloc - Allocate an order-0 page-cache folio from a PISA user allocator
+ * @mapping: Page-cache address_space
+ * @gfp: GFP flags
+ * @is_write: True when allocating on the buffered write path
+ *
+ * Return: Newly allocated order-0 folio, or NULL on failure.
+ */
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+ gfp_t gfp, bool is_write)
+{
+ enum dma_data_direction dir = is_write ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+ struct page *page;
+ int nid;
+
+ if (!__pisa_mapping_enabled(mapping, dir))
+ return NULL;
+ if (is_write && __pisa_mapping_enabled(mapping, DMA_FROM_DEVICE))
+ dir = DMA_BIDIRECTIONAL;
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!smp_load_acquire(&pisa_user_ready) &&
+ ((gfp & GFP_KERNEL) != GFP_KERNEL || !pisa_user_init()))
+ return NULL;
+ if (pisa_user_budget() <= 0)
+ return NULL;
+ nid = pisa_sanitize_nid(numa_node_id());
+ page = pisa_alloc(pisa_nodes[nid].user_pa[dir], gfp);
+ if (!page)
+ return NULL;
+ if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
+ put_page(page);
+ return NULL;
+ }
+ return page_folio(page);
+}
+
+/**
+ * pisa_folio_upgrade - Migrate an unlocked order-0 folio to a PISA allocator page
+ * @folio: Folio to migrate (caller holds 1 ref, dropped only when returning true)
+ * @dir: Required DMA direction (DMA_TO_DEVICE or DMA_FROM_DEVICE)
+ * @fbatch: Optional folio_batch holding @folio's reference to release before migration
+ *
+ * Return: true if @folio was isolated and caller's reference was dropped
+ * (caller must re-lookup in mapping); false if @folio is untouched.
+ */
+bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
+ struct folio_batch *fbatch)
+{
+ LIST_HEAD(list);
+
+ if (!pisa_user_init() ||
+ !pisa_user_want_dir(folio, true, dir, true) ||
+ (!is_pisa_pfn(folio_pfn(folio)) && pisa_user_budget() <= 0))
+ return false;
+ if (!folio_test_lru(folio))
+ lru_add_drain();
+ if (pisa_user_busy(folio, 1) || !isolate_folio_to_list(folio, &list))
+ return false;
+ /* Drop caller's reference so migrate_pages() sees only expected refs. */
+ if (fbatch)
+ folio_batch_release(fbatch);
+ else
+ folio_put(folio);
+ pisa_user_migrate_dir(&list, true, dir, false,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
return true;
}
+bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir)
+{
+ struct device *dev;
+
+ if (!bdev || !bdev->bd_disk)
+ return false;
+ for (dev = disk_to_dev(bdev->bd_disk); dev; dev = dev->parent) {
+ if (dir == DMA_FROM_DEVICE ?
+ READ_ONCE(dev->pisa_rx_bufs) :
+ READ_ONCE(dev->pisa_tx_bufs))
+ return true;
+ }
+ return false;
+}
+
+bool __pisa_mapping_enabled(const struct address_space *mapping, enum dma_data_direction dir)
+{
+ struct super_block *sb;
+
+ if (!mapping || !mapping->host || !mapping->host->i_sb)
+ return false;
+ sb = mapping->host->i_sb;
+ return __pisa_bdev_enabled(sb_is_blkdev_sb(sb) ?
+ I_BDEV(mapping->host) : sb->s_bdev, dir);
+}
+
#define PISA_USER_BATCH 512
+static unsigned long pisa_user_scrub_alloc(struct pisa_alloc *pa,
+ enum dma_data_direction dir,
+ unsigned int mode,
+ unsigned long budget)
+{
+ unsigned int wasted_pct = READ_ONCE(pisa_wasted_pct);
+ unsigned int old_ms = READ_ONCE(pisa_old_ms);
+ unsigned long bpp = PISA_BLOCKS(0), in_use, free_blocks;
+ unsigned long max_age, base_pfn = 0, flags;
+ DECLARE_BITMAP(inuse_map, PISA_MAX_BLOCKS);
+ struct list_head *lists[] = { &pa->partial_dirty, &pa->partial };
+ struct pisa_meta *meta, *target = NULL;
+ unsigned int i, idx, count = 0;
+ bool should_release = false;
+ unsigned long migrated = 0;
+ LIST_HEAD(list);
+ long held;
+
+ if (!wasted_pct || !old_ms)
+ return 0;
+
+ pisa_pcp_drain(pa);
+
+ held = pisa_alloc_held_pages(pa);
+ in_use = READ_ONCE(pa->blocks_in_use);
+ free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
+ (unsigned long)held * bpp - in_use : 0;
+ if (held <= 0 || free_blocks * 100 <= (unsigned long)held * bpp * wasted_pct)
+ return 0;
+
+ max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < ARRAY_SIZE(lists) && !target; i++) {
+ list_for_each_entry_reverse(meta, lists[i], list) {
+ if (mode == 1 && &meta->list == lists[i]->next)
+ break;
+ if (time_after(jiffies, meta->alloc_jiffies + max_age)) {
+ target = meta;
+ break;
+ }
+ }
+ }
+ if (target) {
+ base_pfn = pisa_meta_to_pfn(target);
+ bitmap_or(inuse_map, target->clean_bitmap, target->dirty_bitmap, bpp);
+ bitmap_complement(inuse_map, inuse_map, bpp);
+ WRITE_ONCE(target->on_list, PISA_LIST_FULL);
+ list_move(&target->list, &pa->full);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (!target)
+ return 0;
+
+ for_each_set_bit(idx, inuse_map, bpp) {
+ unsigned long pfn = base_pfn + idx;
+ struct folio *folio;
+
+ rcu_read_lock();
+ if (!is_pisa_pfn(pfn)) {
+ rcu_read_unlock();
+ continue;
+ }
+ folio = page_folio(pfn_to_page(pfn));
+ if (!folio_try_get(folio)) {
+ rcu_read_unlock();
+ continue;
+ }
+ if (!is_pisa_pfn(pfn) || folio != page_folio(pfn_to_page(pfn))) {
+ rcu_read_unlock();
+ folio_put(folio);
+ continue;
+ }
+ rcu_read_unlock();
+
+ if (!pisa_user_want_dir(folio, false, dir, true) ||
+ pisa_user_busy(folio, 1) ||
+ !isolate_folio_to_list(folio, &list)) {
+ folio_put(folio);
+ continue;
+ }
+ folio_put(folio);
+ if (++count >= min_t(unsigned long, budget, PISA_USER_BATCH))
+ break;
+ }
+
+ if (!list_empty(&list)) {
+ migrated = pisa_user_migrate_dir(&list, mode == 1, dir, true,
+ GFP_KERNEL | __GFP_NOWARN);
+ pisa_pcp_drain(pa);
+ }
+
+ spin_lock_irqsave(&pa->lock, flags);
+ if (READ_ONCE(target->pa) == pa) {
+ WRITE_ONCE(target->alloc_jiffies, jiffies);
+ should_release = pisa_place_meta(target);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return migrated;
+}
+
+bool pisa_user_scrub_node(int nid, unsigned long budget, u64 deadline)
+{
+ unsigned int mode = READ_ONCE(pisa_scrub_migrate);
+ unsigned long done = 0;
+ unsigned int dir;
+
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!mode || mode > 2 || !smp_load_acquire(&pisa_user_ready))
+ return false;
+
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+ unsigned long n;
+
+ if (!pa)
+ continue;
+ while ((n = pisa_user_scrub_alloc(pa, dir, mode, budget - done))) {
+ done += n;
+ if (done >= budget || local_clock() >= deadline)
+ return true;
+ cond_resched();
+ }
+ }
+ return false;
+}
+
/**
* pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
* @vma: VMA, with the mmap lock held for reading
@@ -262,17 +583,17 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (pisa_user_want(folio, to_pisa) && !pisa_user_busy(folio, 0) &&
- isolate_folio_to_list(folio, &list))
+ if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false) &&
+ !pisa_user_busy(folio, 0) && isolate_folio_to_list(folio, &list))
n++;
folio_walk_end(&fw, vma);
}
if (n == PISA_USER_BATCH) {
- pisa_user_migrate(&list, to_pisa, gfp);
+ pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
n = 0;
}
cond_resched();
}
- pisa_user_migrate(&list, to_pisa, gfp);
+ pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
return 0;
}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index d11fd3d663dc1..1d9b07370cc75 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -10,9 +10,13 @@
#include <linux/device.h>
#include <linux/mm.h>
#include <linux/numa.h>
+#include <linux/folio_batch.h>
#include <linux/rcupdate.h>
+struct address_space;
+struct block_device;
struct device;
+struct folio_batch;
struct iommu_domain;
#ifdef CONFIG_PISA
@@ -134,7 +138,15 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
/* User pages in PISA blocks, see drivers/iommu/pisa-user.c */
DECLARE_STATIC_KEY_FALSE(pisa_user_key);
-bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr);
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir, bool pinned);
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+ gfp_t gfp, bool is_write);
+bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
+ struct folio_batch *fbatch);
+bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir);
+bool __pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir);
int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa);
@@ -144,6 +156,49 @@ static inline bool pisa_user_enabled(void)
return static_branch_unlikely(&pisa_user_key);
}
+static inline bool is_bdev_pisa_enabled(struct block_device *bdev,
+ enum dma_data_direction dir)
+{
+ return pisa_user_enabled() && __pisa_bdev_enabled(bdev, dir);
+}
+
+static inline bool pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir)
+{
+ return pisa_user_enabled() && __pisa_mapping_enabled(mapping, dir);
+}
+
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+ struct folio *folio, bool write_wait)
+{
+ return write_wait && pisa_mapping_enabled(mapping, DMA_TO_DEVICE) &&
+ pisa_folio_upgrade(folio, DMA_TO_DEVICE, NULL);
+}
+
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+ struct folio *folio, pgoff_t *index,
+ struct folio_batch *fbatch)
+{
+ pgoff_t idx;
+
+ if (!pisa_mapping_enabled(mapping, DMA_TO_DEVICE))
+ return false;
+ idx = folio->index;
+ if (!pisa_folio_upgrade(folio, DMA_TO_DEVICE, fbatch))
+ return false;
+ *index = idx;
+ return true;
+}
+
+static inline struct folio *
+pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
+ unsigned int order, bool is_write)
+{
+ if (order || !pisa_user_enabled())
+ return NULL;
+ return __pisa_filemap_alloc(mapping, gfp, is_write);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -230,11 +285,52 @@ static inline bool pisa_user_enabled(void)
return false;
}
-static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir,
+ bool pinned)
{
return false;
}
+static inline bool pisa_folio_upgrade(struct folio *folio,
+ enum dma_data_direction dir,
+ struct folio_batch *fbatch)
+{
+ return false;
+}
+
+static inline bool is_bdev_pisa_enabled(struct block_device *bdev,
+ enum dma_data_direction dir)
+{
+ return false;
+}
+
+static inline bool pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir)
+{
+ return false;
+}
+
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+ struct folio *folio, bool write_wait)
+{
+ return false;
+}
+
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+ struct folio *folio, pgoff_t *index,
+ struct folio_batch *fbatch)
+{
+ return false;
+}
+
+static inline struct folio *
+pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
+ unsigned int order, bool is_write)
+{
+ return NULL;
+}
+
static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa)
{
diff --git a/mm/filemap.c b/mm/filemap.c
index b04be64a5069f..3860b0c6db4ec 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -15,6 +15,7 @@
#include <linux/dax.h>
#include <linux/fs.h>
#include <linux/sched/signal.h>
+#include <linux/pisa.h>
#include <linux/uaccess.h>
#include <linux/capability.h>
#include <linux/kernel_stat.h>
@@ -1976,6 +1977,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
pgoff_t index, fgf_t fgp_flags, gfp_t gfp, struct mempolicy *policy)
{
+ bool tried_pisa = false;
struct folio *folio;
repeat:
@@ -1985,6 +1987,13 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
if (!folio)
goto no_page;
+ if (!tried_pisa) {
+ tried_pisa = true;
+ if (pisa_filemap_upgrade(mapping, folio,
+ (fgp_flags & (FGP_WRITE | FGP_NOWAIT)) == FGP_WRITE))
+ goto repeat;
+ }
+
if (fgp_flags & FGP_LOCK) {
if (fgp_flags & FGP_NOWAIT) {
if (!folio_trylock(folio)) {
@@ -2044,7 +2053,11 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
err = -ENOMEM;
if (order > min_order)
alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
- folio = filemap_alloc_folio(alloc_gfp, order, policy);
+ folio = pisa_filemap_alloc(mapping, alloc_gfp, order,
+ fgp_flags & FGP_WRITE);
+ if (!folio)
+ folio = filemap_alloc_folio(alloc_gfp, order,
+ policy);
if (!folio)
continue;
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 0694ec4098f6d..39cc28f9762c6 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -37,6 +37,7 @@
#include <linux/sched/rt.h>
#include <linux/sched/signal.h>
#include <linux/mm_inline.h>
+#include <linux/pisa.h>
#include <linux/shmem_fs.h>
#include <trace/events/writeback.h>
@@ -2409,6 +2410,7 @@ static pgoff_t wbc_end(struct writeback_control *wbc)
static struct folio *writeback_get_folio(struct address_space *mapping,
struct writeback_control *wbc)
{
+ pgoff_t upgraded = ULONG_MAX;
struct folio *folio;
retry:
@@ -2423,6 +2425,12 @@ static struct folio *writeback_get_folio(struct address_space *mapping,
return NULL;
}
+ if (folio->index != upgraded &&
+ pisa_writeback_upgrade(mapping, folio, &wbc->index, &wbc->fbatch)) {
+ upgraded = wbc->index;
+ goto retry;
+ }
+
folio_lock(folio);
if (unlikely(!folio_prepare_writeback(mapping, wbc, folio))) {
folio_unlock(folio);
diff --git a/mm/readahead.c b/mm/readahead.c
index 6e5563290287e..f5cfc69dc9909 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -114,6 +114,7 @@
*/
#include <linux/blk_plug.h>
+#include <linux/pisa.h>
#include <linux/kernel.h>
#include <linux/dax.h>
#include <linux/gfp.h>
@@ -197,7 +198,9 @@ static struct folio *ractl_alloc_folio(struct readahead_control *ractl,
{
struct folio *folio;
- folio = filemap_alloc_folio(gfp_mask, order, NULL);
+ folio = pisa_filemap_alloc(ractl->mapping, gfp_mask, order, false);
+ if (!folio)
+ folio = filemap_alloc_folio(gfp_mask, order, NULL);
if (folio && ractl->dropbehind)
__folio_set_dropbehind(folio);
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 40f20d0faa859..8ba9c39cb05ec 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -651,7 +651,8 @@ static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, s
return false;
/* Move user pages into PISA memory, then look them up again. */
- if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE))) {
+ if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE),
+ DMA_TO_DEVICE, false)) {
iov_iter_revert(from, *copied);
*copied = iov_iter_get_pages2(from, pages, *length, max_pages, start);
if (*copied < 0) {
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (48 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Document the PISA IOMMU 2MB-hugepage-backed DMA memory
subsystem in Documentation/core-api/pisa.rst.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/pisa.rst | 426 ++++++++++++++++++
3 files changed, 433 insertions(+)
create mode 100644 Documentation/core-api/pisa.rst
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..cb978469a7df3 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5345,6 +5345,12 @@ Kernel parameters
pirq= [SMP,APIC] Manual mp-table setup
See Documentation/arch/x86/i386/IO-APIC.rst.
+ pisa.*= [IOMMU] Pre-mapped IOVA Superpage Allocator (PISA)
+ tuning parameters (max_bits, max_pages, slots, max_idle,
+ idle_ms, reservoir_max, scrub_max, compact, wasted_pct,
+ old_ms, pcp_batch, user_max_4K_pages, scrub_migrate).
+ See Documentation/core-api/pisa.rst.
+
plip= [PPT,NET] Parallel port network link
Format: { parport<nr> | timid | 0 }
See also Documentation/admin-guide/parport.rst.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..4955f477ed797 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
dma-attributes
dma-isa-lpc
swiotlb
+ pisa
mm-api
cgroup
genalloc
diff --git a/Documentation/core-api/pisa.rst b/Documentation/core-api/pisa.rst
new file mode 100644
index 0000000000000..6ed4bd9951f8a
--- /dev/null
+++ b/Documentation/core-api/pisa.rst
@@ -0,0 +1,426 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=============================================
+PISA: Pre-mapped IOVA Superpage Allocator
+=============================================
+
+Overview
+========
+
+PISA is a memory allocator that provides DMA-able memory backed
+by large physically contiguous pages (typically 2MB), and mapped by
+IOMMU leaves of the same size.
+Devices can opt-in to use PISA memory for descriptor rings,
+header buffers, tx and rx data buffers, and zerocopy I/O.
+
+PISA has the following features:
+
+- greatly reduced IOTLB pressure by using fewer and much larger
+ IOMMU entries
+- greatly reduced DMA map/unmap costs. Mapping only happens when the first
+ block from a 2MB page is used. Unmapping is a no-op for the caller,
+ and happens in a separate worker the 2MB page is fully unused
+- much better security than DMA-FQ, because buffers are not returned
+ to the buddy allocator (where they could be used for code or sensitive
+ data) until all IOMMU and IOTLB mappings have been flushed.
+ Note that the mappings are kept active during the lifetime of buffers,
+ but this is the same thing that happens e.g. with pagepool, even in strict
+ IOMMU mode (DMA).
+- can remove the cost of copying into bounce buffers e.g. for overcommitted
+ or Confidential Computing VMs.
+
+For devices that do not opt-in to PISA, the IOMMU keeps using the
+native IOMMU mode (DMA or DMA-FQ).
+
+Note that PISA is fully integrated with relevant kernel APIs so
+devices can use it without device-specific code changes.
+
+What PISA is not
+================
+For avoidance of doubt: PISA is essentially a specialized memory allocator,
+modeled after alloc_pages() and related APIs. The main difference is that
+it enforces some properties on the memory it provides so that other operations
+(IOMMU mapping and unmapping, high rate allocations, identification...)
+can be implemented efficiently and with the desired security properties.
+
+It is not a specialized cache or pagepool or slab allocator:
+we have already plenty of them, built on top of the buddy allocator.
+PISA sits below these existing subsystems and acts as a memory provider.
+
+It is not a custom netdev feature. While network devices are the primary
+user, disks and other compute accelerators can use it, again, as a replacement
+for the alloc_pages() allocator.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+ (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+ access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+ exceeds the IOTLB size, backpressuring the bus via flow control.
+ This is one of the biggest bottlenecks for high speed devices:
+ typical network devices have very little IOVA locality, and that makes
+ the IOTLB almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA
+ map/unmap costs. This is common practice for command, completion and
+ buffer queues, as well as disk I/O and network receive buffers. Network
+ TX buffers are a noticeable exception. They normally come from
+ ``alloc_pages()`` calls unaware of the leaf device, and require
+ mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+ very expensive operation (up to several microseconds to wait for
+ completion, necessary for security), but the way it is implemented
+ in DMA-FQ compromises on security: IOTLB flush requests are run
+ periodically in batches, but the underlying memory is recycled without
+ waiting. During that window, the physical memory is still accessible,
+ opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+ blocks, e.g. 2MB large pages instead of the usual 4KB (PTE). This is not
+ currently pursued by the kernel.
+
+PISA addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map,
+ unmap, and free device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB PISA pages and 2MB IOMMU
+ mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused PISA pages
+ to the system
+
+Execution is what makes PISA practical: the implementation intercepts
+a small set of kernel APIs so that device will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since PISA intercepts allocations of device-accessible
+memory, it can transparently configure unencrypted mode, required for
+confidential computing. Under a paravirtualized IOMMU, the long-lived 2MB
+mappings also make host-side pinning of DMA memory a rare event.
+
+Limitations
+===========
+
+PISA is currently limited to systems with 2MB large pages, which
+is the vast majority of existing platforms. This is enforced at compile time.
+
+Decoupling the allocation size from 2MB can be made without much effort,
+so the mechanisms can be extended to systems with 16K/64K page size.
+
+The implementation assumes non-preemptible spinlocks, so PISA is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+PISA can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers follow the direction the allocator was created
+with. Receive pools (``page_pool``) are ``DMA_BIDIRECTIONAL``, so received
+buffers can be retransmitted (XDP_TX, forwarding) without a new mapping.
+Allocators that only hold transmit data, such as socket TX fragments, are
+``DMA_TO_DEVICE``. Allocators created
+``DMA_TO_DEVICE`` or ``DMA_FROM_DEVICE`` have their PISA pages mapped device
+read-only or write-only, and a mapping that requests other access falls back
+to a per-buffer mapping.
+
+IOMMU mappings for queues are also ``DMA_BIDIRECTIONAL``, though stricter modes
+can be implemented trivially.
+
+Implementation details
+======================
+
+Most of the code size and design complexity relates to the handling
+of exceptional events (device teardown, memory hotplug) and to
+the safe release of pages through deferred tasks, RCU and grace
+periods.
+
+The implementation lives in ``drivers/iommu/pisa-*.c`` and
+``include/linux/pisa.h``, and relies on the components described below.
+We use the name "PISA" to refer to the second-level physically contiguous
+pages (typically 2MB) used for I/O memory, but the design is not tightly
+bound to the use of a PISA page.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, socket TX page frags, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+
+and most devices will automatically inherit the benefits.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of PISA pages, based on either their Physical
+ Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to PISA page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **PISA metadata** (``struct pisa_meta``):
+
+ Identification of PISA pages could be done in principle with a flag in
+ ``struct page``, but that would not solve the management of per-page
+ metadata.
+
+ PISA addresses both requirements without depending on ``struct page``
+ using an idea similar to ``pageblock_flags``. Each PISA page used as a
+ buffer requires only 256B of metadata, or 0.01% overhead. For metadata,
+ we reserve a sparse virtual memory region of approximate size
+ ``max_possible_pfn * 256 / PISA_SIZE`` (``pisa.max_bits=N`` on the
+ command line covers ``2^N`` bytes of physical address space instead; RAM
+ above the covered range is not used by PISA), so the metadata can be accessed
+ directly using the PISA page index. A metadata page is backed when a
+ allocator first takes a PISA page in its 32MB chunk, from any context: the
+ page tables are allocated beforehand for present RAM, and by a memory
+ hotplug notifier for memory added later. A compact bitmap (1 bit per
+ 32MB chunk, or 4KB per 1TB of address space) tracks which metadata
+ pages are backed, allowing fast lockless validation and direct array
+ indexing.
+
+ ``struct pisa_meta`` has flags to mark the page status (used as
+ PISA buffer, or decrypted for confidential computing), and tracks
+ which subpages are free, which domains have an active IOMMU mapping for
+ the page, and holds list/RCU linkage to manage its lifetime.
+
+- **Per-domain reserved PISA IOVA ranges**:
+
+ Another key requirement is to quickly resolve the PA-IOVA mapping for a
+ given domain. Since a PISA page can be mapped in multiple domains (e.g.
+ a network buffer used to route packets between different NICs), it would
+ be too expensive to manage random PA-IOVA mappings created on the fly.
+ Each PISA page gets a dense slot number when it enters PISA, and is
+ mapped at the same offset, ``slot * PISA_SIZE``, in
+ every domain's window. PISA reserves on each domain an IOVA range of
+ ``pisa.slots * 2MB`` (by default 2 * ``max_pages`` slots, i.e. a
+ quarter of RAM): its size does not depend on the physical memory layout,
+ and IOVAs do not expose physical addresses. The IOVA allocation happens
+ the first time a domain is used, hence a IOVA within the reserved range
+ will uniquely identify a PISA page. Slots are handed out lowest first
+ and reused only after the page has been unmapped from every domain.
+
+- **alloc_pages() compatible pisa_alloc allocator for I/O buffers**:
+
+ PISA implements a pisa_alloc object for fixed-order page allocations
+ that is a natural fit for the allocation of NIC receive buffers and tx
+ socket buffers. The API is similar to ``alloc_pages()`` (in fact, it is an
+ almost direct replacement) and pisa_allocs are instantiated as follows:
+
+ - per node, order-0 ``DMA_BIDIRECTIONAL`` (``__GFP_PISA_RW``), with per-CPU
+ block caches, to provide RX buffer pages
+ - per node, ``DMA_TO_DEVICE`` (``__GFP_PISA_RO``), with per-CPU block caches
+ to provide order-3 (``SKB_FRAG_PAGE_ORDER``) pages to refill tx socket buffers; on a miss,
+ an ordinary order-0 page is used. Order-2 ``DMA_TO_DEVICE`` (without per-CPU
+ caches) is also provided for small skb heads on systems with few CPUs.
+
+ - per node, order-0 ``DMA_BIDIRECTIONAL`` (``pisa_alloc_coherent()``),
+ carving naturally aligned contiguous 4KB page ranges (up to ``PISA_SIZE``)
+ to back coherent ``dma_alloc_attrs()`` queues and header bounce buffers
+ when ``dev->pisa_coherent`` is enabled.
+
+
+- **Fine grained control**
+
+ Both for experimentation and production use, it is useful to have
+ some form of control on which devices want to use PISA and for
+ what. PISA exposes the following controls. Keep in mind that full
+ performance can only be achieved if all regions are mapped using PISA.
+
+ - ``/sys/devices/*/*/pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads}``
+ per-device entries that control whether a device uses PISA for coherent
+ allocations, RX buffers, TX socket page fragments, and small TX skb heads
+
+- **Safe release of memory to the system**
+
+ A key feature of PISA is that memory is not released to the system
+ while there are active IOMMU mappings. Release of buffers is managed
+ as follows
+
+ 1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to
+ PISA, IOMMU mappings are still active, and they can be recycled
+
+ 2. when all blocks in a PISA page are unused, the page is moved to the
+ allocator's "idle" list, still with active mappings and available for reuse,
+ unless the allocator already holds max_idle_pages idle pages
+
+ 3. pages above that per-allocator watermark are released by a reclaim work
+ item, and under memory pressure the shrinker also releases idle pages
+ from all allocators: the iommu mappings are removed and a synchronous IOTLB
+ flush is issued. The pages are no longer usable for allocations but not
+ returned to the buddy allocator yet
+
+ 4. once the previous step is complete, the system schedules ``call_rcu()``
+ and waits for an RCU grace period (and re-encrypts the page if it was
+ decrypted) before returning the PISA page to the buddy allocator.
+
+- **Handling domain destruction**
+
+ When a device/domain is destroyed, all I/O issued by the device must have
+ been quiesced and memory released via ``__free_pages()`` or ``dma_free*()``.
+ PISA hooks into the domain destructor (``iommu_put_dma_cookie()``) and
+ forgets the disappearing domain: its bit is cleared in every active
+ PISA page, without touching the domain's page tables, which the owner
+ frees wholesale. The domain index is reused only after an SRCU grace
+ period.
+
+- **Handling loss of the last DMA API user**
+
+ ``dma_unmap_*()`` never removes window PTEs, so when the last driver using
+ a group's default domain through the DMA API unbinds, or the group's DMA
+ ownership is taken (VFIO, iommufd), ``pisa_domain_quiesce()`` unmaps every
+ active PISA page from that domain and flushes the IOTLB. The window
+ and domain index are kept; PTEs are re-installed on demand if a driver binds
+ again. In-flight page_pool pages, which ``page_pool_scrub()`` cannot unmap,
+ are covered the same way.
+
+- **Allocation and compound splitting**:
+
+ An allocator that needs memory calls ``pisa_alloc_split_page()`` to allocate a
+ physically contiguous PISA page on the calling CPU's NUMA node, and
+ ``split_page_compound()`` to split it into independent blocks of
+ ``pa->order``. These blocks can be independently passed through the
+ networking stack.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+PISA integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of PISA memory is opt-in via per-device sysfs
+attributes or module parameters. Mapping, unmapping, and freeing automatically detects
+whether a buffer or IOVA belongs to PISA:
+
+- **Streaming DMA map and unmap** (``dma_map_page_attrs()``,
+ ``dma_map_single_attrs()``, ``dma_map_sg_attrs()`` and
+ ``dma_unmap_page_attrs()``, ``dma_unmap_single_attrs()``,
+ ``dma_unmap_sg_attrs()`` via ``drivers/iommu/dma-iommu.c`` and
+ ``kernel/dma/direct.c``):
+
+ - On map, any buffer belonging to a PISA page
+ (``is_pisa_phys(phys)``) is mapped via the domain's reserved
+ PISA IOVA window with a simple addition, typically reusing
+ existing mappings. Under ``dma-direct``, decrypted PISA pages
+ (``is_pisa_direct(phys, size)``) can be mapped directly without
+ ``swiotlb`` bounce.
+
+ On unmap, IOVAs can be identified as PISA buffers with a simple
+ range check and that results in a no-operation.
+
+- **Coherent DMA allocation and free** (``dma_alloc_attrs()`` /
+ ``dma_alloc_coherent()`` and ``dma_free_attrs()`` / ``dma_free_coherent()``
+ via ``drivers/iommu/dma-iommu.c`` and ``kernel/dma/direct.c``):
+
+ - PISA is enabled per device via ``/sys/devices/.../pisa_coherent``
+ (``dev->pisa_coherent``). When set, sleepable ``GFP_KERNEL`` coherent
+ allocations up to ``PISA_SIZE`` are rounded up to 4KB, carved out of
+ the per-node order-0 ``pisa_alloc`` via ``pisa_alloc_coherent()``, and
+ mapped through the domain's PISA IOVA window (or directly at their
+ physical/decrypted DMA address for ``dma-direct``). ``dma_free_attrs()``
+ releases the order-0 pages via ``pisa_free_coherent()``, which recycles
+ them back to the allocator.
+
+- **Page allocator release** (``__free_pages()`` / ``put_page()`` via
+ ``__free_pages_prepare()`` in ``mm/page_alloc.c``):
+
+ - ``pisa_free_page()`` checks ``is_pisa_page(page)``
+ and recycles PISA subpages back to their owning ``pisa_alloc``
+ instead of releasing them to the buddy allocator while their 2MB IOMMU
+ mappings remain active.
+
+- **IOMMU domain teardown** (``iommu_put_dma_cookie()`` in
+ ``drivers/iommu/dma-iommu.c``):
+
+ - A call to ``pisa_domain_release()`` before freeing a
+ domain's IOVA cookie and page tables clears cached domain state across
+ all allocators. Existing IOMMU mappings and IOTLB flush has already happened.
+
+- **Networking ``page_pool``** (``net/core/page_pool.c``):
+
+ - PISA is enabled per device via ``/sys/devices/.../pisa_rx_bufs``
+ (``dev->pisa_rx_bufs``). ``__page_pool_alloc_netmems_slow()`` adds
+ ``__GFP_PISA_RW`` via ``pisa_rx_gfp()`` when ``PP_FLAG_DMA_MAP`` is set,
+ allocating RX pages from the per-node ``DMA_BIDIRECTIONAL`` PISA
+ allocator and falling back to the buddy allocator on exhaustion.
+
+- **Socket TX page fragments** (``sk_page_frag_dst()`` and
+ ``skb_page_frag_refill()`` in ``net/core/sock.c``):
+
+ - When enabled per device via ``/sys/devices/.../pisa_tx_bufs``,
+ socket TX fragments are allocated from per-node ``pisa_alloc`` instances,
+ with per-CPU block caches in front of them.
+ These allocators are ``DMA_TO_DEVICE``: devices can only read them, and
+ callers that map fragments for device writes get per-buffer mappings.
+ - Only data routed to a device with a DMA parent, outside IPsec, is copied
+ into them (TCP, UDP/raw), through a dedicated
+ per-task frag. Loopback, virtual devices, kTLS, sockmap, KCM and MPTCP
+ use ordinary pages. ESP output uses the allocators through
+ ``skb_page_frag_refill_dev()``, when their egress device can use them.
+
+- **Small skb heads** (``kmalloc_reserve()`` in ``net/core/skbuff.c``):
+
+ - When enabled per device via ``/sys/devices/.../pisa_tx_skb_heads``,
+ small skb heads come from the ``skbuff_small_head_pisa`` slab cache,
+ whose slab pages are blocks of per-node TX-only ``pisa_alloc``
+ instances (created ``DMA_TO_DEVICE``), zeroed when taken from the allocator.
+ Mappings of these heads that request device write access fall back to
+ per-buffer IOMMU mappings.
+
+- **NIC driver RX buffers**
+
+ - Controlled per device by ``/sys/devices/.../pisa_rx_bufs``
+ (``dev->pisa_rx_bufs``) for queue modes managing their own page rings
+ (e.g. ``gve`` GQI-RDA).
+
+- **User pages for MSG_ZEROCOPY** (``drivers/iommu/pisa-user.c``):
+
+ - ``madvise(MADV_PISA)`` migrates the present private anonymous pages
+ of a range into blocks of per-node ``DMA_TO_DEVICE`` allocators, so that
+ zerocopy sends from them use the PISA window instead of per-send IOMMU
+ mappings or hypervisor pinning. ``MADV_NOPISA`` migrates them back to
+ ordinary pages. Both are allowed through ``process_madvise()``.
+ Freeing the pages (``munmap()``, ``MADV_DONTNEED``, exit, reclaim)
+ returns the blocks to their allocators.
+ - ``/sys/module/pisa/parameters/user_max_4K_pages``: Soft cap on user
+ pages held in PISA blocks, in 4 KB pages (``max_pages`` counts PISA
+ pages). When non-zero, eligible pages are also migrated on
+ ``MSG_ZEROCOPY`` sends (best effort, non-blocking, skipping pages still
+ referenced by in-flight skbs); 0 (default) disables both. With
+ ``pisa_tx_bufs`` also enabled, pages that are not in PISA
+ after that (cap reached, busy, large, shared or file-backed) are copied
+ by TCP into PISA socket frags instead of being sent zerocopy (other
+ senders still send them zerocopy); the copy is not reported to the
+ application.
+ - Not available on CoCo guests: decrypted blocks cannot be mapped to user
+ space, and whole pages would be shared with the host.
+
+- **Global memory limit and observability:**
+
+ - ``/sys/module/pisa/parameters/max_pages``: Global cap on PISA
+ pages held by all allocators and the per-node reservoirs
+ (defaults to 1/8th of physical RAM, at least 16 pages).
+ - ``/sys/module/pisa/parameters/reservoir_max``: Per-NUMA-node reservoir
+ target of pre-allocated PISA pages (defaults to 16 per memory node).
+ - ``/sys/module/pisa/parameters/scrub_max``: Max idle PISA pages
+ scrubbed per background pass (0 disables background scrubbing).
+ - ``/sys/kernel/debug/pisa/allocs``: Debugfs summary of global, per-node,
+ user-page, and per-allocator allocation, mapping, and fallback counters.
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread* [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
` (49 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
50 siblings, 0 replies; 86+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA allocation and mapping failures silently fall back to standard page
allocation and per-buffer DMA mappings, making performance regressions
hard to diagnose without observability.
Add per-allocator, per-node reservoir, user-page migration, and global
shrinker/reclaim counters, and expose them via
/sys/kernel/debug/pisa/allocs.
Also emit a one-shot dmesg warning on the first 2MB buddy allocation
failure.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 298 +++++++++++++++++++++++++++++++++----
drivers/iommu/pisa-kunit.c | 7 +-
drivers/iommu/pisa-map.c | 79 +++++++++-
drivers/iommu/pisa-node.c | 43 +++++-
drivers/iommu/pisa-priv.h | 127 +++++++++++++++-
drivers/iommu/pisa-user.c | 77 +++++++++-
include/linux/pisa.h | 5 +
net/core/datagram.c | 2 +
8 files changed, 589 insertions(+), 49 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index cedb1a62d20e2..91b9f00d21985 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -11,6 +11,7 @@
#include <linux/cache.h>
#include <linux/cc_platform.h>
#include <linux/cpuhotplug.h>
+#include <linux/debugfs.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
#include <linux/dma-map-ops.h>
@@ -31,6 +32,7 @@
#include <linux/rcupdate.h>
#include <linux/sched/clock.h>
#include <linux/sched/mm.h>
+#include <linux/seq_file.h>
#include <linux/set_memory.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -85,6 +87,13 @@ module_param_named(old_ms, pisa_old_ms, uint, 0644);
unsigned int pisa_scrub_max __read_mostly;
#define PISA_SCRUB_MAX_LIMIT 65536U
+/* Reclaim statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats pisa_reclaim_stats;
+
+char pisa_pages_2m[16] __ro_after_init;
+char pisa_pages_4k[16] __ro_after_init;
+char pisa_no_pages_2m[16] __ro_after_init;
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -115,6 +124,7 @@ static const struct kernel_param_ops pisa_scrub_ops = {
.get = param_get_uint,
};
module_param_cb(scrub_max, &pisa_scrub_ops, &pisa_scrub_max, 0644);
+
/* Return true if a new PISA page should be decrypted. */
bool pisa_want_decrypt(void)
{
@@ -179,7 +189,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
unsigned long flags;
spin_lock_irqsave(&meta->pa->lock, flags);
- meta->pa->stats.page_free_cnt++;
+ pisa_stat_inc(meta->pa->stats.page_free_cnt);
spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
@@ -190,6 +200,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Still shared with the host, or possibly still mapped by a device. */
if (meta->flags & (PISA_DECRYPTED | PISA_QUARANTINED))
return;
+
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -238,6 +249,9 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
struct pisa_meta *meta, *tmp;
struct llist_node *node;
struct pisa_alloc *pa;
+ unsigned int n = 0;
+
+ pisa_stat_inc(pisa_reclaim_stats.reclaim_run);
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
@@ -263,8 +277,11 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
read_unlock(&pisa_allocs_lock);
node = llist_del_all(&pisa_free_list);
- llist_for_each_entry_safe(meta, tmp, node, llnode)
+ llist_for_each_entry_safe(meta, tmp, node, llnode) {
pisa_release_page(meta);
+ n++;
+ }
+ pisa_stat_add(pisa_reclaim_stats.reclaim_pages, n);
if (next_delay != MAX_JIFFY_OFFSET)
queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
@@ -469,6 +486,7 @@ static unsigned long pisa_scrub_alloc(struct pisa_alloc *pa)
for (idx = 0; idx < count; idx++)
__set_bit(idxs[idx], meta->clean_bitmap);
meta->nr_clean += count;
+ pisa_stat_add(pa->stats.block_scrub_cnt, count);
pa->scrub_demand -= min(count, pa->scrub_demand);
should_release = pisa_place_meta(meta);
spin_unlock_irqrestore(&pa->lock, flags);
@@ -603,6 +621,7 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
meta->nr_dirty++;
}
pa->blocks_in_use--;
+ pisa_stat_inc(pa->stats.block_free_cnt);
*release |= pisa_place_meta(meta);
}
@@ -729,7 +748,8 @@ static const struct kernel_param_ops pisa_pcp_batch_ops = {
module_param_cb(pcp_batch, &pisa_pcp_batch_ops, &pisa_pcp_batch, 0644);
/* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
-static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n,
+ bool is_free)
{
unsigned int i, batch = READ_ONCE(pisa_pcp_batch);
struct pisa_pcp *pcp;
@@ -738,6 +758,8 @@ static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned
local_irq_save(flags);
pcp = this_cpu_ptr(pa->pcp);
spin_lock(&pcp->lock);
+ if (is_free)
+ pisa_stat_add(pcp->free_cnt, n);
for (i = 0; i < n; i++) {
if (pcp->count == PISA_PCP_HIGH) {
/* Keep the most recently freed, likely still in cache. */
@@ -799,18 +821,37 @@ void pisa_pcp_drain(struct pisa_alloc *pa)
}
}
-/* Blocks in @pa's caches, for shrinker and debugfs. */
-static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+/* Aggregate statistics across @pa's per-CPU caches, for shrinker and debugfs. */
+static unsigned long pisa_pcp_stats(struct pisa_alloc *pa, u64 *alloc, u64 *miss,
+ u64 *free, u64 *ks, u64 *as)
{
+ u64 a = 0, m = 0, f = 0, s_k = 0, s_a = 0;
unsigned long n = 0;
int cpu;
- if (!pa->pcp)
- return 0;
- for_each_possible_cpu(cpu) {
- /* A racy sum is good enough for reclaim and statistics. */
- n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+ if (pa->pcp) {
+ for_each_possible_cpu(cpu) {
+ const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+
+ /* A racy sum is good enough for reclaim and statistics. */
+ n += READ_ONCE(pcp->count);
+ a += READ_ONCE(pcp->alloc_cnt);
+ m += READ_ONCE(pcp->miss_cnt);
+ f += READ_ONCE(pcp->free_cnt);
+ s_k += READ_ONCE(pcp->alloc_ks);
+ s_a += READ_ONCE(pcp->alloc_as);
+ }
}
+ if (alloc)
+ *alloc = a;
+ if (miss)
+ *miss = m;
+ if (free)
+ *free = f;
+ if (ks)
+ *ks = s_k;
+ if (as)
+ *as = s_a;
return n;
}
@@ -1206,6 +1247,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
}
*countp -= used;
pa->blocks_in_use += used;
+ pisa_stat_add(pa->stats.block_alloc_cnt, used);
return got;
}
@@ -1254,9 +1296,11 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
/* Extract up to @want blocks from @meta under @pa->lock. */
static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
- struct page **out, bool zero, bool frozen)
+ struct page **out, bool can_block, bool zero,
+ bool frozen)
{
unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
+ unsigned long base_pfn = pisa_meta_to_pfn(meta);
struct pisa_alloc *pa = meta->pa;
lockdep_assert_held(&pa->lock);
@@ -1269,12 +1313,23 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
for (i = got; i < got + dirty; i++)
set_page_private(out[i], 1);
+ if (can_block) {
+ pisa_stat_add(pa->stats.alloc_kc, got);
+ pisa_stat_add(pa->stats.alloc_ks, dirty);
+ } else {
+ pisa_stat_add(pa->stats.alloc_ac, got);
+ pisa_stat_add(pa->stats.alloc_as, dirty);
+ }
got += dirty;
pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
} else {
/* First choice are dirty blocks, followed by clean */
got += pisa_scan_bitmap(meta, true, want, out, !frozen);
got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+ if (can_block)
+ pisa_stat_add(pa->stats.alloc_kd, got);
+ else
+ pisa_stat_add(pa->stats.alloc_ad, got);
}
used = avail_before - pisa_meta_avail(meta);
@@ -1284,6 +1339,8 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
meta->nr_dirty = 0;
}
pisa_place_meta(meta);
+ if (page_to_nid(pfn_to_page(base_pfn)) != pa->nid)
+ pisa_stat_add(pa->stats.numa_mismatch_cnt, used);
return got;
}
@@ -1292,6 +1349,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
struct page **pages, bool zero, bool frozen)
{
+ bool can_block = gfpflags_allow_blocking(gfp);
bool should_scrub = false;
unsigned long n, flags, i;
struct pisa_meta *meta;
@@ -1299,7 +1357,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
/* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
spin_lock_irqsave(&pa->lock, flags);
- for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+ for (n = 0; n < nr_pages;
+ n += take_blocks(meta, want, &pages[n], can_block, zero, frozen)) {
want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
meta = pisa_pick_meta(pa, zero);
if (meta)
@@ -1310,15 +1369,31 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
spin_unlock_irqrestore(&pa->lock, flags);
/* With @zero, fresh dirty blocks are zeroed below. */
meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ long err = PTR_ERR(meta);
+
+ if (err == -EAGAIN) {
+ pisa_stat_atomic_inc(pa->stats.fail_backoff);
+ } else if (err == -ENOSPC) {
+ pisa_stat_atomic_inc(pa->stats.fail_budget);
+ } else if (err == -ENOMEM) {
+ pisa_stat_atomic_inc(pa->stats.fail_nomem);
+ pr_warn_once("pisa: order-%u allocation failed for allocator %p; further failures are counted in debugfs only\n",
+ PISA_ORDER, pa);
+ } else if (err == -EBUSY) {
+ pisa_stat_atomic_inc(pa->stats.fail_split);
+ }
+ if (!n)
+ pisa_stat_atomic_inc(pa->stats.block_alloc_fail);
goto out;
+ }
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
WRITE_ONCE(meta->alloc_jiffies, jiffies);
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
- pa->stats.page_alloc_cnt++;
+ pisa_stat_inc(pa->stats.page_alloc_cnt);
}
/* Queue the scrubber when the inline zeroing crosses the trigger. */
should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
@@ -1346,9 +1421,11 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
struct page **page_array, bool frozen)
{
unsigned int pcp_batch = READ_ONCE(pisa_pcp_batch);
- unsigned long got, n, i, j;
+ bool zero = want_init_on_alloc(gfp);
+ unsigned long got, local, n, i, j;
got = pisa_pcp_pop(pa, nr_pages, page_array);
+ local = got;
if (got < nr_pages) {
if (nr_pages - got >= pcp_batch) {
got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
@@ -1360,7 +1437,7 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
for (i = 0; i < n && got < nr_pages; i++)
page_array[got++] = batch[i];
if (i < n)
- pisa_pcp_push(pa, batch + i, n - i);
+ pisa_pcp_push(pa, batch + i, n - i, false);
}
}
@@ -1369,10 +1446,13 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
page_array[i] = NULL;
/* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
- if (pisa_block_poisoned(pa, block))
+ if (pisa_block_poisoned(pa, block)) {
+ if (i < local)
+ local--;
continue;
+ }
pisa_block_prep(block);
- if (want_init_on_alloc(gfp)) {
+ if (zero) {
memset(page_address(block), 0, PAGE_SIZE << pa->order);
if (frozen) /* Order zeroing before caller publishes the page. */
smp_wmb();
@@ -1381,6 +1461,15 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
page_ref_unfreeze(block, 1);
page_array[j++] = block;
}
+ this_cpu_add(pa->pcp->alloc_cnt, j);
+ if (zero) {
+ if (gfpflags_allow_blocking(gfp))
+ this_cpu_add(pa->pcp->alloc_ks, j);
+ else
+ this_cpu_add(pa->pcp->alloc_as, j);
+ }
+ if (j > local)
+ this_cpu_inc(pa->pcp->miss_cnt);
return j;
}
@@ -1400,10 +1489,14 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
{
unsigned long i = 0, n, got;
- if (unlikely(!pa ||
- (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+ if (unlikely(!pa))
goto skip;
+ if (unlikely(gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))) {
+ pisa_stat_atomic_inc(pa->stats.fail_gfp);
+ goto skip;
+ }
+
while (i < nr_pages) {
if (page_array[i]) {
i++;
@@ -1500,6 +1593,9 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
page_ref_unfreeze(page + i, 1);
}
pa->blocks_in_use += nblocks;
+ pisa_stat_add(pa->stats.block_alloc_cnt, nblocks);
+ pisa_stat_add(pa->stats.alloc_kc, nblocks - dirty);
+ pisa_stat_add(pa->stats.alloc_ks, dirty);
pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
pisa_place_meta(meta);
return page;
@@ -1577,7 +1673,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
meta->prot = pa->prot;
WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
- pa->stats.page_alloc_cnt++;
+ pisa_stat_inc(pa->stats.page_alloc_cnt);
page = pisa_take_contig(meta, nblocks, align_mask);
if (unlikely(!page))
should_release = pisa_place_meta(meta);
@@ -1591,6 +1687,11 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
pisa_schedule_scrub(pa->nid);
if (!page)
return NULL;
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp)) {
+ this_cpu_add(pa->pcp->alloc_cnt, nblocks);
+ this_cpu_inc(pa->pcp->miss_cnt);
+ }
*pagep = page;
return memset(page_address(page), 0, size);
}
@@ -1631,9 +1732,10 @@ bool __pisa_free_page(struct page *page)
if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
page_to_nid(page) == numa_mem_id()) {
if (pisa_page_can_recycle(page)) {
- pisa_pcp_push(pa, &page, 1);
+ pisa_pcp_push(pa, &page, 1, true);
return true;
}
+ this_cpu_inc(pa->pcp->free_cnt);
if (this_cpu_read(pa->pcp->count))
pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
}
@@ -1657,7 +1759,7 @@ static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_co
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
- nr += pisa_pcp_count(pa) << pa->order;
+ nr += pisa_pcp_stats(pa, NULL, NULL, NULL, NULL, NULL) << pa->order;
}
read_unlock(&pisa_allocs_lock);
@@ -1685,11 +1787,14 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
{
struct pisa_meta *meta, *tmp;
+ unsigned long freed, res_freed;
struct pisa_alloc *pa;
- unsigned long freed;
LIST_HEAD(release_list);
freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+ res_freed = freed;
+ pisa_stat_atomic_inc(pisa_reclaim_stats.shrink_scan);
+ pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_res, res_freed);
if (freed < sc->nr_to_scan) {
read_lock(&pisa_allocs_lock);
@@ -1704,6 +1809,7 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
break;
}
read_unlock(&pisa_allocs_lock);
+ pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_idle, freed - res_freed);
}
/* Unmap outside locks; defer decrypted pages to workqueue outside fs_reclaim. */
@@ -1723,6 +1829,141 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
return freed ?: SHRINK_STOP;
}
+/* See pisa_seq_kv(). */
+void __pisa_seq_kv(struct seq_file *s, const char *title, const struct pisa_kv *st, unsigned int n)
+{
+ seq_printf(s, " %-12s", title);
+ for (; n; n--, st++)
+ seq_printf(s, " %-10s%11lld", st->name, st->val);
+ seq_putc(s, '\n');
+}
+
+/* Output global and active per-allocator statistics in /sys/kernel/debug/pisa/allocs. */
+static int pisa_allocs_show(struct seq_file *s, void *unused)
+{
+ struct pisa_reclaim_stats *rs = &pisa_reclaim_stats;
+ struct pisa_alloc *pa;
+
+ seq_puts(s, "global (allocators with no activity are omitted)\n");
+ pisa_seq_kv(s, pisa_pages_2m, { "in_use", atomic_long_read(&pisa_nr_pages) },
+ { "max", pisa_max_pages },
+ { "slots", pisa_nr_slots });
+ pisa_domains_show(s);
+ pisa_seq_kv(s, "reservoir", { pisa_pages_2m, pisa_reservoir_count() },
+ { "max_node", pisa_reservoir_max });
+ pisa_seq_kv(s, "shrinker",
+ { "scan", pisa_stat_atomic_read(rs->shrink_scan) },
+ { "from_res", pisa_stat_atomic_read(rs->pte_pages_res) },
+ { "from_idle", pisa_stat_atomic_read(rs->pte_pages_idle) });
+ pisa_seq_kv(s, "reclaim",
+ { "runs", pisa_stat_read(rs->reclaim_run) },
+ { pisa_pages_2m, pisa_stat_read(rs->reclaim_pages) },
+ { "unmap_fail", pisa_stat_atomic_read(rs->unmap_fail) });
+ pisa_reservoir_show(s);
+ pisa_user_show(s);
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ u64 alloc_pages, free_pages, block_alloc, block_free, block_scrub;
+ u64 pages_in_use, blocks_in_use, used_pct = 0, pcp_alloc, pcp_miss;
+ u64 block_fail, fail_gfp, numa_mismatch, pcp_free, cached;
+ u64 alloc_kd, alloc_kc, alloc_ks, alloc_ad, alloc_ac, alloc_as;
+ u64 pcp_ks, pcp_as;
+ struct pisa_alloc_stats *st = &pa->stats;
+ unsigned int num_idle, max_idle;
+ unsigned long flags;
+
+ block_fail = pisa_stat_atomic_read(st->block_alloc_fail);
+ fail_gfp = pisa_stat_atomic_read(st->fail_gfp);
+ cached = pisa_pcp_stats(pa, &pcp_alloc, &pcp_miss, &pcp_free, &pcp_ks, &pcp_as);
+
+ spin_lock_irqsave(&pa->lock, flags);
+ alloc_pages = st->page_alloc_cnt;
+ free_pages = st->page_free_cnt;
+ block_alloc = st->block_alloc_cnt;
+ block_free = st->block_free_cnt;
+ block_scrub = st->block_scrub_cnt;
+ numa_mismatch = st->numa_mismatch_cnt;
+ alloc_kd = st->alloc_kd;
+ alloc_kc = st->alloc_kc;
+ alloc_ks = st->alloc_ks;
+ alloc_ad = st->alloc_ad;
+ alloc_ac = st->alloc_ac;
+ alloc_as = st->alloc_as;
+ num_idle = pa->num_idle_pages;
+ max_idle = pa->max_idle_pages;
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ /* Omit allocators that have never allocated or failed an allocation. */
+ if (!alloc_pages && !block_alloc && !block_fail && !fail_gfp)
+ continue;
+
+ pages_in_use = alloc_pages > free_pages ? alloc_pages - free_pages : 0;
+ blocks_in_use = block_alloc > block_free + cached ?
+ block_alloc - block_free - cached : 0;
+ if (pages_in_use)
+ used_pct = div64_u64((blocks_in_use << pa->order) * 100,
+ pages_in_use << PISA_ORDER);
+
+ seq_printf(s, "alloc %s order %u nid %d%s\n", pa->name, pa->order,
+ pa->nid, pa->prot == IOMMU_READ ? " tx-only" :
+ pa->prot == IOMMU_WRITE ? " rx-only" : "");
+ pisa_seq_kv(s, pisa_pages_2m, { "got", alloc_pages },
+ { "put", free_pages }, { "in_use", pages_in_use },
+ { "idle", num_idle }, { "idle_max", max_idle });
+ pisa_seq_kv(s, pisa_no_pages_2m,
+ { "backoff", pisa_stat_atomic_read(st->fail_backoff) },
+ { "budget", pisa_stat_atomic_read(st->fail_budget) },
+ { "nomem", pisa_stat_atomic_read(st->fail_nomem) },
+ { "split", pisa_stat_atomic_read(st->fail_split) });
+ pisa_seq_kv(s, "blocks", { "alloc", block_alloc },
+ { "free", block_free }, { "in_use", blocks_in_use },
+ { "used_pct", used_pct }, { "scrub", block_scrub });
+ pisa_seq_kv(s, "blocks", { "failed", block_fail },
+ { "gfp", fail_gfp }, { "numa_miss", numa_mismatch });
+ pisa_seq_kv(s, "allocations",
+ { "KD", alloc_kd }, { "KC", alloc_kc }, { "KS", alloc_ks },
+ { "AD", alloc_ad }, { "AC", alloc_ac }, { "AS", alloc_as });
+ if (pa->pcp) {
+ int cpu;
+
+ pisa_seq_kv(s, "pcpu", { "alloc", pcp_alloc },
+ { "miss", pcp_miss },
+ { "free", pcp_free }, { "cached", cached },
+ { "KS", pcp_ks }, { "AS", pcp_as });
+ for_each_possible_cpu(cpu) {
+ const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+ u64 a = READ_ONCE(pcp->alloc_cnt);
+ u64 m = READ_ONCE(pcp->miss_cnt);
+ u64 f = READ_ONCE(pcp->free_cnt);
+ unsigned int c = READ_ONCE(pcp->count);
+ char title[16];
+
+ if (!a && !f && !c)
+ continue;
+ snprintf(title, sizeof(title), "pcpu_%u", cpu);
+ pisa_seq_kv(s, title, { "alloc", a },
+ { "miss", m },
+ { "free", f }, { "cached", c });
+ }
+ }
+#ifdef CONFIG_IOMMU_DMA
+ pisa_seq_kv(s, "domains",
+ { "mapped", pisa_stat_atomic_read(st->page_map_cnt) });
+ pisa_seq_kv(s, "map_fail",
+ { "no_window", pisa_stat_atomic_read(st->fallback_nowindow) },
+ { "no_idx", pisa_stat_atomic_read(st->fallback_noidx) },
+ { "no_huge", pisa_stat_atomic_read(st->fallback_nohuge) },
+ { "map_err", pisa_stat_atomic_read(st->fallback_maperr) },
+ { "prot", pisa_stat_atomic_read(st->fallback_prot) });
+#endif
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(pisa_allocs);
+
/* Return the blocks a dead CPU cached, which nothing else would. */
static int pisa_cpu_dead(unsigned int cpu)
{
@@ -1751,6 +1992,10 @@ static int __init pisa_init(void)
struct shrinker *shrink;
int nid, i;
+ snprintf(pisa_pages_2m, sizeof(pisa_pages_2m), "pages_%luM", PISA_SIZE >> 20);
+ snprintf(pisa_pages_4k, sizeof(pisa_pages_4k), "pages_%luK", PAGE_SIZE >> 10);
+ snprintf(pisa_no_pages_2m, sizeof(pisa_no_pages_2m), "no_pages_%luM", PISA_SIZE >> 20);
+
pisa_node_init();
/* Default ceiling is 1/8th of RAM (floor of 16 PISA pages). */
@@ -1787,7 +2032,6 @@ static int __init pisa_init(void)
pisa_nodes[nid].pa[ro][order] = pa;
}
}
-
pisa_user_boot_enable();
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
@@ -1799,6 +2043,10 @@ static int __init pisa_init(void)
pisa_cpu_dead) < 0)
pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
+ /* Built-in and never unloaded; debugfs stubs are no-ops when disabled. */
+ debugfs_create_file("allocs", 0444, debugfs_create_dir("pisa", NULL),
+ NULL, &pisa_allocs_fops);
+
shrink = shrinker_alloc(0, "pisa");
if (!shrink) {
pr_warn("pisa: shrinker registration failed\n");
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 3fc17024bc9d0..22f4966412feb 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -291,9 +291,9 @@ static int fake_domain_setup(struct kunit *test)
static void test_map_and_domain_lifecycle(struct kunit *test)
{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
.coherent_dma_mask = DMA_BIT_MASK(64) };
- struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct pisa_window *win;
struct pisa_meta *meta;
struct pisa_alloc *pa;
@@ -574,14 +574,15 @@ static void test_gfp_pisa(struct kunit *test)
static void test_reservoir_and_scrubber(struct kunit *test)
{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
unsigned int old_res = READ_ONCE(pisa_reservoir_max);
unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
- struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct pisa_meta *meta;
struct pisa_alloc *pa;
struct page *p;
int nid;
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
pisa_shrink_scan(NULL, &sc);
/* Refill reservoir to 1 PISA page per memory node, then consume one. */
@@ -607,9 +608,9 @@ static void test_reservoir_and_scrubber(struct kunit *test)
WRITE_ONCE(pisa_scrub_max, 512);
pisa_scrub_node(pa->nid);
KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ KUNIT_EXPECT_GE(test, pa->stats.block_scrub_cnt, 1ULL);
}
}
- pisa_shrink_scan(NULL, &sc);
/* Trim reservoirs back to 0. */
WRITE_ONCE(pisa_reservoir_max, 0);
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index fc58c639f3288..2f5ea446b7de5 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -10,12 +10,11 @@
#include <linux/dma-map-ops.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
-#include <linux/export.h>
#include <linux/iommu.h>
#include <linux/iova.h>
-#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
+#include <linux/seq_file.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/srcu.h>
@@ -106,6 +105,8 @@ static struct {
struct iommu_domain *domain;
dma_addr_t base;
spinlock_t map_lock; /* serialises iommu_map() in __pisa_map_phys() */
+ atomic_t slots;
+ char name[32];
} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
static DEFINE_SPINLOCK(pisa_domains_lock);
@@ -115,12 +116,56 @@ DEFINE_STATIC_SRCU(pisa_srcu);
/* Bumped under pisa_domains_lock whenever a domain index is released. */
unsigned int pisa_idx_gen __read_mostly;
+void pisa_domains_show(struct seq_file *s)
+{
+ unsigned int active = 0, draining = 0;
+ char bmap[PISA_MAX_DOMAINS];
+ unsigned long flags;
+ int idx;
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (pisa_domains[idx].domain) {
+ active++;
+ bmap[idx] = '1';
+ } else if (pisa_domains[idx].base) {
+ draining++;
+ bmap[idx] = 'd';
+ } else {
+ bmap[idx] = '.';
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ pisa_seq_kv(s, "domains", { "active", active }, { "draining", draining },
+ { "max", PISA_MAX_DOMAINS });
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if ((idx & 63) == 0)
+ seq_printf(s, " %-12s", idx ? "" : "domains_map");
+ if ((idx & 7) == 0)
+ seq_putc(s, ' ');
+ seq_putc(s, bmap[idx]);
+ if ((idx & 63) == 63 || idx == PISA_MAX_DOMAINS - 1)
+ seq_putc(s, '\n');
+ }
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (!pisa_domains[idx].domain)
+ continue;
+ seq_printf(s, " domain_%-5d base %pad size %lluM slots %d name %s\n",
+ idx, &pisa_domains[idx].base, pisa_window_size >> 20,
+ atomic_read(&pisa_domains[idx].slots), pisa_domains[idx].name);
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+}
/* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
static void pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
int idx, dma_addr_t iova, size_t unmapped)
{
pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
&iova, domain, idx, unmapped);
+ pisa_stat_atomic_inc(pisa_reclaim_stats.unmap_fail);
WRITE_ONCE(meta->flags, meta->flags | PISA_QUARANTINED);
}
@@ -142,6 +187,7 @@ int pisa_unmap_all(struct pisa_meta *meta)
if (!test_and_clear_bit(idx, meta->domains_mapped))
continue;
+ atomic_dec(&pisa_domains[idx].slots);
spin_lock_irqsave(&pisa_domains_lock, flags);
domain = pisa_domains[idx].domain;
@@ -217,6 +263,8 @@ void pisa_domain_release(struct iommu_domain *domain)
/* Mark slot FREE and bump @pisa_idx_gen so NOIDX domains retry. */
spin_lock_irqsave(&pisa_domains_lock, flags);
pisa_domains[idx].base = 0;
+ atomic_set(&pisa_domains[idx].slots, 0);
+ pisa_domains[idx].name[0] = '\0';
WRITE_ONCE(pisa_idx_gen, pisa_idx_gen + 1);
spin_unlock_irqrestore(&pisa_domains_lock, flags);
@@ -239,6 +287,7 @@ static unsigned int pisa_unmap_domain_locked(struct pisa_meta *meta,
if (!test_and_clear_bit(idx, meta->domains_mapped))
return 0;
+ atomic_dec(&pisa_domains[idx].slots);
unmapped = iommu_unmap_fast(domain, iova, PISA_SIZE, gather);
if (unlikely(unmapped != PISA_SIZE))
@@ -388,6 +437,9 @@ int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct p
pisa_domains[idx].domain = domain;
pisa_domains[idx].base = base;
spin_lock_init(&pisa_domains[idx].map_lock);
+ atomic_set(&pisa_domains[idx].slots, 0);
+ strscpy(pisa_domains[idx].name, dev_name(dev),
+ sizeof(pisa_domains[idx].name));
WRITE_ONCE(win->domain_idx, idx + PISA_IDX_FIRST);
/* Publish .domain_idx before .base (paired with is_pisa_iova()). */
smp_store_release(&win->base, base);
@@ -435,8 +487,10 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
/* Every leaf PTE of the page carries IOMMU_CACHE | @meta->prot. */
block_prot = IOMMU_CACHE | meta->prot;
- if (unlikely(prot & ~block_prot))
+ if (unlikely(prot & ~block_prot)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_prot);
return -EACCES;
+ }
/* Acquire pairs with the .base release in pisa_window_assign(). */
if (unlikely(!smp_load_acquire(&win->base))) {
@@ -445,8 +499,15 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
if (likely(!ret))
ret = pisa_window_assign(dev, domain, win);
- if (ret)
+ if (ret) {
+ if (ret == -EOPNOTSUPP)
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nohuge);
+ else if (ret == -EBUSY)
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_noidx);
+ else
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nowindow);
return ret;
+ }
}
addr = pisa_phys_to_iova(win, meta, phys);
@@ -454,8 +515,10 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
/* Check @dev's own DMA mask and bus limit against the window IOVA. */
limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
limit = min_not_zero(limit, dev->coherent_dma_mask);
- if (unlikely(addr + size - 1 > limit))
+ if (unlikely(addr + size - 1 > limit)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nowindow);
return -ENOSPC;
+ }
domain_idx = win->domain_idx - PISA_IDX_FIRST;
if (likely(test_bit(domain_idx, meta->domains_mapped))) {
@@ -471,10 +534,14 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
- if (likely(!ret)) {
+ if (unlikely(ret)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_maperr);
+ } else {
/* Ensure leaf PTE is visible before setting the domain bit. */
smp_mb__before_atomic();
set_bit(domain_idx, meta->domains_mapped);
+ atomic_inc(&pisa_domains[domain_idx].slots);
+ pisa_stat_atomic_inc(meta->pa->stats.page_map_cnt);
}
}
spin_unlock_irqrestore(&pisa_domains[domain_idx].map_lock, flags);
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 7469674133620..66280715ba504 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -15,6 +15,7 @@
#include <linux/nodemask.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/seq_file.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
#include <linux/workqueue.h>
@@ -139,8 +140,10 @@ static void pisa_reservoir_refill_node(int nid)
meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
pisa_want_decrypt(), &clean);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ pisa_stat_inc(res->stats.refill_err);
break;
+ }
/* Decrypted and init_on_alloc/init_on_free pages are already zeroed. */
if (!clean)
@@ -149,6 +152,7 @@ static void pisa_reservoir_refill_node(int nid)
spin_lock_irqsave(&res->lock, flags);
list_add(&meta->list, &res->pages);
res->nr_pages++;
+ pisa_stat_inc(res->stats.refill);
spin_unlock_irqrestore(&res->lock, flags);
cond_resched();
@@ -169,6 +173,7 @@ static void pisa_reservoir_trim_node(int nid, unsigned int max)
if (meta) {
list_del_init(&meta->list);
res->nr_pages--;
+ pisa_stat_inc(res->stats.trim);
}
spin_unlock_irqrestore(&res->lock, flags);
if (!meta)
@@ -184,8 +189,7 @@ void pisa_schedule_scrub(int nid)
{
struct pisa_node *node;
- if ((!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate)) ||
- nid == NUMA_NO_NODE)
+ if (!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate))
return;
node = &pisa_nodes[nid];
@@ -253,6 +257,7 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool
if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
list_del_init(&meta->list);
res->nr_pages--;
+ pisa_stat_inc(res->stats.hit);
refill = res->nr_pages <= max / 2;
} else {
meta = NULL;
@@ -282,6 +287,7 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *cle
*clean = true;
return meta;
}
+ pisa_stat_atomic_inc(pisa_nodes[nid].stats.miss);
/* Empty: pop_node() only schedules after a successful pop. */
pisa_schedule_reservoir_node(nid);
}
@@ -339,3 +345,34 @@ void __init pisa_node_init(void)
INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
}
}
+
+/* Print the reservoir of each node that has seen any activity. */
+void pisa_reservoir_show(struct seq_file *s)
+{
+ bool header = false;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ struct pisa_node *res = &pisa_nodes[nid];
+ unsigned int nr = READ_ONCE(res->nr_pages);
+ s64 refill = pisa_stat_read(res->stats.refill);
+ s64 refill_err = pisa_stat_read(res->stats.refill_err);
+ s64 miss = pisa_stat_atomic_read(res->stats.miss);
+ char title[16];
+
+ /* Pages served or trimmed were refilled first. */
+ if (!nr && !refill && !refill_err && !miss)
+ continue;
+ if (!header) {
+ seq_puts(s, "reservoir\n");
+ header = true;
+ }
+ snprintf(title, sizeof(title), "node_%d", nid);
+ pisa_seq_kv(s, title, { pisa_pages_2m, nr },
+ { "hit", pisa_stat_read(res->stats.hit) },
+ { "miss", miss });
+ pisa_seq_kv(s, title, { "refill", refill },
+ { "refill_err", refill_err },
+ { "trim", pisa_stat_read(res->stats.trim) });
+ }
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 1ecfebbaeb381..8eec8996bbca7 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -8,9 +8,7 @@
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/mem_encrypt.h>
-#include <linux/nodemask.h>
#include <linux/spinlock.h>
-#include <linux/topology.h>
#include <linux/types.h>
#include <linux/workqueue.h>
@@ -136,13 +134,78 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
struct pisa_pcp {
spinlock_t lock; /* by this CPU, or by a drain from any CPU */
unsigned int count;
+ u64 alloc_cnt;
+ u64 miss_cnt;
+ u64 free_cnt;
+ u64 alloc_ks;
+ u64 alloc_as;
struct page *blocks[PISA_PCP_HIGH];
};
-/* Per-allocator statistics, see pisa_allocs_show(). */
+#define pisa_stat_add(var, val) ((var) += (val))
+#define pisa_stat_inc(var) pisa_stat_add(var, 1)
+#define pisa_stat_dec(var) pisa_stat_add(var, -1)
+#define pisa_stat_read(var) READ_ONCE(var)
+
+#define pisa_stat_atomic_add(var, val) atomic64_add(val, &(var))
+#define pisa_stat_atomic_inc(var) atomic64_inc(&(var))
+#define pisa_stat_atomic_read(var) atomic64_read(&(var))
+
+/**
+ * struct pisa_alloc_stats - Per-allocator statistics, see pisa_allocs_show()
+ * @page_alloc_cnt: PISA pages allocated for this allocator
+ * @page_free_cnt: PISA pages released back to the buddy allocator
+ * @block_alloc_cnt: Block allocations satisfied
+ * @block_free_cnt: Block frees recycled into allocator
+ * @block_scrub_cnt: Dirty blocks zeroed by background scrubber
+ * @numa_mismatch_cnt: Block allocations satisfied from a non-target NUMA node
+ * @alloc_kd: Sleepable non-zeroing block allocations
+ * @alloc_kc: Sleepable zeroing block allocations served from clean_bitmap
+ * @alloc_ks: Sleepable zeroing block allocations zeroed inline
+ * @alloc_ad: Atomic non-zeroing block allocations
+ * @alloc_ac: Atomic zeroing block allocations served from clean_bitmap
+ * @alloc_as: Atomic zeroing block allocations zeroed inline
+ * @fail_backoff: PISA page acquisitions skipped due to atomic backoff
+ * @fail_budget: PISA page acquisitions refused by global pisa_max_pages
+ * @fail_nomem: PISA page acquisitions that ran out of memory
+ * @fail_split: split_page_compound() rejections
+ * @block_alloc_fail: PISA page acquisitions that failed on empty allocator
+ * @fail_gfp: pisa_alloc() calls declined due to unsupported GFP
+ * @page_map_cnt: PISA page mappings established across all domains
+ * @fallback_nowindow: Buffers mapped individually due to no usable IOVA window
+ * @fallback_noidx: Buffers mapped individually due to exhausted domain indices
+ * @fallback_nohuge: Buffers mapped individually because domain cannot use PISA
+ * @fallback_maperr: Buffers mapped individually because PISA mapping failed
+ * @fallback_prot: Buffers mapped individually due to protection mismatch
+ */
struct pisa_alloc_stats {
+ /* Updated under @pa->lock. */
u64 page_alloc_cnt;
u64 page_free_cnt;
+ u64 block_alloc_cnt;
+ u64 block_free_cnt;
+ u64 block_scrub_cnt;
+ u64 numa_mismatch_cnt;
+ u64 alloc_kd;
+ u64 alloc_kc;
+ u64 alloc_ks;
+ u64 alloc_ad;
+ u64 alloc_ac;
+ u64 alloc_as;
+
+ /* Updated without @pa->lock (e.g. pisa_alloc_split_page() and map path). */
+ atomic64_t fail_backoff;
+ atomic64_t fail_budget;
+ atomic64_t fail_nomem;
+ atomic64_t fail_split;
+ atomic64_t block_alloc_fail;
+ atomic64_t fail_gfp;
+ atomic64_t page_map_cnt;
+ atomic64_t fallback_nowindow;
+ atomic64_t fallback_noidx;
+ atomic64_t fallback_nohuge;
+ atomic64_t fallback_maperr;
+ atomic64_t fallback_prot;
};
/**
@@ -195,6 +258,18 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
READ_ONCE(pa->num_idle_pages);
}
+/* Reservoir statistics, in PISA pages, see pisa_reservoir_show(). */
+struct pisa_reservoir_stats {
+ /* Updated under @res->lock or by the per-node singleton @res->work. */
+ u64 hit; /* requests for this node it served */
+ u64 refill; /* PISA pages added by the refill */
+ u64 refill_err; /* refill allocations that failed */
+ u64 trim; /* PISA pages released by the trim */
+
+ /* Updated without @res->lock when the reservoir is empty. */
+ atomic64_t miss; /* requests for this node found it empty */
+};
+
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
@@ -206,8 +281,28 @@ struct pisa_node {
unsigned int nr_pages;
unsigned long flags; /* PISA_NODE_* requests */
struct work_struct work; /* refill/trim/scrub, on this node */
+ struct pisa_reservoir_stats stats;
} ____cacheline_aligned_in_smp;
+/* Global reclaim and shrinker statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats {
+ /* Updated by singleton pisa_reclaim_work. */
+ u64 reclaim_run; /* pisa_reclaim_work_fn() runs */
+ u64 reclaim_pages; /* PISA pages it released */
+
+ /* Updated locklessly. */
+ atomic64_t shrink_scan; /* pisa_shrink_scan() calls */
+ atomic64_t pte_pages_res; /* 4KB pages taken from the reservoirs */
+ atomic64_t pte_pages_idle; /* 4KB pages taken from idle allocators */
+ atomic64_t unmap_fail; /* PISA page unmaps that failed */
+};
+
+extern struct pisa_reclaim_stats pisa_reclaim_stats;
+
+extern char pisa_pages_2m[16];
+extern char pisa_pages_4k[16];
+extern char pisa_no_pages_2m[16];
+
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
@@ -255,8 +350,6 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
bool pisa_want_decrypt(void);
-bool pisa_place_meta(struct pisa_meta *meta);
-void pisa_schedule_reclaim(void);
int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
int pisa_page_unprepare(struct pisa_meta *meta);
@@ -267,6 +360,8 @@ __printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
int pisa_enable_pcp(struct pisa_alloc *pa);
void pisa_pcp_drain(struct pisa_alloc *pa);
+bool pisa_place_meta(struct pisa_meta *meta);
+void pisa_schedule_reclaim(void);
void pisa_release_page(struct pisa_meta *meta);
unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
unsigned long nr_pages, struct page **page_array, bool frozen);
@@ -300,8 +395,28 @@ void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+
+struct seq_file;
+void pisa_reservoir_show(struct seq_file *s);
+void pisa_user_show(struct seq_file *s);
void pisa_user_boot_enable(void);
+/* One name/value pair of a debugfs statistics line. */
+struct pisa_kv {
+ const char *name;
+ s64 val;
+};
+
+void __pisa_seq_kv(struct seq_file *s, const char *title, const struct pisa_kv *st, unsigned int n);
+
+/* Print @title and fixed-width name/value pairs to debugfs seq_file @s. */
+#define pisa_seq_kv(s, title, ...) \
+do { \
+ const struct pisa_kv __st[] = { __VA_ARGS__ }; \
+ \
+ __pisa_seq_kv(s, title, __st, ARRAY_SIZE(__st)); \
+} while (0)
+
#endif /* CONFIG_PISA */
#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
@@ -316,6 +431,7 @@ void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
int pisa_unmap_all(struct pisa_meta *meta);
+void pisa_domains_show(struct seq_file *s);
int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win);
int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
struct iommu_domain *domain, phys_addr_t phys,
@@ -364,6 +480,7 @@ static inline void pisa_slots_init(void) {}
static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+static inline void pisa_domains_show(struct seq_file *s) {}
static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
struct iommu_domain *domain, phys_addr_t phys,
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
index 7dca3bf066f0d..5cf199943f82c 100644
--- a/drivers/iommu/pisa-user.c
+++ b/drivers/iommu/pisa-user.c
@@ -21,6 +21,7 @@
#include <linux/pagewalk.h>
#include <linux/folio_batch.h>
#include <linux/sched/clock.h>
+#include <linux/seq_file.h>
#include <linux/swap.h>
#include "pisa-priv.h"
@@ -88,6 +89,26 @@ module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 064
static bool pisa_user_ready __read_mostly;
static DEFINE_MUTEX(pisa_user_lock);
+/* Statistics, see pisa_user_show(). */
+struct pisa_user_stats {
+ atomic64_t in; /* pages migrated into PISA blocks */
+ atomic64_t in_fail; /* ... isolated but not migrated */
+ atomic64_t out; /* pages migrated back to ordinary pages */
+ atomic64_t out_fail; /* ... isolated but not migrated */
+ atomic64_t no_budget; /* pages not moved in: cap reached */
+ atomic64_t busy; /* pages skipped: referenced elsewhere */
+ atomic64_t large; /* pages skipped: in a large anon folio */
+ atomic64_t copy; /* TCP chunks copied: page not in PISA */
+};
+
+static struct pisa_user_stats pisa_user_stats;
+
+/* Called by zerocopy_fill_skb_from_iter() when it asks TCP to copy. */
+void pisa_stats_inc_user_copy(void)
+{
+ pisa_stat_atomic_inc(pisa_user_stats.copy);
+}
+
/* Lazily create per-node user allocators on first use (process context). */
static bool pisa_user_init(void)
{
@@ -141,6 +162,9 @@ static unsigned long pisa_user_in_use(void)
unsigned long used = 0;
int nid;
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!smp_load_acquire(&pisa_user_ready))
+ return 0;
for_each_node_state(nid, N_MEMORY) {
unsigned int dir;
@@ -255,7 +279,7 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
.same_alloc = same_alloc,
};
enum migrate_mode mode = MIGRATE_SYNC;
- unsigned int done = 0;
+ unsigned int nr, done = 0;
if (list_empty(list))
return 0;
@@ -264,9 +288,17 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
if (!gfpflags_allow_blocking(gfp))
mode = MIGRATE_ASYNC;
}
+ nr = list_count_nodes(list); /* all order-0 */
if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
NULL, (unsigned long)&mtc, mode, MR_SYSCALL, &done))
putback_movable_pages(list);
+ if (to_pisa) {
+ pisa_stat_atomic_add(pisa_user_stats.in, done);
+ pisa_stat_atomic_add(pisa_user_stats.in_fail, nr - done);
+ } else {
+ pisa_stat_atomic_add(pisa_user_stats.out, done);
+ pisa_stat_atomic_add(pisa_user_stats.out_fail, nr - done);
+ }
return done;
}
@@ -285,7 +317,7 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
int extra = pinned ? GUP_PIN_COUNTING_BIAS : 1;
bool drained = false;
LIST_HEAD(list);
- unsigned int i, want = 0, need_budget = 0;
+ unsigned int i, want = 0, need_budget = 0, large = 0;
if (!pisa_user_init())
return false;
@@ -293,15 +325,21 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
+ /* THP/mTHP heap pages are never migrated, see pisa_user_want() */
+ large += folio_test_large(folio) && folio_test_anon(folio);
if (pisa_user_want_dir(folio, true, dir, false)) {
want++;
need_budget += !is_pisa_pfn(folio_pfn(folio));
}
}
+ if (large)
+ pisa_stat_atomic_add(pisa_user_stats.large, large);
if (!want)
return false;
- if (need_budget && pisa_user_budget() <= 0)
+ if (need_budget && pisa_user_budget() <= 0) {
+ pisa_stat_atomic_add(pisa_user_stats.no_budget, need_budget);
return false;
+ }
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
@@ -312,8 +350,10 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
lru_add_drain();
drained = true;
}
- if (pisa_user_busy(folio, extra))
+ if (pisa_user_busy(folio, extra)) {
+ pisa_stat_atomic_inc(pisa_user_stats.busy);
continue;
+ }
isolate_folio_to_list(folio, &list);
}
if (list_empty(&list))
@@ -583,9 +623,14 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false) &&
- !pisa_user_busy(folio, 0) && isolate_folio_to_list(folio, &list))
- n++;
+ if (to_pisa && folio_test_large(folio) && folio_test_anon(folio)) {
+ pisa_stat_atomic_inc(pisa_user_stats.large);
+ } else if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false)) {
+ if (pisa_user_busy(folio, 0))
+ pisa_stat_atomic_inc(pisa_user_stats.busy);
+ else if (isolate_folio_to_list(folio, &list))
+ n++;
+ }
folio_walk_end(&fw, vma);
}
if (n == PISA_USER_BATCH) {
@@ -597,3 +642,21 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
return 0;
}
+
+/* Print user page state and migration statistics to debugfs seq_file @s. */
+void pisa_user_show(struct seq_file *s)
+{
+ seq_puts(s, "user\n");
+ pisa_seq_kv(s, pisa_pages_4k, { "in_use", pisa_user_in_use() },
+ { "max", pisa_user_max_4k_pages });
+ pisa_seq_kv(s, "migrate",
+ { "in", pisa_stat_atomic_read(pisa_user_stats.in) },
+ { "in_fail", pisa_stat_atomic_read(pisa_user_stats.in_fail) },
+ { "out", pisa_stat_atomic_read(pisa_user_stats.out) },
+ { "out_fail", pisa_stat_atomic_read(pisa_user_stats.out_fail) });
+ pisa_seq_kv(s, "skipped",
+ { "no_budget", pisa_stat_atomic_read(pisa_user_stats.no_budget) },
+ { "busy", pisa_stat_atomic_read(pisa_user_stats.busy) },
+ { "large", pisa_stat_atomic_read(pisa_user_stats.large) },
+ { "copy", pisa_stat_atomic_read(pisa_user_stats.copy) });
+}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 1d9b07370cc75..667321b18ae47 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -147,6 +147,7 @@ bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir);
bool __pisa_mapping_enabled(const struct address_space *mapping,
enum dma_data_direction dir);
+void pisa_stats_inc_user_copy(void);
int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa);
@@ -331,6 +332,10 @@ pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
return NULL;
}
+static inline void pisa_stats_inc_user_copy(void)
+{
+}
+
static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa)
{
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 8ba9c39cb05ec..360ed445fe779 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -693,6 +693,8 @@ static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, s
if (keep)
return false;
+ if (skb->len == orig_len)
+ pisa_stats_inc_user_copy();
*ret = (skb->len != orig_len) ? 0 : -EXDEV;
return true;
}
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 86+ messages in thread