From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.218.72 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794248744; b=4s/Gs100aP+T36L8SdcGvAmEs6CH2lD6d4iKIwpUv3T6A5JOA/1Qhg/1g4efRCX/0Qmf 5x8VvxNb/5cBrZ7+kbr6idnUcC3ni3mv1amn9j43HBBD5M1XW/7GUuyoXREwZfoM8NS+b m60/1UiUYVpl+z0HdUbQbfdJ9r38SF6vKnq1NF8Q80In/1CDvb+Bb2g9IA9P2p8tWrBJo JtS5dejr1tojLHf6/Dl3S+1OLyN6/bqH+Mr/WkZEiSqa0CYMOsMfqBWc9a8EivjJNPDH3 3EbHIN3bmZMUu2DPeM5am/DXFom6qEVD0drAdycTSKw0WHMrciXgaDRfGejhQmoP7CA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794248744; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=54ExQeOShcGppfyJua980qoDBOFtyYHHsHDIcQmw4W4=; b=5t7Z/KFO/tm4RowKRKixuFphM0ncEquiWX76Li9dXPIzGMoNri85aP3+b9FgMu6eoZ3E 2EGK2Bx5TzIUCoMvoGydShUv3Z0jL4HTBNL0BojjRu9OBouI5Gp5ZFD4zfmgQI4OOzRzG zMQW+hcOGLeYcEo6jvnIZBKHIFqJXcnCH7VL4s4RnDum82cFPNobV3wxln8F2xXTYqnrb lQ05SP2MtqjNP9LBHcEAlpGfD38g/Dv11SmG1NpapxsOhv0bdrzHcNsVtdRxhCFmr4Ysa rb/LrJm8K+JADs0ys7yDZ27zloQP6C1SDJmUaoBcFJq5dK+Y6vnj6XyeCH5B/x2cKuA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=p2jSV5qO; arc=none smtp.remote-ip=209.85.218.72 Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2272560A382B for ; Mon, 9 Nov 2026 18:25:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 2272560A382B Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=p2jSV5qO Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-c4d58203e40so98520266b.1 for ; Mon, 09 Nov 2026 10:25:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1794248743; x=1794853543; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=54ExQeOShcGppfyJua980qoDBOFtyYHHsHDIcQmw4W4=; b=p2jSV5qO0XBa1qgKeWrpSAaOlwovte9S4H8sptAoi/MFd2T/v6+jDkMxZdVEIzwA7Y Aw2NPfYCWLO0hjKZpPoGMMoKJuCBRrCj9GyykHMZpHcWEsWig0dSZ0vnvvzEGqs8WjPP 4qOx2dORsulpFdpjIpi/nI+dJCoFRoc+0nbOUa/URO41gCsIobaQpGwawpJAXsSpMhwy FVKcIiqjbDtxkcdF0710b0VoBzmniP45mD7lqefgEZEO5oJzmOpNXd0x3ydA43MztdDJ UxvvD4nxs/pCPFMhvI9Nrbu1ExneZT2+IW5GzdmMIHxCwdWso4jnWxM+5aUVnMqY+y8i o07A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794248743; x=1794853543; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=54ExQeOShcGppfyJua980qoDBOFtyYHHsHDIcQmw4W4=; b=dCeMfOHg8JpkNlYqR2z+n2Njfut4E+R0WwJRdV1jumQtkYpvlAwELjKQrmDwz1gv6t GyoSzEcJf1R6TPlfotFW06gpNG66R88eqcN0YABEgP5EY4XXhtD+7mf9WIqf6ivgZCf4 gW70UYuXnigtmaaj941OT2lgpgzAZt4eqtofW+avFsoeIrWyeF3MDehYIsu08vOENY07 uzvYbWLDYeP5ULO7FiemueA6001pc44jTKL2J4M7LOqw6jh/qWSAbmnErbvZ2h14Dkeh 7PJNS6sMmmXswxaL/GA0o2n15dbalH/3e4HMpoYWQIvFgtY1SanH5/3oyQFov+xjlft1 zxiw== X-Forwarded-Encrypted: i=1; AL+hErG1VRfXdkPhkSruqcYMMLr12KqQYyEL8PW6DXckrw0j/2G76gZAwV6O9WSMO4/obbFovOe/CKi5gczbeAQ=@vger.kernel.org X-Gm-Message-State: APuU4Z8SWSa0IawN4e3icTS7F8koVkBTXb/vG1FUIHxu6AZQeZqG57P8 87ezOXu2BjEI4BbZjjn/GyOLwtkRFK6EgPyLuYiNKY8+cZqWiC3tNxRVx4PUfhtk2U/oDrk7tO9 uj/p1jA== X-Received: from ejblh15.prod.google.com ([2002:a17:906:f8cf:b0:c34:bb1f:3bc4]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a17:906:c155:b0:c2d:fade:e089 with SMTP id a640c23a62f3a-c5048c88b3cmr21975766b.0.1794248742269; Mon, 09 Nov 2026 10:25:42 -0800 (PST) Date: Mon, 9 Nov 2026 18:25:13 +0000 In-Reply-To: <20261003212241.3432303-1-lrizzo@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261003212241.3432303-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.0.542.g3ef89b7020-goog Message-ID: <20261109182533.3594984-6-lrizzo@google.com> Subject: [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages From: Luigi Rizzo To: Luigi Rizzo , Joerg Roedel , Will Deacon , Robin Murphy , Christoph Hellwig , Marek Szyprowski , Andrew Morton , Vlastimil Babka , David Hildenbrand , Jens Axboe , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Matthew Wilcox , Lorenzo Stoakes , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Alexander Lobakin , iommu@lists.linux.dev, netdev@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, driver-core@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Add two mechanisms to bound the number of unused PISA pages: - a global ceiling, pisa.max_pages, on the number of PISA pages pinned across all allocators and reservoirs, defaulting to one eighth of RAM, with a floor for small machines. - a shrinker that, under reclaim, drains the per-node reservoirs first and then completely idle pages across all allocators. Idle pages are detached under the allocator lock but retired outside it, to avoid holding the lock during the expensive unmaps/iotlb flush. Signed-off-by: Luigi Rizzo --- drivers/iommu/pisa-alloc.c | 91 ++++++++++++++++++++++++++++++++++++++ drivers/iommu/pisa-kunit.c | 15 +++++++ drivers/iommu/pisa-node.c | 25 +++++++++++ drivers/iommu/pisa-priv.h | 15 +++++++ 4 files changed, 146 insertions(+) diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c index 1f1b99023094c..de8009fd6fff3 100644 --- a/drivers/iommu/pisa-alloc.c +++ b/drivers/iommu/pisa-alloc.c @@ -4,6 +4,7 @@ * See Documentation/core-api/pisa.rst for the architecture overview. */ +#include #include #include #include @@ -22,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -40,6 +42,11 @@ static LLIST_HEAD(pisa_free_list); /* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */ static struct workqueue_struct *pisa_wq __ro_after_init; +/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */ +atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp; +unsigned long pisa_max_pages __read_mostly; +module_param_named(max_pages, pisa_max_pages, ulong, 0644); + /* Max completely idle PISA pages kept per allocator before async release. */ static unsigned int pisa_max_idle __read_mostly = 1; module_param_named(max_idle, pisa_max_idle, uint, 0644); @@ -81,6 +88,7 @@ static void pisa_release_page_rcu(struct rcu_head *head) WRITE_ONCE(meta->pa, NULL); + atomic_long_dec(&pisa_nr_pages); for (i = 0; i < PISA_BLOCKS(order); i++) { block = page + (i << order); page_ref_unfreeze(block, 1); @@ -93,6 +101,7 @@ void pisa_release_page(struct pisa_meta *meta) { if (!meta->pa) { __free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER); + atomic_long_dec(&pisa_nr_pages); return; } @@ -688,6 +697,72 @@ bool __pisa_free_page(struct page *page) return true; } +/* Shrinker: releases reservoir pages first, then idle allocator pages. */ +static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + unsigned long nr = pisa_reservoir_count(); + struct pisa_alloc *pa; + + read_lock(&pisa_allocs_lock); + list_for_each_entry(pa, &pisa_allocs, node) + nr += READ_ONCE(pa->num_idle_pages); + read_unlock(&pisa_allocs_lock); + + return nr << PISA_ORDER; +} + +/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */ +unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list, + unsigned int *nr, struct list_head *release_list, + unsigned long nr_to_scan) +{ + unsigned long freed = 0, flags; + + spin_lock_irqsave(lock, flags); + while (freed < nr_to_scan && !list_empty(list)) { + list_move(list->next, release_list); + (*nr)--; + freed += 1UL << PISA_ORDER; + } + spin_unlock_irqrestore(lock, flags); + + return freed; +} + +VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + struct pisa_meta *meta, *tmp; + struct pisa_alloc *pa; + unsigned long freed; + LIST_HEAD(release_list); + + freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan); + + if (freed < sc->nr_to_scan) { + read_lock(&pisa_allocs_lock); + list_for_each_entry(pa, &pisa_allocs, node) { + freed += pisa_shrink_list(&pa->lock, &pa->idle, + &pa->num_idle_pages, &release_list, + sc->nr_to_scan - freed); + + if (freed >= sc->nr_to_scan) + break; + } + read_unlock(&pisa_allocs_lock); + } + + /* Retiring must not run under @pa->lock. */ + list_for_each_entry_safe(meta, tmp, &release_list, list) { + list_del_init(&meta->list); + pisa_release_page(meta); + } + + /* Charge do_shrink_slab() for the full PISA pages reclaimed. */ + sc->nr_scanned = freed; + + return freed ?: SHRINK_STOP; +} + static int __init pisa_init(void) { const struct { @@ -698,10 +773,15 @@ static int __init pisa_init(void) { 2, true }, { SKB_FRAG_PAGE_ORDER, true }, }; + struct shrinker *shrink; int nid, i; pisa_node_init(); + /* Default ceiling is 1/8th of RAM (floor of 16 PISA pages). */ + if (!pisa_max_pages) + pisa_max_pages = max(totalram_pages() >> (PISA_ORDER + 3), 16UL); + /* Preallocate the supported per-node allocators. */ for_each_node_state(nid, N_MEMORY) { for (i = 0; i < ARRAY_SIZE(cfg); i++) { @@ -724,6 +804,17 @@ static int __init pisa_init(void) if (!pisa_wq) pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n"); + shrink = shrinker_alloc(0, "pisa"); + if (!shrink) { + pr_warn("pisa: shrinker registration failed\n"); + return 0; + } + + shrink->count_objects = pisa_shrink_count; + shrink->scan_objects = pisa_shrink_scan; + shrink->seeks = DEFAULT_SEEKS; + shrinker_register(shrink); + return 0; } subsys_initcall(pisa_init); diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c index bdc52e6497b05..96bc8dbc38fb4 100644 --- a/drivers/iommu/pisa-kunit.c +++ b/drivers/iommu/pisa-kunit.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -98,8 +99,10 @@ static void test_alloc_and_recycle(struct kunit *test) { 2, true }, { SKB_FRAG_PAGE_ORDER, true }, }; + struct shrink_control sc = { .nr_to_scan = ULONG_MAX }; struct page *p1, *p2, *bulk[3]; struct pisa_alloc *pa; + unsigned long old_max; unsigned int i; KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, 0, DMA_NONE)); @@ -115,6 +118,7 @@ static void test_alloc_and_recycle(struct kunit *test) pa = pisa_node_pa(first_memory_node, order, cases[i].ro); KUNIT_ASSERT_NOT_NULL(test, pa); + pisa_shrink_scan(NULL, &sc); KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT)); KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32)); KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE)); @@ -155,6 +159,7 @@ static void test_alloc_and_recycle(struct kunit *test) if (p2) __free_pages(p2, order); + pisa_shrink_scan(NULL, &sc); } /* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */ @@ -162,6 +167,16 @@ static void test_alloc_and_recycle(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, p1); KUNIT_EXPECT_FALSE(test, is_pisa_page(p1)); __free_pages(p1, 0); + + /* Global budget cap (pisa_max_pages = 0) refuses new PISA pages. */ + pa = pisa_node_pa(first_memory_node, 2, true); + KUNIT_ASSERT_NOT_NULL(test, pa); + old_max = READ_ONCE(pisa_max_pages); + WRITE_ONCE(pisa_max_pages, 0); + WRITE_ONCE(pisa_reservoir_max, 0); + KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL)); + WRITE_ONCE(pisa_reservoir_max, 16); + WRITE_ONCE(pisa_max_pages, old_max); } static void test_reservoir_and_scrubber(struct kunit *test) diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c index 524f0d4457ad7..0f23b94a7a6f8 100644 --- a/drivers/iommu/pisa-node.c +++ b/drivers/iommu/pisa-node.c @@ -74,6 +74,10 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block, return ERR_PTR(-EAGAIN); } + /* Racy check against global ceiling across allocators and reservoirs. */ + if (atomic_long_read(&pisa_nr_pages) >= READ_ONCE(pisa_max_pages)) + return ERR_PTR(-ENOSPC); + /* * Long-lived active pages must be unmovable, lowmem, non-compound, and * must not dip into emergency reserves or trigger OOM/warnings. @@ -102,6 +106,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block, meta = pisa_meta_of_pfn(page_to_pfn(page)); memset(meta, 0, sizeof(*meta)); + + atomic_long_inc(&pisa_nr_pages); *clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free(); return meta; @@ -256,6 +262,25 @@ unsigned long pisa_reservoir_count(void) return nr; } +/* Detach up to @nr_to_scan base pages from reservoirs onto @release_list. */ +unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan) +{ + unsigned long freed = 0; + int nid; + + for_each_node_state(nid, N_MEMORY) { + struct pisa_node *res = &pisa_nodes[nid]; + + if (!READ_ONCE(res->nr_pages)) + continue; + freed += pisa_shrink_list(&res->lock, &res->pages, &res->nr_pages, + release_list, nr_to_scan - freed); + if (freed >= nr_to_scan) + break; + } + return freed; +} + void __init pisa_node_init(void) { int nid; diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h index f15934ccdeea4..d5ea83745ba47 100644 --- a/drivers/iommu/pisa-priv.h +++ b/drivers/iommu/pisa-priv.h @@ -115,6 +115,16 @@ int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block); struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn); unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta); +struct shrink_control; +struct shrinker; +#if IS_ENABLED(CONFIG_KUNIT) +unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc); +#endif + +/* Global PISA page budget across all allocators and per-node reservoirs. */ +extern atomic_long_t pisa_nr_pages; +extern unsigned long pisa_max_pages; + static inline int pisa_sanitize_nid(int nid) { if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY)) @@ -149,12 +159,17 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp) return page; } +unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list, + unsigned int *nr, struct list_head *release_list, + unsigned long nr_to_scan); + /* pisa-node.c: per-NUMA-node PISA page reservoir and work item */ extern unsigned int pisa_reservoir_max; void pisa_node_init(void); void pisa_schedule_reservoir(void); struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean); unsigned long pisa_reservoir_count(void); +unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan); #endif /* CONFIG_PISA */ #endif /* _DRIVERS_IOMMU_PISA_PRIV_H */ -- 2.56.0.542.g3ef89b7020-goog