From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.128.53 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794423576; b=VinG30MzTUYVBGFp0rhhpLIboz3BaU1VLx+TTOHgzSovMUlvKiKs5X45Q8fUoCOhja4V ElOgVUkuet3rzIbF8AMwFmTWYkShp/j7pFE5/0S3DNQcGgHwwEd+WWovpntklh9W3QoUW 4yLV1eO5j7La7kBd0GWhNejLqV6P0QwdNOC5lq6YBwcgJOFX0yfhDhukmhFCOEHpvMHBy aTJwlM0jCZLe9FrAl4fR9dZuBkNlkFDu2wjWDdGR7fEGcdCBaFuDZ1TxjaDGt2tDsq4O7 oSU6OLFZWze9EdhXD68ZB6COOtNGnupkM5jdkO7GLJF2RRzs24IPGZWhrnlVDw5KnPw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794423576; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=bNaWmjzZL0cMs+uJYhUIAi6U7mf0NFDrLgrotrfyBqc=; b=4IT2OFagl9PQIhtb4rnwYXKLM7i+YTy1TXjJ993mjaJY4D0gyCi3Larlhb/UEBlOh4/q ouonK7ljn4TF5J+/FjM67GX7bBeb7GfJf0BKpSwtyPB8VGO8TdW0wPWgqqksUqfjCIM4s 9BTi1A4QuCnC0w478kDl6AgHPhL7vpFcZNssPe2NdyjOLSet/AQ4WHxdotJCCI7PUL6Nx ZTGXm6JHeutP7CvDTrdRaFnznrbz5m6h2sbfjDOXkgHGY83llyD+BUCM0x7DS8m4EFdFZ kNIH4JPALnZuDmjatJrP4FRQ/EaTZCeUv4ntVPnzquituJPqmc2Y7XHa3apSVGhaqSA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=LUvRyQPn; arc=none smtp.remote-ip=209.85.128.53 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 988A8606A337 for ; Wed, 11 Nov 2026 18:59:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 988A8606A337 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=LUvRyQPn Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4b79502c27aso34997365e9.1 for ; Wed, 11 Nov 2026 10:59:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794423574; x=1795028374; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bNaWmjzZL0cMs+uJYhUIAi6U7mf0NFDrLgrotrfyBqc=; b=LUvRyQPnrpphOY/2BqfF9pJnRMvSpapj4ntjHVC+N+B9p++8Hw/+VZiKChctsKSjnk pYUftqd4+zEI8XO6nu+0c939JedO2legBCLD9m9kBCoGuZSu/cw3998tHpOSIa5kzWwB uT6Oafs7RjfsmBvFjxGcd/d7TzTAMTtakWf23cNaPISaL4QILByPN+URkqQnFqArmA+B xdOCp2zz5Rifg9/TeSx9WpdCbXaPAR8owdLhFyI4vgxh68ySPMRyctdLDha5q3OyE+fA sh6F+W/oyPVbWi8XWWcvzddQzbhpYOnn0uIZt4D8CK80MR9NWgiTSvhwXpVFsJEuVKTZ s5/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794423574; x=1795028374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=bNaWmjzZL0cMs+uJYhUIAi6U7mf0NFDrLgrotrfyBqc=; b=reaUnFlcKoLP1jeD0Y0+IKrgX+Luv5tk4DrZGpWiUVv2F1HhoHL8ss6Jym0G5mnXHl 82klY2EROv05W1fNJocPlFlJAC1TIQHrnolJzyZ6J3g9hr1JU0XpKa0wnwT42Wsd3qJu CH2wDfNyJTnenLNXJ7YmNzOTizdoDax3ckzlof44jDoIm7UflrGjenvTEsckzF+qW7Ze G2UqS4E3T7HkHDocIAMyWe8DVr8fTnmi6PnRpeC4+OQGDuhy/YTVg225l8eJmX3QvQ6q f93PvK0amtI4vnXIqEHfO1YXQas2BYCG5XlyRLZkLICGHEOrjbF6/pLVlCXuRtA7XW11 keXg== X-Forwarded-Encrypted: i=1; AL+hErEtW+OY+KfXJSbNK2RTUnSiqHv0UXuDvcaO/b08XFfgdN7Y/RDi02RvOAoMgreNbEcJ1RU7r+9h42nTKOI=@vger.kernel.org X-Gm-Message-State: APuU4Z+CWJdvjyVenCTZZapMsX4uLafY5Phc7T/GzM1ObspUmXbwRTFR o+gWI6p9Pu3a/lUXuBQPGq2HOJX/bOh/ZwlTFBWhMm73vQDrjvr2pEwC X-Gm-Gg: AYBFou3jFV5Aq+HnbJHCclCEk4okL45p86EZHJXjOuBM7pCuKI2UdOXnpLnLt4BTw/n tj9xIvB/obBL3FheiUXUKBlKsLvwTf2Kq/xdcpokdzHpppyjpkj0YMFZml/gUxD6AJKVpEjxa+Z ew5A8Ja7X1hJwyGN45DumhDd9R8yP+vQBMaOHr/V35NdjU0VmsS9u/8ULWqxNVz41qZhGCCIonn Uzj5wIMuXi7UWYgSHwtcoaTZNiefxugCa55b7euW8bAPMJVT3+Y9AjkX6JrkdhlBQ5I+Eqp2GV8 zQrOw4LvA8W8z9y4Z5sKfZuI9JH/bvgLf/uRub32omHNOPyrXpcLKe40TZ6zCXAH5C4/9i2zKdY VgTSM8MBxmXj6xhKkVOutExJ6lHqWor05UXBuYZCDVgKVfTyLD6giFVPPicMxLZBEiBDE5X3Nrw iLiZdK/5di4EoM7n8zlMAl4/kveu6p8djy6cdGeZq3D1er9uzBIG/Y1H9qhDtLAxqwS+nX/Pf5P Tj2UFGtTT6zP+b2LFhMJK9hRgAxKKRaMM7cocIFV7J2QvUJo5FrzxKDq4pFbRw1DkN4WjbrAUpe xb6KWwNgK2K3ONID9bssYeSKwQpGQamFrRhhucI5txK0jZFSTc/xgmxHMTa/CHdZZriuXNEcdIJ ce0sYJNokVYbivbLdHTYLYHhB1MykDxb6wC/bBwmyMGKpyLxF260Y0Kv7m7sFzBnuag== X-Received: by 2002:a05:600c:81c5:b0:4a4:3a2d:15da with SMTP id 5b1f17b1804b1-4ba093753a6mr56638565e9.15.1794423573896; Wed, 11 Nov 2026 10:59:33 -0800 (PST) Received: from dev-dsk-fgriffo-1c-93421965.eu-west-1.amazon.com (54-240-197-234.amazon.com. [54.240.197.234]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4bb4e265a0csm9161435e9.9.2026.11.11.10.59.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Nov 2026 10:59:33 -0800 (PST) From: Fred Griffoul To: Joerg Roedel , Will Deacon , Jason Gunthorpe , Kevin Tian Cc: Robin Murphy , Shuah Khan , Mostafa Saleh , Guanghui Feng , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, nh-open-source@amazon.com Subject: [PATCH 2/3] iommufd: Use iommu_unmap_sparse() to unpin domain pages Date: Wed, 11 Nov 2026 18:59:30 +0000 Message-ID: <20261111185931.94882-3-griffoul@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261111185931.94882-1-griffoul@gmail.com> References: <20261111185931.94882-1-griffoul@gmail.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Fred Griffoul iommufd keeps no copy of the PFNs it pinned for an area, so iopt_area_unpin_domain() reads each one back from the domain with iommu_iova_to_phys(), one CPU page at a time, and then unmaps the range in a second walk. With 4 KiB pages, a 1 GiB IOPTE costs 262144 lookups before the unmap starts. Call iommu_unmap_sparse() instead, and add the PFNs of each IOPTE it removes to the pfn_batch, so one walk both unmaps the range and collects the PFNs. The last IOPTE removed can hold pages beyond the ones this pass unpins. The loop already handles that case: it keeps those pages in the batch and carries them over to the next pass. Domains that do not use iommupt keep the per-page read-back, and so do areas whose IOVA or memory starts or ends inside a CPU page. Such areas exist only when the smallest page size of every domain in the IOAS is smaller than PAGE_SIZE. Signed-off-by: Fred Griffoul --- drivers/iommu/iommufd/pages.c | 98 +++++++++++++++++++++++++++++++++-- 1 file changed, 93 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c index 2f6153108add7..c2a04a8845cc8 100644 --- a/drivers/iommu/iommufd/pages.c +++ b/drivers/iommu/iommufd/pages.c @@ -468,6 +468,79 @@ static void batch_from_domain_continue(struct pfn_batch *batch, batch->array_size = array_size; } +struct batch_unmap { + struct pfn_batch *batch; + unsigned long first_iova; + unsigned long last_iova; +}; + +static bool batch_record_unmapped(void *priv, unsigned long iova, + phys_addr_t paddr, size_t size) +{ + struct batch_unmap *unmap = priv; + unsigned long first, end; + + /* Stop after the entry that holds the last page to unpin */ + if (iova > unmap->last_iova) + return false; + + /* + * Count each CPU page once, from the entry that holds its first byte. + * Pages before first_iova belong to an access and stay pinned. + */ + first = max(ALIGN(iova, PAGE_SIZE), unmap->first_iova); + end = ALIGN(iova + size, PAGE_SIZE); + if (first >= end) + return true; + return batch_add_pfn_num(unmap->batch, PHYS_PFN(paddr + (first - iova)), + (end - first) >> PAGE_SHIFT, BATCH_CPU_MEMORY); +} + +/* True if the area starts and ends on CPU pages, in both IOVA and memory */ +static bool iopt_area_pages_aligned(struct iopt_area *area) +{ + return !area->page_offset && + PAGE_ALIGNED(iopt_area_iova(area) | + (iopt_area_last_iova(area) + 1)); +} + +/* + * Unmap from unmapped_index and collect the PFNs from start_index on into the + * batch in one walk. The walk ends after the entry that holds last_index, which + * can reach past last_index, or before an entry the batch has no room for. + * Returns the number of indexes unmapped, 0 if the caller has to read the PFNs + * back with batch_from_domain() instead. + */ +static unsigned long batch_unmap_from_domain(struct pfn_batch *batch, + struct iommu_domain *domain, + struct iopt_area *area, + unsigned long unmapped_index, + unsigned long start_index, + unsigned long last_index, + unsigned long real_last_index) +{ + unsigned long start = max(start_index, unmapped_index); + struct batch_unmap unmap = { + .batch = batch, + .first_iova = iopt_area_index_to_iova(area, start), + .last_iova = iopt_area_index_to_iova_last(area, last_index), + }; + unsigned long iova = iopt_area_index_to_iova(area, unmapped_index); + unsigned long last = + iopt_area_index_to_iova_last(area, real_last_index); + size_t unmapped; + + if (!iopt_area_pages_aligned(area)) + return 0; + + /* The range runs to the area end, batch_record_unmapped() stops it */ + unmapped = iommu_unmap_sparse(domain, iova, last - iova + 1, + batch_record_unmapped, &unmap); + if (IS_ENABLED(CONFIG_IOMMUFD_TEST)) + WARN_ON(unmapped % PAGE_SIZE); + return unmapped >> PAGE_SHIFT; +} + /* * This is part of the VFIO compatibility support for VFIO_TYPE1_IOMMU. That * mode permits splitting a mapped area up, and then one of the splits is @@ -1689,19 +1762,33 @@ iopt_area_unpin_domain(struct pfn_batch *batch, struct iopt_area *area, { while (start_index <= last_index) { unsigned long batch_last_index; + bool read_ahead = true; if (*unmapped_end_index <= last_index) { unsigned long start = max(start_index, *unmapped_end_index); + unsigned long unmapped, batch_end; if (IS_ENABLED(CONFIG_IOMMUFD_TEST) && batch->total_pfns) WARN_ON(*unmapped_end_index - batch->total_pfns != start_index); - batch_from_domain(batch, domain, area, start, - last_index); - batch_last_index = start_index + batch->total_pfns - 1; + unmapped = batch_unmap_from_domain(batch, domain, area, + *unmapped_end_index, + start_index, + last_index, + real_last_index); + if (unmapped) { + /* The walk ended on an entry boundary */ + *unmapped_end_index += unmapped; + read_ahead = false; + } else { + batch_from_domain(batch, domain, area, start, + last_index); + } + batch_end = start_index + batch->total_pfns - 1; + batch_last_index = min(batch_end, last_index); } else { batch_last_index = last_index; } @@ -1716,9 +1803,10 @@ iopt_area_unpin_domain(struct pfn_batch *batch, struct iopt_area *area, * middle of the domains we need to keep reading pfns until we * find a cut point to do the unmap. The pfns we read are * carried over and either skipped or integrated into the next - * batch. + * batch. batch_unmap_from_domain() already ends on an entry + * boundary and needs no read-ahead. */ - if (batch_last_index == last_index && + if (read_ahead && batch_last_index == last_index && last_index != real_last_index) batch_from_domain_continue(batch, domain, area, last_index + 1, -- 2.47.3