From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E299367B7C for ; Sun, 14 Jun 2026 23:37:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781480254; cv=none; b=HgdCiYVM46X4txEXWGaBCb5IZYJG2gYeara0rcumPnJQKkZ3jrlXmqn8J1f1mPVkCawTl5SoOSBtbOB5bp5qLCLqkJYEWwUTXojCgCuvuPAiXxKPzv1BSp0iV+891Fo7ZyXXEOGtRiQ2KKKpBQ9KeiuMJ9UwUHCmLC0spdDEHlk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781480254; c=relaxed/simple; bh=xMPsKlCTNd5P5cxrBc/7gJCAEAOuwwlCGWqTr1ezf7c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lSd3SptCtXZDfrLjb1bxhZmX7Gs3w13PvNvrlnav82yhFY5o1vAW2Aff4wL2CfzLr+6faZPSkmW7SWsz9LfyLOKRmWR1+GLKwWYdMXx2cQ/av59RIOzG7kpNf2V06PukwptCFZC7ALCa7Oh8M+1eWTrZaMNkbjqqy+GVX+hR9qs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NMwnVNLL; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NMwnVNLL" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-36b9106b0fbso2504835a91.0 for ; Sun, 14 Jun 2026 16:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1781480252; x=1782085052; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=96+QZxG3nyVoZV3L32zhxFaOkaRZ33wW/16vuSmDbtA=; b=NMwnVNLLoSWFPYaAmr3zlCy3uy56rrOKdcEExThiB6D61f3uoGXTV2ZTQsrZ9oYNNc RvbMwTGuX3HfN9ebC/JqHqb0qK8ZdT4X5WQ+16k/1DaehOxW3OuVMry//wOr645zy09F 2Hv1T5bgE72rVdxgVl3mJQh9aUvaOw20VHXuqgrSqi81zCb/+Sd6YStiB7pwNQzbZcrW Xa/Na/rtexWiadWjjZR193hCplMeKG0e9VwF8c5IZLR0XK33x7K2x3LqD6bpwJ0404Pn PDwXWshSI8dJisog/oO6AKBOaUgGFr7ZNosvWvCu0efFuUU2mQV0hhbwPh9RCIDUL1kV zBrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781480252; x=1782085052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=96+QZxG3nyVoZV3L32zhxFaOkaRZ33wW/16vuSmDbtA=; b=XyxcnfIVKTe/kJ/vHVACLD1ILw22mCfuKtZB5o1u/uVbPAdWdSk2SrtSB1wupMAK2T v9biBHavnlXvU86ytOHt4ObWUqiGeGVs2QLAhPg+ya8g6qvT4ne16jwpixgm5QNbLLR0 k/3/QeOItuPJaO/nrHaleitraK+La4uzEoC/kdZTme/z1w6EoRbKvNd3fBisevKG96Uh o1567QGPV3c7cm//owCq9Tf0TXy09YAeNxDvfHGlWFQ6Ymf6Ob0b81DOoZbYbgtOyu5B i82BlvCvKshkiE0K3Hx+mFFrG7X0RO5wBqfy/qlEHjE7E3MAFOizFMcPrU3eM90Q2YTj RTEQ== X-Forwarded-Encrypted: i=1; AFNElJ/HrbUJwrodpdPKcGmgPj4/vbOqTkyzWTxWGt0tEbfn7ECXWlfi5BhY4k1iR+0QHBrRljAvnJVVCwHiPtw=@vger.kernel.org X-Gm-Message-State: AOJu0YzYg5EUyLBGiWBBP4jh4Z+nwOthviCMUJCe40LH3ZEGT/3KeVVN FimRezyyc7slKTMPfcBxuM7k3tMfWeBhukUkQpR9agnOQthS1f+Cpm/AYo4MMlfPV9UuxvJGd9A N0BLFg9PCdPWNPA== X-Received: from plpo14.prod.google.com ([2002:a17:903:3e0e:b0:2b2:488e:b71]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:3510:b0:2c0:d94f:50c7 with SMTP id d9443c01a7336-2c3fb4ebb17mr94855085ad.7.1781480251720; Sun, 14 Jun 2026 16:37:31 -0700 (PDT) Date: Sun, 14 Jun 2026 23:37:13 +0000 In-Reply-To: <20260614233728.2212104-1-skhawaja@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260614233728.2212104-1-skhawaja@google.com> X-Mailer: git-send-email 2.54.0.1136.gdb2ca164c4-goog Message-ID: <20260614233728.2212104-4-skhawaja@google.com> Subject: [PATCH v3 03/18] iommu/pages: Add APIs to preserve/unpreserve/restore iommu pages From: Samiullah Khawaja To: David Woodhouse , Lu Baolu , Joerg Roedel , Will Deacon , Jason Gunthorpe Cc: Samiullah Khawaja , Robin Murphy , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Pratyush Yadav , Pasha Tatashin , David Matlack , Andrew Morton , Pranjal Shrivastava , Vipin Sharma Content-Type: text/plain; charset="UTF-8" IOMMU pages are allocated/freed using APIs using struct ioptdesc. For the proper preservation and restoration of ioptdesc add helper functions. Reviewed-by: Pranjal Shrivastava Signed-off-by: Samiullah Khawaja --- drivers/iommu/iommu-pages.c | 108 ++++++++++++++++++++++++++++++++++-- drivers/iommu/iommu-pages.h | 30 ++++++++++ 2 files changed, 134 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommu-pages.c b/drivers/iommu/iommu-pages.c index 3bab175d8557..96cc935fb4e1 100644 --- a/drivers/iommu/iommu-pages.c +++ b/drivers/iommu/iommu-pages.c @@ -6,6 +6,7 @@ #include "iommu-pages.h" #include #include +#include #include #define IOPTDESC_MATCH(pg_elm, elm) \ @@ -28,6 +29,13 @@ static inline size_t ioptdesc_mem_size(struct ioptdesc *desc) return 1UL << (folio_order(ioptdesc_folio(desc)) + PAGE_SHIFT); } +static inline void iommu_folio_update_stats(struct folio *folio, + unsigned long nr_pages) +{ + mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, nr_pages); + lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, nr_pages); +} + /** * iommu_alloc_pages_node_sz - Allocate a zeroed page of a given size from * specific NUMA node @@ -80,8 +88,7 @@ void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t size) * rather large, i.e. multiple gigabytes in size. */ pgcnt = 1UL << order; - mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, pgcnt); - lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, pgcnt); + iommu_folio_update_stats(folio, pgcnt); return folio_address(folio); } @@ -95,8 +102,7 @@ static void __iommu_free_desc(struct ioptdesc *iopt) if (IOMMU_PAGES_USE_DMA_API) WARN_ON_ONCE(iopt->incoherent); - mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, -pgcnt); - lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, -pgcnt); + iommu_folio_update_stats(folio, -pgcnt); folio_put(folio); } @@ -131,6 +137,100 @@ void iommu_put_pages_list(struct iommu_pages_list *list) } EXPORT_SYMBOL_GPL(iommu_put_pages_list); +#if IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE) +/** + * iommu_unpreserve_pages - Unpreserve pages that were preserved in KHO + * @virt: Virtual address of page to be unpreserved + */ +void iommu_unpreserve_pages(void *virt) +{ + kho_unpreserve_folio(ioptdesc_folio(virt_to_ioptdesc(virt))); +} +EXPORT_SYMBOL_GPL(iommu_unpreserve_pages); + +/** + * iommu_preserve_pages - Preserve pages using KHO + * @virt: Virtual address of page to preserve + * + * Returns 0 on success, negative error on failure + */ +int iommu_preserve_pages(void *virt) +{ + return kho_preserve_folio(ioptdesc_folio(virt_to_ioptdesc(virt))); +} +EXPORT_SYMBOL_GPL(iommu_preserve_pages); + +/** + * iommu_unpreserve_pages_list - Unpreserve a list of KHO preserved pages + * @list: List of pages to unpreserve + */ +void iommu_unpreserve_pages_list(struct iommu_pages_list *list) +{ + struct ioptdesc *iopt; + + list_for_each_entry(iopt, &list->pages, iopt_freelist_elm) + kho_unpreserve_folio(ioptdesc_folio(iopt)); +} +EXPORT_SYMBOL_GPL(iommu_unpreserve_pages_list); + +/** + * iommu_restore_pages - Restore pages that were preserved in KHO + * @phys: Physical address of page to restore + */ +void iommu_restore_pages(u64 phys) +{ + struct ioptdesc *iopt; + struct folio *folio; + unsigned long pgcnt; + unsigned int order; + + folio = kho_restore_folio(phys); + BUG_ON(!folio); + + iopt = folio_ioptdesc(folio); + + /* + * For the restored pages incoherent is set to false as these are not + * mapped using the DMA_API. The remapping of these pages using DMA_API + * is not needed as these are not going to be written to by the new + * kernel. + */ + iopt->incoherent = false; + + order = folio_order(folio); + pgcnt = 1UL << order; + iommu_folio_update_stats(folio, pgcnt); +} +EXPORT_SYMBOL_GPL(iommu_restore_pages); + +/** + * iommu_preserve_pages_list - Preserve list of pages using KHO + * @list: List of pages to preserve + * + * Returns 0 on success, negative error on failure + */ +int iommu_preserve_pages_list(struct iommu_pages_list *list) +{ + struct ioptdesc *iopt; + int ret; + + list_for_each_entry(iopt, &list->pages, iopt_freelist_elm) { + ret = kho_preserve_folio(ioptdesc_folio(iopt)); + if (ret) + goto err; + } + + return 0; + +err: + list_for_each_entry_continue_reverse(iopt, &list->pages, iopt_freelist_elm) + kho_unpreserve_folio(ioptdesc_folio(iopt)); + + return ret; +} +EXPORT_SYMBOL_GPL(iommu_preserve_pages_list); +#endif + /** * iommu_pages_start_incoherent - Setup the page for cache incoherent operation * @virt: The page to setup diff --git a/drivers/iommu/iommu-pages.h b/drivers/iommu/iommu-pages.h index ae9da4f571f6..adf597306145 100644 --- a/drivers/iommu/iommu-pages.h +++ b/drivers/iommu/iommu-pages.h @@ -53,6 +53,36 @@ void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t size); void iommu_free_pages(void *virt); void iommu_put_pages_list(struct iommu_pages_list *list); +#if IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE) +int iommu_preserve_pages(void *virt); +void iommu_unpreserve_pages(void *virt); +int iommu_preserve_pages_list(struct iommu_pages_list *list); +void iommu_unpreserve_pages_list(struct iommu_pages_list *list); +void iommu_restore_pages(u64 phys); +#else +static inline int iommu_preserve_pages(void *virt) +{ + return -EOPNOTSUPP; +} + +static inline void iommu_unpreserve_pages(void *virt) +{ +} + +static inline int iommu_preserve_pages_list(struct iommu_pages_list *list) +{ + return -EOPNOTSUPP; +} + +static inline void iommu_unpreserve_pages_list(struct iommu_pages_list *list) +{ +} + +static inline void iommu_restore_pages(u64 phys) +{ +} +#endif + /** * iommu_pages_list_add - add the page to a iommu_pages_list * @list: List to add the page to -- 2.54.0.1136.gdb2ca164c4-goog