From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.214.200 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792894779; b=VWz3SON8WusXyOBtKv94P1Aj057RSnYctCxAWnTtmhJXpZv58j5wJGNjB0aZpFiebaOV dZmlQdw/BXD0aHw9CnY7pcOEnT6/6UsFU/xxk9UvKHkl+qVqyjVN+qMwsdwNvWz3YOJ0B S00ww++1+M3x/QxH9t2FVSg/sY0VSUz1s3/8ko6zBVrMc2/X4iQms9dmssFGdUvCD3eLi qN8kTBxvuEU1vvNKQ/UsyUHRm4lrdG8Dub7KT2XX66m8IkW/loZoEmueUVecczDJmBJS7 WW8kT2i6oHInnoFOLx7VZq6XiK+b0Y2Li3SptgzPfDFE3INhbz5s52toxrj1mmY1ZIA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792894779; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=4tNPBuMcA871ilhSXZeeD0cmUC+Z67Q8xk9SGMtX3nE=; b=KfGLF0cf0CWuLO4qZazaKtsb7RqruIQ9rkW52NgZoBvH9F0EmR3J69IKS1Ikbbdv19vZ 7msupvmxzfvlpXtQaXLX8+yHDk3D6JrWdrH73kAODiK9knzD8q1SR51apC7c2MGZpwV+x el2/cQx0KLhpl4Slf+WNzlAtL9HZH14EsA+QHPUeKqIfHEW8T7TV/kBTJSDCaQ4/O4kAF h63EUw/JA4jbIBSq3q7XmeIClI/ul7n3ykOJnSLH7fo02A1Ifq8FzkTWxfz535trXEOXV TaIxoLWqABw8xTUsW6h+VDHlyBYSHCbgoD8QeN926dZykiSYjaIasCU+E8TpWUBLQtw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=UXpHr79s; arc=none smtp.remote-ip=209.85.214.200 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F5E560A2639 for ; Sun, 25 Oct 2026 02:19:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 3F5E560A2639 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=UXpHr79s Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2e93f67358eso20180185ad.1 for ; Sat, 24 Oct 2026 19:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792894779; x=1793499579; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=4tNPBuMcA871ilhSXZeeD0cmUC+Z67Q8xk9SGMtX3nE=; b=UXpHr79syOT95JBFaVbMBLfYVb7dfrWWZt3qGnzCF+kz6415F7rsc8PIh4+k6W9Yxt APX29Rucy5n7BRSJPqJ2edbVQVZfDrmKMJiJ89HsILsDp1xG52LGfDnKtWBGh5e7QYok voBFlHFbJuOOc7niLK+BH6Uu2wdu0B3/eLwm5laBHcAMKFJ1H4AF4w6TsQxvTaJoNJsT toOc6aSdAKGe0DUdMoF/hPPFODFSCalUEict8SE49HQ3h1aBck4Hj0yG/w/EpoxwJRwg DzHIO2fkAFT9fUoH0qWygmHQ7mfHUrjKKjbmGTmKBOf+7L6c3Iflt3TK9+U8XYLwMT65 lqYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792894779; x=1793499579; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4tNPBuMcA871ilhSXZeeD0cmUC+Z67Q8xk9SGMtX3nE=; b=L0fx9vCNUMgA+8mO64B4wbKQP9RCIde/pWOi6F8ywU+vk3S/nrEK6/MpoUjAcAWg2r GE7ujT/xOF4MekNuxLFKECZdBXfnqeyz7p6K9SudfOShtiUeZIBiPpDSOBlspz85+BAA ziDMAdZjp+gYlOfO8jOXax8p48MnFtwwmN2tpYK2ZhpTuS8Qv3rwG7SxOjH6BbNTw7p0 avaqnatkltKY2+f9+9/FhCovohMbH7TQQS9LnruQH83bb82HpZQ9SMt+pJ2yekyI5iuB k1kG+YfjMSmwvcUYXY2onZSQUrSJSvmr9+9gbUzMMZx+uIsu2G31VVT7iileIaCM+Zj9 NnqQ== X-Forwarded-Encrypted: i=1; AKwUvBzKjicCUOsnaAREYb1RZPr/0H9iD3RBnLAoF48KowP30kI6zD2T6MLC98s8nDDCvVo8tjr4+xq1KR7LEsA=@vger.kernel.org X-Gm-Message-State: AFq9FYJvJtqbPXv5PPQxq/GATXUzAo9EzseTBtNGiqKvaSzBf+fwmn2v 7gVyc44GUUeuIdp7vHn/tljVfKeKB1lDf/lrVUwSaBFj9H8QuqJYlvwRHv5kPGjxU43zKq4ynNg vpikwqV7wXBdGpg== X-Received: from plka14.prod.google.com ([2002:a17:903:f8e:b0:2e2:f8c9:e138]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:dac4:b0:2e5:cf33:80ca with SMTP id d9443c01a7336-2ea9c96ded1mr25523625ad.52.1792894778208; Sat, 24 Oct 2026 19:19:38 -0700 (PDT) Date: Sun, 25 Oct 2026 02:19:14 +0000 In-Reply-To: <20261025021930.674647-1-skhawaja@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261025021930.674647-1-skhawaja@google.com> X-Mailer: git-send-email 2.56.0.257.gf7497c7d27-goog Message-ID: <20261025021930.674647-4-skhawaja@google.com> Subject: [PATCH v6 03/18] iommupt: Implement preserve/unpreserve/restore callbacks From: Samiullah Khawaja To: Joerg Roedel , Will Deacon , Lu Baolu , Jason Gunthorpe , David Woodhouse Cc: Samiullah Khawaja , Robin Murphy , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Pratyush Yadav , Pasha Tatashin , David Matlack , Andrew Morton , Pranjal Shrivastava , Vipin Sharma , Ankit Soni , Nicolin Chen , John Starks Content-Type: text/plain; charset="UTF-8" Add iommupt ops for presevation, unpresevation and restoration of iommu page tables for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables. Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore, the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features. Signed-off-by: Samiullah Khawaja --- drivers/iommu/generic_pt/iommu_pt.h | 155 ++++++++++++++++++++++ drivers/iommu/generic_pt/kunit_iommu_pt.h | 30 +++++ include/linux/generic_pt/iommu.h | 30 +++++ 3 files changed, 215 insertions(+) diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h index 07ec2b3ab986..716eca2c47e1 100644 --- a/drivers/iommu/generic_pt/iommu_pt.h +++ b/drivers/iommu/generic_pt/iommu_pt.h @@ -16,6 +16,7 @@ #include "../iommu-pages.h" #include #include +#include enum { SW_BIT_CACHE_FLUSH_DONE = 0, @@ -1005,6 +1006,155 @@ static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova, return ret; } +#ifdef CONFIG_IOMMU_LIVEUPDATE +static void NS(unpreserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range = pt_all_range(common); + struct pt_iommu_collect_args collect = { + .pending.free_list = IOMMU_PAGES_LIST_INIT( + collect.pending.free_list), + }; + + iommu_pages_list_add(&collect.pending.free_list, range.top_table); + + /* + * pt_walk_range() will never fail as check_mapped is not set in the + * collect args. + */ + pt_walk_range(&range, __collect_tables, &collect); + + iommu_unpreserve_pages_list(&collect.pending.free_list); +} + +static int NS(preserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range = pt_all_range(common); + struct pt_iommu_collect_args collect = { + .pending.free_list = IOMMU_PAGES_LIST_INIT( + collect.pending.free_list), + }; + int ret; + + iommu_pages_list_add(&collect.pending.free_list, range.top_table); + + /* + * pt_walk_range() will never fail as check_mapped is not set in the + * collect args. + */ + pt_walk_range(&range, __collect_tables, &collect); + + ret = iommu_preserve_pages_list(&collect.pending.free_list); + if (ret) + return ret; + + ser->top_table_phys = virt_to_phys(range.top_table); + ser->top_level = range.top_level; + + /* + * VASZ and SIGN_EXTEND will be needed in next kernel for collector page + * table walk to restore and free pages. + * + * Use the max_vasz_lg2 from range, as that is the current one if + * DYNAMIC_TOP is supported. + */ + ser->max_vasz_lg2 = range.max_vasz_lg2; + ser->sign_extend = pt_feature(common, PT_FEAT_SIGN_EXTEND); + + return 0; +} + +static int __restore_tables(struct pt_range *range, void *arg, + unsigned int level, struct pt_table_p *table) +{ + struct pt_state pts = pt_init(range, level, table); + int ret; + + if (!pt_can_have_table(&pts)) + return 0; + + for_each_pt_level_entry(&pts) { + if (pts.type == PT_ENTRY_TABLE) { + iommu_restore_pages(virt_to_phys(pts.table_lower)); + + /* + * pt_descend can only fail if pts.table_lower is not + * init. So the if statement below is dead code. + */ + ret = pt_descend(&pts, arg, __restore_tables); + if (ret) + return ret; + } + } + + return 0; +} + +static void NS(deinit)(struct pt_iommu *iommu_table); +static const struct pt_iommu_ops NS(ops_immutable) = { + .deinit = NS(deinit), +}; + +static int NS(restore)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range; + + if (ser->max_vasz_lg2 > common->max_vasz_lg2 || + ser->top_level > PT_MAX_TOP_LEVEL) + return -EINVAL; + + /* + * Override the max_vasz_lg2 here to the one calculated using top_range + * in previous kernel. This makes sure that any later page walks use the + * clamped max_vasz_lg2 instead of using the full hardware maximum + * max_vasz_lg2. + */ + common->max_vasz_lg2 = ser->max_vasz_lg2; + + /* + * Restored page tables are strictly transient and only permitted to be + * destroyed via deinit() op. Because we only preserve user-assigned + * devices utilizing pass-through frameworks (VFIO / IOMMUFD), any + * concurrent or subsequent map/unmap operations on a restored domain + * are explicitly blocked at the subsystem boundary (e.g., via IOAS + * immutability). + */ + iommu_table->ops = &NS(ops_immutable); + + /* + * It is safe to override this here since this domain is immutable and + * can only be freed. This also means that for non-coherent pages DMA + * mappings will not be created using the DMA API. + * + * Note that the FMT specific bits are not cleared as they might be + * set/preserved/restored by the driver to handle special cases that are + * required for page walks. + */ + common->features &= ~GENMASK(PT_FEAT_FMT_START - 1, 0); + if (ser->sign_extend) + common->features |= BIT(PT_FEAT_SIGN_EXTEND); + + range = pt_all_range(common); + iommu_restore_pages(ser->top_table_phys); + + /* + * The old top_table can be freed here as it is not used for any IOMMU + * mappings yet, because the restore happens during device probe. + */ + iommu_pages_free_incoherent(range.top_table, + iommu_table->iommu_device); + + /* Set the restored top table */ + pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level); + + /* Restore all pages*/ + range = pt_all_range(common); + return pt_walk_range(&range, __restore_tables, NULL); +} +#endif + struct pt_unmap_args { struct iommupt_pending_gather pending; pt_vaddr_t unmapped; @@ -1188,6 +1338,11 @@ static const struct pt_iommu_ops NS(ops) = { #endif .get_info = NS(get_info), .deinit = NS(deinit), +#ifdef CONFIG_IOMMU_LIVEUPDATE + .preserve = NS(preserve), + .unpreserve = NS(unpreserve), + .restore = NS(restore), +#endif }; static int pt_init_common(struct pt_common *common) diff --git a/drivers/iommu/generic_pt/kunit_iommu_pt.h b/drivers/iommu/generic_pt/kunit_iommu_pt.h index 9fc71db74234..7306de44eb00 100644 --- a/drivers/iommu/generic_pt/kunit_iommu_pt.h +++ b/drivers/iommu/generic_pt/kunit_iommu_pt.h @@ -427,6 +427,35 @@ static void test_mixed(struct kunit *test) check_iova(test, start, oa, len); } +static void test_restore_free(struct kunit *test) +{ + struct kunit_iommu_priv *priv = test->priv; + struct pt_range top_range = pt_top_range(priv->common); + u64 start = 0x3fe400ULL << 12; + u64 end = 0x4c0600ULL << 12; + pt_vaddr_t len = end - start; + + if (top_range.last_va <= start || sizeof(unsigned long) == 4) + kunit_skip(test, "range is too small"); + if ((priv->safe_pgsize_bitmap & GENMASK(30, 21)) != (BIT(30) | BIT(21))) + kunit_skip(test, "incompatible psize"); + + /* Map a large mixed range to populate multiple levels of page tables */ + do_map(test, start, start, len); + + /* + * Simulate a restored state by clearing all features except SIGN_EXTEND + * and DMA_INCOHERENT. This verifies that the generic page table free + * walker can correctly tear down a populated domain when other features + * are zeroed. Also set max_vasz_lg2 as done by the actual restore() + * function. + */ + priv->common->features &= (BIT(PT_FEAT_SIGN_EXTEND) | BIT(PT_FEAT_DMA_INCOHERENT)); + priv->common->max_vasz_lg2 = top_range.max_vasz_lg2; + + /* The domain will be freed when the test exits. */ +} + static struct kunit_case iommu_test_cases[] = { KUNIT_CASE_FMT(test_increase_level), KUNIT_CASE_FMT(test_map_simple), @@ -435,6 +464,7 @@ static struct kunit_case iommu_test_cases[] = { KUNIT_CASE_FMT(test_random_map), KUNIT_CASE_FMT(test_pgsize_boundary), KUNIT_CASE_FMT(test_mixed), + KUNIT_CASE_FMT(test_restore_free), {}, }; diff --git a/include/linux/generic_pt/iommu.h b/include/linux/generic_pt/iommu.h index 13ebb72a67de..617fb9af6bef 100644 --- a/include/linux/generic_pt/iommu.h +++ b/include/linux/generic_pt/iommu.h @@ -13,6 +13,7 @@ struct iommu_iotlb_gather; struct pt_iommu_ops; struct pt_iommu_driver_ops; struct iommu_dirty_bitmap; +struct iommu_domain_ser; /** * DOC: IOMMU Radix Page Table @@ -166,6 +167,35 @@ struct pt_iommu_ops { * table from all HW access and all caches. */ void (*deinit)(struct pt_iommu *iommu_table); + + /** + * @preserve: Preserve the iommu page table for liveupdate + * @iommu_table: Table to preserve + * @ser: Serialization struct to fill with preserved state + * + * Preserve iommu page table and the relevant state for liveupdate. The + * caller must make sure that the page table is not updated during and + * after preservation. + */ + int (*preserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + + /** + * @unpreserve: Unpreserve the iommu page table + * @iommu_table: Table to unpreserve + * @ser: Serialization struct that contains preserved state + */ + void (*unpreserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + + /** + * @restore: Restore the iommu page table after liveupdate + * @iommu_table: Table to restore the state into + * @ser: Serialization struct that contains preserved state + * + * The iommu_table is back filled with the restored state that was + * preserved in the serialization struct. + */ + int (*restore)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + }; /** -- 2.56.0.257.gf7497c7d27-goog