From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.167.197 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794283718; b=5PMH6plMVsTJCW+n+r9CXRHxv4ihgk2inCja6uqh/Gd4FNz26318rzWhL06XXAHpMUzC tVGFRA0BTDHeabqkSP0XKf5ZhjGDPsydfpHMbBYWivawsMuRRWfTK2Sl9Z31EgGsvxKf2 xWfrySfik5i0+ALgFEcRCJD8gyusJruFJ+PNpIIEG/BwUUBTHvgQLw7Erl+yXrpKPkeI1 C1XQHv/CIseXQYANVcYsVrs/f28vIZDdCGj7rVXnR6BgHLbcvMfWpLcUJM9LX2Za4+Rco IKqAUtQrkZjtShKekVtMJgr3erUk+DwoZFA0zMDqJVxpSzCzr1yTjvFs0EV++hxu5fg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794283718; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=B9lAyV/x5/hfGaUSiw6mbDUfBF5DuSO6OL83gmt2yFA=; b=FWlQHFA3tsfRe2vfl9uncAKaJIEh3i8PicSKjutDFUPPgEPOB5b48fEm0/avXwXyoduf +cuTthizIxYEY70AgOFFyfbOZLytxHysv0NnMHa85SraGkr+X960rc8DZHXChPMlZYjjg zTIGpBCQpbWmmf6Qyo6QAnGrtqhnHE+0hTX1aFUZiZNNgItLPh2p+oRTP7AG4y41HsKLS GiG97tA65DkfAzvAg29HBO9zAsVHVg4hvertAOJnzCHtZzNS/WsgJwbPccNke7Hm3GSNp usUkFR70vgqRUjU5//wEMD7GT83hEQE+h60URZSurAcYeb7mmu0pOEIS5J11y8nwwRg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=MieX1COt; arc=none smtp.remote-ip=209.85.167.197 Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1135F60A2629 for ; Tue, 10 Nov 2026 04:08:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 1135F60A2629 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--skhawaja.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=MieX1COt Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-4dab94dc3e7so2959165b6e.0 for ; Mon, 09 Nov 2026 20:08:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1794283716; x=1794888516; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=B9lAyV/x5/hfGaUSiw6mbDUfBF5DuSO6OL83gmt2yFA=; b=MieX1COtFeYfJ07qmAUo3QP7BakfoyE/XRoVphWWjsEwweBUj/ZfruwN0SL1l4sYER iIuCH3rqmrUvfvM65mdI1grCiV232bup+qkU1mx9Nv675FKJgSiajRUkTfRLmFGqB/Zw nb4wXSNzovq+4VhYfeyrKefXxqXT1XnxX484v4I3yz765F47Zfh4MSzT6FZ0OxZt0ym+ auHJQVRR8bHNqUxBBGnhKjdgPCiP7i1TBUTZsyoNmdMz5OXdXQh4yhY5sRlfyX2mu3/F aWB2DDIUYm8aindZ4/fJyhs8eRVMCHVwADgrN+ogOzsAJ91UqcI87OmCQxgF1GSDAJmF +FJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794283716; x=1794888516; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=B9lAyV/x5/hfGaUSiw6mbDUfBF5DuSO6OL83gmt2yFA=; b=v9mkEa2Z616+ISifT1AuE1Sj8zFOArSj6nE9UOjysL5kzsWMxziuMbuNXtFw8TjHKH pMx8DPTO8kMtWRdJuVphiOgdv7n3MMAMd3zDuptBHA9NfZ7svJ2+QTREH+2u1IjPuyRG ICJm7nP8iKZL10cbEfsjVVwZPEcwQ2tTmcLfstDEaa+LrT97SLOHZCcUWBoBM8VbXZRo DtyqX6Oq5H0kfaplNIizmzvB00VuYvjCpEmV8XZpY/P9yQcyeKVPVuFRHpJc7FROWcKC VUTYiGCIKJDMZn+7mLMs0Q67LlSw+jbA64dJX8i6UvDws4dePWl4HA8g5xGGgIXpMwii 7HbQ== X-Forwarded-Encrypted: i=1; AL+hErFddZ86Hrt9fHNUZhfD89WgX1Va5dP/jVvF6me1LKaeJqseO0+xBWziEd18+zTp8Dfqw9ZEQOaj357c960=@vger.kernel.org X-Gm-Message-State: APuU4Z90dH1QfnwhbG2BVgupC99n7kgNEQop1fgABwwWevMs75iseTSV rVYnJ+hQMH8IHdc5IPBhESsxZL+YG0lMmLEf1pVEAKYe12XP0Djn00krBHXR+HLIjIUFzntZqF0 u65D8IJiy7PYbNQ== X-Received: from ilpy14.prod.google.com ([2002:a92:c74e:0:b0:542:e9f4:1733]) (user=skhawaja job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6808:f12:b0:4ec:d521:9124 with SMTP id 5614622812f47-5683cc817ffmr1095684b6e.30.1794283715550; Mon, 09 Nov 2026 20:08:35 -0800 (PST) Date: Tue, 10 Nov 2026 04:08:09 +0000 In-Reply-To: <20261110040828.1774343-1-skhawaja@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261110040828.1774343-1-skhawaja@google.com> X-Mailer: git-send-email 2.56.0.506.g506a7eabc5-goog Message-ID: <20261110040828.1774343-4-skhawaja@google.com> Subject: [PATCH v7 03/18] iommupt: Implement preserve/unpreserve/restore callbacks From: Samiullah Khawaja To: Joerg Roedel , Will Deacon , Lu Baolu , Jason Gunthorpe , David Woodhouse Cc: Samiullah Khawaja , Robin Murphy , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Pratyush Yadav , Pasha Tatashin , David Matlack , Andrew Morton , Pranjal Shrivastava , Vipin Sharma , Ankit Soni , Nicolin Chen , John Starks Content-Type: text/plain; charset="UTF-8" Add iommupt ops for preservation, unpreservation and restoration of iommu page tables for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables. Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore, the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features. Signed-off-by: Samiullah Khawaja Reviewed-by: Lu Baolu --- drivers/iommu/generic_pt/iommu_pt.h | 155 ++++++++++++++++++++++ drivers/iommu/generic_pt/kunit_iommu_pt.h | 30 +++++ include/linux/generic_pt/iommu.h | 30 +++++ 3 files changed, 215 insertions(+) diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h index 837d835e4fad..b50dc94231f8 100644 --- a/drivers/iommu/generic_pt/iommu_pt.h +++ b/drivers/iommu/generic_pt/iommu_pt.h @@ -16,6 +16,7 @@ #include "../iommu-pages.h" #include #include +#include enum { SW_BIT_CACHE_FLUSH_DONE = 0, @@ -991,6 +992,155 @@ static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova, return ret; } +#ifdef CONFIG_IOMMU_LIVEUPDATE +static void NS(unpreserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range = pt_all_range(common); + struct pt_iommu_collect_args collect = { + .pending.free_list = IOMMU_PAGES_LIST_INIT( + collect.pending.free_list), + }; + + iommu_pages_list_add(&collect.pending.free_list, range.top_table); + + /* + * pt_walk_range() will never fail as check_mapped is not set in the + * collect args. + */ + pt_walk_range(&range, __collect_tables, &collect); + + iommu_unpreserve_pages_list(&collect.pending.free_list); +} + +static int NS(preserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range = pt_all_range(common); + struct pt_iommu_collect_args collect = { + .pending.free_list = IOMMU_PAGES_LIST_INIT( + collect.pending.free_list), + }; + int ret; + + iommu_pages_list_add(&collect.pending.free_list, range.top_table); + + /* + * pt_walk_range() will never fail as check_mapped is not set in the + * collect args. + */ + pt_walk_range(&range, __collect_tables, &collect); + + ret = iommu_preserve_pages_list(&collect.pending.free_list); + if (ret) + return ret; + + ser->top_table_phys = virt_to_phys(range.top_table); + ser->top_level = range.top_level; + + /* + * VASZ and SIGN_EXTEND will be needed in next kernel for collector page + * table walk to restore and free pages. + * + * Use the max_vasz_lg2 from range, as that is the current one if + * DYNAMIC_TOP is supported. + */ + ser->max_vasz_lg2 = range.max_vasz_lg2; + ser->sign_extend = pt_feature(common, PT_FEAT_SIGN_EXTEND); + + return 0; +} + +static int __restore_tables(struct pt_range *range, void *arg, + unsigned int level, struct pt_table_p *table) +{ + struct pt_state pts = pt_init(range, level, table); + int ret; + + if (!pt_can_have_table(&pts)) + return 0; + + for_each_pt_level_entry(&pts) { + if (pts.type == PT_ENTRY_TABLE) { + iommu_restore_pages(virt_to_phys(pts.table_lower)); + + /* + * pt_descend can only fail if pts.table_lower is not + * init. So the if statement below is dead code. + */ + ret = pt_descend(&pts, arg, __restore_tables); + if (ret) + return ret; + } + } + + return 0; +} + +static void NS(deinit)(struct pt_iommu *iommu_table); +static const struct pt_iommu_ops NS(ops_immutable) = { + .deinit = NS(deinit), +}; + +static int NS(restore)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_range range; + + if (ser->max_vasz_lg2 > common->max_vasz_lg2 || + ser->top_level > PT_MAX_TOP_LEVEL) + return -EINVAL; + + /* + * Override the max_vasz_lg2 here to the one calculated using top_range + * in previous kernel. This makes sure that any later page walks use the + * clamped max_vasz_lg2 instead of using the full hardware maximum + * max_vasz_lg2. + */ + common->max_vasz_lg2 = ser->max_vasz_lg2; + + /* + * Restored page tables are strictly transient and only permitted to be + * destroyed via deinit() op. Because we only preserve user-assigned + * devices utilizing pass-through frameworks (VFIO / IOMMUFD), any + * concurrent or subsequent map/unmap operations on a restored domain + * are explicitly blocked at the subsystem boundary (e.g., via IOAS + * immutability). + */ + iommu_table->ops = &NS(ops_immutable); + + /* + * It is safe to override this here since this domain is immutable and + * can only be freed. This also means that for non-coherent pages DMA + * mappings will not be created using the DMA API. + * + * Note that the FMT specific bits are not cleared as they might be + * set/preserved/restored by the driver to handle special cases that are + * required for page walks. + */ + common->features &= ~GENMASK(PT_FEAT_FMT_START - 1, 0); + if (ser->sign_extend) + common->features |= BIT(PT_FEAT_SIGN_EXTEND); + + range = pt_all_range(common); + iommu_restore_pages(ser->top_table_phys); + + /* + * The old top_table can be freed here as it is not used for any IOMMU + * mappings yet, because the restore happens during device probe. + */ + iommu_pages_free_incoherent(range.top_table, + iommu_table->iommu_device); + + /* Set the restored top table */ + pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level); + + /* Restore all pages*/ + range = pt_all_range(common); + return pt_walk_range(&range, __restore_tables, NULL); +} +#endif + struct pt_unmap_args { struct iommupt_pending_gather pending; pt_vaddr_t unmapped; @@ -1174,6 +1324,11 @@ static const struct pt_iommu_ops NS(ops) = { #endif .get_info = NS(get_info), .deinit = NS(deinit), +#ifdef CONFIG_IOMMU_LIVEUPDATE + .preserve = NS(preserve), + .unpreserve = NS(unpreserve), + .restore = NS(restore), +#endif }; static int pt_init_common(struct pt_common *common) diff --git a/drivers/iommu/generic_pt/kunit_iommu_pt.h b/drivers/iommu/generic_pt/kunit_iommu_pt.h index 9fc71db74234..7306de44eb00 100644 --- a/drivers/iommu/generic_pt/kunit_iommu_pt.h +++ b/drivers/iommu/generic_pt/kunit_iommu_pt.h @@ -427,6 +427,35 @@ static void test_mixed(struct kunit *test) check_iova(test, start, oa, len); } +static void test_restore_free(struct kunit *test) +{ + struct kunit_iommu_priv *priv = test->priv; + struct pt_range top_range = pt_top_range(priv->common); + u64 start = 0x3fe400ULL << 12; + u64 end = 0x4c0600ULL << 12; + pt_vaddr_t len = end - start; + + if (top_range.last_va <= start || sizeof(unsigned long) == 4) + kunit_skip(test, "range is too small"); + if ((priv->safe_pgsize_bitmap & GENMASK(30, 21)) != (BIT(30) | BIT(21))) + kunit_skip(test, "incompatible psize"); + + /* Map a large mixed range to populate multiple levels of page tables */ + do_map(test, start, start, len); + + /* + * Simulate a restored state by clearing all features except SIGN_EXTEND + * and DMA_INCOHERENT. This verifies that the generic page table free + * walker can correctly tear down a populated domain when other features + * are zeroed. Also set max_vasz_lg2 as done by the actual restore() + * function. + */ + priv->common->features &= (BIT(PT_FEAT_SIGN_EXTEND) | BIT(PT_FEAT_DMA_INCOHERENT)); + priv->common->max_vasz_lg2 = top_range.max_vasz_lg2; + + /* The domain will be freed when the test exits. */ +} + static struct kunit_case iommu_test_cases[] = { KUNIT_CASE_FMT(test_increase_level), KUNIT_CASE_FMT(test_map_simple), @@ -435,6 +464,7 @@ static struct kunit_case iommu_test_cases[] = { KUNIT_CASE_FMT(test_random_map), KUNIT_CASE_FMT(test_pgsize_boundary), KUNIT_CASE_FMT(test_mixed), + KUNIT_CASE_FMT(test_restore_free), {}, }; diff --git a/include/linux/generic_pt/iommu.h b/include/linux/generic_pt/iommu.h index 13ebb72a67de..617fb9af6bef 100644 --- a/include/linux/generic_pt/iommu.h +++ b/include/linux/generic_pt/iommu.h @@ -13,6 +13,7 @@ struct iommu_iotlb_gather; struct pt_iommu_ops; struct pt_iommu_driver_ops; struct iommu_dirty_bitmap; +struct iommu_domain_ser; /** * DOC: IOMMU Radix Page Table @@ -166,6 +167,35 @@ struct pt_iommu_ops { * table from all HW access and all caches. */ void (*deinit)(struct pt_iommu *iommu_table); + + /** + * @preserve: Preserve the iommu page table for liveupdate + * @iommu_table: Table to preserve + * @ser: Serialization struct to fill with preserved state + * + * Preserve iommu page table and the relevant state for liveupdate. The + * caller must make sure that the page table is not updated during and + * after preservation. + */ + int (*preserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + + /** + * @unpreserve: Unpreserve the iommu page table + * @iommu_table: Table to unpreserve + * @ser: Serialization struct that contains preserved state + */ + void (*unpreserve)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + + /** + * @restore: Restore the iommu page table after liveupdate + * @iommu_table: Table to restore the state into + * @ser: Serialization struct that contains preserved state + * + * The iommu_table is back filled with the restored state that was + * preserved in the serialization struct. + */ + int (*restore)(struct pt_iommu *iommu_table, struct iommu_domain_ser *ser); + }; /** -- 2.56.0.506.g506a7eabc5-goog