From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f227.google.com (mail-pl1-f227.google.com [209.85.214.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64CF93BB661 for ; Wed, 16 Sep 2026 05:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.227 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535163; cv=none; b=Xjbr1oJQSC5dOfaC74SqC8gQZ/G1HeR/ajMH0pJXFnExao8gVZInxSEDbNgowxF5A+Gor6vNu1q52Vyc/fhWb3EXSqEsCsFqua3M3x3LroH3lHLgcd5qBFqytDQWS0UhGyNPBCeGCmxFSN6l/Mx/sleiLy0lFIrsRw34KhX6J20= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535163; c=relaxed/simple; bh=ZCYsk3b8be7YzwPtZc9AlEHE3JpuvSauytIqzViEWP8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lNJUIMEkc6AhhuG+eUdQA0vUdKgRpjyKLLfil+g0GYePl8k/nY9qNzrwAh8sg5lrw5V3jGk4z6J6607ADWwz4EGxFgOoO/ye3z9wusY4cPubioc0/F1yQrGsVveGu/3BQ1F5ysmnFYkOGb+vhcm3RlMmMJMLnEyT3/a2i8andc8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=YW777VKk; arc=none smtp.client-ip=209.85.214.227 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="YW777VKk" Received: by mail-pl1-f227.google.com with SMTP id d9443c01a7336-2d6d28aa26cso34054605ad.2 for ; Tue, 15 Sep 2026 22:06:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789535161; x=1790139961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qEqGUwhuGtd/IQ00br0ru4/E+DR8RR02AnrUv6Zwxpk=; b=q2CZ0jCABdyd+63njMzNbMsgtWcKuWTF0iw4gFn5NjqoDLrRsrdHkemJRjMV99zipe mSOmkj6kpZUYcT5QvsuldwAOSnG5Am+Na96SbDEEemFvBKU7yRKROyjr5k2S507CTzNr 02Jh9mH4n96PSXHI/SBxNSFUL6q5yu/XVZXQeygrfrQ/+xgxrRgu0u3t3dhsmwlFgbGv AuRZelzmgDTI+gHNyN/jsnPFTrfC2BsFDRvky6xdtDWGVgGIEXCYzWuk04wpaKAgO0k7 Unh9JgQpPZIEZPOhkRM6W6JGoWvS/R99Qu798mCFpX3SXyPR5uZLFbEAvZtBNxvGpCvp A8TA== X-Forwarded-Encrypted: i=1; AKwUvByiDucF00RuCSssQ5mnR+esfuZhK+nH6FwHDKkCVeW90BDmKOYFmUMNbkOrjVApPB9dSmBHRyUHfTqU5V8=@vger.kernel.org X-Gm-Message-State: AFuF++k+kMNx2ByjLDvwGebPmjZR7B7iRPj/C29Cu1g0w8fltiHMPrKZ c7AQPm1ORQEyfFzDv9SzE7I+O1wQUimS6LBl0lWKQe6E1It1F0Pf7T1nOEd+2ptnhjvUM0NNi0v e0U22LTQ4DEA+T6frXnrgJ1czKMib/QkVmFo9i7OvhD/IxAM1Q+RiGeHlZfKl+hwXSGDw+G8Daf 7kXp7OHZ2iL4LWOHMOpJ+FAbinzeXJmgfpjggAREOCT5ACdOXTf62EwxJTnAOl+wAnY+zgNFWWK zXzdcKBplqpGGsl X-Gm-Gg: AYBFou0m7C3eDekjCBQx4hGmY8OPm/Zs4h6fN9hmkArUU076BWjNHC8IRjiT+XLe/yB FtFjKIk0j8tHzqVjHkumT0JUUFC0dycC7UMVDpmd+kLkPcNrvE8a/bJKlROMiDncTmgTby+H4jc IsRTfUYMRQkGXH3tCBAVUBa2iX4VEU7JykfXmAI29Iw/GLWqTMHtHdAyssFKLonmxG3lOdIQcHa v9I+hbPsJxkMvPi5T6j/U1gHsYBtwEI7e2uimPWG9nDESgqOACPWsPCpUeB3q6098s7tMMLVw/v HpI0loeIvLyfXko4vqs05RPn0ddHOTsZzJcxb/gQv5+jujrYZ612NYKtkIPh0lm7QgUPhG8u0jd yhY5YodOucjPb4mpkfiE2VKvBPu0u7lV2PbQUR57hGy5qTSQ46L85lrXR/Lu0Hq3X/TZqlf9Wrh gRZX4DgeCyvArBnzeE0JwBiwawcaEmfCG7 X-Received: by 2002:a17:902:e74b:b0:2dd:3ddf:b895 with SMTP id d9443c01a7336-2dd8dd0b2fcmr22960145ad.3.1789535160712; Tue, 15 Sep 2026 22:06:00 -0700 (PDT) Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-29.dlp.protect.broadcom.com. [144.49.247.29]) by smtp-relay.gmail.com with ESMTPS id d9443c01a7336-2dd89e94c81sm749675ad.12.2026.09.15.22.05.59 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Sep 2026 22:06:00 -0700 (PDT) X-Relaying-Domain: broadcom.com X-CFilter-Loop: Reflected Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cc1a439db36so4351578a12.2 for ; Tue, 15 Sep 2026 22:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1789535158; x=1790139958; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qEqGUwhuGtd/IQ00br0ru4/E+DR8RR02AnrUv6Zwxpk=; b=YW777VKkuthOhJll13Vfzz2TFuo7Dg4wmScpv2bjhDyzJ/f/p/MOQx8huOil4+PGWN WtMxrZuFv75DzHRdMmW8yrE7IHNh9G3HDENQQC02VhyktO6g++ZPQ8J/0inkvJ4CgURj R07zG3rTlx/SnJGV0+elbsCoqb3Xe0YjId85M= X-Forwarded-Encrypted: i=1; AKwUvBzquDHwQGCDhqsysPQ6fbpAyoPYtweQyjkTHZtvtN7Rj62ne7HPio4taAwQUc0gdCo4y1DjD7IeK65pnNk=@vger.kernel.org X-Received: by 2002:a05:6a21:1343:b0:3da:80a1:f2f with SMTP id adf61e73a8af0-3dd5f5dfd69mr2927550637.14.1789535158380; Tue, 15 Sep 2026 22:05:58 -0700 (PDT) X-Received: by 2002:a05:6a21:1343:b0:3da:80a1:f2f with SMTP id adf61e73a8af0-3dd5f5dfd69mr2927490637.14.1789535157892; Tue, 15 Sep 2026 22:05:57 -0700 (PDT) Received: from vertex.localdomain ([192.19.144.250]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-33bf5e40494sm3951386eec.15.2026.09.15.22.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 22:05:56 -0700 (PDT) From: Zack Rusin To: Borislav Petkov , Ajay Kaher , Alexey Makhalov , x86@kernel.org, Dennis Zhou , Tejun Heo , Arnd Bergmann , Kiryl Shutsemau , Rick Edgecombe Cc: Thomas Gleixner , Ingo Molnar , Dave Hansen , "H . Peter Anvin" , virtualization@lists.linux.dev, bcm-kernel-feedback-list@broadcom.com, linux-kernel@vger.kernel.org, Christoph Lameter , Andrew Morton , Tom Lendacky , Bo Gan , linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-coco@lists.linux.dev, kvm@vger.kernel.org, Zack Rusin Subject: [PATCH v1 2/2] x86/vmware: Decrypt steal-time storage before sharing it Date: Wed, 16 Sep 2026 01:05:40 -0400 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e VMware's steal-time counter must be in shared memory so the host can update it. Encrypted guests currently register its address without decrypting the storage. Move their setup to an early initcall, when the memory allocator is available for page-table splitting, but before secondary CPUs start. Leave ordinary guests' setup unchanged. Convert every possible CPU's storage before registering any address. Zero each object after conversion, since its contents may not survive conversion and the host initializes only the counter. Register the boot CPU with preemption disabled; the existing hotplug callbacks handle the others. If conversion or boot-CPU registration fails, attempt to re-encrypt all affected ranges, including a partially converted failing range, and disable steal time. TDX's conversion path requires directly mapped memory. Check all per-CPU objects before converting anything, and disable steal time if a TDX guest uses the page per-CPU allocator, which supplies vmalloc mappings. This also covers automatic fallback from the embedded allocator. AMD guests are unaffected by this restriction because their conversion path supports those mappings. Link: https://lore.kernel.org/r/20260309235250.2611115-5-alexey.makhalov@broadcom.com Co-developed-by: Bo Gan Signed-off-by: Bo Gan Co-developed-by: Alexey Makhalov Signed-off-by: Alexey Makhalov Signed-off-by: Zack Rusin --- arch/x86/kernel/cpu/vmware.c | 96 +++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c index 34b73573b108..f49898275d87 100644 --- a/arch/x86/kernel/cpu/vmware.c +++ b/arch/x86/kernel/cpu/vmware.c @@ -25,14 +25,19 @@ #include #include #include +#include #include #include +#include +#include #include +#include #include #include #include #include #include +#include #include #include #include @@ -147,6 +152,7 @@ static struct cyc2ns_data vmware_cyc2ns __ro_after_init; static bool vmw_sched_clock __initdata = true; static DEFINE_PER_CPU_DECRYPTED(struct vmware_steal_time, vmw_steal_time) __aligned(64); static bool has_steal_clock; +static bool vmw_steal_time_ready; static bool steal_acc __initdata = true; /* steal time accounting */ static __init int setup_vmw_sched_clock(char *s) @@ -280,7 +286,9 @@ static void vmware_disable_steal_time(void) static void vmware_guest_cpu_init(void) { - if (has_steal_clock) + if (has_steal_clock && + (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) || + vmw_steal_time_ready)) vmware_register_steal_time(); } @@ -325,6 +333,92 @@ static int vmware_cpu_down_prepare(unsigned int cpu) } #endif +static void __init vmware_steal_time_range(int cpu, unsigned long *start, + int *numpages) +{ + unsigned long addr = (unsigned long)per_cpu_ptr(&vmw_steal_time, cpu); + + *start = addr & PAGE_MASK; + *numpages = DIV_ROUND_UP(offset_in_page(addr) + + sizeof(struct vmware_steal_time), PAGE_SIZE); +} + +static int __init vmware_decrypt_steal_time(void) +{ + int cpu, failed_cpu, numpages, ret; + unsigned long start; + + if (!has_steal_clock || + !cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) + return 0; + + /* + * TDX's conversion callback derives the physical range with __pa(), + * so the storage must live in the direct map. The page per-CPU + * allocator, which is also the automatic fallback when the embedding + * allocator fails, hands out vmalloc addresses instead. Reject those + * before converting anything, so no page is left converted. + */ + if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { + for_each_possible_cpu(cpu) { + if (!is_vmalloc_addr(per_cpu_ptr(&vmw_steal_time, cpu))) + continue; + + pr_warn("steal time disabled: TDX requires a direct mapping\n"); + has_steal_clock = false; + return 0; + } + } + + for_each_possible_cpu(cpu) { + vmware_steal_time_range(cpu, &start, &numpages); + ret = set_memory_decrypted(start, numpages); + if (ret) { + failed_cpu = cpu; + goto rollback; + } + + /* + * Conversion need not preserve the zeroes. The host + * initializes only the counter on enable, so the guest + * must initialize the reserved words itself. + */ + memset(per_cpu_ptr(&vmw_steal_time, cpu), 0, + sizeof(struct vmware_steal_time)); + } + + vmw_steal_time_ready = true; + preempt_disable(); + vmware_guest_cpu_init(); + preempt_enable(); + if (!has_steal_clock) { + pr_warn("failed to register boot CPU steal-time memory\n"); + failed_cpu = nr_cpu_ids; + goto rollback_pages; + } + return 0; + +rollback: + pr_warn("failed to decrypt steal-time memory for CPU %d: %d\n", + failed_cpu, ret); + +rollback_pages: + for_each_possible_cpu(cpu) { + vmware_steal_time_range(cpu, &start, &numpages); + ret = set_memory_encrypted(start, numpages); + if (ret) + pr_warn("failed to re-encrypt steal-time memory for CPU %d: %d\n", + cpu, ret); + if (cpu == failed_cpu) + break; + } + + vmw_steal_time_ready = false; + has_steal_clock = false; + return 0; +} +early_initcall(vmware_decrypt_steal_time); + static __init int activate_jump_labels(void) { if (has_steal_clock) { -- 2.53.0