From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D0EFC64E7C for ; Tue, 17 Nov 2020 17:12:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BAE6B238E6 for ; Tue, 17 Nov 2020 17:12:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="N5DzzsTi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729739AbgKQRM3 (ORCPT ); Tue, 17 Nov 2020 12:12:29 -0500 Received: from mail-dm6nam12on2071.outbound.protection.outlook.com ([40.107.243.71]:24061 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728705AbgKQRM1 (ORCPT ); Tue, 17 Nov 2020 12:12:27 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C9zwQuKatpYfH8Sag8ufjVfCVo8MIOYk23MZpfKUDv7uXF/TfV2Lcwf9lc9Y2IGi9XBFXv7bSxoa4GXk8ss8G76mYpDkgzCpCE3sMvb8LM6DAn74uPM0ZlYDgd9lDXVVxLfUsjiTXrmCPnPJTzW5B5Hrh2XwICM96RaXIemG7AR3oiz1cVbSZlfWSlfs0HS9MZrPMtFoD6tR+w118vMPLqDS/7bFlFYjBjE8wjnl7ikQce4iB455U0ffvZrFGeh+l1OXJIiCVpTGbtXzDTlBmlvEDqVgzu12yKsHFnLhkvPTt7aZQ1/PhLoix4ngh/lv5s7TjLEXpoODCYu+PVsK1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4ktpBbqBaxlFUbmN0pBId3j2iC2J5+O0rE9ohbalHr4=; b=dLozh+al94WzoOrN0GHI2Guz8hnB671cVHBnzBlIpYSSOfG27j23x+LzQOiSVR55UKPF1sxDvaJfUU4m2Sh98UoCQLz/4jaRxIDlLD193g28c9VW2wYpxuwcYP7zAVgYAYzagRPOIrSYrTQnAALZCEKHZ3qnyTZOJ2IynHriqSWvtcLUEYLMqn3QdD8YXt4fpGD5HzVamiOf+chWEp4LbRF9XaPa5bf7hmhkb9mq34aN0a4Dk+wFZeGP3Xh2D5em9alkFMsVYQ08A3XWTh+SnPPfRWOuY/FLwnWQS1RTUJ4dMOSrqwh34uYc5HBs6pL4Cmzt2Wdh0EClD70K3q69jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4ktpBbqBaxlFUbmN0pBId3j2iC2J5+O0rE9ohbalHr4=; b=N5DzzsTiny7mv6z7iRc/5w/xlBeHGzWp18w2LK00MuQ1z3AQdfWBGPgu3tbW7kJbLW03/S7E2xO5FGFXK7nSj4eaNQMLHi7tI3VmOrC4AhGpwK0PCje1Yq34KQUkJfuHSXG9mNofpmMdl0GZDIRQ2qAwW9+nF9deR6Eq4ikcXls= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from DM5PR12MB1355.namprd12.prod.outlook.com (2603:10b6:3:6e::7) by DM5PR12MB1772.namprd12.prod.outlook.com (2603:10b6:3:107::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.25; Tue, 17 Nov 2020 17:12:19 +0000 Received: from DM5PR12MB1355.namprd12.prod.outlook.com ([fe80::dcda:c3e8:2386:e7fe]) by DM5PR12MB1355.namprd12.prod.outlook.com ([fe80::dcda:c3e8:2386:e7fe%12]) with mapi id 15.20.3564.028; Tue, 17 Nov 2020 17:12:19 +0000 From: Tom Lendacky To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Paolo Bonzini , Jim Mattson , Joerg Roedel , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Borislav Petkov , Ingo Molnar , Thomas Gleixner , Brijesh Singh Subject: [PATCH v4 32/34] KVM: SVM: Provide support for SEV-ES vCPU loading Date: Tue, 17 Nov 2020 11:07:35 -0600 Message-Id: <5e19b1fd4a11ee87261e6dcfa97d1d4250766500.1605632857.git.thomas.lendacky@amd.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: SN4PR0801CA0014.namprd08.prod.outlook.com (2603:10b6:803:29::24) To DM5PR12MB1355.namprd12.prod.outlook.com (2603:10b6:3:6e::7) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from tlendack-t1.amd.com (165.204.77.1) by SN4PR0801CA0014.namprd08.prod.outlook.com (2603:10b6:803:29::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.28 via Frontend Transport; Tue, 17 Nov 2020 17:12:18 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 0383353c-cf4a-486b-0592-08d88b1bf0d1 X-MS-TrafficTypeDiagnostic: DM5PR12MB1772: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F6jP2CiH4ijhZNVoPEz8rVUBhMlAy7EN77kAD8EAIDivPEgzmeCHkMXpBmqUZSGpMUAKqtx/Jfa7GruYm0n3zlC/QY1+KiJvNIkzmkKCA8DFqMEAwZdM5CbDHvPW4FpSE9yI0O+ATeZFyVadLzkTKnVD7Z2If8NBtzOYYcf+UXaubjhS3TE6IZwgbkjemUs/HD6g13FYl61b0uxre4olXqCltcWhu0x07sUeAabEu00NiwBo+6rwCYvzCu5Q4Acp3n6ehfGzf5PwvUyDV3za3HRiRSmHh290ZeYHwZF9eqigLh26g8aVupBXZpeEh4PP X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM5PR12MB1355.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(366004)(346002)(376002)(39860400002)(136003)(6486002)(66476007)(66946007)(8936002)(83380400001)(5660300002)(26005)(478600001)(66556008)(16526019)(6666004)(186003)(36756003)(86362001)(8676002)(316002)(956004)(52116002)(4326008)(54906003)(7696005)(7416002)(2616005)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: ecFGkxecpUkx3fSs0B1vxhkWWVjTUBRu715fnSjGF8gduKqQvePE+VSOaG/IGCHJazdYYZpOn0jqDjQNjA+3Sye8QVb21jZ/yudpxRPJgyc5b+/eP5l0lrvKpVS8Wikeop7FrGDXDO0Fnc36vfqFoIoirhm0e3XmlVfZq+dGOBkP+cdEFrDpyAu7NmhH9ZtVxxwpKK/+MAxLFX8rMSsKjIzK3/k0RjqBcb2IU1Aus3trXnXHhXW1PvwRrNCKMAwoq9LhQfCq64NSE3D2XcgaLS2s1h9FD7USeRUwHUgGXdh4DmezTgDsUsnYLHFC/cRaI88zTaqufLuRblXqF6Q9A9mQwdCZtNfBQTHXh0ngoHYWyvSzNpl8cJQWx+pMVQdGhdDO7D4lmxLIPRwFFUNm3/DqV6RyNcuholpQqUtGBPDH4mB78qIDkAnWxWdI39f3DUmHprtUMXWLJzr8iQSceqjT5b59zjBxetv/9DH4+rudIm5k4BD6hi8fEn0pENDn0JWu6AZ/9oheMkF0e+7Ddglq0k6p7SmVNwIRlIOd9t6H27/uHahzDH7qWjmbe//uyvhI7NiahJuAZC+PeZDnmvzUNWb5CeumnNJ/0WV8dNaokZsnUNq+rB1+wY9ormYj0jlMH0dazaI3/8O2tuS7xA== X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0383353c-cf4a-486b-0592-08d88b1bf0d1 X-MS-Exchange-CrossTenant-AuthSource: DM5PR12MB1355.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2020 17:12:19.6861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x63ISA2Kfkg49Bqzbe+QDX8sfn6GogM0U8hh1beZ4FMGdOhbcPjnlN2AxOMI4XUeR58ZSjZ4BIhO1TfuULS4sw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1772 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Tom Lendacky An SEV-ES vCPU requires additional VMCB vCPU load/put requirements. SEV-ES hardware will restore certain registers on VMEXIT, but not save them on VMRUM (see Table B-3 and Table B-4 of the AMD64 APM Volume 2), so make the following changes: General vCPU load changes: - During vCPU loading, perform a VMSAVE to the per-CPU SVM save area and save the current values of XCR0, XSS and PKRU to the per-CPU SVM save area as these registers will be restored on VMEXIT. General vCPU put changes: - Do not attempt to restore registers that SEV-ES hardware has already restored on VMEXIT. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/svm.h | 10 ++++--- arch/x86/kvm/svm/sev.c | 54 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.c | 36 ++++++++++++++++--------- arch/x86/kvm/svm/svm.h | 22 +++++++++++----- arch/x86/kvm/x86.c | 3 ++- arch/x86/kvm/x86.h | 1 + 6 files changed, 103 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index a57331de59e2..1c561945b426 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -234,7 +234,8 @@ struct vmcb_save_area { u8 cpl; u8 reserved_2[4]; u64 efer; - u8 reserved_3[112]; + u8 reserved_3[104]; + u64 xss; /* Valid for SEV-ES only */ u64 cr4; u64 cr3; u64 cr0; @@ -265,9 +266,12 @@ struct vmcb_save_area { /* * The following part of the save area is valid only for - * SEV-ES guests when referenced through the GHCB. + * SEV-ES guests when referenced through the GHCB or for + * saving to the host save area. */ - u8 reserved_7[104]; + u8 reserved_7[80]; + u32 pkru; + u8 reserved_7a[20]; u64 reserved_8; /* rax already available at 0x01f8 */ u64 rcx; u64 rdx; diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index e34d3a6dba80..225f18dbf522 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -18,12 +18,15 @@ #include #include +#include #include "x86.h" #include "svm.h" #include "cpuid.h" #include "trace.h" +#define __ex(x) __kvm_handle_fault_on_reboot(x) + static u8 sev_enc_bit; static int sev_flush_asids(void); static DECLARE_RWSEM(sev_deactivate_lock); @@ -1902,3 +1905,54 @@ void sev_es_create_vcpu(struct vcpu_svm *svm) GHCB_VERSION_MIN, sev_enc_bit)); } + +void sev_es_vcpu_load(struct vcpu_svm *svm, int cpu) +{ + struct svm_cpu_data *sd = per_cpu(svm_data, cpu); + struct vmcb_save_area *hostsa; + unsigned int i; + + /* + * As an SEV-ES guest, hardware will restore the host state on VMEXIT, + * of which one step is to perform a VMLOAD. Since hardware does not + * perform a VMSAVE on VMRUN, the host savearea must be updated. + */ + asm volatile(__ex("vmsave") : : "a" (__sme_page_pa(sd->save_area)) : "memory"); + + /* + * Certain MSRs are restored on VMEXIT, only save ones that aren't + * restored. + */ + for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) { + if (host_save_user_msrs[i].sev_es_restored) + continue; + + rdmsrl(host_save_user_msrs[i].index, svm->host_user_msrs[i]); + } + + /* XCR0 is restored on VMEXIT, save the current host value */ + hostsa = (struct vmcb_save_area *)(page_address(sd->save_area) + 0x400); + hostsa->xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK); + + /* PKRU is restored on VMEXIT, save the curent host value */ + hostsa->pkru = read_pkru(); + + /* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */ + hostsa->xss = host_xss; +} + +void sev_es_vcpu_put(struct vcpu_svm *svm) +{ + unsigned int i; + + /* + * Certain MSRs are restored on VMEXIT and were saved with vmsave in + * sev_es_vcpu_load() above. Only restore ones that weren't. + */ + for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) { + if (host_save_user_msrs[i].sev_es_restored) + continue; + + wrmsrl(host_save_user_msrs[i].index, svm->host_user_msrs[i]); + } +} diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 252e10de0950..b9c4d8b28423 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1417,15 +1417,20 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vmcb_mark_all_dirty(svm->vmcb); } + if (sev_es_guest(svm->vcpu.kvm)) { + sev_es_vcpu_load(svm, cpu); + } else { #ifdef CONFIG_X86_64 - rdmsrl(MSR_GS_BASE, to_svm(vcpu)->host.gs_base); + rdmsrl(MSR_GS_BASE, to_svm(vcpu)->host.gs_base); #endif - savesegment(fs, svm->host.fs); - savesegment(gs, svm->host.gs); - svm->host.ldt = kvm_read_ldt(); + savesegment(fs, svm->host.fs); + savesegment(gs, svm->host.gs); + svm->host.ldt = kvm_read_ldt(); - for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) - rdmsrl(host_save_user_msrs[i], svm->host_user_msrs[i]); + for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) + rdmsrl(host_save_user_msrs[i].index, + svm->host_user_msrs[i]); + } if (static_cpu_has(X86_FEATURE_TSCRATEMSR)) { u64 tsc_ratio = vcpu->arch.tsc_scaling_ratio; @@ -1453,19 +1458,24 @@ static void svm_vcpu_put(struct kvm_vcpu *vcpu) avic_vcpu_put(vcpu); ++vcpu->stat.host_state_reload; - kvm_load_ldt(svm->host.ldt); + if (sev_es_guest(svm->vcpu.kvm)) { + sev_es_vcpu_put(svm); + } else { + kvm_load_ldt(svm->host.ldt); #ifdef CONFIG_X86_64 - loadsegment(fs, svm->host.fs); - wrmsrl(MSR_KERNEL_GS_BASE, current->thread.gsbase); - load_gs_index(svm->host.gs); + loadsegment(fs, svm->host.fs); + wrmsrl(MSR_KERNEL_GS_BASE, current->thread.gsbase); + load_gs_index(svm->host.gs); #else #ifdef CONFIG_X86_32_LAZY_GS - loadsegment(gs, svm->host.gs); + loadsegment(gs, svm->host.gs); #endif #endif - for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) - wrmsrl(host_save_user_msrs[i], svm->host_user_msrs[i]); + for (i = 0; i < NR_HOST_SAVE_USER_MSRS; i++) + wrmsrl(host_save_user_msrs[i].index, + svm->host_user_msrs[i]); + } } static unsigned long svm_get_rflags(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 48e4cfaf0a69..5229c5763a30 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -23,15 +23,23 @@ #define __sme_page_pa(x) __sme_set(page_to_pfn(x) << PAGE_SHIFT) -static const u32 host_save_user_msrs[] = { +static const struct svm_host_save_msrs { + u32 index; /* Index of the MSR */ + bool sev_es_restored; /* True if MSR is restored on SEV-ES VMEXIT */ +} host_save_user_msrs[] = { #ifdef CONFIG_X86_64 - MSR_STAR, MSR_LSTAR, MSR_CSTAR, MSR_SYSCALL_MASK, MSR_KERNEL_GS_BASE, - MSR_FS_BASE, + { .index = MSR_STAR, .sev_es_restored = true }, + { .index = MSR_LSTAR, .sev_es_restored = true }, + { .index = MSR_CSTAR, .sev_es_restored = true }, + { .index = MSR_SYSCALL_MASK, .sev_es_restored = true }, + { .index = MSR_KERNEL_GS_BASE, .sev_es_restored = true }, + { .index = MSR_FS_BASE, .sev_es_restored = true }, #endif - MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, - MSR_TSC_AUX, + { .index = MSR_IA32_SYSENTER_CS, .sev_es_restored = true }, + { .index = MSR_IA32_SYSENTER_ESP, .sev_es_restored = true }, + { .index = MSR_IA32_SYSENTER_EIP, .sev_es_restored = true }, + { .index = MSR_TSC_AUX, .sev_es_restored = false }, }; - #define NR_HOST_SAVE_USER_MSRS ARRAY_SIZE(host_save_user_msrs) #define MAX_DIRECT_ACCESS_MSRS 18 @@ -582,5 +590,7 @@ int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in); void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); void sev_es_init_vmcb(struct vcpu_svm *svm); void sev_es_create_vcpu(struct vcpu_svm *svm); +void sev_es_vcpu_load(struct vcpu_svm *svm, int cpu); +void sev_es_vcpu_put(struct vcpu_svm *svm); #endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index a0eca41eaa33..6e0599ae517f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -197,7 +197,8 @@ EXPORT_SYMBOL_GPL(host_efer); bool __read_mostly allow_smaller_maxphyaddr = 0; EXPORT_SYMBOL_GPL(allow_smaller_maxphyaddr); -static u64 __read_mostly host_xss; +u64 __read_mostly host_xss; +EXPORT_SYMBOL_GPL(host_xss); u64 __read_mostly supported_xss; EXPORT_SYMBOL_GPL(supported_xss); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index f46bb286def5..a922d950638d 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -278,6 +278,7 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu); extern u64 host_xcr0; extern u64 supported_xcr0; +extern u64 host_xss; extern u64 supported_xss; static inline bool kvm_mpx_supported(void) -- 2.28.0