From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932948AbdBPPso (ORCPT ); Thu, 16 Feb 2017 10:48:44 -0500 Received: from mail-bn3nam01on0081.outbound.protection.outlook.com ([104.47.33.81]:55662 "EHLO NAM01-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932620AbdBPPsc (ORCPT ); Thu, 16 Feb 2017 10:48:32 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; From: Tom Lendacky Subject: [RFC PATCH v4 27/28] x86: Add support to encrypt the kernel in-place To: , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Brijesh Singh , Ingo Molnar , Alexander Potapenko , Andy Lutomirski , "H. Peter Anvin" , Borislav Petkov , Andrey Ryabinin , Thomas Gleixner , Larry Woodman , Dmitry Vyukov Date: Thu, 16 Feb 2017 09:48:08 -0600 Message-ID: <20170216154808.19244.475.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> References: <20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: BLUPR0401CA0040.namprd04.prod.outlook.com (10.162.114.178) To MWHPR12MB1150.namprd12.prod.outlook.com (10.169.204.14) X-MS-Office365-Filtering-Correlation-Id: c4326c1a-5d39-462f-5d53-08d456833924 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081);SRVR:MWHPR12MB1150; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1150;3:6vty0BNUUQ0oVYYaXwVVWfj4PkZKthR3nBlzdurukqbJBb1tU1gqW5Lx8ttHm8o5uMxXGJjdPxDWo+sdUZFMG1l2hpKoFb0YIWrMxOGnkecNbGBWkj6xGHL1yf0t97z5tCRTlh1ZRTuLFF+1SlK/5S7VO/nbUoxc7Jjk9J4I29oRpNt78cLhVpCD0K8EPHIdnxQVr6adxw75ym35zFQp4Crb1QxZRXvpXvihLPoYiOKP8ULyPE1taVwMBvRMl/ZHtf8sAPvAS9zctYAnVRlPD3sQSBh2hJR1xywcsLQFvGQ=;25:cqNJTVeZ8mxlMF5wArT7XtUeUOjQLVAxylR5NIBWIIlw9qCGLlsKK5Zv91NJY0D20X59EMcma7khVdmAnKpfyPBRZfXRYGNYjxjGHZj0hEVzL94RvR4RK+fhOhLHFQFqUdlfkTQL+g3BnP6X+MQz8iIDx+2GHSon+q6IvgN763bz0VSalTjy+JlOJElSoCU2VUhgaBellR8cM3VLXH0WEHAdN+aNEczLW/Cw69O22/zuzu1HMeTIpB6WlLMwKqk0WFMJcwKIsrJ4l5Orsk2ikDtmHkORfr88I6Dq0OXCy+eYfoH6y9ZDP6oWOB4mnhl+RcS2c9M1LpZJc1cLYl8xdu32q6DwgqRaRsO7FQW+fy3j6OWVups939B6mekFjKhw/3baCIAJXrfTvdW6d0vfVRyllDWJxEro5Xmf+0xLUeErteboa22GdrhmdDxfwus+l5LuN/3TcJXqIqIX+D2lnQ== X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1150;31:oMR+XtfFgrw0vjJFIX8QdEXJq3qnn4BxbmumOhz1vn40h2NirDgx+n28LQMjMx/UaS22g9Ml7wqWtb+nzfo5TOu43TE0rT/QSnnZLt5Vju3s0/3axxuli7u6syhmxbNuCuYKvanOVpp+WJzoOtBTMfgUnnqLPNmAGGOl1Mt97FQ5WPt+GvtU/ACxNolQ39ypu8zXETshYEhI8NLVubZN3lueZb+hp74Gir+xjr8PMjY=;20:YAvyZJK+zyMwohZegNyFFarCGtxuLdpRR6Nn8LcJFhCE3PDDzMX4SrlA5DhPx2Z78gTkyUwDSY0n1DhX5s151ITF2uuLLgiomwqnbWRjpnUULQrnWhOaAjvqtW2BzLWtFs6kanVgqB7FShSP+RrRNnq91mzyV97zvlR3bCHuVa+S4VvMhgaLR8PQDb+M9fAjYe/NK2B8oERykS9mw2FhsDu7OxPPjFtP+JtFObFhaUIJGxl7kaCdUz+O84anjqgLFiVlA+LKs+xZUwVqfYvd/kqicJSYjyJ4UokRNcoF+l3b5+u2qBrg7W4mXMxedExuiHujKAaZ30/Air6XCXYex8nVLKFGVBHtkbpkq18vKobJMqs+r76TxR6+AdShc+nYaI9yWfb5dHm2Dq1LEF9vdZZ9CZrgABSlA7oEJKtE4aOTdpkRYccSCGvtTHRkP9vGTsZX+7iPuQNkNKNmq8BOjhxvHi3A72q0JeF9smkjq4MrlBO9mr//gIVSRdUFr3Gn X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123560025)(20161123564025)(20161123558025)(20161123562025)(20161123555025)(6072148);SRVR:MWHPR12MB1150;BCL:0;PCL:0;RULEID:;SRVR:MWHPR12MB1150; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1150;4:EEYUtpDxWsX7a2xWNi2FUKpEuF2fqin2ZXRjUlUawoQU0rgbfQDaSmPJVUWHsl8li6ie8vZb10tVyBbSTJ8wYzw0gapu1iygsWD7KRHZCN7etGVeiKrG7eae3nNH7Jf7R+nv0Q10J5mbZQNnnjUFT+5vqK77eEEHrM2SwtNgsmzPLxZ1skqimlw5zIWp4XeO47Sax6/DdGucyLhDzaTNUHSN/3rYwrK5sQINRffZl7tBOZ5asdvbPhR/n0Zlkl7a2yL6VrQNTVtvWLCneUzFCXL6twX6prcW9TaupCxhaXEpZtW1BHyjnoXJkVJiCOKV+jTpxDqnzxKHtEPGRxf7irxXzSrAecePdPo0jJ13ZDq9LwtUmTvre/zxoKt4RRy9gwGSwmQ1RVnmluFtLDrcCYF3vfahLDe9knLlXjnNDXXIhblSTsMfYmw3hP0s4jjLy5mcXHcMnJDYEBJjIPuEtM70ojUX6X1Iio6PRIAd11xA3CxqnckyUs7riZ0pXmDBBcIjkHwFmZjXyMZs34xlvaCgdznSHWbOKGmhrSgQJciGkW10MiJbEN/iFVp4Ewhu2NUeAMv0UOqvan66ok1tp9gGlSAYVvYl4jfwF1D+4K06djnEQueyBugD9WTGCZb+8RrX12XShUPkFJ52O1OkDhe8JloLDjk5A7zMnb+zlPA= X-Forefront-PRVS: 0220D4B98D X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(39410400002)(39850400002)(39860400002)(39840400002)(39450400003)(189002)(199003)(86362001)(25786008)(33646002)(2201001)(53416004)(42186005)(5660300001)(97746001)(9686003)(54356999)(50986999)(106356001)(2906002)(55016002)(101416001)(54906002)(6506006)(76176999)(105586002)(8676002)(92566002)(81156014)(47776003)(50466002)(81166006)(103116003)(6666003)(230700001)(189998001)(83506001)(4001350100001)(38730400002)(68736007)(4326007)(23676002)(1076002)(7416002)(97736004)(305945005)(2950100002)(7736002)(6116002)(69596002)(3846002)(53936002)(389900003)(66066001)(2004002)(71626007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR12MB1150;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjEyTUIxMTUwOzIzOmtVV2E3Sm95VE9xWFJuZXo4dElwR3hnbjdR?= =?utf-8?B?aVBhelVzd2dVS0FKOGt4ZWZlUlY0YzBQb2tHNWxPd1RIYnEwdXZRcWtEaTZC?= =?utf-8?B?cTJOb1FNcWhHYlpUSlpXMmhucWdwaW9DUGJHTnNWRHNSQlJOUGtSRGk5citk?= =?utf-8?B?R1hvVWpaelNWaTVQMmZOYkRiWHQ3ZzAyc0dqOTlHbUJya3FZRnBoV1NPeDBW?= =?utf-8?B?eUpZU0lPajA0eXVxTW0zVWc3amw2ZGZQczB1N3gxRkVHMTk1QzY5Y09ZcUY1?= =?utf-8?B?Z3A4L081ejJEeXNZdFBTWU1XaUY4bFBCMm1zMGhyS0NHbFNVQlA1dzV5aUFD?= =?utf-8?B?SDNaaTdJMStxdnlrcWR6dUhhYVA0MVVvMVg2aERTdlN4Rzc4NlpnMHlCdTNR?= =?utf-8?B?UUVHdVprK3VGTzVxbmd0NndTMDN6MGZlZ1dKRWtyTGNIQU5aaDRmZEpjcW83?= =?utf-8?B?RzBaV0JaeGdCaGdNOU9VRVdsaXpYaElJL2I2NnRFdUxrODY4dElQNHIrMjcx?= =?utf-8?B?ZGZ4Z000Y0h5YldIRzN5dnpVYXIxK0xCTWM3VU0xWG5KMEppdGdRUk5rNVJW?= =?utf-8?B?RXl0VGlBR3dOOWdkQndwb2dOcENkUnlOb2NrU0VBTGxNSytpTlRQNDczaUE1?= =?utf-8?B?UVZvYVYwSHMxMUd6aStEVDRScUlmZlVuUEdZSnM0S3MrZ2dmbDhSaGhHM01W?= =?utf-8?B?Si9Obzg1RHhHQS9vTDRaeERDMWZaSkRBUlljR3U4Wkk3VmUzZmJsSmoyT0d1?= =?utf-8?B?dS9YQUVveGZRMkpKTjJGTjRlZ09mbjRiVDJSMTBvcHpnSnRWYnFaaC92aDRr?= =?utf-8?B?WnhnR1hRSy9hV1M5M1AxWlRsa1B3SmdLekIvdk4wTmppdzVIWFVnQmI4cUw1?= =?utf-8?B?R0xXMkQyeTF2eWpJQ3RNK2t6T2cxbG5HOVB3RjRWWVV0d3FjdGMrT1QyWm1B?= =?utf-8?B?UkcxUjlyRFdmRStWN1NUMzRpRVVFWWcxTkhaeks4cE15Rm1EbVNPOWxRbUZW?= =?utf-8?B?VG1JckdRb2F0VWlVOXlOUWh5VUpmRFFWU0ZCSE9lVVJLWnRqTlI3Nk1ITG44?= =?utf-8?B?dGEremhKZldTVmVCV0l0dnlNZVFlY2plSzhvMDBXblRaN3pYNXV6eE02YkZ1?= =?utf-8?B?N2d0QmtWVGtVNkpkL0lrSTROM2NMdDZnSmNUTlgrbkU3MDNXeWZmeVJ0VlQv?= =?utf-8?B?QnVhM0RNNFcwTmJIVCtDU0crdmw0eG1rRE1TVWtKaUwxUm1zakVrNkFSVm1p?= =?utf-8?B?NkpwY0Y1VFhrdFQ4ZW90Ykt0TDhHMG1FZysyZkRvbytWTUJaLzRMQklBWExK?= =?utf-8?B?NC9YMVY4NE5MVGRiNnp2M0I2d3M2VzgyOW8rSDJhTUJqZ1RsRUJrMU4xeEMz?= =?utf-8?B?ZXBWcFhFbndQMGEyVEdLNEs5L3dvd1dPR2gwMCsxcExUOUVoa0dMTjIrY0Jh?= =?utf-8?B?RXJ1cTllYlRyeTE0NFBLUUJtcmRyRENWM0ZyZ29UR2owNHRxUm41ZXVVRWFN?= =?utf-8?B?Ky9mRkhVanMxaFd2Z2ZRc2ZnWVl6cDUrRHlXdFBXenFWTy9qZS9hTkNRNnl3?= =?utf-8?B?T1lOTi9uaVNHK2pSczVSNEZKenVDRXBoWWMzdUpYU3N3UGpYaU5aWXB1SHNH?= =?utf-8?B?cE1RQ08wak1udHRIb2JkK21aMGFQV2t0RzJ0cUpxeHEzWEZvRmUwM09pbDVP?= =?utf-8?B?V1BsMVh5bEVpeTQvSEpJY2d4OURwVGE2S25OaCtTZjF3TUpVZkMwZnJTblJi?= =?utf-8?B?MHVsNVl2Qmt4K0NHVFVGL2hING8zWFF5QXlLTjFnN1lOUUhuUmRIMGdnZEdC?= =?utf-8?B?U0JiQktwZnhDdWRscUEvd1hPbTRkeVl5VTI5Nk5nb0IrRVFxNTZ1TkljalNP?= =?utf-8?B?RzhNRlRjclA2YW9uWnZPL2o3aFNoeHhKd08wT2NqTHg5UTI2MXZLRVBFQzNs?= =?utf-8?Q?BpdmNRQWdUG1XpzfMzw9+GWhJqI804=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1150;6:rshB7sN7SwP0wv9CVQICKBkqKtGiqfq2CNLn0DDWrbdfGNYahapEB9XhOBd8v8gYaIewq7y8wIQ7Jhh4cIoqLbOza0Dk0NDe8qidvKYKX7RcsIVe4S9SS+klHtTG6/8BmFAcpEObp7E4E2BGl0paUMV0Qds4ns3rOq40XAktevEpWoUIjeIhMhr7Sl1ikamvX0R8XLIKVbOptoBr6qsmwafmldu6EWsyRTw5WfmAe9otVIhFijwuP7ZNsKQKBWsfyJJQZAoLIhQYxznV9zCL7yRhyfrOo7Qssf6M4JZBWSvB1DCMniPSyO0giscvKy2AlSS5RaaRGUaLm9cSlROby7ZRiotAyJq09/suRCJ8dE6nSSvGTyrD93PF/CJqo7JJXbVZbkmuNWkulWRf0bMvXclv1pmPtfHCy1/RghPm+Rg=;5:dUqeMJKZXqXjqOuteLtIclqRLFt1upypoHTBIE88l9Z0HlsvH8cpx1uFkbDM9QKOxa0eP0Owef/33yzXOef8JLFO7mIYEmVv9MXHocx1AlrFLQGxPG6axHBCm3eGZXPV44QHqe9s0WeqduvUAV6xEQ==;24:f0nZ3qStgoeIq6g578+TWb9qs2cFJWhH3LqPf5m1e82T56DDYnfYwOivLiqC6RrVydrd67f4ChleUG5H0DwiFeAkZlm1Gltoy7LOOQwJFIY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1150;7:BZb5/P9aF6ThVE162kiF1DmcIMABXcfrW2GmqcKKNmmLqnDR2JEDZnwybk7txKvk53M1A797e9eleu23cVyrwKKKL6jwSYp7ZOWfgI560cvQKdd4zkFaySvjb6G4/YfKNBCiTr1Fucz4byDEJwo+nOfdtkOuLu/hiKTgV0+fJkTJYWHGomSiDAysflIsxg7j9o5UU56hOt0bI67Nw9j280sSyja3TYDwzoJmbDFba0IAA+SNWFESZxCH76BhdoWM5lhKgTD6+3GvpRhaJybEqcERapQFDpz4ZIpFOdOfbRoBSiLLwF+AnWmfjLem1id0coT1B24oN0z3m7dZO76ISg==;20:BlCsh6Djq0SRDRO5FNWqtTRdz3Mgos7LF5CKE9oLWtB+eaDj61qiZqnU8BNlVSdBnjgoU7+VEr1avuIgdLBS6cuHprctV1ErIgER8yRaVlXVZzfN3nDnL4kkNTFN7ivOyqsZXvegUF2+0hgMchpyYW2T3HRwO5VwxbMYXXfJSxfsnUwtjuTsP6mkmVaUtYPAmq3aqVlZTnNI10XCg1ceyHjDn5VtFfUyXrJj2kqJOO/IrtT8mqubBpOMQMpmG402 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2017 15:48:12.8630 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1150 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch adds the support to encrypt the kernel in-place. This is done by creating new page mappings for the kernel - a decrypted write-protected mapping and an encrypted mapping. The kernel is encyrpted by copying the kernel through a temporary buffer. Signed-off-by: Tom Lendacky --- arch/x86/kernel/Makefile | 1 arch/x86/kernel/mem_encrypt_boot.S | 156 +++++++++++++++++++++++++++++ arch/x86/kernel/mem_encrypt_init.c | 191 ++++++++++++++++++++++++++++++++++++ 3 files changed, 348 insertions(+) create mode 100644 arch/x86/kernel/mem_encrypt_boot.S diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 33af80a..dc3ed84 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -142,4 +142,5 @@ ifeq ($(CONFIG_X86_64),y) obj-y += vsmp_64.o obj-y += mem_encrypt_init.o + obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o endif diff --git a/arch/x86/kernel/mem_encrypt_boot.S b/arch/x86/kernel/mem_encrypt_boot.S new file mode 100644 index 0000000..58e1756 --- /dev/null +++ b/arch/x86/kernel/mem_encrypt_boot.S @@ -0,0 +1,156 @@ +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2016 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + + .text + .code64 +ENTRY(sme_encrypt_execute) + +#ifdef CONFIG_AMD_MEM_ENCRYPT + /* + * Entry parameters: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - length of kernel + * RCX - address of the encryption workarea + * - stack page (PAGE_SIZE) + * - encryption routine page (PAGE_SIZE) + * - intermediate copy buffer (PMD_PAGE_SIZE) + * R8 - address of the pagetables to use for encryption + */ + + /* Set up a one page stack in the non-encrypted memory area */ + movq %rcx, %rax + addq $PAGE_SIZE, %rax + movq %rsp, %rbp + movq %rax, %rsp + push %rbp + + push %r12 + push %r13 + + movq %rdi, %r10 + movq %rsi, %r11 + movq %rdx, %r12 + movq %rcx, %r13 + + /* Copy encryption routine into the workarea */ + movq %rax, %rdi + leaq .Lencrypt_start(%rip), %rsi + movq $(.Lencrypt_stop - .Lencrypt_start), %rcx + rep movsb + + /* Setup registers for call */ + movq %r10, %rdi + movq %r11, %rsi + movq %r8, %rdx + movq %r12, %rcx + movq %rax, %r8 + addq $PAGE_SIZE, %r8 + + /* Call the encryption routine */ + call *%rax + + pop %r13 + pop %r12 + + pop %rsp /* Restore original stack pointer */ +.Lencrypt_exit: +#endif /* CONFIG_AMD_MEM_ENCRYPT */ + + ret +ENDPROC(sme_encrypt_execute) + +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * Routine used to encrypt kernel. + * This routine must be run outside of the kernel proper since + * the kernel will be encrypted during the process. So this + * routine is defined here and then copied to an area outside + * of the kernel where it will remain and run decrypted + * during execution. + * + * On entry the registers must be: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - address of the pagetables to use for encryption + * RCX - length of kernel + * R8 - intermediate copy buffer + * + * RAX - points to this routine + * + * The kernel will be encrypted by copying from the non-encrypted + * kernel space to an intermediate buffer and then copying from the + * intermediate buffer back to the encrypted kernel space. The physical + * addresses of the two kernel space mappings are the same which + * results in the kernel being encrypted "in place". + */ +.Lencrypt_start: + /* Enable the new page tables */ + mov %rdx, %cr3 + + /* Flush any global TLBs */ + mov %cr4, %rdx + andq $~X86_CR4_PGE, %rdx + mov %rdx, %cr4 + orq $X86_CR4_PGE, %rdx + mov %rdx, %cr4 + + /* Set the PAT register PA5 entry to write-protect */ + push %rcx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + push %rdx /* Save original PAT value */ + andl $0xffff00ff, %edx /* Clear PA5 */ + orl $0x00000500, %edx /* Set PA5 to WP */ + wrmsr + pop %rdx /* RDX contains original PAT value */ + pop %rcx + + movq %rcx, %r9 /* Save length */ + movq %rdi, %r10 /* Save destination address */ + movq %rsi, %r11 /* Save source address */ + + wbinvd /* Invalidate any cache entries */ + + /* Copy/encrypt 2MB at a time */ +1: + movq %r11, %rsi + movq %r8, %rdi + movq $PMD_PAGE_SIZE, %rcx + rep movsb + + movq %r8, %rsi + movq %r10, %rdi + movq $PMD_PAGE_SIZE, %rcx + rep movsb + + addq $PMD_PAGE_SIZE, %r11 + addq $PMD_PAGE_SIZE, %r10 + subq $PMD_PAGE_SIZE, %r9 + jnz 1b + + /* Restore PAT register */ + push %rdx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + pop %rdx + wrmsr + + ret +.Lencrypt_stop: +#endif /* CONFIG_AMD_MEM_ENCRYPT */ diff --git a/arch/x86/kernel/mem_encrypt_init.c b/arch/x86/kernel/mem_encrypt_init.c index 25af15d..07cbb90 100644 --- a/arch/x86/kernel/mem_encrypt_init.c +++ b/arch/x86/kernel/mem_encrypt_init.c @@ -16,9 +16,200 @@ #ifdef CONFIG_AMD_MEM_ENCRYPT #include +#include + +#include + +extern void sme_encrypt_execute(unsigned long, unsigned long, unsigned long, + void *, pgd_t *); + +#define PGD_FLAGS _KERNPG_TABLE_NOENC +#define PUD_FLAGS _KERNPG_TABLE_NOENC +#define PMD_FLAGS __PAGE_KERNEL_LARGE_EXEC + +static void __init *sme_pgtable_entry(pgd_t *pgd, void *next_page, + void *vaddr, pmdval_t pmd_val) +{ + pud_t *pud; + pmd_t *pmd; + + pgd += pgd_index((unsigned long)vaddr); + if (pgd_none(*pgd)) { + pud = next_page; + memset(pud, 0, sizeof(*pud) * PTRS_PER_PUD); + native_set_pgd(pgd, + native_make_pgd((unsigned long)pud + PGD_FLAGS)); + next_page += sizeof(*pud) * PTRS_PER_PUD; + } else { + pud = (pud_t *)(native_pgd_val(*pgd) & ~PTE_FLAGS_MASK); + } + + pud += pud_index((unsigned long)vaddr); + if (pud_none(*pud)) { + pmd = next_page; + memset(pmd, 0, sizeof(*pmd) * PTRS_PER_PMD); + native_set_pud(pud, + native_make_pud((unsigned long)pmd + PUD_FLAGS)); + next_page += sizeof(*pmd) * PTRS_PER_PMD; + } else { + pmd = (pmd_t *)(native_pud_val(*pud) & ~PTE_FLAGS_MASK); + } + + pmd += pmd_index((unsigned long)vaddr); + if (pmd_none(*pmd) || !pmd_large(*pmd)) + native_set_pmd(pmd, native_make_pmd(pmd_val)); + + return next_page; +} + +static unsigned long __init sme_pgtable_calc(unsigned long start, + unsigned long end) +{ + unsigned long addr, total; + + total = 0; + addr = start; + while (addr < end) { + unsigned long pgd_end; + + pgd_end = (addr & PGDIR_MASK) + PGDIR_SIZE; + if (pgd_end > end) + pgd_end = end; + + total += sizeof(pud_t) * PTRS_PER_PUD * 2; + + while (addr < pgd_end) { + unsigned long pud_end; + + pud_end = (addr & PUD_MASK) + PUD_SIZE; + if (pud_end > end) + pud_end = end; + + total += sizeof(pmd_t) * PTRS_PER_PMD * 2; + + addr = pud_end; + } + + addr = pgd_end; + } + total += sizeof(pgd_t) * PTRS_PER_PGD; + + return total; +} void __init sme_encrypt_kernel(void) { + pgd_t *pgd; + void *workarea, *next_page, *vaddr; + unsigned long kern_start, kern_end, kern_len; + unsigned long index, paddr, pmd_flags; + unsigned long exec_size, full_size; + + /* If SME is not active then no need to prepare */ + if (!sme_active()) + return; + + /* Set the workarea to be after the kernel */ + workarea = (void *)ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE); + + /* + * Prepare for encrypting the kernel by building new pagetables with + * the necessary attributes needed to encrypt the kernel in place. + * + * One range of virtual addresses will map the memory occupied + * by the kernel as encrypted. + * + * Another range of virtual addresses will map the memory occupied + * by the kernel as decrypted and write-protected. + * + * The use of write-protect attribute will prevent any of the + * memory from being cached. + */ + + /* Physical address gives us the identity mapped virtual address */ + kern_start = __pa_symbol(_text); + kern_end = ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE) - 1; + kern_len = kern_end - kern_start + 1; + + /* + * Calculate required number of workarea bytes needed: + * executable encryption area size: + * stack page (PAGE_SIZE) + * encryption routine page (PAGE_SIZE) + * intermediate copy buffer (PMD_PAGE_SIZE) + * pagetable structures for workarea (in case not currently mapped) + * pagetable structures for the encryption of the kernel + */ + exec_size = (PAGE_SIZE * 2) + PMD_PAGE_SIZE; + + full_size = exec_size; + full_size += ALIGN(exec_size, PMD_PAGE_SIZE) / PMD_PAGE_SIZE * + sizeof(pmd_t) * PTRS_PER_PMD; + full_size += sme_pgtable_calc(kern_start, kern_end + exec_size); + + next_page = workarea + exec_size; + + /* Make sure the current pagetables have entries for the workarea */ + pgd = (pgd_t *)native_read_cr3(); + paddr = (unsigned long)workarea; + while (paddr < (unsigned long)workarea + full_size) { + vaddr = (void *)paddr; + next_page = sme_pgtable_entry(pgd, next_page, vaddr, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + native_write_cr3(native_read_cr3()); + + /* Calculate a PGD index to be used for the decrypted mapping */ + index = (pgd_index(kern_end + full_size) + 1) & (PTRS_PER_PGD - 1); + index <<= PGDIR_SHIFT; + + /* Set and clear the PGD */ + pgd = next_page; + memset(pgd, 0, sizeof(*pgd) * PTRS_PER_PGD); + next_page += sizeof(*pgd) * PTRS_PER_PGD; + + /* Add encrypted (identity) mappings for the kernel */ + pmd_flags = PMD_FLAGS | _PAGE_ENC; + paddr = kern_start; + while (paddr < kern_end) { + vaddr = (void *)paddr; + next_page = sme_pgtable_entry(pgd, next_page, vaddr, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* Add decrypted (non-identity) mappings for the kernel */ + pmd_flags = (PMD_FLAGS & ~_PAGE_CACHE_MASK) | (_PAGE_PAT | _PAGE_PWT); + paddr = kern_start; + while (paddr < kern_end) { + vaddr = (void *)(paddr + index); + next_page = sme_pgtable_entry(pgd, next_page, vaddr, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* Add the workarea to both mappings */ + paddr = kern_end + 1; + while (paddr < (kern_end + exec_size)) { + vaddr = (void *)paddr; + next_page = sme_pgtable_entry(pgd, next_page, vaddr, + paddr + PMD_FLAGS); + + vaddr = (void *)(paddr + index); + next_page = sme_pgtable_entry(pgd, next_page, vaddr, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Perform the encryption */ + sme_encrypt_execute(kern_start, kern_start + index, kern_len, + workarea, pgd); + } unsigned long __init sme_get_me_mask(void)