From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934149AbeAXPAN (ORCPT ); Wed, 24 Jan 2018 10:00:13 -0500 Received: from mail-sn1nam01on0065.outbound.protection.outlook.com ([104.47.32.65]:38148 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S934128AbeAXPAJ (ORCPT ); Wed, 24 Jan 2018 10:00:09 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Subject: Re: [PATCH 1/3] x86/mm/encrypt: Move sme_populate_pgd*() into separate translation unit To: "Kirill A. Shutemov" , Ingo Molnar , x86@kernel.org, Thomas Gleixner , "H. Peter Anvin" Cc: Linus Torvalds , Borislav Petkov , linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20180123171910.55841-1-kirill.shutemov@linux.intel.com> <20180123171910.55841-2-kirill.shutemov@linux.intel.com> From: Tom Lendacky Message-ID: <1dee5138-fe8e-70ec-61d1-86f802e68e7c@amd.com> Date: Wed, 24 Jan 2018 08:59:58 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <20180123171910.55841-2-kirill.shutemov@linux.intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.78.1] X-ClientProxiedBy: DM5PR21CA0060.namprd21.prod.outlook.com (10.175.112.150) To BN6PR12MB1139.namprd12.prod.outlook.com (10.168.226.141) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: f09bb81f-3dc6-457d-880d-08d5633b2764 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(48565401081)(5600026)(4604075)(2017052603307)(7153060)(7193020);SRVR:BN6PR12MB1139; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;3:gfcAGqcOHPa0ZiQm2tGsWU0wAO8Wrjeyr8STc3Et6JWFALPgfoMNqp5ds26/UExkHD/bCVPH7kAyClE22C4URUbm3Z85CzXIaL5lkeBfXQG+suHI0BR5ZDA4ktMu5Ay6xkQ5mKnH/QzmgfdqDoVc6J4PoeRed+NNJOgZBlNEG3wX8JezZPjYucziNc2MqK+pLOQg6QEJItMpif0f5+HrpivruBEffQ/LPD0mhyqW579A+dhOXltIOFdDmeO3bWq9;25:54hHNhLohnQvu3z9a8TUNVvctEQjhgoK3ylZj/3WY54Szpl4OInVSDMTlotAFpU8vnXxHeOLxOJoQXrxp5UKGoRwAl83JDocyz9XH75h/frvTp/H3p+8fNUvJfQGjzSxLA90lWX22KNhYzEFaXNELGpoWxriv2r0pZ8OoM2ETTf+ovs9h2YPw+qkZcbIRYiSsmiVk5ur08Q/Ay4RgCZmKcnxiM87zmUDhuscSziBRW9K+HWt941N/Ip6XUHt7J/RC06Y1eFWv5qAWi1pCrm6y3RK/itTFdMISjGEWUBpjyCGVdqQO+2SqJDU7F1eXHUYjNdEXYb1f5CEW0BzQqw84w==;31:HYT117nKXbAf1ZvgA4b/auNWq6mXIlgQIzVPVSg7Td59ukm7FXi9yd1nNji3LDNeFQLu/RJZYidCGDNKZFYnWDZtJBrXSbAKR/+ZKkpA+rdSJtW9fCe/ZZm6WPkwcElDKoLJsRLCvVhK5x6tZItOOsxCmUuu9L0Vc8J1JMWVrBPNTbMEFBko4UKUS5WLT45XeRbylBHZG2GzVH2pE2ly3kaObadVDKWxduBqYqAZtN0= X-MS-TrafficTypeDiagnostic: BN6PR12MB1139: X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;20:ZUtSZ08grK9nt2Y3X0E5zNFR0g9qhMq3sHozV7RaXI7twynTPcneGiieUBUXDD3fTrRQ8Fb5o414WwFARpq2PxFsV7YpqPKU45VdSxFUXNRu1bvHUJuVQzECOHs6zZKbr1yJOOkWzAlTzc+CB44QPHoLpe4CpCkDNedCNW++Rj48RUXkzr+QsxnPl74OyIm1n1iZiz/TWaftwiQH4N64xCD+o9c9AYgrdrwohOB9z+hI8P3GSis8bCeKu8cU+ScdgDc72xWbtusXZDNqgj41Um7HJjP8z+gc/l5geYOvPEinxpCq+LtZIe6CIu/wf9c3GRY/E5b/c3/RHt/GMf12jc6S11uuRMOxbeJEjQ/pgtDltRt2M9MY0C7yP1jDLMPYyYEI4mHh1+LMjRMyA5hZWUWKbB/DY+CWxNhImmY01pWWBbexPfeuJE8aPfRfNzlu4xx/M9HIAlSQooSr/UqjyEje2XjmJb3/kda8GCQ6IRobmSxexaQakNVQxDrtyG0u;4:fWlz02yNQzKIFYra9zLoD0UILcfo5DsZsIMDgydxJe2QpHC64CvezH9Uwj1nBgwysT8n+5u+aHJrIi7Nfs99EJ2KXzvrvkKcjKIIVZjLNoAp/oERazfLNC1BX8mPsXMfIl9+tbcYnOYmGLpueYB38k+GaS+lE1G4HZojRGp1DxyB/Q32xN4CMrJ5zWWszP+hgQKYSUkp1JDX/kaSRr+cXgswC/P3NtD7wbJBBTVcKvj9rzwlaSgAgJF8qiwfPdj4DVx0P9yosQbG2yIhKvwysde+pJkhTJSXV1CWF0hzWx8ZZ1PzABeiAZjc9x2jsnyofFkY/sTzd1AvnXjqrbD1EA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040501)(2401047)(5005006)(8121501046)(10201501046)(3231023)(2400081)(944501161)(93006095)(93001095)(3002001)(6055026)(6041288)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(6072148)(201708071742011);SRVR:BN6PR12MB1139;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1139; X-Forefront-PRVS: 056297E276 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(39860400002)(346002)(376002)(39380400002)(396003)(366004)(199004)(189003)(36756003)(47776003)(5660300001)(65956001)(65806001)(7736002)(26005)(31686004)(72206003)(66066001)(478600001)(16526018)(77096007)(105586002)(97736004)(386003)(230700001)(53546011)(2906002)(2950100002)(305945005)(6666003)(65826007)(86362001)(6486002)(6246003)(68736007)(59450400001)(3260700006)(31696002)(25786009)(53936002)(3846002)(90366009)(54906003)(316002)(6116002)(110136005)(81156014)(8936002)(81166006)(58126008)(8676002)(16576012)(52116002)(106356001)(4326008)(83506002)(64126003)(52146003)(229853002)(23676004)(50466002)(76176011)(2486003)(2004002);DIR:OUT;SFP:1101;SCL:1;SRVR:BN6PR12MB1139;H:[10.236.65.116];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjEyTUIxMTM5OzIzOjhPR1ZLUy84Z3ZyNDNQZUdocXVWU0Nhb2lk?= =?utf-8?B?NFZaZSsrNjRyQ1c3Z3BsUkZ5T2taZVQ5K2tuKzIycWpZUUEwY1lQejB4T1I4?= =?utf-8?B?YjNockNwTVpSRWM2NjdvaFlxODNXa2ZhZlk5M2pUWkZYNFB4VXcrWWhyNUFI?= =?utf-8?B?M3diTlR1S3BqbU04OVBRSXloL0I0OXhXaWZuSE5HNittRmxlc1p1eFpjMkl4?= =?utf-8?B?Y3VHekt3Uy9EY0RsSnh2ZDlLMTFpMXE4U2FSNXl2TmdCQ1pYRG9Ub0RnTkZt?= =?utf-8?B?cGtKWVp5OGhsVTl6cStkYlU2WnVSMERvdmEwSmpnQ2g5a0NwS0psL0JPY0M0?= =?utf-8?B?QVhMWTdEOE1HaUVXRzVGZE1ld0t6UUY2Y2xXcEc4S24wdWtUK1ZPODkydVRU?= =?utf-8?B?TmR3dUNKaGFqb1ZFRTVYR0RBeTBEMzQ0L2JMOXI0ZzVZa2gzMjZXVEJlc2pV?= =?utf-8?B?WXErM2FCQlVndWNaOG1OZ2ZKM0RjNExuL0NNM29XUUJYYU1lZXUrbVhKRTI2?= =?utf-8?B?c1F3M1BXaDR3NUp2eUkrRlBmMC9LbjI5RmFuYnhjanhndmJhRmIyQUNUTmdE?= =?utf-8?B?dURUSTkyR0d0R1N0Z2hpTHVaa0ZWM1RrOW1EVmsrVmRWeTFHcUlXQ0xOZjNT?= =?utf-8?B?Z1RydG5pQTNMRm1SWnd4alFhZitEdUF4cElMQVpyM2NCQkF1SEt4bldBdkZ4?= =?utf-8?B?WlRpOGxOZ2VLRStzYzZqNnRqS2FvSStKbFVHL0dKWmJ6aG5TOEQxRWpvRmh1?= =?utf-8?B?VE1tZHFUMks4b3NPdnY4NjZnTndia2tGVWJtbGg4MmhUbzRzaUVXS3Iva0NN?= =?utf-8?B?dFFqZk1JWTk4U0VOVmtnK0pIL2dnSVdZenNFMTlIMzNORFhLb0kzNjFpMVdj?= =?utf-8?B?My9QSmlKRk9ic1ppTlA4dUh5YVNWUWFNVWY2TnpMWENLdlNPem9GNEV5OGlt?= =?utf-8?B?ZlVScVFxRlNGTUFpK2pDSHdJREhwbHd0dmw2ekkxaDRNcU5VTkdaK3BDWDZn?= =?utf-8?B?ZmZyNE5YYjhHcWNLY0lhcDdjenplZFRFZzVYTStEZGpqZlNxR0lCaGpzWkFC?= =?utf-8?B?NXg4azlnNFpNeFN3L1JFUUZLTW5reHRXSGNjdWtocEhGZWUxWGF5RkZjT0VE?= =?utf-8?B?Tk8zT1BUTU9NNlF0S0cwZHFXVXZsRzEzUzJHWEdsT0QvZTNxQUREcjFib1pV?= =?utf-8?B?emI5QkpROTRFaG1EdlJzZVVsSUkwQkZ0NTRFVEx0bG5VVWJzQ21FeTlqSk5N?= =?utf-8?B?Y0xzbGJUWjJ5YVcwK3VhZ1B6QndDK1lIV0FyOFdvakFTck9RMlB5ZDZRQlN5?= =?utf-8?B?UFhBVVN5cFZBQXlDQ2x6ak5sMjVwOVIreTdTRE95NFUxWVFxcDFFUWdzK3dX?= =?utf-8?B?VTBzSU1ZWEdmRTkvT3dMVk1FU0tIVWxTSHBENUdEUlprdVdrTFdMMUUya0dJ?= =?utf-8?B?c043OUxjT1ZvVnlWYk41SWtxQy9XQWtrTGZJZUtqRk9NM0NjTFdWTUQ1ZmNx?= =?utf-8?B?ejdnTDhTRWsyM1kzRHZaOW5CNXozTVVTM2l2Ujk4YVdmc1FKV0VGZXlFUzdO?= =?utf-8?B?YnZ4cUp4bjhtdllTRmxkc3B3UGVoR1VXSU9oOTBvMFJSaDA4L0N4clFwODhV?= =?utf-8?B?eVY5QTR4MG5wOXV1cDk3TmU3U0huUEdsRmplSEN2d1plckxJK1I2NlNsUDA5?= =?utf-8?B?NEozRXJML2t2QzJWRU1OaGNDamZaNzYxM0pVUDlqUzhWMG1MNitSTElML2JW?= =?utf-8?B?dGdHVGRmMkVMUGYrSTVtKzAwQmpROVo2S003Q1MxWGFUUWR6eHJiWCtxQlBZ?= =?utf-8?B?T1liL1g2RkcxL1ozcVMzbWRUNnNPcTVJblNPdTk0WUZ3NzhlcUI4M25tTFV4?= =?utf-8?B?RWh5Q09VMjhMQUFIQVJ1dVAxb0pxdElheDNXTmxLRForT2U1bldYNTltQUFI?= =?utf-8?B?VWFILzRuRjV6N3J5Mmh0MmVMS0FlMTRkYS9CWWd6KzVwOWdCNkpvV1VoU01l?= =?utf-8?B?OTYxdWJxSXBkTmNCaXVCQkdZa3FON3hkckVxdz09?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;6:gZk+IFE++eO8T9y2lTpz6AOonto2pthhmvPOOQXdGsBwuZAYsRlc03e8fC3hv3ZtrXuBQabTWnfIL1cPJclRu5nCSadinZXkPMGz2u4/pXO62d/fCP1cqCyQJlT0xGmkNUmHJnG3IokP2g8HxCJz8hcdcqU0gTGa/lFd6UR1dLfQvkNCcqtl7k4W1I475Zb5q5EtktOoJ/lulEgRmrhZBybAgG0JGUtNFBnAhDf/0ubak2HhDqEqyCWHxu/QQVPnZS9M8BW1Ir9IE9Qe0L52jTjEFFQ1RAy2QJSRTWnzDLx8CA4vR/7alqm+/lKybJ5EgDLeJKsXQ8Kv1khMRT9jZZ6FifBrq8PPV9zsZS8KVp4=;5:EdWizas7k5TOaTZG2Y4bQW0Xl5p0fxF+xqKoPMyjDtTvbT8cxDnMn1yklU5EATPbk3vKUEsDhANG+T9KkG8i5KXE2uXexxc4ULLgxWFthmERWfRW7wT4VPBQUVJxt/XU3PuE0mWlyg46qda7q4Do7TlCpyCyvED2cMBmPklMkII=;24:Cnjvm9KQ52+SqK0FhW9Hf756q4twB/uK6biscfoG1ydDoKD3g2KkWZLEwMFYee3jynHmjTyjOfj8jfgpDPbo9bR79t5P+/Y07UaL9Z1pLAU=;7:md1+NdiOsiFsoyIMqheAwQWiX8jJ9G8uhyao67YT5sXCG+BrIKvdtEs9A+mbiTG/l5xMTuPLZKzWMVurCYDf04C+8axG8cp2erfze4QEqERSGH6TCUB9pj1YeiVhRN8+ebRH+IAFssyEdzyMpgM00hNcMrAR4SDY85aV1wBeDsN/zVVaw0ssKZui9n44+tvqJMm6/EgcwyMg6Ou9eOS9w8KhQuDadVxmQyaMJlYx5ThlNl6qLbk8+heoxjSWUFru SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;20:n+NdED8hd306bdACmbJZ1PQX2izHeHQdBR157kLSWV+0bJVFSbGd1Uq1bQ3Jkyvr2KL34aVnDMgEps7IAB/tteMdEKPidm7Sk4bcLvVSwA7n367P0c3Ky+lovO7HYrsh3LlzTYugEJmfvchNvRKYSgI2bTFqStA1DH38BqJJPyh8KHBlXmbZaKiUcDVJuWmR5Jjk4w9C8F9CyVebxM9XMrC6PY0+FbZpJY1ILauF3S61ysoLbdltMHh1LKxFfqju X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2018 15:00:04.7444 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f09bb81f-3dc6-457d-880d-08d5633b2764 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1139 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 1/23/2018 11:19 AM, Kirill A. Shutemov wrote: > sme_populate_pgd() and sme_populate_pgd_large() operate on the identity > mapping, which means they want virtual addresses to be equal to physical > one, without PAGE_OFFSET shift. > > We also need to avoid paravirtualizaion call there. > > Getting this done is tricky. We cannot use usual page table helpers. > It forces us to open-code a lot of things. It makes code ugly and hard > to modify. > > We can get it work with the page table helpers, but it requires few > preprocessor tricks. These tricks may have side effects for the rest of > the file. > > Let's isolate sme_populate_pgd() and sme_populate_pgd_large() into own > translation unit. > > Signed-off-by: Kirill A. Shutemov > --- > arch/x86/mm/Makefile | 13 ++-- > arch/x86/mm/mem_encrypt.c | 129 ----------------------------------- > arch/x86/mm/mem_encrypt_identity.c | 134 +++++++++++++++++++++++++++++++++++++ > arch/x86/mm/mm_internal.h | 14 ++++ > 4 files changed, 156 insertions(+), 134 deletions(-) > create mode 100644 arch/x86/mm/mem_encrypt_identity.c > > diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile > index 27e9e90a8d35..51e364ef12d9 100644 > --- a/arch/x86/mm/Makefile > +++ b/arch/x86/mm/Makefile > @@ -1,12 +1,14 @@ > # SPDX-License-Identifier: GPL-2.0 > -# Kernel does not boot with instrumentation of tlb.c and mem_encrypt.c > -KCOV_INSTRUMENT_tlb.o := n > -KCOV_INSTRUMENT_mem_encrypt.o := n > +# Kernel does not boot with instrumentation of tlb.c and mem_encrypt*.c > +KCOV_INSTRUMENT_tlb.o := n > +KCOV_INSTRUMENT_mem_encrypt.o := n > +KCOV_INSTRUMENT_mem_encrypt_identity.o := n > > -KASAN_SANITIZE_mem_encrypt.o := n > +KASAN_SANITIZE_mem_encrypt.o := n > +KASAN_SANITIZE_mem_encrypt_identity.o := n > > ifdef CONFIG_FUNCTION_TRACER > -CFLAGS_REMOVE_mem_encrypt.o = -pg You found this one already. > +CFLAGS_REMOVE_mem_encrypt_identity.o = -pg > endif > > obj-y := init.o init_$(BITS).o fault.o ioremap.o extable.o pageattr.o mmap.o \ > @@ -47,4 +49,5 @@ obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o > obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o > > obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o > +obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o > obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o > diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c > index e1d61e8500f9..740b8a54f616 100644 > --- a/arch/x86/mm/mem_encrypt.c > +++ b/arch/x86/mm/mem_encrypt.c > @@ -464,18 +464,6 @@ void swiotlb_set_mem_attributes(void *vaddr, unsigned long size) > set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT); > } > > -struct sme_populate_pgd_data { > - void *pgtable_area; > - pgd_t *pgd; > - > - pmdval_t pmd_flags; > - pteval_t pte_flags; > - unsigned long paddr; > - > - unsigned long vaddr; > - unsigned long vaddr_end; > -}; > - > static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) If you're going to move some of the functions, did you look at moving the sme_enable(), sme_encrypt_kernel(), etc., too? I believe everything below the sme_populate_pgd_data structure is used during early identity-mapped boot time. If you move everything, then mm_internal.h doesn't need to be updated and all of the identity-mapped early boot code ends up in one file. You'd have to move the command line declarations and make sev_enabled not a static, but it should be doable. Thanks, Tom > { > unsigned long pgd_start, pgd_end, pgd_size; > @@ -491,11 +479,6 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) > memset(pgd_p, 0, pgd_size); > } > > -#define PGD_FLAGS _KERNPG_TABLE_NOENC > -#define P4D_FLAGS _KERNPG_TABLE_NOENC > -#define PUD_FLAGS _KERNPG_TABLE_NOENC > -#define PMD_FLAGS _KERNPG_TABLE_NOENC > - > #define PMD_FLAGS_LARGE (__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL) > > #define PMD_FLAGS_DEC PMD_FLAGS_LARGE > @@ -512,118 +495,6 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) > > #define PTE_FLAGS_ENC (PTE_FLAGS | _PAGE_ENC) > > -static pmd_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) > -{ > - pgd_t *pgd_p; > - p4d_t *p4d_p; > - pud_t *pud_p; > - pmd_t *pmd_p; > - > - pgd_p = ppd->pgd + pgd_index(ppd->vaddr); > - if (native_pgd_val(*pgd_p)) { > - if (IS_ENABLED(CONFIG_X86_5LEVEL)) > - p4d_p = (p4d_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); > - else > - pud_p = (pud_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); > - } else { > - pgd_t pgd; > - > - if (IS_ENABLED(CONFIG_X86_5LEVEL)) { > - p4d_p = ppd->pgtable_area; > - memset(p4d_p, 0, sizeof(*p4d_p) * PTRS_PER_P4D); > - ppd->pgtable_area += sizeof(*p4d_p) * PTRS_PER_P4D; > - > - pgd = native_make_pgd((pgdval_t)p4d_p + PGD_FLAGS); > - } else { > - pud_p = ppd->pgtable_area; > - memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); > - ppd->pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; > - > - pgd = native_make_pgd((pgdval_t)pud_p + PGD_FLAGS); > - } > - native_set_pgd(pgd_p, pgd); > - } > - > - if (IS_ENABLED(CONFIG_X86_5LEVEL)) { > - p4d_p += p4d_index(ppd->vaddr); > - if (native_p4d_val(*p4d_p)) { > - pud_p = (pud_t *)(native_p4d_val(*p4d_p) & ~PTE_FLAGS_MASK); > - } else { > - p4d_t p4d; > - > - pud_p = ppd->pgtable_area; > - memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); > - ppd->pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; > - > - p4d = native_make_p4d((pudval_t)pud_p + P4D_FLAGS); > - native_set_p4d(p4d_p, p4d); > - } > - } > - > - pud_p += pud_index(ppd->vaddr); > - if (native_pud_val(*pud_p)) { > - if (native_pud_val(*pud_p) & _PAGE_PSE) > - return NULL; > - > - pmd_p = (pmd_t *)(native_pud_val(*pud_p) & ~PTE_FLAGS_MASK); > - } else { > - pud_t pud; > - > - pmd_p = ppd->pgtable_area; > - memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); > - ppd->pgtable_area += sizeof(*pmd_p) * PTRS_PER_PMD; > - > - pud = native_make_pud((pmdval_t)pmd_p + PUD_FLAGS); > - native_set_pud(pud_p, pud); > - } > - > - return pmd_p; > -} > - > -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) > -{ > - pmd_t *pmd_p; > - > - pmd_p = sme_prepare_pgd(ppd); > - if (!pmd_p) > - return; > - > - pmd_p += pmd_index(ppd->vaddr); > - if (!native_pmd_val(*pmd_p) || !(native_pmd_val(*pmd_p) & _PAGE_PSE)) > - native_set_pmd(pmd_p, native_make_pmd(ppd->paddr | ppd->pmd_flags)); > -} > - > -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) > -{ > - pmd_t *pmd_p; > - pte_t *pte_p; > - > - pmd_p = sme_prepare_pgd(ppd); > - if (!pmd_p) > - return; > - > - pmd_p += pmd_index(ppd->vaddr); > - if (native_pmd_val(*pmd_p)) { > - if (native_pmd_val(*pmd_p) & _PAGE_PSE) > - return; > - > - pte_p = (pte_t *)(native_pmd_val(*pmd_p) & ~PTE_FLAGS_MASK); > - } else { > - pmd_t pmd; > - > - pte_p = ppd->pgtable_area; > - memset(pte_p, 0, sizeof(*pte_p) * PTRS_PER_PTE); > - ppd->pgtable_area += sizeof(*pte_p) * PTRS_PER_PTE; > - > - pmd = native_make_pmd((pteval_t)pte_p + PMD_FLAGS); > - native_set_pmd(pmd_p, pmd); > - } > - > - pte_p += pte_index(ppd->vaddr); > - if (!native_pte_val(*pte_p)) > - native_set_pte(pte_p, native_make_pte(ppd->paddr | ppd->pte_flags)); > -} > - > static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) > { > while (ppd->vaddr < ppd->vaddr_end) { > diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c > new file mode 100644 > index 000000000000..dbf7a98f657d > --- /dev/null > +++ b/arch/x86/mm/mem_encrypt_identity.c > @@ -0,0 +1,134 @@ > +/* > + * AMD Memory Encryption Support > + * > + * Copyright (C) 2016 Advanced Micro Devices, Inc. > + * > + * Author: Tom Lendacky > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + */ > + > +#define DISABLE_BRANCH_PROFILING > + > +#include > +#include > +#include "mm_internal.h" > + > +#define PGD_FLAGS _KERNPG_TABLE_NOENC > +#define P4D_FLAGS _KERNPG_TABLE_NOENC > +#define PUD_FLAGS _KERNPG_TABLE_NOENC > +#define PMD_FLAGS _KERNPG_TABLE_NOENC > + > +static pmd_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) > +{ > + pgd_t *pgd_p; > + p4d_t *p4d_p; > + pud_t *pud_p; > + pmd_t *pmd_p; > + > + pgd_p = ppd->pgd + pgd_index(ppd->vaddr); > + if (native_pgd_val(*pgd_p)) { > + if (IS_ENABLED(CONFIG_X86_5LEVEL)) > + p4d_p = (p4d_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); > + else > + pud_p = (pud_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); > + } else { > + pgd_t pgd; > + > + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { > + p4d_p = ppd->pgtable_area; > + memset(p4d_p, 0, sizeof(*p4d_p) * PTRS_PER_P4D); > + ppd->pgtable_area += sizeof(*p4d_p) * PTRS_PER_P4D; > + > + pgd = native_make_pgd((pgdval_t)p4d_p + PGD_FLAGS); > + } else { > + pud_p = ppd->pgtable_area; > + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); > + ppd->pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; > + > + pgd = native_make_pgd((pgdval_t)pud_p + PGD_FLAGS); > + } > + native_set_pgd(pgd_p, pgd); > + } > + > + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { > + p4d_p += p4d_index(ppd->vaddr); > + if (native_p4d_val(*p4d_p)) { > + pud_p = (pud_t *)(native_p4d_val(*p4d_p) & ~PTE_FLAGS_MASK); > + } else { > + p4d_t p4d; > + > + pud_p = ppd->pgtable_area; > + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); > + ppd->pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; > + > + p4d = native_make_p4d((pudval_t)pud_p + P4D_FLAGS); > + native_set_p4d(p4d_p, p4d); > + } > + } > + > + pud_p += pud_index(ppd->vaddr); > + if (native_pud_val(*pud_p)) { > + if (native_pud_val(*pud_p) & _PAGE_PSE) > + return NULL; > + > + pmd_p = (pmd_t *)(native_pud_val(*pud_p) & ~PTE_FLAGS_MASK); > + } else { > + pud_t pud; > + > + pmd_p = ppd->pgtable_area; > + memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); > + ppd->pgtable_area += sizeof(*pmd_p) * PTRS_PER_PMD; > + > + pud = native_make_pud((pmdval_t)pmd_p + PUD_FLAGS); > + native_set_pud(pud_p, pud); > + } > + > + return pmd_p; > +} > + > +void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) > +{ > + pmd_t *pmd_p; > + > + pmd_p = sme_prepare_pgd(ppd); > + if (!pmd_p) > + return; > + > + pmd_p += pmd_index(ppd->vaddr); > + if (!native_pmd_val(*pmd_p) || !(native_pmd_val(*pmd_p) & _PAGE_PSE)) > + native_set_pmd(pmd_p, native_make_pmd(ppd->paddr | ppd->pmd_flags)); > +} > + > +void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) > +{ > + pmd_t *pmd_p; > + pte_t *pte_p; > + > + pmd_p = sme_prepare_pgd(ppd); > + if (!pmd_p) > + return; > + > + pmd_p += pmd_index(ppd->vaddr); > + if (native_pmd_val(*pmd_p)) { > + if (native_pmd_val(*pmd_p) & _PAGE_PSE) > + return; > + > + pte_p = (pte_t *)(native_pmd_val(*pmd_p) & ~PTE_FLAGS_MASK); > + } else { > + pmd_t pmd; > + > + pte_p = ppd->pgtable_area; > + memset(pte_p, 0, sizeof(*pte_p) * PTRS_PER_PTE); > + ppd->pgtable_area += sizeof(*pte_p) * PTRS_PER_PTE; > + > + pmd = native_make_pmd((pteval_t)pte_p + PMD_FLAGS); > + native_set_pmd(pmd_p, pmd); > + } > + > + pte_p += pte_index(ppd->vaddr); > + if (!native_pte_val(*pte_p)) > + native_set_pte(pte_p, native_make_pte(ppd->paddr | ppd->pte_flags)); > +} > diff --git a/arch/x86/mm/mm_internal.h b/arch/x86/mm/mm_internal.h > index 4e1f6e1b8159..b3ab82ae9b12 100644 > --- a/arch/x86/mm/mm_internal.h > +++ b/arch/x86/mm/mm_internal.h > @@ -19,4 +19,18 @@ extern int after_bootmem; > > void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache); > > +struct sme_populate_pgd_data { > + void *pgtable_area; > + pgd_t *pgd; > + > + pmdval_t pmd_flags; > + pteval_t pte_flags; > + unsigned long paddr; > + > + unsigned long vaddr; > + unsigned long vaddr_end; > +}; > + > +void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd); > +void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd); > #endif /* __X86_MM_INTERNAL_H */ >