mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "tip-bot2 for Kirill A. Shutemov" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
	"Borislav Petkov (AMD)" <bp@alien8.de>,
	Rick Edgecombe <rick.p.edgecombe@intel.com>,
	Kai Huang <kai.huang@intel.com>, Tao Liu <ltao@redhat.com>,
	x86@kernel.org, linux-kernel@vger.kernel.org
Subject: [tip: x86/cc] x86/tdx: Convert shared memory back to private on kexec
Date: Tue, 18 Jun 2024 14:01:40 -0000	[thread overview]
Message-ID: <171871930069.10875.8795812694370524886.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20240614095904.1345461-12-kirill.shutemov@linux.intel.com>

The following commit has been merged into the x86/cc branch of tip:

Commit-ID:     859e63b789d6b17b3c64e51a0aabdc58752a0254
Gitweb:        https://git.kernel.org/tip/859e63b789d6b17b3c64e51a0aabdc58752a0254
Author:        Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
AuthorDate:    Fri, 14 Jun 2024 12:58:56 +03:00
Committer:     Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Mon, 17 Jun 2024 17:46:05 +02:00

x86/tdx: Convert shared memory back to private on kexec

TDX guests allocate shared buffers to perform I/O. It is done by allocating
pages normally from the buddy allocator and converting them to shared with
set_memory_decrypted().

The second, kexec-ed kernel has no idea what memory is converted this way. It
only sees E820_TYPE_RAM.

Accessing shared memory via private mapping is fatal. It leads to unrecoverable
TD exit.

On kexec, walk direct mapping and convert all shared memory back to private. It
makes all RAM private again and second kernel may use it normally.

The conversion occurs in two steps: stopping new conversions and unsharing all
memory. In the case of normal kexec, the stopping of conversions takes place
while scheduling is still functioning. This allows for waiting until any ongoing
conversions are finished. The second step is carried out when all CPUs except one
are inactive and interrupts are disabled. This prevents any conflicts with code
that may access shared memory.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Tested-by: Tao Liu <ltao@redhat.com>
Link: https://lore.kernel.org/r/20240614095904.1345461-12-kirill.shutemov@linux.intel.com
---
 arch/x86/coco/tdx/tdx.c           | 94 ++++++++++++++++++++++++++++++-
 arch/x86/include/asm/pgtable.h    |  5 ++-
 arch/x86/include/asm/set_memory.h |  3 +-
 arch/x86/mm/pat/set_memory.c      | 42 ++++++++++++-
 4 files changed, 141 insertions(+), 3 deletions(-)

diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
index 979891e..078e2ba 100644
--- a/arch/x86/coco/tdx/tdx.c
+++ b/arch/x86/coco/tdx/tdx.c
@@ -7,6 +7,7 @@
 #include <linux/cpufeature.h>
 #include <linux/export.h>
 #include <linux/io.h>
+#include <linux/kexec.h>
 #include <asm/coco.h>
 #include <asm/tdx.h>
 #include <asm/vmx.h>
@@ -14,6 +15,7 @@
 #include <asm/insn.h>
 #include <asm/insn-eval.h>
 #include <asm/pgtable.h>
+#include <asm/set_memory.h>
 
 /* MMIO direction */
 #define EPT_READ	0
@@ -831,6 +833,95 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages,
 	return 0;
 }
 
+/* Stop new private<->shared conversions */
+static void tdx_kexec_begin(void)
+{
+	if (!IS_ENABLED(CONFIG_KEXEC_CORE))
+		return;
+
+	/*
+	 * Crash kernel reaches here with interrupts disabled: can't wait for
+	 * conversions to finish.
+	 *
+	 * If race happened, just report and proceed.
+	 */
+	if (!set_memory_enc_stop_conversion())
+		pr_warn("Failed to stop shared<->private conversions\n");
+}
+
+/* Walk direct mapping and convert all shared memory back to private */
+static void tdx_kexec_finish(void)
+{
+	unsigned long addr, end;
+	long found = 0, shared;
+
+	if (!IS_ENABLED(CONFIG_KEXEC_CORE))
+		return;
+
+	lockdep_assert_irqs_disabled();
+
+	addr = PAGE_OFFSET;
+	end  = PAGE_OFFSET + get_max_mapped();
+
+	while (addr < end) {
+		unsigned long size;
+		unsigned int level;
+		pte_t *pte;
+
+		pte = lookup_address(addr, &level);
+		size = page_level_size(level);
+
+		if (pte && pte_decrypted(*pte)) {
+			int pages = size / PAGE_SIZE;
+
+			/*
+			 * Touching memory with shared bit set triggers implicit
+			 * conversion to shared.
+			 *
+			 * Make sure nobody touches the shared range from
+			 * now on.
+			 */
+			set_pte(pte, __pte(0));
+
+			/*
+			 * Memory encryption state persists across kexec.
+			 * If tdx_enc_status_changed() fails in the first
+			 * kernel, it leaves memory in an unknown state.
+			 *
+			 * If that memory remains shared, accessing it in the
+			 * *next* kernel through a private mapping will result
+			 * in an unrecoverable guest shutdown.
+			 *
+			 * The kdump kernel boot is not impacted as it uses
+			 * a pre-reserved memory range that is always private.
+			 * However, gathering crash information could lead to
+			 * a crash if it accesses unconverted memory through
+			 * a private mapping which is possible when accessing
+			 * that memory through /proc/vmcore, for example.
+			 *
+			 * In all cases, print error info in order to leave
+			 * enough bread crumbs for debugging.
+			 */
+			if (!tdx_enc_status_changed(addr, pages, true)) {
+				pr_err("Failed to unshare range %#lx-%#lx\n",
+				       addr, addr + size);
+			}
+
+			found += pages;
+		}
+
+		addr += size;
+	}
+
+	__flush_tlb_all();
+
+	shared = atomic_long_read(&nr_shared);
+	if (shared != found) {
+		pr_err("shared page accounting is off\n");
+		pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found);
+	}
+}
+
 void __init tdx_early_init(void)
 {
 	struct tdx_module_args args = {
@@ -890,6 +981,9 @@ void __init tdx_early_init(void)
 	x86_platform.guest.enc_cache_flush_required  = tdx_cache_flush_required;
 	x86_platform.guest.enc_tlb_flush_required    = tdx_tlb_flush_required;
 
+	x86_platform.guest.enc_kexec_begin	     = tdx_kexec_begin;
+	x86_platform.guest.enc_kexec_finish	     = tdx_kexec_finish;
+
 	/*
 	 * TDX intercepts the RDMSR to read the X2APIC ID in the parallel
 	 * bringup low level code. That raises #VE which cannot be handled
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 65b8e5b..e39311a 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -140,6 +140,11 @@ static inline int pte_young(pte_t pte)
 	return pte_flags(pte) & _PAGE_ACCESSED;
 }
 
+static inline bool pte_decrypted(pte_t pte)
+{
+	return cc_mkdec(pte_val(pte)) == pte_val(pte);
+}
+
 #define pmd_dirty pmd_dirty
 static inline bool pmd_dirty(pmd_t pmd)
 {
diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h
index 9aee318..4b2abce 100644
--- a/arch/x86/include/asm/set_memory.h
+++ b/arch/x86/include/asm/set_memory.h
@@ -49,8 +49,11 @@ int set_memory_wb(unsigned long addr, int numpages);
 int set_memory_np(unsigned long addr, int numpages);
 int set_memory_p(unsigned long addr, int numpages);
 int set_memory_4k(unsigned long addr, int numpages);
+
+bool set_memory_enc_stop_conversion(void);
 int set_memory_encrypted(unsigned long addr, int numpages);
 int set_memory_decrypted(unsigned long addr, int numpages);
+
 int set_memory_np_noalias(unsigned long addr, int numpages);
 int set_memory_nonglobal(unsigned long addr, int numpages);
 int set_memory_global(unsigned long addr, int numpages);
diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index a7a7a6c..443a97e 100644
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -2227,12 +2227,48 @@ vmm_fail:
 	return ret;
 }
 
+/*
+ * The lock serializes conversions between private and shared memory.
+ *
+ * It is taken for read on conversion. A write lock guarantees that no
+ * concurrent conversions are in progress.
+ */
+static DECLARE_RWSEM(mem_enc_lock);
+
+/*
+ * Stop new private<->shared conversions.
+ *
+ * Taking the exclusive mem_enc_lock waits for in-flight conversions to complete.
+ * The lock is not released to prevent new conversions from being started.
+ */
+bool set_memory_enc_stop_conversion(void)
+{
+	/*
+	 * In a crash scenario, sleep is not allowed. Try to take the lock.
+	 * Failure indicates that there is a race with the conversion.
+	 */
+	if (oops_in_progress)
+		return down_write_trylock(&mem_enc_lock);
+
+	down_write(&mem_enc_lock);
+
+	return true;
+}
+
 static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)
 {
-	if (cc_platform_has(CC_ATTR_MEM_ENCRYPT))
-		return __set_memory_enc_pgtable(addr, numpages, enc);
+	int ret = 0;
 
-	return 0;
+	if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
+		if (!down_read_trylock(&mem_enc_lock))
+			return -EBUSY;
+
+		ret = __set_memory_enc_pgtable(addr, numpages, enc);
+
+		up_read(&mem_enc_lock);
+	}
+
+	return ret;
 }
 
 int set_memory_encrypted(unsigned long addr, int numpages)

  reply	other threads:[~2024-06-18 14:01 UTC|newest]

Thread overview: 75+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-06-14  9:58 [PATCHv12 00/19] x86/tdx: Add kexec support Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 01/19] x86/acpi: Extract ACPI MADT wakeup code into a separate file Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 02/19] x86/apic: Mark acpi_mp_wake_* variables as __ro_after_init Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 03/19] cpu/hotplug: Add support for declaring CPU offlining not supported Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 04/19] cpu/hotplug, x86/acpi: Disable CPU offlining for ACPI MADT wakeup Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 05/19] x86/relocate_kernel: Use named labels for less confusion Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Borislav Petkov
2024-06-14  9:58 ` [PATCHv12 06/19] x86/kexec: Keep CR4.MCE set during kexec for TDX guest Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 07/19] x86/mm: Make x86_platform.guest.enc_status_change_*() return errno Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] x86/mm: Make x86_platform.guest.enc_status_change_*() return an error tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 08/19] x86/mm: Return correct level from lookup_address() if pte is none Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 09/19] x86/tdx: Account shared memory Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 10/19] x86/mm: Add callbacks to prepare encrypted memory for kexec Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 11/19] x86/tdx: Convert shared memory back to private on kexec Kirill A. Shutemov
2024-06-18 14:01   ` tip-bot2 for Kirill A. Shutemov [this message]
2024-06-14  9:58 ` [PATCHv12 12/19] x86/mm: Make e820__end_ram_pfn() cover E820_TYPE_ACPI ranges Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:58 ` [PATCHv12 13/19] x86/mm: Do not zap page table entries mapping unaccepted memory table during kdump Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Ashish Kalra
2024-06-14  9:58 ` [PATCHv12 14/19] x86/acpi: Rename fields in acpi_madt_multiproc_wakeup structure Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] x86/acpi: Rename fields in the " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:59 ` [PATCHv12 15/19] x86/acpi: Do not attempt to bring up secondary CPUs in kexec case Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] x86/acpi: Do not attempt to bring up secondary CPUs in the " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:59 ` [PATCHv12 16/19] x86/smp: Add smp_ops.stop_this_cpu() callback Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:59 ` [PATCHv12 17/19] x86/mm: Introduce kernel_ident_mapping_free() Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-14  9:59 ` [PATCHv12 18/19] x86/acpi: Add support for CPU offlining for ACPI MADT wakeup method Kirill A. Shutemov
2024-06-14  9:59 ` [PATCHv12 19/19] ACPI: tables: Print MULTIPROC_WAKEUP when MADT is parsed Kirill A. Shutemov
2024-06-18 14:01   ` [tip: x86/cc] " tip-bot2 for Kirill A. Shutemov
2024-06-17 21:13 ` [PATCH v8 0/2] x86/snp: Add kexec support Ashish Kalra
2024-06-17 21:15   ` [PATCH v8 1/2] x86/boot/compressed: Skip Video Memory access in Decompressor for SEV-ES/SNP Ashish Kalra
2024-06-19 10:22     ` Borislav Petkov
2024-06-17 21:15   ` [PATCH v8 2/2] x86/snp: Convert shared memory back to private on kexec Ashish Kalra
2024-06-20 22:22 ` [PATCH v9 0/3] x86/snp: Add kexec support Ashish Kalra
2024-06-20 22:23   ` [PATCH v9 1/3] x86/sev: Move SEV compilation units Ashish Kalra
2024-06-20 22:23   ` [PATCH v9 2/3] x86/boot: Skip video memory access in the decompressor for SEV-ES/SNP Ashish Kalra
2024-06-24 15:03     ` Tom Lendacky
2024-06-20 22:23   ` [PATCH v9 3/3] x86/snp: Convert shared memory back to private on kexec Ashish Kalra
2024-06-24 15:18     ` Tom Lendacky
2024-06-24 18:26     ` Borislav Petkov
2024-06-24 20:57       ` Kalra, Ashish
2024-06-25  3:59         ` Borislav Petkov
2024-06-28  4:27           ` Kalra, Ashish
2024-06-28 14:01             ` Tom Lendacky
2024-06-28 19:14               ` Kalra, Ashish
2024-06-28 20:33       ` Kalra, Ashish
2024-06-24 18:21 ` [PATCH v10 0/2] x86/snp: Add kexec support Ashish Kalra
2024-06-24 18:21   ` [PATCH v10 1/2] x86/boot: Skip video memory access in the decompressor for SEV-ES/SNP Ashish Kalra
2024-06-24 18:22   ` [PATCH v10 2/2] Subject: [PATCH v9 3/3] x86/snp: Convert shared memory back to private on kexec Ashish Kalra
2024-07-02 19:56 ` [PATCH v11 0/3] x86/snp: Add kexec support Ashish Kalra
2024-07-02 19:57   ` [PATCH v11 1/3] x86/boot: Skip video memory access in the decompressor for SEV-ES/SNP Ashish Kalra
2024-07-02 19:57   ` [PATCH v11 2/3] x86/mm: refactor __set_clr_pte_enc() Ashish Kalra
2024-07-05 14:26     ` Borislav Petkov
2024-07-02 19:58   ` [PATCH v11 3/3] x86/snp: Convert shared memory back to private on kexec Ashish Kalra
2024-07-05 14:28     ` Borislav Petkov
2024-07-05 14:29     ` Borislav Petkov
2024-07-10 20:12       ` Kalra, Ashish
2024-07-30 19:20 ` [PATCH v12 0/3] x86/snp: Add kexec support Ashish Kalra
2024-07-30 19:21   ` [PATCH v12 1/3] x86/boot: Skip video memory access in the decompressor for SEV-ES/SNP Ashish Kalra
2024-07-30 19:21   ` [PATCH v12 2/3] x86/mm: refactor __set_clr_pte_enc() Ashish Kalra
2024-07-30 19:22   ` [PATCH v12 3/3] x86/snp: Convert shared memory back to private on kexec Ashish Kalra
2024-08-01 19:14 ` [PATCH v13 0/3] x86/snp: Add kexec support Ashish Kalra
2024-08-01 19:14   ` [PATCH v13 1/3] x86/boot: Skip video memory access in the decompressor for SEV-ES/SNP Ashish Kalra
2024-08-01 19:14   ` [PATCH v13 2/3] x86/mm: refactor __set_clr_pte_enc() Ashish Kalra
2024-10-28 16:15     ` Tom Lendacky
2024-08-01 19:14   ` [PATCH v13 3/3] x86/snp: Convert shared memory back to private on kexec Ashish Kalra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=171871930069.10875.8795812694370524886.tip-bot2@tip-bot2 \
    --to=tip-bot2@linutronix.de \
    --cc=bp@alien8.de \
    --cc=kai.huang@intel.com \
    --cc=kirill.shutemov@linux.intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-tip-commits@vger.kernel.org \
    --cc=ltao@redhat.com \
    --cc=rick.p.edgecombe@intel.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®