mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [patch V4 00/30] x86/microcode: Cleanup and late loading enhancements
@ 2023-10-02 11:59 Thomas Gleixner
  2023-10-02 11:59 ` [patch V4 01/30] x86/microcode/32: Move early loading after paging enable Thomas Gleixner
                   ` (30 more replies)
  0 siblings, 31 replies; 104+ messages in thread
From: Thomas Gleixner @ 2023-10-02 11:59 UTC (permalink / raw)
  To: LKML
  Cc: x86, Borislav Petkov, Chang S. Bae, Arjan van de Ven, Nikolay Borisov

This is a follow up on:

  https://lore.kernel.org/lkml/20230912065249.695681286@linutronix.de

Late microcode loading is desired by enterprise users. Late loading is
problematic as it requires detailed knowledge about the change and an
analysis whether this change modifies something which is already in use by
the kernel. Large enterprise customers have engineering teams and access to
deep technical vendor support. The regular admin does not have such
resources, so the kernel has always tainted the kernel after late loading.

Intel recently added a new previously reserved field to the microcode
header which contains the minimal microcode revision which must be running
on the CPU to make the load safe. This field is 0 in all older microcode
revisions, which the kernel assumes to be unsafe. Minimal revision checking
can be enforced via Kconfig or kernel command line. It then refuses to load
an unsafe revision. The default loads unsafe revisions like before and
taints the kernel. If a safe revision is loaded the kernel is not tainted.

But that does not solve all other known problems with late loading:

    - Late loading on current Intel CPUs is unsafe vs. NMI when
      hyperthreading is enabled. If a NMI hits the secondary sibling while
      the primary loads the microcode, the machine can crash.

    - Soft offline SMT siblings which are playing dead with MWAIT can cause
      damage too when the microcode update modifies MWAIT. That's a
      realistic scenario in the context of 'nosmt' mitigations. :(

Neither the core code nor the Intel specific code handles any of this at all.

While trying to implement this, I stumbled over disfunctional, horribly
complex and redundant code, which I decided to clean up first so the new
functionality can be added on a clean slate.

So the series has several sections:

   1) Move the 32bit early loading after paging enable

   2) Cleanup of the Intel specific code

   3) Implementation of proper core control logic to handle the NMI safe
      requirements

   4) Support for minimal revision check in the core and the Intel specific
      parts.

Changes vs. V3:

  - Rebased on v6.6-rc1

  - Remove the early load magic which was required for physical address
    mode from the AMD code.

  - Address the review comments from Borislav, which is mostly naming,
    comments and change logs. No functional changes vs. v3

The series is also available from git:

   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git ucode-v4

Thanks,

	tglx
---
 Documentation/admin-guide/kernel-parameters.txt |    5 
 arch/x86/Kconfig                                |   25 
 arch/x86/include/asm/apic.h                     |    5 
 arch/x86/include/asm/cpu.h                      |   20 
 arch/x86/include/asm/microcode.h                |   19 
 arch/x86/kernel/Makefile                        |    1 
 arch/x86/kernel/apic/apic_flat_64.c             |    2 
 arch/x86/kernel/apic/ipi.c                      |    8 
 arch/x86/kernel/apic/x2apic_cluster.c           |    1 
 arch/x86/kernel/apic/x2apic_phys.c              |    1 
 arch/x86/kernel/cpu/common.c                    |   12 
 arch/x86/kernel/cpu/microcode/amd.c             |  129 +---
 arch/x86/kernel/cpu/microcode/core.c            |  637 ++++++++++++++--------
 arch/x86/kernel/cpu/microcode/intel.c           |  682 +++++++-----------------
 arch/x86/kernel/cpu/microcode/internal.h        |   32 -
 arch/x86/kernel/head32.c                        |    6 
 arch/x86/kernel/head_32.S                       |   10 
 arch/x86/kernel/nmi.c                           |    9 
 arch/x86/kernel/smpboot.c                       |   12 
 drivers/platform/x86/intel/ifs/load.c           |    8 
 include/linux/cpuhotplug.h                      |    1 
 21 files changed, 788 insertions(+), 837 deletions(-)

^ permalink raw reply	[flat|nested] 104+ messages in thread
* [patch V5 30/39] x86/microcode: Add per CPU result state
@ 2023-10-17 21:24 Thomas Gleixner
  2023-10-20 11:37 ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
  2023-10-24 13:20 ` tip-bot2 for Thomas Gleixner
  0 siblings, 2 replies; 104+ messages in thread
From: Thomas Gleixner @ 2023-10-17 21:24 UTC (permalink / raw)
  To: LKML; +Cc: x86, Borislav Petkov

From: Thomas Gleixner <tglx@linutronix.de>

The microcode rendezvous is purely acting on global state, which does
not allow to analyze fails in a coherent way.

Introduce per CPU state where the results are written into, which allows to
analyze the return codes of the individual CPUs.

Initialize the state when walking the cpu_present_mask in the online check
to avoid another for_each_cpu() loop.

Enhance the result print out with that.

The structure is intentionally named ucode_ctrl as it will gain control
fields in subsequent changes.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/microcode/core.c     |  112 ++++++++++++++++++-------------
 arch/x86/kernel/cpu/microcode/internal.h |    1 
 2 files changed, 67 insertions(+), 46 deletions(-)
---
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -252,6 +252,11 @@ static struct platform_device	*microcode
  *   requirement can be relaxed in the future. Right now, this is conservative
  *   and good.
  */
+struct microcode_ctrl {
+	enum ucode_state	result;
+};
+
+static DEFINE_PER_CPU(struct microcode_ctrl, ucode_ctrl);
 static atomic_t late_cpus_in, late_cpus_out;
 
 static bool wait_for_cpus(atomic_t *cnt)
@@ -274,23 +279,19 @@ static bool wait_for_cpus(atomic_t *cnt)
 	return false;
 }
 
-/*
- * Returns:
- * < 0 - on error
- *   0 - success (no update done or microcode was updated)
- */
-static int __reload_late(void *info)
+static int load_cpus_stopped(void *unused)
 {
 	int cpu = smp_processor_id();
-	enum ucode_state err;
-	int ret = 0;
+	enum ucode_state ret;
 
 	/*
 	 * Wait for all CPUs to arrive. A load will not be attempted unless all
 	 * CPUs show up.
 	 * */
-	if (!wait_for_cpus(&late_cpus_in))
-		return -1;
+	if (!wait_for_cpus(&late_cpus_in)) {
+		this_cpu_write(ucode_ctrl.result, UCODE_TIMEOUT);
+		return 0;
+	}
 
 	/*
 	 * On an SMT system, it suffices to load the microcode on one sibling of
@@ -299,17 +300,11 @@ static int __reload_late(void *info)
 	 * loading attempts happen on multiple threads of an SMT core. See
 	 * below.
 	 */
-	if (cpumask_first(topology_sibling_cpumask(cpu)) == cpu)
-		err = microcode_ops->apply_microcode(cpu);
-	else
+	if (cpumask_first(topology_sibling_cpumask(cpu)) != cpu)
 		goto wait_for_siblings;
 
-	if (err >= UCODE_NFOUND) {
-		if (err == UCODE_ERROR) {
-			pr_warn("Error reloading microcode on CPU %d\n", cpu);
-			ret = -1;
-		}
-	}
+	ret = microcode_ops->apply_microcode(cpu);
+	this_cpu_write(ucode_ctrl.result, ret);
 
 wait_for_siblings:
 	if (!wait_for_cpus(&late_cpus_out))
@@ -321,19 +316,18 @@ static int __reload_late(void *info)
 	 * per-cpu cpuinfo can be updated with right microcode
 	 * revision.
 	 */
-	if (cpumask_first(topology_sibling_cpumask(cpu)) != cpu)
-		err = microcode_ops->apply_microcode(cpu);
+	if (cpumask_first(topology_sibling_cpumask(cpu)) == cpu)
+		return 0;
 
-	return ret;
+	ret = microcode_ops->apply_microcode(cpu);
+	this_cpu_write(ucode_ctrl.result, ret);
+	return 0;
 }
 
-/*
- * Reload microcode late on all CPUs. Wait for a sec until they
- * all gather together.
- */
-static int microcode_reload_late(void)
+static int load_late_stop_cpus(void)
 {
-	int old = boot_cpu_data.microcode, ret;
+	unsigned int cpu, updated = 0, failed = 0, timedout = 0, siblings = 0;
+	int old_rev = boot_cpu_data.microcode;
 	struct cpuinfo_x86 prev_info;
 
 	pr_err("Attempting late microcode loading - it is dangerous and taints the kernel.\n");
@@ -348,26 +342,47 @@ static int microcode_reload_late(void)
 	 */
 	store_cpu_caps(&prev_info);
 
-	ret = stop_machine_cpuslocked(__reload_late, NULL, cpu_online_mask);
+	stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask);
+
+	/* Analyze the results */
+	for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) {
+		switch (per_cpu(ucode_ctrl.result, cpu)) {
+		case UCODE_UPDATED:	updated++; break;
+		case UCODE_TIMEOUT:	timedout++; break;
+		case UCODE_OK:		siblings++; break;
+		default:		failed++; break;
+		}
+	}
 
 	if (microcode_ops->finalize_late_load)
-		microcode_ops->finalize_late_load(ret);
+		microcode_ops->finalize_late_load(!updated);
 
-	if (!ret) {
-		pr_info("Reload succeeded, microcode revision: 0x%x -> 0x%x\n",
-			old, boot_cpu_data.microcode);
-		microcode_check(&prev_info);
-		add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
-	} else {
-		pr_info("Reload failed, current microcode revision: 0x%x\n",
-			boot_cpu_data.microcode);
+	if (!updated) {
+		/* Nothing changed. */
+		if (!failed && !timedout)
+			return 0;
+		pr_err("update failed: %u CPUs failed %u CPUs timed out\n",
+		       failed, timedout);
+		return -EIO;
 	}
-	return ret;
+
+	add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
+	pr_info("load: updated on %u primary CPUs with %u siblings\n", updated, siblings);
+	if (failed || timedout) {
+		pr_err("load incomplete. %u CPUs timed out or failed\n",
+		       num_online_cpus() - (updated + siblings));
+	}
+	pr_info("revision: 0x%x -> 0x%x\n", old_rev, boot_cpu_data.microcode);
+	microcode_check(&prev_info);
+
+	return updated + siblings == num_online_cpus() ? 0 : -EIO;
 }
 
 /*
- *  Ensure that all required CPUs which are present and have been booted
- *  once are online.
+ * This function does two things:
+ *
+ * 1) Ensure that all required CPUs which are present and have been booted
+ *    once are online.
  *
  *    To pass this check, all primary threads must be online.
  *
@@ -378,9 +393,12 @@ static int microcode_reload_late(void)
  *    behaviour is undefined. The default play_dead() implementation on
  *    modern CPUs uses MWAIT, which is also not guaranteed to be safe
  *    against a microcode update which affects MWAIT.
+ *
+ * 2) Initialize the per CPU control structure
  */
-static bool ensure_cpus_are_online(void)
+static bool setup_cpus(void)
 {
+	struct microcode_ctrl ctrl = { .result = -1, };
 	unsigned int cpu;
 
 	for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) {
@@ -390,18 +408,20 @@ static bool ensure_cpus_are_online(void)
 				return false;
 			}
 		}
+		/* Initialize the per CPU state */
+		per_cpu(ucode_ctrl, cpu) = ctrl;
 	}
 	return true;
 }
 
-static int ucode_load_late_locked(void)
+static int load_late_locked(void)
 {
-	if (!ensure_cpus_are_online())
+	if (!setup_cpus())
 		return -EBUSY;
 
 	switch (microcode_ops->request_microcode_fw(0, &microcode_pdev->dev)) {
 	case UCODE_NEW:
-		return microcode_reload_late();
+		return load_late_stop_cpus();
 	case UCODE_NFOUND:
 		return -ENOENT;
 	default:
@@ -421,7 +441,7 @@ static ssize_t reload_store(struct devic
 		return -EINVAL;
 
 	cpus_read_lock();
-	ret = ucode_load_late_locked();
+	ret = load_late_locked();
 	cpus_read_unlock();
 
 	return ret ? : size;
--- a/arch/x86/kernel/cpu/microcode/internal.h
+++ b/arch/x86/kernel/cpu/microcode/internal.h
@@ -16,6 +16,7 @@ enum ucode_state {
 	UCODE_UPDATED,
 	UCODE_NFOUND,
 	UCODE_ERROR,
+	UCODE_TIMEOUT,
 };
 
 struct microcode_ops {


^ permalink raw reply	[flat|nested] 104+ messages in thread
end of thread, other threads:[~2023-10-24 13:31 UTC | newest]

Thread overview: 104+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-10-02 11:59 [patch V4 00/30] x86/microcode: Cleanup and late loading enhancements Thomas Gleixner
2023-10-02 11:59 ` [patch V4 01/30] x86/microcode/32: Move early loading after paging enable Thomas Gleixner
2023-10-02 11:59 ` [patch V4 02/30] x86/boot/32: Disable stackprotector and tracing for mk_early_pgtbl_32() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:38   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 03/30] x86/microcode/intel: Rip out mixed stepping support for Intel CPUs Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Ashok Raj
2023-10-02 11:59 ` [patch V4 04/30] x86/microcode/intel: Simplify scan_microcode() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:38   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 05/30] x86/microcode/intel: Simplify and rename generic_load_microcode() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:38   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 06/30] x86/microcode/intel: Cleanup code further Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:38   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 07/30] x86/microcode/intel: Simplify early loading Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 08/30] x86/microcode/intel: Save the microcode only after a successful late-load Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:21   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 09/30] x86/microcode/intel: Switch to kvmalloc() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:21   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 10/30] x86/microcode/intel: Unify microcode apply() functions Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 11/30] x86/microcode/intel: Rework intel_cpu_collect_info() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 12/30] x86/microcode/intel: Reuse intel_cpu_collect_info() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:21   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 13/30] x86/microcode/intel: Rework intel_find_matching_signature() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:21   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 14/30] x86/microcode/amd: Read revision from hardware in collect_cpu_info_amd() Thomas Gleixner
2023-10-04  8:32   ` Borislav Petkov
2023-10-02 11:59 ` [patch V4 15/30] x86/microcode: Remove pointless apply() invocation Thomas Gleixner
2023-10-06 13:26   ` Borislav Petkov
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 16/30] x86/microcode: Get rid of the schedule work indirection Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 17/30] x86/microcode: Clean up mc_cpu_down_prep() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 18/30] x86/microcode: Handle "nosmt" correctly Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 19/30] x86/microcode: Clarify the late load logic Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 11:59 ` [patch V4 20/30] x86/microcode: Sanitize __wait_for_cpus() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 21/30] x86/microcode: Add per CPU result state Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 22/30] x86/microcode: Add per CPU control field Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 23/30] x86/microcode: Provide new control functions Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 24/30] x86/microcode: Replace the all in one rendevouz handler Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] x86/microcode: Replace the all-in-one rendevous handler tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 25/30] x86/microcode: Rendezvous and load in NMI Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 26/30] x86/microcode: Protect against instrumentation Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 27/30] x86/apic: Provide apic_force_nmi_on_cpu() Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 28/30] x86/microcode: Handle "offline" CPUs correctly Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-20 11:37   ` tip-bot2 for Thomas Gleixner
2023-10-24 13:20   ` tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 29/30] x86/microcode: Prepare for minimal revision check Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-02 12:00 ` [patch V4 30/30] x86/microcode/intel: Add a minimum required revision for late-loads Thomas Gleixner
2023-10-09 12:29   ` [tip: x86/microcode] x86/microcode/intel: Add a minimum required revision for late loading tip-bot2 for Ashok Raj
2023-10-20 11:37   ` tip-bot2 for Ashok Raj
2023-10-24 13:20   ` tip-bot2 for Ashok Raj
2023-10-08  8:54 ` [patch V4 00/30] x86/microcode: Cleanup and late loading enhancements Qiuxu Zhuo
2023-10-08 13:08   ` Borislav Petkov
2023-10-09  5:03     ` Zhuo, Qiuxu
2023-10-10  8:00     ` Zhuo, Qiuxu
2023-10-10  8:11       ` Borislav Petkov
2023-10-17 21:24 [patch V5 30/39] x86/microcode: Add per CPU result state Thomas Gleixner
2023-10-20 11:37 ` [tip: x86/microcode] " tip-bot2 for Thomas Gleixner
2023-10-24 13:20 ` tip-bot2 for Thomas Gleixner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®