Re: [PATCH 1/1] x86/microcode: Check for offline CPUs before checking for microcode update

Re: [PATCH 1/1] x86/microcode: Check for offline CPUs before checking for microcode update

[Pontes, Otavio]

Sorry, I forgot to copy LKML

________________________________________
From: Pontes, Otavio <otavio.pontes@intel.com>
Sent: Friday, March 19, 2021 9:55 AM
To: x86@kernel.org
Cc: Borislav Petkov; Thomas Gleixner; Pontes, Otavio; Raj, Ashok; Luck, Tony
Subject: [PATCH 1/1] x86/microcode: Check for offline CPUs before checking for microcode update

Checking for any offline CPUs to abort microcode update must be done before
kernel caching a new microcode from the filesystem. Otherwise when offlined
CPUs are onlined later, those cores are going to be updated with the new
microcode while CPUs previously onine will continue to run with the older
microcode. If update is aborted before checking for the microcode file,
offline CPUs that are onlined later will get the same microcode as online
CPUs.

Turn off one core (2 threads)
$ echo 0 > /sys/devices/system/cpu/cpu3/online
$ echo 0 > /sys/devices/system/cpu/cpu1/online

Install the ucode fails because there's one core off
$ cp intel-ucode/06-8e-09 /lib/firmware/intel-ucode/
$ echo 1 > /sys/devices/system/cpu/microcode/reload
bash: echo: write error: Invalid argument

Turn the core back on
$ echo 1 > /sys/devices/system/cpu/cpu3/online
$ echo 1 > /sys/devices/system/cpu/cpu1/online
$ cat /proc/cpuinfo |grep microcode
microcode : 0x30
microcode : 0xde
microcode : 0x30
microcode : 0xde

Fixes: 30ec26da9967 ("x86/microcode: Do not upload microcode if CPUs are offline")
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Acked-by: Ashok Raj <ashok.raj@intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
---
 arch/x86/kernel/cpu/microcode/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index b935e1b5f115..6a6318e9590c 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -629,16 +629,16 @@ static ssize_t reload_store(struct device *dev,
        if (val != 1)
                return size;

-       tmp_ret = microcode_ops->request_microcode_fw(bsp, &microcode_pdev->dev, true);
-       if (tmp_ret != UCODE_NEW)
-               return size;
-
        get_online_cpus();

        ret = check_online_cpus();
        if (ret)
                goto put;

+       tmp_ret = microcode_ops->request_microcode_fw(bsp, &microcode_pdev->dev, true);
+       if (tmp_ret != UCODE_NEW)
+               goto put;
+
        mutex_lock(&microcode_mutex);
        ret = microcode_reload_late();
        mutex_unlock(&microcode_mutex);
--
2.30.2

Re: [PATCH 1/1] x86/microcode: Check for offline CPUs before checking for microcode update

[Borislav Petkov]

On Fri, Mar 19, 2021 at 06:48:14PM +0000, Pontes, Otavio wrote:
> Turn off one core (2 threads)
> $ echo 0 > /sys/devices/system/cpu/cpu3/online
> $ echo 0 > /sys/devices/system/cpu/cpu1/online
> 
> Install the ucode fails because there's one core off
> $ cp intel-ucode/06-8e-09 /lib/firmware/intel-ucode/
> $ echo 1 > /sys/devices/system/cpu/microcode/reload
> bash: echo: write error: Invalid argument
> 
> Turn the core back on
> $ echo 1 > /sys/devices/system/cpu/cpu3/online
> $ echo 1 > /sys/devices/system/cpu/cpu1/online
> $ cat /proc/cpuinfo |grep microcode
> microcode : 0x30
> microcode : 0xde
> microcode : 0x30
> microcode : 0xde

Yeah, I'm looking at that check_online_cpus() thing and wondering why we
even need that:

0. So you have CPUs 1 and 3 offline.
1. We can update on the subset of cores which are online
2. If a core is offline and comes online, we have the hotplug notifier:

        cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "x86/microcode:online",
                                  mc_cpu_online, mc_cpu_down_prep);

which takes care of updating the microcode when that CPU comes online.

So unless your microcode folks don't come back with a real requirement
why all CPUs must absolutely be online for a late update, then the
proper fix is to get rid of check_online_cpus() altogether and update
what's online and the rest will get updated when they come online.

I know Ashok did:

commit 30ec26da9967d0d785abc24073129a34c3211777
Author: Ashok Raj <ashok.raj@intel.com>
Date:   Wed Feb 28 11:28:43 2018 +0100

    x86/microcode: Do not upload microcode if CPUs are offline

    Avoid loading microcode if any of the CPUs are offline, and issue a
    warning. Having different microcode revisions on the system at any time
    is outright dangerous.

but those cores are offlined so they're executing some idle routine...

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/1] x86/microcode: Check for offline CPUs before checking for microcode update

[Raj, Ashok]

On Sat, Mar 20, 2021 at 03:55:46PM +0100, Borislav Petkov wrote:
[snip]
> > microcode : 0x30
> > microcode : 0xde
> > microcode : 0x30
> > microcode : 0xde
> 
> Yeah, I'm looking at that check_online_cpus() thing and wondering why we
> even need that:
> 
> 0. So you have CPUs 1 and 3 offline.
> 1. We can update on the subset of cores which are online
> 2. If a core is offline and comes online, we have the hotplug notifier:
> 
>         cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "x86/microcode:online",
>                                   mc_cpu_online, mc_cpu_down_prep);
> 
> which takes care of updating the microcode when that CPU comes online.
> 
> So unless your microcode folks don't come back with a real requirement
> why all CPUs must absolutely be online for a late update, then the
> proper fix is to get rid of check_online_cpus() altogether and update
> what's online and the rest will get updated when they come online.

Its true we update them during the online flow, but the core is still
behind compared to other cores. It still participates when it enters SMM,
or when running MCE for instance. Unless its in WAIT_FOR_SIPI state its
best to not leave a core behind when updating microcode.

-- 
Cheers,
Ashok

[Forgiveness is the attribute of the STRONG - Gandhi]

[tip: x86/microcode] x86/microcode: Check for offline CPUs before requesting new microcode

[tip-bot2 for Otavio Pontes]

The following commit has been merged into the x86/microcode branch of tip:

Commit-ID:     7189b3c11903667808029ec9766a6e96de5012a5
Gitweb:        https://git.kernel.org/tip/7189b3c11903667808029ec9766a6e96de5012a5
Author:        Otavio Pontes <otavio.pontes@intel.com>
AuthorDate:    Fri, 19 Mar 2021 09:55:15 -07:00
Committer:     Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 22 Mar 2021 22:29:40 +01:00

x86/microcode: Check for offline CPUs before requesting new microcode

Currently, the late microcode loading mechanism checks whether any CPUs
are offlined, and, in such a case, aborts the load attempt.

However, this must be done before the kernel caches new microcode from
the filesystem. Otherwise, when offlined CPUs are onlined later, those
cores are going to be updated through the CPU hotplug notifier callback
with the new microcode, while CPUs previously onine will continue to run
with the older microcode.

For example:

Turn off one core (2 threads):

  echo 0 > /sys/devices/system/cpu/cpu3/online
  echo 0 > /sys/devices/system/cpu/cpu1/online

Install the ucode fails because a primary SMT thread is offline:

  cp intel-ucode/06-8e-09 /lib/firmware/intel-ucode/
  echo 1 > /sys/devices/system/cpu/microcode/reload
  bash: echo: write error: Invalid argument

Turn the core back on

  echo 1 > /sys/devices/system/cpu/cpu3/online
  echo 1 > /sys/devices/system/cpu/cpu1/online
  cat /proc/cpuinfo |grep microcode
  microcode : 0x30
  microcode : 0xde
  microcode : 0x30
  microcode : 0xde

The rationale for why the update is aborted when at least one primary
thread is offline is because even if that thread is soft-offlined
and idle, it will still have to participate in broadcasted MCE's
synchronization dance or enter SMM, and in both examples it will execute
instructions so it better have the same microcode revision as the other
cores.

 [ bp: Heavily edit and extend commit message with the reasoning behind all
   this. ]

Fixes: 30ec26da9967 ("x86/microcode: Do not upload microcode if CPUs are offline")
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Ashok Raj <ashok.raj@intel.com>
Link: https://lkml.kernel.org/r/20210319165515.9240-2-otavio.pontes@intel.com
---
 arch/x86/kernel/cpu/microcode/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index b935e1b..6a6318e 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -629,16 +629,16 @@ static ssize_t reload_store(struct device *dev,
 	if (val != 1)
 		return size;
 
-	tmp_ret = microcode_ops->request_microcode_fw(bsp, &microcode_pdev->dev, true);
-	if (tmp_ret != UCODE_NEW)
-		return size;
-
 	get_online_cpus();
 
 	ret = check_online_cpus();
 	if (ret)
 		goto put;
 
+	tmp_ret = microcode_ops->request_microcode_fw(bsp, &microcode_pdev->dev, true);
+	if (tmp_ret != UCODE_NEW)
+		goto put;
+
 	mutex_lock(&microcode_mutex);
 	ret = microcode_reload_late();
 	mutex_unlock(&microcode_mutex);