[PATCH RFC v1 0/8] x86/microcode: Enable uniform feature

["Chang S. Bae" <chang.seok.bae@intel.com>]

Hi all,

This series enables another Intel microcode loading feature.

While this initial posting is marked with RFC, it is also expected to
provide patches to the people who are interested in testing/using this
feature. Tested-by is anticipated from that end.

So, x86 maintainers, feel free to ignore this round. Having said that,
feedback will be appreciated and always welcomed.

== Introduction ==

Traditionally, a single trigger updates a core-scoped microcode, which
thus requires executing WRMSR0x79 on every core. This also means that
there is a possibility to load different microcode patches between cores.
Despite this fact, the kernel currently enforces loading the same image
across the CPUs.

This uniform microcode conceptually eliminates a chance of running a
different microcode within a scope of CPUs. From the loader perspective,
the uniform feature extends the loading scope to a larger number of cores
than one core.

A few points worth calling out about the feature:

  * The CPU enumerates the update scope such as package-wide or system-
    wide, depending on the implementation. The scope is advertised
    via MSR and isn't programmable.

  * The scope reduces the number of triggers, whereas staging primarily
    reduces the amount of work under the WRMSR window. Unlike staging,
    uniform loading is applicable to both early- and late-loading paths.

  * For early loading, only the parallel CPU bringup is relevant. In the
    legacy serial bringup, once the first CPU in a scope completes the
    update, subsequent CPUs will observe the updated revision so skip
    WRMSR0x79.

  * Staging introduced a new loading process. But uniform loading extends
    the semantics of the existing flow. Software that assumes the legacy
    scope remains supported. The next section discusses this
    compatibility aspect in more detail.

== Backward Compatibility ==

Older kernels assume a per-core scope, being ignorant of the uniform
loading scope. So, they trigger loading via WRMSR0x79 on every core. And
the spec [1] has the following statement, in Section 2.4 "Uniform
Microcode Update":

   NOTE [*]
   ... It is always allowed to load the update on more logical processors
   than necessary, which may result in unnecessary additional latency.

So this means legacy kernels remain functional on uniform systems. To
provide more context, folks involved in the implementation agreed to
share additional implementation details with the community. Their
write-up is attached at the end of this cover letter.

== Latency Trade-off ==

The above spec also states `additional latency` which in fact is a
side-effect from preserving the backward compatibility. Excessive update
triggers may elevate lock contention and waiting time inside the
microcode updater.

For example, consider a system with 100 cores per package where hardware
advertises package scope. Ideally, only one WRMSR is enough instead of
100. However, if all 100 CPUs attempt the update concurrently, the
update mechanism has to secure an atomic operation where only one CPU
participates in the actual update while the others wait.

This effect is measurable when comparing kernels with and without
uniform support. The exact numbers depend on the patch characteristic,
the lock contention, and the underlying synchronization implementation.
On multiple implementations, uniform loading showed loading-time
reductions especially during the late loading.

On the other hand, early loading showed only marginal gains from parallel
bringup. CPU0 already updates the scoped APs before they come online. In
addition, unlike stop_machine()-based late loading, parallel bringup is
not strictly simultaneous but instead iterates through CPUs for bringup.
Thus, the degree of contention appears much lower during early loading.

== Call for Reviews ==

Below are some aspects to collect feedback:

  * Early-loading support

    Supporting uniform scopes during early loading appears consistent
    with late loading support. But there is also a trade-off to consider:
    code complexity vs benefit. As mentioned above, the measurable
    latency impact looks marginal. And at the same time infrastructure
    change (patches 1/2) also appears moderate (except for the new
    cpumask -- see below).

    If the consensus ends up skipping uniform support for early loading,
    then at least the microcode implementation details need to be
    documented instead.

  * New cpumask: primary core CPUs

    Introducing a new topology cpumask was not the preferred option. But,
    during early loading CPU0 must determine which APs participate in the
    primary bringup phase before AP topology state is fully established.

    For both core- and system-scopes, the selection is straightforward.
    But package scope cannot reliably depend on package IDs at that stage
    because logical package IDs are established later during AP bringup.
    Patch3 has more detail.

  * Error handling on firmware misconfiguration

    If firmware is expected to configure the feature but leaves the
    system in an incomplete state, then that could be an indication of
    an unreliable situation. This version disables the loader entirely
    (patch8).

    Alternatively, instead of being paranoid, tainting could be an option
    too.

But any review beyond these points are definitely welcome too.

== Patchset and Validation ==

This series can be divided into two parts:

  * Part 1, patch 1-3: Preparatory infrastructure changes
  * Part 2, patch 4-8: Uniform-loading enablement

Testing was performed on one primary test machine. Since the feature is
architectural, the expected semantics should remain consistent across
implementations.

The patch set is available in this repository:
  git://github.com/intel-staging/microcode.git uniform_rfc-v1

Thanks,
Chang

== Reference ==

[1] Intel Runtime Microcode Update Technical Paper
    https://cdrdv2.intel.com/v1/dl/getContent/782715

[*] The NOTE paragraph primarily discusses certain unusual configurations
    where the uniform scope is not explicitly exported but effectively
    active. But the quoted compatibility statement should stand
    generally.

== Appendix: Microcode Implementation Note ==

The uniform update protocol is an optimization for boot/runtime microcode
update. It is backward compatible with existing microarchitecture of
core/thread scope update and any OS MCU drivers that rely on legacy
method of update.

With uniform update, if multiple logical processors attempt to load an
update simultaneously, there is a race to an internal semaphore within
the microcode. The winner of the race assumes control of the update
process and sends an internal interrupt to all other threads (if only one
thread initiates the update, it is the winner by default).

All other logical processors receive the internal interrupt at an
architectural instruction boundary and proceed to load the update under
the coordination of the winner. This ensures that the responding threads
load the update in a controlled manner while at a well-defined
architectural instruction boundary. If a higher priority interrupt or a
fault happens, all logical processors will see it either before the
microcode patch has been applied or after. In either case, all logical
processors will see the same microcode revision and nothing intermediate.

Chang S. Bae (8):
  cpu/hotplug: Allow architecture-specific primary CPU bringup
  x86/hotplug: Implement SMT-primary selection for parallel bringup
  x86/cpu/topology: Introduce primary core mask
  x86/microcode: Extend struct microcode_ops for uniform loading
  x86/microcode: Clarify online enforcement with uniform loading
  x86/microcode: Support uniform scope for late loading
  x86/microcode/intel: Support uniform scope for early loading
  x86/microcode/intel: Enable uniform loading

 arch/Kconfig                             |   4 +
 arch/x86/Kconfig                         |   1 +
 arch/x86/include/asm/microcode.h         |   2 +
 arch/x86/include/asm/msr-index.h         |  10 ++
 arch/x86/include/asm/topology.h          |   3 +
 arch/x86/kernel/cpu/microcode/core.c     |  76 ++++++++++-
 arch/x86/kernel/cpu/microcode/intel.c    | 153 +++++++++++++++++++++--
 arch/x86/kernel/cpu/microcode/internal.h |  15 ++-
 arch/x86/kernel/cpu/topology.c           |  23 +++-
 arch/x86/kernel/cpu/topology_common.c    |   9 ++
 include/linux/cpu.h                      |   9 ++
 kernel/cpu.c                             |   8 +-
 12 files changed, 291 insertions(+), 22 deletions(-)


base-commit: df2908090cda368b01ff43709f51890076c56157
-- 
2.53.0