[PATCH v6 00/18] x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

[Babu Moger <babu.moger@amd.com>]

Hi All,

This series adds support for AMD's Privilege-Level Zero Association
(PLZA) so kernel work can use a separate resource allocation and/or
monitoring association from the user task, and wires it up through a
small generic "kernel mode" (kmode) layer in fs/resctrl so future
architectures can plug in without touching (hopefully) core resctrl.

The features are documented in:

   AMD64 Zen6 Platform Quality of Service (PQOS) Extensions,
   Publication # 69193 Revision 1.00, Issue Date March 2026

available at https://bugzilla.kernel.org/show_bug.cgi?id=206537

The patches are based on top of commit (tip/master):

  8cf07e484290 Merge branch into tip/master: 'x86/tdx'

Any feedback is appreciated.

Background
==========

When memory bandwidth associated with a CLOSID is aggressively throttled,
and a task with that CLOSID moves into kernel mode, the kernel operations
are also aggressively throttled. This can stall forward progress and
eventually degrade overall system performance.

Privilege-Level Zero Association (PLZA) allows the user to specify a CLOSID
and/or RMID for execution at Privilege Level Zero. When PLZA is enabled on
a CPU, kernel work at PL0 uses the CLOSID and/or RMID from MSR
PQR_PLZA_ASSOC; otherwise, the CPU uses the CLOSID and RMID from PQR_ASSOC.

Design
======

A new sysfs file, info/kernel_mode, selects the kernel mode for resource
allocation and monitoring of kernel work and, when the mode needs one, the
resource group that backs it. Reads list the supported modes and the
currently active association; writes select the mode or the group. Look
at the thread below for design discussion.
https://lore.kernel.org/lkml/14a8ad0a-e842-4268-871a-0762f1169e03@intel.com/
https://lore.kernel.org/lkml/e4c95002-ae8a-48d0-bedd-772db58b937e@intel.com/

Two kernel modes are exposed:

- inherit_user: kernel work uses the allocation and monitoring
  associations of the user-space task on whose behalf it runs.

- global_enable_per_cpu: kernel work uses the allocation and/or
  monitoring association of a backing CTRL_MON or MON group. ctrl= and
  mon= select assign or inherit for each, and group= names the group
  with <CTRL_MON>/<MON>/ path syntax. An omitted parameter keeps the
  value the file currently shows. A monitor group requires mon=assign.
  A failed write leaves the current association in place.

Per-rdtgroup files kmode_cpus and kmode_cpus_list scope the association
to a subset of online CPUs. They are visible only on the group that
backs the active association. A write enables the association on CPUs
added to the mask and disables it on CPUs removed from the mask. A new
global_enable_per_cpu association sets the mask to every online CPU. A
CPU that comes online later joins the association. A CPU that goes
offline leaves it only when that CPU is in the mask.

A group that backs the association, or a control group whose monitor
group does, cannot change mode, including entering pseudo-lock setup.
The backing group cannot be renamed until the association is cleared.

The arch hook, resctrl_arch_configure_kmode_global(), keeps the
fs/resctrl layer arch-neutral.

resctrl_set_kmode_support() lets architecture code register each extra
mode during resctrl initialization. inherit_user is already registered
when the platform can allocate or monitor. On AMD, PLZA registers
global_enable_per_cpu when the feature is available.

Only AMD PLZA is wired up here; Intel and ARM can add their own support
later by implementing the hooks.

Examples
========

(See Documentation/filesystems/resctrl.rst, "kernel_mode", "kmode_cpus",
and "Examples on working with kernel_mode", for the full UAPI.)

  # Mount resctrl
  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

  # Read the supported modes.  The active mode is bracketed for display
  # only; do not include brackets when writing.
  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

  # Create a CTRL_MON group and back kernel-mode allocation with it.
  # mkdir ctrl1
  # echo "global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1//" \
          > info/kernel_mode
  # cat info/kernel_mode
  inherit_user
  [global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1//]

  # kmode_cpus and kmode_cpus_list are visible only on the backing group.
  # ls ctrl1/kmode_cpus*
  ctrl1/kmode_cpus  ctrl1/kmode_cpus_list

  # Restrict the association to a CPU subset. The association is enabled
  # on CPUs added to the mask and disabled on CPUs removed from it.
  # echo 0-3 > ctrl1/kmode_cpus_list
  # cat ctrl1/kmode_cpus
  f
  # cat ctrl1/kmode_cpus_list
  0-3

  # Return to inherit_user.
  # echo "inherit_user" > info/kernel_mode
  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

Tested on AMD with PLZA; builds on x86 without PLZA and does not expose
global_enable_per_cpu unless the feature is available.

Layout
======

  01-03  x86: PLZA CPU feature, command-line option, and MSR definitions.
  04-07  fs/resctrl: kernel mode enum, arch hook, kmode state, and
         resctrl_set_kmode_support().
  08     x86/resctrl: register global_enable_per_cpu when PLZA is
         available.
  09     fs/resctrl: info/kernel_mode read interface.
  10     fs/resctrl: hidden rdtgroup files.
  11     fs/resctrl: per-rdtgroup kmode_cpus/kmode_cpus_list.
  12     fs/resctrl: CPU hotplug for the active association.
  13     fs/resctrl: deactivate the association when a group is removed.
  14     fs/resctrl: show or hide kmode_cpus on the backing group.
  15     fs/resctrl: reject mode changes while a group backs the
         association.
  16     fs/resctrl: info/kernel_mode write interface.
  17     fs/resctrl: kmode_cpus/kmode_cpus_list writes.
  18     fs/resctrl: documentation and examples.


Changelog
=========
v6:
  - Addressed all the comments as discussed in v5. Anything missing
    is not intentional. Please feel to comment on any mistake.
  - Rename assign_global_enable_per_cpu to global_enable_per_cpu and
    describe the interface as a kernel mode rather than a policy.
  - Flatten kernel mode state into resctrl_kmode. The arch hook is
    resctrl_arch_configure_kmode_global().
  - An omitted ctrl=, mon=, or group= keeps the value shown by reading
    info/kernel_mode. inherit_user takes no parameters.
    ctrl=inherit;mon=inherit remains global_enable_per_cpu.
  - Selecting a new global_enable_per_cpu association sets that group's
    kmode_cpus to every online CPU. A CPU that comes online later joins
    the association even if user space had removed it from the mask.
  - kmode_cpus files are created hidden and shown only while the group
    backs the active association.
  - A monitor group requires mon=assign. Removing a control group
    detaches a child that backs the association before the parent's
    CLOSID is freed.
  - Reject a mode change when the group or one of its monitor groups
    backs the association.
  - Split kmode_cpus visibility and the mode-change rejection into
    their own patches.
  - Use the parsing similar to ctrlmondata.c:parse_line(),
    ctrlmondata.c:resctrl_io_alloc_parse_line(),
    monitor.c:resctrl_parse_mbm_assignment().
  - Re-wrote the changelog as a feature and not as a bug-fix.
  - Remove implementation details in function headers.

v5:
  - Collapse the two v4 global-assign modes into a single
    assign_global_enable_per_cpu policy with ctrl= and mon= options.
  - Use "association" terminology consistently in code, errors, and
    documentation.
  - Register assign_global_enable_per_cpu once during x86 resource
    discovery when PLZA is available.
  - Reject mon=inherit when group= selects a monitor group.
  - Split hidden-file support, deactivation-on-teardown, and PLZA mode
    registration into separate patches for easier review.
  - Fix msr_pqr_plza_assoc truncation on 32-bit systems by using u64.
  - Refresh documentation and add a readable end-to-end example walkthrough.
  - Block pseudo-lock setup on kernel-mode-associated groups.
  - Reject mode changes and rename while a group backs the active
    association.

v4:
  - Reorder and split the series into 15 patches: separate read-only
    info/kernel_mode display from the write path; add hotplug support
    when a CPU comes online; add an end-to-end documentation/examples
    patch.
  - Introduced resctrl_set_kmode_support() so architecture code can
    register supported kernel-mode policies during resctrl initialization.
  - info/kernel_mode write: validate group type; run fail paths before
    tearing down the active association so errors retain the old state.
  - kmode_cpus / kmode_cpus_list: writable with incremental enable/disable
    deltas; empty masks allowed; offline CPUs rejected.
  - Hotplug: newly online CPUs are added to the associated group's
    kmode_cpu_mask and programmed when assign_global_enable_per_cpu is
    active.

v3:
  - Generalise the layer beyond AMD: rename "PLZA mode" to "kernel mode"
    (kmode) in code, sysfs, and Documentation.
  - Reset the association when the associated rdtgroup is removed,
    instead of leaving stale state.

v2:
  - Similar to RFC with a new proposal; interface names were not final.
  - Separated Global Bandwidth Enforcement (GLBE) from PLZA; this series
    only adds PLZA support.
  - Used "kmode" instead of "PLZA" in the generic layer.

Previous versions:
v5: https://lore.kernel.org/lkml/cover.1787772750.git.babu.moger@amd.com/
v4: https://lore.kernel.org/lkml/cover.1783461016.git.babu.moger@amd.com/
v3: https://lore.kernel.org/lkml/cover.1777591496.git.babu.moger@amd.com/
v2: https://lore.kernel.org/lkml/cover.1773347820.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1769029977.git.babu.moger@amd.com/


Babu Moger (18):
  x86/cpufeatures: Support Privilege Level Zero Association (PLZA)
  x86/resctrl: Add PLZA support to command-line options
  x86/resctrl: Add PLZA configuration definitions and data structures
  fs/resctrl: Introduce kernel mode enum
  x86,arm,fs/resctrl: Introduce architecture hook to program global
    kernel mode
  fs/resctrl: Introduce kernel mode states for resctrl
  fs/resctrl: Introduce resctrl_set_kmode_support() to register
    supported modes
  x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is
    available
  fs/resctrl: Add interface to display kernel mode status
  fs/resctrl: Add support for hidden resource group files
  fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup
  fs/resctrl: Add CPU hotplug support for kernel mode associations
  fs/resctrl: Deactivate kernel mode associations when a group is
    removed
  fs/resctrl: Control visibility of global per-CPU kernel mode group
    files
  fs/resctrl: Reject mode changes for groups backing kernel mode
  fs/resctrl: Add interface to modify kernel mode via info/kernel_mode
  fs/resctrl: Allow user space to write kmode_cpus/kmode_cpus_list
  fs/resctrl: Add kernel mode documentation and examples

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 209 +++++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   6 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  38 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  49 ++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 drivers/resctrl/mpam_resctrl.c                |   7 +
 fs/resctrl/internal.h                         |  35 +
 fs/resctrl/rdtgroup.c                         | 713 ++++++++++++++++++
 include/linux/resctrl.h                       |  81 ++
 12 files changed, 1142 insertions(+), 1 deletion(-)

-- 
2.43.0