[PATCH v7 00/19] x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

[PATCH v7 00/19] x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

[Babu Moger]

Hi All,

This series adds support for AMD's Privilege-Level Zero Association (PLZA),
which allows the CPU to use a different CLOSID and/or RMID association
while executing in kernel mode than it does in user mode.

The implementation is integrated through a generic kernel mode ("kmode")
framework in fs/resctrl, enabling future architectures to support similar
functionality without requiring changes to the core resctrl code. The
intent is to provide a common infrastructure that architecture-specific
implementations can hook into, minimizing future resctrl core
modifications.

The features are documented in:

   AMD64 Zen6 Platform Quality of Service (PQOS) Extensions,
   Publication # 69193 Revision 1.00, Issue Date March 2026

available at https://bugzilla.kernel.org/show_bug.cgi?id=206537

The patches are based on top of commit (tip/master):

  c6279551a44c Merge branch into tip/master: 'x86/sev'

Any feedback is appreciated.

Background
==========

When memory bandwidth associated with a CLOSID is aggressively throttled,
and a task with that CLOSID moves into kernel mode, the kernel operations
are also aggressively throttled. This can stall forward progress and
eventually degrade overall system performance.

Privilege-Level Zero Association (PLZA) allows the user to specify a CLOSID
and/or RMID for execution at Privilege Level Zero. When PLZA is enabled on
a CPU, kernel work at PL0 uses the CLOSID and/or RMID from MSR
PQR_PLZA_ASSOC; otherwise, the CPU uses the CLOSID and RMID from PQR_ASSOC.

Design
======

A new sysfs file, info/kernel_mode, selects the kernel mode for resource
allocation and monitoring of kernel work and, when the mode needs one, the
resource group that backs it. Reads list the supported modes, with the
active mode in square brackets. Writes use the same line without the
brackets. Look at the thread below for design discussion.
https://lore.kernel.org/lkml/14a8ad0a-e842-4268-871a-0762f1169e03@intel.com/
https://lore.kernel.org/lkml/e4c95002-ae8a-48d0-bedd-772db58b937e@intel.com/

Two kernel modes are exposed:

- inherit_user: kernel work uses the allocation and monitoring
  associations of the user task on whose behalf it runs. It takes no
  parameters.

- global_enable_per_cpu: kernel work uses the allocation and/or
  monitoring association of a backing CTRL_MON or MON group. ctrl= and
  mon= select assign or inherit for each. group= names the group and
  must be the last parameter, because a group name may contain ';'.

  When mon=assign, a CTRL_MON group is shown as <CTRL_MON>// and a MON
  group as <CTRL_MON>/<MON>/. When mon=inherit, or monitoring is
  unsupported, a CTRL_MON group is shown as <CTRL_MON>/.

  An omitted ctrl=, mon=, or group= selects the default: assign where
  the platform supports it, and the default CTRL_MON group. A partial
  write therefore restores the omitted fields and programs every online
  CPU. ctrl=inherit together with mon=inherit is rejected. A monitor
  group requires mon=assign. A failed write leaves the current
  association in place.

Per-rdtgroup files kmode_cpus and kmode_cpus_list scope the association
to a subset of online CPUs. They are visible only on the group that
backs the active association. A write enables the association on CPUs
added to the mask and disables it on CPUs removed from the mask. An
empty mask disables the association on every currently online CPU. A
new global_enable_per_cpu association sets the mask to every online
CPU. A CPU that comes online later joins the association, including
after a narrowed or empty mask. A CPU that goes offline leaves it only
when that CPU is in the mask.

A group that backs the association, or a control group whose monitor
group does, cannot change mode, including entering pseudo-lock setup.
Reparenting a backing monitor group is rejected. The backing group
cannot be renamed until the association is cleared.

The arch hook, resctrl_arch_configure_kmode_global(), keeps the
fs/resctrl layer arch-neutral. enable=false disables the mode on the
given CPUs and still writes the same CLOSID, RMID, and per-ID enables,
which is what the PLZA domain rule requires. It does not by itself
switch the active mode back to inherit_user.

resctrl_set_kmode_support() lets architecture code register each extra
mode during resctrl initialization. inherit_user is registered by
default. On AMD, PLZA registers global_enable_per_cpu when the feature
is available and the platform can allocate or monitor.

Only AMD PLZA is wired up here; Intel and ARM can add their own support
later by implementing the hooks.

Examples
========

(See Documentation/filesystems/resctrl.rst, "kernel_mode", "kmode_cpus",
and "Examples on working with kernel_mode", for the full UAPI.)

  # Mount resctrl
  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

  # Read the supported modes.  The active mode is bracketed for display
  # only; do not include brackets when writing.
  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

  # Create a CTRL_MON group and back kernel-mode allocation with it.
  # mon=inherit, so the group is shown as <CTRL_MON>/.
  # mkdir ctrl1
  # echo "global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/" \
          > info/kernel_mode
  # cat info/kernel_mode
  inherit_user
  [global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/]

  # kmode_cpus and kmode_cpus_list are visible only on the backing group.
  # ls ctrl1/kmode_cpus*
  ctrl1/kmode_cpus  ctrl1/kmode_cpus_list

  # Restrict the association to a CPU subset. The association is enabled
  # on CPUs added to the mask and disabled on CPUs removed from it.
  # echo 0-3 > ctrl1/kmode_cpus_list
  # cat ctrl1/kmode_cpus
  f
  # cat ctrl1/kmode_cpus_list
  0-3

  # Return to inherit_user. The inactive global_enable_per_cpu line
  # shows the defaults again.
  # echo "inherit_user" > info/kernel_mode
  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

Tested on AMD with PLZA; builds on x86 without PLZA and does not expose
global_enable_per_cpu unless the feature is available.

Layout
======

  01-03  x86: PLZA CPU feature, command-line option, and MSR definitions.
  04-07  fs/resctrl: kernel mode enum, arch hook, kmode state, and
         resctrl_set_kmode_support().
  08     x86/resctrl: register global_enable_per_cpu when PLZA is
         available.
  09     fs/resctrl: document last_cmd_status under a top-level info
         section.
  10     fs/resctrl: info/kernel_mode read interface.
  11     fs/resctrl: hidden rdtgroup files.
  12     fs/resctrl: per-rdtgroup kmode_cpus/kmode_cpus_list.
  13     x86,fs/resctrl: CPU hotplug for the active association, and
         clear PLZA across kexec.
  14     fs/resctrl: deactivate the association when a group is removed.
  15     fs/resctrl: info/kernel_mode write interface.
  16     fs/resctrl: reject mode changes and reparenting while a group
         backs the association.
  17     fs/resctrl: show or hide kmode_cpus on the backing group.
  18     fs/resctrl: kmode_cpus/kmode_cpus_list writes.
  19     fs/resctrl: documentation and examples.


Changelog
=========
v7:
  - Used "CLOSID and/or RMID association" throughout the series because
    feature operates on the MSR_IA32_PQR_PLZA_ASSOC association register.
    The association concept is already known in the resctrl subsystem.
    Please let me know if you see a better alternative.
  - Rebase onto tip/master c6279551a44c ("Merge branch into tip/master:
    'x86/sev'").
  - Document last_cmd_status in a new "Top level files" section of the
    info directory, before kernel_mode.
  - info/kernel_mode includes the monitor component of group= only when
    mon=assign. A CTRL_MON group then reads as <CTRL_MON>//. When
    mon=inherit, or monitoring is unsupported, it reads as <CTRL_MON>/.
  - An omitted ctrl=, mon=, or group= selects the defaults (assign
    where the platform supports it, and the default CTRL_MON group).
    A partial write restores the omitted fields and programs every
    online CPU.
  - Reject ctrl=inherit together with mon=inherit. With both enables
    clear, CPL0 keeps the user task's IDs, which is inherit_user.
  - group= is the last parameter, so a group name may contain ';'.
  - Drop resctrl_kmode_switch(). Deactivation restores inherit_user
    and the default ctrl and mon values.
  - Move kmode_cpus visibility to after activation and deactivation,
    and move the mode-change and reparent checks to after the
    kernel_mode write interface.
  - Clear MSR_IA32_PQR_PLZA_ASSOC from clear_closid_rmid() so a kexec
    boot does not keep a previous kernel's PLZA state.

v6:
  - Addressed all the comments as discussed in v5. Anything missing
    is not intentional. Please feel free to comment on any mistake.
  - Rename assign_global_enable_per_cpu to global_enable_per_cpu and
    describe the interface as a kernel mode rather than a policy.
  - Flatten kernel mode state into resctrl_kmode. The arch hook is
    resctrl_arch_configure_kmode_global().
  - An omitted ctrl=, mon=, or group= keeps the value shown by reading
    info/kernel_mode. inherit_user takes no parameters.
    ctrl=inherit;mon=inherit remains global_enable_per_cpu.
  - Selecting a new global_enable_per_cpu association sets that group's
    kmode_cpus to every online CPU. A CPU that comes online later joins
    the association even if user space had removed it from the mask.
  - kmode_cpus files are created hidden and shown only while the group
    backs the active association.
  - A monitor group requires mon=assign. Removing a control group
    detaches a child that backs the association before the parent's
    CLOSID is freed.
  - Reject a mode change when the group or one of its monitor groups
    backs the association.
  - Split kmode_cpus visibility and the mode-change rejection into
    their own patches.
  - Use the parsing similar to ctrlmondata.c:parse_line(),
    ctrlmondata.c:resctrl_io_alloc_parse_line(),
    monitor.c:resctrl_parse_mbm_assignment().
  - Re-wrote the changelog as a feature and not as a bug-fix.
  - Remove implementation details in function headers.

v5:
  - Collapse the two v4 global-assign modes into a single
    assign_global_enable_per_cpu policy with ctrl= and mon= options.
  - Use "association" terminology consistently in code, errors, and
    documentation.
  - Register assign_global_enable_per_cpu once during x86 resource
    discovery when PLZA is available.
  - Reject mon=inherit when group= selects a monitor group.
  - Split hidden-file support, deactivation-on-teardown, and PLZA mode
    registration into separate patches for easier review.
  - Fix msr_pqr_plza_assoc truncation on 32-bit systems by using u64.
  - Refresh documentation and add a readable end-to-end example walkthrough.
  - Block pseudo-lock setup on kernel-mode-associated groups.
  - Reject mode changes and rename while a group backs the active
    association.

v4:
  - Reorder and split the series into 15 patches: separate read-only
    info/kernel_mode display from the write path; add hotplug support
    when a CPU comes online; add an end-to-end documentation/examples
    patch.
  - Introduced resctrl_set_kmode_support() so architecture code can
    register supported kernel-mode policies during resctrl initialization.
  - info/kernel_mode write: validate group type; run fail paths before
    tearing down the active association so errors retain the old state.
  - kmode_cpus / kmode_cpus_list: writable with incremental enable/disable
    deltas; empty masks allowed; offline CPUs rejected.
  - Hotplug: newly online CPUs are added to the associated group's
    kmode_cpu_mask and programmed when assign_global_enable_per_cpu is
    active.

v3:
  - Generalise the layer beyond AMD: rename "PLZA mode" to "kernel mode"
    (kmode) in code, sysfs, and Documentation.
  - Reset the association when the associated rdtgroup is removed,
    instead of leaving stale state.

v2:
  - Similar to RFC with a new proposal; interface names were not final.
  - Separated Global Bandwidth Enforcement (GLBE) from PLZA; this series
    only adds PLZA support.
  - Used "kmode" instead of "PLZA" in the generic layer.

Previous versions:
v6: https://lore.kernel.org/lkml/cover.1790867441.git.babu.moger@amd.com/
v5: https://lore.kernel.org/lkml/cover.1787772750.git.babu.moger@amd.com/
v4: https://lore.kernel.org/lkml/cover.1783461016.git.babu.moger@amd.com/
v3: https://lore.kernel.org/lkml/cover.1777591496.git.babu.moger@amd.com/
v2: https://lore.kernel.org/lkml/cover.1773347820.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1769029977.git.babu.moger@amd.com/

Babu Moger (19):
  x86/cpufeatures: Support Privilege Level Zero Association (PLZA)
  x86/resctrl: Add PLZA support to command-line options
  x86/resctrl: Add PLZA MSR definition and field layout
  fs/resctrl: Introduce kernel mode enum
  arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU
    kernel mode
  fs/resctrl: Introduce data structures for kernel mode configuration
  fs/resctrl: Introduce resctrl_set_kmode_support() to register
    supported modes
  x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is
    available
  fs/resctrl: Rearrange top-level files in info directory
  fs/resctrl: Add interface to display kernel mode status
  fs/resctrl: Add support for hidden resource group files
  fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup
  x86,fs/resctrl: Add CPU hotplug support for kernel mode
  fs/resctrl: Deactivate kernel mode when the group is removed
  fs/resctrl: Extend info/kernel_mode to configure the kernel mode
  fs/resctrl: Reject mode changes for groups backing kernel mode
  fs/resctrl: Control visibility of global per-CPU kernel mode group
    files
  fs/resctrl: Allow userspace to write kmode_cpus/kmode_cpus_list
  fs/resctrl: Add kernel mode documentation and examples

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 255 ++++++-
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   9 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  38 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  49 ++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 drivers/resctrl/mpam_resctrl.c                |   7 +
 fs/resctrl/internal.h                         |  32 +
 fs/resctrl/rdtgroup.c                         | 693 ++++++++++++++++++
 include/linux/resctrl.h                       |  65 ++
 12 files changed, 1141 insertions(+), 12 deletions(-)

-- 
2.43.0

[PATCH v7 01/19] x86/cpufeatures: Support Privilege Level Zero Association (PLZA)

[Babu Moger]

When memory bandwidth associated with a CLOSID is aggressively throttled,
and a task with that CLOSID moves into kernel mode, the kernel operations
are also aggressively throttled. This can stall forward progress and
eventually degrade overall system performance.

AMD hardware supports a feature Privilege Level Zero Association (PLZA),
which allows the CPU's CLOSID and/or RMID association to be changed during
the transition from user mode to kernel mode. This allows the kernel to run
using a different CLOSID and/or RMID than user space, which can improve
system performance and/or allow kernel work to be monitored separately.

Note that PLZA feature associates CLOSID and/or RMID to kernel work, and
therefore influences resource allocation and/or monitoring for all
resources, not only memory bandwidth.

The feature is detected via CPUID_Fn80000020_EBX_x00 [Bit 9]: Privilege
Level Zero Association (PLZA).

The PLZA feature details are documented in [1] available from [2].

[1] AMD64 Zen6 Platform Quality of Service (PQOS) Extensions:
    Publication # 69193 Revision: 1.00, Issue Date: March 2026

Signed-off-by: Babu Moger <babu.moger@amd.com>
Acked-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 # [2]
---
v7: Added Reviewed-by: tag(Reinette).

v6: Highlight that CLOSID/RMID association affects all resources,
    not just memory bandwidth.

v5: Added Acked-by from Boris.
    Updated the changelog to include cache allocation and monitoring.
    Moved the Link tag last.
    Updated subject line for cpufeatures.

v4: Split the patch into 2. This patch only handles x86 changes.
    Re-wrote the changelog along the ABMC changes.

v3: Code did not change. Patch order changed.
    Added documentation link.

v2: Rebased on top of the latest tip.
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/scattered.c    | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 303467a498b8..3090fb28ce06 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -536,6 +536,7 @@
 						      * and purposes if CLEAR_CPU_BUF_VM is set).
 						      */
 #define X86_FEATURE_X2AVIC_EXT		(21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
+#define X86_FEATURE_PLZA		(21*32+21) /* Privilege Level Zero Association */
 
 /*
  * BUG word(s)
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index c990245d29d6..be387ba1cbea 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -62,6 +62,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
 	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
+	{ X86_FEATURE_PLZA,			CPUID_EBX,  9, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_TSA_SQ_NO,		CPUID_ECX,  1, 0x80000021, 0 },
 	{ X86_FEATURE_TSA_L1_NO,		CPUID_ECX,  2, 0x80000021, 0 },
-- 
2.43.0

[PATCH v7 02/19] x86/resctrl: Add PLZA support to command-line options

[Babu Moger]

Add "plza" to the rdt= command-line parameter so PLZA (Privilege Level Zero
Association) exposure to resctrl can be enabled or disabled at boot. PLZA
is exposed by default when supported; rdt=!plza suppresses it.

Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
---
v7: Removed repetative lines.
    Added Reviewed-by: (Reinette).

v6: Updated the changelog.

v5: Updated the changelog to mention about rdt= options.

v4: Split the patch 1 from v3 into 2 patches.
    This patch contains changes for resctrl subsystem.
---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 Documentation/filesystems/resctrl.rst           | 1 +
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd9..b8c680f7f638 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6460,7 +6460,7 @@ Kernel parameters
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec, abmc, sdciae, energy[:guid],
+			mba, smba, bmec, abmc, sdciae, plza, energy[:guid],
 			perf[:guid].
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 2750f2eabf70..d13a4998ee7e 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -28,6 +28,7 @@ SMBA (Slow Memory Bandwidth Allocation)				""
 BMEC (Bandwidth Monitoring Event Configuration)			""
 ABMC (Assignable Bandwidth Monitoring Counters)			""
 SDCIAE (Smart Data Cache Injection Allocation Enforcement)	""
+PLZA (Privilege Level Zero Association)				""
 =============================================================== ================================
 
 Historically, new features were made visible by default in /proc/cpuinfo. This
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 3294b18ba45f..e64e9191184e 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -772,6 +772,7 @@ enum {
 	RDT_FLAG_BMEC,
 	RDT_FLAG_ABMC,
 	RDT_FLAG_SDCIAE,
+	RDT_FLAG_PLZA,
 };
 
 #define RDT_OPT(idx, n, f)	\
@@ -799,6 +800,7 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
 	RDT_OPT(RDT_FLAG_ABMC,	    "abmc",	X86_FEATURE_ABMC),
 	RDT_OPT(RDT_FLAG_SDCIAE,    "sdciae",	X86_FEATURE_SDCIAE),
+	RDT_OPT(RDT_FLAG_PLZA,	    "plza",	X86_FEATURE_PLZA),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
-- 
2.43.0

[PATCH v7 03/19] x86/resctrl: Add PLZA MSR definition and field layout

[Babu Moger]

Privilege Level Zero Association (PLZA) enables a CPU to use a different
CLOSID and/or RMID association when executing in kernel mode than those
used in user mode.

PLZA is programmed per-CPU through MSR_IA32_PQR_PLZA_ASSOC, which contains
the kernel mode CLOSID and RMID, an enable bit for each, and a bit that
enables PLZA on the CPU. All fields except the PLZA enable bit must be
programmed identically on every CPU in a domain.

Make the register and its field layout available in preparation for resctrl
support of PLZA.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated changelog(Reinette) with minor changes.
    Minor code comment update.

v6: Updated the code comments to match general format (members first and
    details after).
    changelog update.

v5: Sashiko reported:
    https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=3
    Fix the union truncate issue on 32 bit systems by using u64.
    Updated the changelog and code comments.

v4: Re-wrote the changelog and code comment.

v3: No code changes. Patch order changed. Improved changelog.

v2: No changes. Just rebasing on top of the latest tip branch.
---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 49 ++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index c0914829a925..13f81a0587b4 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -1306,6 +1306,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_PQR_PLZA_ASSOC		0xc00003fc
 #define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
 #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 03cb22f4e607..d75013ede600 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -226,6 +226,55 @@ union l3_qos_abmc_cfg {
 	unsigned long full;
 };
 
+/*
+ * Privilege Level Zero Association (PLZA) is configured via
+ * MSR_IA32_PQR_PLZA_ASSOC.
+ *
+ * @rmid		: RMID programmed for PLZA.
+ * @reserved1		: Reserved.
+ * @rmid_en		: Enable RMID for PLZA.
+ * @closid		: CLOSID programmed for PLZA.
+ * @reserved2		: Reserved.
+ * @closid_en		: Enable CLOSID for PLZA.
+ * @reserved3		: Reserved.
+ * @plza_en		: Enable PLZA. When enabled, PLZA applies to the
+ *			  given CPU.
+ *
+ * The MSR_IA32_PQR_PLZA_ASSOC must be programmed on every CPU in a QoS
+ * domain. Each domain supports a single PLZA CLOSID and/or RMID, and all
+ * the fields except @plza_en must be programmed identically in a domain
+ * to ensure consistent behavior.
+ *
+ * CLOSID and RMID selections are controlled independently. When PLZA is
+ * enabled (@plza_en = 1), CPL0 uses CLOSID and RMID values from
+ * MSR_IA32_PQR_PLZA_ASSOC depending on the corresponding enable bits. When
+ * PLZA is disabled (@plza_en = 0), the CLOSID and RMID are always taken
+ * from MSR_IA32_PQR_ASSOC regardless of privilege level.
+ *
+ * - @rmid_en = 1: use the RMID from MSR_IA32_PQR_PLZA_ASSOC if @plza_en = 1.
+ * - @rmid_en = 0: use the RMID from MSR_IA32_PQR_ASSOC.
+ * - @closid_en = 1: use the CLOSID from MSR_IA32_PQR_PLZA_ASSOC if @plza_en = 1.
+ * - @closid_en = 0: use the CLOSID from MSR_IA32_PQR_ASSOC.
+ *
+ * This allows PLZA to override only RMID, only CLOSID, or both. If both
+ * @rmid_en and @closid_en are clear, the effective behavior is the same as
+ * when PLZA is disabled, with RMID and CLOSID used from MSR_IA32_PQR_ASSOC
+ * irrespective of privilege level.
+ */
+union msr_pqr_plza_assoc {
+	struct {
+		u64	rmid		:12,
+			reserved1	:19,
+			rmid_en		: 1,
+			closid		: 4,
+			reserved2	:11,
+			closid_en	: 1,
+			reserved3	:15,
+			plza_en		: 1;
+	} split;
+	u64	full;
+};
+
 void rdt_ctrl_update(void *arg);
 
 int rdt_get_l3_mon_config(struct rdt_resource *r);
-- 
2.43.0

[PATCH v7 04/19] fs/resctrl: Introduce kernel mode enum

[Babu Moger]

Kernel work need not use the CLOSID and RMID of the user task on whose
behalf it runs. On x86, Privilege Level Zero Association (PLZA) enables a
CPU to use a different CLOSID and/or RMID association while executing at
privilege level zero.

Resctrl needs common definitions of how kernel work is managed so that
architectures can advertise the kernel modes they support and user space
can query them and select the one to use.

Introduce enum resctrl_kernel_mode:

- RESCTRL_INHERIT_USER: Kernel work uses the CLOSID and RMID of the user
  task.

- RESCTRL_GLOBAL_ENABLE_PER_CPU: All the kernel work uses the CLOSID
  and/or RMID of a single resource group, which may also have user tasks
  assigned to it. The mode can be enabled or disabled per-CPU; on a CPU
  where it is disabled, kernel work inherits the user task's CLOSID and
  RMID.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog update (Reinette) with minor changes.
    Code comment update.

v6: changelog update and code comment update.
    Renamed RESCTRL_ASSIGN_GLOBAL_ENABLE_PER_CPU to RESCTRL_GLOBAL_ENABLE_PER_CPU.

v5: Collapse the two global-assign modes into a single
    RESCTRL_ASSIGN_GLOBAL_ENABLE_PER_CPU value. Update the changelog to
    match the enum names and descriptions in the code.

v4: Updated the changelog to be generic as possible.
    Moved the enum resctrl_kernel_mode to include/linux/resctrl.h.
    Updated the code comments to be generic.
    Removed resctrl_kmode_cfg from the code. This definition can be fs
    specific only and architectures dont need to know.
    Changed enum name to resctrl_kernel_mode from resctrl_kernel_modes.

v3: Removed resctrl_kmode definition.
    Changed the kernel mode definitions to enum resctrl_kernel_modes.
    Used BIT() to set/test the features.
    Added details to changelog.

v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.
    https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
---
 include/linux/resctrl.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index b6b4519a5190..6f3e57ec4aa3 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -716,6 +716,31 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
  */
 bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
 
+/**
+ * enum resctrl_kernel_mode - Kernel modes
+ *
+ * @RESCTRL_INHERIT_USER:
+ *     Kernel work inherits the CLOSID and RMID of the user task.
+ *
+ * @RESCTRL_GLOBAL_ENABLE_PER_CPU:
+ *     All kernel work uses the CLOSID and/or RMID of a single resource
+ *     group, which may also have user tasks assigned to it. The mode can
+ *     be enabled/disabled per-CPU, all online CPUs are enabled by default.
+ *     When the mode is disabled on a CPU then the kernel work on that CPU
+ *     inherits the CLOSID and RMID from the user task, effectively the
+ *     kernel mode becomes RESCTRL_INHERIT_USER on that CPU.
+ *
+ *     On x86, CLOSID and RMID are configured independently; either or both
+ *     may differ from the user task.
+ */
+enum resctrl_kernel_mode {
+	RESCTRL_INHERIT_USER,
+	RESCTRL_GLOBAL_ENABLE_PER_CPU,
+	RESCTRL_KMODE_LAST = RESCTRL_GLOBAL_ENABLE_PER_CPU
+};
+
+#define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
-- 
2.43.0

[PATCH v7 05/19] arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU kernel mode

[Babu Moger]

RESCTRL_GLOBAL_ENABLE_PER_CPU enables a CPU to use a different CLOSID
and/or RMID association when executing in kernel mode than those used in
user mode. On x86, this mode corresponds to PLZA (Privilege Level Zero
Association).

PLZA is programmed per-CPU through MSR_IA32_PQR_PLZA_ASSOC, which contains
the kernel mode CLOSID and RMID, an enable bit for each, and a bit that
enables PLZA on the CPU. All fields except the PLZA enable bit must be
programmed identically on every CPU in a domain.

The mode is enabled independently on each CPU, so the filesystem must
program hardware when the mode is activated, when CPUs are added or
removed, and when it is deactivated. CLOSID and RMID can be programmed
independently, so the architecture callback provides separate enable
options for each.

Add resctrl_arch_configure_kmode_global() with an x86 PLZA implementation
and an empty MPAM stub for ARM systems that do not support this mode.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the changelog with consistant terms.
    Update changelog based on comments from Reinette with minor changes.

v6: Rename to resctrl_arch_configure_kmode_global(). Rewrite changelog
    for RESCTRL_GLOBAL_ENABLE_PER_CPU only (not a bugfix, not a code
    restatement). Add arm to the subject. Document parameters first;
    closid/rmid are always programmed. Split fs/arch contract from
    call scenarios and name the mode explicitly.

v5: Fix kernel-doc for resctrl_arch_configure_kmode(), wire assign_ctrl
    through to PLZA closid_en, and update the changelog.
    Add calling context in API doc.

v4: Added assign_mon parameter in resctrl_arch_configure_kmode() to program the RMID
    as discussed in below.
    https://lore.kernel.org/lkml/20260605100642.1103628-1-qinyuntan@linux.alibaba.com/
    Reintroduce independent monitoring assignment via assign_mon after removing
    the task-based PLZA approach in v3.
    Changed cpumask type to "const struct cpumask *cpu_mask".
    Added MPAM stub to avoid any linking issues when resctrl_arch_configure_kmode()
    is called from FS layer. Thanks to Qinyun.
    Re-wrote the changelog to be generic.
    Updated code comments.
v3: Removed task based PLZA implementation so related changes are removed.
    Per-task RMID handling is replaced by group type selection in the fs layer.
    Updated the change log with details.
    Removed resctrl_arch_set_kmode() as arch only provides the modes supported.
    It is FS which decided which mode to apply.
v2: Updated the commit message to include the sequence of steps to enable PLZA.
    Added mode code comments for clarity.
    Added kmode to function names to be generic.
---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 +++++++++++++++++++++++
 drivers/resctrl/mpam_resctrl.c            |  7 +++++
 include/linux/resctrl.h                   | 28 +++++++++++++++++
 3 files changed, 73 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
index 62044489b052..79b6fdcbc553 100644
--- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
+++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
@@ -137,3 +137,41 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
 
 	return 0;
 }
+
+static void resctrl_kmode_set_one_amd(void *arg)
+{
+	union msr_pqr_plza_assoc *plza = arg;
+
+	wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, plza->full);
+}
+
+/*
+ * resctrl_arch_configure_kmode_global - Program Privilege Level Zero
+ *					 Association (PLZA).
+ * @cpu_mask:	CPUs whose MSR_IA32_PQR_PLZA_ASSOC should be updated.
+ * @closid:	CLOSID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_ctrl is false.
+ * @assign_ctrl: Whether PLZA should program the kernel mode CLOSID.
+ * @rmid:	RMID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_mon is false.
+ * @assign_mon:	Whether PLZA should program the kernel mode RMID.
+ * @enable:	Whether PLZA should program the kernel mode.
+ *
+ * Program PLZA on @cpu_mask for %RESCTRL_GLOBAL_ENABLE_PER_CPU. See
+ * union msr_pqr_plza_assoc for the hardware field meanings.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+	union msr_pqr_plza_assoc plza = { 0 };
+
+	plza.split.rmid = rmid;
+	plza.split.rmid_en = assign_mon;
+	plza.split.closid = closid;
+	plza.split.closid_en = assign_ctrl;
+	plza.split.plza_en = enable;
+
+	on_each_cpu_mask(cpu_mask, resctrl_kmode_set_one_amd, &plza, 1);
+}
diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
index 5468367776e2..548bfb336be5 100644
--- a/drivers/resctrl/mpam_resctrl.c
+++ b/drivers/resctrl/mpam_resctrl.c
@@ -139,6 +139,13 @@ bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
 	return false;
 }
 
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+}
+
 void resctrl_arch_pre_mount(void)
 {
 }
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 6f3e57ec4aa3..e43415ae6e0e 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -741,6 +741,34 @@ enum resctrl_kernel_mode {
 
 #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 
+/**
+ * resctrl_arch_configure_kmode_global() - Program RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *
+ * @cpu_mask:	Online CPUs on which to update the CLOSID and/or RMID.
+ * @closid:	CLOSID of the resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *		mode. Always provided, even when @assign_ctrl is false.
+ * @assign_ctrl: true to update the @closid for kernel mode; false to inherit the
+ *		CLOSID from the user task.
+ * @rmid:	RMID of the resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *		mode. Always provided, even when @assign_mon is false.
+ * @assign_mon:	true to update the @rmid for kernel mode; false to inherit the
+ *		RMID from the user task.
+ * @enable:	true to enable %RESCTRL_GLOBAL_ENABLE_PER_CPU on CPUs in @cpu_mask;
+ *		false to disable @RESCTRL_GLOBAL_ENABLE_PER_CPU and reset the
+ *		kernel mode to %RESCTRL_INHERIT_USER.
+ *
+ * The callback programs %RESCTRL_GLOBAL_ENABLE_PER_CPU. A configuration starts
+ * when %RESCTRL_GLOBAL_ENABLE_PER_CPU is activated and ends when the mode is
+ * deactivated or userspace changes the backing resource group, @assign_ctrl, or
+ * @assign_mon. Each call for a given configuration uses the same @closid,
+ * @assign_ctrl, @rmid, and @assign_mon; only @cpu_mask and @enable differ.
+ * Before a configuration ends, it is disabled on every CPU where it is enabled.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
-- 
2.43.0

[PATCH v7 06/19] fs/resctrl: Introduce data structures for kernel mode configuration

[Babu Moger]

The resctrl filesystem needs to track the supported kernel modes, the
active mode, and whether the backing group's CLOSID and RMID are assigned
or inherited.

Introduce struct resctrl_kmode_cfg to represent supported kernel modes and
their configuration.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changed kmode_sup -> supported, kmode_cur -> active.
    Updated the changelog.
    Removed resctrl_kmode_init().
    Updated the code comments.

v6: Drop ctrl_en/mon_en and use resctrl_arch_alloc_capable() /
    resctrl_arch_mon_capable() directly. Flatten caps/active into
    resctrl_kmode_cfg. Rename resctrl_kcfg to resctrl_kmode, ctrl_mode
    to ctrl, mon_mode to mon, and k_rdtgrp to rdtgrp. Comments describe
    the objects, not a member list. Rewrite changelog.

v5: Replace per-mode registration helpers with a single resctrl_kmode_cfg
    structure. Derive ctrl_en and mon_en from resctrl_arch_alloc_capable()
    and resctrl_arch_mon_capable() during filesystem init. Update code
    comments and changelog.

v4: New patch to initialize supported kernel modes.
    https://lore.kernel.org/lkml/737a4228-52fb-4583-ac64-8efe79c107e6@intel.com/
    Changed the kmode_cur to enum resctrl_kernel_mode in resctrl_kmode_cfg.
    Moved the resctrl_kmode_cfg to filesystem code.
---
 fs/resctrl/internal.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index e62a277dee85..38aa0c0cfb7e 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -314,6 +314,33 @@ struct mbm_state {
 	u32	prev_bw;
 };
 
+/**
+ * enum kmode_state - Kernel mode association state.
+ * @KMODE_INHERIT:    Inherit the CLOSID or RMID from the user task.
+ * @KMODE_ASSIGN:     Use the CLOSID or RMID from the backing resource group.
+ */
+enum kmode_state {
+	KMODE_INHERIT,
+	KMODE_ASSIGN
+};
+
+/**
+ * struct resctrl_kmode_cfg - Kernel mode configuration
+ * @supported:	Bitmap of supported kernel modes.
+ * @active:	Currently active kernel mode.
+ * @ctrl:	CLOSID assign or inherit in %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ * @mon:	RMID assign or inherit in %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ * @rdtgrp:	Resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ *		%NULL while the mode is not selected.
+ */
+struct resctrl_kmode_cfg {
+	DECLARE_BITMAP(supported, RESCTRL_NUM_KERNEL_MODES);
+	enum resctrl_kernel_mode	active;
+	enum kmode_state		ctrl;
+	enum kmode_state		mon;
+	struct rdtgroup			*rdtgrp;
+};
+
 extern struct mutex rdtgroup_mutex;
 
 static inline const char *rdt_kn_name(const struct kernfs_node *kn)
-- 
2.43.0

[PATCH v7 07/19] fs/resctrl: Introduce resctrl_set_kmode_support() to register supported modes

[Babu Moger]

Architectures may support different kernel modes that need to be exposed to
user space via resctrl filesystem.

Initialize the resctrl_kmode data structure and add
resctrl_set_kmode_support() to allow architecture to register its supported
kernel modes. This is preparatory infrastructure for future kernel mode
support in resctrl.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Moved the static initialization for resctrl_kmode in here as it is used
    here first.
    Updated the changelog and code comments.

v6: Use resctrl_kmode.kmode_sup after the struct flatten. Drop duplicate
    kernel-doc from the .c definition and place the function next to
    resctrl_kmode_init(). Rewrite changelog (not a bugfix).

v5: New patch split from resctrl_kmode_cfg initialization. Register modes
    in resctrl_kmode_cfg.caps.kmode_sup using RESCTRL_INHERIT_USER and
    RESCTRL_ASSIGN_GLOBAL_ENABLE_PER_CPU. Ignore out-of-range mode values.

v4: New patch to initialize supported kernel modes.
    https://lore.kernel.org/lkml/737a4228-52fb-4583-ac64-8efe79c107e6@intel.com/
---
 fs/resctrl/rdtgroup.c   | 14 ++++++++++++++
 include/linux/resctrl.h | 12 ++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 443b41a4d467..28bd4c4d98f2 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -76,6 +76,14 @@ static void rdtgroup_destroy_root(void);
 
 struct dentry *debugfs_resctrl;
 
+static struct resctrl_kmode_cfg resctrl_kmode = {
+	.supported = { BIT(RESCTRL_INHERIT_USER) },
+	.active = RESCTRL_INHERIT_USER,
+	.ctrl = KMODE_ASSIGN,
+	.mon = KMODE_ASSIGN,
+	.rdtgrp = NULL,
+};
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when
@@ -2297,6 +2305,12 @@ static void io_alloc_init(void)
 	}
 }
 
+void resctrl_set_kmode_support(enum resctrl_kernel_mode kmode)
+{
+	if (kmode < RESCTRL_NUM_KERNEL_MODES)
+		__set_bit(kmode, resctrl_kmode.supported);
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index e43415ae6e0e..519c28c63bc7 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -741,6 +741,18 @@ enum resctrl_kernel_mode {
 
 #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 
+/**
+ * resctrl_set_kmode_support() - Register a supported kernel mode
+ * @kmode:	Kernel mode supported by the architecture.
+ *
+ * %RESCTRL_INHERIT_USER is a default mode and does not need to be
+ * registered.
+ *
+ * Architectures use this interface during resctrl initialization to
+ * advertise supported kernel modes.
+ */
+void resctrl_set_kmode_support(enum resctrl_kernel_mode kmode);
+
 /**
  * resctrl_arch_configure_kmode_global() - Program RESCTRL_GLOBAL_ENABLE_PER_CPU
  *
-- 
2.43.0

[PATCH v7 08/19] x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is available

[Babu Moger]

PLZA (Privilege Level Zero Association) provides the global per-CPU kernel
mode, RESCTRL_GLOBAL_ENABLE_PER_CPU, which enables a CPU to use a different
CLOSID and/or RMID association when executing in kernel mode than those
used in user mode.

Register the RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode with the resctrl
kernel mode infrastructure when PLZA is supported.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog Update for consistancy.

v6: Use RESCTRL_GLOBAL_ENABLE_PER_CPU.
    Rewrite changelog (not a bugfix).

v5: Update the patch to adapt new data structure and resctrl_set_kmode_support()
    changes.

v4: New patch to set the supported features during arch init.
---
 arch/x86/kernel/cpu/resctrl/core.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index e64e9191184e..14530c11bbd7 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -990,6 +990,10 @@ static __init bool get_rdt_resources(void)
 	rdt_alloc_capable = get_rdt_alloc_resources();
 	rdt_mon_capable = get_rdt_mon_resources();
 
+	if (rdt_cpu_has(X86_FEATURE_PLZA) &&
+	    (rdt_alloc_capable || rdt_mon_capable))
+		resctrl_set_kmode_support(RESCTRL_GLOBAL_ENABLE_PER_CPU);
+
 	return (rdt_mon_capable || rdt_alloc_capable);
 }
 
-- 
2.43.0

[PATCH v7 09/19] fs/resctrl: Rearrange top-level files in info directory

[Babu Moger]

The "last_cmd_status" file reports the status of the last command
executed by the resctrl subsystem. Since it is documented after all
other files in the info directory, it is easy to overlook.

Create a dedicated section for top-level files in the info directory
and move the "last_cmd_status" documentation there.

Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7:  New patch to move the last_cmd_status in new section.
---
 Documentation/filesystems/resctrl.rst | 29 +++++++++++++++++----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index d13a4998ee7e..2d672bbb69be 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -529,18 +529,25 @@ with the following files:
 The upper bound for how many "CTRL_MON" + "MON" can be created
 is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values.
 
-Finally, in the top level of the "info" directory there is a file
-named "last_cmd_status". This is reset with every "command" issued
-via the file system (making new directories or writing to any of the
-control files). If the command was successful, it will read as "ok".
-If the command failed, it will provide more information that can be
-conveyed in the error returns from file operations. E.g.
-::
+Top level files
+---------------
+
+The top level of the "info" directory contains the following file(s):
+
+"last_cmd_status":
+	Status of the last command executed by the resctrl subsystem.
+
+	This file is reset with every "command" issued via the file
+	system (making new directories or writing to any of the control
+	files). If the command was successful, it will read as "ok".
+	If the command failed, it will provide more information that
+	can be conveyed in the error returns from file operations.
+	For example::
 
-	# echo L3:0=f7 > schemata
-	bash: echo: write error: Invalid argument
-	# cat info/last_cmd_status
-	mask f7 has non-consecutive 1-bits
+		# echo L3:0=f7 > schemata
+		bash: echo: write error: Invalid argument
+		# cat info/last_cmd_status
+		mask f7 has non-consecutive 1-bits
 
 Resource alloc and monitor groups
 =================================
-- 
2.43.0

[PATCH v7 10/19] fs/resctrl: Add interface to display kernel mode status

[Babu Moger]

Kernel mode support allows the CPU's CLOSID and/or RMID association to
change when transitioning from user mode to kernel mode.

Add info/kernel_mode to expose the supported and active kernel modes
through the resctrl filesystem. Supported modes are listed one per line,
with the active mode enclosed in square brackets.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the changelog.
    kernel_mode prints the monitoring group only when mon == assign.

v6: Rewrote the changelog to make it as feature and not as a bug fix.
    Update the code to use the new name "global_enable_per_cpu".
    Display ctrl or mon only when allocation or monitoring is supportd.

v5: Use resctrl_kcfg.caps and resctrl_kcfg.active after the struct split
    from patch 0006. Rename resctrl_kmode_bind_str() to
    resctrl_kmode_state_str() and ctrl_bind/mon_bind to ctrl_state/mon_state.
    Used info_kn_lock()/info_kn_unlock().
    Add documentation update in resctrl.rst.

v4: Fixed the display of inherit_ctrl_and_mon policy. It is not
    associated with any group.
    Added "uninitialized" for the non-active group.
    Rewrote the changelog.

v3: New patch to handle the changed interface file info/kernel_mode.
    Changed the group name to "none" if kmode binding is not done.
    Reinette suggested "uninitialized". "none" seemed more relevent.
---
 Documentation/filesystems/resctrl.rst |  56 ++++++++++++++
 fs/resctrl/rdtgroup.c                 | 105 ++++++++++++++++++++++++++
 2 files changed, 161 insertions(+)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 2d672bbb69be..033d405338c0 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -549,6 +549,62 @@ The top level of the "info" directory contains the following file(s):
 		# cat info/last_cmd_status
 		mask f7 has non-consecutive 1-bits
 
+"kernel_mode":
+	Lists the kernel modes supported by the system, one per line, with
+	the active mode enclosed in square brackets.
+
+	A task running on a CPU is associated with an allocation and
+	monitoring ID pair (ctrl_hw_id, mon_hw_id), which determines the
+	resource allocation and monitoring applied to that task. Kernel
+	modes allow the CPU's allocation and/or monitoring association to
+	change during the transition from user mode to kernel mode.
+
+	Possible kernel modes are:
+
+	"inherit_user":
+		Default mode at boot. Kernel work uses the allocation
+		and monitoring of the user task on whose behalf it runs.
+
+	"global_enable_per_cpu":
+		Kernel work uses the allocation and/or monitoring of a backing
+		CTRL_MON or MON group. The mode can be enabled or disabled
+		per-CPU. The line contains group= and whichever of ctrl= and
+		mon= the platform supports::
+
+		  global_enable_per_cpu:ctrl=<assign|inherit>;mon=<assign|inherit>;group=<CTRL_MON>/<MON>/
+
+		"ctrl":
+			"assign" if kernel work uses the resource group's
+			allocation, or "inherit" if it uses the user task's
+			allocation. Present only on systems that support allocation.
+
+		"mon":
+			"assign" if kernel work uses the resource group's monitoring,
+			or "inherit" if it uses the user task's monitoring. Present
+			only on systems that support monitoring.
+
+		"group":
+			Path of the backing resource group, relative to the resctrl
+			mount point.
+
+			A MON group is shown as "<CTRL_MON>/<MON>/".
+
+			A CTRL_MON group is shown as "<CTRL_MON>//" when "mon=assign".
+
+			When "mon=inherit", or monitoring is unsupported, a CTRL_MON group
+			is shown as "<CTRL_MON>/".
+
+		When the mode is not active, the output lists the supported parameters
+		and their default values, which are used if the mode is activated
+		without new values.
+
+	Example of a system that supports allocation and monitoring, with default mode
+	set to inherit_user::
+
+		# cat info/kernel_mode
+		[inherit_user]
+		global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
 Resource alloc and monitor groups
 =================================
 
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 28bd4c4d98f2..afbe0f94940a 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -998,6 +998,104 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/* Names used in info/kernel_mode; indexed by enum resctrl_kernel_mode */
+static const char * const resctrl_kmode_str[] = {
+	[RESCTRL_INHERIT_USER]		= "inherit_user",
+	[RESCTRL_GLOBAL_ENABLE_PER_CPU]	= "global_enable_per_cpu"
+};
+
+static_assert(ARRAY_SIZE(resctrl_kmode_str) == RESCTRL_NUM_KERNEL_MODES);
+
+static const char *resctrl_kmode_state_str(enum kmode_state state)
+{
+	return state == KMODE_ASSIGN ? "assign" : "inherit";
+}
+
+static void resctrl_kmode_group_path(struct rdtgroup *rdtgrp,
+				     const char **ctrl, const char **mon)
+{
+	*ctrl = "";
+	*mon = "";
+
+	if (rdtgrp->type == RDTMON_GROUP) {
+		*ctrl = rdt_kn_name(rdtgrp->mon.parent->kn);
+		*mon = rdt_kn_name(rdtgrp->kn);
+	} else {
+		*ctrl = rdt_kn_name(rdtgrp->kn);
+	}
+}
+
+/**
+ * resctrl_kernel_mode_show() - Display supported and active kernel modes
+ * @of:		kernfs open file
+ * @seq:	output seq_file
+ * @v:		unused
+ *
+ * Lists one line per mode and brackets the active mode. inherit_user
+ * has no parameters. global_enable_per_cpu prints ctrl= only when
+ * allocation is supported and mon= only when monitoring is supported,
+ * then group=. The MON group is included in group= only when monitoring
+ * is supported and mon is assign.
+ *
+ * Return: 0 on success, or -ENOENT on error.
+ */
+static int resctrl_kernel_mode_show(struct kernfs_open_file *of,
+				    struct seq_file *seq, void *v)
+{
+	const char *ctrl_grp, *mon_grp;
+	enum resctrl_kernel_mode mode;
+	bool active;
+	char sep;
+
+	if (!info_kn_lock(of->kn))
+		return -ENOENT;
+
+	for (mode = 0; mode < RESCTRL_NUM_KERNEL_MODES; mode++) {
+		if (!test_bit(mode, resctrl_kmode.supported))
+			continue;
+
+		active = resctrl_kmode.active == mode;
+
+		seq_printf(seq, active ? "[%s" : "%s", resctrl_kmode_str[mode]);
+
+		if (mode == RESCTRL_GLOBAL_ENABLE_PER_CPU) {
+			ctrl_grp = "";
+			mon_grp = "";
+			if (active) {
+				if (resctrl_kmode.rdtgrp)
+					resctrl_kmode_group_path(resctrl_kmode.rdtgrp,
+								 &ctrl_grp, &mon_grp);
+				else
+					pr_warn("%s active without a resource group\n",
+						resctrl_kmode_str[mode]);
+			}
+
+			sep = ':';
+			if (resctrl_arch_alloc_capable()) {
+				seq_printf(seq, "%cctrl=%s", sep,
+					   resctrl_kmode_state_str(resctrl_kmode.ctrl));
+				sep = ';';
+			}
+			if (resctrl_arch_mon_capable()) {
+				seq_printf(seq, "%cmon=%s", sep,
+					   resctrl_kmode_state_str(resctrl_kmode.mon));
+				sep = ';';
+			}
+			if (resctrl_arch_mon_capable() &&
+			    resctrl_kmode.mon != KMODE_INHERIT)
+				seq_printf(seq, "%cgroup=%s/%s/", sep, ctrl_grp, mon_grp);
+			else
+				seq_printf(seq, "%cgroup=%s/", sep, ctrl_grp);
+		}
+
+		seq_puts(seq, active ? "]\n" : "\n");
+	}
+
+	info_kn_unlock(of->kn);
+
+	return 0;
+}
+
 void *rdt_kn_parent_priv(struct kernfs_node *kn)
 {
 	/*
@@ -1983,6 +2081,13 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdt_last_cmd_status_show,
 		.fflags		= RFTYPE_TOP_INFO,
 	},
+	{
+		.name		= "kernel_mode",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_kernel_mode_show,
+		.fflags		= RFTYPE_TOP_INFO,
+	},
 	{
 		.name		= "mbm_assign_on_mkdir",
 		.mode		= 0644,
-- 
2.43.0

[PATCH v7 11/19] fs/resctrl: Add support for hidden resource group files

[Babu Moger]

A resctrl file is visible to user space as soon as its resource group
directory is activated. A file can only be hidden after it has been
created, as is done for the info directory's BMEC event configuration
files.

In the global_enable_per_cpu kernel mode, kernel work running on CPUs where
the mode is active uses the CLOSID and/or RMID as part of a single backing
resource group. The files needed to display and modify the CPUs on which
this kernel mode is active are only meaningful in that group. Visible in
every resource group, the files would return errors when accessed in any
group other than the one backing the active kernel mode. Creating the files
when the kernel mode is activated, and removing them when it is
deactivated, would add failure paths to kernel mode activation that showing
and hiding existing files does not have.

Allow a resource group file to be marked as hidden so that it is hidden
when created, before its directory is activated, and is never visible to
user space until resctrl restores its visibility. This provides the
foundation in preparation for kernel mode support that shows the kmode_cpus
and kmode_cpus_list files only in the resource group backing the active
kernel mode.

Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the changelog (Reinette's input with minor update in second
    paragrapj).

v6: Removed resctrl_hidden_files_set_visible() from the patch.
    It will be separate patch.
    Re-wrote the changelog.

v5: New patch to add support for creating hidden files.
---
 fs/resctrl/internal.h | 2 ++
 fs/resctrl/rdtgroup.c | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index 38aa0c0cfb7e..909511e985a1 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -283,6 +283,7 @@ extern int max_name_width;
  * @kf_ops:	File operations
  * @flags:	File specific RFTYPE_FLAGS_* flags
  * @fflags:	File specific RFTYPE_* flags
+ * @hidden:	Hide file at creation; visibility may be restored later
  * @seq_show:	Show content of the file
  * @write:	Write to the file
  */
@@ -292,6 +293,7 @@ struct rftype {
 	const struct kernfs_ops	*kf_ops;
 	unsigned long		flags;
 	unsigned long		fflags;
+	bool			hidden;
 
 	int (*seq_show)(struct kernfs_open_file *of,
 			struct seq_file *sf, void *v);
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index afbe0f94940a..9235f094245c 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -313,6 +313,9 @@ static int rdtgroup_add_file(struct kernfs_node *parent_kn, struct rftype *rft)
 		return ret;
 	}
 
+	if (rft->hidden)
+		kernfs_show(kn, false);
+
 	return 0;
 }
 
-- 
2.43.0

[PATCH v7 12/19] fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup

[Babu Moger]

The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows CPUs to use different
CLOSID and/or RMID associations when transitioning from user mode to kernel
mode. When active, it is enabled or disabled on a per-CPU basis through a
backing resource group.

Add kmode_cpus and kmode_cpus_list to display the CPUs in a resource group
backing the active kernel mode.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog updated with consistant terms.
    Code comment update.

v6: Drop rdtgrp::kmode. Rewrite changelog (not a bugfix; association).
    https://lore.kernel.org/lkml/ef8804d5-f20c-49a1-87cf-ab39437a5f0d@intel.com/

v5: Updated the changelog.
    Removed the RDT_MODE_PSEUDO_LOCKED check and added rdtgroup->kmode check.
    Files are created hidden.

v4: Rewrote the change to be generic.

v3: New patch to add "kmode_cpus" and "kmode_cpus_list" to support
    kernel_modes.
---
 fs/resctrl/internal.h |  3 +++
 fs/resctrl/rdtgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index 909511e985a1..a473b408dd9d 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -216,6 +216,8 @@ struct mongroup {
  * @mon:			mongroup related data
  * @mode:			mode of resource group
  * @mba_mbps_event:		input monitoring event id when mba_sc is enabled
+ * @kmode_cpu_mask:		list CPUs in the group when backing the active
+ *				RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode
  * @plr:			pseudo-locked region
  */
 struct rdtgroup {
@@ -229,6 +231,7 @@ struct rdtgroup {
 	struct mongroup			mon;
 	enum rdtgrp_mode		mode;
 	enum resctrl_event_id		mba_mbps_event;
+	struct cpumask			kmode_cpu_mask;
 	struct pseudo_lock_region	*plr;
 };
 
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 9235f094245c..16f68f4317ef 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -390,6 +390,32 @@ static int rdtgroup_cpus_show(struct kernfs_open_file *of,
 	return ret;
 }
 
+/*
+ * Display CPU mask of the kernel mode backing the resource group.
+ * Supports both "kmode_cpus" (bitmap format) and "kmode_cpus_list"
+ * (range list format); the output format is selected accordingly.
+ *
+ * Returns -ENOENT on error.
+ */
+static int rdtgroup_kmode_cpus_show(struct kernfs_open_file *of,
+				    struct seq_file *s, void *v)
+{
+	struct rdtgroup *rdtgrp;
+	int ret = 0;
+
+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+
+	/* Verify the group is the active kernel mode backing group */
+	if (!rdtgrp || rdtgrp != resctrl_kmode.rdtgrp)
+		ret = -ENOENT;
+	else
+		seq_printf(s, is_cpu_list(of) ? "%*pbl\n" : "%*pb\n",
+			   cpumask_pr_args(&rdtgrp->kmode_cpu_mask));
+	rdtgroup_kn_unlock(of->kn);
+
+	return ret;
+}
+
 /*
  * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
  *
@@ -2266,6 +2292,23 @@ static struct rftype res_common_files[] = {
 		.flags		= RFTYPE_FLAGS_CPUS_LIST,
 		.fflags		= RFTYPE_BASE,
 	},
+	{
+		.name		= "kmode_cpus",
+		.mode		= 0444,
+		.hidden		= true,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_kmode_cpus_show,
+		.fflags		= RFTYPE_BASE,
+	},
+	{
+		.name		= "kmode_cpus_list",
+		.mode		= 0444,
+		.hidden		= true,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_kmode_cpus_show,
+		.flags		= RFTYPE_FLAGS_CPUS_LIST,
+		.fflags		= RFTYPE_BASE,
+	},
 	{
 		.name		= "tasks",
 		.mode		= 0644,
-- 
2.43.0

[PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode

[Babu Moger]

The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows the CPU's CLOSID
and/or RMID association to change when transitioning from user mode to
kernel mode. The mode is enabled or disabled on a per-CPU basis through a
backing resource group. CPUs that come online are brought into a state
consistent with the active kernel mode, while CPUs that go offline are
removed from its CPU mask.

Integrate RESCTRL_GLOBAL_ENABLE_PER_CPU mode with CPU hotplug handling so
that CPUs automatically join the active kernel mode when they come online
and are removed from it when they go offline. Also, ensure the register
state remains in a consistent reset state across kexec boot transitions.
This ensures that kernel mode behavior remains consistent across CPU
hotplug events and kexec transitions.

Introduce resctrl_kmode_global_online_cpu() and
resctrl_kmode_global_offline_cpu(), and hook them into the existing
resctrl_online_cpu() and resctrl_offline_cpu() paths. Also update
clear_closid_rmid() to reset MSR_IA32_PQR_PLZA_ASSOC register across kexec
transitions.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog update.
    Code comment update.
    Added changes to reset the PLZA register during the bootup.

v6: Update the changelog in non-bugfix form.
    Update the code to remove rdtgroup->kmode.
    Introduce new wrapper resctrl_configure_kmode_global() to consolidate
    architecture calls.

v5: Added resctrl_kmode_offline_cpu() to disable the kernel mode
    association and update kmode_cpu_mask when an associated CPU goes
    offline.
    This addresses sashiko comments:
    https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=11
    https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=10
    Also moved the patch earlier than patch 10.
    [RESEND PATCH v4 10/15] fs/resctrl: Reset the kernel mode binding when an rdtgroup is removed
    That seemed much more clear.

v4: New patch in the series. Patch taken from:
    https://lore.kernel.org/lkml/20260611111706.1981788-5-qinyuntan@linux.alibaba.com/
    Updated the code to enable kernel mode by default whenever a CPU comes
    online when one of global-assign mode is enabled.
---
 arch/x86/kernel/cpu/resctrl/core.c |  3 ++
 fs/resctrl/rdtgroup.c              | 56 ++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 14530c11bbd7..d7faf30c2885 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -704,6 +704,9 @@ static void clear_closid_rmid(int cpu)
 	state->cur_closid = RESCTRL_RESERVED_CLOSID;
 	state->cur_rmid = RESCTRL_RESERVED_RMID;
 	wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
+
+	if (boot_cpu_has(X86_FEATURE_PLZA))
+		wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
 }
 
 static int resctrl_arch_online_cpu(unsigned int cpu)
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 16f68f4317ef..af4c929d7ed4 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -416,6 +416,17 @@ static int rdtgroup_kmode_cpus_show(struct kernfs_open_file *of,
 	return ret;
 }
 
+static void resctrl_configure_kmode_global(const struct cpumask *mask,
+					   struct rdtgroup *rdtgrp,
+					   bool enable)
+{
+	resctrl_arch_configure_kmode_global(mask, rdtgrp->closid,
+					    resctrl_kmode.ctrl == KMODE_ASSIGN,
+					    rdtgrp->mon.rmid,
+					    resctrl_kmode.mon == KMODE_ASSIGN,
+					    enable);
+}
+
 /*
  * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
  *
@@ -1125,6 +1136,48 @@ static int resctrl_kernel_mode_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/*
+ * resctrl_kmode_global_online_cpu() - Program kernel mode association for @cpu
+ * @cpu: CPU that has just been brought online
+ *
+ * When RESCTRL_GLOBAL_ENABLE_PER_CPU is active, add @cpu to the active kernel
+ * mode and program the corresponding CLOSID and/or RMID association.
+ */
+static void resctrl_kmode_global_online_cpu(unsigned int cpu)
+{
+	struct rdtgroup *rdtgrp = resctrl_kmode.rdtgrp;
+
+	if (resctrl_kmode.active != RESCTRL_GLOBAL_ENABLE_PER_CPU || !rdtgrp)
+		return;
+
+	resctrl_configure_kmode_global(cpumask_of(cpu), rdtgrp, true);
+
+	cpumask_set_cpu(cpu, &rdtgrp->kmode_cpu_mask);
+}
+
+/*
+ * resctrl_kmode_global_offline_cpu() - Clear kernel mode association for @cpu
+ * @cpu: CPU being taken offline.
+ *
+ * When RESCTRL_GLOBAL_ENABLE_PER_CPU is active and @cpu is in kmode_cpu_mask,
+ * the association is disabled on @cpu and @cpu is removed from that mask.
+ * A CPU outside the mask is left unchanged.
+ */
+static void resctrl_kmode_global_offline_cpu(unsigned int cpu)
+{
+	struct rdtgroup *rdtgrp = resctrl_kmode.rdtgrp;
+
+	if (resctrl_kmode.active != RESCTRL_GLOBAL_ENABLE_PER_CPU || !rdtgrp)
+		return;
+
+	if (!cpumask_test_cpu(cpu, &rdtgrp->kmode_cpu_mask))
+		return;
+
+	cpumask_clear_cpu(cpu, &rdtgrp->kmode_cpu_mask);
+
+	resctrl_configure_kmode_global(cpumask_of(cpu), rdtgrp, false);
+}
+
 void *rdt_kn_parent_priv(struct kernfs_node *kn)
 {
 	/*
@@ -4882,6 +4935,7 @@ void resctrl_online_cpu(unsigned int cpu)
 	mutex_lock(&rdtgroup_mutex);
 	/* The CPU is set in default rdtgroup after online. */
 	cpumask_set_cpu(cpu, &rdtgroup_default.cpu_mask);
+	resctrl_kmode_global_online_cpu(cpu);
 	mutex_unlock(&rdtgroup_mutex);
 }
 
@@ -4925,6 +4979,8 @@ void resctrl_offline_cpu(unsigned int cpu)
 		}
 	}
 
+	resctrl_kmode_global_offline_cpu(cpu);
+
 	if (!l3->mon_capable)
 		goto out_unlock;
 
-- 
2.43.0

[PATCH v7 14/19] fs/resctrl: Deactivate kernel mode when the group is removed

[Babu Moger]

The RESCTRL_GLOBAL_ENABLE_PER_CPU allows CPUs to use different CLOSID
and/or RMID association when transitioning from user mode to kernel mode.
When active, the mode is enabled or disabled on a per-CPU basis as part a
backing resource group.

When the backing group is removed, the configured kernel mode settings are
no longer valid. Deactivate the kernel mode and restore it to
RESCTRL_INHERIT_USER, the default mode in which kernel work inherits the
user task's CLOSID and/or RMID.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Removed rdtgroup_kmode_global_detach() and merged everything to
    rdtgroup_kmode_global_deactivate().
    Removed detach from rdtgroup_rmdir_ctrl().
    Updated the changelog and code comments.

v6: Rewrite changelog (not a bugfix).
    Shorten the code comments.

v5: Took care of couple of Sashiko comments on previous patch.
    https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger@amd.com?part=10
    Re-wrote the changelog.
    Renamed rdtgroup_config_kmode_reset() to rdtgroup_kmode_deactivate().

v4: Re-wrote the changelog.
    Added the call free_all_child_rdtgrp() and rmdir_all_sub()
    Simplified the code comments.

v3: New patch to handle the kernel_mode clean up.
---
 Documentation/filesystems/resctrl.rst |  3 +++
 fs/resctrl/rdtgroup.c                 | 30 +++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 033d405338c0..e57361dd342e 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -594,6 +594,9 @@ The top level of the "info" directory contains the following file(s):
 			When "mon=inherit", or monitoring is unsupported, a CTRL_MON group
 			is shown as "<CTRL_MON>/".
 
+			Removing the backing group deactivates the current kernel mode and
+			restores to "inherit_user".
+
 		When the mode is not active, the output lists the supported parameters
 		and their default values, which are used if the mode is activated
 		without new values.
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index af4c929d7ed4..97d99406de00 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1178,6 +1178,29 @@ static void resctrl_kmode_global_offline_cpu(unsigned int cpu)
 	resctrl_configure_kmode_global(cpumask_of(cpu), rdtgrp, false);
 }
 
+/*
+ * rdtgroup_kmode_global_deactivate() - Deactivate global kernel mode
+ * @rdtgrp: Resctrl group backing the active kernel mode.
+ *
+ * If @rdtgrp backs the active RESCTRL_GLOBAL_ENABLE_PER_CPU mode, disable
+ * the mode, clear kernel mode state, and restore the default
+ * RESCTRL_INHERIT_USER configuration.
+ */
+static void rdtgroup_kmode_global_deactivate(struct rdtgroup *rdtgrp)
+{
+	if (resctrl_kmode.active != RESCTRL_GLOBAL_ENABLE_PER_CPU ||
+	    resctrl_kmode.rdtgrp != rdtgrp)
+		return;
+
+	resctrl_configure_kmode_global(&rdtgrp->kmode_cpu_mask, rdtgrp, false);
+	cpumask_clear(&rdtgrp->kmode_cpu_mask);
+
+	resctrl_kmode.rdtgrp = NULL;
+	resctrl_kmode.active = RESCTRL_INHERIT_USER;
+	resctrl_kmode.ctrl = KMODE_ASSIGN;
+	resctrl_kmode.mon = KMODE_ASSIGN;
+}
+
 void *rdt_kn_parent_priv(struct kernfs_node *kn)
 {
 	/*
@@ -3214,6 +3237,7 @@ static void free_all_child_rdtgrp(struct rdtgroup *rdtgrp)
 
 	head = &rdtgrp->mon.crdtgrp_list;
 	list_for_each_entry_safe(sentry, stmp, head, mon.crdtgrp_list) {
+		rdtgroup_kmode_global_deactivate(sentry);
 		rdtgroup_unassign_cntrs(sentry);
 		free_rmid(sentry->closid, sentry->mon.rmid);
 		list_del(&sentry->mon.crdtgrp_list);
@@ -3251,6 +3275,7 @@ static void rmdir_all_sub(void)
 		cpumask_or(&rdtgroup_default.cpu_mask,
 			   &rdtgroup_default.cpu_mask, &rdtgrp->cpu_mask);
 
+		rdtgroup_kmode_global_deactivate(rdtgrp);
 		rdtgroup_unassign_cntrs(rdtgrp);
 
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP ||
@@ -3346,6 +3371,7 @@ static void resctrl_fs_teardown(void)
 		return;
 
 	rmdir_all_sub();
+	rdtgroup_kmode_global_deactivate(&rdtgroup_default);
 	rdtgroup_unassign_cntrs(&rdtgroup_default);
 	mon_put_kn_priv();
 	rdt_pseudo_lock_release();
@@ -4381,6 +4407,8 @@ static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)
 	u32 closid, rmid;
 	int cpu;
 
+	rdtgroup_kmode_global_deactivate(rdtgrp);
+
 	/* Give any tasks back to the parent group */
 	rdt_move_group_tasks(rdtgrp, prdtgrp, tmpmask);
 
@@ -4431,6 +4459,8 @@ static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)
 	u32 closid, rmid;
 	int cpu;
 
+	rdtgroup_kmode_global_deactivate(rdtgrp);
+
 	/* Give any tasks back to the default group */
 	rdt_move_group_tasks(rdtgrp, &rdtgroup_default, tmpmask);
 
-- 
2.43.0

[PATCH v7 15/19] fs/resctrl: Extend info/kernel_mode to configure the kernel mode

[Babu Moger]

Kernel mode support allows the CPU's CLOSID and/or RMID associations to
change during the transition from user mode to kernel mode. The resctrl
file info/kernel_mode lists the current kernel mode configuration.

Extend info/kernel_mode with write support so user space can configure the
active kernel mode. Validate kernel mode configuration before activation.
Restrict backing groups to shareable mode so the kernel can use the group's
CLOSID and/or RMID, which are shared with the group's tasks.

Document the interface in Documentation/filesystems/resctrl.rst.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Refactored rdtgroup_kmode_global_activate and rdtgroup_kmode_global_deactivate().
    Removed resctrl_kmode_switch().
    Refactored resctrl_kmode_parse_group().
    Handled the "group=" as the last parameter.

v6: Use flattened resctrl_kmode and global_enable_per_cpu. Omitted
    parameters keep the values shown by reading the file. Drop the
    ctrl=inherit;mon=inherit no-op. Parse options with strsep(), as the
    other resctrl writers do, and split validation out of the write path.
    https://lore.kernel.org/lkml/3bc39dd7-b7b0-4203-a493-0bd872fdc644@intel.com/

v5: Adapt to the resctrl_kcfg caps and active split. Accept
    assign_global_enable_per_cpu:ctrl=;mon=;group=<ctrl>/<mon>/ writes.
    Treat both ctrl=inherit and mon=inherit as a no-op. Use group=// for
    the default control group. Show associated kmode files through
    resctrl_hidden_files_set_visible().
    Added kmode check in rdtgroup_locksetup_enter(), rdtgroup_mode_write()
    and rdtgroup_rename().
    Simplify kernel-doc for write and activate helpers.

v4: Validate input before tearing down the active association so failures
    retain the previous association. Allow assign_mon on CTRL_MON and MON
    groups; require a control group when control assignment is inherit.

v3: New patch to make info/kernel_mode writable.
---
 Documentation/filesystems/resctrl.rst |  40 ++++
 fs/resctrl/rdtgroup.c                 | 274 +++++++++++++++++++++++++-
 2 files changed, 313 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index e57361dd342e..1a1877222624 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -608,6 +608,46 @@ The top level of the "info" directory contains the following file(s):
 		[inherit_user]
 		global_enable_per_cpu:ctrl=assign;mon=assign;group=//
 
+	Writes use the same line format as reads, without square brackets and
+	with a trailing newline. Errors are reported in "info/last_cmd_status".
+
+	"inherit_user":
+		Written without parameters.
+
+	"global_enable_per_cpu":
+		"ctrl=", "mon=", and "group=" are optional.
+
+		An omitted "ctrl=", "mon=" and "group=" keeps the default values.
+
+		If "mon=inherit", or monitoring is unsupported, "group=" must
+		reference a CTRL_MON group.
+
+		The "ctrl=inherit" together with "mon=inherit" is rejected.
+
+		A MON group requires "mon=assign".
+
+		"group=" must be the last parameter. The group must be in shareable mode.
+
+		Selecting a new kernel mode, parameter, or group replaces the active
+		mode and updates "kmode_cpus" to include all online CPUs.
+
+		Empty values are rejected.
+
+		Example::
+
+			# mkdir ctrl1
+			# echo "global_enable_per_cpu:ctrl=assign;mon=assign;group=ctrl1//" \
+				> info/kernel_mode
+
+			# cat info/kernel_mode
+			inherit_user
+			[global_enable_per_cpu:ctrl=assign;mon=assign;group=ctrl1//]
+
+			# echo "inherit_user" > info/kernel_mode
+			# cat info/kernel_mode
+			[inherit_user]
+			global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
 Resource alloc and monitor groups
 =================================
 
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 97d99406de00..872755532fb4 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1201,6 +1201,277 @@ static void rdtgroup_kmode_global_deactivate(struct rdtgroup *rdtgrp)
 	resctrl_kmode.mon = KMODE_ASSIGN;
 }
 
+/* Program @rdtgrp on every online CPU and show its kmode_cpus files. */
+static void rdtgroup_kmode_global_activate(struct rdtgroup *rdtgrp,
+					   enum resctrl_kernel_mode mode,
+					   enum kmode_state ctrl,
+					   enum kmode_state mon)
+{
+	resctrl_kmode.active = mode;
+	resctrl_kmode.ctrl = ctrl;
+	resctrl_kmode.mon = mon;
+	resctrl_kmode.rdtgrp = rdtgrp;
+
+	cpumask_copy(&rdtgrp->kmode_cpu_mask, cpu_online_mask);
+	resctrl_configure_kmode_global(&rdtgrp->kmode_cpu_mask, rdtgrp, true);
+}
+
+static int resctrl_kmode_parse_state(char *val, enum kmode_state *state)
+{
+	if (!strcmp(val, "assign")) {
+		*state = KMODE_ASSIGN;
+		return 0;
+	}
+	if (!strcmp(val, "inherit")) {
+		*state = KMODE_INHERIT;
+		return 0;
+	}
+	return -EINVAL;
+}
+
+/*
+ * group is "<CTRL_MON>/" or "<CTRL_MON>/<MON>/". The MON component is
+ * optional: mon=inherit and systems without monitoring do not require it.
+ * An empty name selects the default CTRL_MON group. Names are not trimmed;
+ * "/" cannot appear in a resource group name.
+ */
+static int resctrl_kmode_parse_group(char *group, struct rdtgroup **rdtgrp)
+{
+	struct rdtgroup *rdtg, *parent = &rdtgroup_default;
+	char *mon = NULL, *slash;
+	size_t len;
+
+	len = strlen(group);
+	if (!len || group[len - 1] != '/') {
+		rdt_last_cmd_puts("Group must be <CTRL_MON>/ or <CTRL_MON>/<MON>/\n");
+		return -EINVAL;
+	}
+	group[len - 1] = '\0';
+
+	slash = strchr(group, '/');
+	if (slash) {
+		*slash = '\0';
+		mon = slash + 1;
+		if (strchr(mon, '/')) {
+			rdt_last_cmd_puts("Group must be <CTRL_MON>/ or <CTRL_MON>/<MON>/\n");
+			return -EINVAL;
+		}
+	}
+
+	if (group[0] != '\0') {
+		parent = NULL;
+		list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list) {
+			if (!strcmp(rdt_kn_name(rdtg->kn), group)) {
+				parent = rdtg;
+				break;
+			}
+		}
+	}
+	if (!parent)
+		goto not_found;
+
+	if (!mon || mon[0] == '\0') {
+		*rdtgrp = parent;
+		return 0;
+	}
+
+	list_for_each_entry(rdtg, &parent->mon.crdtgrp_list, mon.crdtgrp_list) {
+		if (!strcmp(rdt_kn_name(rdtg->kn), mon)) {
+			*rdtgrp = rdtg;
+			return 0;
+		}
+	}
+
+not_found:
+	rdt_last_cmd_puts("Group not found\n");
+	return -EINVAL;
+}
+
+static int resctrl_kmode_parse_parameters(char *params, enum kmode_state *ctrl,
+					  enum kmode_state *mon,
+					  struct rdtgroup **rdtgrp)
+{
+	char *key, *tok;
+
+	while ((tok = strsep(&params, ";"))) {
+		key = strsep(&tok, "=");
+		if (!tok) {
+			if (strim(key)[0] == '\0')
+				continue;
+			rdt_last_cmd_puts("Missing '=' in kernel mode parameter\n");
+			return -EINVAL;
+		}
+		key = strim(key);
+		if (key[0] == '\0') {
+			rdt_last_cmd_puts("Empty kernel mode parameters\n");
+			return -EINVAL;
+		}
+
+		/*
+		 * group= is the last parameter and may contain ';'.
+		 * Restore it if stripped.
+		 */
+		if (!strcmp(key, "group") && params)
+			params[-1] = ';';
+
+		tok = strim(tok);
+		if (tok[0] == '\0') {
+			rdt_last_cmd_puts("Empty kernel mode parameters\n");
+			return -EINVAL;
+		}
+
+		if (!strcmp(key, "group"))
+			return resctrl_kmode_parse_group(tok, rdtgrp);
+
+		if (!strcmp(key, "ctrl")) {
+			if (!resctrl_arch_alloc_capable()) {
+				rdt_last_cmd_puts("ctrl= is not supported\n");
+				return -EINVAL;
+			}
+			if (resctrl_kmode_parse_state(tok, ctrl)) {
+				rdt_last_cmd_puts("Invalid ctrl= parameter\n");
+				return -EINVAL;
+			}
+		} else if (!strcmp(key, "mon")) {
+			if (!resctrl_arch_mon_capable()) {
+				rdt_last_cmd_puts("mon= is not supported\n");
+				return -EINVAL;
+			}
+			if (resctrl_kmode_parse_state(tok, mon)) {
+				rdt_last_cmd_puts("Invalid mon= parameter\n");
+				return -EINVAL;
+			}
+		} else {
+			rdt_last_cmd_puts("Unsupported kernel mode parameter\n");
+			return -EINVAL;
+		}
+	}
+	return 0;
+}
+
+static int resctrl_kmode_global_validate(struct rdtgroup *rdtgrp,
+					 enum kmode_state ctrl,
+					 enum kmode_state mon)
+{
+	if (ctrl == KMODE_ASSIGN && !resctrl_arch_alloc_capable()) {
+		rdt_last_cmd_puts("ctrl=assign is not supported\n");
+		return -EINVAL;
+	}
+	if (mon == KMODE_ASSIGN && !resctrl_arch_mon_capable()) {
+		rdt_last_cmd_puts("mon=assign is not supported\n");
+		return -EINVAL;
+	}
+	if (mon == KMODE_INHERIT && rdtgrp->type == RDTMON_GROUP) {
+		rdt_last_cmd_puts("A monitor group requires mon=assign parameter\n");
+		return -EINVAL;
+	}
+	if (rdtgrp->mode != RDT_MODE_SHAREABLE) {
+		rdt_last_cmd_puts("The group is not in shareable mode\n");
+		return -EINVAL;
+	}
+	if (rdtgrp->type == RDTMON_GROUP &&
+	    rdtgrp->mon.parent->mode != RDT_MODE_SHAREABLE) {
+		rdt_last_cmd_puts("The parent group is not in shareable mode\n");
+		return -EINVAL;
+	}
+	if (ctrl == KMODE_INHERIT && mon == KMODE_INHERIT) {
+		rdt_last_cmd_puts("ctrl=inherit and mon=inherit is not supported\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * resctrl_kernel_mode_write() - Set the active kernel mode
+ * @of:		kernfs open file
+ * @buf:	Kernel mode line written by user space
+ * @nbytes:	length of @buf
+ * @off:	unused
+ *
+ * Return: @nbytes on success, negative errno on error.
+ */
+static ssize_t resctrl_kernel_mode_write(struct kernfs_open_file *of,
+					 char *buf, size_t nbytes, loff_t off)
+{
+	enum resctrl_kernel_mode mode;
+	enum kmode_state ctrl, mon;
+	char *mode_str, *params;
+	struct rdtgroup *rdtgrp;
+	int ret = 0;
+
+	if (!info_kn_lock(of->kn))
+		return -ENOENT;
+
+	rdt_last_cmd_clear();
+
+	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
+		rdt_last_cmd_puts("kernel_mode: Invalid input\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	buf[nbytes - 1] = '\0';
+
+	mode_str = strsep(&buf, ":");
+	mode_str = strim(mode_str);
+	params = buf;
+
+	for (mode = 0; mode < RESCTRL_NUM_KERNEL_MODES; mode++) {
+		if (!strcmp(mode_str, resctrl_kmode_str[mode]))
+			break;
+	}
+	if (mode == RESCTRL_NUM_KERNEL_MODES) {
+		rdt_last_cmd_puts("Unknown kernel mode\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	if (!test_bit(mode, resctrl_kmode.supported)) {
+		rdt_last_cmd_puts("Kernel mode not available\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	if (mode == RESCTRL_INHERIT_USER) {
+		/* "inherit_user" does not support parameters */
+		if (params && strim(params)[0] != '\0') {
+			rdt_last_cmd_puts("Unsupported parameter\n");
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+		if (resctrl_kmode.active == RESCTRL_GLOBAL_ENABLE_PER_CPU)
+			rdtgroup_kmode_global_deactivate(resctrl_kmode.rdtgrp);
+		goto out_unlock;
+	}
+
+	/* Initialize the default values */
+	ctrl = resctrl_arch_alloc_capable() ? KMODE_ASSIGN : KMODE_INHERIT;
+	mon = resctrl_arch_mon_capable() ? KMODE_ASSIGN : KMODE_INHERIT;
+	rdtgrp = &rdtgroup_default;
+
+	if (params) {
+		ret = resctrl_kmode_parse_parameters(params, &ctrl, &mon, &rdtgrp);
+		if (ret)
+			goto out_unlock;
+	}
+
+	ret = resctrl_kmode_global_validate(rdtgrp, ctrl, mon);
+	if (ret)
+		goto out_unlock;
+
+	if (resctrl_kmode.active == mode && resctrl_kmode.rdtgrp == rdtgrp &&
+	    resctrl_kmode.ctrl == ctrl && resctrl_kmode.mon == mon)
+		goto out_unlock;
+
+	if (resctrl_kmode.active == RESCTRL_GLOBAL_ENABLE_PER_CPU)
+		rdtgroup_kmode_global_deactivate(resctrl_kmode.rdtgrp);
+
+	rdtgroup_kmode_global_activate(rdtgrp, mode, ctrl, mon);
+
+out_unlock:
+	info_kn_unlock(of->kn);
+	return ret ?: nbytes;
+}
+
 void *rdt_kn_parent_priv(struct kernfs_node *kn)
 {
 	/*
@@ -2188,9 +2459,10 @@ static struct rftype res_common_files[] = {
 	},
 	{
 		.name		= "kernel_mode",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= resctrl_kernel_mode_show,
+		.write		= resctrl_kernel_mode_write,
 		.fflags		= RFTYPE_TOP_INFO,
 	},
 	{
-- 
2.43.0

[PATCH v7 16/19] fs/resctrl: Reject mode changes for groups backing kernel mode

[Babu Moger]

The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode uses the CLOSID and/or
RMID of a backing group when transitioning from user mode to kernel
mode. Since the CLOSID and RMID are shared with the group's tasks, the
backing group must remain shareable while kernel mode is active.

Reject mode changes for a group backing the active kernel mode and for
the parent of a backing monitor group.

Also reject reparenting a backing monitor group, as it would change the
monitor group's parent and implicitly modify the active kernel mode
configuration.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog update.
    Moved the patch after the kernel_mode modify interface.
    Added code comments to justify reparenting.

v6: New patch to fix sashiko feedback.
    https://sashiko.dev/#/patchset/cover.1787772750.git.babu.moger%40amd.com
---
 fs/resctrl/rdtgroup.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 872755532fb4..1ad169e714a4 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1994,6 +1994,27 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 	return true;
 }
 
+/* True when @rdtgrp or one of its monitor groups backs the kernel mode. */
+static bool rdtgroup_has_kmode_global(struct rdtgroup *rdtgrp)
+{
+	struct rdtgroup *child;
+
+	if (!resctrl_kmode.rdtgrp)
+		return false;
+
+	if (resctrl_kmode.rdtgrp == rdtgrp)
+		return true;
+
+	if (rdtgrp->type != RDTCTRL_GROUP)
+		return false;
+
+	list_for_each_entry(child, &rdtgrp->mon.crdtgrp_list, mon.crdtgrp_list)
+		if (child == resctrl_kmode.rdtgrp)
+			return true;
+
+	return false;
+}
+
 /*
  * rdtgroup_mode_write - Modify the resource group's mode
  */
@@ -2034,6 +2055,12 @@ static ssize_t rdtgroup_mode_write(struct kernfs_open_file *of,
 		goto out;
 	}
 
+	if (rdtgroup_has_kmode_global(rdtgrp)) {
+		rdt_last_cmd_puts("Cannot change mode for kernel mode backing group\n");
+		ret = -EINVAL;
+		goto out;
+	}
+
 	if (!strcmp(buf, "shareable")) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			ret = rdtgroup_locksetup_exit(rdtgrp);
@@ -4922,6 +4949,19 @@ static int rdtgroup_rename(struct kernfs_node *kn,
 		goto out;
 	}
 
+	/*
+	 * Changing the parent of a group backing the active kernel mode changes
+	 * the effective CLOSID and/or RMID configuration applied to kernel work.
+	 * Reject such moves and require kernel mode to be reconfigured explicitly
+	 * instead.
+	 */
+	if (rdtgrp == resctrl_kmode.rdtgrp &&
+	    rdtgrp->mon.parent != new_prdtgrp) {
+		rdt_last_cmd_puts("Cannot move a group that backs kernel mode\n");
+		ret = -EPERM;
+		goto out;
+	}
+
 	/*
 	 * Allocate the cpumask for use in mongrp_reparent() to avoid the
 	 * possibility of failing to allocate it after kernfs_rename() has
-- 
2.43.0

[PATCH v7 17/19] fs/resctrl: Control visibility of global per-CPU kernel mode group files

[Babu Moger]

The kmode_cpus and kmode_cpus_list files are specific to the global per-CPU
kernel mode. These files are created in a hidden state when an rdtgroup is
created.

Introduce resctrl_kmode_global_files_show() to control the visibility of
these files for individual resource groups. Show the files only for the
group backing the active global per-CPU kernel mode and hide them for all
other groups.

Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the prototype rdtgroup_kmode_global_files_show().
    Moved the patch after both rdtgroup_kmode_global_activate() and
    rdtgroup_kmode_global_deactivate() is Introduced, so the change is
    symmetrical.
    Updated the changelog.

v6: Helper named for the global per-CPU files instead of unhiding every
    hidden file on the group.
    https://lore.kernel.org/lkml/34aa4824-810b-4c81-8188-f2f37e1dc277@intel.com/
---
 fs/resctrl/rdtgroup.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 1ad169e714a4..86bd6a66812b 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1136,6 +1136,24 @@ static int resctrl_kernel_mode_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/* Show or hide kmode_cpus and kmode_cpus_list on @kn. */
+static void resctrl_kmode_global_files_show(struct rdtgroup *rdtgrp, bool show)
+{
+	struct kernfs_node *kn_file;
+
+	kn_file = kernfs_find_and_get(rdtgrp->kn, "kmode_cpus");
+	if (kn_file) {
+		kernfs_show(kn_file, show);
+		kernfs_put(kn_file);
+	}
+
+	kn_file = kernfs_find_and_get(rdtgrp->kn, "kmode_cpus_list");
+	if (kn_file) {
+		kernfs_show(kn_file, show);
+		kernfs_put(kn_file);
+	}
+}
+
 /*
  * resctrl_kmode_global_online_cpu() - Program kernel mode association for @cpu
  * @cpu: CPU that has just been brought online
@@ -1194,6 +1212,7 @@ static void rdtgroup_kmode_global_deactivate(struct rdtgroup *rdtgrp)
 
 	resctrl_configure_kmode_global(&rdtgrp->kmode_cpu_mask, rdtgrp, false);
 	cpumask_clear(&rdtgrp->kmode_cpu_mask);
+	resctrl_kmode_global_files_show(rdtgrp, false);
 
 	resctrl_kmode.rdtgrp = NULL;
 	resctrl_kmode.active = RESCTRL_INHERIT_USER;
@@ -1214,6 +1233,7 @@ static void rdtgroup_kmode_global_activate(struct rdtgroup *rdtgrp,
 
 	cpumask_copy(&rdtgrp->kmode_cpu_mask, cpu_online_mask);
 	resctrl_configure_kmode_global(&rdtgrp->kmode_cpu_mask, rdtgrp, true);
+	resctrl_kmode_global_files_show(rdtgrp, true);
 }
 
 static int resctrl_kmode_parse_state(char *val, enum kmode_state *state)
-- 
2.43.0

[PATCH v7 18/19] fs/resctrl: Allow userspace to write kmode_cpus/kmode_cpus_list

[Babu Moger]

The global per-CPU kernel mode allows CPUs to use different CLOSID and/or
RMID associations when transitioning from user mode to kernel mode. The
mode is enabled on a per-CPU basis, and kmode_cpus and kmode_cpus_list
expose the CPUs used by active kernel mode.

Add write support so userspace can update the CPUs used by the active
kernel mode.

Document the interface in Documentation/filesystems/resctrl.rst.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog update.
    Last_cmd_status message update to keep the messages consistant.
    Minor changes to handle resctrl_kmode data structure changes.

v6: Use flattened resctrl_kmode and global_enable_per_cpu.
    Fix the sashiko reported issue.
    https://sashiko.dev/#/patchset/cover.1787772750.git.babu.moger%40amd.com?part=15
    Update the changelog.

v5: Adapt to the resctrl_kcfg caps and active split. Use
    resctrl_kcfg.active.ctrl_mode and mon_mode in kmode_cpus_write().
    Use full ctrl=/mon= syntax in the kmode_cpus example.

v4: Empty masks are now allowed and updated masks are in rdtgroup->kmode_cpu_mask.
    Updated the changelog.

v3: New patch to add "kmode_cpus" and "kmode_cpus_list" to support
    kernel_modes.
---
 Documentation/filesystems/resctrl.rst |  30 +++++++
 fs/resctrl/rdtgroup.c                 | 114 +++++++++++++++++++++++++-
 2 files changed, 142 insertions(+), 2 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 1a1877222624..2f0f6f01766f 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -716,6 +716,36 @@ All groups contain the following files:
 "cpus_list":
 	Just like "cpus", only using ranges of CPUs instead of bitmasks.
 
+"kmode_cpus":
+	Accessible only within the resource group backing the active global
+	per-CPU kernel mode (see "info/kernel_mode").
+
+	Bitmask of the logical CPUs used by resource group backing the kernel
+	mode. When kernel mode is activated through info/kernel_mode, every
+	currently online CPU is included. CPUs that come online later are
+	automatically added to the group.
+
+	Writing a mask replaces the CPU mask. The mode is enabled on CPUs
+	added to the mask and disabled on CPUs removed from it. An empty mask
+	disables the mode on all currently online CPUs. CPUs that come online
+	later are still automatically added to the group. The mask must
+	contain only online CPUs; masks naming offline CPUs are rejected.
+	Errors are reported in "info/last_cmd_status".
+	Example::
+
+	  # mkdir ctrl1
+	  # echo "global_enable_per_cpu:ctrl=assign;mon=assign;group=ctrl1//" \
+	        > info/kernel_mode
+	  # echo 0-3 > ctrl1/kmode_cpus_list
+	  # cat ctrl1/kmode_cpus
+	  f
+	  # cat ctrl1/kmode_cpus_list
+	  0-3
+
+"kmode_cpus_list":
+	Just like "kmode_cpus", only using ranges of CPUs instead of bitmasks.
+	Writable with the same semantics and restrictions as "kmode_cpus".
+
 
 When control is enabled all CTRL_MON groups will also contain:
 
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 86bd6a66812b..0d67e7086e25 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -427,6 +427,114 @@ static void resctrl_configure_kmode_global(const struct cpumask *mask,
 					    enable);
 }
 
+/**
+ * kmode_cpus_write() - Update @rdtgrp's kmode_cpu_mask from @newmask
+ * @rdtgrp:	Resctrl group whose kmode_cpu_mask is being updated.
+ * @newmask:	New CPUs for @rdtgrp's kernel mode association.
+ * @tmpmask:	Caller-allocated scratch cpumask used to compute the
+ *		incremental enable/disable deltas.
+ *
+ * Enable the kernel mode on CPUs added to the mask and disable it on
+ * CPUs removed from the mask.
+ */
+static void kmode_cpus_write(struct rdtgroup *rdtgrp,
+			     cpumask_var_t newmask, cpumask_var_t tmpmask)
+{
+	/* CPUs dropped from this group: old & ~newmask. */
+	cpumask_andnot(tmpmask, &rdtgrp->kmode_cpu_mask, newmask);
+	if (!cpumask_empty(tmpmask))
+		resctrl_configure_kmode_global(tmpmask, rdtgrp, false);
+
+	/* CPUs newly added: newmask & ~old. */
+	cpumask_andnot(tmpmask, newmask, &rdtgrp->kmode_cpu_mask);
+	if (!cpumask_empty(tmpmask))
+		resctrl_configure_kmode_global(tmpmask, rdtgrp, true);
+
+	cpumask_copy(&rdtgrp->kmode_cpu_mask, newmask);
+}
+
+/**
+ * rdtgroup_kmode_cpus_write() - Write kmode_cpus or kmode_cpus_list
+ * @of: kernfs open file
+ * @buf: CPU bitmask or list from userspace
+ * @nbytes: length of @buf
+ * @off: unused
+ *
+ * Update the CPU mask of the group that backs the kernel mode.
+ *
+ * Return: @nbytes on success, negative errno on error.
+ */
+static ssize_t rdtgroup_kmode_cpus_write(struct kernfs_open_file *of,
+					 char *buf, size_t nbytes, loff_t off)
+{
+	cpumask_var_t tmpmask = CPUMASK_VAR_NULL, newmask = CPUMASK_VAR_NULL;
+	struct rdtgroup *rdtgrp;
+	int ret;
+
+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+	if (!rdtgrp) {
+		ret = -ENOENT;
+		goto out_unlock;
+	}
+
+	if (!buf) {
+		rdt_last_cmd_printf("%s: Invalid input\n",
+				    is_cpu_list(of) ? "kmode_cpus_list" :
+						      "kmode_cpus");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL) ||
+	    !zalloc_cpumask_var(&newmask, GFP_KERNEL)) {
+		rdt_last_cmd_printf("%s: Kernel allocation failure\n",
+				    is_cpu_list(of) ? "kmode_cpus_list" :
+						      "kmode_cpus");
+		ret = -ENOMEM;
+		goto out_free;
+	}
+
+	if (resctrl_kmode.active != RESCTRL_GLOBAL_ENABLE_PER_CPU) {
+		rdt_last_cmd_puts("No active global per-cpu kernel mode\n");
+		ret = -EINVAL;
+		goto out_free;
+	}
+
+	if (resctrl_kmode.rdtgrp != rdtgrp) {
+		rdt_last_cmd_puts("Group is not backing the global per-cpu kernel mode\n");
+		ret = -EINVAL;
+		goto out_free;
+	}
+
+	if (is_cpu_list(of))
+		ret = cpulist_parse(buf, newmask);
+	else
+		ret = cpumask_parse(buf, newmask);
+
+	if (ret) {
+		rdt_last_cmd_puts("Bad CPU list/mask\n");
+		goto out_free;
+	}
+
+	/* kernel-mode association is only programmed on online CPUs. */
+	cpumask_andnot(tmpmask, newmask, cpu_online_mask);
+	if (!cpumask_empty(tmpmask)) {
+		rdt_last_cmd_puts("Can only assign online CPUs\n");
+		ret = -EINVAL;
+		goto out_free;
+	}
+
+	kmode_cpus_write(rdtgrp, newmask, tmpmask);
+
+out_free:
+	free_cpumask_var(tmpmask);
+	free_cpumask_var(newmask);
+out_unlock:
+	rdtgroup_kn_unlock(of->kn);
+
+	return ret ?: nbytes;
+}
+
 /*
  * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
  *
@@ -2689,17 +2797,19 @@ static struct rftype res_common_files[] = {
 	},
 	{
 		.name		= "kmode_cpus",
-		.mode		= 0444,
+		.mode		= 0644,
 		.hidden		= true,
 		.kf_ops		= &rdtgroup_kf_single_ops,
+		.write		= rdtgroup_kmode_cpus_write,
 		.seq_show	= rdtgroup_kmode_cpus_show,
 		.fflags		= RFTYPE_BASE,
 	},
 	{
 		.name		= "kmode_cpus_list",
-		.mode		= 0444,
+		.mode		= 0644,
 		.hidden		= true,
 		.kf_ops		= &rdtgroup_kf_single_ops,
+		.write		= rdtgroup_kmode_cpus_write,
 		.seq_show	= rdtgroup_kmode_cpus_show,
 		.flags		= RFTYPE_FLAGS_CPUS_LIST,
 		.fflags		= RFTYPE_BASE,
-- 
2.43.0

[PATCH v7 19/19] fs/resctrl: Add kernel mode documentation and examples

[Babu Moger]

Kernel mode configuration is exposed through info/kernel_mode,
kmode_cpus, and kmode_cpus_list.

Provide examples showing how to query supported modes, configure kernel
modes, and manage per-CPU kernel mode associations.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Changelog update.
    Documentation update.

v6: Renamed assign_global_enable_per_cpu to global_enable_per_cpu in the
    examples. Replaced "policy" with "mode" throughout.

v5: Update the example for the collapsed kernel mode model:
    inherit_user and assign_global_enable_per_cpu with ctrl=/mon=
    association options, and <CTRL_MON group>/<MON group>/ paths.

v4: New patch. Added examples in the documentation.
---
 Documentation/filesystems/resctrl.rst | 96 +++++++++++++++++++++++++++
 1 file changed, 96 insertions(+)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 2f0f6f01766f..d161c51fe6e8 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -1933,6 +1933,102 @@ View the llc occupancy snapshot::
   11234000
 
 
+Examples on working with kernel_mode
+====================================
+The kernel-mode interface spans three files:
+
+- ``info/kernel_mode`` - select the kernel mode, configuration and backing group
+- ``kmode_cpus`` / ``kmode_cpus_list`` - view or adjust the CPU scope of the
+  active mode (visible only on a backing group)
+
+The walkthrough below covers a typical workflow on a platform with PLZA
+(Privilege Level Zero Association) support:
+
+a. Discover supported kernel modes
+::
+
+  # mount -t resctrl resctrl /sys/fs/resctrl/
+
+  # cat /sys/fs/resctrl/info/kernel_mode
+  [inherit_user]
+  global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
+``inherit_user`` is the active mode. Kernel work inherits the resource
+allocation and monitoring of the current user task. The second line
+shows that ``global_enable_per_cpu`` is available. In later reads, the
+square brackets move to the active mode.
+
+b. Create a control group "ctrl1". Activate global_enable_per_cpu mode on the
+control group.
+::
+
+  # mkdir /sys/fs/resctrl/ctrl1
+  # echo "global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/" > \
+        /sys/fs/resctrl/info/kernel_mode
+
+  # cat /sys/fs/resctrl/info/kernel_mode
+  inherit_user
+  [global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/]
+
+``ctrl=assign`` uses ctrl1's allocation.
+
+``mon=inherit`` uses the monitoring of the user task.
+
+The group path uses the ``<CTRL_MON>/`` syntax; a control group alone is
+written as ``ctrl1/``.
+
+When the mode is active:
+
+- ``ctrl1/kmode_cpus`` and ``ctrl1/kmode_cpus_list`` become visible.
+- All currently online CPUs are added initially.
+- CPUs that come online later are added automatically.
+
+c. Narrow the CPU scope to 0-3
+::
+
+  # echo 0-3 > /sys/fs/resctrl/ctrl1/kmode_cpus_list
+  # cat /sys/fs/resctrl/ctrl1/kmode_cpus
+  f
+  # cat /sys/fs/resctrl/ctrl1/kmode_cpus_list
+  0-3
+
+The write replaces the CPU mask:
+
+- The mode is enabled on CPUs added to the mask.
+- The mode is disabled on CPUs removed from the mask.
+- Offline CPUs in the mask are rejected.
+- An empty mask disables the mode on all currently online CPUs, but later
+  hotplug CPUs are still added automatically.
+
+d. Create a monitor group and activate the mode on the monitor group.
+::
+
+  # mkdir /sys/fs/resctrl/ctrl1/mon_groups/mon1
+  # echo "global_enable_per_cpu:ctrl=assign;mon=assign;group=ctrl1/mon1/" > \
+        /sys/fs/resctrl/info/kernel_mode
+
+  # cat /sys/fs/resctrl/info/kernel_mode
+  inherit_user
+  [global_enable_per_cpu:ctrl=assign;mon=assign;group=ctrl1/mon1/]
+
+Kernel mode work uses ctrl1's allocation (``ctrl=assign``) and mon1's monitoring
+(``mon=assign``).
+
+- Resets the CPU scope to all currently online CPUs.
+- Hides ``kmode_cpus`` and ``kmode_cpus_list`` on the previous group.
+- Shows them under ``ctrl1/mon_groups/mon1/``.
+
+e. Return to inherit_user
+::
+
+  # echo "inherit_user" > /sys/fs/resctrl/info/kernel_mode
+  # cat /sys/fs/resctrl/info/kernel_mode
+  [inherit_user]
+  global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
+Writing ``inherit_user`` clears the active kernel mode and hides ``kmode_cpus``
+and ``kmode_cpus_list`` because no group is backing the kernel mode.
+
 Examples on working with mbm_assign_mode
 ========================================
 
-- 
2.43.0