[PATCH v7 05/19] arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU kernel mode

[Babu Moger <babu.moger@amd.com>]

RESCTRL_GLOBAL_ENABLE_PER_CPU enables a CPU to use a different CLOSID
and/or RMID association when executing in kernel mode than those used in
user mode. On x86, this mode corresponds to PLZA (Privilege Level Zero
Association).

PLZA is programmed per-CPU through MSR_IA32_PQR_PLZA_ASSOC, which contains
the kernel mode CLOSID and RMID, an enable bit for each, and a bit that
enables PLZA on the CPU. All fields except the PLZA enable bit must be
programmed identically on every CPU in a domain.

The mode is enabled independently on each CPU, so the filesystem must
program hardware when the mode is activated, when CPUs are added or
removed, and when it is deactivated. CLOSID and RMID can be programmed
independently, so the architecture callback provides separate enable
options for each.

Add resctrl_arch_configure_kmode_global() with an x86 PLZA implementation
and an empty MPAM stub for ARM systems that do not support this mode.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the changelog with consistant terms.
    Update changelog based on comments from Reinette with minor changes.

v6: Rename to resctrl_arch_configure_kmode_global(). Rewrite changelog
    for RESCTRL_GLOBAL_ENABLE_PER_CPU only (not a bugfix, not a code
    restatement). Add arm to the subject. Document parameters first;
    closid/rmid are always programmed. Split fs/arch contract from
    call scenarios and name the mode explicitly.

v5: Fix kernel-doc for resctrl_arch_configure_kmode(), wire assign_ctrl
    through to PLZA closid_en, and update the changelog.
    Add calling context in API doc.

v4: Added assign_mon parameter in resctrl_arch_configure_kmode() to program the RMID
    as discussed in below.
    https://lore.kernel.org/lkml/20260605100642.1103628-1-qinyuntan@linux.alibaba.com/
    Reintroduce independent monitoring assignment via assign_mon after removing
    the task-based PLZA approach in v3.
    Changed cpumask type to "const struct cpumask *cpu_mask".
    Added MPAM stub to avoid any linking issues when resctrl_arch_configure_kmode()
    is called from FS layer. Thanks to Qinyun.
    Re-wrote the changelog to be generic.
    Updated code comments.
v3: Removed task based PLZA implementation so related changes are removed.
    Per-task RMID handling is replaced by group type selection in the fs layer.
    Updated the change log with details.
    Removed resctrl_arch_set_kmode() as arch only provides the modes supported.
    It is FS which decided which mode to apply.
v2: Updated the commit message to include the sequence of steps to enable PLZA.
    Added mode code comments for clarity.
    Added kmode to function names to be generic.
---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 +++++++++++++++++++++++
 drivers/resctrl/mpam_resctrl.c            |  7 +++++
 include/linux/resctrl.h                   | 28 +++++++++++++++++
 3 files changed, 73 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
index 62044489b052..79b6fdcbc553 100644
--- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
+++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
@@ -137,3 +137,41 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
 
 	return 0;
 }
+
+static void resctrl_kmode_set_one_amd(void *arg)
+{
+	union msr_pqr_plza_assoc *plza = arg;
+
+	wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, plza->full);
+}
+
+/*
+ * resctrl_arch_configure_kmode_global - Program Privilege Level Zero
+ *					 Association (PLZA).
+ * @cpu_mask:	CPUs whose MSR_IA32_PQR_PLZA_ASSOC should be updated.
+ * @closid:	CLOSID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_ctrl is false.
+ * @assign_ctrl: Whether PLZA should program the kernel mode CLOSID.
+ * @rmid:	RMID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_mon is false.
+ * @assign_mon:	Whether PLZA should program the kernel mode RMID.
+ * @enable:	Whether PLZA should program the kernel mode.
+ *
+ * Program PLZA on @cpu_mask for %RESCTRL_GLOBAL_ENABLE_PER_CPU. See
+ * union msr_pqr_plza_assoc for the hardware field meanings.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+	union msr_pqr_plza_assoc plza = { 0 };
+
+	plza.split.rmid = rmid;
+	plza.split.rmid_en = assign_mon;
+	plza.split.closid = closid;
+	plza.split.closid_en = assign_ctrl;
+	plza.split.plza_en = enable;
+
+	on_each_cpu_mask(cpu_mask, resctrl_kmode_set_one_amd, &plza, 1);
+}
diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
index 5468367776e2..548bfb336be5 100644
--- a/drivers/resctrl/mpam_resctrl.c
+++ b/drivers/resctrl/mpam_resctrl.c
@@ -139,6 +139,13 @@ bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
 	return false;
 }
 
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+}
+
 void resctrl_arch_pre_mount(void)
 {
 }
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 6f3e57ec4aa3..e43415ae6e0e 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -741,6 +741,34 @@ enum resctrl_kernel_mode {
 
 #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 
+/**
+ * resctrl_arch_configure_kmode_global() - Program RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *
+ * @cpu_mask:	Online CPUs on which to update the CLOSID and/or RMID.
+ * @closid:	CLOSID of the resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *		mode. Always provided, even when @assign_ctrl is false.
+ * @assign_ctrl: true to update the @closid for kernel mode; false to inherit the
+ *		CLOSID from the user task.
+ * @rmid:	RMID of the resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *		mode. Always provided, even when @assign_mon is false.
+ * @assign_mon:	true to update the @rmid for kernel mode; false to inherit the
+ *		RMID from the user task.
+ * @enable:	true to enable %RESCTRL_GLOBAL_ENABLE_PER_CPU on CPUs in @cpu_mask;
+ *		false to disable @RESCTRL_GLOBAL_ENABLE_PER_CPU and reset the
+ *		kernel mode to %RESCTRL_INHERIT_USER.
+ *
+ * The callback programs %RESCTRL_GLOBAL_ENABLE_PER_CPU. A configuration starts
+ * when %RESCTRL_GLOBAL_ENABLE_PER_CPU is activated and ends when the mode is
+ * deactivated or userspace changes the backing resource group, @assign_ctrl, or
+ * @assign_mon. Each call for a given configuration uses the same @closid,
+ * @assign_ctrl, @rmid, and @assign_mon; only @cpu_mask and @enable differ.
+ * Before a configuration ends, it is disabled on every CPU where it is enabled.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
-- 
2.43.0