[PATCH 4/4] fs/resctrl: program PLZA on a CPU that comes online under a binding

[Qinyun Tan <qinyuntan@linux.alibaba.com>]

The kernel-mode (PLZA) MSR (MSR_IA32_PQR_PLZA_ASSOC) is per-CPU and is
only written when a binding is configured via info/kernel_mode or when a
group's kmode_cpus mask is changed.  Both paths only touch the CPUs that
are online at that moment, and the CPU hotplug online callback does not
reprogram it.

As a result, a CPU that comes online *after* a binding is in place -- a
hot-added vCPU, or a CPU that was offline at bind time -- runs with PLZA
disabled even though an empty kmode_cpu_mask ("all online CPUs") or an
explicit mask says it is in scope.  Its CPL0 traffic then uses the
default PQR_ASSOC CLOSID/RMID instead of the bound kernel-mode group, so
the kernel-mode protection is silently off on that CPU while
info/kernel_mode still reports the binding as active.

Make resctrl_online_cpu() the single authoritative sync point: on every
CPU online, idempotently drive the CPU to the state it should have under
the current binding -- enable PLZA with the bound CLOSID/RMID when the
CPU is in scope, otherwise explicitly clear it so stale enable bits left
on a CPU that was offline while it was removed from the scope are cleared
too (offline CPUs are skipped by the IPI-based reprogram paths and the
MSR is not reset by a soft offline).  Bail out early on platforms without
kernel-mode support so the MSR is never touched there.

offline needs no counterpart: a CPU's PLZA state is fully determined by
this online-time sync.

Signed-off-by: Qinyun Tan <qinyuntan@linux.alibaba.com>
Reviewed-by: Xunlei Pang <xlpang@linux.alibaba.com>
---
 fs/resctrl/rdtgroup.c | 53 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index cef6ab75c2b57..35cc0083d1b2f 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -5336,11 +5336,64 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 	return err;
 }
 
+/*
+ * resctrl_kmode_online_cpu() - Sync a freshly onlined CPU to its PLZA state
+ * @cpu: The CPU that just came online.
+ *
+ * The kernel-mode (PLZA) MSR is per-CPU and is only written when a binding is
+ * configured or its kmode_cpus mask is changed; nothing reprograms it for a
+ * CPU that comes online afterwards.  Make the online callback the single
+ * authoritative sync point: idempotently drive @cpu to the state it should
+ * have under the current binding.  This both programs a CPU that is onlined
+ * after a binding is in place (e.g. a hot-added vCPU, or a CPU that was
+ * offline at bind time) and clears stale enable bits left on a CPU that was
+ * offline while it was removed from the scope (offline CPUs are skipped by
+ * the IPI-based reprogram paths and the MSR is not reset by a soft offline).
+ *
+ * Scope: an empty kmode_cpu_mask means "all online CPUs" (so @cpu is always
+ * included), a non-empty mask means only the CPUs explicitly listed there.
+ *
+ * Context: Caller must hold rdtgroup_mutex.  Runs on @cpu during the CPU
+ * hotplug online callback, so the MSR write lands on the right CPU and no
+ * cross-CPU IPI is needed.
+ */
+static void resctrl_kmode_online_cpu(unsigned int cpu)
+{
+	struct rdtgroup *rdtgrp = resctrl_kcfg.k_rdtgrp;
+	u32 closid = 0, rmid = 0;
+	bool assign_rmid = false;
+	bool enable = false;
+
+	/* Platform without kernel-mode (PLZA) support: never touch the MSR. */
+	if (!(resctrl_kcfg.kmode & ~BIT(INHERIT_CTRL_AND_MON)))
+		return;
+
+	/* Compute this CPU's correct state under the active binding. */
+	if (resctrl_kcfg.kmode_cur != BIT(INHERIT_CTRL_AND_MON) && rdtgrp &&
+	    (cpumask_empty(&rdtgrp->kmode_cpu_mask) ||
+	     cpumask_test_cpu(cpu, &rdtgrp->kmode_cpu_mask))) {
+		if (rdtgrp->type == RDTMON_GROUP) {
+			closid = rdtgrp->mon.parent->closid;
+			rmid = rdtgrp->mon.rmid;
+		} else {
+			closid = rdtgrp->closid;
+			rmid = rdtgrp->mon.rmid;
+		}
+		assign_rmid = (resctrl_kcfg.kmode_cur == BIT(GLOBAL_ASSIGN_CTRL_ASSIGN_MON_PER_CPU));
+		enable = true;
+	}
+
+	/* Idempotent sync: enable when in scope, otherwise clear stale bits. */
+	resctrl_arch_configure_kmode(cpumask_of(cpu), closid, rmid, assign_rmid, enable);
+}
+
 void resctrl_online_cpu(unsigned int cpu)
 {
 	mutex_lock(&rdtgroup_mutex);
 	/* The CPU is set in default rdtgroup after online. */
 	cpumask_set_cpu(cpu, &rdtgroup_default.cpu_mask);
+	/* Restore the kernel-mode (PLZA) binding on a CPU that just came online. */
+	resctrl_kmode_online_cpu(cpu);
 	mutex_unlock(&rdtgroup_mutex);
 }
 
-- 
2.43.7