Re: [PATCH v9 09/26] x86/resctrl: Introduce interface to display number of monitoring counters

[Peter Newman <peternewman@google.com>]

Hi Babu,

On Mon, Nov 18, 2024 at 03:31:28PM -0600, Moger, Babu wrote:
> Hi Reinette,
> 
> On 11/15/24 18:06, Reinette Chatre wrote:
> > Hi Babu,
> > 
> > On 10/29/24 4:21 PM, Babu Moger wrote:
> >> The mbm_cntr_assign mode provides an option to the user to assign a
> >> counter to an RMID, event pair and monitor the bandwidth as long as
> >> the counter is assigned. Number of assignments depend on number of
> >> monitoring counters available.
> >>
> >> Provide the interface to display the number of monitoring counters
> >> supported. The interface file 'num_mbm_cntrs' is available when an
> >> architecture supports mbm_cntr_assign mode.
> >>
> > 
> > As mentioned in previous patch, do you think it may be possible to
> > have a value for num_mbm_cntrs for non-ABMC AMD systems? If that is
> > available and always exposed to user space (irrespective of
> > mbm_cntr_assign mode) then it would be clear to user space on
> > benefits/risks of running a "default" mode.
> 
> I am trying the work-around to get the number of max active RMIDs in
> default mode. The method is to loop through all of the recently assigned
> RMID's to see if any of their QM_CTR.U bits transition from 0->1.
> 
> I am not successful in getting it to work so far. I remember Peter was
> trying this before in soft-ABMC. Peter, Any success with that?

Sorry I missed this question before. By now maybe you've already debugged your
own implementation...

Here's what I've been using:

-- >8 --
Subject: [PATCH] x86/resctrl: Detect MBM counters on pre-ABMC AMD
 implementations

This procedure is based on information provided directly by AMD
which cannot currently be corroborated by any public documentation.

In particular, it assumes that deallocation of MBM counters for RMIDs
is driven directly by writing a new RMID value into PQR_ASSOC.

Signed-off-by: Peter Newman <peternewman@google.com>
---
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 arch/x86/kernel/cpu/resctrl/monitor.c  | 83 ++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 377b5db667930..4a13eab110510 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -705,6 +705,7 @@ int closids_supported(void);
 void closid_free(int closid);
 int alloc_rmid(u32 closid);
 void free_rmid(u32 closid, u32 rmid);
+int amd_detect_mbm_counters(void);
 int __init rdt_get_mon_l3_config(struct rdt_resource *r);
 void __exit rdt_put_mon_l3_config(void);
 bool __init rdt_cpu_has(int flag);
diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
index 2dd6c47c9276a..f4a59251134d3 100644
--- a/arch/x86/kernel/cpu/resctrl/monitor.c
+++ b/arch/x86/kernel/cpu/resctrl/monitor.c
@@ -1172,6 +1172,89 @@ static __init int snc_get_config(void)
 	return ret;
 }
 
+static void zero_rmid(void *unused)
+{
+	wrmsr(MSR_IA32_PQR_ASSOC, 0, 0);
+}
+
+int amd_detect_mbm_counters(void)
+{
+	struct cpu_cacheinfo *ci;
+	u32 rmid, test_rmid;
+	int llc_index;
+	u64 ctr;
+
+	/*
+	 * The following detection mechanism below provided by AMD. It applies
+	 * only to pre-ABMC models (Rome, Milan, Genoa).
+	 */
+	if (WARN_ON(boot_cpu_data.x86_vendor != X86_VENDOR_AMD))
+		return -1;
+	if (WARN_ON(rdt_cpu_has(X86_FEATURE_ABMC)))
+		return -1;
+
+	/*
+	 * Must not migrate to another CCX during this test. Assume no IRQ
+	 * handler would access the MSRs used below before resctrl is
+	 * initialized.
+	 */
+	ci = get_cpu_cacheinfo(get_cpu());
+	llc_index = ci->num_leaves - 1;
+
+	/* Ensure PQR_ASSOC.RMID = 0 in this CCX. */
+	if (ci->cpu_map_populated)
+		on_each_cpu_mask(&ci->info_list[llc_index].shared_cpu_map,
+				 zero_rmid, NULL, true);
+
+	rmid = 0;
+	while (true) {
+		wrmsr(MSR_IA32_PQR_ASSOC, rmid, 0);
+		wrmsr(MSR_IA32_QM_EVTSEL, QOS_L3_MBM_TOTAL_EVENT_ID, rmid);
+
+		/*
+		 * Ensure that a counter has been allocated on this RMID. The
+		 * loop below is expected to complete in two iterations.
+		 */
+		do {
+			rdmsrl(MSR_IA32_QM_CTR, ctr);
+
+			if (WARN_ON(ctr & RMID_VAL_ERROR)) {
+				pr_err("failed to read total event on rmid %u\n",
+				       rmid);
+				put_cpu();
+				return 0;
+			}
+		} while (ctr & RMID_VAL_UNAVAIL);
+
+		/*
+		 * The order in which counters are reused is not predictable, so
+		 * check all previously-assigned counters. If any loses its
+		 * value, then too many are in use as a result of the last
+		 * PQR_ASSOC write.
+		 */
+		for (test_rmid = 0; test_rmid < rmid; test_rmid++) {
+			wrmsr(MSR_IA32_QM_EVTSEL, QOS_L3_MBM_TOTAL_EVENT_ID,
+			      test_rmid);
+			rdmsrl(MSR_IA32_QM_CTR, ctr);
+
+			/*
+			 * As soon as a previous counter loses a value, we have
+			 * determined the number of RMIDs which can hold a value
+			 * simultaneously in this CCX.
+			 */
+			if (ctr & RMID_VAL_UNAVAIL) {
+				put_cpu();
+				return rmid;
+			}
+		}
+		rmid++;
+	}
+
+	put_cpu();
+
+	return 0;
+}
+
 int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;