Re: [PATCH v5 2/2] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 10/21/26 4:58 PM, Babu Moger wrote:
> The mbm_event mode assigns hardware MBM counters to RMID/event pairs.
> It is intended for deployments that need to manage counter assignment
> on platforms where the number of monitoring groups exceeds the available
> hardware counters. Hardware counters are scarce on these platforms, so
> mbm_event mode suits workflows that monitor a subset of groups at a time
> and rotate assignments as needed.
> 
> resctrl enables the mbm_event mode by default on hardware that supports

nit: "resctrl enables" -> "x86 resctrl enables" to highlight this is not the
resctrl filesystem behavior but the x86 arch driver behavior, which is different
from the MPAM driver. 

> it. This breaks the pqos tool [1], which assumes the historical default
> mode. It creates 16 or more monitoring groups and uses two counters per
> group. On a platform with 32 counters per domain, that exhausts the
> mbm_event counters pool, causing additional groups to read "Unassigned".
> pqos interprets a non-numeric event read as zero bandwidth and reports 0
> MB/s for those groups:
> 
>   $sudo pqos -m all:0-4
>   CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>      0        0.69         16k       192.0         0.0         0.0
>      1        0.40          3k         0.0         0.0         0.0
>      2        0.44          1k        64.0         0.0         0.0
>      3        0.39          1k        64.0         0.0         0.0
>      4        0.43          1k         0.0         0.0         0.0
> 
> Leave mbm_assign_mode in default mode during initialization. Default mode
> uses one counter per monitoring group. On existing AMD platforms, the
> default active counters pool is 64 and may be larger on newer hardware.
> The active counters pool is the number of counters that the hardware can
> actively track in default mode.
> 
> Deployments within the active counters pool receive accurate bandwidth
> measurements in the default mode. Users can create more monitoring groups
> (for example, 4096 on current hardware) than there are active counters in
> the pool. Beyond that pool, readings may be misleading or report as
> "Unavailable", with no user-visible indication.
> 
> Default mode is still limited once the number of monitoring groups exceeds
> that active counters pool. After hardware re-allocates a counter, a read
> may return "Unavailable". pqos still treats "Unavailable" as zero.
> Successive reads can be a count, "Unavailable", then another count. pqos
> treats those jumps as wraparound and can report an inconsistent rate:
> 
>   $sudo pqos -m all:0-4
>   CORE     IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>   0        0.76         60k        32.0         0.0         0.0
>   1        0.45          1k        32.0         0.0         0.0
>   2        1.57        107k        64.0         0.0 17592186044184.9
>   3        1.62        276k      4928.0         3.2         0.5
> 
> Users that need stable measurements beyond the active counters pool should
> use mbm_event mode and rotate assignments as needed. Enable it with:
> 
>   $echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

nit: "$echo mbm_event" -> "# echo mbm_event"

> 
> Fixes: 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported")
> Closes: https://github.com/intel/intel-cmt-cat/issues/311
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Cc: stable@vger.kernel.org
> Link: https://github.com/intel/intel-cmt-cat # [1]
> ---

...

> ---
>  Documentation/filesystems/resctrl.rst | 101 ++++++++++++++++----------
>  arch/x86/kernel/cpu/resctrl/monitor.c |   1 -
>  2 files changed, 63 insertions(+), 39 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index b52795e03303..fc3ced99e3b7 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -354,8 +354,8 @@ with the following files:
>  	::
>  
>  	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> -	  [mbm_event]
> -	  default
> +	  [default]
> +	  mbm_event
>  
>  	"mbm_event":
>  
> @@ -373,20 +373,35 @@ with the following files:
>  	Moving to mbm_event counter assignment mode requires users to assign the counters
>  	to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
>  
> -	The mode is beneficial for AMD platforms that support more CTRL_MON
> -	and MON groups than available hardware counters. By default, this
> -	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
> -	Monitoring Counters) capability, ensuring counters remain assigned even
> -	when the corresponding RMID is not actively used by any processor.
> +	It is intended for deployments that need to manage counter assignment on
> +	platforms where the number of monitoring groups exceeds the available
> +	hardware counters. Hardware counters are scarce on these platforms, so
> +	mbm_event mode suits workflows that monitor a subset of groups at a time and
> +	rotate assignments as needed. The mbm_event mode ensures counters remain
> +	assigned even when the corresponding RMID is not actively monitored.

An assigned counter implies that the RMID,event is monitored. Considering this, the last
sentence appears to have a contradication:
	The mbm_event mode ensures counters remain assigned even when the
	corresponding RMID is _not_ actively _monitored_.

Should "monitored" rather be "used"? This would read:
	The mbm_event mode ensures counters remain assigned even when the
	corresponding RMID is not actively used.
  >  	"default":
>  
>  	In default mode, resctrl assumes there is a hardware counter for each
> -	event within every CTRL_MON and MON group. On AMD platforms, it is
> -	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
> -	events between reads resulting from hardware re-allocating counters. This can
> -	result in misleading values or display "Unavailable" if no counter is assigned
> -	to the event.
> +	event within every CTRL_MON and MON group. This mode is enabled by default.

Please drop "This mode is enabled by default." since this is general documentation and
this is not true for MPAM.

> +
> +	On AMD platforms that support more CTRL_MON and MON groups than hardware
> +	counters, hardware dynamically shares a smaller pool of counters among
> +	RMIDs. One counter from this pool is used per monitoring group and counts
> +	every MBM event of that group's RMID. The size of the pool is not
> +	enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be
> +	much larger. For example, 64 counters in this pool monitor 64 groups.
> +	Newer hardware may provide a larger pool.
> +
> +	While the number of monitoring groups does not exceed that pool, a counter
> +	stays attached to each RMID and readings remain accurate. Creating more
> +	groups than the pool can cause hardware to re-allocate those counters
> +	between successive reads of an event. Bandwidth values may then be
> +	misleading, or a read may return "Unavailable" if no counter is allocated
> +	to the RMID. There is no user-visible indication when this begins.
> +	Users who need stable readings beyond that pool should switch to mbm_event
> +	mode, if supported, and assign counters to the groups of interest
> +	(rotating assignments as needed).

Above are two AMD architecture specific paragraphs inserted in the middle of generic
documentation. On top of that, the second paragraph does not have any architecture
specific "markers" and this could be confused as being generic.

Could this be re-organized to seperate the generic and arch specific documentation?
This could be done similar to how the io_alloc documentation used "See architecture
specific notes below." text. The two AMD specific paragraphs can then be linked.

For example (changes on top of this patch),
---8<---
diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index fc3ced99e3b7..f33c06612a77 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -350,7 +350,7 @@ with the following files:
 "mbm_assign_mode":
 	The supported counter assignment modes. The enclosed brackets indicate which mode
 	is enabled. The MBM events associated with counters may reset when "mbm_assign_mode"
-	is changed.
+	is changed. See architecture specific notes below.
 	::
 
 	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
@@ -383,25 +383,7 @@ with the following files:
 	"default":
 
 	In default mode, resctrl assumes there is a hardware counter for each
-	event within every CTRL_MON and MON group. This mode is enabled by default.
-
-	On AMD platforms that support more CTRL_MON and MON groups than hardware
-	counters, hardware dynamically shares a smaller pool of counters among
-	RMIDs. One counter from this pool is used per monitoring group and counts
-	every MBM event of that group's RMID. The size of the pool is not
-	enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be
-	much larger. For example, 64 counters in this pool monitor 64 groups.
-	Newer hardware may provide a larger pool.
-
-	While the number of monitoring groups does not exceed that pool, a counter
-	stays attached to each RMID and readings remain accurate. Creating more
-	groups than the pool can cause hardware to re-allocate those counters
-	between successive reads of an event. Bandwidth values may then be
-	misleading, or a read may return "Unavailable" if no counter is allocated
-	to the RMID. There is no user-visible indication when this begins.
-	Users who need stable readings beyond that pool should switch to mbm_event
-	mode, if supported, and assign counters to the groups of interest
-	(rotating assignments as needed).
+	event within every CTRL_MON and MON group.
 
 	* To enable "mbm_event" counter assignment mode:
 	  ::
@@ -413,6 +395,24 @@ with the following files:
 
 	    # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
 
+	In default mode on AMD platforms that support more CTRL_MON and MON groups
+	than hardware counters, hardware dynamically shares a smaller pool of
+	counters among RMIDs. One counter from this pool is used per monitoring
+	group and counts every MBM event of that group's RMID. The size of the pool
+	is not enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may
+	be much larger. For example, 64 counters in this pool monitor 64 groups.
+	Newer hardware may provide a larger pool.
+
+	On such platforms, while the number of monitoring groups does not exceed
+	that pool, a counter stays attached to each RMID and readings remain
+	accurate. Creating more groups than the pool can cause hardware to
+	re-allocate those counters between successive reads of an event. Bandwidth
+	values may then be misleading, or a read may return "Unavailable" if no
+	counter is allocated to the RMID. There is no user-visible indication when
+	this begins. Users who need stable readings beyond that pool should switch
+	to mbm_event mode, if supported, and assign counters to the groups of
+	interest (rotating assignments as needed).
+
 "num_mbm_cntrs":
 	The maximum number of counters (total of available and assigned counters) in
 	each domain when the system supports mbm_event mode.
---8<---



>  
>  	* To enable "mbm_event" counter assignment mode:
>  	  ::
> @@ -1784,62 +1799,72 @@ View the llc occupancy snapshot::
>  Examples on working with mbm_assign_mode
>  ========================================
>  
> -a. Check if MBM counter assignment mode is supported.
> +a. Check if MBM counter assignment mode is supported. The boot mode is "default".

Same comment as earlier, please drop the "The boot mode is "default"." that
does not apply to MPAM.

>  ::
>  
> -  # mount -t resctrl resctrl /sys/fs/resctrl/

Since this drops the "mount" command. Should the final "unmount" step be dropped also?

> +  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> +  [default]
> +  mbm_event
> +
> +The "default" mode is enabled. "mbm_event" is supported but not enabled.
>  
> +b. Enable "mbm_event" mode. Switching the mbm_assign_mode may
> +reset all the MBM counters (and thus all MBM events) of all the resctrl groups.
> +::
> +
> +  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>    # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>    [mbm_event]
>    default
>  

The above changes could be merged together under a single step "a" that
will significantly reduce the churn when considering this is a stable patch.

> -The "mbm_event" mode is detected and enabled.
> -
> -b. Check how many assignable counters are supported.
> +c. Check how many assignable counters are supported.
>  ::
>  
>    # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
>    0=32;1=32
>  
> -c. Check how many assignable counters are available for assignment in each domain.
> +d. Check how many assignable counters are available for assignment in each domain.
>  ::
>  
>    # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
> -  0=30;1=30
> +  0=32;1=32
>  
> -d. To list the default group's assign states.
> +e. To list the default group's assign states. Counters are not assigned
> +automatically. They are unassigned when the mode is switched to "mbm_event".

I think "Counters are not assigned automatically." contradict mbm_assign_on_mkdir
that is forced true on the switch. Can it just be dropped? That would just leave:

	e. To list the default group's assign states. Counters are unassigned when the mode is switched to "mbm_event".

>  ::
>  
>    # cat /sys/fs/resctrl/mbm_L3_assignments
> -  mbm_total_bytes:0=e;1=e
> -  mbm_local_bytes:0=e;1=e
> +  mbm_total_bytes:0=_;1=_
> +  mbm_local_bytes:0=_;1=_
>  
> -e.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
> +f. To assign the counter associated with the mbm_total_bytes event on domain 0
> +in exclusive mode.
>  ::
>  
> -  # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
> +  # echo "mbm_total_bytes:0=e" > /sys/fs/resctrl/mbm_L3_assignments
>    # cat /sys/fs/resctrl/mbm_L3_assignments
> -  mbm_total_bytes:0=_;1=e
> -  mbm_local_bytes:0=e;1=e
> +  mbm_total_bytes:0=e;1=_
> +  mbm_local_bytes:0=_;1=_
>  
> -f. To unassign the counter associated with the mbm_total_bytes event on all domains.
> +g. To unassign the counter associated with the mbm_total_bytes event on all domains.
>  ::
>  
>    # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
> -  # cat /sys/fs/resctrl/mbm_L3_assignment
> +  # cat /sys/fs/resctrl/mbm_L3_assignments
>    mbm_total_bytes:0=_;1=_
> -  mbm_local_bytes:0=e;1=e
> +  mbm_local_bytes:0=_;1=_
>  
> -g. To assign a counter associated with the mbm_total_bytes event on all domains in
> -exclusive mode.
> +h. To assign a counter associated with the mbm_total_bytes and mbm_local_bytes
> +events on all domains in exclusive mode.
>  ::
>  
>    # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
> +  # echo "mbm_local_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
>    # cat /sys/fs/resctrl/mbm_L3_assignments
>    mbm_total_bytes:0=e;1=e
>    mbm_local_bytes:0=e;1=e
>  
> -h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
> +i. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
>  no change in reading the events with the assignment.
>  ::

Could you please remind me if the first read after assignment returns
"Unavailable" or a valid count?

>  
> @@ -1852,7 +1877,7 @@ no change in reading the events with the assignment.
>    # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
>    121212144
>  
> -i. Check the event configurations.
> +j. Check the event configurations.
>  ::
>  
>    # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
> @@ -1862,7 +1887,7 @@ i. Check the event configurations.
>    # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
>    local_reads,local_non_temporal_writes,local_reads_slow_memory
>  
> -j. Change the event configuration for mbm_local_bytes.
> +k. Change the event configuration for mbm_local_bytes.
>  ::
>  
>    # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
> @@ -1871,7 +1896,7 @@ j. Change the event configuration for mbm_local_bytes.
>    # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
>    local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads
>  
> -k. Now read the local events again. The first read may come back with "Unavailable"
> +l. Now read the local events again. The first read may come back with "Unavailable"
>  status. The subsequent read of mbm_local_bytes will display the current value.
>  ::
>  
> @@ -1884,17 +1909,17 @@ status. The subsequent read of mbm_local_bytes will display the current value.
>    # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
>    1566565
>  
> -l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
> +m. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
>  done using the following command. Note that switching the mbm_assign_mode may reset all
>  the MBM counters (and thus all MBM events) of all the resctrl groups.
>  ::
>  
>    # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>    # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> -  mbm_event
>    [default]
> +  mbm_event
>  
> -m. Unmount the resctrl filesystem.
> +n. Unmount the resctrl filesystem.
>  ::
>  

As mentioned earlier this is now asymmetric after dropping the "mount" in the first step.
Should this unmount step be dropped now?

>    # umount /sys/fs/resctrl/
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 5d0d3b18f9b8..a6a9090c4808 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -472,7 +472,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
>  		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
>  		/* cntr_id is 12 bits and can only encode 4096 counters. */
>  		r->mon.num_mbm_cntrs = min((ebx & GENMASK(15, 0)) + 1, BIT(12));
> -		hw_res->mbm_cntr_assign_enabled = true;
>  	}
>  
>  	r->mon_capable = true;

Reinette