Re: [PATCH v5 2/2] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot

[Babu Moger <babu.moger@amd.com>]

Hi Reinette,

Thanks for the review.

On 10/26/26 19:15, Reinette Chatre wrote:
> Hi Babu,
> 
> On 10/21/26 4:58 PM, Babu Moger wrote:
>> The mbm_event mode assigns hardware MBM counters to RMID/event pairs.
>> It is intended for deployments that need to manage counter assignment
>> on platforms where the number of monitoring groups exceeds the available
>> hardware counters. Hardware counters are scarce on these platforms, so
>> mbm_event mode suits workflows that monitor a subset of groups at a time
>> and rotate assignments as needed.
>>
>> resctrl enables the mbm_event mode by default on hardware that supports
> 
> nit: "resctrl enables" -> "x86 resctrl enables" to highlight this is not the
> resctrl filesystem behavior but the x86 arch driver behavior, which is different
> from the MPAM driver.

Sure.

> 
>> it. This breaks the pqos tool [1], which assumes the historical default
>> mode. It creates 16 or more monitoring groups and uses two counters per
>> group. On a platform with 32 counters per domain, that exhausts the
>> mbm_event counters pool, causing additional groups to read "Unassigned".
>> pqos interprets a non-numeric event read as zero bandwidth and reports 0
>> MB/s for those groups:
>>
>>    $sudo pqos -m all:0-4
>>    CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>>       0        0.69         16k       192.0         0.0         0.0
>>       1        0.40          3k         0.0         0.0         0.0
>>       2        0.44          1k        64.0         0.0         0.0
>>       3        0.39          1k        64.0         0.0         0.0
>>       4        0.43          1k         0.0         0.0         0.0
>>
>> Leave mbm_assign_mode in default mode during initialization. Default mode
>> uses one counter per monitoring group. On existing AMD platforms, the
>> default active counters pool is 64 and may be larger on newer hardware.
>> The active counters pool is the number of counters that the hardware can
>> actively track in default mode.
>>
>> Deployments within the active counters pool receive accurate bandwidth
>> measurements in the default mode. Users can create more monitoring groups
>> (for example, 4096 on current hardware) than there are active counters in
>> the pool. Beyond that pool, readings may be misleading or report as
>> "Unavailable", with no user-visible indication.
>>
>> Default mode is still limited once the number of monitoring groups exceeds
>> that active counters pool. After hardware re-allocates a counter, a read
>> may return "Unavailable". pqos still treats "Unavailable" as zero.
>> Successive reads can be a count, "Unavailable", then another count. pqos
>> treats those jumps as wraparound and can report an inconsistent rate:
>>
>>    $sudo pqos -m all:0-4
>>    CORE     IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>>    0        0.76         60k        32.0         0.0         0.0
>>    1        0.45          1k        32.0         0.0         0.0
>>    2        1.57        107k        64.0         0.0 17592186044184.9
>>    3        1.62        276k      4928.0         3.2         0.5
>>
>> Users that need stable measurements beyond the active counters pool should
>> use mbm_event mode and rotate assignments as needed. Enable it with:
>>
>>    $echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> 
> nit: "$echo mbm_event" -> "# echo mbm_event"
> 

Sure.

>>
>> Fixes: 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported")
>> Closes: https://github.com/intel/intel-cmt-cat/issues/311
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> Cc: stable@vger.kernel.org
>> Link: https://github.com/intel/intel-cmt-cat # [1]
>> ---
> 
> ...
> 
>> ---
>>   Documentation/filesystems/resctrl.rst | 101 ++++++++++++++++----------
>>   arch/x86/kernel/cpu/resctrl/monitor.c |   1 -
>>   2 files changed, 63 insertions(+), 39 deletions(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index b52795e03303..fc3ced99e3b7 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -354,8 +354,8 @@ with the following files:
>>   	::
>>   
>>   	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>> -	  [mbm_event]
>> -	  default
>> +	  [default]
>> +	  mbm_event
>>   
>>   	"mbm_event":
>>   
>> @@ -373,20 +373,35 @@ with the following files:
>>   	Moving to mbm_event counter assignment mode requires users to assign the counters
>>   	to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
>>   
>> -	The mode is beneficial for AMD platforms that support more CTRL_MON
>> -	and MON groups than available hardware counters. By default, this
>> -	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
>> -	Monitoring Counters) capability, ensuring counters remain assigned even
>> -	when the corresponding RMID is not actively used by any processor.
>> +	It is intended for deployments that need to manage counter assignment on
>> +	platforms where the number of monitoring groups exceeds the available
>> +	hardware counters. Hardware counters are scarce on these platforms, so
>> +	mbm_event mode suits workflows that monitor a subset of groups at a time and
>> +	rotate assignments as needed. The mbm_event mode ensures counters remain
>> +	assigned even when the corresponding RMID is not actively monitored.
> 
> An assigned counter implies that the RMID,event is monitored. Considering this, the last
> sentence appears to have a contradication:
> 	The mbm_event mode ensures counters remain assigned even when the
> 	corresponding RMID is _not_ actively _monitored_.
> 
> Should "monitored" rather be "used"? This would read:
> 	The mbm_event mode ensures counters remain assigned even when the
> 	corresponding RMID is not actively used.

Yes. Sounds good.

>    >  	"default":
>>   
>>   	In default mode, resctrl assumes there is a hardware counter for each
>> -	event within every CTRL_MON and MON group. On AMD platforms, it is
>> -	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
>> -	events between reads resulting from hardware re-allocating counters. This can
>> -	result in misleading values or display "Unavailable" if no counter is assigned
>> -	to the event.
>> +	event within every CTRL_MON and MON group. This mode is enabled by default.
> 
> Please drop "This mode is enabled by default." since this is general documentation and
> this is not true for MPAM.

Sure.

> 
>> +
>> +	On AMD platforms that support more CTRL_MON and MON groups than hardware
>> +	counters, hardware dynamically shares a smaller pool of counters among
>> +	RMIDs. One counter from this pool is used per monitoring group and counts
>> +	every MBM event of that group's RMID. The size of the pool is not
>> +	enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be
>> +	much larger. For example, 64 counters in this pool monitor 64 groups.
>> +	Newer hardware may provide a larger pool.
>> +
>> +	While the number of monitoring groups does not exceed that pool, a counter
>> +	stays attached to each RMID and readings remain accurate. Creating more
>> +	groups than the pool can cause hardware to re-allocate those counters
>> +	between successive reads of an event. Bandwidth values may then be
>> +	misleading, or a read may return "Unavailable" if no counter is allocated
>> +	to the RMID. There is no user-visible indication when this begins.
>> +	Users who need stable readings beyond that pool should switch to mbm_event
>> +	mode, if supported, and assign counters to the groups of interest
>> +	(rotating assignments as needed).
> 
> Above are two AMD architecture specific paragraphs inserted in the middle of generic
> documentation. On top of that, the second paragraph does not have any architecture
> specific "markers" and this could be confused as being generic.
> 
> Could this be re-organized to seperate the generic and arch specific documentation?
> This could be done similar to how the io_alloc documentation used "See architecture
> specific notes below." text. The two AMD specific paragraphs can then be linked.

Yes. Looks good. Thanks for the patch.

> 
> For example (changes on top of this patch),
> ---8<---
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index fc3ced99e3b7..f33c06612a77 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -350,7 +350,7 @@ with the following files:
>   "mbm_assign_mode":
>   	The supported counter assignment modes. The enclosed brackets indicate which mode
>   	is enabled. The MBM events associated with counters may reset when "mbm_assign_mode"
> -	is changed.
> +	is changed. See architecture specific notes below.
>   	::
>   
>   	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> @@ -383,25 +383,7 @@ with the following files:
>   	"default":
>   
>   	In default mode, resctrl assumes there is a hardware counter for each
> -	event within every CTRL_MON and MON group. This mode is enabled by default.
> -
> -	On AMD platforms that support more CTRL_MON and MON groups than hardware
> -	counters, hardware dynamically shares a smaller pool of counters among
> -	RMIDs. One counter from this pool is used per monitoring group and counts
> -	every MBM event of that group's RMID. The size of the pool is not
> -	enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be
> -	much larger. For example, 64 counters in this pool monitor 64 groups.
> -	Newer hardware may provide a larger pool.
> -
> -	While the number of monitoring groups does not exceed that pool, a counter
> -	stays attached to each RMID and readings remain accurate. Creating more
> -	groups than the pool can cause hardware to re-allocate those counters
> -	between successive reads of an event. Bandwidth values may then be
> -	misleading, or a read may return "Unavailable" if no counter is allocated
> -	to the RMID. There is no user-visible indication when this begins.
> -	Users who need stable readings beyond that pool should switch to mbm_event
> -	mode, if supported, and assign counters to the groups of interest
> -	(rotating assignments as needed).
> +	event within every CTRL_MON and MON group.
>   
>   	* To enable "mbm_event" counter assignment mode:
>   	  ::
> @@ -413,6 +395,24 @@ with the following files:
>   
>   	    # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>   
> +	In default mode on AMD platforms that support more CTRL_MON and MON groups
> +	than hardware counters, hardware dynamically shares a smaller pool of
> +	counters among RMIDs. One counter from this pool is used per monitoring
> +	group and counts every MBM event of that group's RMID. The size of the pool
> +	is not enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may
> +	be much larger. For example, 64 counters in this pool monitor 64 groups.
> +	Newer hardware may provide a larger pool.
> +
> +	On such platforms, while the number of monitoring groups does not exceed
> +	that pool, a counter stays attached to each RMID and readings remain
> +	accurate. Creating more groups than the pool can cause hardware to
> +	re-allocate those counters between successive reads of an event. Bandwidth
> +	values may then be misleading, or a read may return "Unavailable" if no
> +	counter is allocated to the RMID. There is no user-visible indication when
> +	this begins. Users who need stable readings beyond that pool should switch
> +	to mbm_event mode, if supported, and assign counters to the groups of
> +	interest (rotating assignments as needed).
> +
>   "num_mbm_cntrs":
>   	The maximum number of counters (total of available and assigned counters) in
>   	each domain when the system supports mbm_event mode.
> ---8<---
> 
> 
> 
>>   
>>   	* To enable "mbm_event" counter assignment mode:
>>   	  ::
>> @@ -1784,62 +1799,72 @@ View the llc occupancy snapshot::
>>   Examples on working with mbm_assign_mode
>>   ========================================
>>   
>> -a. Check if MBM counter assignment mode is supported.
>> +a. Check if MBM counter assignment mode is supported. The boot mode is "default".
> 
> Same comment as earlier, please drop the "The boot mode is "default"." that
> does not apply to MPAM.

Sure.

> 
>>   ::
>>   
>> -  # mount -t resctrl resctrl /sys/fs/resctrl/
> 
> Since this drops the "mount" command. Should the final "unmount" step be dropped also?

Yes.

> 
>> +  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>> +  [default]
>> +  mbm_event
>> +
>> +The "default" mode is enabled. "mbm_event" is supported but not enabled.
>>   
>> +b. Enable "mbm_event" mode. Switching the mbm_assign_mode may
>> +reset all the MBM counters (and thus all MBM events) of all the resctrl groups.
>> +::
>> +
>> +  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>     # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>     [mbm_event]
>>     default
>>   
> 
> The above changes could be merged together under a single step "a" that
> will significantly reduce the churn when considering this is a stable patch.

Yes. We can. But because of auto assign is not enabled on mode switch, 
we still have to touch few steps. Your choice.

> 
>> -The "mbm_event" mode is detected and enabled.
>> -
>> -b. Check how many assignable counters are supported.
>> +c. Check how many assignable counters are supported.
>>   ::
>>   
>>     # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
>>     0=32;1=32
>>   
>> -c. Check how many assignable counters are available for assignment in each domain.
>> +d. Check how many assignable counters are available for assignment in each domain.
>>   ::
>>   
>>     # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
>> -  0=30;1=30
>> +  0=32;1=32
>>   
>> -d. To list the default group's assign states.
>> +e. To list the default group's assign states. Counters are not assigned
>> +automatically. They are unassigned when the mode is switched to "mbm_event".
> 
> I think "Counters are not assigned automatically." contradict mbm_assign_on_mkdir
> that is forced true on the switch. Can it just be dropped? That would just leave:
> 
> 	e. To list the default group's assign states. Counters are unassigned when the mode is switched to "mbm_event".

Yes. Sure.

> 
>>   ::
>>   
>>     # cat /sys/fs/resctrl/mbm_L3_assignments
>> -  mbm_total_bytes:0=e;1=e
>> -  mbm_local_bytes:0=e;1=e
>> +  mbm_total_bytes:0=_;1=_
>> +  mbm_local_bytes:0=_;1=_
>>   
>> -e.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
>> +f. To assign the counter associated with the mbm_total_bytes event on domain 0
>> +in exclusive mode.
>>   ::
>>   
>> -  # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
>> +  # echo "mbm_total_bytes:0=e" > /sys/fs/resctrl/mbm_L3_assignments
>>     # cat /sys/fs/resctrl/mbm_L3_assignments
>> -  mbm_total_bytes:0=_;1=e
>> -  mbm_local_bytes:0=e;1=e
>> +  mbm_total_bytes:0=e;1=_
>> +  mbm_local_bytes:0=_;1=_
>>   
>> -f. To unassign the counter associated with the mbm_total_bytes event on all domains.
>> +g. To unassign the counter associated with the mbm_total_bytes event on all domains.
>>   ::
>>   
>>     # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
>> -  # cat /sys/fs/resctrl/mbm_L3_assignment
>> +  # cat /sys/fs/resctrl/mbm_L3_assignments
>>     mbm_total_bytes:0=_;1=_
>> -  mbm_local_bytes:0=e;1=e
>> +  mbm_local_bytes:0=_;1=_
>>   
>> -g. To assign a counter associated with the mbm_total_bytes event on all domains in
>> -exclusive mode.
>> +h. To assign a counter associated with the mbm_total_bytes and mbm_local_bytes
>> +events on all domains in exclusive mode.
>>   ::
>>   
>>     # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
>> +  # echo "mbm_local_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
>>     # cat /sys/fs/resctrl/mbm_L3_assignments
>>     mbm_total_bytes:0=e;1=e
>>     mbm_local_bytes:0=e;1=e
>>   
>> -h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
>> +i. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
>>   no change in reading the events with the assignment.
>>   ::
> 
> Could you please remind me if the first read after assignment returns
> "Unavailable" or a valid count?


As far as I have observed, mbm_event mode (ABMC) returns either 0 when 
there is no activity or a valid count. I have not personally encountered 
an Unavailable value. However, the specification allows Unavailable to 
be returned when the count is not available.

> 
>>   
>> @@ -1852,7 +1877,7 @@ no change in reading the events with the assignment.
>>     # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
>>     121212144
>>   
>> -i. Check the event configurations.
>> +j. Check the event configurations.
>>   ::
>>   
>>     # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
>> @@ -1862,7 +1887,7 @@ i. Check the event configurations.
>>     # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
>>     local_reads,local_non_temporal_writes,local_reads_slow_memory
>>   
>> -j. Change the event configuration for mbm_local_bytes.
>> +k. Change the event configuration for mbm_local_bytes.
>>   ::
>>   
>>     # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
>> @@ -1871,7 +1896,7 @@ j. Change the event configuration for mbm_local_bytes.
>>     # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
>>     local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads
>>   
>> -k. Now read the local events again. The first read may come back with "Unavailable"
>> +l. Now read the local events again. The first read may come back with "Unavailable"
>>   status. The subsequent read of mbm_local_bytes will display the current value.
>>   ::
>>   
>> @@ -1884,17 +1909,17 @@ status. The subsequent read of mbm_local_bytes will display the current value.
>>     # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
>>     1566565
>>   
>> -l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
>> +m. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
>>   done using the following command. Note that switching the mbm_assign_mode may reset all
>>   the MBM counters (and thus all MBM events) of all the resctrl groups.
>>   ::
>>   
>>     # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>     # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>> -  mbm_event
>>     [default]
>> +  mbm_event
>>   
>> -m. Unmount the resctrl filesystem.
>> +n. Unmount the resctrl filesystem.
>>   ::
>>   
> 
> As mentioned earlier this is now asymmetric after dropping the "mount" in the first step.
> Should this unmount step be dropped now?

Sure.

Thanks
Babu