Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Ben Horgan <ben.horgan@arm.com>]

Hi Reinette,

On 7/14/26 23:06, Reinette Chatre wrote:
> Hi Fenghua,
> 
> On 7/10/26 1:59 PM, Fenghua Yu wrote:
>> On 6/25/26 08:43, Reinette Chatre wrote:
>>> On 6/24/26 6:26 PM, Fenghua Yu wrote:
>>>> On 6/24/26 15:22, Reinette Chatre wrote:
>>>>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>>>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>>>>
>>>>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>>>>> features that may need to change schemata interface. The CPU-less
>>>>>> feature was discussed on LPC (although the interfaces will be
>>>>>> slightly different from the LPC).
>>>>>
>>>>> I know. Here is where I tried to engage with you on needed interfaces after LPC:
>>>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>>>
>>>> MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
>>>> 1. L3 and memory together on each processor MSC
>>>> 2. L3 in processor MSC and memory control/monitoring in different memory MSCs.
>>
>> Ben said there is type 3 platform:
>> 3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.
>>
>>>>
>>>> On type 1 platform, schemata is legacy:
>>>> MB:1=100;2=100  <-- cache id 1 and 2 as domain id
>>>>
>>>> On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
>>>> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
>>>>                             26 as domain id
>>>> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
>>>> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
>>>> memory nodes allocation.
>>
>> On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:
>>
>>      MB:1=100;2=100                 <-- cache id 1 and 2 as domain id
>> MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10,
>>                                                    18, 26 as domain id
>>>
>>> (to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it
>>> explicit that it is memory bandwidth allocation at node scope)
>>>
>>> I am trying to consider how this can be accomplished while also considering all the other
>>> new hardware features that resctrl need to support. Consider, for example, AMD's "Global
>>> MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/) that throttles
>>> memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time
>>> the plan is to support this with a second control associated with the MB resource that can
>>> allocate memory bandwidth at node scope. See
>>> https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@amd.com/
>>>
>>> If resctrl creates a new resource for node scoped memory bandwidth allocations to support these
>>> "type 2" systems then that will result in an inconsistent interface between architectures that
>>> we should avoid.
>>>
>>> Have you been listening in on the discussions surrounding emulated controls? Considering that,
>>> would it be possible to support the "MB" control on a type "2" system but have it be backed by
>>> (emulated by) the underlying "MB_NODE" control?
>>>
>>> resctrl could expose both controls on these "type 2" systems but make it clear that "MB"
>>> is emulated by "MB_NODE". For example:
>>>
>>> info/
>>> └── MB/
>>>      └── resource_schemata/
>>>          └── MB/
>>>              └── MB_NODE/
>>>
>>> User will see both controls in schemata file but when changes are made to "MB" control it
>>> will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control
>>> that will establish familiarity with the interface at which point resctrl can drop the
>>> "MB" control from the schemata file on these "type 2" systems.
>>>
>>> Having the MB resource available with an MB control will keep resctrl backward compatible
>>> if there are any tools that expect that. If backward compatibility is not of concern then
>>> resctrl could initialize with the emulated control disabled by default. See discussion at
>>> https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@intel.com/
>>> that describes a new resctrl capability in support of RISC-V and RDT.
>>> With this resctrl could initialize with:
>>>
>>> info/
>>> └── MB/
>>>      └── resource_schemata/
>>>          ├── MB/
>>>          │   ├── MB_NODE/
>>>          │   │   └── status:enabled
>>>          │   └── status:disabled
>>>          └── mode:legacy [native]
>>>
>>> With above a "type 2" system will boot with its schemata file just containing the "MB_NODE"
>>> control while info/MB describes the memory bandwidth resource.
>>>
>>
>> On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.
>>
>> Is this directory OK?
>>
>>  info/
>>  └── MB/
>>       └── resource_schemata/
>>           ├── MB/
>>           │   ├── MB_NODE/
>>           │   │   └── status:disabled
>>           │   └── status:enabled
>>           ├── MB_NODE/
>>           └── mode:node
>>
>> 1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/
> 
> I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE
> hardware control exists. 
> 
>> 2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node
> 
> I assume you mean "scope" instead of "mode"? (more below)
> 
>> 3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)
> 
> Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.
> 
> From what I understand these "type 3" machines could be simplified to:
> 
> info/
> └── MB/
>     └── resource_schemata/
>         ├── MB/
>         │   └── scope:L3
>         └── MB_NODE/
>             └── scope:NODE
> 
> Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want
> to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface
> may end up looking like:
> 
> info/
> └── MB/
>     └── resource_schemata/
>         ├── MB/
>         │   ├── MB_MAX/
>         │   │   └── scope:L3
>         │   └── scope:L3
>         └── MB_NODE/
>             └── scope:NODE
> 
> A system like above will thus have three schemata file entries:
> MB
> MB_MAX
> MB_NODE
> 
> Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control
> so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE
> on these systems.
> > Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume
we should expect that the user might read back the whole file, modify what they expect to change and
then write everything back. If legacy software does this and wants to change the MB schemata then
the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and
fail the write if two dependent schemata are changed at the same time?

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
"mode" and can just toggle back if they want to use "MB" directly again.

Thanks,

Ben

> 
> ... 
> 
>>>>>> There is another MPAM feature called MBW Max hardlimit which sets
>>>>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>>>>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>>>>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>>>>>> percentage on each domain. By default hardlimit is off (0) and can
>>>>>> be turned on to set MBW Max hardlimit on a domain.
>>>>>
>>>>> ack. This sounds like a new control associated with the MB resource.
>>>>> This is a boolean control as Dave highlighted in previous discussion so
>>>>> resctrl would need to know its properties.
>>>>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
>>>>>
>>>>
>>>> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)
>>>>
>>>>>> For exmple:
>>>>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>>>>>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>>>>>
>>>>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>>>>>>
>>>>>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>>>>>>
>>>>>
>>>>> ack.
>>>>>
>>>>>>>            L3:0=fff;1=fff
>>>>>>> # echo 'MB_MIN:0=50' > schemata
>>>>>>> # cat schemata
>>>>>>>            MB_MAX:0=100;1=100
>>>>>>>            MB_MIN:0=50;1=100
>>>>>>>            MB:0=100;1=100
>>>>>>>            L3:0=fff;1=fff
>>>>>>>
>>>>>>> Writing to the dummy control will call a dummy callback that just prints to the
>>>>>>> kernel log:
>>>>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>>>>>
>>>>>>>
>>>>>>> Example output of info/MB/:
>>>>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>>>>>> /sys/fs/resctrl/info/MB/num_closids:15
>>>>>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>>>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>>>>>
>>>>>> Add two new MB info RO files:
>>>>>> 1. /sys/fs/resctrl/info/MB/domain_id
>>>>>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
>>>>>
>>>>> This proposal introduces a *global* property to the MB *resource*? It does not seem as though
>>>>> this takes into account *anything* about how resctrl can support new hardware that has been
>>>>> discussed before, during, or after LPC. You have not participated in these discussions and
>>>>> now make an orthogonal proposal that does not take into account *any* of the requirements
>>>>> that we have been struggling with for months.
>>>>>
>>>>> Why should this proposal be taken seriously? In your absence folks have been trying to
>>>>> accommodate how these upcoming products and be supported and the "scope" file associated with
>>>>> a control is intended to communicate to user space how the domain ID should be interpreted.
>>>>>
>>>>> Why are you proposing something entirely different here without even acknowledging current
>>>>> approach and explaining why it does not work for you?
>>>>>
>>>>
>>>> So can I change this part to adding the following files in info dirctory?
>>>>
>>>> 1. For numa memory bw allocation (MBN):
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
>>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
>>>
>>> This is not just about adding files to the info directory. The files, directories, their relationships,
>>> and content have meaning. All I see from these proposals is an attempt to slap some new files into
>>> resctrl without any consideration to present consistent interface to users and without consideration of
>>> other architectures that need to be supported by resctrl.
>>>
>>> resctrl needs to provide a generic and consistent interface to user space irrespective of the
>>> underlying architecture. Architectures cannot just slap some new files for their convenience.
>>>
>>>>
>>>>>> 2. /sys/fs/resctrl/info/MB/max_lim
>>>>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
>>>>>
>>>>> Again this adds another *global* property to the MB resource but then above you
>>>>> describe the new "MB_HLIM" schemata file entry that implies that it is a new control
>>>>> for the MB resource. Having it be a new control for the MB resource matches earlier
>>>>> discussions. To support this I thus expect it to be exposed as a new control with
>>>>> potentially a new type if any of the existing planned types do not suffice.
>>>>>
>>>>
>>>> How about adding these MB_HLIM dir and files in info?
>>>>
>>>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
>>>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
>>>
>>> This presents "MB_HLIM" as a *resource* to user space. It is not a resource
>>> but a *control* of a resource, no? I thus expect it to instead look something like
>>> below that makes it clear that MB_HARDMAX is a control of the MB resource.
>>>
>>> info
>>> └── MB
>>>      └── resource_schemata
>>>          ├── MB
>>>          └── MB_HARDMAX
>>
>> Yes, this makes sense. I have changed to this hierarchy.
> 
> Thank you very much for considering this approach.
> 
> Reinette
> 
>