Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Ben, Chenyu, and Tony,

On 7/16/26 7:59 AM, Ben Horgan wrote:
> On 7/15/26 16:41, Reinette Chatre wrote:
>> On 7/15/26 1:34 AM, Ben Horgan wrote:
>>> On 7/14/26 23:06, Reinette Chatre wrote:

...

>>>> Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want
>>>> to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface
>>>> may end up looking like:
>>>>
>>>> info/
>>>> └── MB/
>>>>     └── resource_schemata/
>>>>         ├── MB/
>>>>         │   ├── MB_MAX/
>>>>         │   │   └── scope:L3
>>>>         │   └── scope:L3
>>>>         └── MB_NODE/
>>>>             └── scope:NODE
>>>>
>>>> A system like above will thus have three schemata file entries:
>>>> MB
>>>> MB_MAX
>>>> MB_NODE
>>>>
>>>> Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control
>>>> so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE
>>>> on these systems.
>>>>> Would that work for these systems?
>>>
>>> I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume
>>> we should expect that the user might read back the whole file, modify what they expect to change and
>>> then write everything back. If legacy software does this and wants to change the MB schemata then
>>> the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and
>>> fail the write if two dependent schemata are changed at the same time?
>>
>> An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file
>> with "#":
>> https://lore.kernel.org/lkml/aNv53UmFGDBL0z3O@e133380.arm.com/
> 
> I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was
> being taken forward.
> 
>> This was one element of original discussion that fizzled a bit since it was not obvious that user
>> space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:
>>
>> 	Reading/writing the schemata file
>> 	---------------------------------
>> 	... When writing you only need to specify those values which you wish to change. ..
> 
> Yeah, I don't know but it does seem like something a user could reasonably expect to work.

ok. 

>>> Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
>>> need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
>>> "mode" and can just toggle back if they want to use "MB" directly again.
>> Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
>> legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
>> resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
>> into the controls it is being emulated with. The only insight to this would continue to be via the
>> hierarchy in the info directory, which would show relationship but not the actual control values.
>> Do you think there could be a need for users (excluding validation?) that may want to see the underlying
>> control values used to emulate a legacy control?
> 
> They may want to see the underlying values to be able to migrate there legacy configuration to the
> native configuration but they can just write their legacy configuration and then toggle the mode to
> see the values in the native mode.
> 
>>
>> I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
>> here would be that users familiar with the new controls would forever need to switch the mode before being
>> able to use them.
> 
> This does also make the new controls slightly harder to discover.
> 
What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be
"legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
easier to discover and let user space see the underlying values, but not break a user space that
may read schemata file, change a few values, and write entire file back.

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
MPAM and region aware MBA?

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

MPAM
====
1) System default

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
  TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.
- MB_MAX control values would show the current active control values of MB_MAX control used to emulate the MB control

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_MAX:...

- Legacy control not displayed at all
- no # prefix

RDT region aware when using MSR interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:disabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...


RDT region aware when using ACPI interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_REGION0_MAX:...
# MB_REGION1_MAX:...
# MB_REGION2_MAX:...
# MB_REGION3_MAX:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/
└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...

Reinette