Re: [PATCH] fs/resctrl,x86/resctrl: Factor mba rounding to be per-arch

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Dave,

On 9/30/25 8:40 AM, Dave Martin wrote:
> On Mon, Sep 29, 2025 at 09:09:35AM -0700, Reinette Chatre wrote:
>> On 9/29/25 6:56 AM, Dave Martin wrote:
>>> On Thu, Sep 25, 2025 at 03:58:35PM -0700, Luck, Tony wrote:
>>>> On Mon, Sep 22, 2025 at 04:04:40PM +0100, Dave Martin wrote:
>>
>> ...
>>
>>>> The region aware h/w supports separate bandwidth controls for each
>>>> region. We could hope (or perhaps update the spec to define) that
>>>> region0 is always node-local DDR memory and keep the "MB" tag for
>>>> that.
>>>
>>> Do you have concerns about existing software choking on the #-prefixed
>>> lines?
>>
>> I am trying to understand the purpose of the #-prefix. I see two motivations
>> for the #-prefix with the primary point that multiple schema apply to the same
>> resource. 
>>
>> 1) Commented schema are "inactive"
>> This is unclear to me. In the MB example the commented lines show the 
>> finer grained controls. Since the original MB resource is an approximation
>> and the hardware must already be configured to support it, would the #-prefixed
>> lines not show the actual "active" configuration?
> 
> They would show the active configuration (possibly more precisely than
> "MB" does).

That is how I see it also. This is specific to MB as we try to maintain
backward compatibility.

If we are going to make user interface changes to resource allocation then
ideally it should consider all known future usage. I am trying to navigate
and understand the discussion on how resctrl can support MPAM and this
RDT region aware requirements. 

I scanned the MPAM spec and from what I understand a resource may support
multiple controls at the same time, each with its own properties, and then
there was this:

	When multiple partitioning controls are active, each affects the partition’s
	bandwidth usage. However, some combinations of controls may not make sense,
	because the regulation of that pair of controls cannot be made to work in concert.

resctrl may thus present an "active configuration" that is not a configuration
that "makes sense" ... this may be ok as resctrl would present what hardware
supports combined with what user requested.

> If not, it's not clear how userspace that is trying to use MB_HW (say)
> could read out the current configuration.
> 
> The # is intended to make resctrl ignore the lines when the file
> is written by userspace.  This is done so that userspace has to
> actually change those lines in order for them to take effect when
> writing.  Old userspace can just pass them through without modification,
> without anything unexpected happening.

Thank you for highlighting this. I did not consider this use case.

> 
> The reason why I think that this convention may be needed is that we
> never told (old) userspace what it was supposed to do with schemata
> entries that it does not recognise.
> 
> 
>> 2) Commented schema are "conflicting"
>> The original proposal mentioned "write them back instead of (or in addition to)
>> the conflicting entries". I do not know how resctrl will be able to
>> handle a user requesting a change to both "MB" and "MB_HW". This seems like
>> something that should fail?
> 
> If userspace is asking for two incompatible things at the same time, we
> can either pick one of them and ignore the rest, or do nothing, or fail
> explicitly.
> 
> If we think that it doesn't really matter what happens, then resctrl
> could just dumbly process the entries in the order given.  If the
> result is not what userspace wanted, that's not our problem.
> 
> (Today, nothing prevents userspace writing multiple "MB" lines at the
> same time: resctrl will process them all, but only the final one will
> have a lasting effect.  So, the fact that a resctrl write can contain
> mutually incompatible requests does not seem to be new.)

Good point.

> 
> 
>> On a high level it is not clear to me why the # prefix is needed. As I understand the
>> schemata names will always be unique and the new features made backward
>> compatible to existing schemata names. That is, existing MB, L3, etc.
>> will also have the new info files that describe their values/ranges.
> 
> Regarding backwards compatibility for the existing controls:
> 
> This proposal is only about numeric controls.  L3 wouldn't change, but
> we could still add info/ metadata for bitmap control at the same time
> as adding it for numeric controls.

I think we should. At least we should leave space for such an addition since
it is not obvious to me how multiple resources with different controls or
single resource with multiple controls should be communicated to user space. 

To be specific, the original proposal [1] introduced a set of files for
a numeric control and that seems to work for existing and upcoming 
schema that need a value in a range. Different controls need different
parameters so to integrate this solution I think it needs another parameter
(presented as a directory, a file, or within a file) that indicates the
type of the control so that user space knows which files/parameters to expect
and how to interpret them. 

Since different controls have different parameters we need to consider
whether it is easier to create/parse unique files for each control or
present all the parameters within one file with another file noting the type
of control.

I understand the files/parameters are intended to be in the schema's info directory
but how this will look is not obvious to me. Part of the MPAM refactoring transitioned
the top level info directories to represent the schema entries that currently reflect
the resources. When we start having multiple schema entries (multiple controls) for a
single resource the simplest implementation may result in a top level info
directory for every schema entry ... but the expectation is that these top
level directories should be per resource, no?

At this time I am envisioning the proposal to result in something like below where
there is one resource directory and one directory per schema entry with a (added by me)
"schema_type" file to help user find out what the schema type is to know which files are present:

MB
├── bandwidth_gran
├── delay_linear
├── MB
│   ├── map
│   ├── max
│   ├── min
│   ├── scale
│   ├── schema_type
│   └── unit
├── MB_HW
│   ├── map
│   ├── max
│   ├── min
│   ├── scale
│   ├── schema_type
│   └── unit
├── MB_MAX
│   └── tbd
├── MB_MIN
│   └── tbd
├── min_bandwidth
├── num_closids
└── thread_throttle_mode

Something else related to control that caught my eye in MPAM spec is this gem:
	MPAM provides discoverable vendor extensions to permit partners
	to invent partitioning controls.
 
> MB may be hard to describe in a useful way, though -- at least in the
> MPAM case, where the number of steps does not divide into 100, and the
> AMD cases where the meaning of the MB control values is different.

Above I do assume that MB would be represented in a new interface since it
is a schema entry, if that causes trouble then we could drop it.

> 
> MB and MB_HW are not interchangeable.  To obtain predictable results
> from MB, userspace would need to know precisely how the kernel is going
> to round the value.  This feels like an implementation detail that
> doesn't belong in the ABI.

ack

...

> Anyway, going back to the "#" convention:
> 
> If the initial read of schemata has the new entries "pre-commented",
> then userspace wouldn't need to know about the new entries.  It could
> just tweak the MB entry (which it knows about), and write the file back:
> 
> 	MB: 0=43
> 	# MB_HW: 0=2
> 	# MB_MIN: 0=1
> 	# MB_MAX: 0=2
> 
> then resctrl knows to ignore the hashed lines, and so reading the file
> back gives:
> 
> 	MB: 0=43
> 	# MB_HW: 0=3
> 	# MB_MIN: 0=2
> 	# MB_MAX: 0=3

Thank you for the example. This seems reasonable. I would like to go back
to what you wrote in [1]:

> Software that understands the new entries can uncomment the conflicting
> entries and write them back instead of (or in addition to) the
> conflicting entries.  For example, userspace might write the following:
> 
> MB_MIN: 0=16, 1=16
> MB_MAX: 0=32, 1=32
> 
> Which might then read back as follows:
> 
> MB: 0=50, 1=50
> # MB_HW: 0=32, 1=32
> # MB_MIN: 0=16, 1=16
> # MB_MAX: 0=32, 1=32

Could/should resctrl uncomment the lines after userspace modified them?

> 
> (For hardware-specific reasons, the MPAM driver currently internally
> programs the MIN bound to be a bit less than the MAX bound, when
> userspace writes an "MB" entry into schemata.  The key thing is that
> writing MB may cause the MB_MIN/MB_MAX entries to change -- at the
> resctrl level, I don't that that we necessarily need to make promises
> about what they can change _to_.  The exact effect of MIN and MAX
> bounds is likely to be hardware-dependent anyway.)

MPAM has the "HARDLIM" distinction associated with these MAX values
and from what I can tell this is per PARTID. Is this something that needs
to be supported? To do this resctrl will need to support modifying
control properties per resource group.

> 
> 
> Regarding new userspce:
> 
> Going forward, we can explicitly document that there should be no
> conflicting or "passenger" entries in a schemata write: don't include
> an entry for somehing that you don't explicitly want to set, and if
> multiple entries affect the same resource, we don't promise what
> happens.
> 
> (But sadly, we can't impose that rule on existing software after the
> fact.)

It may thus not be worth it to make such a rule.

> 
> 
> One final note: I have not provided any way to indicate that all those
> entries control the same hardware resource.  The common "MB" prefix is
> intended as a clue, but ultimately, userspace needs to know what an
> entry controls before tweaking it.
> 
> We could try to describe the relationships explicitly, but I'm not sure
> that it is useful...

What other relationships should we consider for MPAM? I see that each
MPAM allows per-PARTID configurations for secure/non-secure, physical/virtual,
... ? Is it expected that MPAM's support of these should be exposed via resctrl?
Have you considered how to express if user wants hardware to have different
allocations for, for example, same PARTID at different execution levels?

Reinette

[1] https://lore.kernel.org/lkml/aNFliMZTTUiXyZzd@e133380.arm.com/