Resctrl and I/O

Resctrl and I/O

[Luck, Tony]

Patches for ARM[1] and RISC-V[2] are both at RFC status.

They appear to be based on similar h/w capabilities where architecture
specific tags can be added to transactions from I/O devices by the IOMMU
servicing the device. Device enumeration is handled outside of the resctrl
file system.

ARM docs say "the device name is the one listed under /sys/kernel/iommu_groups/<id>/devices/"

RISC-V says architecture specific device tokens are "devices or device groups"

In both cases assignments are made with:

	# echo {device} > {group}/devices

So now I'm looking at the capabilities of Intel RDT hardware. I'm ignoring
the implementation on the Granite Rapids CPU, and looking at what is described
in the RDT Architecture Specification[3]. Here are the highlights:

1) Enumeration is split between two ACPI tables. IRDT enumerates device groups
and provides MMIO addresses to assign CLOSID/RMID to each group. ERDT provides
MMIO addresses to read event counts for RMIDs and to configure cache allocation
bit masks and memory bandwidth limits.

2) All I/O monitoring and control is with respect to I/O caches, not to the L3
caches that legacy RDT features operate on.

3) The number of RMIDs supported for I/O is lower (maybe considearbly lower)
than the number supported for CPU monitoring.

4) Monitoring events are I/O cache occupancy, total i/o bandwidth, and i/o miss
bandwidth.

5) Current revision (1.4) of the specification doesn't include details of
I/O memory bandwidth control (The "IBAD" subtable of ERDT is reserved for this).


Fitting this into resctrl
-------------------------

RISC-V description of "architecture specific device tokens" sounds great.
I'll have to find some ares of /sys to provide the mapping of tokens to
"device groups". There's no association with IOMMUs, so this will be some
other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
from ACPI IRDT table.

Need an info file to tell users the format and source for device tokens.

I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
events} files in resctrl group directories when rdtgroup::mon::rmid >=
num_io_rmids? But that feels clumsy.

Domains are based on I/O L3 cache instances. So a new rdt_resource with
new line in the schemata file for IOL3 cache masks per domain. In the
future when there are I/O MBA controls, new line(s) for those too.

-Tony


[1] https://lore.kernel.org/linux-iommu/20260901140802.1215508-1-xiaqinxin@huawei.com/
[2] https://lore.kernel.org/lkml/20260714130657.46963-1-zhangzhanpeng.jasper@bytedance.com/
[3] Linked from the bottom of https://www.intel.com/sdm under "Related Specifications", or
direct link: https://cdrdv2.intel.com/v1/dl/getContent/789566?erpm_id=1158389_ts1790091453223

Re: Resctrl and I/O

[Drew Fustini]

On Tue, Sep 22, 2026 at 09:51:11AM -0700, Luck, Tony wrote:
> Patches for ARM[1] and RISC-V[2] are both at RFC status.
> 
> They appear to be based on similar h/w capabilities where architecture
> specific tags can be added to transactions from I/O devices by the IOMMU
> servicing the device. Device enumeration is handled outside of the resctrl
> file system.

Thanks for starting a discussion on this topic.

> RISC-V description of "architecture specific device tokens" sounds great.
> I'll have to find some ares of /sys to provide the mapping of tokens to
> "device groups". There's no association with IOMMUs, so this will be some
> other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
> from ACPI IRDT table.

resctrl could parse iommu_group:<id> itself which should cover the IOMMU
case on ARM and RISC-V. Maybe other prefixes could be passed on to arch
code?

> Need an info file to tell users the format and source for device tokens.

I agree and it could list which token namespaces the platform uses.

> I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
> number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
> events} files in resctrl group directories when rdtgroup::mon::rmid >=
> num_io_rmids? But that feels clumsy.

There's no separate I/O ID space on RISC-V that could be smaller. The
IDs a device carries are the same IDs the hart (e.g. cpu) carries. The
number of RCIDs (e.g. CLOSID) and MCIDs (e.g. RMID) is a property of
each capacity or bandwidth (CBQRI) controller. The RISC-V IOMMU spec
also requires the IOMMU to support RCID and MCID at least as wide as any
hart.

Thanks,
Drew

Re: Resctrl and I/O

[James Morse]

Hi Tony,

On 22/09/2026 17:51, Luck, Tony wrote:
> Patches for ARM[1] and RISC-V[2] are both at RFC status.
> 
> They appear to be based on similar h/w capabilities where architecture
> specific tags can be added to transactions from I/O devices by the IOMMU
> servicing the device. Device enumeration is handled outside of the resctrl
> file system.
> 
> ARM docs say "the device name is the one listed under /sys/kernel/iommu_groups/<id>/devices/"

I think using existing stuff in sysfs is a good idea.
(Creating new things in sysfs fills me with dread!)

ARM/MPAM also has platform devices that can add the CLOSID/RMID metadata.
e.g. the GIC ITS [4] - a kind of translator for MSIs.

For these we considered just using the device name as found in /sys/bus/platform/devices/,
in the hope that can be tied back to a struct device - and we could add a callback for
resctrl to configure the device with a CLOSID/RMID.
(but - I've not seen a use-case for this yet - those devices generate very little traffic)


Robin M set me straight on the iommu_group:
Strictly that is the wrong abstraction as devices are in the same group if the
driver can't stop them communicating, e.g. PCIe peer2peer. This doesn't mean it
can't tell the traffic apart for CLOSID/RMID labelling.
I picked this because its the device granularity VFIO works with.

Equally, a device may create multiple streams of traffic, where only the driver
will know what each one corresponds to.


Using iommu_group like this may need a bit more thinking....
If we can fall back to using a device as seen by sysfs, then the arch differences
here could be invisible to user-space.


> RISC-V says architecture specific device tokens are "devices or device groups"
> 
> In both cases assignments are made with:
> 
> 	# echo {device} > {group}/devices
> 
> So now I'm looking at the capabilities of Intel RDT hardware. I'm ignoring
> the implementation on the Granite Rapids CPU, and looking at what is described
> in the RDT Architecture Specification[3]. Here are the highlights:


> 1) Enumeration is split between two ACPI tables. IRDT enumerates device groups
> and provides MMIO addresses to assign CLOSID/RMID to each group.

For ARM/MPAM - this would always be driver specific. The fields may be share a register
with other things the driver needs to consider.
The IOMMU abstracts this for its devices, but this is still something the IOMMU driver
needs to know about.

A driver hook, with a drivers/acpi fallback would be ideal.


Do you have arbitrary grouping of devices? e.g. a USB controller and the UART.
Or would the grouping make some sense to the OS. e.g. all the USB devices behind
this port.

Having the UART show up when you add a USB controller to a control group is a bit odd.


> ERDT provides
> MMIO addresses to read event counts for RMIDs and to configure cache allocation
> bit masks and memory bandwidth limits.

Arm's IOMMU does have controls/monitors that match the spec the MPAM driver needs,
but we currently have no way to expose them through resctrl.
The caches on the IOMMU are also relatively small - I've not yet seen anyone show that
setting the controls there makes a big difference.


> 2) All I/O monitoring and control is with respect to I/O caches, not to the L3
> caches that legacy RDT features operate on.

Do you have I/O traffic that can turn up in the CPU L3 cache? I think PCIe calls
this 'cache stashing'.

Arm's IOMMU is the same - but the traffic may also show up in the L3 depending on how
the system is put together.


> 3) The number of RMIDs supported for I/O is lower (maybe considearbly lower)
> than the number supported for CPU monitoring.

MPAM can have this happen - and today it will sanitise the values to the lowest
common value. Considerably lower becomes considerably upsetting!

I'd be interested if we have an approach for this - the IOMMU designers would argue
they only need one RMID per device - but the system wide sanitised value means they
have to match the rest of the system.


For MPAM systems this shape is a big headache because on the platforms we've seen,
the monitoring is done at the memory-controllers - so any RMID spaces get joined
up eventually.


> 4) Monitoring events are I/O cache occupancy, total i/o bandwidth, and i/o miss
> bandwidth.
> 
> 5) Current revision (1.4) of the specification doesn't include details of
> I/O memory bandwidth control (The "IBAD" subtable of ERDT is reserved for this).
> 
> 
> Fitting this into resctrl
> -------------------------
> 
> RISC-V description of "architecture specific device tokens" sounds great.
> I'll have to find some ares of /sys to provide the mapping of tokens to
> "device groups".
> There's no association with IOMMUs, so this will be some
> other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
> from ACPI IRDT table.

What about using the device behind the IOMMU instead? Presumably the kernel
could allocate ids that don't clash with those in the IRDT, and use those
for the IOMMU.


> Need an info file to tell users the format and source for device tokens.

I'd prefer we have one consistent way of doing this across architectures.
Especially if we're going to add something new to sysfs.


The sysfs path probably covers all bases - but I wouldn't want to parse that
in the kernel.


> I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
> number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
> events} files in resctrl group directories when rdtgroup::mon::rmid >=
> num_io_rmids? But that feels clumsy.

I had a half baked idea on how to do this. (which Ben has poked some holes in)

Assigning a device to a ctrl/mon group needs to be able to fail. We can use this
to re-allocate the RMID into the range the new device can use. The CPU tasks can
be repainted to use the new value - but the obvious problem is the old data in
the caches that uses the old RMID.
As no new traffic will be generated for that RMID, it might be possible to keep
it associated with the ctrl/mon group (which would now have two RMID) to keep the
user-space view of the counters correct until the old RMID falls below the
reallocation threshold, and it can be free'd.

It's worse if we ever needed/wanted to do this with CLOSID. Two CAT bitmaps
can be configured the same - but MBA doesn't alias in the same way. Tasks would
be able to burst higher than the configured limit - but the size is limited by
the size of the caches. And if no new traffic can be generated with the old
CLOSID/RMID, then the effect can't be made to last forever.


For a totally wild suggestion (that I don't think works here) - I had
mooted having a second mount of resctrl for islands of the SoC where the
properties are different. I don't think it works in this case as the IO-RMID
presumably gets seen by some other monitors later.


> Domains are based on I/O L3 cache instances. So a new rdt_resource with
> new line in the schemata file for IOL3 cache masks per domain. In the
> future when there are I/O MBA controls, new line(s) for those too.


Thanks,

James


[4] https://support.arm.com/documentation/ihi0069/hb/?lang=en search for "MPAM" in the pdf.

> 
> [1] https://lore.kernel.org/linux-iommu/20260901140802.1215508-1-xiaqinxin@huawei.com/
> [2] https://lore.kernel.org/lkml/20260714130657.46963-1-zhangzhanpeng.jasper@bytedance.com/
> [3] Linked from the bottom of https://www.intel.com/sdm under "Related Specifications", or
> direct link: https://cdrdv2.intel.com/v1/dl/getContent/789566?erpm_id=1158389_ts1790091453223

Re: Resctrl and I/O

[Jason Gunthorpe]

On Wed, Sep 23, 2026 at 06:13:27PM +0100, James Morse wrote:

> Using iommu_group like this may need a bit more thinking....
> If we can fall back to using a device as seen by sysfs, then the arch differences
> here could be invisible to user-space.

Please don't codify iommu_groups in any new uAPI, that has proven to
be a problematic choice for VFIO. We undid it.. Lets not repeat it.

Jason

Re: [External] Re: Resctrl and I/O

[yunhui cui]

Hi James,

On Thu, Sep 24, 2026 at 1:13 AM James Morse <james.morse@arm.com> wrote:
>
> Hi Tony,
>
> On 22/09/2026 17:51, Luck, Tony wrote:
> > Patches for ARM[1] and RISC-V[2] are both at RFC status.
> >
> > They appear to be based on similar h/w capabilities where architecture
> > specific tags can be added to transactions from I/O devices by the IOMMU
> > servicing the device. Device enumeration is handled outside of the resctrl
> > file system.
> >
> > ARM docs say "the device name is the one listed under /sys/kernel/iommu_groups/<id>/devices/"
>
> I think using existing stuff in sysfs is a good idea.
> (Creating new things in sysfs fills me with dread!)
>
> ARM/MPAM also has platform devices that can add the CLOSID/RMID metadata.
> e.g. the GIC ITS [4] - a kind of translator for MSIs.
>
> For these we considered just using the device name as found in /sys/bus/platform/devices/,
> in the hope that can be tied back to a struct device - and we could add a callback for
> resctrl to configure the device with a CLOSID/RMID.
> (but - I've not seen a use-case for this yet - those devices generate very little traffic)
>
>
> Robin M set me straight on the iommu_group:
> Strictly that is the wrong abstraction as devices are in the same group if the
> driver can't stop them communicating, e.g. PCIe peer2peer. This doesn't mean it
> can't tell the traffic apart for CLOSID/RMID labelling.
> I picked this because its the device granularity VFIO works with.
>
> Equally, a device may create multiple streams of traffic, where only the driver
> will know what each one corresponds to.
>
>
> Using iommu_group like this may need a bit more thinking....
> If we can fall back to using a device as seen by sysfs, then the arch differences
> here could be invisible to user-space.
>
>
> > RISC-V says architecture specific device tokens are "devices or device groups"
> >
> > In both cases assignments are made with:
> >
> >       # echo {device} > {group}/devices
> >
> > So now I'm looking at the capabilities of Intel RDT hardware. I'm ignoring
> > the implementation on the Granite Rapids CPU, and looking at what is described
> > in the RDT Architecture Specification[3]. Here are the highlights:
>
>
> > 1) Enumeration is split between two ACPI tables. IRDT enumerates device groups
> > and provides MMIO addresses to assign CLOSID/RMID to each group.
>
> For ARM/MPAM - this would always be driver specific. The fields may be share a register
> with other things the driver needs to consider.
> The IOMMU abstracts this for its devices, but this is still something the IOMMU driver
> needs to know about.
>
> A driver hook, with a drivers/acpi fallback would be ideal.
>
>
> Do you have arbitrary grouping of devices? e.g. a USB controller and the UART.
> Or would the grouping make some sense to the OS. e.g. all the USB devices behind
> this port.
>
> Having the UART show up when you add a USB controller to a control group is a bit odd.
>
>
> > ERDT provides
> > MMIO addresses to read event counts for RMIDs and to configure cache allocation
> > bit masks and memory bandwidth limits.
>
> Arm's IOMMU does have controls/monitors that match the spec the MPAM driver needs,
> but we currently have no way to expose them through resctrl.
> The caches on the IOMMU are also relatively small - I've not yet seen anyone show that
> setting the controls there makes a big difference.
>
>
> > 2) All I/O monitoring and control is with respect to I/O caches, not to the L3
> > caches that legacy RDT features operate on.
>
> Do you have I/O traffic that can turn up in the CPU L3 cache? I think PCIe calls
> this 'cache stashing'.
>
> Arm's IOMMU is the same - but the traffic may also show up in the L3 depending on how
> the system is put together.
>
>
> > 3) The number of RMIDs supported for I/O is lower (maybe considearbly lower)
> > than the number supported for CPU monitoring.
>
> MPAM can have this happen - and today it will sanitise the values to the lowest
> common value. Considerably lower becomes considerably upsetting!
>
> I'd be interested if we have an approach for this - the IOMMU designers would argue
> they only need one RMID per device - but the system wide sanitised value means they
> have to match the rest of the system.
>

  This lowest-common-denominator issue suggests that resctrl may need a
  logical CLOSID/RMID namespace that is independent of the ID space
  implemented by each hardware controller.

  A resctrl group would retain a system-wide logical CLOSID/RMID, while
  each resource or architecture backend maps those logical IDs to its
  controller-local allocation entries and monitoring counters. This would
  avoid unconditionally limiting the number of resctrl groups to the
  minimum ID count across all enabled resources.

  The current resctrl allocation model uses that minimum as the
  system-wide CLOSID limit, which is safe for direct indexing but leaves
  additional entries in larger controllers unused.

  On the RISC-V CBQRI side, we are discussing a hardware mechanism that
  could provide such per-controller mapping while requests continue to
  carry the logical RCIDs/MCIDs:

  https://lists.riscv.org/g/sig-datacenter/message/286


Thanks，
Yunhui

>
> For MPAM systems this shape is a big headache because on the platforms we've seen,
> the monitoring is done at the memory-controllers - so any RMID spaces get joined
> up eventually.
>
>
> > 4) Monitoring events are I/O cache occupancy, total i/o bandwidth, and i/o miss
> > bandwidth.
> >
> > 5) Current revision (1.4) of the specification doesn't include details of
> > I/O memory bandwidth control (The "IBAD" subtable of ERDT is reserved for this).
> >
> >
> > Fitting this into resctrl
> > -------------------------
> >
> > RISC-V description of "architecture specific device tokens" sounds great.
> > I'll have to find some ares of /sys to provide the mapping of tokens to
> > "device groups".
> > There's no association with IOMMUs, so this will be some
> > other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
> > from ACPI IRDT table.
>
> What about using the device behind the IOMMU instead? Presumably the kernel
> could allocate ids that don't clash with those in the IRDT, and use those
> for the IOMMU.
>
>
> > Need an info file to tell users the format and source for device tokens.
>
> I'd prefer we have one consistent way of doing this across architectures.
> Especially if we're going to add something new to sysfs.
>
>
> The sysfs path probably covers all bases - but I wouldn't want to parse that
> in the kernel.
>
>
> > I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
> > number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
> > events} files in resctrl group directories when rdtgroup::mon::rmid >=
> > num_io_rmids? But that feels clumsy.
>
> I had a half baked idea on how to do this. (which Ben has poked some holes in)
>
> Assigning a device to a ctrl/mon group needs to be able to fail. We can use this
> to re-allocate the RMID into the range the new device can use. The CPU tasks can
> be repainted to use the new value - but the obvious problem is the old data in
> the caches that uses the old RMID.
> As no new traffic will be generated for that RMID, it might be possible to keep
> it associated with the ctrl/mon group (which would now have two RMID) to keep the
> user-space view of the counters correct until the old RMID falls below the
> reallocation threshold, and it can be free'd.
>
> It's worse if we ever needed/wanted to do this with CLOSID. Two CAT bitmaps
> can be configured the same - but MBA doesn't alias in the same way. Tasks would
> be able to burst higher than the configured limit - but the size is limited by
> the size of the caches. And if no new traffic can be generated with the old
> CLOSID/RMID, then the effect can't be made to last forever.
>
>
> For a totally wild suggestion (that I don't think works here) - I had
> mooted having a second mount of resctrl for islands of the SoC where the
> properties are different. I don't think it works in this case as the IO-RMID
> presumably gets seen by some other monitors later.
>
>
> > Domains are based on I/O L3 cache instances. So a new rdt_resource with
> > new line in the schemata file for IOL3 cache masks per domain. In the
> > future when there are I/O MBA controls, new line(s) for those too.
>
>
> Thanks,
>
> James
>
>
> [4] https://support.arm.com/documentation/ihi0069/hb/?lang=en search for "MPAM" in the pdf.
>
> >
> > [1] https://lore.kernel.org/linux-iommu/20260901140802.1215508-1-xiaqinxin@huawei.com/
> > [2] https://lore.kernel.org/lkml/20260714130657.46963-1-zhangzhanpeng.jasper@bytedance.com/
> > [3] Linked from the bottom of https://www.intel.com/sdm under "Related Specifications", or
> > direct link: https://cdrdv2.intel.com/v1/dl/getContent/789566?erpm_id=1158389_ts1790091453223
>