Resctrl and I/O

Resctrl and I/O

[Luck, Tony]

Patches for ARM[1] and RISC-V[2] are both at RFC status.

They appear to be based on similar h/w capabilities where architecture
specific tags can be added to transactions from I/O devices by the IOMMU
servicing the device. Device enumeration is handled outside of the resctrl
file system.

ARM docs say "the device name is the one listed under /sys/kernel/iommu_groups/<id>/devices/"

RISC-V says architecture specific device tokens are "devices or device groups"

In both cases assignments are made with:

	# echo {device} > {group}/devices

So now I'm looking at the capabilities of Intel RDT hardware. I'm ignoring
the implementation on the Granite Rapids CPU, and looking at what is described
in the RDT Architecture Specification[3]. Here are the highlights:

1) Enumeration is split between two ACPI tables. IRDT enumerates device groups
and provides MMIO addresses to assign CLOSID/RMID to each group. ERDT provides
MMIO addresses to read event counts for RMIDs and to configure cache allocation
bit masks and memory bandwidth limits.

2) All I/O monitoring and control is with respect to I/O caches, not to the L3
caches that legacy RDT features operate on.

3) The number of RMIDs supported for I/O is lower (maybe considearbly lower)
than the number supported for CPU monitoring.

4) Monitoring events are I/O cache occupancy, total i/o bandwidth, and i/o miss
bandwidth.

5) Current revision (1.4) of the specification doesn't include details of
I/O memory bandwidth control (The "IBAD" subtable of ERDT is reserved for this).


Fitting this into resctrl
-------------------------

RISC-V description of "architecture specific device tokens" sounds great.
I'll have to find some ares of /sys to provide the mapping of tokens to
"device groups". There's no association with IOMMUs, so this will be some
other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
from ACPI IRDT table.

Need an info file to tell users the format and source for device tokens.

I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
events} files in resctrl group directories when rdtgroup::mon::rmid >=
num_io_rmids? But that feels clumsy.

Domains are based on I/O L3 cache instances. So a new rdt_resource with
new line in the schemata file for IOL3 cache masks per domain. In the
future when there are I/O MBA controls, new line(s) for those too.

-Tony


[1] https://lore.kernel.org/linux-iommu/20260901140802.1215508-1-xiaqinxin@huawei.com/
[2] https://lore.kernel.org/lkml/20260714130657.46963-1-zhangzhanpeng.jasper@bytedance.com/
[3] Linked from the bottom of https://www.intel.com/sdm under "Related Specifications", or
direct link: https://cdrdv2.intel.com/v1/dl/getContent/789566?erpm_id=1158389_ts1790091453223

Re: Resctrl and I/O

[Drew Fustini]

On Tue, Sep 22, 2026 at 09:51:11AM -0700, Luck, Tony wrote:
> Patches for ARM[1] and RISC-V[2] are both at RFC status.
> 
> They appear to be based on similar h/w capabilities where architecture
> specific tags can be added to transactions from I/O devices by the IOMMU
> servicing the device. Device enumeration is handled outside of the resctrl
> file system.

Thanks for starting a discussion on this topic.

> RISC-V description of "architecture specific device tokens" sounds great.
> I'll have to find some ares of /sys to provide the mapping of tokens to
> "device groups". There's no association with IOMMUs, so this will be some
> other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
> from ACPI IRDT table.

resctrl could parse iommu_group:<id> itself which should cover the IOMMU
case on ARM and RISC-V. Maybe other prefixes could be passed on to arch
code?

> Need an info file to tell users the format and source for device tokens.

I agree and it could list which token namespaces the platform uses.

> I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
> number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
> events} files in resctrl group directories when rdtgroup::mon::rmid >=
> num_io_rmids? But that feels clumsy.

There's no separate I/O ID space on RISC-V that could be smaller. The
IDs a device carries are the same IDs the hart (e.g. cpu) carries. The
number of RCIDs (e.g. CLOSID) and MCIDs (e.g. RMID) is a property of
each capacity or bandwidth (CBQRI) controller. The RISC-V IOMMU spec
also requires the IOMMU to support RCID and MCID at least as wide as any
hart.

Thanks,
Drew