Re: [PATCH v4 1/6] x86/resctrl: Parse ACPI ERDT table and map RMDD domains by L3 cache ID

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Chenyu,

On 6/23/26 11:04 PM, Chen, Yu C wrote:
> On 6/24/2026 12:05 PM, Chen, Yu C wrote:
>> On 6/24/2026 12:46 AM, Reinette Chatre wrote:
>>> Hi Chenyu,
>>>
>>> On 6/23/26 4:43 AM, Chen, Yu C wrote:
>>>> On 6/23/2026 2:29 PM, Chen, Yu C wrote:
>>>>>>>>    From what I can tell it is tmp that is essentially returned by this function and it
>>>>>>>> is the result of get_cpu_cacheinfo_id() that returns the cache ID initialized from
>>>>>>>> CPUID leaf 0x4 ... the function name ("get_l3_cache_id_from_cacd()") and function comment
>>>>>>>> "Resolve L3 cache ID from CACD subtable" implies differently and the way this is done
>>>>>>>> makes me wonder why CACD needs to be parsed at all.
>>>>>>>>
>>>>>>>> resctrl already has a hotplug handler and it dynamically determines the domain ID
>>>>>>>> based on the cache ID returned from get_cpu_cacheinfo_id(). If the domain ID used by the
>>>>>>>> new ERDT tables can be, as it appears is done here, trusted to what Linux already
>>>>>>>> treating as domain ID then why is it needed to parse CACD at all? Why not just
>>>>>>>> pick the domain ID from RMDD directly and use it directly as domain ID?
>>>>>>>>
>>>>>>>> Later the domain ID from RMDD is just used in error messages ... never actually used
>>>>>>>> as a domain ID to guide the implementation so the correlation appears to be implicit with
>>>>>>>> this CACD parsing not intuitive.
>>>>>>>>
>>>>>>>
>>>>>>> You are right that the cache ID ultimately comes from get_cpu_cacheinfo_id(),
>>>>>>> not from CACD directly - the function name is misleading and should be improved.
>>>>>>>
>>>>>>> However, my understanding is that CACD parsing acts as the bridge between ACPI's
>>>>>>> RMDD domain namespace and the kernel's L3 domain ID: CACD enumerates which CPUs
>>>>>>> belong to each RMDD, and from those CPUs we derive the kernel-side L3 cache ID.
>>>>>>> The specification defines an RMDD DomainID as an identifier unique within the
>>>>>>> ERDT table, yet it does not guarantee that this ID matches the kernel’s cache ID
>>>>>>> obtained via CPUID leaf 0x4. For this reason, we omitted the RMDD domain ID in
>>>>>>> the current release.
>>>>>>
>>>>>> get_l3_cache_id_from_cacd() implies a straightforward query and the function
>>>>>> comments supports this. At the same time the function includes what appears to be
>>>>>> sanity checks but from what you say turns out to be essential enforcement of the
>>>>>> RMDD to CPUID leaf 0x4 mapping. Is this correct? As sanity checks these additional
>>>>>> checks are ok but as RMDD to CPUID leaf 0x4 mapping enforcement I find them inadequate
>>>>>> because its correctness requires all CPUs of all domains to be online which cannot be
>>>>>> guaranteed when this initialization runs.
>>>>>>
>>>>>> Consider for example a scenario where RMDD and CPUID do not agree on which CPUs form
>>>>>> part of a domain. If only one CPU of this RMDD domain is online at the time this
>>>>>> initialization is done then the mismatch will never be noticed and resctrl will just
>>>>>> silently use the CPUID mapping, no?
>>>>>>
>>>>>
>>>>> I suppose you are referring to the following scenario:
>>>>> CACD says CPUs {0,1,2,3} belong to RMDD-A, and CPUID says
>>>>> CPUs {0,1} share L3-X while CPUs {2,3} share L3-Y (a mismatch).
>>>>> If only CPU 0 is brought online:
>>>>>
>>>>> get_l3_cache_id_from_cacd() iterates CACD's X2APIC list: {0,1,2,3}
>>>>> CPU 1, 2, 3 are offline -> skipped
>>>>> CPU 0 is online -> cache_id = get_cpu_cacheinfo_id(0) = L3-X,
>>>>> all passed.
>>>>>
>>>>> Then later CPU2 is online, resctrl's hotplug creates domain L3-Y.
>>>>> But the xarray has no entry for L3-Y, so erdt_mon_read() will return
>>>>> -EIO for that domain.
>>>
>>> An error would at least give indication of a problem but I also see a possible
>>> scenario where the data will happily be read from the wrong register. Consider the
>>> following change to the scenario you present:
>>> CACD: RMDD-A = {0, 2}; RMDD-B = {1, 3}
>>> CPUID: L3-X = {0, 1}; L3-Y = {2, 3}
>>>
>>> If ACPI enumeration is done with CPUs 1, 2, and 3 offline then they are skipped.
>>> CPU 0 is online -> cache_id = get_cpu_cacheinfo_id(0) = L3-X
>>>
>>> Later when CPU 1 comes online resctrl will consider it to be in L3-X and
>>> also find an xarray entry for L3-X which will result in erdt_mon_read() to
>>> read the RMID data from RMDD-A instead of RMDD-B?
>>>
>>
>> Yes, this approach is feasible. However, I am uncertain whether RDT should
>> be enabled. The underlying issue is that the firmware(BIOS) and silicon(CPUID)
>> will be out of sync, leaving RDT in an unstable state.
>>
> 
> That said, there remains an open question as to whether we ought to enable
> RDT on platforms where the domain information returned by firmware (BIOS)
> is inconsistent with the CPUID values exposed by the silicon.
I consider a mismatch between BIOS and CPUID to be a hardware bug and should
never happen. If it does resctrl can complain loudly (via WARN) and not
attempt to use the conflicting resources. I do not see how resctrl could
"not enable RDT" in this scenario since a mismatch may only be discovered
during CPU online that may only occur some time after resctrl is mounted.

On 6/23/26 9:05 PM, Chen, Yu C wrote:
...
> 
> Thanks for the detailed feedback. If I understand correctly, separating
> ACPI enumeration from CPUID enumeration would look something like the following.
> Could you let me know if this approach is feasible?
> 
> ----------
> 1. At ACPI parse time , record the full CPU mask from CACD for each
>    RMDD domain without checking online state or resolving L3 cache IDs.

ack.

>    Store a per-CPU pointer (erdt_cpu_domain[cpu]) for O(1) lookup.

Why is such optimization needed? From what I can tell this pointer is
only accessed *once*. I do not think onlining a CPU is a hot path?
Could this be simplified to just be a list of ERDT domains? 

> 2. At CPU online time (resctrl_arch_online_cpu), validate that the CPU's
>    CPUID-derived L3 ID is consistent with other CPUs in the same RMDD:
>    - First online CPU in an RMDD establishes the L3 ID for that domain
>      and inserts the domain into erdt_domain_xa keyed by L3 cache ID.
>    - Subsequent CPUs must match. On mismatch, the CPU is refused from resctrl.

Considering that this now establishes 1:1 between RMDD and resctrl domain and that
resctrl already maintains a list of domains, could the x86 arch specific monitoring
domain (struct rdt_hw_l3_mon_domain) perhaps just point to the ERDT monitoring data
for that domain instead of maintaining a separate array of domains? So, when x86 is
requested to read monitoring data for a domain there would be no need to query an
array for the ERDT domain info since the containing structure already has a pointer
to it.

This may need more care to organize the ERDT per-domain monitoring data structure when
considering the references to upcoming additions that are not clear to me at this time.

> 
> 3. erdt_mon_read() looks up erdt_domain_xa by hdr->id (L3 cache ID) as before,
>    but the xarray is now populated lazily at CPU online time.

I do not see the need for an xarray. As new domains are created during CPU online their
initialization can just point to the needed data, no?

Reinette