Re: [PATCH v4 1/6] x86/resctrl: Parse ACPI ERDT table and map RMDD domains by L3 cache ID

["Chen, Yu C" <yu.c.chen@intel.com>]

On 6/24/2026 12:46 AM, Reinette Chatre wrote:
> Hi Chenyu,
> 
> On 6/23/26 4:43 AM, Chen, Yu C wrote:
>> On 6/23/2026 2:29 PM, Chen, Yu C wrote:
>>>>>>    From what I can tell it is tmp that is essentially returned by this function and it
>>>>>> is the result of get_cpu_cacheinfo_id() that returns the cache ID initialized from
>>>>>> CPUID leaf 0x4 ... the function name ("get_l3_cache_id_from_cacd()") and function comment
>>>>>> "Resolve L3 cache ID from CACD subtable" implies differently and the way this is done
>>>>>> makes me wonder why CACD needs to be parsed at all.
>>>>>>
>>>>>> resctrl already has a hotplug handler and it dynamically determines the domain ID
>>>>>> based on the cache ID returned from get_cpu_cacheinfo_id(). If the domain ID used by the
>>>>>> new ERDT tables can be, as it appears is done here, trusted to what Linux already
>>>>>> treating as domain ID then why is it needed to parse CACD at all? Why not just
>>>>>> pick the domain ID from RMDD directly and use it directly as domain ID?
>>>>>>
>>>>>> Later the domain ID from RMDD is just used in error messages ... never actually used
>>>>>> as a domain ID to guide the implementation so the correlation appears to be implicit with
>>>>>> this CACD parsing not intuitive.
>>>>>>
>>>>>
>>>>> You are right that the cache ID ultimately comes from get_cpu_cacheinfo_id(),
>>>>> not from CACD directly - the function name is misleading and should be improved.
>>>>>
>>>>> However, my understanding is that CACD parsing acts as the bridge between ACPI's
>>>>> RMDD domain namespace and the kernel's L3 domain ID: CACD enumerates which CPUs
>>>>> belong to each RMDD, and from those CPUs we derive the kernel-side L3 cache ID.
>>>>> The specification defines an RMDD DomainID as an identifier unique within the
>>>>> ERDT table, yet it does not guarantee that this ID matches the kernel’s cache ID
>>>>> obtained via CPUID leaf 0x4. For this reason, we omitted the RMDD domain ID in
>>>>> the current release.
>>>>
>>>> get_l3_cache_id_from_cacd() implies a straightforward query and the function
>>>> comments supports this. At the same time the function includes what appears to be
>>>> sanity checks but from what you say turns out to be essential enforcement of the
>>>> RMDD to CPUID leaf 0x4 mapping. Is this correct? As sanity checks these additional
>>>> checks are ok but as RMDD to CPUID leaf 0x4 mapping enforcement I find them inadequate
>>>> because its correctness requires all CPUs of all domains to be online which cannot be
>>>> guaranteed when this initialization runs.
>>>>
>>>> Consider for example a scenario where RMDD and CPUID do not agree on which CPUs form
>>>> part of a domain. If only one CPU of this RMDD domain is online at the time this
>>>> initialization is done then the mismatch will never be noticed and resctrl will just
>>>> silently use the CPUID mapping, no?
>>>>
>>>
>>> I suppose you are referring to the following scenario:
>>> CACD says CPUs {0,1,2,3} belong to RMDD-A, and CPUID says
>>> CPUs {0,1} share L3-X while CPUs {2,3} share L3-Y (a mismatch).
>>> If only CPU 0 is brought online:
>>>
>>> get_l3_cache_id_from_cacd() iterates CACD's X2APIC list: {0,1,2,3}
>>> CPU 1, 2, 3 are offline -> skipped
>>> CPU 0 is online -> cache_id = get_cpu_cacheinfo_id(0) = L3-X,
>>> all passed.
>>>
>>> Then later CPU2 is online, resctrl's hotplug creates domain L3-Y.
>>> But the xarray has no entry for L3-Y, so erdt_mon_read() will return
>>> -EIO for that domain.
> 
> An error would at least give indication of a problem but I also see a possible
> scenario where the data will happily be read from the wrong register. Consider the
> following change to the scenario you present:
> CACD: RMDD-A = {0, 2}; RMDD-B = {1, 3}
> CPUID: L3-X = {0, 1}; L3-Y = {2, 3}
> 
> If ACPI enumeration is done with CPUs 1, 2, and 3 offline then they are skipped.
> CPU 0 is online -> cache_id = get_cpu_cacheinfo_id(0) = L3-X
> 
> Later when CPU 1 comes online resctrl will consider it to be in L3-X and
> also find an xarray entry for L3-X which will result in erdt_mon_read() to
> read the RMID data from RMDD-A instead of RMDD-B?
> 

Yes, this approach is feasible. However, I am uncertain whether RDT should
be enabled. The underlying issue is that the firmware(BIOS) and 
silicon(CPUID)
will be out of sync, leaving RDT in an unstable state.

>>>
>>> If this is the case, I wonder if we could run the sanity check during
>>> CPU hotplug to reparse the CACD table whenever a CPU comes online. My
>>> thought would be to call get_l3_cache_id_from_cacd() within
>>> resctrl_arch_online_cpu().
>>>
>>
>> In fact, parsing the CACD table during CPU hotplug may not be feasible,
>> as the ACPI tables could have already been released. We could instead
>> perform the check directly against the erdt_domain_xa xarray. If a
>> mismatch is detected, disable ERDT and fall back to legacy MSR-based
>> access:
>>
>> /* Called from resctrl_arch_online_cpu() */
>> void erdt_verify_cpu_domain(int cpu)
>> {
>>      int cache_id;
>>
>>      if (!__erdt_enabled)
>>          return;
>>
>>      cache_id = get_cpu_cacheinfo_id(cpu, RESCTRL_L3_CACHE);
>>      if (cache_id < 0)
>>          return;
>>
>>      if (!xa_load(&erdt_domain_xa, cache_id)) {
>>          pr_warn(FW_BUG "CPU%d L3 domain %d not described by ERDT, disabling ERDT\n",
>>                 cpu, cache_id);
>>          __erdt_enabled = false;
>>      }
>> }
> When considering the example scenario above I do not believe that this implements a
> robust transition from ACPI to CPUID.
> 
> Additionally, disabling ERDT during a CPU online has broader implications: if ERDT was
> enabled earlier then the associated events will be enabled at this point and have
> mov_evt::any_cpu set. From what I can tell this would just implicitly expect that the
> events could still be read via MSR but not actually route the read to the correct CPU?
> 
> I think it would be simpler to separate the ACPI enumeration from the CPUID enumeration
> instead of wedging in these snippets at various touch points. The ACPI enumeration could,
> for example, just record the ERDT RMDD domain <-> cpu_mask mapping without taking online CPUs
> into account. Depending on which tables are found in ACPI, if ERDT is the intended mechanism then
> any duplicate CPUID enumeration should be avoided. This means that there will be two unique
> flows to enabling resctrl events, one for ACPI and one for CPUID. The x86 online handler can
> perform the sanity check when a CPU comes online and since RMDD and CPUID are expected to
> correlate the CPU could not be added to resctrl if it is not in the expected domain. To me
> this seems simpler and more robust.
> 

Thanks for the detailed feedback. If I understand correctly, separating
ACPI enumeration from CPUID enumeration would look something like the 
following.
Could you let me know if this approach is feasible?

----------
1. At ACPI parse time , record the full CPU mask from CACD for each
    RMDD domain without checking online state or resolving L3 cache IDs.
    Store a per-CPU pointer (erdt_cpu_domain[cpu]) for O(1) lookup.

2. At CPU online time (resctrl_arch_online_cpu), validate that the CPU's
    CPUID-derived L3 ID is consistent with other CPUs in the same RMDD:
    - First online CPU in an RMDD establishes the L3 ID for that domain
      and inserts the domain into erdt_domain_xa keyed by L3 cache ID.
    - Subsequent CPUs must match. On mismatch, the CPU is refused from 
resctrl.

3. erdt_mon_read() looks up erdt_domain_xa by hdr->id (L3 cache ID) as 
before,
    but the xarray is now populated lazily at CPU online time.


Pseudocode:

/* ACPI parse time: store full CPU mask, no online check */
cacd_record_cpus(cacd, domain_info):
	for each x2apic_id in cacd:
		cpu = topo_lookup_cpuid(x2apic_id)
		cpumask_set_cpu(cpu, domain_info->cpu_mask)
		erdt_cpu_domain[cpu] = domain_info
		domain_info->l3_id = -1   /* unknown until first CPU comes online */


/* CPU online time via resctrl_arch_online_cpu():
  * validate and establish real L3 mapping
  */
erdt_verify_cpu_domain(cpu):
	dom = erdt_cpu_domain[cpu]
	if (!dom)
		return -EINVAL        /* CPU not in any ERDT domain */

	cpuid_l3_id = get_cpu_cacheinfo_id(cpu)

	if (dom->l3_id == -1) {
		dom->l3_id = cpuid_l3_id
		xa_store(&erdt_domain_xa, cpuid_l3_id, dom)
	} else if (dom->l3_id != cpuid_l3_id) {
		return -EINVAL        /* mismatch: refuse CPU */
	}
	return 0

thanks,
Chenyu