Re: [PATCH v4 0/8] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 5/5/25 12:54 PM, Moger, Babu wrote:
> On 5/5/25 11:22, Reinette Chatre wrote:
>> On 5/2/25 5:53 PM, Moger, Babu wrote:
>>> On 5/2/2025 4:20 PM, Reinette Chatre wrote:
>>>> On 4/21/25 3:43 PM, Babu Moger wrote:

...

>>>>
>>>> Then "bit_usage" will look like:
>>>>
>>>> # cat /sys/fs/resctrl/info/L3/bit_usage
>>>> 0=HHHHXXXXSSSS0000
>>>
>>> It is confusing here. To make it clear we may have to print all the CLOSes in each domain.
>>
>> Could you please elaborate how this is confusing?
> 
> # cat /sys/fs/resctrl/info/L3/bit_usage
> 0=HHHHXXXXSSSS0000
> 
> This may give the impression that the all CLOSes in all domains carries
> this property, but in reality, it applies only to one CLOS(15) within each
> domain.
> 
> Example below....
> 

...

> Here is the behaviour after these cahnges.
> 
> === Before io_alloc enabled==============================
> 
> #cd /sys/fs/resctrl/L3/
> # cat io_alloc
> disabled
> 
> # cat shareable_bits
> 0  (This is always 0 for AMD)
> 
> # cat bit_usage
> 0=SSSSSSSSSSSSSSSS;1=SSSSSSSSSSSSSSSS;2=SSSSSSSSSSSSSSSS;3=SSSSSSSSSSSSSSSS

Please note that the "S" in above does not have anything to do with
"shareable_bits" at this point. The "S" indicates that all L3 instances
are currently used by software and that sharing is allowed.

"bit_usage" gives insight to user space how all L3 instances are used.

If at this point a new resource group is created and it has an "exclusive"
allocation then "bit_usage" will change to reflect that.  For example,
you can try this on the system you are testing on:

# echo 'L3:0=fff0;1=fff0;2=fff0;3=fff0' > /sys/fs/resctrl/schemata
# mkdir /sys/fs/resctrl/g1
# echo 'L3:0=f;1=f;2=f;3=f' > /sys/fs/resctrl/g1/schemata
# echo 'exclusive' > /sys/fs/resctrl/g1/mode

The above isolates a portion of all L3 instances for exclusive use by g1.
After above changes:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=SSSSSSSSSSSSEEEE;1=SSSSSSSSSSSSEEEE;2=SSSSSSSSSSSSEEEE;3=SSSSSSSSSSSSEEEE

Note that there is no "closid" or resource group information but instead,
"bit_usage" shows to user space how each cache instance is being used
across all resource groups and hardware (IO) allocations.

> 
> ==== After io_alloc enabled=================================
> 
> # echo 1 > io_alloc
> 
> # cat io_alloc
> enabled
> 
> # cat io_alloc_cbm
> L3:0=ffff;1=ffff;2=ffff;3=ffff
> 
> #cat bit_usage
> 0=XXXXXXXXXXXXXXXX;1=XXXXXXXXXXXXXXXX;2=XXXXXXXXXXXXXXXX;3=XXXXXXXXXXXXXXXX

Looks accurate to me. It shows that both hardware and software can
allocate into all portions of all caches.

> 
> ==== After changing io_alloc_cbm ============================
> 
> #echo "L3:0=ff00;1=ff00;2=ff00;3=ff00 > io_alloc_cbm
> 
> # cat io_alloc_cbm
> L3:0=ff00;1=ff00;2=ff00;3=ff00
> 
> #cat bit_usage
> 0=XXXXXXXXSSSSSSSS;1=XXXXXXXXSSSSSSSS;2=XXXXXXXXSSSSSSSS;3=XXXXXXXXSSSSSSSS

Looks accurate to me.

> =============================================================
> 
> My concern here is, this may imply that the property is present across all
> CLOSes in all the domains, while in fact, it only applies to a single
> CLOS(15) within each domain.

If a user wants a resource group specific view then the schemata should be used.
"bit_usage" presents the view from the cache instance perspective and reflects
how each L3 cache instance is being used at that moment in time. It helps
system administrator answer the question "how are the caches used at the moment"?
"bit_usage" does so by presenting a summary of all allocations across all resource
groups and any hardware allocations that may exist. This file helps user space
to understand how the cache is being used without needing to correlate the CBMs
of all resource groups and IO allocations. For example, "bit_usage" is to be used
by system administrator to ensure cache is used optimally (for example, there are
no unused portions). Also, a user may be investigating a performance issue in
a particular resource group and "bit_usage" will help with that to see if
the tasks in that resource group may be competing with IO.

Reinette