Re: [PATCH v4 0/8] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

["Moger, Babu" <bmoger@amd.com>]

Hi Reinette,

On 5/5/2025 4:13 PM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 5/5/25 12:54 PM, Moger, Babu wrote:
>> On 5/5/25 11:22, Reinette Chatre wrote:
>>> On 5/2/25 5:53 PM, Moger, Babu wrote:
>>>> On 5/2/2025 4:20 PM, Reinette Chatre wrote:
>>>>> On 4/21/25 3:43 PM, Babu Moger wrote:
> 
> ...
> 
>>>>>
>>>>> Then "bit_usage" will look like:
>>>>>
>>>>> # cat /sys/fs/resctrl/info/L3/bit_usage
>>>>> 0=HHHHXXXXSSSS0000
>>>>
>>>> It is confusing here. To make it clear we may have to print all the CLOSes in each domain.
>>>
>>> Could you please elaborate how this is confusing?
>>
>> # cat /sys/fs/resctrl/info/L3/bit_usage
>> 0=HHHHXXXXSSSS0000
>>
>> This may give the impression that the all CLOSes in all domains carries
>> this property, but in reality, it applies only to one CLOS(15) within each
>> domain.
>>
>> Example below....
>>
> 
> ...
> 
>> Here is the behaviour after these cahnges.
>>
>> === Before io_alloc enabled==============================
>>
>> #cd /sys/fs/resctrl/L3/
>> # cat io_alloc
>> disabled
>>
>> # cat shareable_bits
>> 0  (This is always 0 for AMD)
>>
>> # cat bit_usage
>> 0=SSSSSSSSSSSSSSSS;1=SSSSSSSSSSSSSSSS;2=SSSSSSSSSSSSSSSS;3=SSSSSSSSSSSSSSSS
> 
> Please note that the "S" in above does not have anything to do with
> "shareable_bits" at this point. The "S" indicates that all L3 instances
> are currently used by software and that sharing is allowed.
> 
> "bit_usage" gives insight to user space how all L3 instances are used.
> 
> If at this point a new resource group is created and it has an "exclusive"
> allocation then "bit_usage" will change to reflect that.  For example,
> you can try this on the system you are testing on:
> 
> # echo 'L3:0=fff0;1=fff0;2=fff0;3=fff0' > /sys/fs/resctrl/schemata
> # mkdir /sys/fs/resctrl/g1
> # echo 'L3:0=f;1=f;2=f;3=f' > /sys/fs/resctrl/g1/schemata
> # echo 'exclusive' > /sys/fs/resctrl/g1/mode
> 
> The above isolates a portion of all L3 instances for exclusive use by g1.
> After above changes:
> # cat /sys/fs/resctrl/info/L3/bit_usage
> 0=SSSSSSSSSSSSEEEE;1=SSSSSSSSSSSSEEEE;2=SSSSSSSSSSSSEEEE;3=SSSSSSSSSSSSEEEE
> 

Yes. I see the same output.

> Note that there is no "closid" or resource group information but instead,
> "bit_usage" shows to user space how each cache instance is being used
> across all resource groups and hardware (IO) allocations.

Ok. Got it.

> 
>>
>> ==== After io_alloc enabled=================================
>>
>> # echo 1 > io_alloc
>>
>> # cat io_alloc
>> enabled
>>
>> # cat io_alloc_cbm
>> L3:0=ffff;1=ffff;2=ffff;3=ffff
>>
>> #cat bit_usage
>> 0=XXXXXXXXXXXXXXXX;1=XXXXXXXXXXXXXXXX;2=XXXXXXXXXXXXXXXX;3=XXXXXXXXXXXXXXXX
> 
> Looks accurate to me. It shows that both hardware and software can
> allocate into all portions of all caches.
> 
>>
>> ==== After changing io_alloc_cbm ============================
>>
>> #echo "L3:0=ff00;1=ff00;2=ff00;3=ff00 > io_alloc_cbm
>>
>> # cat io_alloc_cbm
>> L3:0=ff00;1=ff00;2=ff00;3=ff00
>>
>> #cat bit_usage
>> 0=XXXXXXXXSSSSSSSS;1=XXXXXXXXSSSSSSSS;2=XXXXXXXXSSSSSSSS;3=XXXXXXXXSSSSSSSS
> 
> Looks accurate to me.
> 
>> =============================================================
>>
>> My concern here is, this may imply that the property is present across all
>> CLOSes in all the domains, while in fact, it only applies to a single
>> CLOS(15) within each domain.
> 
> If a user wants a resource group specific view then the schemata should be used.
> "bit_usage" presents the view from the cache instance perspective and reflects
> how each L3 cache instance is being used at that moment in time. It helps
> system administrator answer the question "how are the caches used at the moment"?
> "bit_usage" does so by presenting a summary of all allocations across all resource
> groups and any hardware allocations that may exist. This file helps user space
> to understand how the cache is being used without needing to correlate the CBMs
> of all resource groups and IO allocations. For example, "bit_usage" is to be used
> by system administrator to ensure cache is used optimally (for example, there are
> no unused portions). Also, a user may be investigating a performance issue in
> a particular resource group and "bit_usage" will help with that to see if
> the tasks in that resource group may be competing with IO.
> 

Ok, "bit_usage" is a summary across all the groups. That is a good 
point. Thanks for the detailed explanation.

Will make those changes in next revision.

Thank you.
Babu