Re: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling

["Chen, Yu C" <yu.c.chen@intel.com>]

On 10/2/2025 7:50 PM, Peter Zijlstra wrote:
> On Thu, Oct 02, 2025 at 07:31:40PM +0800, Chen, Yu C wrote:
>> On 10/1/2025 9:17 PM, Peter Zijlstra wrote:
>>> On Sat, Aug 09, 2025 at 01:03:10PM +0800, Chen Yu wrote:
>>>> From: Tim Chen <tim.c.chen@linux.intel.com>
>>>>
>>>> Cache-aware scheduling is designed to aggregate threads into their
>>>> preferred LLC, either via the task wake up path or the load balancing
>>>> path. One side effect is that when the preferred LLC is saturated,
>>>> more threads will continue to be stacked on it, degrading the workload's
>>>> latency. A strategy is needed to prevent this aggregation from going too
>>>> far such that the preferred LLC is too overloaded.
>>>
>>> So one of the ideas was to extend the preferred llc number to a mask.
>>> Update the preferred mask with (nr_threads / llc_size) bits, indicating
>>> the that many top llc as sorted by occupancy.
>>>
>>>
>>
>> Having more than one preferred LLC helps prevent aggregation from going
>> too far on a single preferred LLC.
>>
>> One question would be: if one LLC cannot hold all the threads of a process,
>> does a second preferred LLC help in this use case? Currently, this patch
>> gives up task aggregation and falls back to legacy load balancing if the
>> preferred LLC is overloaded. If we place threads across two preferred LLCs,
>> these threads might encounter cross-LLC latency anyway - so we may as well
>> let
>> legacy load balancing spread them out IMO.
> 
> Well, being stuck on 2 LLCs instead of being spread across 10 still
> seems like a win, no?
> 
> Remember, our friends at AMD have *MANY* LLCs.
> 

I see, this makes sense.

>> Another issue that Patch 7 tries to address is avoiding task
>> bouncing between preferred LLCs and non-preferred LLCs. If we
>> introduce a preferred LLC priority list, logic to prevent task
>> bouncing between different preferred LLCs might be needed in
>> load balancing, which could become complicated.
> 
> It doesn't really become more difficult to tell preferred LLC from
> non-preferred LLC with a asm. So why should things get more complicatd?
> 

Besides distinguishing between preferred LLCs and non-preferred LLCs,
we might also want to distinguish between the ith preferred LLC and
the jth preferred LLC—this would help avoid task migration
bouncing between them by using hysteresis.

> 
> Anyway, it was just one of the 'random' ideas I had kicking about.
> Reality always ruins things, *shrug* :-)

Yes, multiple preferred LLCs is a promising direction if the data
shows good results. We are planning to clean up the RFC patch and
send a refreshed version to summarize the current status. Meanwhile,
we will evaluate the multi-preferred LLC approach internally.
Thanks for providing this idea.

thanks,
Chenyu