Re: [PATCH v2 sched_ext/for-6.13] sched_ext: Do not enable LLC/NUMA optimizations when domains overlap

[Andrea Righi <arighi@nvidia.com>]

On Thu, Nov 07, 2024 at 09:04:56AM -1000, Tejun Heo wrote:
> Hello, Andrea.
> 
> Some nits below:
> 
> On Thu, Nov 07, 2024 at 09:48:03AM +0100, Andrea Righi wrote:
> > +static bool llc_and_numa_mismatch(void)
> > +{
> ...
> > +     for_each_online_cpu(cpu) {
> > +             sd = cpu_rq(cpu)->sd;
> > +
> > +             while (sd) {
> 
> This can be for_each_domain(cpu, sd).

Oh that's nicer, thanks!

> 
> > +                     bool is_llc = sd->flags & SD_SHARE_LLC;
> > +                     bool is_numa = sd->flags & SD_NUMA;
> > +
> > +                     if (is_llc != is_numa)
> > +                             return true;
> > +
> > +                     sd = sd->parent;
> > +             }
> > +     }
> > +
> > +     return false;
> > +}
> > +
> >  /*
> >   * Initialize topology-aware scheduling.
> >   *
> >   * Detect if the system has multiple LLC or multiple NUMA domains and enable
> >   * cache-aware / NUMA-aware scheduling optimizations in the default CPU idle
> >   * selection policy.
> > + *
> > + * Assumption: under normal circumstances we can assume that each CPU belongs
> > + * to a single NUMA domain and a single LLC domain.
> > + *
> > + * However, in complex or highly specialized systems (e.g., multi-socket,
> > + * chiplet-based, or virtualized systems), the relationship between NUMA and
> > + * LLC domains can become more intricate, though each CPU is still considered
> > + * to belong to a single NUMA and LLC domain in the kernel's internal
> > + * representation.
> > + *
> > + * Another assumption is that each LLC domain is always fully contained within
> > + * a single NUMA domain. In reality, in chiplet-based or virtualized systems,
> > + * LLC domains may logically span multiple NUMA nodes. However, the kernel’s
> 
> Are there any actual systems that have a single LLC spanning multiple NUMA
> nodes? I think it'd be sufficient to state that the kernel assumes that a
> CPU belongs to a single LLC and a single LLC belongs to a single socket.

I've searched quite a bit, but haven't found any architecture that
explicitly shows an LLC shared across different NUMA nodes. While there
are technologies that enable L3 cache coherency / communication between
multiple CCDs (such as AMD's Infinity Fabric in EPYC processors or
Intel's UPI in some Xeon models), these are not technically LLCs
spanning multiple NUMA nodes.

So, I think it's fine to just state that the kernel is assuming the
hierarchy CPU -> single LLC -> single NUMA.

I'll apply these changes and send a v3, thanks!

-Andrea

> 
> Otherwise, looks great to me.
> 
> Thanks.
> 
> --
> tejun