Re: [tip: sched/core] sched/fair: Multi-LLC select_idle_sibling()

[Chen Yu <yu.c.chen@intel.com>]

On 2023-06-05 at 21:07:46 +0200, Peter Zijlstra wrote:
> On Mon, Jun 05, 2023 at 05:25:30PM +0200, Marek Szyprowski wrote:
> 
> > nfortunately it causes 
> > regression on my ARM 64bit Exynos5433-based TM2e test board during the 
> > CPU hotplug tests. 
> 
> Can you elucidate an ARM illiterate on the actual topology of that
> machine?
> 
> 
> > CPU: 6 PID: 43 Comm: cpuhp/6 Not tainted 6.4.0-rc1+ #13640
> > Hardware name: Samsung TM2E board (DT)
> > pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > pc : __bitmap_and+0x4c/0x78
> > lr : select_idle_cpu+0x64/0x450
> 
> Btw, where is lr at? Is that perhaps per_cpu(sd_llc) being NULL  or
> something?
> 
> 
> > > ---
> > >   kernel/sched/fair.c     | 38 ++++++++++++++++++++++++++++++++++++++
> > >   kernel/sched/features.h |  1 +
> > >   2 files changed, 39 insertions(+)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 48b6f0c..0172458 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -7028,6 +7028,38 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
> > >   }
> > >   
> > >   /*
> > > + * For the multiple-LLC per node case, make sure to try the other LLC's if the
> > > + * local LLC comes up empty.
> > > + */
> > > +static int
> > > +select_idle_node(struct task_struct *p, struct sched_domain *sd, int target)
> > > +{
> > > +	struct sched_domain *parent = sd->parent;
> > > +	struct sched_group *sg;
> > > +
> > > +	/* Make sure to not cross nodes. */
> > > +	if (!parent || parent->flags & SD_NUMA)
> > > +		return -1;
> > > +
> > > +	sg = parent->groups;
> > > +	do {
> > > +		int cpu = cpumask_first(sched_group_span(sg));
> > > +		struct sched_domain *sd_child;
> > > +
> > > +		sd_child = per_cpu(sd_llc, cpu);
> 
> IOW, sd_child end up NULL?
>
Just wonder if we can use rcu_dereference(per_cpu(sd_llc, cpu)),
and cpu offline/online could modify sd_llc pointer. (We used rcu dereference
for sd_llc in other places)

thanks,
Chenyu