Re: New NUMA scheduler and hotplug CPU

[Andrew Theurer <habanero@us.ibm.com>]

> >To me, it'd make more sense to add the CPUs to the scheduler structures
> >as they get brought online. I can also imagine machines where you have
> >a massive (infinite?) variety of possible CPUs that could appear -
> >like an NUMA box where you could just plug arbitrary numbers of new
> >nodes in as you wanted.
>
> I guess so, but you'd still need NR_CPUS to be >= that arbitrary
> number.
>
> >Moreover, as the CPUs aren't fixed numbers in advance, how are you going
> >to know which node to put them in, etc? Setting up every possible thing
> >in advance seems like an infeasible way to do hotplug to me.
>
> Well this would be the problem. I guess its quite possible that
> one doesn't know the topology of newly added CPUs before hand.
>
> Well OK, this would require a per architecture function to handle
> CPU hotplug. It could possibly just default to arch_init_sched_domains,
> and just completely reinitialise everything which would be the simplest.

Call me crazy, but why not let the topology be determined via userspace at a 
more appropriate time?  When you hotplug, you tell it where in the scheduler 
to plug it.  Have structures in the scheduler which represent the 
nodes-runqueues-cpus topology (in the past I tried a node/rq/cpu structs with 
simple pointers), but let the topology be built based on user's desires thru 
hotplug.  

For example, you boot on just the boot cpu, which by default is in the first 
node on the first runqueue.  All other cpus, whether being "booted" for the 
for the first time or hotplugged (maybe now there's really no difference), 
the hotplugging tells where the cpu should be, in what node and what 
runqueue.  HT cpus work even better, because you can hotplug siblings, once 
at a time if you wanted, to the same runqueue.  Or you have cpus sharing a 
die, same thing, lots of choices here.  This removes any per-arch updates to 
the kernel for things like scheduler topology, and lets them go somewhere 
else more easily changes, like userspace. 

Forgive me if this sounds stupid; I have not been following the discussion 
closely.