Re: cpusets not cpu hotplug aware

[Andrew Morton <akpm@osdl.org>]

On Mon, 21 Aug 2006 14:01:48 -0700
Paul Jackson <pj@sgi.com> wrote:

> Anton wrote:
> 
> > Maybe the notifier is the right way to go, but it seems strange to
> > create two copies of cpu_online_map (with the associated possibiliy of
> > the two getting out of sync).
> 
> Every cpuset in the system, of which this top_cpuset is just the first,
> has a set of cpus and memory nodes on which tasks in that cpuset are
> allowed to operate.  It's not just top_cpuset that we need to understand
> how to relate to hotplug and unplug.
> 
> I'll bet there is more hidden state in mm/mempolicy.c, for mbind()
> and set_mempolicy(), and in kernel/sched.c for the sched_setaffinity(),
> which was derived from what memory nodes or cpus were online.  For
> example, I see several fields in 'struct mempolicy' that contain
> node numbers in some form, and the 'cpus_allowed' field in the task
> struct that sched_setaffinity sets.
> 
> How does hotplug and unplug interact with these various saved states?
> 
> > Its up to the cpusets code to register a hotplug notifier to update the
> > top_cpuset maps.
> 
> That, or user level code, when it adds or removes a cpu or a memory
> node, needs to be responsible for adding or removing that cpu or node
> to or from whichever cpusets are affected.
> 
> For example, if you just added cpu 31, to a system that had been
> running on cpus 0 to 30, you can add cpu 31 to the top cpuset by
> doing:
> 
> 	mkdir /dev/cpuset		 	# if not already done
> 	mount -t cpuset cpuset /dev/cpuset	# if not already done
> 	/bin/echo 0-31 > /dev/cpsuet/cpus
> 
> > If cpuset_cpus_allowed doesnt return the current online mask and we want
> > to schedule on a cpu that has been added since boot it looks like we
> > will fail.
> 
> In general, on systems actually using cpusets, that -is- what should
> happen.  Just because a cpu was brought online doesn't mean it was
> intended to be allowed in any given tasks current cpuset.
> 
> Granted, I would guess users of systems not using cpusets (but
> still have cpusets configured in their kernel, as is common in some
> distro kernels), would expect the behaviour you expected - bringing
> a cpu (or memory node) on or offline would make it available (or
> not) for something like a sched_setaffinity (or mbind/set_mempolicy)
> immediately, without having to invoke some magic cpuset voodoo.
> 
> Offhand, this sounds to me like a choice of two modes of operation.
> 
>     If you aren't actually using cpusets (so every task is in the
>     original top_cpuset) then you'd expect that cpuset to "get out
>     of the way", meaning top_cpuset (the only cpuset, in this case)
>     tracked whatever cpus and nodes were online at the moment.
> 
>     If instead you start messing with cpusets (creating more than one
>     of them and moving tasks between them) then you'd expect cpusets
>     to be enforced, without automatically adding newly added cpus or
>     memory nodes to existing cpusets.  Only the user knows which
>     cpusets should get the added cpus or memory nodes in this case.
> 
> I don't jump for joy over yet another modal state flag.  But I don't see
> a better alternative -- do you?
> 

If the kernel provider (ie: distro) has enabled cpusets then it would be
appropriate that they also provide a hotplug script which detects whether their
user is actually using cpusets and if not, to take some sensible default action. 
ie: add the newly-added CPU to the system's single cpuset, no?

iow: perhaps send a patch against the upstream udev package.