Re: cpusets not cpu hotplug aware

[Nathan Lynch <ntl@pobox.com>]

Paul Jackson wrote:
> Nathan wrote:
> > I think it would be more sensible for the default (i.e. user hasn't
> > explicitly configured any cpusets) behavior on a CONFIG_CPUSETS=y
> > kernel to match the behavior with a CONFIG_CPUSETS=n kernel. 
> 
> Basically, in this situation, that would mean that the code that
> masks a requested sched_setaffinity cpumask with the tasks
> cpuset_cpus_allowed():
> 
> 	cpus_allowed = cpuset_cpus_allowed(p);
> 	cpus_and(new_mask, new_mask, cpus_allowed);
> 
> would not be executed in the case of a kernel configured for cpusets,
> when running on a system that wasn't using cpusets.
> 
> That's quite doable, and for systems not actually using cpusets, makes
> good sense.
> 
> But it makes a bit of discontinuity in the system behaviour when
> someone starts using cpusets.  If someone makes a cpuset, then suddenly
> tasks in the top cpuset start seeing failed sched_setaffinity calls
> for any CPU that was brought online after system boot.
> 
> If there is some decent way I can get the cpus_allowed of the top
> cpuset to track the cpu_online_map, then we avoid this discontinuity
> in system behaviour.


How about this?  I've verified it fixes the issue but I'm nervous
about the locking.


--- cpuhp-sched_setaffinity.orig/kernel/cpuset.c
+++ cpuhp-sched_setaffinity/kernel/cpuset.c
@@ -2033,6 +2033,31 @@ out:
 	return err;
 }
 
+static int cpuset_handle_cpuhp(struct notifier_block *nb,
+				unsigned long phase, void *_cpu)
+{
+	unsigned long cpu = (unsigned long)_cpu;
+
+	mutex_lock(&manage_mutex);
+	lock_cpu_hotplug();
+	mutex_lock(&callback_mutex);
+
+	switch (phase) {
+	case CPU_ONLINE:
+		cpu_set(cpu, top_cpuset.cpus_allowed);
+		break;
+	case CPU_DEAD:
+		cpu_clear(cpu, top_cpuset.cpus_allowed);
+		break;
+	}
+
+	mutex_unlock(&callback_mutex);
+	unlock_cpu_hotplug();
+	mutex_unlock(&manage_mutex);
+
+	return 0;
+}
+
 /**
  * cpuset_init_smp - initialize cpus_allowed
  *
@@ -2043,6 +2068,8 @@ void __init cpuset_init_smp(void)
 {
 	top_cpuset.cpus_allowed = cpu_online_map;
 	top_cpuset.mems_allowed = node_online_map;
+
+	hotcpu_notifier(cpuset_handle_cpuhp, 0);
 }
 
 /**