[RFC][PATCH] avoid cpu hot removal if busy take3

[RFC][PATCH] avoid cpu hot removal if busy take3

[KAMEZAWA Hiroyuki]

Hi,

At first I'm sorry that a patch I sent was too messy.
This is updated one. tested on ia64 SMP and works well.

This is the log of failure of cpu-hot-removal. 
==
Jun 23 16:14:26 casares kernel: cpu(1) is busy because of task(6512)
Jun 23 16:14:26 casares kernel: adjust task(6512)'s cpu affinity  or set cpu_remo
va_migration to 1 to remove cpu 1
Jun 23 16:14:26 casares kernel: cpu_down: attempt to take down CPU 1 failed
==
I think this includes enough information for sysadmin.

Changelog V2 -> V3
- changes the name and the meaning of sysctl. 
- bug fixes

-Kame
==

Now, cpu hot remove migrates all tasks on target cpu by force.

During cpu-hot-remove, if tsk->cpus_allowed contains the only target
cpu of removal, tsk->cpus_allowd is disposed and the kernel migrate it to
any cpu at random. It's obvious that user-land configuration before cpu hot
removal was bad. This looks a realisitc workaround, but this is not good in
carefully scheduled environment.

In this case,
1. ignore bad configuration in user-land just do warnings.
2. cancel cpu hot removal and warn users to fix the problem and retry.
seems to be a realisitc workaround. Killing the problematic process may
cause some trouble in user-land (dead-lock etc..)

This patch adds sysctl cpu_removal_migration.
If cpu_removal_migration == 1, all tasks are migrated by force.
If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
tasks.

Note: cpu scheduler's notifier chain has the highest priority. then, this
      failure detection will be done at first.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>



 include/linux/sysctl.h |    1 +
 kernel/sched.c         |   44 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/sysctl.c        |   13 +++++++++++++
 3 files changed, 58 insertions(+)

Index: linux-2.6.17.cputest/kernel/sched.c
===================================================================
--- linux-2.6.17.cputest.orig/kernel/sched.c	2006-06-18 10:49:35.000000000 +0900
+++ linux-2.6.17.cputest/kernel/sched.c	2006-06-23 15:25:21.000000000 +0900
@@ -4562,6 +4562,46 @@
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
+/*
+ * if cpu_removal_migration=0(sysctl), cpu-hot-remove will fail if cpu is busy.
+ * Default value is 1. all tasks are forced to migrate.
+ */
+int cpu_removal_migration = 1;
+
+/*
+ * test there are tasks tightly coupled to the target cpu.
+ * This is called only when cpu_removal_migration = 0.
+ */
+static int test_cpu_busy(int cpu)
+{
+	cpumask_t mask;
+	int ret = 0;
+	pid_t pid = 0;
+	struct task_struct *p;
+	cpus_clear(mask);
+	cpu_set(cpu, mask);
+
+	read_lock(&tasklist_lock);
+	for_each_process(p) {
+		if (p == current)
+			continue;
+		if (p->mm && cpus_equal(mask, p->cpus_allowed)) {
+			ret = 1;
+			pid = p->pid;
+			break;
+		}
+	}
+	read_unlock(&tasklist_lock);
+	if (ret) {
+		printk(KERN_ERR "cpu(%d) is busy because of task(%d)\n",
+			cpu, pid);
+		printk(KERN_ERR "adjust task(%d)'s cpu affinity  or set "
+			"cpu_remova_migration to 1 to remove cpu %d\n",
+			pid, cpu);
+	}
+	return ret;
+}
+
 /* Figure out where task on dead CPU should go, use force if neccessary. */
 static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *tsk)
 {
@@ -4752,6 +4792,10 @@
 		kthread_stop(cpu_rq(cpu)->migration_thread);
 		cpu_rq(cpu)->migration_thread = NULL;
 		break;
+	case CPU_DOWN_PREPARE:
+		if (!cpu_removal_migration && test_cpu_busy(cpu))
+			return NOTIFY_BAD;
+		break;
 	case CPU_DEAD:
 		migrate_live_tasks(cpu);
 		rq = cpu_rq(cpu);
Index: linux-2.6.17.cputest/kernel/sysctl.c
===================================================================
--- linux-2.6.17.cputest.orig/kernel/sysctl.c	2006-06-18 10:49:35.000000000 +0900
+++ linux-2.6.17.cputest/kernel/sysctl.c	2006-06-23 15:25:21.000000000 +0900
@@ -78,6 +78,9 @@
 extern int proc_unknown_nmi_panic(ctl_table *, int, struct file *,
 				  void __user *, size_t *, loff_t *);
 #endif
+#ifdef CONFIG_HOTPLUG_CPU
+extern int cpu_removal_migration;
+#endif
 
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
 static int maxolduid = 65535;
@@ -683,6 +686,16 @@
 		.proc_handler	= &proc_dointvec,
 	},
 #endif
+#ifdef CONFIG_HOTPLUG_CPU
+	{
+		.ctl_name	= KERN_CPU_REMOVAL_MIGRATION,
+		.procname	= "cpu_removal_migration",
+		.data		= &cpu_removal_migration,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
+#endif
 	{ .ctl_name = 0 }
 };
 
Index: linux-2.6.17.cputest/include/linux/sysctl.h
===================================================================
--- linux-2.6.17.cputest.orig/include/linux/sysctl.h	2006-06-18 10:49:35.000000000 +0900
+++ linux-2.6.17.cputest/include/linux/sysctl.h	2006-06-23 15:33:34.000000000 +0900
@@ -148,6 +148,7 @@
 	KERN_SPIN_RETRY=70,	/* int: number of spinlock retries */
 	KERN_ACPI_VIDEO_FLAGS=71, /* int: flags for setting up video after ACPI sleep */
 	KERN_IA64_UNALIGNED=72, /* int: ia64 unaligned userland trap enable */
+	KERN_CPU_REMOVAL_MIGRATION=73, /* int: allow forced migration at cpu removal */
 };
 
 

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[Nathan Lynch]

KAMEZAWA Hiroyuki wrote:
> 
> Now, cpu hot remove migrates all tasks on target cpu by force.
> 
> During cpu-hot-remove, if tsk->cpus_allowed contains the only target
> cpu of removal, tsk->cpus_allowd is disposed and the kernel migrate it to
> any cpu at random. It's obvious that user-land configuration before cpu hot
> removal was bad. This looks a realisitc workaround, but this is not good in
> carefully scheduled environment.
> 
> In this case,
> 1. ignore bad configuration in user-land just do warnings.
> 2. cancel cpu hot removal and warn users to fix the problem and retry.
> seems to be a realisitc workaround. Killing the problematic process may
> cause some trouble in user-land (dead-lock etc..)
> 
> This patch adds sysctl cpu_removal_migration.
> If cpu_removal_migration == 1, all tasks are migrated by force.
> If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> tasks.
> 
> Note: cpu scheduler's notifier chain has the highest priority. then, this
>       failure detection will be done at first.

I'm still not convinced that this is a good thing to do.  I reiterate:
this can be implemented in userspace (probably with fewer lines of
code, even).  Why should this policy be in the kernel?

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[KAMEZAWA Hiroyuki]

On Fri, 23 Jun 2006 09:27:46 -0500
Nathan Lynch <ntl@pobox.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> > This patch adds sysctl cpu_removal_migration.
> > If cpu_removal_migration == 1, all tasks are migrated by force.
> > If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> > tasks.
> > 
> > Note: cpu scheduler's notifier chain has the highest priority. then, this
> >       failure detection will be done at first.
> 
> I'm still not convinced that this is a good thing to do.  I reiterate:
> this can be implemented in userspace (probably with fewer lines of
> code, even).  Why should this policy be in the kernel?
> 
I don't think so.
If we can expect all things can be maintained by user-space in proper way,
why we need forced migration ? This patch is just one of possible workarounds. 
and implemtns, "success always" and "fail if busy" policy to cpu-hot-remove.

-Kame

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[Pavel Machek]

On Fri 2006-06-23 23:35:25, KAMEZAWA Hiroyuki wrote:
> On Fri, 23 Jun 2006 09:27:46 -0500
> Nathan Lynch <ntl@pobox.com> wrote:
> 
> > KAMEZAWA Hiroyuki wrote:
> > > This patch adds sysctl cpu_removal_migration.
> > > If cpu_removal_migration == 1, all tasks are migrated by force.
> > > If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> > > tasks.
> > > 
> > > Note: cpu scheduler's notifier chain has the highest priority. then, this
> > >       failure detection will be done at first.
> > 
> > I'm still not convinced that this is a good thing to do.  I reiterate:
> > this can be implemented in userspace (probably with fewer lines of
> > code, even).  Why should this policy be in the kernel?
> > 
> I don't think so.
> If we can expect all things can be maintained by user-space in proper way,
> why we need forced migration ? This patch is just one of possible workarounds. 
> and implemtns, "success always" and "fail if busy" policy to cpu-hot-remove.

So... we have piece of policy in kernel, that maybe should not be
there (forced migration). Now, you want to make that policy optional,
and add second piece of policy?

No no, I'm afraid.							Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[KAMEZAWA Hiroyuki]

On Fri, 23 Jun 2006 17:10:05 +0200
Pavel Machek <pavel@suse.cz> wrote:

> On Fri 2006-06-23 23:35:25, KAMEZAWA Hiroyuki wrote:
> > I don't think so.
> > If we can expect all things can be maintained by user-space in proper way,
> > why we need forced migration ? This patch is just one of possible workarounds. 
> > and implemtns, "success always" and "fail if busy" policy to cpu-hot-remove.
> 
> So... we have piece of policy in kernel, that maybe should not be
> there (forced migration). Now, you want to make that policy optional,
> and add second piece of policy?
> 
> No no, I'm afraid.							Pavel
> 

Ah, okay. the kernel has the policy in it, so we don't need another policy...
Hmm...
I want another one policy which is more conservative,
but if people says  "don't do that", I'll give up this patch..
and consider how to explain users.

Thanks,
-Kame

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[Ashok Raj]

On Fri, Jun 23, 2006 at 04:40:42PM +0900, KAMEZAWA Hiroyuki wrote:
> 
> In this case,
> 1. ignore bad configuration in user-land just do warnings.
> 2. cancel cpu hot removal and warn users to fix the problem and retry.
> seems to be a realisitc workaround. Killing the problematic process may
> cause some trouble in user-land (dead-lock etc..)
> 
> This patch adds sysctl cpu_removal_migration.
> If cpu_removal_migration == 1, all tasks are migrated by force.
> If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> tasks.

Having this dual behaviour is a concern. Probably we should have the tasks
decide if they want to terminate themselves if its not *OK* to run on a 
different CPU, and not have a policy in kernel decide which way the 
behaviour should be. The kernel policy should be to always force
the cpu removal to happen. Admin should decide what processes should terminate
ahead of time before the removal force migrates them.

Once the kernel/admin chooses to perform cpu offline, it should not be possible
for some process to veto the removal and fail the removal. Removal was probably
choosen since we would like to offline a failing cpu, and dont want some
thing in the way to make that happen.

> + */
> +static int test_cpu_busy(int cpu)
> +{
> +	cpumask_t mask;
> +	int ret = 0;

 Deleted....

> +
> +	read_lock(&tasklist_lock);
> +	for_each_process(p) {
> +		if (p == current)
> +			continue;
> +		if (p->mm && cpus_equal(mask, p->cpus_allowed)) {
> +			ret = 1;
> +			pid = p->pid;
> +			break;
> +		}

Do you want to scan and print all possible id's? otherwise printk will
have just 1, and next attempt will show another pid now... in case the
admin wants to do something useful with this list, probably better to 
give it all out?


-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[KAMEZAWA Hiroyuki]

On Fri, 23 Jun 2006 12:09:50 -0700
Ashok Raj <ashok.raj@intel.com> wrote:

> On Fri, Jun 23, 2006 at 04:40:42PM +0900, KAMEZAWA Hiroyuki wrote:
> > This patch adds sysctl cpu_removal_migration.
> > If cpu_removal_migration == 1, all tasks are migrated by force.
> > If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> > tasks.
> 
> Having this dual behaviour is a concern. Probably we should have the tasks
> decide if they want to terminate themselves if its not *OK* to run on a 
> different CPU, and not have a policy in kernel decide which way the 
> behaviour should be. The kernel policy should be to always force
> the cpu removal to happen. Admin should decide what processes should terminate
> ahead of time before the removal force migrates them.
> 
Hmm..I wish this forcced migration will not break resource isolation sub system
in future.

> Once the kernel/admin chooses to perform cpu offline, it should not be possible
> for some process to veto the removal and fail the removal. Removal was probably
> choosen since we would like to offline a failing cpu, and dont want some
> thing in the way to make that happen.

But in dynamic reconfiguration case (ex. VM resiging for load balancing)
the demand is not so heavy. And if we want to remove cpu by force, just adding
one line to script, echo 1 > /proc/sys/kernel/cpu_removal_migration is enough.
This is not so big obstacle.


> > +
> > +	read_lock(&tasklist_lock);
> > +	for_each_process(p) {
> > +		if (p == current)
> > +			continue;
> > +		if (p->mm && cpus_equal(mask, p->cpus_allowed)) {
> > +			ret = 1;
> > +			pid = p->pid;
> > +			break;
> > +		}
> 
> Do you want to scan and print all possible id's? otherwise printk will
> have just 1, and next attempt will show another pid now... in case the
> admin wants to do something useful with this list, probably better to 
> give it all out?
> 
Hmm, maybe useful. I'll consider again this if I can go ahead. 

Thanks,
-Kame

Re: [RFC][PATCH] avoid cpu hot removal if busy take3

[Randy.Dunlap]

On Fri, 23 Jun 2006 12:09:50 -0700 Ashok Raj wrote:

> On Fri, Jun 23, 2006 at 04:40:42PM +0900, KAMEZAWA Hiroyuki wrote:
> > 
> > In this case,
> > 1. ignore bad configuration in user-land just do warnings.
> > 2. cancel cpu hot removal and warn users to fix the problem and retry.
> > seems to be a realisitc workaround. Killing the problematic process may
> > cause some trouble in user-land (dead-lock etc..)
> > 
> > This patch adds sysctl cpu_removal_migration.
> > If cpu_removal_migration == 1, all tasks are migrated by force.
> > If cpu_removal_migration == 0, cpu_hotremoval can fail because of not-migratable
> > tasks.
> 
> Having this dual behaviour is a concern. Probably we should have the tasks
> decide if they want to terminate themselves if its not *OK* to run on a 
> different CPU, and not have a policy in kernel decide which way the 
> behaviour should be. The kernel policy should be to always force
> the cpu removal to happen. Admin should decide what processes should terminate
> ahead of time before the removal force migrates them.

You are making a value call as you see it.  Others have
disagreed.

> Once the kernel/admin chooses to perform cpu offline, it should not be possible
> for some process to veto the removal and fail the removal. Removal was probably
> choosen since we would like to offline a failing cpu, and dont want some
> thing in the way to make that happen.

Just how slowly or quickly to CPUs fail?  Does an admin have time
to look up the list of processes that are bound to which CPUs,
move them or kill them, etc., before giving the offline-that-CPU
command?  or would she/he rather tell the system how to handle
CPU-offlining conflicts and then just issue the one command?


> > + */
> > +static int test_cpu_busy(int cpu)
> > +{
> > +	cpumask_t mask;
> > +	int ret = 0;
> 
>  Deleted....
> 
> > +
> > +	read_lock(&tasklist_lock);
> > +	for_each_process(p) {
> > +		if (p == current)
> > +			continue;
> > +		if (p->mm && cpus_equal(mask, p->cpus_allowed)) {
> > +			ret = 1;
> > +			pid = p->pid;
> > +			break;
> > +		}
> 
> Do you want to scan and print all possible id's? otherwise printk will
> have just 1, and next attempt will show another pid now... in case the
> admin wants to do something useful with this list, probably better to 
> give it all out?

Yep, I noticed that too, but didn't comment on it.

---
~Randy