load balancing regression since commit 367456c7

load balancing regression since commit 367456c7

[Tim Chen]

Peter,

We noticed in a hackbench test (./hackbench 100 process 2000)
on a Sandy bridge 2 socket server, there has been a slow down
by a factor of 4 since commit 367456c7 was applied
(sched: Ditch per cgroup task lists for load-balancing).

The commit 5d6523e (sched: Fix load-balance wreckage) did
not fix the regression.

In the profile, there is heavy spin lock contention in the load_balance path of 3.4-rc2
where it was less than .003% of cpu before commit 367456c7.

When we looked into /proc/schedstat for 3.4-rc2 for the run duration, 
on cpu0 schedule was called 13x more often, and schedule call which
left the processor idle was 530x as much.

There was also a big increase in try to wake up remote (sd->ttwu_wake_remote) count.
		
		increase in sd->ttwu_wake_remote for cpu0
domain 0	540%
domain 1 	7570%
domain 2	4426%

Wonder if there is unnecessary load balancing to remote cpu?

Tim


profile for 3.4-rc2

     7.16%     hackbench  [kernel.kallsyms]      [k] _raw_spin_lock
               |
               --- _raw_spin_lock
                  |
                  |--56.52%-- load_balance
                  |          idle_balance
                  |          __schedule
                  |          schedule
                  |          |
                  |          |--98.73%-- schedule_timeout
                  |          |          |
                  |          |          |--97.80%-- unix_stream_recvmsg
                  |          |          |          sock_aio_read.part.7
                  |          |          |          sock_aio_read
                  |          |          |          do_sync_read
                  |          |          |          vfs_read
                  |          |          |          sys_read
                  |          |          |          system_call
                  |          |          |          __read_nocancel
                  |          |          |          create_worker
                  |          |          |          group
                  |          |          |          main
                  |          |          |          __libc_start_main
                  |          |          |

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Tue, 2012-04-10 at 18:06 -0700, Tim Chen wrote:
> Peter,
> 
> We noticed in a hackbench test (./hackbench 100 process 2000)
> on a Sandy bridge 2 socket server, there has been a slow down
> by a factor of 4 since commit 367456c7 was applied
> (sched: Ditch per cgroup task lists for load-balancing).
> 
> The commit 5d6523e (sched: Fix load-balance wreckage) did
> not fix the regression.
> 
> In the profile, there is heavy spin lock contention in the load_balance path of 3.4-rc2
> where it was less than .003% of cpu before commit 367456c7.

I can't actually reproduce but does the below help? 

If not, can you shoot your .config over?

---
 kernel/sched/fair.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0d97ebd..e1da5c6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3215,6 +3215,8 @@ static int move_one_task(struct lb_env *env)
 
 static unsigned long task_h_load(struct task_struct *p);
 
+static const unsigned int sched_nr_migrate_break = IS_ENABLED(CONFIG_PREEMPT) ? 8 : 32;
+
 /*
  * move_tasks tries to move up to load_move weighted load from busiest to
  * this_rq, as part of a balancing operation within domain "sd".
@@ -3242,7 +3244,7 @@ static int move_tasks(struct lb_env *env)
 
 		/* take a breather every nr_migrate tasks */
 		if (env->loop > env->loop_break) {
-			env->loop_break += sysctl_sched_nr_migrate;
+			env->loop_break += sched_nr_migrate_break;
 			env->flags |= LBF_NEED_BREAK;
 			break;
 		}
@@ -4407,7 +4409,8 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 		.dst_cpu	= this_cpu,
 		.dst_rq		= this_rq,
 		.idle		= idle,
-		.loop_break	= sysctl_sched_nr_migrate,
+		.loop_break	= sched_nr_migrate_break,
+		.loop_max	= sysctl_sched_nr_migrate,
 	};
 
 	cpumask_copy(cpus, cpu_active_mask);
@@ -4448,7 +4451,6 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 		env.load_move = imbalance;
 		env.src_cpu = busiest->cpu;
 		env.src_rq = busiest;
-		env.loop_max = busiest->nr_running;
 
 more_balance:
 		local_irq_save(flags);

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Tue, 2012-04-10 at 18:06 -0700, Tim Chen wrote:
>                   |--56.52%-- load_balance
>                   |          idle_balance
>                   |          __schedule
>                   |          schedule 

Ahh, I know why I didn't see it, I have a CONFIG_PREEMPT kernel and
idle_balancing stops once its gotten a single task over instead of
achieving proper balance.

And since hackbench generates insanely long runqueues and the patch that
caused your regression 'fixed' the lock-breaking it will now iterate the
entire runqueue if needed to achieve balance, which hurts.

I think the patch I send ought to work, let me try disabling
CONFIG_PREEMPT.

Re: load balancing regression since commit 367456c7

[Tim Chen]

On Tue, 2012-04-17 at 14:09 +0200, Peter Zijlstra wrote:
> On Tue, 2012-04-10 at 18:06 -0700, Tim Chen wrote:
> >                   |--56.52%-- load_balance
> >                   |          idle_balance
> >                   |          __schedule
> >                   |          schedule 
> 
> Ahh, I know why I didn't see it, I have a CONFIG_PREEMPT kernel and
> idle_balancing stops once its gotten a single task over instead of
> achieving proper balance.
> 
> And since hackbench generates insanely long runqueues and the patch that
> caused your regression 'fixed' the lock-breaking it will now iterate the
> entire runqueue if needed to achieve balance, which hurts.
> 
> I think the patch I send ought to work, let me try disabling
> CONFIG_PREEMPT.
> --

yes, CONFIG_PREEMPT is turned off on my side.  With the patch that you
sent, the slowed down went from a factor of 4 down to a factor 2. 

So the run time is now twice as long vs four time as long vs v3.3
kernel.

Tim

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Tue, 2012-04-17 at 09:44 -0700, Tim Chen wrote:
> On Tue, 2012-04-17 at 14:09 +0200, Peter Zijlstra wrote:
> > On Tue, 2012-04-10 at 18:06 -0700, Tim Chen wrote:
> > >                   |--56.52%-- load_balance
> > >                   |          idle_balance
> > >                   |          __schedule
> > >                   |          schedule 
> > 
> > Ahh, I know why I didn't see it, I have a CONFIG_PREEMPT kernel and
> > idle_balancing stops once its gotten a single task over instead of
> > achieving proper balance.
> > 
> > And since hackbench generates insanely long runqueues and the patch that
> > caused your regression 'fixed' the lock-breaking it will now iterate the
> > entire runqueue if needed to achieve balance, which hurts.
> > 
> > I think the patch I send ought to work, let me try disabling
> > CONFIG_PREEMPT.
> > --
> 
> yes, CONFIG_PREEMPT is turned off on my side.  With the patch that you
> sent, the slowed down went from a factor of 4 down to a factor 2. 
> 
> So the run time is now twice as long vs four time as long vs v3.3
> kernel.

Ok, so I can't reproduce this on my WSM-EP.. even !PREEMPT kernels are
consistent with hackbench times with or without that patch.

Can you still send your full .config? Also, do you have cpu-cgroup muck
enabled and are you using that systemd shite?

What does the below patch (on top of the previous) do?

---
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -784,7 +784,7 @@ account_entity_enqueue(struct cfs_rq *cf
 		update_load_add(&rq_of(cfs_rq)->load, se->load.weight);
 #ifdef CONFIG_SMP
 	if (entity_is_task(se))
-		list_add_tail(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
+		list_add(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
 #endif
 	cfs_rq->nr_running++;
 }

Re: load balancing regression since commit 367456c7

[Tim Chen]

On Fri, 2012-04-20 at 16:00 +0200, Peter Zijlstra wrote:

> 
> Ok, so I can't reproduce this on my WSM-EP.. even !PREEMPT kernels are
> consistent with hackbench times with or without that patch.
> 
> Can you still send your full .config? Also, do you have cpu-cgroup muck
> enabled and are you using that systemd shite?
> 
> What does the below patch (on top of the previous) do?
> 

There is a slight 10% to 15% improvement with the patch.  However,
the change in performance difficult to quantify precisely as the 
hackbench runtime has large variations (up to 50%) on our Sandy Bridge EP
server machines since commit 367456c7.

We also do not see regression for hackbench on WSM-EP, but only 
on machines with Sandy-Bridge EP (2 socket, 8 cores/socket, HT enabled).  
We are not running hackbench in cgroup for this test.

The Sandy Bridge EP machines installed has FC16, so I think it uses 
systemd. The machine on our WSM EP has FC15, which also has systemd.

I'm sending you the .config in a separate mail.  Thanks for taking a look.
The .config aligns closely with standard Fedora settings.

Tim

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Fri, 2012-04-20 at 09:40 -0700, Tim Chen wrote:
> We also do not see regression for hackbench on WSM-EP, but only 
> on machines with Sandy-Bridge EP (2 socket, 8 cores/socket, HT enabled).  

Argh, Suresh, any idea what could be different and relevant to this
issue between WSM and SNB -EP ?

Tim, can you see the problem on the desktop SNB part? That's the only
SNB I have available.

> We are not running hackbench in cgroup for this test.
> 
> The Sandy Bridge EP machines installed has FC16, so I think it uses 
> systemd. The machine on our WSM EP has FC15, which also has systemd.

Ah, but your .config has:

CONFIG_CGROUP_SCHED=y
CONFIG_SCHED_AUTOGROUP=y

and systemd, when cpu-cgroup is available, will automagically use it.

Could you disable those two CONFIG knobs and see if it persists?

If it goes away, its the cgroup balancing, if its still there its
elsewhere.

Re: load balancing regression since commit 367456c7

[Tim Chen]

On Fri, 2012-04-20 at 18:53 +0200, Peter Zijlstra wrote:
> On Fri, 2012-04-20 at 09:40 -0700, Tim Chen wrote:
> > We also do not see regression for hackbench on WSM-EP, but only 
> > on machines with Sandy-Bridge EP (2 socket, 8 cores/socket, HT enabled).  
> 
> Argh, Suresh, any idea what could be different and relevant to this
> issue between WSM and SNB -EP ?
> 
> Tim, can you see the problem on the desktop SNB part? That's the only
> SNB I have available.
> 
> > We are not running hackbench in cgroup for this test.
> > 
> > The Sandy Bridge EP machines installed has FC16, so I think it uses 
> > systemd. The machine on our WSM EP has FC15, which also has systemd.
> 
> Ah, but your .config has:
> 
> CONFIG_CGROUP_SCHED=y
> CONFIG_SCHED_AUTOGROUP=y
> 
> and systemd, when cpu-cgroup is available, will automagically use it.
> 
> Could you disable those two CONFIG knobs and see if it persists?
> 

Turning those off did recover the regression.

Tim

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Fri, 2012-04-20 at 10:13 -0700, Tim Chen wrote:
> On Fri, 2012-04-20 at 18:53 +0200, Peter Zijlstra wrote:
> > On Fri, 2012-04-20 at 09:40 -0700, Tim Chen wrote:
> > > We also do not see regression for hackbench on WSM-EP, but only 
> > > on machines with Sandy-Bridge EP (2 socket, 8 cores/socket, HT enabled).  
> > 
> > Argh, Suresh, any idea what could be different and relevant to this
> > issue between WSM and SNB -EP ?
> > 
> > Tim, can you see the problem on the desktop SNB part? That's the only
> > SNB I have available.
> > 
> > > We are not running hackbench in cgroup for this test.
> > > 
> > > The Sandy Bridge EP machines installed has FC16, so I think it uses 
> > > systemd. The machine on our WSM EP has FC15, which also has systemd.
> > 
> > Ah, but your .config has:
> > 
> > CONFIG_CGROUP_SCHED=y
> > CONFIG_SCHED_AUTOGROUP=y
> > 
> > and systemd, when cpu-cgroup is available, will automagically use it.
> > 
> > Could you disable those two CONFIG knobs and see if it persists?
> > 
> 
> Turning those off did recover the regression.

OK, I'll go stare at the cgroup part then.. Thanks!

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

On Fri, 2012-04-20 at 19:33 +0200, Peter Zijlstra wrote:
> 
> OK, I'll go stare at the cgroup part then.. Thanks!
> 
Ok, I could reproduce when using cgroups, the below fixes it for me, can
you confirm?

---
Subject: sched: Fix more load-balance fallout
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Tue Apr 17 13:38:40 CEST 2012

Commits 367456c756a6 ("sched: Ditch per cgroup task lists for
load-balancing") and 5d6523ebd ("sched: Fix load-balance wreckage")
left some more wreckage.

By setting loop_max unconditionally to ->nr_running load-balancing
could take a lot of time on very long runqueues (hackbench!). So keep
the sysctl as max limit of the amount of tasks we'll iterate.

Furthermore, the min load filter for migration completely fails with
cgroups since inequality in per-cpu state can easily lead to such
small loads :/

Furthermore the change to add new tasks to the tail of the queue
instead of the head seems to have some effect.. not quite sure I
understand why.

Combined these fixes solve the huge hackbench regression reported by
Tim when hackbench is ran in a cgroup.

Reported-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 kernel/sched/fair.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -784,7 +784,7 @@ account_entity_enqueue(struct cfs_rq *cf
 		update_load_add(&rq_of(cfs_rq)->load, se->load.weight);
 #ifdef CONFIG_SMP
 	if (entity_is_task(se))
-		list_add_tail(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
+		list_add(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
 #endif
 	cfs_rq->nr_running++;
 }
@@ -3215,6 +3215,14 @@ static int move_one_task(struct lb_env *
 
 static unsigned long task_h_load(struct task_struct *p);
 
+static const unsigned int sched_nr_migrate_break =
+#ifdef CONFIG_PREEMPT
+	8
+#else
+	32
+#endif
+	;
+
 /*
  * move_tasks tries to move up to load_move weighted load from busiest to
  * this_rq, as part of a balancing operation within domain "sd".
@@ -3242,7 +3250,7 @@ static int move_tasks(struct lb_env *env
 
 		/* take a breather every nr_migrate tasks */
 		if (env->loop > env->loop_break) {
-			env->loop_break += sysctl_sched_nr_migrate;
+			env->loop_break += sched_nr_migrate_break;
 			env->flags |= LBF_NEED_BREAK;
 			break;
 		}
@@ -3252,7 +3260,7 @@ static int move_tasks(struct lb_env *env
 
 		load = task_h_load(p);
 
-		if (load < 16 && !env->sd->nr_balance_failed)
+		if (sched_feat(LB_MIN) && load < 16 && !env->sd->nr_balance_failed)
 			goto next;
 
 		if ((load / 2) > env->load_move)
@@ -4407,7 +4415,7 @@ static int load_balance(int this_cpu, st
 		.dst_cpu	= this_cpu,
 		.dst_rq		= this_rq,
 		.idle		= idle,
-		.loop_break	= sysctl_sched_nr_migrate,
+		.loop_break	= sched_nr_migrate_break,
 	};
 
 	cpumask_copy(cpus, cpu_active_mask);
@@ -4448,7 +4456,8 @@ static int load_balance(int this_cpu, st
 		env.load_move = imbalance;
 		env.src_cpu = busiest->cpu;
 		env.src_rq = busiest;
-		env.loop_max = busiest->nr_running;
+		env.loop_max = min_t(unsigned long,
+				sysctl_sched_nr_migrate, busiest->nr_running);
 
 more_balance:
 		local_irq_save(flags);

Re: load balancing regression since commit 367456c7

[Tim Chen]

On Wed, 2012-04-25 at 16:56 +0200, Peter Zijlstra wrote:
> On Fri, 2012-04-20 at 19:33 +0200, Peter Zijlstra wrote:
> > 
> > OK, I'll go stare at the cgroup part then.. Thanks!
> > 
> Ok, I could reproduce when using cgroups, the below fixes it for me, can
> you confirm?
> 
> ---

Mmm... I got the following error when I try to compile the patch on latest kernel tip:

kernel/sched/fair.c:3263:3: error: implicit declaration of function ‘static_branch_LB_MIN’ [-Werror=implicit-function-declaration]
kernel/sched/fair.c:3263:46: error: ‘__SCHED_FEAT_LB_MIN’ undeclared (first use in this function)
kernel/sched/fair.c:3263:46: note: each undeclared identifier is reported only once for each function it appears in
cc1: some warnings being treated as errors
make[2]: *** [kernel/sched/fair.o] Error 1
make[1]: *** [kernel/sched] Error 2
make: *** [kernel] Error 2

Tim

Re: load balancing regression since commit 367456c7

[Peter Zijlstra]

Gargh.. lost the change to kernel/sched/features.h, now included.

Sorry for that.

---
Subject: sched: Fix more load-balance fallout
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Tue Apr 17 13:38:40 CEST 2012

Commits 367456c756a6 ("sched: Ditch per cgroup task lists for
load-balancing") and 5d6523ebd ("sched: Fix load-balance wreckage")
left some more wreckage.

By setting loop_max unconditionally to ->nr_running load-balancing
could take a lot of time on very long runqueues (hackbench!). So keep
the sysctl as max limit of the amount of tasks we'll iterate.

Furthermore, the min load filter for migration completely fails with
cgroups since inequality in per-cpu state can easily lead to such
small loads :/

Furthermore the change to add new tasks to the tail of the queue
instead of the head seems to have some effect.. not quite sure I
understand why.

Combined these fixes solve the huge hackbench regression reported by
Tim when hackbench is ran in a cgroup.

Reported-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1335365763.28150.267.camel@twins
---
 kernel/sched/fair.c     |   19 ++++++++++++++-----
 kernel/sched/features.h |    1 +
 2 files changed, 15 insertions(+), 5 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -784,7 +784,7 @@ account_entity_enqueue(struct cfs_rq *cf
 		update_load_add(&rq_of(cfs_rq)->load, se->load.weight);
 #ifdef CONFIG_SMP
 	if (entity_is_task(se))
-		list_add_tail(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
+		list_add(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
 #endif
 	cfs_rq->nr_running++;
 }
@@ -3215,6 +3215,14 @@ static int move_one_task(struct lb_env *
 
 static unsigned long task_h_load(struct task_struct *p);
 
+static const unsigned int sched_nr_migrate_break =
+#ifdef CONFIG_PREEMPT
+	8
+#else
+	32
+#endif
+	;
+
 /*
  * move_tasks tries to move up to load_move weighted load from busiest to
  * this_rq, as part of a balancing operation within domain "sd".
@@ -3242,7 +3250,7 @@ static int move_tasks(struct lb_env *env
 
 		/* take a breather every nr_migrate tasks */
 		if (env->loop > env->loop_break) {
-			env->loop_break += sysctl_sched_nr_migrate;
+			env->loop_break += sched_nr_migrate_break;
 			env->flags |= LBF_NEED_BREAK;
 			break;
 		}
@@ -3252,7 +3260,7 @@ static int move_tasks(struct lb_env *env
 
 		load = task_h_load(p);
 
-		if (load < 16 && !env->sd->nr_balance_failed)
+		if (sched_feat(LB_MIN) && load < 16 && !env->sd->nr_balance_failed)
 			goto next;
 
 		if ((load / 2) > env->load_move)
@@ -4407,7 +4415,7 @@ static int load_balance(int this_cpu, st
 		.dst_cpu	= this_cpu,
 		.dst_rq		= this_rq,
 		.idle		= idle,
-		.loop_break	= sysctl_sched_nr_migrate,
+		.loop_break	= sched_nr_migrate_break,
 	};
 
 	cpumask_copy(cpus, cpu_active_mask);
@@ -4448,7 +4456,8 @@ static int load_balance(int this_cpu, st
 		env.load_move = imbalance;
 		env.src_cpu = busiest->cpu;
 		env.src_rq = busiest;
-		env.loop_max = busiest->nr_running;
+		env.loop_max = min_t(unsigned long,
+				sysctl_sched_nr_migrate, busiest->nr_running);
 
 more_balance:
 		local_irq_save(flags);
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -68,3 +68,4 @@ SCHED_FEAT(TTWU_QUEUE, true)
 
 SCHED_FEAT(FORCE_SD_OVERLAP, false)
 SCHED_FEAT(RT_RUNTIME_SHARE, true)
+SCHED_FEAT(LB_MIN, false)

Re: load balancing regression since commit 367456c7

[Tim Chen]

On Wed, 2012-04-25 at 19:43 +0200, Peter Zijlstra wrote:
> Gargh.. lost the change to kernel/sched/features.h, now included.
> 
> Sorry for that.
> 
> ---
> Subject: sched: Fix more load-balance fallout
> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Tue Apr 17 13:38:40 CEST 2012
> 
> Commits 367456c756a6 ("sched: Ditch per cgroup task lists for
> load-balancing") and 5d6523ebd ("sched: Fix load-balance wreckage")
> left some more wreckage.
> 
> By setting loop_max unconditionally to ->nr_running load-balancing
> could take a lot of time on very long runqueues (hackbench!). So keep
> the sysctl as max limit of the amount of tasks we'll iterate.
> 
> Furthermore, the min load filter for migration completely fails with
> cgroups since inequality in per-cpu state can easily lead to such
> small loads :/
> 
> Furthermore the change to add new tasks to the tail of the queue
> instead of the head seems to have some effect.. not quite sure I
> understand why.
> 
> Combined these fixes solve the huge hackbench regression reported by
> Tim when hackbench is ran in a cgroup.
> 
> Reported-by: Tim Chen <tim.c.chen@linux.intel.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Link: http://lkml.kernel.org/r/1335365763.28150.267.camel@twins

The patch fixed the regression for me.  

Acked-by: Tim Chen <tim.c.chen@linux.intel.com>

[tip:sched/urgent] sched: Fix more load-balancing fallout

[tip-bot for Peter Zijlstra]

Commit-ID:  eb95308ee2a69403909e111837b9068c64cfc349
Gitweb:     http://git.kernel.org/tip/eb95308ee2a69403909e111837b9068c64cfc349
Author:     Peter Zijlstra <a.p.zijlstra@chello.nl>
AuthorDate: Tue, 17 Apr 2012 13:38:40 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 26 Apr 2012 12:54:52 +0200

sched: Fix more load-balancing fallout

Commits 367456c756a6 ("sched: Ditch per cgroup task lists for
load-balancing") and 5d6523ebd ("sched: Fix load-balance wreckage")
left some more wreckage.

By setting loop_max unconditionally to ->nr_running load-balancing
could take a lot of time on very long runqueues (hackbench!). So keep
the sysctl as max limit of the amount of tasks we'll iterate.

Furthermore, the min load filter for migration completely fails with
cgroups since inequality in per-cpu state can easily lead to such
small loads :/

Furthermore the change to add new tasks to the tail of the queue
instead of the head seems to have some effect.. not quite sure I
understand why.

Combined these fixes solve the huge hackbench regression reported by
Tim when hackbench is ran in a cgroup.

Reported-by: Tim Chen <tim.c.chen@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Link: http://lkml.kernel.org/r/1335365763.28150.267.camel@twins
[ got rid of the CONFIG_PREEMPT tuning and made small readability edits ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/fair.c     |   18 ++++++++++--------
 kernel/sched/features.h |    1 +
 2 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0d97ebd..e955364 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -784,7 +784,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 		update_load_add(&rq_of(cfs_rq)->load, se->load.weight);
 #ifdef CONFIG_SMP
 	if (entity_is_task(se))
-		list_add_tail(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
+		list_add(&se->group_node, &rq_of(cfs_rq)->cfs_tasks);
 #endif
 	cfs_rq->nr_running++;
 }
@@ -3215,6 +3215,8 @@ static int move_one_task(struct lb_env *env)
 
 static unsigned long task_h_load(struct task_struct *p);
 
+static const unsigned int sched_nr_migrate_break = 32;
+
 /*
  * move_tasks tries to move up to load_move weighted load from busiest to
  * this_rq, as part of a balancing operation within domain "sd".
@@ -3242,7 +3244,7 @@ static int move_tasks(struct lb_env *env)
 
 		/* take a breather every nr_migrate tasks */
 		if (env->loop > env->loop_break) {
-			env->loop_break += sysctl_sched_nr_migrate;
+			env->loop_break += sched_nr_migrate_break;
 			env->flags |= LBF_NEED_BREAK;
 			break;
 		}
@@ -3252,7 +3254,7 @@ static int move_tasks(struct lb_env *env)
 
 		load = task_h_load(p);
 
-		if (load < 16 && !env->sd->nr_balance_failed)
+		if (sched_feat(LB_MIN) && load < 16 && !env->sd->nr_balance_failed)
 			goto next;
 
 		if ((load / 2) > env->load_move)
@@ -4407,7 +4409,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 		.dst_cpu	= this_cpu,
 		.dst_rq		= this_rq,
 		.idle		= idle,
-		.loop_break	= sysctl_sched_nr_migrate,
+		.loop_break	= sched_nr_migrate_break,
 	};
 
 	cpumask_copy(cpus, cpu_active_mask);
@@ -4445,10 +4447,10 @@ redo:
 		 * correctly treated as an imbalance.
 		 */
 		env.flags |= LBF_ALL_PINNED;
-		env.load_move = imbalance;
-		env.src_cpu = busiest->cpu;
-		env.src_rq = busiest;
-		env.loop_max = busiest->nr_running;
+		env.load_move	= imbalance;
+		env.src_cpu	= busiest->cpu;
+		env.src_rq	= busiest;
+		env.loop_max	= min_t(unsigned long, sysctl_sched_nr_migrate, busiest->nr_running);
 
 more_balance:
 		local_irq_save(flags);
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index e61fd73..de00a48 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -68,3 +68,4 @@ SCHED_FEAT(TTWU_QUEUE, true)
 
 SCHED_FEAT(FORCE_SD_OVERLAP, false)
 SCHED_FEAT(RT_RUNTIME_SHARE, true)
+SCHED_FEAT(LB_MIN, false)