[RFC PATCH v1 0/5] Modular find_busiest_group()

[RFC PATCH v1 0/5] Modular find_busiest_group()

[Vaidyanathan Srinivasan]

Hi,

I have been building tunable sched_mc=n patches on top of existing
sched_mc_power_savings code and adding more stuff to
find_busiest_group().

Reference:

[1]Making power policy just work
http://lwn.net/Articles/287924/ 

[2][RFC v1] Tunable sched_mc_power_savings=n
http://lwn.net/Articles/287882/

[3][RFC PATCH v2 0/7] Tunable sched_mc_power_savings=n
http://lwn.net/Articles/297306/

Peter Zijlstra had suggested that it is a good idea to cleanup the
current code in find_busiest_group() before building on the existing
power saving balance infrastructure [4]. This becomes even more
important from the fact that there have been recent bugs in the power
savings code that was hard to detect and fix [5][6].

[4] http://lkml.org/lkml/2008/9/8/103

Reference to bugs:

[5] sched: Fix __load_balance_iterator() for cfq with only one task
http://lkml.org/lkml/2008/9/5/135

[6]sched: arch_reinit_sched_domains() must destroy domains to force rebuild
http://lkml.org/lkml/2008/8/29/191
http://lkml.org/lkml/2008/8/29/343

In an attempt to make the find_busiest_group() function modular and
extensible to more complex load balance decision, I have defined new data
structures and functions and made the find_busiest_group() function small
and readable.

** This is RFC patch, just compiled and booted, may have logic errors

Please let me know if the approach is correct.  I will continue to fix
the logic errors and make it functionally correct.

Thanks,
Vaidy

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>

After applying the patch series, the function will look like this:

/*
 * find_busiest_group finds and returns the busiest CPU group within the
 * domain. It calculates and returns the amount of weighted load which
 * should be moved to restore balance via the imbalance parameter.
 */
static struct sched_group *
find_busiest_group(struct sched_domain *sd, int this_cpu,
		   unsigned long *imbalance, enum cpu_idle_type idle,
		   int *sd_idle, const cpumask_t *cpus, int *balance)
{
	struct sched_group *group = sd->groups;
	unsigned long max_pull;
	int load_idx;
	struct group_loads gl;
	struct sd_loads sdl;

	memset(&sdl, 0, sizeof(sdl));
	sdl.sd = sd;

	/* Get the load index corresponding to cpu idle state */
	load_idx = get_load_idx(sd, idle);

	do {
		int need_balance;

		need_balance = get_group_loads(group, this_cpu, cpus, idle,
					       load_idx, &gl);

		if (*sd_idle && gl.nr_running)
			*sd_idle = 0;

		if (!need_balance && balance) {
			*balance = 0;
			*imbalance = 0;
			return NULL;
		}

		/* Compare groups and find busiest non-local group */
		update_sd_loads(&sdl, &gl);
		/* Compare groups and find power saving candidates */
		update_powersavings_group_loads(&sdl, &gl, idle);

		group = group->next;
	} while (group != sd->groups);

	if (!sdl.busiest.group ||
	     sdl.local.load >= sdl.max_load ||
	     sdl.busiest.nr_running == 0)
		goto out_balanced;

	sdl.avg_load = (SCHED_LOAD_SCALE * sdl.load) / sdl.cpu_power;

	if (sdl.local.load >= sdl.avg_load ||
			100*sdl.load <= sd->imbalance_pct*sdl.local.load)
		goto out_balanced;

	if (sdl.busiest.group_imbalance)
		sdl.busiest.avg_load_per_task =
			min(sdl.busiest.avg_load_per_task, sdl.avg_load);

	/*
	 * We're trying to get all the cpus to the average_load, so we don't
	 * want to push ourselves above the average load, nor do we wish to
	 * reduce the max loaded cpu below the average load, as either of these
	 * actions would just result in more rebalancing later, and ping-pong
	 * tasks around. Thus we look for the minimum possible imbalance.
	 * Negative imbalances (*we* are more loaded than anyone else) will
	 * be counted as no imbalance for these purposes -- we can't fix that
	 * by pulling tasks to us. Be careful of negative numbers as they'll
	 * appear as very large values with unsigned longs.
	 */
	if (sdl.max_load <= sdl.busiest.avg_load_per_task)
		goto out_balanced;

	/*
	 * In the presence of smp nice balancing, certain scenarios can have
	 * max load less than avg load(as we skip the groups at or below
	 * its cpu_power, while calculating max_load..)
	 * In this condition attempt to adjust the imbalance parameter
	 * in the small_imbalance functions.
	 *
	 * Now if max_load is more than avg load, balancing is needed,
	 * find the exact number of tasks to be moved.
	 */
	if (sdl.max_load >= sdl.avg_load) {

		/* Don't want to pull so many tasks that
		 * a group would go idle
		 */
		max_pull = min(sdl.max_load - sdl.avg_load,
				sdl.max_load - sdl.busiest.avg_load_per_task);

		/* How much load to actually move to equalise the imbalance */
		*imbalance = min(max_pull * sdl.busiest.group->__cpu_power,
				(sdl.avg_load - sdl.local.load) *
				 sdl.local.group->__cpu_power) /
				 SCHED_LOAD_SCALE;

		/* If we have adjusted the required imbalance, then return */
		if (*imbalance >= sdl.busiest.avg_load_per_task)
			return sdl.busiest.group;

	}

	/*
	 * if *imbalance is less than the average load per runnable task
	 * there is no gaurantee that any tasks will be moved so we'll have
	 * a think about bumping its value to force at least one task to be
	 * moved
	 */
	*imbalance = 0;  /* Will be adjusted below */

	if (small_imbalance_one_task(&sdl, imbalance))
		return sdl.busiest.group;

	/* Further look for effective cpu power utilisation */
	small_imbalance_optimize_cpu_power(&sdl, imbalance);

	/*
	 * Un conditional return, we have tries all possible means to adjust
	 * the imbalance for effective task move
	 */
	return sdl.busiest.group;

out_balanced:
	/* Try opportuinity for power save balance */
	return powersavings_balance_group(&sdl, &gl, idle, imbalance);
}


---

Vaidyanathan Srinivasan (5):
      Split find_busiest_group()
      Small imbalance corrections
      Collect statistics required for powersave balance
      Calculate statistics for current load balance domain
      Load calculation for each group


 kernel/sched.c |  612 ++++++++++++++++++++++++++++++++++----------------------
 1 files changed, 370 insertions(+), 242 deletions(-)

-- 

[RFC PATCH v1 1/5] Load calculation for each group

[Vaidyanathan Srinivasan]

Add data structures for per group stats, and function
to calculate the required stats.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
---

 kernel/sched.c |  114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 114 insertions(+), 0 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 9889080..abf60ea 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3064,6 +3064,120 @@ static int move_one_task(struct rq *this_rq, int this_cpu, struct rq *busiest,
 	return 0;
 }
 
+/* Helper functions for find_busiest_group */
+
+int get_load_idx(struct sched_domain *sd, enum cpu_idle_type idle)
+{
+	if (idle == CPU_NOT_IDLE)
+		return sd->busy_idx;
+	else if (idle == CPU_NEWLY_IDLE)
+		return sd->newidle_idx;
+	else
+		return sd->idle_idx;
+}
+
+/* Struct to return group stats */
+
+struct group_loads {
+	struct sched_group *group;
+	unsigned long nr_running;
+	unsigned long load;
+	unsigned long load_per_cpu;
+	unsigned long weighted_load;
+	unsigned long avg_load_per_task;
+	unsigned int group_imbalance;
+	int local_group;
+	int balance_cpu;
+};
+
+/* Helper function to calculate basic group level stats */
+
+int get_group_loads(struct sched_group *group, int this_cpu,
+		    const cpumask_t *valid_cpus, enum cpu_idle_type idle,
+		    int load_idx,
+		    struct group_loads *gl)
+{
+	struct rq *rq;
+	unsigned long load, min_load, max_load, avg_load_per_task_per_cpu;
+	int cpu;
+	int local_group = 0;
+	int first_idle_cpu = -1;
+	int need_balance = 1;
+
+	gl->group = group;
+	gl->nr_running = 0;
+	gl->load = 0;
+	gl->weighted_load = 0;
+	gl->avg_load_per_task = 0;
+	gl->group_imbalance = 0;
+	gl->balance_cpu = -1;
+	max_load = 0;
+	min_load = ~0UL;
+
+	gl->local_group = cpu_isset(this_cpu, group->cpumask);
+
+	for_each_cpu_mask_nr(cpu, group->cpumask) {
+		if (!cpu_isset(cpu, *valid_cpus))
+			continue;
+
+		rq = cpu_rq(cpu);
+
+		/* Bias balancing toward cpus of our domain */
+		if (gl->local_group) {
+			if (idle_cpu(cpu) && first_idle_cpu == -1)
+				first_idle_cpu = cpu;
+
+			load = target_load(cpu, load_idx);
+		} else {
+			load = source_load(cpu, load_idx);
+			if (load > max_load)
+				max_load = load;
+			if (load < min_load)
+				min_load = load;
+		}
+		gl->nr_running += rq->nr_running;
+		gl->load += load;
+		gl->weighted_load += weighted_cpuload(cpu);
+		gl->avg_load_per_task += cpu_avg_load_per_task(cpu);
+	}
+
+	/*
+	 * Consider the group unbalanced when the imbalance is larger
+	 * than the average weight of two tasks.
+	 *
+	 * APZ: with cgroup the avg task weight can vary wildly and
+	 *      might not be a suitable number - should we keep a
+	 *      normalized nr_running number somewhere that negates
+	 *      the hierarchy?
+	 */
+
+	avg_load_per_task_per_cpu = sg_div_cpu_power(group,
+				gl->avg_load_per_task * SCHED_LOAD_SCALE);
+
+	if (!gl->local_group &&
+	    ((max_load - min_load) > 2*avg_load_per_task_per_cpu))
+			gl->group_imbalance = 1;
+
+	if (local_group) {
+		if (first_idle_cpu != -1)
+			gl->balance_cpu = first_idle_cpu;
+		else
+			gl->balance_cpu = first_cpu(group->cpumask);
+
+		/*
+		 * First idle cpu or the first cpu(busiest) in this sched group
+		 * is eligible for doing load balancing at this and above
+		 * domains. In the newly idle case, we will allow all the cpu's
+		 * to do the newly idle load balance.
+		 */
+		if (idle != CPU_NEWLY_IDLE && gl->balance_cpu != this_cpu)
+			need_balance = 0;
+	}
+	gl->load_per_cpu = sg_div_cpu_power(group, gl->load * SCHED_LOAD_SCALE);
+
+	return need_balance;
+}
+
 /*
  * find_busiest_group finds and returns the busiest CPU group within the
  * domain. It calculates and returns the amount of weighted load which

[RFC PATCH v1 2/5] Calculate statistics for current load balance domain

[Vaidyanathan Srinivasan]

Add data structures and function to collect per sched domain 
statistics required for load balance decision.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
---

 kernel/sched.c |   41 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index abf60ea..8c394a4 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3178,6 +3178,47 @@ int get_group_loads(struct sched_group *group, int this_cpu,
 	return need_balance;
 }
 
+/* Struct to hold sched domain level loads */
+
+struct sd_loads {
+	struct sched_domain *sd;
+	unsigned long load;
+	unsigned long cpu_power;
+	unsigned long max_load;
+	unsigned long avg_load;
+	struct group_loads local;
+	struct group_loads busiest;
+	/* sched_mc balance groups */
+	struct group_loads min_load_group;
+	struct group_loads power_save_leader_group;
+};
+
+/* Function to aggregate per group loads to sched domain loads */
+
+void update_sd_loads(struct sd_loads *sdl, struct group_loads *gl)
+{
+	int group_capacity = gl->group->__cpu_power / SCHED_LOAD_SCALE;
+
+	if (gl->local_group)
+		sdl->local = *gl;
+	else {
+		sdl->load += gl->load;
+		sdl->cpu_power += gl->group->__cpu_power;
+
+
+		/* Find the busiest */
+		if (gl->load > sdl->load &&
+			(gl->nr_running > group_capacity ||
+			 gl->group_imbalance)) {
+
+			sdl->load = gl->load;
+			sdl->busiest = *gl;
+		}
+	}
+
+
+}
+
 /*
  * find_busiest_group finds and returns the busiest CPU group within the
  * domain. It calculates and returns the amount of weighted load which

[RFC PATCH v1 3/5] Collect statistics required for powersave balance

[Vaidyanathan Srinivasan]

Update sched domain level statistics with the minimum load and
group leader who can pull more tasks. Also suggest a powersave
movement if the domain is otherwise balanced.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
---

 kernel/sched.c |   97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 97 insertions(+), 0 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 8c394a4..dd87061 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3219,6 +3219,103 @@ void update_sd_loads(struct sd_loads *sdl, struct group_loads *gl)
 
 }
 
+#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
+void update_powersavings_group_loads(struct sd_loads *sdl,
+				     struct group_loads *gl,
+				     enum cpu_idle_type idle)
+{
+	int group_capacity = gl->group->__cpu_power / SCHED_LOAD_SCALE;
+
+	/*
+	 * Busy processors will not participate in power savings
+	 * balance.
+	 */
+	if (idle == CPU_NOT_IDLE ||
+			!(sdl->sd->flags & SD_POWERSAVINGS_BALANCE))
+		return;
+
+	/*
+	 * If the local group is idle or completely loaded
+	 * no need to do power savings balance at this domain
+	 * Same case of non-local groups as well
+	 */
+	if (gl->nr_running >= group_capacity || gl->nr_running == 0)
+		return;
+
+	/*
+	 * Calculate the group which has the least non-idle load.
+	 * This is the group from where we need to pick up the load
+	 * for saving power
+	 */
+	if (!sdl->min_load_group.group)
+		sdl->min_load_group = *gl;
+	else {
+		if (gl->nr_running < sdl->min_load_group.nr_running)
+			sdl->min_load_group = *gl;
+		/* If the loads are equal, then prefer the cpu with
+		 * less logical number
+		 */
+		else if (gl->nr_running == sdl->min_load_group.nr_running &&
+			 first_cpu(gl->group->cpumask) <
+			 first_cpu(sdl->min_load_group.group->cpumask))
+			sdl->min_load_group = *gl;
+	}
+
+	/*
+	 * Calculate the group which is almost near its
+	 * capacity but still has some space to pick up some load
+	 * from other group and save more power
+	 */
+
+	if (gl->nr_running > 0 && gl->nr_running <= group_capacity - 1) {
+		if (!sdl->power_save_leader_group.group)
+			sdl->power_save_leader_group = *gl;
+		else {
+			if (gl->nr_running >
+			    sdl->power_save_leader_group.nr_running)
+				sdl->power_save_leader_group = *gl;
+			else if (gl->nr_running ==
+				 sdl->power_save_leader_group.nr_running &&
+				 first_cpu(gl->group->cpumask) <
+				 first_cpu(sdl->min_load_group.group->cpumask))
+				sdl->power_save_leader_group = *gl;
+		}
+	}
+}
+
+static struct sched_group *powersavings_balance_group(struct sd_loads *sdl,
+	struct group_loads *gl, enum cpu_idle_type idle,
+	unsigned long *imbalance)
+{
+	*imbalance = 0;
+	if (idle == CPU_NOT_IDLE || !(sdl->sd->flags & SD_POWERSAVINGS_BALANCE))
+		return NULL;
+
+	if (sdl->local.group == sdl->power_save_leader_group.group &&
+		sdl->power_save_leader_group.group !=
+		sdl->min_load_group.group) {
+		*imbalance = sdl->min_load_group.avg_load_per_task;
+		return sdl->min_load_group.group;
+	}
+
+	return NULL;
+}
+#else
+void update_powersavings_group_loads(struct sd_loads *sdl,
+			struct group_loads *gl, enum cpu_idle_type idle)
+{
+	return;
+}
+
+static struct sched_group *powersavings_balance_group(struct sd_loads *sdl,
+	struct group_loads *gl, enum cpu_idle_type idle,
+	unsigned long *imbalance)
+{
+	*imbalance = 0;
+	return NULL;
+}
+#endif
+
 /*
  * find_busiest_group finds and returns the busiest CPU group within the
  * domain. It calculates and returns the amount of weighted load which

[RFC PATCH v1 4/5] Small imbalance corrections

[Vaidyanathan Srinivasan]

Add functions to bump up the imbalance to eventually initiate 
a task move to balance across groups.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
---

 kernel/sched.c |   72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index dd87061..1f38e2c 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3219,6 +3219,78 @@ void update_sd_loads(struct sd_loads *sdl, struct group_loads *gl)
 
 }
 
+/* Bump up imbalance to one task so that some task movement can happen */
+
+int small_imbalance_one_task(struct sd_loads *sdl, unsigned long *imbalance)
+{
+	unsigned int imbn;
+	imbn = 2;
+	if (sdl->local.nr_running) {
+		if (sdl->busiest.avg_load_per_task >
+			sdl->local.avg_load_per_task)
+			imbn = 1;
+	}
+
+	if (sdl->max_load - sdl->local.load +
+		2*sdl->busiest.avg_load_per_task >=
+				sdl->busiest.avg_load_per_task * imbn) {
+		*imbalance = sdl->busiest.avg_load_per_task;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Adjust imbalance to move task if the result of the move will
+ * yield better use of cpu power
+ */
+
+void small_imbalance_optimize_cpu_power(struct sd_loads *sdl,
+					unsigned long *imbalance)
+{
+	unsigned long tmp, pwr_now, pwr_move;
+	pwr_move = pwr_now = 0;
+
+	/*
+	 * OK, we don't have enough imbalance to justify moving tasks,
+	 * however we may be able to increase total CPU power used by
+	 * moving them.
+	 */
+
+	pwr_now += sdl->busiest.group->__cpu_power *
+			min(sdl->busiest.avg_load_per_task, sdl->max_load);
+	pwr_now += sdl->local.group->__cpu_power *
+			min(sdl->local.avg_load_per_task, sdl->local.load);
+	pwr_now /= SCHED_LOAD_SCALE;
+
+	/* Amount of load we'd subtract */
+	tmp = sg_div_cpu_power(sdl->busiest.group,
+			sdl->busiest.avg_load_per_task * SCHED_LOAD_SCALE);
+	if (sdl->max_load > tmp)
+		pwr_move += sdl->busiest.group->__cpu_power *
+			min(sdl->busiest.avg_load_per_task,
+			    sdl->max_load - tmp);
+
+	/* Amount of load we'd add */
+	if (sdl->max_load * sdl->busiest.group->__cpu_power <
+			sdl->busiest.avg_load_per_task * SCHED_LOAD_SCALE)
+		tmp = sg_div_cpu_power(sdl->local.group,
+				sdl->max_load *
+				sdl->busiest.group->__cpu_power);
+	else
+		tmp = sg_div_cpu_power(sdl->local.group,
+			sdl->busiest.avg_load_per_task * SCHED_LOAD_SCALE);
+	pwr_move += sdl->local.group->__cpu_power *
+			min(sdl->local.avg_load_per_task,
+			sdl->local.load + tmp);
+	pwr_move /= SCHED_LOAD_SCALE;
+
+	/* Move if we gain throughput */
+	if (pwr_move > pwr_now)
+		*imbalance = sdl->busiest.avg_load_per_task;
+
+}
+
 #if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
 void update_powersavings_group_loads(struct sd_loads *sdl,
 				     struct group_loads *gl,

[RFC PATCH v1 5/5] Split find_busiest_group()

[Vaidyanathan Srinivasan]

Make use of the previously defined helper functions and data
structures and split the find_busiest_group() function into
smaller modular functions.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
---

 kernel/sched.c |  324 +++++++++++---------------------------------------------
 1 files changed, 64 insertions(+), 260 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 1f38e2c..1cd2e60 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3387,7 +3387,6 @@ static struct sched_group *powersavings_balance_group(struct sd_loads *sdl,
 	return NULL;
 }
 #endif
-
 /*
  * find_busiest_group finds and returns the busiest CPU group within the
  * domain. It calculates and returns the amount of weighted load which
@@ -3398,208 +3397,55 @@ find_busiest_group(struct sched_domain *sd, int this_cpu,
 		   unsigned long *imbalance, enum cpu_idle_type idle,
 		   int *sd_idle, const cpumask_t *cpus, int *balance)
 {
-	struct sched_group *busiest = NULL, *this = NULL, *group = sd->groups;
-	unsigned long max_load, avg_load, total_load, this_load, total_pwr;
+	struct sched_group *group = sd->groups;
 	unsigned long max_pull;
-	unsigned long busiest_load_per_task, busiest_nr_running;
-	unsigned long this_load_per_task, this_nr_running;
-	int load_idx, group_imb = 0;
-#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
-	int power_savings_balance = 1;
-	unsigned long leader_nr_running = 0, min_load_per_task = 0;
-	unsigned long min_nr_running = ULONG_MAX;
-	struct sched_group *group_min = NULL, *group_leader = NULL;
-#endif
+	int load_idx;
+	struct group_loads gl;
+	struct sd_loads sdl;
 
-	max_load = this_load = total_load = total_pwr = 0;
-	busiest_load_per_task = busiest_nr_running = 0;
-	this_load_per_task = this_nr_running = 0;
+	memset(&sdl, 0, sizeof(sdl));
+	sdl.sd = sd;
 
-	if (idle == CPU_NOT_IDLE)
-		load_idx = sd->busy_idx;
-	else if (idle == CPU_NEWLY_IDLE)
-		load_idx = sd->newidle_idx;
-	else
-		load_idx = sd->idle_idx;
+	/* Get the load index corresponding to cpu idle state */
+	load_idx = get_load_idx(sd, idle);
 
 	do {
-		unsigned long load, group_capacity, max_cpu_load, min_cpu_load;
-		int local_group;
-		int i;
-		int __group_imb = 0;
-		unsigned int balance_cpu = -1, first_idle_cpu = 0;
-		unsigned long sum_nr_running, sum_weighted_load;
-		unsigned long sum_avg_load_per_task;
-		unsigned long avg_load_per_task;
-
-		local_group = cpu_isset(this_cpu, group->cpumask);
-
-		if (local_group)
-			balance_cpu = first_cpu(group->cpumask);
+		int need_balance;
 
-		/* Tally up the load of all CPUs in the group */
-		sum_weighted_load = sum_nr_running = avg_load = 0;
-		sum_avg_load_per_task = avg_load_per_task = 0;
-
-		max_cpu_load = 0;
-		min_cpu_load = ~0UL;
-
-		for_each_cpu_mask_nr(i, group->cpumask) {
-			struct rq *rq;
-
-			if (!cpu_isset(i, *cpus))
-				continue;
+		need_balance = get_group_loads(group, this_cpu, cpus, idle,
+					       load_idx, &gl);
 
-			rq = cpu_rq(i);
+		if (*sd_idle && gl.nr_running)
+			*sd_idle = 0;
 
-			if (*sd_idle && rq->nr_running)
-				*sd_idle = 0;
-
-			/* Bias balancing toward cpus of our domain */
-			if (local_group) {
-				if (idle_cpu(i) && !first_idle_cpu) {
-					first_idle_cpu = 1;
-					balance_cpu = i;
-				}
-
-				load = target_load(i, load_idx);
-			} else {
-				load = source_load(i, load_idx);
-				if (load > max_cpu_load)
-					max_cpu_load = load;
-				if (min_cpu_load > load)
-					min_cpu_load = load;
-			}
-
-			avg_load += load;
-			sum_nr_running += rq->nr_running;
-			sum_weighted_load += weighted_cpuload(i);
-
-			sum_avg_load_per_task += cpu_avg_load_per_task(i);
-		}
-
-		/*
-		 * First idle cpu or the first cpu(busiest) in this sched group
-		 * is eligible for doing load balancing at this and above
-		 * domains. In the newly idle case, we will allow all the cpu's
-		 * to do the newly idle load balance.
-		 */
-		if (idle != CPU_NEWLY_IDLE && local_group &&
-		    balance_cpu != this_cpu && balance) {
+		if (!need_balance && balance) {
 			*balance = 0;
-			goto ret;
+			*imbalance = 0;
+			return NULL;
 		}
 
-		total_load += avg_load;
-		total_pwr += group->__cpu_power;
+		/* Compare groups and find busiest non-local group */
+		update_sd_loads(&sdl, &gl);
+		/* Compare groups and find power saving candidates */
+		update_powersavings_group_loads(&sdl, &gl, idle);
 
-		/* Adjust by relative CPU power of the group */
-		avg_load = sg_div_cpu_power(group,
-				avg_load * SCHED_LOAD_SCALE);
-
-
-		/*
-		 * Consider the group unbalanced when the imbalance is larger
-		 * than the average weight of two tasks.
-		 *
-		 * APZ: with cgroup the avg task weight can vary wildly and
-		 *      might not be a suitable number - should we keep a
-		 *      normalized nr_running number somewhere that negates
-		 *      the hierarchy?
-		 */
-		avg_load_per_task = sg_div_cpu_power(group,
-				sum_avg_load_per_task * SCHED_LOAD_SCALE);
-
-		if ((max_cpu_load - min_cpu_load) > 2*avg_load_per_task)
-			__group_imb = 1;
-
-		group_capacity = group->__cpu_power / SCHED_LOAD_SCALE;
-
-		if (local_group) {
-			this_load = avg_load;
-			this = group;
-			this_nr_running = sum_nr_running;
-			this_load_per_task = sum_weighted_load;
-		} else if (avg_load > max_load &&
-			   (sum_nr_running > group_capacity || __group_imb)) {
-			max_load = avg_load;
-			busiest = group;
-			busiest_nr_running = sum_nr_running;
-			busiest_load_per_task = sum_weighted_load;
-			group_imb = __group_imb;
-		}
-
-#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
-		/*
-		 * Busy processors will not participate in power savings
-		 * balance.
-		 */
-		if (idle == CPU_NOT_IDLE ||
-				!(sd->flags & SD_POWERSAVINGS_BALANCE))
-			goto group_next;
-
-		/*
-		 * If the local group is idle or completely loaded
-		 * no need to do power savings balance at this domain
-		 */
-		if (local_group && (this_nr_running >= group_capacity ||
-				    !this_nr_running))
-			power_savings_balance = 0;
-
-		/*
-		 * If a group is already running at full capacity or idle,
-		 * don't include that group in power savings calculations
-		 */
-		if (!power_savings_balance || sum_nr_running >= group_capacity
-		    || !sum_nr_running)
-			goto group_next;
-
-		/*
-		 * Calculate the group which has the least non-idle load.
-		 * This is the group from where we need to pick up the load
-		 * for saving power
-		 */
-		if ((sum_nr_running < min_nr_running) ||
-		    (sum_nr_running == min_nr_running &&
-		     first_cpu(group->cpumask) <
-		     first_cpu(group_min->cpumask))) {
-			group_min = group;
-			min_nr_running = sum_nr_running;
-			min_load_per_task = sum_weighted_load /
-						sum_nr_running;
-		}
-
-		/*
-		 * Calculate the group which is almost near its
-		 * capacity but still has some space to pick up some load
-		 * from other group and save more power
-		 */
-		if (sum_nr_running <= group_capacity - 1) {
-			if (sum_nr_running > leader_nr_running ||
-			    (sum_nr_running == leader_nr_running &&
-			     first_cpu(group->cpumask) >
-			      first_cpu(group_leader->cpumask))) {
-				group_leader = group;
-				leader_nr_running = sum_nr_running;
-			}
-		}
-group_next:
-#endif
 		group = group->next;
 	} while (group != sd->groups);
 
-	if (!busiest || this_load >= max_load || busiest_nr_running == 0)
+	if (!sdl.busiest.group ||
+	     sdl.local.load >= sdl.max_load ||
+	     sdl.busiest.nr_running == 0)
 		goto out_balanced;
 
-	avg_load = (SCHED_LOAD_SCALE * total_load) / total_pwr;
+	sdl.avg_load = (SCHED_LOAD_SCALE * sdl.load) / sdl.cpu_power;
 
-	if (this_load >= avg_load ||
-			100*max_load <= sd->imbalance_pct*this_load)
+	if (sdl.local.load >= sdl.avg_load ||
+			100*sdl.load <= sd->imbalance_pct*sdl.local.load)
 		goto out_balanced;
 
-	busiest_load_per_task /= busiest_nr_running;
-	if (group_imb)
-		busiest_load_per_task = min(busiest_load_per_task, avg_load);
+	if (sdl.busiest.group_imbalance)
+		sdl.busiest.avg_load_per_task =
+			min(sdl.busiest.avg_load_per_task, sdl.avg_load);
 
 	/*
 	 * We're trying to get all the cpus to the average_load, so we don't
@@ -3612,26 +3458,38 @@ group_next:
 	 * by pulling tasks to us. Be careful of negative numbers as they'll
 	 * appear as very large values with unsigned longs.
 	 */
-	if (max_load <= busiest_load_per_task)
+	if (sdl.max_load <= sdl.busiest.avg_load_per_task)
 		goto out_balanced;
 
 	/*
 	 * In the presence of smp nice balancing, certain scenarios can have
 	 * max load less than avg load(as we skip the groups at or below
 	 * its cpu_power, while calculating max_load..)
+	 * In this condition attempt to adjust the imbalance parameter
+	 * in the small_imbalance functions.
+	 *
+	 * Now if max_load is more than avg load, balancing is needed,
+	 * find the exact number of tasks to be moved.
 	 */
-	if (max_load < avg_load) {
-		*imbalance = 0;
-		goto small_imbalance;
-	}
+	if (sdl.max_load >= sdl.avg_load) {
+
+		/* Don't want to pull so many tasks that
+		 * a group would go idle
+		 */
+		max_pull = min(sdl.max_load - sdl.avg_load,
+				sdl.max_load - sdl.busiest.avg_load_per_task);
+
+		/* How much load to actually move to equalise the imbalance */
+		*imbalance = min(max_pull * sdl.busiest.group->__cpu_power,
+				(sdl.avg_load - sdl.local.load) *
+				 sdl.local.group->__cpu_power) /
+				 SCHED_LOAD_SCALE;
 
-	/* Don't want to pull so many tasks that a group would go idle */
-	max_pull = min(max_load - avg_load, max_load - busiest_load_per_task);
+		/* If we have adjusted the required imbalance, then return */
+		if (*imbalance >= sdl.busiest.avg_load_per_task)
+			return sdl.busiest.group;
 
-	/* How much load to actually move to equalise the imbalance */
-	*imbalance = min(max_pull * busiest->__cpu_power,
-				(avg_load - this_load) * this->__cpu_power)
-			/ SCHED_LOAD_SCALE;
+	}
 
 	/*
 	 * if *imbalance is less than the average load per runnable task
@@ -3639,77 +3497,23 @@ group_next:
 	 * a think about bumping its value to force at least one task to be
 	 * moved
 	 */
-	if (*imbalance < busiest_load_per_task) {
-		unsigned long tmp, pwr_now, pwr_move;
-		unsigned int imbn;
-
-small_imbalance:
-		pwr_move = pwr_now = 0;
-		imbn = 2;
-		if (this_nr_running) {
-			this_load_per_task /= this_nr_running;
-			if (busiest_load_per_task > this_load_per_task)
-				imbn = 1;
-		} else
-			this_load_per_task = cpu_avg_load_per_task(this_cpu);
-
-		if (max_load - this_load + 2*busiest_load_per_task >=
-					busiest_load_per_task * imbn) {
-			*imbalance = busiest_load_per_task;
-			return busiest;
-		}
+	*imbalance = 0;  /* Will be adjusted below */
 
-		/*
-		 * OK, we don't have enough imbalance to justify moving tasks,
-		 * however we may be able to increase total CPU power used by
-		 * moving them.
-		 */
+	if (small_imbalance_one_task(&sdl, imbalance))
+		return sdl.busiest.group;
 
-		pwr_now += busiest->__cpu_power *
-				min(busiest_load_per_task, max_load);
-		pwr_now += this->__cpu_power *
-				min(this_load_per_task, this_load);
-		pwr_now /= SCHED_LOAD_SCALE;
-
-		/* Amount of load we'd subtract */
-		tmp = sg_div_cpu_power(busiest,
-				busiest_load_per_task * SCHED_LOAD_SCALE);
-		if (max_load > tmp)
-			pwr_move += busiest->__cpu_power *
-				min(busiest_load_per_task, max_load - tmp);
-
-		/* Amount of load we'd add */
-		if (max_load * busiest->__cpu_power <
-				busiest_load_per_task * SCHED_LOAD_SCALE)
-			tmp = sg_div_cpu_power(this,
-					max_load * busiest->__cpu_power);
-		else
-			tmp = sg_div_cpu_power(this,
-				busiest_load_per_task * SCHED_LOAD_SCALE);
-		pwr_move += this->__cpu_power *
-				min(this_load_per_task, this_load + tmp);
-		pwr_move /= SCHED_LOAD_SCALE;
+	/* Further look for effective cpu power utilisation */
+	small_imbalance_optimize_cpu_power(&sdl, imbalance);
 
-		/* Move if we gain throughput */
-		if (pwr_move > pwr_now)
-			*imbalance = busiest_load_per_task;
-	}
-
-	return busiest;
+	/*
+	 * Un conditional return, we have tries all possible means to adjust
+	 * the imbalance for effective task move
+	 */
+	return sdl.busiest.group;
 
 out_balanced:
-#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
-	if (idle == CPU_NOT_IDLE || !(sd->flags & SD_POWERSAVINGS_BALANCE))
-		goto ret;
-
-	if (this == group_leader && group_leader != group_min) {
-		*imbalance = min_load_per_task;
-		return group_min;
-	}
-#endif
-ret:
-	*imbalance = 0;
-	return NULL;
+	/* Try opportuinity for power save balance */
+	return powersavings_balance_group(&sdl, &gl, idle, imbalance);
 }
 
 /*

Re: [RFC PATCH v1 3/5] Collect statistics required for powersave balance

[Gautham R Shenoy]

On Wed, Sep 24, 2008 at 09:48:09PM +0530, Vaidyanathan Srinivasan wrote:
> Update sched domain level statistics with the minimum load and
> group leader who can pull more tasks. Also suggest a powersave
> movement if the domain is otherwise balanced.
> 
> Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> ---
> 
>  kernel/sched.c |   97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 97 insertions(+), 0 deletions(-)
> 
> diff --git a/kernel/sched.c b/kernel/sched.c
> index 8c394a4..dd87061 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -3219,6 +3219,103 @@ void update_sd_loads(struct sd_loads *sdl, struct group_loads *gl)
> 
>  }
> 
> +#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
> +void update_powersavings_group_loads(struct sd_loads *sdl,
> +				     struct group_loads *gl,
> +				     enum cpu_idle_type idle)
> +{
> +	int group_capacity = gl->group->__cpu_power / SCHED_LOAD_SCALE;
> +
> +	/*
> +	 * Busy processors will not participate in power savings
> +	 * balance.
> +	 */
> +	if (idle == CPU_NOT_IDLE ||
> +			!(sdl->sd->flags & SD_POWERSAVINGS_BALANCE))
> +		return;
> +
> +	/*
> +	 * If the local group is idle or completely loaded
> +	 * no need to do power savings balance at this domain
> +	 * Same case of non-local groups as well
> +	 */
	Are there groups other than local and non-local ?
	If not, this comment can be condensed so that it can be applied
	to all groups :-)

> +	if (gl->nr_running >= group_capacity || gl->nr_running == 0)
> +		return;
> +
> +	/*
> +	 * Calculate the group which has the least non-idle load.
> +	 * This is the group from where we need to pick up the load
> +	 * for saving power
> +	 */
> +	if (!sdl->min_load_group.group)
> +		sdl->min_load_group = *gl;
> +	else {
> +		if (gl->nr_running < sdl->min_load_group.nr_running)
> +			sdl->min_load_group = *gl;
> +		/* If the loads are equal, then prefer the cpu with
> +		 * less logical number
> +		 */
> +		else if (gl->nr_running == sdl->min_load_group.nr_running &&
> +			 first_cpu(gl->group->cpumask) <
> +			 first_cpu(sdl->min_load_group.group->cpumask))
> +			sdl->min_load_group = *gl;
> +	}
> +
> +	/*
> +	 * Calculate the group which is almost near its
> +	 * capacity but still has some space to pick up some load
> +	 * from other group and save more power
> +	 */
> +
> +	if (gl->nr_running > 0 && gl->nr_running <= group_capacity - 1) {
> +		if (!sdl->power_save_leader_group.group)
> +			sdl->power_save_leader_group = *gl;
> +		else {
> +			if (gl->nr_running >
> +			    sdl->power_save_leader_group.nr_running)
> +				sdl->power_save_leader_group = *gl;
> +			else if (gl->nr_running ==
> +				 sdl->power_save_leader_group.nr_running &&
> +				 first_cpu(gl->group->cpumask) <
> +				 first_cpu(sdl->min_load_group.group->cpumask))
> +				sdl->power_save_leader_group = *gl;
> +		}
> +	}
> +}
> +
> +static struct sched_group *powersavings_balance_group(struct sd_loads *sdl,
> +	struct group_loads *gl, enum cpu_idle_type idle,
> +	unsigned long *imbalance)
> +{
> +	*imbalance = 0;
> +	if (idle == CPU_NOT_IDLE || !(sdl->sd->flags & SD_POWERSAVINGS_BALANCE))
> +		return NULL;
> +
> +	if (sdl->local.group == sdl->power_save_leader_group.group &&
> +		sdl->power_save_leader_group.group !=
> +		sdl->min_load_group.group) {
> +		*imbalance = sdl->min_load_group.avg_load_per_task;
> +		return sdl->min_load_group.group;
> +	}
> +
> +	return NULL;
> +}
> +#else
> +void update_powersavings_group_loads(struct sd_loads *sdl,
> +			struct group_loads *gl, enum cpu_idle_type idle)
> +{
> +	return;
> +}
> +
> +static struct sched_group *powersavings_balance_group(struct sd_loads *sdl,
> +	struct group_loads *gl, enum cpu_idle_type idle,
> +	unsigned long *imbalance)
> +{
> +	*imbalance = 0;
> +	return NULL;
> +}
> +#endif
> +
>  /*
>   * find_busiest_group finds and returns the busiest CPU group within the
>   * domain. It calculates and returns the amount of weighted load which

-- 
Thanks and Regards
gautham