(Re-send for lkml.  For some reason kmail decided HTML mail was cool again).

On Thu, 4 Jun 2009 01:39:38 am Linus Torvalds wrote:
> On Wed, 3 Jun 2009, Rusty Russell wrote:
> > I took my standard config, and turned on AUDIT, CGROUP, all the sched
> > options, all the namespace options, profiling, markers, kprobes,
> > relocatable kernel, 1000Hz, preempt, support for every x86 variant (ie.
> > PAE, NUMA, HIGHMEM64, DISCONTIGMEM).  I turned off kernel debugging and
> > paravirt.  Booted with maxcpus=1.
>
> Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you
> can't compare it to a no-highmem case).

Thanks, your point is demonstrated below.  I don't think HIGHMEM4G is 
unreasonable for a distro tho, so I turned that on instead.

Anyway, so everyone can play along at home, I've extracted the context-switch 
test into a standalone proggie; run it in a loop to make it warm and watch the 
results.  I tested it here on my 3G RAM Core2 Duo laptop (travelling, away 
from previous test box), and here are typical results (./context-switch 1000):

minimal config:
0.001271
0.001279
0.001284
0.001279
0.001290

maximal config (as before, but NUMAQ=n, HIGHMEM4G, maxcpus=1):
0.002476
0.002507
0.002491
0.002518
0.002505

maximum config (as above with mem=880M ie. no actual highmem)
0.001917
0.001893
0.001936
0.001915
0.001925

So we're paying a 48% overhead; microbenchmarks always suffer as code is added, 
and we've added a lot of code with these options.

I've attached my maximal and minimal configs if people are overly curious.
Rusty.

#include <sys/types.h>
#include <signal.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <err.h>
#include <sys/time.h>
#include <stdio.h>

static void do_context_switch(int runs,
			      struct timeval *start, struct timeval *end)
{
	char c = 1;
	int fds1[2], fds2[2], child;

	if (pipe(fds1) != 0 || pipe(fds2) != 0)
		err(1, "Creating pipe");

	child = fork();
	if (child == -1)
		err(1, "forking");

	if (child > 0) {
		close(fds1[0]);
		close(fds2[1]);

		gettimeofday(start, NULL);
		while (runs > 0) {
			write(fds1[1], &c, 1);
			read(fds2[0], &c, 1);
			runs -= 2;
		}
		gettimeofday(end, NULL);

		waitpid(child, NULL, 0);
		close(fds1[1]);
		close(fds2[0]);
	} else {
		close(fds2[0]);
		close(fds1[1]);

		while (runs > 0) {
			read(fds1[0], &c, 1);
			write(fds2[1], &c, 1);
			runs -= 2;
		}
		exit(0);
	}
}

int main(int argc, char *argv[])
{
	struct timeval start, end, diff;

	if (argc != 2)
		errx(1, "Usage: context-switch <num-runs>");

	do_context_switch(atoi(argv[1]), &start, &end);

	timersub(&end, &start, &diff);

	printf("%lu.%06lu\n", diff.tv_sec, diff.tv_usec);
	return 0;
}