High scheduler wake up times

[Shawn Bohrer <shawn.bohrer@gmail.com>]

Hello,

Currently we have a workload that depends on around 50 processes that
wake up 1000 times a second do a small amount of work and go back to
sleep.  This works great on RHEL 5 (2.6.18-164.6.1.el5), but on recent
kernels we are unable to achieve 1000 iterations per second.  Using
the simple test application below on RHEL 5 2.6.18-164.6.1.el5 I can run
500 of these processes on and still achieve 999.99 iterations per
second.  Running just 10 of these processes on the same machine with
2.6.32.6 produces results like:

...
Iterations Per Sec: 905.659667
Iterations Per Sec: 805.099068
Iterations Per Sec: 925.195578
Iterations Per Sec: 759.310773
Iterations Per Sec: 702.849261
Iterations Per Sec: 782.157292
Iterations Per Sec: 917.138031
Iterations Per Sec: 834.770391
Iterations Per Sec: 850.543755
...

I've tried playing with some of the cfs tunables in /proc/sys/kernel/
without success.  Are there any suggestions on how to achieve the
results we are looking for using a recent kernel?

Thanks,
Shawn


#include <sys/epoll.h>
#include <sys/time.h>
#include <stdio.h>
#include <unistd.h>

int main ()
{
	int epfd = epoll_create(1);
	int i, j;
	struct timeval tv;
	unsigned long start, end;
	const unsigned int count = 60000;

	while (1) {
		gettimeofday(&tv, NULL);
		start = tv.tv_sec * 1000000 + tv.tv_usec;

		for (i = 0; i < count; ++i) {
			if (epoll_wait(epfd, 0, 1, 1) == -1)
				perror("epoll failed");

			for (j = 0; j < 10000; ++j)
				/* simulate work */;
		}
		gettimeofday(&tv, NULL);
		end = tv.tv_sec * 1000000 + tv.tv_usec;

		printf("Iterations Per Sec: %f\n", count/((double)(end - start)/1000000));
	}

	close(epfd);
}