sendfile() with 100 simultaneous 100MB files

[Jon Smirl <jonsmirl@gmail.com>]

I was reading this blog post about the lighttpd web server.
http://blog.lighttpd.net/articles/2005/11/11/optimizing-lighty-for-high-concurrent-large-file-downloads
It describes problems they are having downloading 100 simultaneous 100MB files.

In this post they complain about sendfile() getting into seek storms and
ending up in 72% IO wait. As a result they built a user space
mechanism to work around the problems.

I tried looking at how the kernel implements sendfile(), I have
minimal understanding of how the fs code works but it looks to me like
sendfile() is working a page at a time. I was looking for code that
does something like this...

1) Compute an adaptive window size and read ahead the appropriate
number of pages.  A larger window would minimize disk seeks.

2) Something along the lines of as soon as a page is sent age the page
down in to the middle of page ages. That would allow for files that
are repeatedly sent, but also reduce thrashing from files that are not
sent frequently and shouldn't stay in the page cache.

Any other ideas why sendfile() would get into a seek storm?

--
Jon Smirl
jonsmirl@gmail.com