Direct io on block device has performance regression on 2.6.x kernel

["Chen, Kenneth W" <kenneth.w.chen@intel.com>]

I don't know where to start, but let me start with the bombshell:

Direct I/O on block device running 2.6.X kernel is a lot SLOWER
than running on a 2.4 Kernel!


Processing a direct I/O request on 2.6 is taking a lot more cpu
time compare to the same I/O request running on a 2.4 kernel.

The proof: easy.  I started off by having a pseudo disk, a software
disk that has zero access latency.  By hooking this pseudo disk into
the block layer API, I can effectively stress the entire I/O stack
above the block level.  Combined with user level test programs that
simply submit direct I/O in a simple while loop, I can measure how
fast kernel can process these I/O requests.  The performance metric
can be either throughput (# of I/O per second) or per unit of work
(processing time per I/O).  For the data presented below, I'm using
throughput metric (meaning larger number is better performance).
Pay attention to relative percentage as absolute number depends on
platform/CPU that test suite runs on.


			synchronous I/O			AIO
			(pread/pwrite/read/write)	io_submit
2.4.21 based
(RHEL3)		265,122				229,810

2.6.9			218,565				206,917
2.6.10		213,041				205,891
2.6.11		212,284				201,124

>From the above chart, you can see that 2.6 kernel is at least 18%
slower in processing direct I/O (on block device) in the synchronous
path and 10% slower in the AIO path compare to a distributor's 2.4
kernel.  What's worse, with each advance of kernel version, the I/O
path is becoming slower and slower.

Most of the performance regression for 2.6.9 came from dio layer (I
still have to find where the regression came from with 2.6.10 and 2.6.11).
DIO is just overloaded with too many areas to cover.  I think it's better
to break things up a little bit.

For example, by having a set of dedicated functions that do direct I/O
on block device improves the performance dramatically:

			synchronous I/O			AIO
			(pread/pwrite/read/write)	io_submit
2.4.21 based
(RHEL3)		265,122				229,810
2.6.9			218,565				206,917
2.6.9+patches	323,016				268,484

See, we can be actually 22% faster in synchronous path and 17% faster
In the AIO path, if we do it right!

Kernel patch and test suite to follow in the next couple postings.

- Ken