Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Erik Cumps <erik.cumps@esaturnus.com>]

Hi,

we are noticing some strange block device IO performance and our 
investigations are leading us away from the hardware and towards the
kernel. This could be a simple tuning problem or a known issue, so
before taking a deep dive down kernel sources and debian kernel patches
we would like to rule out the simple things first.

The context is a 16 GB 32-bit intel debian workstation, using an ext4
filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
machines. The host system (so not the virual machines) shows sporadic
extremely slow write performance (around 4 megabytes per second).
However, if we use the debian 3.2.0 kernel this problem does not
manifest itself.

We've created a simple IO performance test script to investigate this.
It is basically a smart wrapper around dd, copying data between the
block device under test and a ramdisk filesystem, using flags to select
the usage of cache, sync and direct io, clearing caches before the test
and running the write and read tests three times to account for
transient performance.

The results of these tests are unexpected: we see the expected normal
write performance when using uncached, unsynced, direct IO and very slow
write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!

This seems to rule out the block device and IO controller being at
fault. In fact, other tests showed the same performance discrepancy with
an NFS mounted filesystem and a platter disk.

We also noticed that when the performance is slow, if we shut down the
KVM virtual machines the performance returns to normal.

Maybe there is something going wrong with cache/buffer handling?
Thanks for your insights. 

I've kept this mail intentionally free from too many technical details
but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps