Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Erik Cumps]

Hi,

we are noticing some strange block device IO performance and our 
investigations are leading us away from the hardware and towards the
kernel. This could be a simple tuning problem or a known issue, so
before taking a deep dive down kernel sources and debian kernel patches
we would like to rule out the simple things first.

The context is a 16 GB 32-bit intel debian workstation, using an ext4
filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
machines. The host system (so not the virual machines) shows sporadic
extremely slow write performance (around 4 megabytes per second).
However, if we use the debian 3.2.0 kernel this problem does not
manifest itself.

We've created a simple IO performance test script to investigate this.
It is basically a smart wrapper around dd, copying data between the
block device under test and a ramdisk filesystem, using flags to select
the usage of cache, sync and direct io, clearing caches before the test
and running the write and read tests three times to account for
transient performance.

The results of these tests are unexpected: we see the expected normal
write performance when using uncached, unsynced, direct IO and very slow
write performance using the regular cached IO. The difference is huge:
it is sometimes two orders of magnitude!

This seems to rule out the block device and IO controller being at
fault. In fact, other tests showed the same performance discrepancy with
an NFS mounted filesystem and a platter disk.

We also noticed that when the performance is slow, if we shut down the
KVM virtual machines the performance returns to normal.

Maybe there is something going wrong with cache/buffer handling?
Thanks for your insights. 

I've kept this mail intentionally free from too many technical details
but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps

Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Erik Cumps]

On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <erik.cumps@esaturnus.com> wrote:
>
> we are noticing some strange block device IO performance and our
> investigations are leading us away from the hardware and towards the
> kernel. This could be a simple tuning problem or a known issue, so
> before taking a deep dive down kernel sources and debian kernel patches
> we would like to rule out the simple things first.
>
> The context is a 16 GB 32-bit intel debian workstation, using an ext4
> filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
> recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
> machines. The host system (so not the virual machines) shows sporadic
> extremely slow write performance (around 4 megabytes per second).
> However, if we use the debian 3.2.0 kernel this problem does not
> manifest itself.
>
> We've created a simple IO performance test script to investigate this.
> It is basically a smart wrapper around dd, copying data between the
> block device under test and a ramdisk filesystem, using flags to select
> the usage of cache, sync and direct io, clearing caches before the test
> and running the write and read tests three times to account for
> transient performance.
>
> The results of these tests are unexpected: we see the expected normal
> write performance when using uncached, unsynced, direct IO and very slow
> write performance using the regular cached IO. The difference is huge:
> it is sometimes two orders of magnitude!

Actually, it is the *synchronous*, direct IO that matches the expected
raw write performance of the device.

The "regular IO" test is doing roughly this:

    echo 3 > /proc/sys/vm/drop_caches
    dd if=ramdisk_file of=test_file bs=1M count=100
    dd if=ramdisk_file of=test_file bs=1M count=100
    dd if=ramdisk_file of=test_file bs=1M count=100

The direct IO test is doing roughly this:

    echo 3 > /proc/sys/vm/drop_caches
    dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
    dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
    dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

> This seems to rule out the block device and IO controller being at
> fault. In fact, other tests showed the same performance discrepancy with
> an NFS mounted filesystem and a platter disk.
>
> We also noticed that when the performance is slow, if we shut down the
> KVM virtual machines the performance returns to normal.
>
> Maybe there is something going wrong with cache/buffer handling?
> Thanks for your insights.
>
> I've kept this mail intentionally free from too many technical details
> but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps

Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Dick Streefland]

On Thursday 2015-06-18 09:35, Erik Cumps wrote:
| On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <erik.cumps@esaturnus.com> wrote:
| > The context is a 16 GB 32-bit intel debian workstation, using an ext4
| > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
| > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
| > machines. The host system (so not the virual machines) shows sporadic
| > extremely slow write performance (around 4 megabytes per second).
| > However, if we use the debian 3.2.0 kernel this problem does not
| > manifest itself.
[...]
| Actually, it is the *synchronous*, direct IO that matches the expected
| raw write performance of the device.
| 
| The "regular IO" test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
| 
| The direct IO test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

I'm seeing this as well here on a number of new Dell Optiplex 7020
machines and one older Optiplex 780, all with 8GB RAM and running
Ubuntu 14.04 in 32-bit mode.

A simple dd command shows the problem:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s

In my case, running:

  echo 3 > /proc/sys/vm/drop_caches

will restore the normal speed for a limited time:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s

There is an old Ubuntu bug report describing the same issue:

  https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294

-- 
Dick

Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Dick Streefland]

On Tuesday 2015-09-01 11:15, Dick Streefland wrote:
| I'm seeing this as well here on a number of new Dell Optiplex 7020
| machines and one older Optiplex 780, all with 8GB RAM and running
| Ubuntu 14.04 in 32-bit mode.

It turned out that the dirty thresholds in /proc/vmstat became zero:

nr_dirty_threshold 0
nr_dirty_background_threshold 0

A workaround is:

# sysctl vm.highmem_is_dirtyable=1

-- 
Dick