Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

[Dick Streefland <dick.streefland@altium.nl>]

On Thursday 2015-06-18 09:35, Erik Cumps wrote:
| On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <erik.cumps@esaturnus.com> wrote:
| > The context is a 16 GB 32-bit intel debian workstation, using an ext4
| > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
| > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
| > machines. The host system (so not the virual machines) shows sporadic
| > extremely slow write performance (around 4 megabytes per second).
| > However, if we use the debian 3.2.0 kernel this problem does not
| > manifest itself.
[...]
| Actually, it is the *synchronous*, direct IO that matches the expected
| raw write performance of the device.
| 
| The "regular IO" test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
| 
| The direct IO test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

I'm seeing this as well here on a number of new Dell Optiplex 7020
machines and one older Optiplex 780, all with 8GB RAM and running
Ubuntu 14.04 in 32-bit mode.

A simple dd command shows the problem:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s

In my case, running:

  echo 3 > /proc/sys/vm/drop_caches

will restore the normal speed for a limited time:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s

There is an old Ubuntu bug report describing the same issue:

  https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294

-- 
Dick