From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755569AbbIAJlH (ORCPT ); Tue, 1 Sep 2015 05:41:07 -0400 Received: from 53505047.static.ziggozakelijk.nl ([83.80.80.71]:35374 "EHLO ns5.tasking.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755235AbbIAJlE (ORCPT ); Tue, 1 Sep 2015 05:41:04 -0400 X-Greylist: delayed 1526 seconds by postgrey-1.27 at vger.kernel.org; Tue, 01 Sep 2015 05:41:04 EDT Date: Tue, 1 Sep 2015 11:15:35 +0200 From: Dick Streefland To: Erik Cumps Cc: linux-kernel@vger.kernel.org Subject: Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels Message-ID: <20150901091535.GA21329@altium.nl> References: <1434462995.6161.800.camel@erik-desktop.office> <1434462995.6161.800.camel@erik-desktop.office> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Altium BV, Amersfoort, The Netherlands User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thursday 2015-06-18 09:35, Erik Cumps wrote: | On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps wrote: | > The context is a 16 GB 32-bit intel debian workstation, using an ext4 | > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively | > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual | > machines. The host system (so not the virual machines) shows sporadic | > extremely slow write performance (around 4 megabytes per second). | > However, if we use the debian 3.2.0 kernel this problem does not | > manifest itself. [...] | Actually, it is the *synchronous*, direct IO that matches the expected | raw write performance of the device. | | The "regular IO" test is doing roughly this: | | echo 3 > /proc/sys/vm/drop_caches | dd if=ramdisk_file of=test_file bs=1M count=100 | dd if=ramdisk_file of=test_file bs=1M count=100 | dd if=ramdisk_file of=test_file bs=1M count=100 | | The direct IO test is doing roughly this: | | echo 3 > /proc/sys/vm/drop_caches | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 | dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100 I'm seeing this as well here on a number of new Dell Optiplex 7020 machines and one older Optiplex 780, all with 8GB RAM and running Ubuntu 14.04 in 32-bit mode. A simple dd command shows the problem: $ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s $ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s In my case, running: echo 3 > /proc/sys/vm/drop_caches will restore the normal speed for a limited time: $ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s There is an old Ubuntu bug report describing the same issue: https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294 -- Dick