From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754809AbbKCOzz (ORCPT ); Tue, 3 Nov 2015 09:55:55 -0500 Received: from mx2.suse.de ([195.135.220.15]:40725 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751504AbbKCOzy (ORCPT ); Tue, 3 Nov 2015 09:55:54 -0500 Date: Tue, 3 Nov 2015 15:55:50 +0100 From: Jan Kara To: Jeff Moyer Cc: Jan Kara , Jens Axboe , LKML Subject: Re: Flush requests not going through IO scheduler Message-ID: <20151103145550.GD4063@quack.suse.cz> References: <20151102122009.GE13433@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon 02-11-15 12:52:26, Jeff Moyer wrote: > Jan Kara writes: > > > Hello, > > > > when looking into a performance issue, I've noticed one interesting thing > > in blktrace data: > > Hi Jan, > > Do you have a reproducer I can run? So I don't have a reproducer that would trigger a situation where we are idling while there is request queued. However the following is easy to trigger: 8,0 67 43 0.000064884 1958 D WS 2499459168 + 104 [jbd2/sda5-8] 8,0 67 44 0.000222208 0 C WS 2499459168 + 104 [0] 8,0 67 0 0.000232553 0 m N cfq1958SN / complete rqnoidle 1 8,0 67 0 0.000234002 0 m N cfq1958SN / set_slice=30 8,0 67 0 0.000234984 0 m N cfq1958SN / Not idling. st->count:1 8,0 67 0 0.000236353 0 m N cfq1958SN / arm_idle: 2 group_idle: 1 8,0 67 0 0.000236832 0 m N cfq schedule dispatch 8,0 67 45 0.000259492 1958 A FWFS 2499459272 + 8 <- (8,5) 621525192 8,0 67 46 0.000259994 1958 Q FWFS 2499459272 + 8 [jbd2/sda5-8] 8,0 67 47 0.000262466 1958 G FWFS 2499459272 + 8 [jbd2/sda5-8] 8,0 67 48 0.000263164 1958 I FWFS 2499459272 + 8 [jbd2/sda5-8] 8,0 67 49 0.000389821 0 D WS 2499459272 + 8 [swapper/67] 8,0 67 50 0.000439548 0 C WS 2499459272 + 8 [0] 8,0 67 51 0.000556711 0 C WS 2499459272 [0] 8,0 67 0 0.004387444 0 m N cfq idle timer fired Here we dispatch the flush request quickly but only because we queued kick of the queue in IO completion of the previous request. If jbd2 were 120 us slower in submitting the flush request, it would miss the kick and the request would end up waiting until the idle timer fires. To trigger this you just need a rotational storage with write back cache, slice_idle & group_idle enabled, mount ext4 on top of it. Any trasaction commit will end up looking like this. Honza -- Jan Kara SUSE Labs, CR