From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752580AbdJDNeg (ORCPT ); Wed, 4 Oct 2017 09:34:36 -0400 Received: from mga14.intel.com ([192.55.52.115]:19788 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752534AbdJDNee (ORCPT ); Wed, 4 Oct 2017 09:34:34 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,477,1500966000"; d="scan'208";a="906656016" Subject: Re: [PATCH V9 13/15] mmc: block: Add CQE and blk-mq support To: Linus Walleij Cc: Ulf Hansson , linux-mmc , linux-block , linux-kernel , Bough Chen , Alex Lemberg , Mateusz Nowak , Yuliy Izrailov , Jaehoon Chung , Dong Aisheng , Das Asutosh , Zhangfei Gao , Sahitya Tummala , Harjani Ritesh , Venu Byravarasu , Shawn Lin , Christoph Hellwig References: <1506083824-4024-1-git-send-email-adrian.hunter@intel.com> <1506083824-4024-14-git-send-email-adrian.hunter@intel.com> From: Adrian Hunter Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Message-ID: <5f3e64c4-f16e-5fa9-7ebe-4baa7502c3e1@intel.com> Date: Wed, 4 Oct 2017 16:27:44 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/10/17 10:39, Linus Walleij wrote: > On Fri, Sep 22, 2017 at 2:37 PM, Adrian Hunter wrote: > >> Add CQE support to the block driver, including: >> - optionally using DCMD for flush requests >> - "manually" issuing discard requests >> - issuing read / write requests to the CQE >> - supporting block-layer timeouts >> - handling recovery >> - supporting re-tuning >> >> CQE offers 25% - 50% better random multi-threaded I/O. There is a slight >> (e.g. 2%) drop in sequential read speed but no observable change to sequential >> write. >> >> CQE automatically sends the commands to complete requests. However it only >> supports reads / writes and so-called "direct commands" (DCMD). Furthermore >> DCMD is limited to one command at a time, but discards require 3 commands. >> That makes issuing discards through CQE very awkward, but some CQE's don't >> support DCMD anyway. So for discards, the existing non-CQE approach is >> taken, where the mmc core code issues the 3 commands one at a time i.e. >> mmc_erase(). Where DCMD is used, is for issuing flushes. >> >> For host controllers without CQE support, blk-mq support is extended to >> synchronous reads/writes or, if the host supports CAP_WAIT_WHILE_BUSY, >> asynchonous reads/writes. The advantage of asynchronous reads/writes is >> that it allows the preparation of the next request while the current >> request is in progress. >> >> Signed-off-by: Adrian Hunter > > I am trying to wrap my head around this large patch. The size makes it hard > but I am doing my best. > > Some overarching questions: > > - Is the CQE only available on the MQ path (i.e. if you enabled MQ) or > on both paths? Only MQ. > > I think it is reasonable that if we introduce a new feature like this > it will only > be available for the new block path. This reflects how the block maintainers > e.g. only allow new scheduling policies to be merged on the MQ path. > The old block layer is legacy and should not be extended with new cool > features that can then be regarded as "regressions" if they don't work > properly with MQ. Better to only implement them for MQ then. I don't agree that holding up features for years and years is an ideal outcome. > > - Performance before/after path on MQ? When the host controller supports asynchronous requests, it should be the same, but as I wrote on the cover message, the block layer workqueue had worse latency than a dedicated thread. It seems that is related to it being bound to a CPU. > > I tested this very extensively when working with my (now dormant) MQ > patch set. Better/equal/worse? > (https://marc.info/?l=linux-mmc&m=148665788227015&w=2) > > The reason my patch set contained refactorings of async post-processing, > removed the waitqueues and the context info, up to the point where I > can issue requests in parallel, i.e. complete requests from the ->done() > callback on the host and immediately let the core issue the next one, > was due to performance issues. Asynchronous requests are completed in the ->done() callback. It is not exactly accurate to say that requests can be issued in parallel. eMMC/SD cards can only do one thing at a time. Asynchronous requests facilitate the driver to prepare the next request without first having to wait for the previous request. > > It's these patches from the old patch set: > > mmc: core: move some code in mmc_start_areq() > mmc: core: refactor asynchronous request finalization > mmc: core: refactor mmc_request_done() > mmc: core: move the asynchronous post-processing > mmc: core: add a kthread for completing requests > mmc: core: replace waitqueue with worker > mmc: core: do away with is_done_rcv > mmc: core: do away with is_new_req > mmc: core: kill off the context info > mmc: queue: simplify queue logic > mmc: block: shuffle retry and error handling > mmc: queue: stop flushing the pipeline with NULL > mmc: queue: issue struct mmc_queue_req items > mmc: queue: get/put struct mmc_queue_req > mmc: queue: issue requests in massive parallel > > I.e. I made 15 patches just to make sure the new block layer did not > regress performance. The MQ-switch patch was just the final step of > these 16 patches. > > Most energy went into that and I think it will be necessary still to work > with MQ in the long haul. > > I am worried that this could add a second MQ execution path that > performs worse than the legacy block path for the above reason, i.e. > it doesn't really take advantage of the MQ speedups by marshalling > the requests and not doing away with the waitqueues and not > completing the requests out-of-order, so we get stuck with a lump of > MQ code that doesn't perform and therefore we cannot switch seamlessly > to MQ. The new patches have support for asynchronous requests and, consequently, the preparation of the next request before waiting for the previous request. There is no need for another execution path.