From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756208AbeASTrw (ORCPT ); Fri, 19 Jan 2018 14:47:52 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:59152 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755932AbeASTrp (ORCPT ); Fri, 19 Jan 2018 14:47:45 -0500 X-IronPort-AV: E=Sophos;i="5.46,382,1511798400"; d="scan'208";a="69989640" From: Bart Van Assche To: "ming.lei@redhat.com" CC: "dm-devel@redhat.com" , "hch@infradead.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "osandov@fb.com" , "axboe@kernel.dk" , "snitzer@redhat.com" Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Topic: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Index: AQHTkAXlp7DN5JBZKUeyLfttG5XpOaN52OmAgAADroCAAATDAIAAE3uAgAAE1gCAABc0gIAAaoKAgAAsBICAAChpgIAAzOAA Date: Fri, 19 Jan 2018 19:47:42 +0000 Message-ID: <1516391260.2968.27.camel@wdc.com> References: <20180118024124.8079-1-ming.lei@redhat.com> <20180118170353.GB19734@redhat.com> <1516296056.2676.23.camel@wdc.com> <20180118183039.GA20121@redhat.com> <1516301278.2676.35.camel@wdc.com> <20180119023212.GA25413@ming.t460p> <1516338585.4077.1.camel@wdc.com> <20180119073423.GC25369@ming.t460p> In-Reply-To: <20180119073423.GC25369@ming.t460p> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bart.VanAssche@wdc.com; x-originating-ip: [199.255.44.172] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY1PR0401MB1691;7:hH4EAmbCKaGQOFAoj+j0pwWkXdSXS+6Xvh9Btpz6qikCpPanPggljS8zAa10x7xK4ZUq7vzkxiusZNNeP0xWQCPYnfOImZjIMh/D+et614rQC3Spv9w9SKmehEFnP6bWvorRmRQLZ8vusYfhvztnhF197jVhNSm6inDD+GxhCODkaISFDGV29S8zzUIXiPX0mGP7gBLRBi0aKWSXk2J2pmyeJDLwVBXDbx5V4xyqWYsOIQuxl+S+trTMzOOWwHvN;20:i10mmAnUP/2pG8prAHncglBmDxehTsR2TNEp45hDUHhtu4IZzociXzPdzlizWjaoibXB7Prlgr32RV/lRraCfh5+LW0g5cLT+L1513G2mPlolAZoimfPFD/rinIct7S4ZLscMg8b1ob/+T93w16/XRmLtC3/CW92caCrUghw2C0= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: cdfb4191-92e8-4513-75d8-08d55f758189 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);SRVR:CY1PR0401MB1691; x-ms-traffictypediagnostic: CY1PR0401MB1691: wdcipoutbound: EOP-TRUE x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040498)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231023)(2400081)(944501161)(3002001)(6055026)(6041285)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123564045)(20161123562045)(20161123560045)(6072148)(201708071742011);SRVR:CY1PR0401MB1691;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:CY1PR0401MB1691; x-forefront-prvs: 0557CBAD84 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(346002)(39380400002)(376002)(39860400002)(396003)(366004)(51444003)(377424004)(199004)(189003)(53936002)(6246003)(102836004)(6506007)(6436002)(14454004)(93886005)(6486002)(6512007)(54906003)(316002)(5640700003)(36756003)(25786009)(103116003)(478600001)(4326008)(72206003)(229853002)(106356001)(2501003)(2906002)(97736004)(8936002)(66066001)(6116002)(3846002)(81156014)(81166006)(68736007)(2900100001)(305945005)(3280700002)(105586002)(3660700001)(59450400001)(76176011)(86362001)(6916009)(2351001)(2950100002)(5660300001)(7736002)(77096007)(99286004)(8676002)(26005);DIR:OUT;SFP:1102;SCL:1;SRVR:CY1PR0401MB1691;H:CY1PR0401MB1536.namprd04.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; x-microsoft-antispam-message-info: Cn0mJujgVLYrzekNqRqQUnS3oOEyVRHuX2YzRw9hFcdoWlWG67IVZsk8r0pPVyqE7pzv4zmh89Y0xDHeBSbCaw== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <45B7B73428ED1F44BF70190FC576D038@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-Network-Message-Id: cdfb4191-92e8-4513-75d8-08d55f758189 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jan 2018 19:47:43.0468 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1691 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w0JJlulR004628 On Fri, 2018-01-19 at 15:34 +0800, Ming Lei wrote: > Could you explain a bit when SCSI target replies with BUSY very often? > > Inside initiator, we have limited the max per-LUN requests and per-host > requests already before calling .queue_rq(). That's correct. However, when a SCSI initiator and target system are communicating with each other there is no guarantee that initiator and target queue depth have been tuned properly. If the initiator queue depth and the number of requests that can be in flight according to the network protocol are both larger than the target queue depth and if the target system uses relatively slow storage (e.g. harddisks) then it can happen that the target replies with BUSY often. The Linux iSCSI initiator limits MaxOutstandingR2T (the number of requests an initiator may sent without having received an answer from the target) to one so I don't think this can happen when using iSCSI/TCP. With the SRP initiator however the maximum requests that can be in flight between initiator and target depends on the number of credits that were negotiated during login between initiator and target. Some time ago I modified the SRP initiator such that it limits the initiator queue depth to the number of SRP credits minus one (for task management). That resulted in a performance improvement due to fewer BUSY conditions at the initiator side (see also commit 7ade400aba9a ("IB/srp: Reduce number of BUSY conditions")). Another patch for the SCST SRP target driver limited the number of SRP credits to the queue depth of the block device at the target side. I'm referring to the following code: ch->rq_size = min(MAX_SRPT_RQ_SIZE, scst_get_max_lun_commands(NULL, 0)) (I have not yet had the time to port this change to LIO). Without such tuning across initiator and target it can happen often that the target system sends the reply "BUSY" back to the initiator. I think that's why there is code in the SCSI core to automatically adjust the initiator queue depth if the "BUSY" condition is encountered frequently. See also scsi_track_queue_full(). Bart.