From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755732AbdLVEDB (ORCPT ); Thu, 21 Dec 2017 23:03:01 -0500 Received: from aserp2130.oracle.com ([141.146.126.79]:37451 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752766AbdLVEC7 (ORCPT ); Thu, 21 Dec 2017 23:02:59 -0500 Subject: Re: [PATCH 5/7] blk-mq: remove REQ_ATOM_COMPLETE usages from blk-mq To: Tejun Heo Cc: jbacik@fb.com, jack@suse.cz, axboe@kernel.dk, clm@fb.com, kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, peterz@infradead.org, Bart.VanAssche@wdc.com References: <20171216120726.517153-1-tj@kernel.org> <20171216120726.517153-6-tj@kernel.org> <64dfa760-d433-1537-9bc6-b12cda3c9dc6@oracle.com> <20171221135051.GE1084507@devbig577.frc2.facebook.com> From: "jianchao.wang" Message-ID: <4e5aa629-f341-d077-961b-c778ceb31154@oracle.com> Date: Fri, 22 Dec 2017 12:02:20 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <20171221135051.GE1084507@devbig577.frc2.facebook.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8752 signatures=668651 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1712220056 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Sorry for my non-detailed description. On 12/21/2017 09:50 PM, Tejun Heo wrote: > Hello, > > On Thu, Dec 21, 2017 at 11:56:49AM +0800, jianchao.wang wrote: >> It's worrying that even though the blk_mark_rq_complete() here is intended to synchronize with >> timeout path, but it indeed give the blk_mq_complete_request() the capability to exclude with There could be scenario where the driver itself stop a request itself with blk_mq_complete_request() or some other interface that will invoke it, races with the normal completion path where a same request comes. For example: a reset could be triggered through sysfs on nvme-rdma Then the driver will cancel all the reqs, including in-flight ones. nvme_rdma_reset_ctrl_work() nvme_rdma_shutdown_ctrl() >>>> if (ctrl->ctrl.queue_count > 1) { nvme_stop_queues(&ctrl->ctrl); //quiesce the queue blk_mq_tagset_busy_iter(&ctrl->tag_set, nvme_cancel_request, &ctrl->ctrl); //invoke blk_mq_complete_request() nvme_rdma_destroy_io_queues(ctrl, shutdown); } >>>> These operations could race with the normal completion path of in-flight ones. It should drain all the in-flight ones first here. But there maybe some other places similar with this. >> itself. Maybe this capability should be reserved. > > Can you explain further how that'd help? The problem there is that if > you have two competing completions, where one isn't a timeout, there's > nothing synchronizing the reuse of the request. IOW, the losing on > can easily complete the next recycle instance. The atomic bitops > might feel like more protection but it's just feels. In above case, the request may simultaneously enter requeue and end path. Thanks Jianchao