From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S971005AbdEZP23 (ORCPT ); Fri, 26 May 2017 11:28:29 -0400 Received: from mx2.mpynet.fi ([82.197.21.85]:33559 "EHLO mx2.mpynet.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S943714AbdEZP2M (ORCPT ); Fri, 26 May 2017 11:28:12 -0400 Date: Fri, 26 May 2017 18:28:09 +0300 From: Rakesh Pandit To: Keith Busch CC: , , "Jens Axboe" , Christoph Hellwig , Sagi Grimberg , Andy Lutomirski Subject: Re: [PATCH V2 1/1] nvme: fix multiple ctrl removal scheduling Message-ID: <20170526152809.GA18963@dhcp-216.srv.tuxera.com> References: <20170524142623.GA27721@dhcp-216.srv.tuxera.com> <20170526100613.GE24894@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170526100613.GE24894@localhost.localdomain> User-Agent: Mutt/1.7.1 (2016-10-04) X-ClientProxiedBy: tuxera-exch.ad.tuxera.com (10.20.48.11) To tuxera-exch.ad.tuxera.com (10.20.48.11) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Added Andy Lutomirski to CC (APST related issue) On Fri, May 26, 2017 at 06:06:14AM -0400, Keith Busch wrote: > On Wed, May 24, 2017 at 05:26:25PM +0300, Rakesh Pandit wrote: > > Commit c5f6ce97c1210 tries to address multiple resets but fails as > > work_busy doesn't involve any synchronization and can fail. This is > > reproducible easily as can be seen by WARNING below which is triggered > > with line: > > > > WARN_ON(dev->ctrl.state == NVME_CTRL_RESETTING) > > > > Allowing multiple resets can result in multiple controller removal as > > well if different conditions inside nvme_reset_work fail and which > > might deadlock on device_release_driver. > > > > This patch makes sure that work queue item (reset_work) is added only > > if controller state != NVME_CTRL_RESETTING and that is achieved by > > moving state change outside nvme_reset_work into nvme_reset and > > removing old work_busy call. State change is always synchronizated > > using controller spinlock. > > So, the reason the state is changed when the work is running rather than > queueing is for the window when the state may be set to NVME_CTRL_DELETING, > and we don't want the reset work to proceed in that case. > > What do you think about adding a new state, like NVME_CTRL_SCHED_RESET, > then leaving the NVME_CTRL_RESETTING state change as-is? Thanks. I will give it a go as soon as I have hardware available (have limited access and yesterday was a holiday here) and address issues pointed by Christoph earlier. Also there is a related (because I can reproduce it easily on same device with nvme_remove) but separate issue with APST implementation. PID (undergoing nvme_uninit_ctrl) waits for ever at blk_execute_rq. Controller is in DEAD state and nvme_remove_namespaces just before device_destroy call has killed all queues which seems to eventually make blk_execute_rq sleep for ever as it tries to sync updated latency (0 most likely). [] blk_execute_rq+0x56/0x80 [] __nvme_submit_sync_cmd+0x89/0xf0 [] nvme_set_features+0x5e/0x90 [] nvme_configure_apst+0x166/0x200 [] nvme_set_latency_tolerance+0x35/0x50 [] apply_constraint+0xb1/0xc0 [] dev_pm_qos_constraints_destroy+0xf4/0x1f0 [] dpm_sysfs_remove+0x2a/0x60 [] device_del+0x101/0x320 [] device_unregister+0x1a/0x60 [] device_destroy+0x3c/0x50 [] nvme_uninit_ctrl+0x45/0xa0 [] nvme_remove+0x78/0x110 [] pci_device_remove+0x39/0xb0 [] device_release_driver_internal+0x155/0x210 [] device_release_driver+0x12/0x20 [] nvme_remove_dead_ctrl_work+0x6b/0x70 [] process_one_work+0x18c/0x3a0 [] worker_thread+0x4e/0x3b0 [] kthread+0x109/0x140 [] ret_from_fork+0x2c/0x40 [] 0xffffffffffffffff