From: Sagi Grimberg <sagi@grimberg.me>
To: Logan Gunthorpe <logang@deltatee.com>,
linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org
Cc: Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@fb.com>,
Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning
Date: Thu, 18 Jul 2019 17:50:11 -0700 [thread overview]
Message-ID: <dd287560-2cb3-28ab-c22d-fe9f3682c609@grimberg.me> (raw)
In-Reply-To: <6deea9e7-ff3c-e115-b2f2-8914df0b6da7@deltatee.com>
>>> With multipath enabled, nvme_scan_work() can read from the
>>> device (through nvme_mpath_add_disk()). However, with fabrics,
>>> once ctrl->state is set to NVME_CTRL_DELETING, the reads will hang
>>> (see nvmf_check_ready()).
>>>
>>> After setting the state to deleting, nvme_remove_namespaces() will
>>> hang waiting for scan_work to flush and these tasks will hang.
>>>
>>> To fix this, ensure we take scan_lock before changing the ctrl-state.
>>> Also, ensure the state is checked while the lock is held
>>> in nvme_scan_lock_work().
>>
>> That's a big hammer...
>
> I didn't think the scan_lock was that contested or that
> nvme_change_ctrl_state() was really called that often...
it shouldn't be, but I think it makes the flow more convoluted
as we serialize by flushing the scan_work right after...
The design principal is met as we do get the I/O failing,
but its just that with mpath we simply queue the I/O again
because the head->list happens to not be empty.
Perhaps taking care of that check is cleaner.
>> But this is I/O that we cannot have queued until we have a path..
>>
>> I would rather have nvme_remove_namespaces() requeue all I/Os for
>> namespaces that serve as the current_path and have the make_request
>> routine to fail I/O if all controllers are deleting as well.
>>
>> Would something like [1] (untested) make sense instead?
>
> I'll have to give this a try next week and I'll let you know then. It
> kind of makes sense to me but a number of things I tried to fix this
> that I thought made sense did not work.
Thanks. Do you have a firm reproducer for it?
>>> + mutex_lock(&ctrl->scan_lock);
>>> +
>>> if (ctrl->state != NVME_CTRL_LIVE)
>>> return;
>>
>> unlock
>
> If we unlock here and relock below, we'd have to recheck the ctrl->state
> to avoid any races. If you don't want to call nvme_identify_ctrl with
> the lock held, then it would probably be better to move the state check
> below it.
Meant before the return statement.
>
>>> @@ -3547,7 +3554,6 @@ static void nvme_scan_work(struct work_struct
>>> *work)
>>> if (nvme_identify_ctrl(ctrl, &id))
>>> return;
>>
>> unlock
Same here.
next prev parent reply other threads:[~2019-07-19 0:50 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-18 22:51 [PATCH 1/2] nvme-core: Fix extra device_put() call on error path Logan Gunthorpe
2019-07-18 22:51 ` [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning Logan Gunthorpe
2019-07-19 0:25 ` Sagi Grimberg
2019-07-19 0:31 ` Sagi Grimberg
2019-07-19 20:15 ` Sagi Grimberg
2019-07-19 0:39 ` Logan Gunthorpe
2019-07-19 0:50 ` Sagi Grimberg [this message]
2019-07-24 19:12 ` Logan Gunthorpe
2019-07-25 18:16 ` Sagi Grimberg
2019-07-25 18:20 ` Logan Gunthorpe
2019-07-18 23:52 ` [PATCH 1/2] nvme-core: Fix extra device_put() call on error path Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dd287560-2cb3-28ab-c22d-fe9f3682c609@grimberg.me \
--to=sagi@grimberg.me \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=logang@deltatee.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®