From: Qiliang Yuan <qiliang.yuan@linux.dev>
To: Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
Nilay Shroff <nilay@linux.ibm.com>,
Hannes Reinecke <hare@suse.de>
Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org,
Qiliang Yuan <qiliang.yuan@linux.dev>
Subject: [PATCH v6 2/3] nvme-multipath: apply delayed_removal_secs changes to a pending removal
Date: Thu, 19 Nov 2026 20:25:02 -0500 [thread overview]
Message-ID: <20261119-bug-nvme-mpath-set-live-srcu-v6-2-da85b18b274b@linux.dev> (raw)
In-Reply-To: <20261119-bug-nvme-mpath-set-live-srcu-v6-0-da85b18b274b@linux.dev>
With delayed_removal_secs set, removing the last path of a namespace
head doesn't delete the head right away. nvme_mpath_remove_disk()
queues remove_work with the configured delay, and bios submitted in the
meantime wait on head->requeue_list in case a path comes back.
A write to delayed_removal_secs only takes effect for removals that
start later. A removal that is already pending keeps the old delay, so
clearing the attribute leaves the queued bios waiting until that delay
expires instead of failing them, and lowering or raising it doesn't
move the pending removal either. When paths remain but none of them is
usable, for example because they are ANA inaccessible, nothing
resubmits the queued bios after the attribute is cleared at all.
Requeue a pending remove_work with the new delay when the attribute is
written, cancelling it first so that the module reference taken for it
carries over. Have nvme_init_ns_head() cancel the pending removal
before it drops subsys->lock rather than after, so that a write to the
attribute cannot requeue the removal once a path is back. When the
attribute is cleared, kick requeue_work, so that the queued bios fail
right away if there is still no usable path.
With delayed_removal_secs set and no path left, the attribute is
changed two seconds later and the head is removed after (mean of two
runs, min-max in parentheses):
delay (s) before (ms) after (ms) change
30 -> 0 26311 (26103-26518) 8.5 (8-9) -99.97%
30 -> 5 28281 (28037-28525) 5293 (5198-5388) -81%
10 -> 30 8110 (8097-8122) 30094 (30057-30130) +271%
A direct read queued before the 30 -> 0 write fails this long after
the write, with no path left and with the only path ANA inaccessible
("> 15000" means it was still queued 15 s later):
paths before (ms) after (ms) change
none 26261 (26071-26451) 4.5 (4-5) -99.98%
inaccessible > 15000 4.5 (4-5) < -99.97%
Fixes: 62188639ec160 ("nvme-multipath: introduce delayed removal of the multipath head node")
Suggested-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
---
drivers/nvme/host/core.c | 7 +++++--
drivers/nvme/host/multipath.c | 11 +++++++++++
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index f4aa02869b790..48d432c5d7aeb 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4231,12 +4231,15 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
list_add_tail_rcu(&ns->siblings, &head->list);
ns->head = head;
- mutex_unlock(&ctrl->subsys->lock);
-
#ifdef CONFIG_NVME_MULTIPATH
+ /*
+ * Cancel a pending head removal under subsys->lock, so that
+ * delayed_removal_secs_store() never requeues it once a path is back.
+ */
if (cancel_delayed_work(&head->remove_work))
module_put(THIS_MODULE);
#endif
+ mutex_unlock(&ctrl->subsys->lock);
return 0;
out_put_ns_head:
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 9958d08cd3829..df96933f49459 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -1251,6 +1251,13 @@ static ssize_t delayed_removal_secs_store(struct device *dev,
mutex_lock(&head->subsys->lock);
head->delayed_removal_secs = sec;
assign_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags, sec);
+ /*
+ * Apply the new delay to a removal that is still pending. The module
+ * reference taken for it carries over to the requeued work.
+ */
+ if (cancel_delayed_work(&head->remove_work))
+ queue_delayed_work(nvme_wq, &head->remove_work,
+ nvme_delayed_removal_jiffies(sec));
mutex_unlock(&head->subsys->lock);
/*
* Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen
@@ -1258,6 +1265,10 @@ static ssize_t delayed_removal_secs_store(struct device *dev,
*/
synchronize_srcu(&head->srcu);
+ /* Resubmit queued bios, so that they fail without a usable path */
+ if (!sec)
+ kblockd_schedule_work(&head->requeue_work);
+
return count;
}
--
2.43.0
next prev parent reply other threads:[~2026-11-20 1:25 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 1:25 [PATCH v6 0/3] nvme-multipath: requeue work fixes Qiliang Yuan
2026-11-20 1:25 ` [PATCH v6 1/3] nvme-multipath: keep a large delayed_removal_secs from wrapping around Qiliang Yuan
2026-11-20 1:25 ` Qiliang Yuan [this message]
2026-11-20 1:25 ` [PATCH v6 3/3] nvme-multipath: don't wait for a grace period in nvme_mpath_set_live() Qiliang Yuan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119-bug-nvme-mpath-set-live-srcu-v6-2-da85b18b274b@linux.dev \
--to=qiliang.yuan@linux.dev \
--cc=axboe@kernel.dk \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=nilay@linux.ibm.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®