mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/3] dm era: a metadata device that never comes back
@ 2026-11-18 11:02 Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 1/3] dm persistent data: drop the original block only once it is copied Denis V. Lunev
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Denis V. Lunev @ 2026-11-18 11:02 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, dm-devel
  Cc: bmarzins, thornber, linux-kernel, stable, den

We move running containers between machines without restarting them,
and their block volumes travel with them, so dm-era tells us which
blocks changed between copy passes.

About one move in three ended with the volume dead rather than moved:
the metadata broke, every commit after it failed, the target reported
"era Error", and the writes it tracked began to fail, which takes the
workload's filesystem read-only. Rebuilding the device recovers the
tracking, not the filesystem.

Three unrelated causes, all of which the kernel log blames on the
space map, which is the victim. Two need no bad luck: multipathd asks
every target that appears for its status, so one is asked exactly
while it resumes, and again while it suspends. The third is older and
wider - a transient read failure in the persistent-data library leaves
the space map short, and the next commit makes that permanent. dm-thin
and dm-cache reach it too, where it costs a transaction rather than
the device, because they can abort and dm-era cannot.

Each patch is reproduced on a kernel whose only modification is the
fix under test. Patch 3 must not be backported before patch 2, nor
before cca2c6aebe86, where the resize goes through the worker the two
would refuse.

Denis V. Lunev (3):
  dm persistent data: drop the original block only once it is copied
  dm era: refuse metadata rpcs once the target is suspended
  dm era: construct the target suspended

 drivers/md/dm-era-target.c                          | 10 +++++++++-
 drivers/md/persistent-data/dm-transaction-manager.c | 13 ++++++++-----
 2 files changed, 17 insertions(+), 6 deletions(-)

-- 
2.55.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 1/3] dm persistent data: drop the original block only once it is copied
  2026-11-18 11:02 [PATCH 0/3] dm era: a metadata device that never comes back Denis V. Lunev
@ 2026-11-18 11:02 ` Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 2/3] dm era: refuse metadata rpcs once the target is suspended Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 3/3] dm era: construct the target suspended Denis V. Lunev
  2 siblings, 0 replies; 4+ messages in thread
From: Denis V. Lunev @ 2026-11-18 11:02 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, dm-devel
  Cc: bmarzins, thornber, linux-kernel, stable, den

From: Denis V. Lunev <den@openvz.org>

__shadow_block() releases the original block's reference before it
reads the block, so every error return in between leaves the caller
pointing at a block the space map has given away.

metadata_ll_commit() shows what that costs. It stores the new block in
ll->bitmap_root only when the shadow succeeds, so a failed read leaves
the space map's index root at a block whose count has already reached
zero. Every commit after that shadows the same block, fails to
decrement it, and fails:

  device-mapper: space map common: unable to decrement block
  device-mapper: era: metadata_commit: pre commit failed

The metadata never recovers. dm-era reports "era Error" and fails the
writes it was tracking, which takes the workload's filesystem
read-only.

A btree caller ends up in a different state. dm_btree_insert()
publishes the new root only on success, so the caller keeps a root
whose count has dropped to zero, and the next dm_tm_shadow_block() of
it finds dm_sm_count_is_more_than_one() false and writes the block in
place instead of copying it.

Decrement the original once the copy holds its contents. A read of a
metadata block can fail for a transient reason, and a single failure is
enough to get here.

Fixes: 3241b1d3e0aa ("dm: add persistent data library")
Cc: stable@vger.kernel.org
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 drivers/md/persistent-data/dm-transaction-manager.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/md/persistent-data/dm-transaction-manager.c b/drivers/md/persistent-data/dm-transaction-manager.c
index 7401add01d12..159aac9da07a 100644
--- a/drivers/md/persistent-data/dm-transaction-manager.c
+++ b/drivers/md/persistent-data/dm-transaction-manager.c
@@ -297,10 +297,6 @@ static int __shadow_block(struct dm_transaction_manager *tm, dm_block_t orig,
 	if (r < 0)
 		return r;
 
-	r = dm_sm_dec_block(tm->sm, orig);
-	if (r < 0)
-		return r;
-
 	r = dm_bm_read_lock(tm->bm, orig, v, &orig_block);
 	if (r < 0)
 		return r;
@@ -322,7 +318,14 @@ static int __shadow_block(struct dm_transaction_manager *tm, dm_block_t orig,
 	       dm_bm_block_size(tm->bm));
 
 	dm_bm_unlock(orig_block);
-	return r;
+
+	r = dm_sm_dec_block(tm->sm, orig);
+	if (r < 0) {
+		dm_bm_unlock(*result);
+		return r;
+	}
+
+	return 0;
 }
 
 int dm_tm_shadow_block(struct dm_transaction_manager *tm, dm_block_t orig,
-- 
2.55.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 2/3] dm era: refuse metadata rpcs once the target is suspended
  2026-11-18 11:02 [PATCH 0/3] dm era: a metadata device that never comes back Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 1/3] dm persistent data: drop the original block only once it is copied Denis V. Lunev
@ 2026-11-18 11:02 ` Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 3/3] dm era: construct the target suspended Denis V. Lunev
  2 siblings, 0 replies; 4+ messages in thread
From: Denis V. Lunev @ 2026-11-18 11:02 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, dm-devel
  Cc: bmarzins, thornber, linux-kernel, stable, den

From: Denis V. Lunev <den@openvz.org>

An rpc reaches the worker in two steps: the caller puts it on the rpc
list, then wake_worker() queues the work unless the target is
suspended. stop_worker() raises that flag and drains the workqueue with
no ordering against either step, so an rpc can land on both sides of
the drain.

An rpc queued before the flag is raised, whose wake_worker() runs after
it, is never served. The caller waits for a completion that only the
next resume can deliver, and it waits holding the table srcu, so a
reload blocks in dm_swap_table() and the device can no longer be
reloaded or removed.

An rpc whose wake_worker() read the flag as clear can instead queue the
work after drain_workqueue() has returned. The worker then commits the
metadata beside the commit era_postsuspend() runs in its own thread.
Both commits allocate from the same space map and take the same block,
which corrupts it, and every commit after that fails:

  device-mapper: space map common: unable to decrement block
  device-mapper: era: metadata_commit: pre commit failed

Add the rpc and queue the work under the rpc lock, take that lock to
raise the flag, and refuse an rpc that arrives once it is raised. An
rpc that made it onto the list is then queued before the drain waits
for it.

Userspace sees one change. A status query on a suspended era device
used to block until the device was resumed, and now reports the target
in error.

Fixes: eec40579d848 ("dm: add era target")
Cc: stable@vger.kernel.org
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 drivers/md/dm-era-target.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c
index ea499adca4ce..29b0602e7849 100644
--- a/drivers/md/dm-era-target.c
+++ b/drivers/md/dm-era-target.c
@@ -1389,10 +1389,14 @@ static int perform_rpc(struct era *era, struct rpc *rpc)
 	init_completion(&rpc->complete);
 
 	spin_lock(&era->rpc_lock);
+	if (atomic_read(&era->suspended)) {
+		spin_unlock(&era->rpc_lock);
+		return -EBUSY;
+	}
 	list_add(&rpc->list, &era->rpc_calls);
+	queue_work(era->wq, &era->worker);
 	spin_unlock(&era->rpc_lock);
 
-	wake_worker(era);
 	wait_for_completion(&rpc->complete);
 
 	return rpc->result;
@@ -1427,7 +1431,10 @@ static void start_worker(struct era *era)
 
 static void stop_worker(struct era *era)
 {
+	spin_lock(&era->rpc_lock);
 	atomic_set(&era->suspended, 1);
+	spin_unlock(&era->rpc_lock);
+
 	drain_workqueue(era->wq);
 }
 
-- 
2.55.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 3/3] dm era: construct the target suspended
  2026-11-18 11:02 [PATCH 0/3] dm era: a metadata device that never comes back Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 1/3] dm persistent data: drop the original block only once it is copied Denis V. Lunev
  2026-11-18 11:02 ` [PATCH 2/3] dm era: refuse metadata rpcs once the target is suspended Denis V. Lunev
@ 2026-11-18 11:02 ` Denis V. Lunev
  2 siblings, 0 replies; 4+ messages in thread
From: Denis V. Lunev @ 2026-11-18 11:02 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, dm-devel
  Cc: bmarzins, thornber, linux-kernel, stable, den

From: Denis V. Lunev <den@openvz.org>

era_preresume() resizes the era array and commits the metadata in the
thread that resumes the device, and starts the worker only once that is
done. That is safe only while the worker cannot run, and nothing makes
it so: struct era is zeroed, so era->suspended reads as running from
the moment the target is constructed, and an rpc arriving before the
resume wakes the worker alongside the resize. Both allocate from the
same space map and take the same metadata block, which corrupts it:

 device-mapper: block manager: validator mismatch (old=array vs new=index)
 device-mapper: space map common: unable to decrement block
 device-mapper: era: metadata_commit: pre commit failed

Every commit after that fails the same way, the target reports "era
Error", and the writes it tracks fail, which takes the workload's
filesystem read-only.

Userspace gets there without trying. multipathd asks every
device-mapper target appearing on the node for its status, over its own
uevent socket, to learn whether the target is a multipath map, so a
target is asked exactly while it resumes.

Mark the target suspended when it is constructed, which is the state
start_worker() already assumes it is leaving. The resize and the commit
then run with the worker held back, as they must, and a status query in
that window is refused rather than served beside them.

Fixes: cca2c6aebe86 ("dm era: only resize metadata in preresume")
Cc: stable@vger.kernel.org # v5.12+
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 drivers/md/dm-era-target.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c
index 29b0602e7849..d40a98169b04 100644
--- a/drivers/md/dm-era-target.c
+++ b/drivers/md/dm-era-target.c
@@ -1557,6 +1557,7 @@ static int era_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 
 	spin_lock_init(&era->rpc_lock);
 	INIT_LIST_HEAD(&era->rpc_calls);
+	atomic_set(&era->suspended, 1);
 
 	ti->private = era;
 	ti->num_flush_bios = 1;
-- 
2.55.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-18 11:02 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-18 11:02 [PATCH 0/3] dm era: a metadata device that never comes back Denis V. Lunev
2026-11-18 11:02 ` [PATCH 1/3] dm persistent data: drop the original block only once it is copied Denis V. Lunev
2026-11-18 11:02 ` [PATCH 2/3] dm era: refuse metadata rpcs once the target is suspended Denis V. Lunev
2026-11-18 11:02 ` [PATCH 3/3] dm era: construct the target suspended Denis V. Lunev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®