Re: [PATCH v2 00/20] md/md-llbitmap: support reshape for RAID10 and RAID5

["yu kuai" <yukuai@fygo.io>]

Hi,

在 2026/7/27 2:59, Mykola Marzhan 写道:
> Hi Kuai,
>
> We ran v2 through a QEMU test campaign while preparing our own llbitmap
> patches, which we will rebase on top of this series: raid1/raid5/raid10
> over loop devices, base 55b77337bdd0 ("md/raid5: avoid R5_Overlap races
> while breaking stripe batches") with and without the series applied,
> KASAN kernels for the reshape stress, and per-bit bitmap dumps as the
> oracle. Four problems, worst first — details under each number below:
>
> 1. raid5: writes above dev_sectors stop being tracked (steady state,
>     no reshape involved)                                 [patches 02,12]
> 2. a stopped and re-assembled reshape resumes with the pre-reshape
>     bitmap geometry, and persists it                     [patch 08]
> 3. discard can mark a chunk unwritten while half of it still holds
>     live data (raid1)                                    [patch 10]
> 4. pre-existing: raid5_bitmap_sector() rounds with round_down() on a
>     non-power-of-2 size — bits land in the wrong stripe; fix offered
>                                                          [base, patch 12]
>
> Happy to rerun the whole campaign on v3 and, if it comes back clean,
> follow up with Tested-by. We'll post the test scripts in-thread if
> useful.

Thanks for the report. I'll look into these problems ASAP. BTW, for-7.3/block
branch should be created soon, I'll start applying patches then.

>
> 1. raid5: writes above dev_sectors stop being tracked
>
> llbitmap_map_layout() (patch 12) compares an array sector against
> llbitmap_logical_size(). Outside a reshape llbitmap_reshaping() is
> false, so that falls through llbitmap_personality_sync_size() to
> llbitmap->sync_size — which raid5_run() set from
> mddev->resync_max_sectors = mddev->dev_sectors, a per-device count.
> An array sector compared against a device-sector limit: every write
> above array sector dev_sectors gets *sectors = 0, and with patch 02's
> early return the bitmap is never touched.
>
> On a 4-disk raid5 tracking stops after the first third of the array,
> on a 7-disk after the first sixth; a crash would then leave everything
> above that boundary out of the post-crash resync. raid1/raid10 are
> unaffected (their resync_max_sectors is the array size), and *during*
> a reshape the correct bitmap_array_sectors is consulted — the problem
> only shows in steady state.
>
> Measured (one 4k write below the limit, one above, count new dirty
> bits): patched 4-disk +1/+0, patched 7-disk +1/+0; base +1/+1 both.
>
> Is llbitmap_logical_size() meant to return array sectors
> unconditionally (i.e. consult patch 17's bitmap_array_sectors outside
> reshape too), or should llbitmap_map_layout() convert before
> comparing? The two helpers currently mix coordinate spaces for
> raid456.
>
> 2. resumed reshape reverts the bitmap geometry
>
> Two things combine:
>
> - md_run() calls md_bitmap_create() before assigning mddev->pers, so
>    llbitmap_refresh_reshape() (patch 08) runs with mddev->pers == NULL
>    and llbitmap_personality_sync_size() bails out to the pre-reshape
>    sync_size from the superblock;
> - on a resumed reshape md never calls pers->start_reshape() — with
>    reshape_position != MaxSector only check_reshape() runs — so
>    bitmap_ops->resize() is never re-invoked either.
>
> llbitmap_reshape_finish() then commits the stale geometry and
> llbitmap_update_sb() persists it, so the revert is permanent: pages
> beyond the old used_pages are returned zeroed by llbitmap_read_page()
> instead of read from disk.
>
> Reproduces in ~90s on raid10 4->6: grow, stop mid-reshape,
> re-assemble — llbitmap/metadata chunks read 8128 -> 12192 -> 8128.
> mdadm --grow --continue does not help. raid5 disk-add is immune
> (device-indexed bitmap, the resize is a no-op), which makes it a
> clean control.
>
> Possible directions: re-run llbitmap_refresh_reshape() once
> mddev->pers is set, or have the resume path call
> bitmap_ops->resize(). Moving md_bitmap_create() after the pers
> assignment would also do it, but we did not audit what else depends
> on the current order.
>
> 3. sub-chunk discard marks a live chunk unwritten
>
> llbitmap_start_discard() computes its first bit with
> DIV_ROUND_UP_SECTOR_T(offset, chunksize) precisely so a partially
> discarded head chunk is not touched. Patch 10 makes
> llbitmap_prepare_range() run llbitmap_encode_range() on every op,
> which floors the offset to a chunk boundary first — the DIV_ROUND_UP
> has nothing left to round. raid1/raid10 previously had no
> bitmap_sector hook, so raw offsets reached start_discard and the
> guard worked.
>
> Test, on raid1: write a pattern, then discard [K+0.5 chunks,
> K+1.5 chunks) — a range that fully contains no chunk, so ideally zero
> chunks change state. Base flips 1 (the tail chunk — pre-existing,
> arguably the same class of problem on the other end), patched flips
> 2. The extra one is chunk K, whose first half still holds the
> pattern.
>
> Maybe encode_range should be skipped for discard, or start_discard
> should take the raw offset.
>
> 4. pre-existing: raid5_bitmap_sector() rounding is wrong for
>     non-power-of-2 data-disk counts
>
> 	sectors_per_chunk = conf->chunk_sectors *
> 		(conf->raid_disks - conf->max_degraded);
> 	start = round_down(start, sectors_per_chunk);
> 	end = round_up(end, sectors_per_chunk);
>
> round_down()/round_up() are mask-based and require power-of-2
> alignment; chunk_sectors is a power of 2, but multiplied by 3, 5, 6
> or 7 data disks the product is not. The rounded sector is then not
> stripe-aligned and raid5_compute_sector() maps it into a different
> stripe: the bits marked and the bits covering the write are disjoint.
> Lost bits, not wide bits.
>
> Per-bit dumps, no reshape running: 3 data disks lose 20/20 targeted
> writes, 6 data disks 17/17, the 4-data (power-of-2) control 0/20.
> Byte-identical on the unpatched base, so not this series' bug — but
> patch 12 routes this mapping into the reshape remap path, so it
> matters more afterwards.
>
> The fix needs division-based rounding (sector_div(), these are
> sector_t). We can send a standalone patch, or you can fold it into
> v3 — whichever is easier.
>
> One gap in our coverage: raid10 grow under sustained IO currently
> floods KASAN with pre-existing slab-out-of-bounds reports from the
> r10bio width mismatch (free_r10bio() / raid10_make_request()) —
> reproduced on the unpatched base too; none of the sites is touched by
> your series. That is what Chen Cheng's "md/raid10: fix r10bio width
> mismatches across reshape" fixes:
>
>    https://lore.kernel.org/linux-raid/20260711100352.425177-1-chencheng@fnnas.com/
>
> Our reproduction data is in that thread; we will re-run the
> raid10-grow KASAN leg here once it lands.
>
> Thanks,
> Mykola

-- 
Thanks,
Kuai