From: Qu Wenruo <quwenruo.btrfs@gmx.com>
To: Miroslav Cimerman <mc@raidlab.org>, linux-btrfs@vger.kernel.org
Cc: David Sterba <dsterba@suse.com>, Chris Mason <mason@kernel.org>,
Qu Wenruo <wqu@suse.com>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] btrfs: raid56: fix use-after-free when waking up the rbio I/O waiter
Date: Mon, 16 Nov 2026 11:32:48 +1030 [thread overview]
Message-ID: <29d117e7-8ccd-4d55-a8d5-1ef78d851700@gmx.com> (raw)
In-Reply-To: <20261116001903.63524-1-mc@raidlab.org>
在 2026/11/16 10:49, Miroslav Cimerman 写道:
> The end_io functions of the RAID56 read and write bios decrement
> rbio->stripes_pending and call wake_up(&rbio->io_wait) when it drops
> to zero. The submitter waits with wait_event(rbio->io_wait,
> atomic_read(&rbio->stripes_pending) == 0) and then can drop the last
> reference to the rbio, which frees it.
>
> If the submitter checks the condition after the last decrement and
> before the wake_up(), it does not sleep at all:
>
> CPU0 (endio, irq) CPU1 (submitter, rmw_rbio)
> raid_wait_write_end_io() rmw_rbio()
> | |-submit_write_bios()
> | | \-submit_bio()
> | |
> | |-wait_event()
> |-bio_put() | ^^^ checks the condition without
> | | taking io_wait.lock
> |-atomic_dec_and_test() |
> | ^^^ stripes_pending hits 0 |
> | | ^^^ sees 0, doesn't sleep
> | |
> | \-rbio_orig_end_io()
> | \-kfree()
> | ^^^ last rbio reference, rbio
> | freed
> |
> \-wake_up()
> \-_raw_spin_lock_irqsave()
> ^^^ io_wait.lock is in freed memory
> Too late.
>
> Even when the submitter sleeps, the wake-up takes it off the wait
> queue, so finish_wait() returns without taking the queue's lock while
> wake_up() still holds it:
>
> CPU0 (endio, irq) CPU1 (submitter, rmw_rbio)
> raid_wait_write_end_io() rmw_rbio()
> | \-wait_event()
> | |-prepare_to_wait_event()
> | \-schedule()
> |-atomic_dec_and_test()
> | ^^^ stripes_pending hits 0
> |
> \-wake_up()
> \-__wake_up_common_lock()
> |-_raw_spin_lock_irqsave()
> |-__wake_up_common()
> | ^^^ autoremove_wake_function()
> | takes the waiter off the
> | queue and wakes CPU1 finish_wait()
> | |-list_empty_careful()
> | | ^^^ entry already removed,
> | | returns without taking
> | | io_wait.lock
> | |
> | \-rbio_orig_end_io()
> | \-kfree()
> | ^^^ rbio freed
> |
> \-_raw_spin_unlock_irqrestore()
> ^^^ io_wait.lock is in freed memory
> Too late.
>
> The same happens on the read side (raid_wait_read_end_io() and
> submit_read_wait_bio_list()) and in the parity scrub. On 6.12.117
> (bio_endio() and the end_io function are tail calls, so they do not
> show in the trace):
>
> BUG: kernel NULL pointer dereference, address: 0000000000000000
> RIP: 0010:__wake_up_common+0x25/0xa0
> Call Trace:
> <IRQ>
> __wake_up+0x31/0x60
> blk_update_request+0xfd/0x410
> blk_mq_end_request+0x1d/0x1a0
> Kernel panic - not syncing: Fatal exception in interrupt
>
> With KASAN on v7.4-rc2 and a test-only delay between the decrement
> and the wake-up:
>
> BUG: KASAN: slab-use-after-free in _raw_spin_lock_irqsave+0x76/0xe0
> __wake_up+0x1c/0x60
> raid_wait_read_end_io+0x132/0x10f0
> blk_update_request+0x21e/0xea0
> Allocated by task 82:
> alloc_rbio.isra.0+0xa8/0xbf0
> raid56_parity_recover+0x10a/0x9c0
> Freed by task 84:
> kfree+0x154/0x420
> rbio_orig_end_io+0x1c4/0x340
> recover_rbio+0x393/0x590
>
> Wait with a completion instead. complete() wakes the waiter under the
> completion's lock and does not touch it afterwards, and
> wait_for_completion() returns only after taking that lock, so the
> rbio may be freed as soon as the wait returns. The parity scrub waits
> for its writes only when finish_parity_scrub() submitted them.
>
> Fixes: d817ce35d24a ("btrfs: raid56: switch recovery path to a single function")
> Fixes: 5eb30ee26fa4 ("btrfs: raid56: introduce the main entrance for RMW path")
> Fixes: 6bfd0133bee2 ("btrfs: raid56: switch scrub path to use a single function")
> Cc: stable@vger.kernel.org # 6.2+
> Reviewed-by: Qu Wenruo <wqu@suse.com>
> Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
Pushed to for-next branch.
Thanks,
Qu
> ---
> Changes in v2:
> - ASCII race charts (Qu)
> - drop the FuzzyRAID sentence and the completion.h include (Qu)
>
> Found with FuzzyRAID, a fault-injection tester for RAID that is not
> public yet. The analysis, the fix, the tests and the commit message
> were prepared with the help of an LLM. I reviewed everything.
>
> The crash: 4 of 10,711 fault-injection scenarios on a 3-device data
> RAID5 in QEMU (6.12.117). With KASAN on v7.4-rc2, a test-only delay
> (20 ms between the last decrement and the wake-up, 0.5 ms before the
> read wait) shows the use-after-free above in the first pass of reads
> over the RAID5 with one device dead. With this patch and the same
> delay there is no report and every block reads back correctly.
>
> v1: https://lore.kernel.org/linux-btrfs/20261115232358.58163-1-mc@raidlab.org/
> ---
> fs/btrfs/raid56.c | 30 ++++++++++++++++++++++--------
> fs/btrfs/raid56.h | 3 ++-
> 2 files changed, 24 insertions(+), 9 deletions(-)
>
> diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
> index a8a8004bb14c..60f270c3a339 100644
> --- a/fs/btrfs/raid56.c
> +++ b/fs/btrfs/raid56.c
> @@ -1019,7 +1019,7 @@ static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info,
> }
>
> bio_list_init(&rbio->bio_list);
> - init_waitqueue_head(&rbio->io_wait);
> + init_completion(&rbio->io_done);
> INIT_LIST_HEAD(&rbio->plug_list);
> spin_lock_init(&rbio->bio_list_lock);
> INIT_LIST_HEAD(&rbio->stripe_cache);
> @@ -1619,7 +1619,19 @@ static void raid_wait_read_end_io(struct bio *bio)
>
> bio_put(bio);
> if (atomic_dec_and_test(&rbio->stripes_pending))
> - wake_up(&rbio->io_wait);
> + complete(&rbio->io_done);
> +}
> +
> +/*
> + * Prepare to wait for the nr bios about to be submitted. With none there is
> + * nothing to wait for.
> + */
> +static void rbio_prepare_io_wait(struct btrfs_raid_bio *rbio, int nr)
> +{
> + reinit_completion(&rbio->io_done);
> + atomic_set(&rbio->stripes_pending, nr);
> + if (nr == 0)
> + complete(&rbio->io_done);
> }
>
> static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
> @@ -1627,7 +1639,7 @@ static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
> {
> struct bio *bio;
>
> - atomic_set(&rbio->stripes_pending, bio_list_size(bio_list));
> + rbio_prepare_io_wait(rbio, bio_list_size(bio_list));
> while ((bio = bio_list_pop(bio_list))) {
> bio->bi_end_io = raid_wait_read_end_io;
>
> @@ -1640,7 +1652,7 @@ static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
> submit_bio(bio);
> }
>
> - wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> + wait_for_completion_io(&rbio->io_done);
> }
>
> static int alloc_rbio_data_pages(struct btrfs_raid_bio *rbio)
> @@ -2329,7 +2341,7 @@ static void raid_wait_write_end_io(struct bio *bio)
> rbio_update_error_bitmap(rbio, bio);
> bio_put(bio);
> if (atomic_dec_and_test(&rbio->stripes_pending))
> - wake_up(&rbio->io_wait);
> + complete(&rbio->io_done);
> }
>
> static void submit_write_bios(struct btrfs_raid_bio *rbio,
> @@ -2337,7 +2349,7 @@ static void submit_write_bios(struct btrfs_raid_bio *rbio,
> {
> struct bio *bio;
>
> - atomic_set(&rbio->stripes_pending, bio_list_size(bio_list));
> + rbio_prepare_io_wait(rbio, bio_list_size(bio_list));
> while ((bio = bio_list_pop(bio_list))) {
> bio->bi_end_io = raid_wait_write_end_io;
>
> @@ -2444,7 +2456,7 @@ static void rmw_rbio(struct btrfs_raid_bio *rbio)
> /* We should have at least one bio assembled. */
> ASSERT(bio_list_size(&bio_list));
> submit_write_bios(rbio, &bio_list);
> - wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> + wait_for_completion_io(&rbio->io_done);
>
> /* We may have more errors than our tolerance during the read. */
> for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
> @@ -2915,7 +2927,9 @@ static void scrub_rbio(struct btrfs_raid_bio *rbio)
> * and writeback the good content.
> */
> ret = finish_parity_scrub(rbio);
> - wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> + /* It submitted its writes if and only if it returned 0. */
> + if (ret == 0)
> + wait_for_completion_io(&rbio->io_done);
> for (sector_nr = 0; sector_nr < rbio->stripe_nsectors; sector_nr++) {
> int found_errors;
>
> diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
> index 8700910b8dc9..c86ffd655360 100644
> --- a/fs/btrfs/raid56.h
> +++ b/fs/btrfs/raid56.h
> @@ -175,7 +175,8 @@ struct btrfs_raid_bio {
>
> atomic_t stripes_pending;
>
> - wait_queue_head_t io_wait;
> + /* Completed by the last end_io when stripes_pending drops to zero. */
> + struct completion io_done;
>
> /* Bitmap to record which horizontal stripe has data */
> unsigned long dbitmap;
>
> base-commit: 3aefd67f08f8ed59ed0fa90619b25856df415031
prev parent reply other threads:[~2026-11-16 1:03 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 0:19 Miroslav Cimerman
2026-11-16 1:02 ` Qu Wenruo [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=29d117e7-8ccd-4d55-a8d5-1ef78d851700@gmx.com \
--to=quwenruo.btrfs@gmx.com \
--cc=dsterba@suse.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mason@kernel.org \
--cc=mc@raidlab.org \
--cc=wqu@suse.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®