mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Qu Wenruo <quwenruo.btrfs@gmx.com>
To: Miroslav Cimerman <mc@raidlab.org>, linux-btrfs@vger.kernel.org
Cc: David Sterba <dsterba@suse.com>, Chris Mason <mason@kernel.org>,
	Qu Wenruo <wqu@suse.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] btrfs: raid56: fix use-after-free when waking up the rbio I/O waiter
Date: Mon, 16 Nov 2026 11:32:48 +1030	[thread overview]
Message-ID: <29d117e7-8ccd-4d55-a8d5-1ef78d851700@gmx.com> (raw)
In-Reply-To: <20261116001903.63524-1-mc@raidlab.org>



在 2026/11/16 10:49, Miroslav Cimerman 写道:
> The end_io functions of the RAID56 read and write bios decrement
> rbio->stripes_pending and call wake_up(&rbio->io_wait) when it drops
> to zero. The submitter waits with wait_event(rbio->io_wait,
> atomic_read(&rbio->stripes_pending) == 0) and then can drop the last
> reference to the rbio, which frees it.
> 
> If the submitter checks the condition after the last decrement and
> before the wake_up(), it does not sleep at all:
> 
>    CPU0 (endio, irq)                  CPU1 (submitter, rmw_rbio)
>    raid_wait_write_end_io()           rmw_rbio()
>    |                                  |-submit_write_bios()
>    |                                  | \-submit_bio()
>    |                                  |
>    |                                  |-wait_event()
>    |-bio_put()                        | ^^^ checks the condition without
>    |                                  |     taking io_wait.lock
>    |-atomic_dec_and_test()            |
>    | ^^^ stripes_pending hits 0       |
>    |                                  | ^^^ sees 0, doesn't sleep
>    |                                  |
>    |                                  \-rbio_orig_end_io()
>    |                                    \-kfree()
>    |                                      ^^^ last rbio reference, rbio
>    |                                          freed
>    |
>    \-wake_up()
>      \-_raw_spin_lock_irqsave()
>        ^^^ io_wait.lock is in freed memory
>        Too late.
> 
> Even when the submitter sleeps, the wake-up takes it off the wait
> queue, so finish_wait() returns without taking the queue's lock while
> wake_up() still holds it:
> 
>    CPU0 (endio, irq)                  CPU1 (submitter, rmw_rbio)
>    raid_wait_write_end_io()           rmw_rbio()
>    |                                  \-wait_event()
>    |                                    |-prepare_to_wait_event()
>    |                                    \-schedule()
>    |-atomic_dec_and_test()
>    | ^^^ stripes_pending hits 0
>    |
>    \-wake_up()
>      \-__wake_up_common_lock()
>        |-_raw_spin_lock_irqsave()
>        |-__wake_up_common()
>        | ^^^ autoremove_wake_function()
>        |     takes the waiter off the
>        |     queue and wakes CPU1       finish_wait()
>        |                                |-list_empty_careful()
>        |                                | ^^^ entry already removed,
>        |                                |     returns without taking
>        |                                |     io_wait.lock
>        |                                |
>        |                                \-rbio_orig_end_io()
>        |                                  \-kfree()
>        |                                    ^^^ rbio freed
>        |
>        \-_raw_spin_unlock_irqrestore()
>          ^^^ io_wait.lock is in freed memory
>          Too late.
> 
> The same happens on the read side (raid_wait_read_end_io() and
> submit_read_wait_bio_list()) and in the parity scrub. On 6.12.117
> (bio_endio() and the end_io function are tail calls, so they do not
> show in the trace):
> 
>    BUG: kernel NULL pointer dereference, address: 0000000000000000
>    RIP: 0010:__wake_up_common+0x25/0xa0
>    Call Trace:
>     <IRQ>
>     __wake_up+0x31/0x60
>     blk_update_request+0xfd/0x410
>     blk_mq_end_request+0x1d/0x1a0
>    Kernel panic - not syncing: Fatal exception in interrupt
> 
> With KASAN on v7.4-rc2 and a test-only delay between the decrement
> and the wake-up:
> 
>    BUG: KASAN: slab-use-after-free in _raw_spin_lock_irqsave+0x76/0xe0
>     __wake_up+0x1c/0x60
>     raid_wait_read_end_io+0x132/0x10f0
>     blk_update_request+0x21e/0xea0
>    Allocated by task 82:
>     alloc_rbio.isra.0+0xa8/0xbf0
>     raid56_parity_recover+0x10a/0x9c0
>    Freed by task 84:
>     kfree+0x154/0x420
>     rbio_orig_end_io+0x1c4/0x340
>     recover_rbio+0x393/0x590
> 
> Wait with a completion instead. complete() wakes the waiter under the
> completion's lock and does not touch it afterwards, and
> wait_for_completion() returns only after taking that lock, so the
> rbio may be freed as soon as the wait returns. The parity scrub waits
> for its writes only when finish_parity_scrub() submitted them.
> 
> Fixes: d817ce35d24a ("btrfs: raid56: switch recovery path to a single function")
> Fixes: 5eb30ee26fa4 ("btrfs: raid56: introduce the main entrance for RMW path")
> Fixes: 6bfd0133bee2 ("btrfs: raid56: switch scrub path to use a single function")
> Cc: stable@vger.kernel.org # 6.2+
> Reviewed-by: Qu Wenruo <wqu@suse.com>
> Signed-off-by: Miroslav Cimerman <mc@raidlab.org>

Pushed to for-next branch.

Thanks,
Qu

> ---
> Changes in v2:
> - ASCII race charts (Qu)
> - drop the FuzzyRAID sentence and the completion.h include (Qu)
> 
> Found with FuzzyRAID, a fault-injection tester for RAID that is not
> public yet. The analysis, the fix, the tests and the commit message
> were prepared with the help of an LLM. I reviewed everything.
> 
> The crash: 4 of 10,711 fault-injection scenarios on a 3-device data
> RAID5 in QEMU (6.12.117). With KASAN on v7.4-rc2, a test-only delay
> (20 ms between the last decrement and the wake-up, 0.5 ms before the
> read wait) shows the use-after-free above in the first pass of reads
> over the RAID5 with one device dead. With this patch and the same
> delay there is no report and every block reads back correctly.
> 
> v1: https://lore.kernel.org/linux-btrfs/20261115232358.58163-1-mc@raidlab.org/
> ---
>   fs/btrfs/raid56.c | 30 ++++++++++++++++++++++--------
>   fs/btrfs/raid56.h |  3 ++-
>   2 files changed, 24 insertions(+), 9 deletions(-)
> 
> diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
> index a8a8004bb14c..60f270c3a339 100644
> --- a/fs/btrfs/raid56.c
> +++ b/fs/btrfs/raid56.c
> @@ -1019,7 +1019,7 @@ static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info,
>   	}
>   
>   	bio_list_init(&rbio->bio_list);
> -	init_waitqueue_head(&rbio->io_wait);
> +	init_completion(&rbio->io_done);
>   	INIT_LIST_HEAD(&rbio->plug_list);
>   	spin_lock_init(&rbio->bio_list_lock);
>   	INIT_LIST_HEAD(&rbio->stripe_cache);
> @@ -1619,7 +1619,19 @@ static void raid_wait_read_end_io(struct bio *bio)
>   
>   	bio_put(bio);
>   	if (atomic_dec_and_test(&rbio->stripes_pending))
> -		wake_up(&rbio->io_wait);
> +		complete(&rbio->io_done);
> +}
> +
> +/*
> + * Prepare to wait for the nr bios about to be submitted. With none there is
> + * nothing to wait for.
> + */
> +static void rbio_prepare_io_wait(struct btrfs_raid_bio *rbio, int nr)
> +{
> +	reinit_completion(&rbio->io_done);
> +	atomic_set(&rbio->stripes_pending, nr);
> +	if (nr == 0)
> +		complete(&rbio->io_done);
>   }
>   
>   static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
> @@ -1627,7 +1639,7 @@ static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
>   {
>   	struct bio *bio;
>   
> -	atomic_set(&rbio->stripes_pending, bio_list_size(bio_list));
> +	rbio_prepare_io_wait(rbio, bio_list_size(bio_list));
>   	while ((bio = bio_list_pop(bio_list))) {
>   		bio->bi_end_io = raid_wait_read_end_io;
>   
> @@ -1640,7 +1652,7 @@ static void submit_read_wait_bio_list(struct btrfs_raid_bio *rbio,
>   		submit_bio(bio);
>   	}
>   
> -	wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> +	wait_for_completion_io(&rbio->io_done);
>   }
>   
>   static int alloc_rbio_data_pages(struct btrfs_raid_bio *rbio)
> @@ -2329,7 +2341,7 @@ static void raid_wait_write_end_io(struct bio *bio)
>   		rbio_update_error_bitmap(rbio, bio);
>   	bio_put(bio);
>   	if (atomic_dec_and_test(&rbio->stripes_pending))
> -		wake_up(&rbio->io_wait);
> +		complete(&rbio->io_done);
>   }
>   
>   static void submit_write_bios(struct btrfs_raid_bio *rbio,
> @@ -2337,7 +2349,7 @@ static void submit_write_bios(struct btrfs_raid_bio *rbio,
>   {
>   	struct bio *bio;
>   
> -	atomic_set(&rbio->stripes_pending, bio_list_size(bio_list));
> +	rbio_prepare_io_wait(rbio, bio_list_size(bio_list));
>   	while ((bio = bio_list_pop(bio_list))) {
>   		bio->bi_end_io = raid_wait_write_end_io;
>   
> @@ -2444,7 +2456,7 @@ static void rmw_rbio(struct btrfs_raid_bio *rbio)
>   	/* We should have at least one bio assembled. */
>   	ASSERT(bio_list_size(&bio_list));
>   	submit_write_bios(rbio, &bio_list);
> -	wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> +	wait_for_completion_io(&rbio->io_done);
>   
>   	/* We may have more errors than our tolerance during the read. */
>   	for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
> @@ -2915,7 +2927,9 @@ static void scrub_rbio(struct btrfs_raid_bio *rbio)
>   	 * and writeback the good content.
>   	 */
>   	ret = finish_parity_scrub(rbio);
> -	wait_event(rbio->io_wait, atomic_read(&rbio->stripes_pending) == 0);
> +	/* It submitted its writes if and only if it returned 0. */
> +	if (ret == 0)
> +		wait_for_completion_io(&rbio->io_done);
>   	for (sector_nr = 0; sector_nr < rbio->stripe_nsectors; sector_nr++) {
>   		int found_errors;
>   
> diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
> index 8700910b8dc9..c86ffd655360 100644
> --- a/fs/btrfs/raid56.h
> +++ b/fs/btrfs/raid56.h
> @@ -175,7 +175,8 @@ struct btrfs_raid_bio {
>   
>   	atomic_t stripes_pending;
>   
> -	wait_queue_head_t io_wait;
> +	/* Completed by the last end_io when stripes_pending drops to zero. */
> +	struct completion io_done;
>   
>   	/* Bitmap to record which horizontal stripe has data */
>   	unsigned long dbitmap;
> 
> base-commit: 3aefd67f08f8ed59ed0fa90619b25856df415031


      reply	other threads:[~2026-11-16  1:03 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  0:19 Miroslav Cimerman
2026-11-16  1:02 ` Qu Wenruo [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=29d117e7-8ccd-4d55-a8d5-1ef78d851700@gmx.com \
    --to=quwenruo.btrfs@gmx.com \
    --cc=dsterba@suse.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mason@kernel.org \
    --cc=mc@raidlab.org \
    --cc=wqu@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®