mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Qu Wenruo <wqu@suse.com>
To: Miroslav Cimerman <mc@raidlab.org>, linux-btrfs@vger.kernel.org
Cc: David Sterba <dsterba@suse.com>, Chris Mason <mason@kernel.org>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] btrfs: raid56: fix use-after-free when waking up the rbio I/O waiter
Date: Mon, 16 Nov 2026 10:27:15 +1030	[thread overview]
Message-ID: <8bf058d0-558f-449c-ba96-2b4a1d067dd2@suse.com> (raw)
In-Reply-To: <20261115232358.58163-1-mc@raidlab.org>



在 2026/11/16 09:53, Miroslav Cimerman 写道:
> The end_io functions of the RAID56 read and write bios decrement
> rbio->stripes_pending and call wake_up(&rbio->io_wait) when it drops
> to zero. The submitter waits with wait_event(rbio->io_wait,
> atomic_read(&rbio->stripes_pending) == 0).
> 
> wait_event() checks the condition before it sleeps, without the wait
> queue's lock. If the submitter checks it after the last decrement and
> before the wake_up(), it does not sleep, finishes the operation and
> can drop the last reference to the rbio, which frees it. Even when
> the submitter sleeps, the wake-up removes it from the wait queue, so
> finish_wait() returns without taking the queue's lock while wake_up()
> may still hold it. Either way, wake_up() then uses a wait queue in
> freed memory, in interrupt context. On 6.12.117 (bio_endio() and the
> end_io function are tail calls, so they do not show in the trace):

The analyze looks good to me. Although I'd prefer an ASCII chart showing 
the race window more clearly, e.g:

    CPU0 (endio, irq)                  CPU1 (submitter, rmw_rbio)
    raid_wait_write_end_io()           rmw_rbio()
    |                                  |-submit_write_bios()
    |                                  | \-submit_bio()
    |                                  |
    |                                  |-wait_event()
    |-bio_put()                        | ^^^ checks the condition without
    |                                  |     taking io_wait.lock
    |-atomic_dec_and_test()            |
    | ^^^ stripes_pending hits 0       |
    |                                  | ^^^ sees 0, doesn't sleep
    |                                  |
    |                                  \-rbio_orig_end_io()
    |                                    \-kfree()
    |                                      ^^^ last rbio reference, rbio
    |                                          freed
    |
    \-wake_up()
      \-_raw_spin_lock_irqsave()
        ^^^ io_wait.lock is in freed memory
        Too late.

And the waiter sleeps case:

    CPU0 (endio, irq)                  CPU1 (submitter, rmw_rbio)
    raid_wait_write_end_io()           rmw_rbio()
    |                                  \-wait_event()
    |                                    |-prepare_to_wait_event()
    |                                    \-schedule()
    |-atomic_dec_and_test()
    | ^^^ stripes_pending hits 0
    |
    \-wake_up()
      \-__wake_up_common_lock()
        |-_raw_spin_lock_irqsave()
        |-__wake_up_common()
        | ^^^ autoremove_wake_function()
        |     takes the waiter off the
        |     queue and wakes CPU1       finish_wait()
        |                                |-list_empty_careful()
        |                                | ^^^ entry already removed,
        |                                |     returns without taking
        |                                |     io_wait.lock
        |                                |
        |                                \-rbio_orig_end_io()
        |                                  \-kfree()
        |                                    ^^^ rbio freed
        |
        \-_raw_spin_unlock_irqrestore()
          ^^^ io_wait.lock is in freed memory
          Too late.



> 
>    BUG: kernel NULL pointer dereference, address: 0000000000000000
>    RIP: 0010:__wake_up_common+0x25/0xa0
>    Call Trace:
>     <IRQ>
>     __wake_up+0x31/0x60
>     blk_update_request+0xfd/0x410
>     blk_mq_end_request+0x1d/0x1a0
>    Kernel panic - not syncing: Fatal exception in interrupt
> 
> With KASAN on v7.4-rc2 and a test-only delay between the decrement
> and the wake-up:
> 
>    BUG: KASAN: slab-use-after-free in _raw_spin_lock_irqsave+0x76/0xe0
>     __wake_up+0x1c/0x60
>     raid_wait_read_end_io+0x132/0x10f0
>     blk_update_request+0x21e/0xea0
>    Allocated by task 82:
>     alloc_rbio.isra.0+0xa8/0xbf0
>     raid56_parity_recover+0x10a/0x9c0
>    Freed by task 84:
>     kfree+0x154/0x420
>     rbio_orig_end_io+0x1c4/0x340
>     recover_rbio+0x393/0x590
> 
> Wait with a completion instead. complete() wakes the waiter under the
> completion's lock and does not touch it afterwards, and
> wait_for_completion() returns only after taking that lock, so the
> rbio may be freed as soon as the wait returns. The parity scrub waits
> for its writes only when finish_parity_scrub() submitted them.
> 
> Found with FuzzyRAID, a fault-injection tester for RAID.

Since it's not a public tool, I think you can skip this sentence.>
> Fixes: d817ce35d24a ("btrfs: raid56: switch recovery path to a single function")
> Fixes: 5eb30ee26fa4 ("btrfs: raid56: introduce the main entrance for RMW path")
> Fixes: 6bfd0133bee2 ("btrfs: raid56: switch scrub path to use a single function")
> Cc: stable@vger.kernel.org # 6.2+
> Signed-off-by: Miroslav Cimerman <mc@raidlab.org>

Reviewed-by: Qu Wenruo <wqu@suse.com>

Just a nitpick below related the header.

[...]
> diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
> index 8700910b8dc9..6c82a975eb68 100644
> --- a/fs/btrfs/raid56.h
> +++ b/fs/btrfs/raid56.h
> @@ -12,6 +12,7 @@
>   #include <linux/spinlock.h>
>   #include <linux/bio.h>
>   #include <linux/refcount.h>
> +#include <linux/completion.h>

You can skip this header, "volumes.h" has already pulled in this header.

Thanks for catching and fixing the bug,
Qu

>   #include <linux/workqueue.h>
>   #include "volumes.h"
>   
> @@ -175,7 +176,8 @@ struct btrfs_raid_bio {
>   
>   	atomic_t stripes_pending;
>   
> -	wait_queue_head_t io_wait;
> +	/* Completed by the last end_io when stripes_pending drops to zero. */
> +	struct completion io_done;
>   
>   	/* Bitmap to record which horizontal stripe has data */
>   	unsigned long dbitmap;
> 
> base-commit: 3aefd67f08f8ed59ed0fa90619b25856df415031


      reply	other threads:[~2026-11-15 23:57 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-15 23:23 Miroslav Cimerman
2026-11-15 23:57 ` Qu Wenruo [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8bf058d0-558f-449c-ba96-2b4a1d067dd2@suse.com \
    --to=wqu@suse.com \
    --cc=dsterba@suse.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mason@kernel.org \
    --cc=mc@raidlab.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®