From: Qu Wenruo <wqu@suse.com>
To: Miroslav Cimerman <mc@raidlab.org>, linux-btrfs@vger.kernel.org
Cc: David Sterba <dsterba@suse.com>, Chris Mason <mason@kernel.org>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH] btrfs: raid56: fix use-after-free when waking up the rbio I/O waiter
Date: Mon, 16 Nov 2026 10:27:15 +1030 [thread overview]
Message-ID: <8bf058d0-558f-449c-ba96-2b4a1d067dd2@suse.com> (raw)
In-Reply-To: <20261115232358.58163-1-mc@raidlab.org>
在 2026/11/16 09:53, Miroslav Cimerman 写道:
> The end_io functions of the RAID56 read and write bios decrement
> rbio->stripes_pending and call wake_up(&rbio->io_wait) when it drops
> to zero. The submitter waits with wait_event(rbio->io_wait,
> atomic_read(&rbio->stripes_pending) == 0).
>
> wait_event() checks the condition before it sleeps, without the wait
> queue's lock. If the submitter checks it after the last decrement and
> before the wake_up(), it does not sleep, finishes the operation and
> can drop the last reference to the rbio, which frees it. Even when
> the submitter sleeps, the wake-up removes it from the wait queue, so
> finish_wait() returns without taking the queue's lock while wake_up()
> may still hold it. Either way, wake_up() then uses a wait queue in
> freed memory, in interrupt context. On 6.12.117 (bio_endio() and the
> end_io function are tail calls, so they do not show in the trace):
The analyze looks good to me. Although I'd prefer an ASCII chart showing
the race window more clearly, e.g:
CPU0 (endio, irq) CPU1 (submitter, rmw_rbio)
raid_wait_write_end_io() rmw_rbio()
| |-submit_write_bios()
| | \-submit_bio()
| |
| |-wait_event()
|-bio_put() | ^^^ checks the condition without
| | taking io_wait.lock
|-atomic_dec_and_test() |
| ^^^ stripes_pending hits 0 |
| | ^^^ sees 0, doesn't sleep
| |
| \-rbio_orig_end_io()
| \-kfree()
| ^^^ last rbio reference, rbio
| freed
|
\-wake_up()
\-_raw_spin_lock_irqsave()
^^^ io_wait.lock is in freed memory
Too late.
And the waiter sleeps case:
CPU0 (endio, irq) CPU1 (submitter, rmw_rbio)
raid_wait_write_end_io() rmw_rbio()
| \-wait_event()
| |-prepare_to_wait_event()
| \-schedule()
|-atomic_dec_and_test()
| ^^^ stripes_pending hits 0
|
\-wake_up()
\-__wake_up_common_lock()
|-_raw_spin_lock_irqsave()
|-__wake_up_common()
| ^^^ autoremove_wake_function()
| takes the waiter off the
| queue and wakes CPU1 finish_wait()
| |-list_empty_careful()
| | ^^^ entry already removed,
| | returns without taking
| | io_wait.lock
| |
| \-rbio_orig_end_io()
| \-kfree()
| ^^^ rbio freed
|
\-_raw_spin_unlock_irqrestore()
^^^ io_wait.lock is in freed memory
Too late.
>
> BUG: kernel NULL pointer dereference, address: 0000000000000000
> RIP: 0010:__wake_up_common+0x25/0xa0
> Call Trace:
> <IRQ>
> __wake_up+0x31/0x60
> blk_update_request+0xfd/0x410
> blk_mq_end_request+0x1d/0x1a0
> Kernel panic - not syncing: Fatal exception in interrupt
>
> With KASAN on v7.4-rc2 and a test-only delay between the decrement
> and the wake-up:
>
> BUG: KASAN: slab-use-after-free in _raw_spin_lock_irqsave+0x76/0xe0
> __wake_up+0x1c/0x60
> raid_wait_read_end_io+0x132/0x10f0
> blk_update_request+0x21e/0xea0
> Allocated by task 82:
> alloc_rbio.isra.0+0xa8/0xbf0
> raid56_parity_recover+0x10a/0x9c0
> Freed by task 84:
> kfree+0x154/0x420
> rbio_orig_end_io+0x1c4/0x340
> recover_rbio+0x393/0x590
>
> Wait with a completion instead. complete() wakes the waiter under the
> completion's lock and does not touch it afterwards, and
> wait_for_completion() returns only after taking that lock, so the
> rbio may be freed as soon as the wait returns. The parity scrub waits
> for its writes only when finish_parity_scrub() submitted them.
>
> Found with FuzzyRAID, a fault-injection tester for RAID.
Since it's not a public tool, I think you can skip this sentence.>
> Fixes: d817ce35d24a ("btrfs: raid56: switch recovery path to a single function")
> Fixes: 5eb30ee26fa4 ("btrfs: raid56: introduce the main entrance for RMW path")
> Fixes: 6bfd0133bee2 ("btrfs: raid56: switch scrub path to use a single function")
> Cc: stable@vger.kernel.org # 6.2+
> Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Just a nitpick below related the header.
[...]
> diff --git a/fs/btrfs/raid56.h b/fs/btrfs/raid56.h
> index 8700910b8dc9..6c82a975eb68 100644
> --- a/fs/btrfs/raid56.h
> +++ b/fs/btrfs/raid56.h
> @@ -12,6 +12,7 @@
> #include <linux/spinlock.h>
> #include <linux/bio.h>
> #include <linux/refcount.h>
> +#include <linux/completion.h>
You can skip this header, "volumes.h" has already pulled in this header.
Thanks for catching and fixing the bug,
Qu
> #include <linux/workqueue.h>
> #include "volumes.h"
>
> @@ -175,7 +176,8 @@ struct btrfs_raid_bio {
>
> atomic_t stripes_pending;
>
> - wait_queue_head_t io_wait;
> + /* Completed by the last end_io when stripes_pending drops to zero. */
> + struct completion io_done;
>
> /* Bitmap to record which horizontal stripe has data */
> unsigned long dbitmap;
>
> base-commit: 3aefd67f08f8ed59ed0fa90619b25856df415031
prev parent reply other threads:[~2026-11-15 23:57 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-15 23:23 Miroslav Cimerman
2026-11-15 23:57 ` Qu Wenruo [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8bf058d0-558f-449c-ba96-2b4a1d067dd2@suse.com \
--to=wqu@suse.com \
--cc=dsterba@suse.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mason@kernel.org \
--cc=mc@raidlab.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®