From: Lukas Czerner <lczerner@redhat.com>
To: Jeff Moyer <jmoyer@redhat.com>
Cc: Lukas Czerner <lczerner@redhat.com>,
linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH] block: fix mis-synchronisation in blkdev_issue_zeroout()
Date: Fri, 4 Mar 2011 16:04:16 +0100 (CET) [thread overview]
Message-ID: <alpine.LFD.2.00.1103041552020.5003@dhcp-27-109.brq.redhat.com> (raw)
In-Reply-To: <x49zkpbhszn.fsf@segfault.boston.devel.redhat.com>
On Fri, 4 Mar 2011, Jeff Moyer wrote:
> Lukas Czerner <lczerner@redhat.com> writes:
>
> > BZ29402
> > https://bugzilla.kernel.org/show_bug.cgi?id=29402
> >
> > We can hit serious mis-synchronization in bio completion path of
> > blkdev_issue_zeroout() leading to a panic.
> >
> > The problem is that when we are going to wait_for_completion() in
> > blkdev_issue_zeroout() we check if the bb.done equals issued (number of
> > submitted bios). If it does, we can skip the wait_for_completition()
> > and just out of the function since there is nothing to wait for.
> > However, there is a ordering problem because bio_batch_end_io() is
> > calling atomic_inc(&bb->done) before complete(), hence it might seem to
> > blkdev_issue_zeroout() that all bios has been completed and exit. At
> > this point when bio_batch_end_io() is going to call complete(bb->wait),
> > bb and wait does not longer exist since it was allocated on stack in
> > blkdev_issue_zeroout() ==> panic!
> >
> > (thread 1) (thread 2)
> > bio_batch_end_io() blkdev_issue_zeroout()
> > if(bb) { ...
> > if (bb->end_io) ...
> > bb->end_io(bio, err); ...
> > atomic_inc(&bb->done); ...
> > ... while (issued != atomic_read(&bb.done))
> > ... (let issued == bb.done)
> > ... (do the rest of the function)
> > ... return ret;
> > complete(bb->wait);
> > ^^^^^^^^
> > panic
>
> That's a pretty tight window. The complete is immediately following the
> increment. I'm surprised thread 2 has time to finish up and exit the
> function before the completion is done.
Yeah, maybe that is why it is so hard to hit. Eric was able to hit it
only on 48 core box running ffsb test with 192 threads. Also the panic
backtrace bug "bad spinlock magic" pointed me to this.
>
> > We can fix this easily by simplifying bio_batch and completion counting.
> > We can count completion locally in blkdev_issue_zeroout() without need of
> > locking or atomic operation because we are the only one handling issued
> > variable holding the number of submitted bios. So remove atomic_t done
> > from struct bio_batch.
>
> It seems to me like it might be better to just not complete anything
> until the count is zero. Why issue a wakeup for every bio?
> fs/direct-io does something similar, maybe take a look at the
> dio_bio_end* routines and see if that would fit well here. With your
> scheme, I worry about missing a completion, maybe because the first bio
> completes before you are done submitting bios. Is that possible?
I do not think it is possible. For every bio submitted there is
wait_for_completion called. When bio complete()s completion->done is
incremented (under the wait->lock). In wait_for_completion() we are
waiting for single submitted bio to complete (completion->done > 0),
then completion->done is decremented. It seems like simple
synchronization.
I am not sure what wakeup you have in mind, but thanks for the tip I'll
look in fs/direct-io.
Thanks!
-Lukas
>
> > Also remove bio_end_io_t *end_io since it is not used.
>
> Yeah, no idea why that was in there.
>
> Cheers,
> Jeff
>
next prev parent reply other threads:[~2011-03-04 15:04 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-04 9:46 Lukas Czerner
2011-03-04 14:15 ` Jeff Moyer
2011-03-04 15:04 ` Lukas Czerner [this message]
2011-03-04 15:15 ` Jeff Moyer
2011-03-07 12:25 ` Lukas Czerner
2011-03-07 14:38 ` Jeff Moyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.LFD.2.00.1103041552020.5003@dhcp-27-109.brq.redhat.com \
--to=lczerner@redhat.com \
--cc=jmoyer@redhat.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®