From: Dave Jones <davej@codemonkey.org.uk>
To: Chris Mason <clm@fb.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Jens Axboe <axboe@fb.com>, Andy Lutomirski <luto@amacapital.net>,
Andy Lutomirski <luto@kernel.org>,
Al Viro <viro@zeniv.linux.org.uk>, Josef Bacik <jbacik@fb.com>,
David Sterba <dsterba@suse.com>,
linux-btrfs <linux-btrfs@vger.kernel.org>,
Linux Kernel <linux-kernel@vger.kernel.org>,
Dave Chinner <david@fromorbit.com>
Subject: Re: bio linked list corruption.
Date: Wed, 23 Nov 2016 14:34:19 -0500 [thread overview]
Message-ID: <20161123193419.pq7adje2eanky2wx@codemonkey.org.uk> (raw)
In-Reply-To: <20161031194454.GA49877@clm-mbp.thefacebook.com>
On Mon, Oct 31, 2016 at 01:44:55PM -0600, Chris Mason wrote:
> On Mon, Oct 31, 2016 at 12:35:16PM -0700, Linus Torvalds wrote:
> >On Mon, Oct 31, 2016 at 11:55 AM, Dave Jones <davej@codemonkey.org.uk> wrote:
> >>
> >> BUG: Bad page state in process kworker/u8:12 pfn:4e0e39
> >> page:ffffea0013838e40 count:0 mapcount:0 mapping:ffff8804a20310e0 index:0x100c
> >> flags: 0x400000000000000c(referenced|uptodate)
> >> page dumped because: non-NULL mapping
> >
> >Hmm. So this seems to be btrfs-specific, right?
> >
> >I searched for all your "non-NULL mapping" cases, and they all seem to
> >have basically the same call trace, with some work thread doing
> >writeback and going through btrfs_writepages().
> >
> >Sounds like it's a race with either fallocate hole-punching or
> >truncate. I'm not seeing it, but I suspect it's btrfs, since DaveJ
> >clearly ran other filesystems too but I am not seeing this backtrace
> >for anything else.
>
> Agreed, I think this is a separate bug, almost certainly btrfs specific.
> I'll work with Dave on a better reproducer.
Ok, let's try this..
Here's the usual dump, from the current tree:
[ 317.689216] BUG: Bad page state in process kworker/u8:8 pfn:4d8fd4
[ 317.702235] page:ffffea001363f500 count:0 mapcount:0 mapping:ffff8804eef8e0e0 index:0x29
[ 317.718068] flags: 0x400000000000000c(referenced|uptodate)
[ 317.731086] page dumped because: non-NULL mapping
[ 317.744118] CPU: 3 PID: 1558 Comm: kworker/u8:8 Not tainted 4.9.0-rc6-think+ #1
[ 317.770404] Workqueue: writeback wb_workfn
[ 317.783490] (flush-btrfs-3)
[ 317.783581] ffffc90000ff3798
[ 317.796651] ffffffff813b69bc
[ 317.796742] ffffea001363f500
[ 317.796798] ffffffff81c474ee
[ 317.796854] ffffc90000ff37c0
[ 317.809910] ffffffff811b30c4
[ 317.810001] 0000000000000000
[ 317.810055] ffffea001363f500
[ 317.810112] 0000000000000003
[ 317.823230] ffffc90000ff37d0
[ 317.823320] ffffffff811b318f
[ 317.823376] ffffc90000ff3818
[ 317.823431] Call Trace:
[ 317.836645] [<ffffffff813b69bc>] dump_stack+0x4f/0x73
[ 317.850063] [<ffffffff811b30c4>] bad_page+0xc4/0x130
[ 317.863426] [<ffffffff811b318f>] free_pages_check_bad+0x5f/0x70
[ 317.876805] [<ffffffff811b61f5>] free_hot_cold_page+0x305/0x3b0
[ 317.890132] [<ffffffff811b660e>] free_hot_cold_page_list+0x7e/0x170
[ 317.903410] [<ffffffff811c08b7>] release_pages+0x2e7/0x3a0
[ 317.916600] [<ffffffff811c2067>] __pagevec_release+0x27/0x40
[ 317.929817] [<ffffffffa00b0a55>] extent_write_cache_pages.isra.44.constprop.59+0x355/0x430 [btrfs]
[ 317.943073] [<ffffffffa00b0fcd>] extent_writepages+0x5d/0x90 [btrfs]
[ 317.956033] [<ffffffffa008ea60>] ? btrfs_releasepage+0x40/0x40 [btrfs]
[ 317.968902] [<ffffffffa008b6a8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 317.981744] [<ffffffff811be6d1>] do_writepages+0x21/0x30
[ 317.994524] [<ffffffff812727cf>] __writeback_single_inode+0x7f/0x6e0
[ 318.007333] [<ffffffff81894cc1>] ? _raw_spin_unlock+0x31/0x50
[ 318.020090] [<ffffffff8127356d>] writeback_sb_inodes+0x31d/0x6c0
[ 318.032844] [<ffffffff812739a2>] __writeback_inodes_wb+0x92/0xc0
[ 318.045623] [<ffffffff81273e97>] wb_writeback+0x3f7/0x530
[ 318.058304] [<ffffffff81274a38>] wb_workfn+0x148/0x620
[ 318.070949] [<ffffffff810a4b54>] ? process_one_work+0x194/0x690
[ 318.083549] [<ffffffff812748f5>] ? wb_workfn+0x5/0x620
[ 318.096160] [<ffffffff810a4bf2>] process_one_work+0x232/0x690
[ 318.108722] [<ffffffff810a4b54>] ? process_one_work+0x194/0x690
[ 318.121225] [<ffffffff810a509e>] worker_thread+0x4e/0x490
[ 318.133680] [<ffffffff810a5050>] ? process_one_work+0x690/0x690
[ 318.146126] [<ffffffff810a5050>] ? process_one_work+0x690/0x690
[ 318.158507] [<ffffffff810aa7e2>] kthread+0x102/0x120
[ 318.170880] [<ffffffff810aa6e0>] ? kthread_park+0x60/0x60
[ 318.183254] [<ffffffff81895822>] ret_from_fork+0x22/0x30
This time, I managed to get my ftrace magic working, so we have this
trace from just before this happened. Does this shed any light ?
https://codemonkey.org.uk/junk/trace.txt
(note that some of the spinlock/rcu calls will be missing here. I chose
to add them to ftrace's exclude list because they dominate the trace
buffer so much, especially in debug builds)
Dave
next prev parent reply other threads:[~2016-11-23 19:34 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-11 14:45 btrfs " Dave Jones
2016-10-11 15:11 ` Al Viro
2016-10-11 15:19 ` Dave Jones
2016-10-11 15:20 ` Chris Mason
2016-10-11 15:49 ` Dave Jones
2016-10-11 15:54 ` Chris Mason
2016-10-11 16:25 ` Dave Jones
2016-10-12 13:47 ` Dave Jones
2016-10-12 14:40 ` Dave Jones
2016-10-12 14:42 ` Chris Mason
2016-10-13 18:16 ` Dave Jones
2016-10-13 21:18 ` Chris Mason
2016-10-13 21:56 ` Dave Jones
2016-10-16 0:42 ` Dave Jones
2016-10-18 1:07 ` Chris Mason
2016-10-18 22:42 ` Dave Jones
2016-10-18 23:12 ` Jens Axboe
2016-10-18 23:31 ` Chris Mason
2016-10-18 23:36 ` Jens Axboe
2016-10-18 23:39 ` Linus Torvalds
2016-10-18 23:42 ` Chris Mason
2016-10-19 0:10 ` Linus Torvalds
2016-10-19 0:19 ` Chris Mason
2016-10-19 0:28 ` Linus Torvalds
2016-10-20 22:48 ` Dave Jones
2016-10-19 1:05 ` Andy Lutomirski
2016-10-20 22:50 ` Dave Jones
2016-10-20 23:01 ` Andy Lutomirski
2016-10-20 23:03 ` Dave Jones
2016-10-20 23:23 ` Andy Lutomirski
2016-10-21 20:02 ` Dave Jones
2016-10-21 20:17 ` Chris Mason
2016-10-21 20:23 ` Dave Jones
2016-10-21 20:38 ` Chris Mason
2016-10-21 20:41 ` Josef Bacik
2016-10-21 21:11 ` Dave Jones
2016-10-22 15:20 ` Dave Jones
2016-10-23 21:32 ` Chris Mason
2016-10-24 4:40 ` Dave Jones
2016-10-24 13:42 ` Chris Mason
2016-10-26 0:27 ` Dave Jones
2016-10-26 1:33 ` Linus Torvalds
2016-10-26 1:39 ` Linus Torvalds
2016-10-26 16:30 ` Dave Jones
2016-10-26 16:48 ` Linus Torvalds
2016-10-26 18:18 ` Dave Jones
2016-10-26 18:42 ` Dave Jones
2016-10-26 19:06 ` Linus Torvalds
2016-10-26 20:00 ` Chris Mason
2016-10-26 21:52 ` Chris Mason
2016-10-26 22:21 ` Linus Torvalds
2016-10-26 22:40 ` Dave Jones
2016-10-26 22:51 ` Linus Torvalds
2016-10-26 22:55 ` Jens Axboe
2016-10-26 22:58 ` Linus Torvalds
2016-10-26 23:03 ` Jens Axboe
2016-10-26 23:07 ` Dave Jones
2016-10-26 23:08 ` Linus Torvalds
2016-10-26 23:20 ` Jens Axboe
2016-10-26 23:38 ` Chris Mason
2016-10-26 23:47 ` Dave Jones
2016-10-27 0:00 ` Jens Axboe
2016-10-27 13:33 ` Chris Mason
2016-10-31 18:55 ` Dave Jones
2016-10-31 19:35 ` Linus Torvalds
2016-10-31 19:44 ` Chris Mason
2016-11-06 16:55 ` btrfs btree_ctree_super fault Dave Jones
2016-11-08 14:59 ` Dave Jones
2016-11-08 15:08 ` Chris Mason
2016-11-10 14:35 ` Dave Jones
2016-11-10 15:27 ` Chris Mason
2016-11-23 19:34 ` Dave Jones [this message]
2016-11-23 19:58 ` bio linked list corruption Dave Jones
2016-12-01 15:32 ` btrfs_destroy_inode warn (outstanding extents) Dave Jones
2016-12-03 16:48 ` Dave Jones
2016-12-07 16:15 ` Dave Jones
2016-12-09 21:12 ` Steven Rostedt
2016-12-04 23:04 ` bio linked list corruption Vegard Nossum
2016-12-05 11:10 ` Vegard Nossum
2016-12-05 17:09 ` Vegard Nossum
2016-12-05 17:21 ` Dave Jones
2016-12-05 17:55 ` Linus Torvalds
2016-12-05 19:11 ` Vegard Nossum
2016-12-05 20:10 ` Linus Torvalds
2016-12-05 20:35 ` Linus Torvalds
2016-12-05 21:33 ` Vegard Nossum
2016-12-06 8:42 ` Vegard Nossum
2016-12-06 8:16 ` Peter Zijlstra
2016-12-06 8:36 ` Ingo Molnar
2016-12-06 16:33 ` Linus Torvalds
2016-12-05 20:10 ` Vegard Nossum
2016-12-05 18:11 ` Andy Lutomirski
2016-12-05 18:25 ` Linus Torvalds
2016-12-05 18:26 ` Vegard Nossum
2016-10-26 23:19 ` Chris Mason
2016-10-26 23:21 ` Jens Axboe
2016-10-27 6:33 ` Christoph Hellwig
2016-10-27 16:34 ` Linus Torvalds
2016-10-27 16:36 ` Jens Axboe
2016-10-26 23:01 ` Dave Jones
2016-10-26 23:05 ` Jens Axboe
2016-10-26 22:52 ` Jens Axboe
2016-10-26 22:07 ` Linus Torvalds
2016-10-26 22:54 ` Chris Mason
2016-10-27 5:41 ` Dave Chinner
2016-10-27 17:23 ` Dave Jones
2016-10-24 20:06 ` Andy Lutomirski
2016-10-24 20:46 ` Linus Torvalds
2016-10-24 21:17 ` Linus Torvalds
2016-10-24 21:50 ` Linus Torvalds
2016-10-24 22:02 ` Chris Mason
2016-10-24 22:42 ` Andy Lutomirski
2016-10-25 0:00 ` Linus Torvalds
2016-10-25 1:09 ` Andy Lutomirski
2016-10-19 17:09 ` Philipp Hahn
2016-10-19 17:43 ` Linus Torvalds
2016-10-20 6:52 ` Ingo Molnar
2016-10-20 7:17 ` Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161123193419.pq7adje2eanky2wx@codemonkey.org.uk \
--to=davej@codemonkey.org.uk \
--cc=axboe@fb.com \
--cc=clm@fb.com \
--cc=david@fromorbit.com \
--cc=dsterba@suse.com \
--cc=jbacik@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@amacapital.net \
--cc=luto@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®