CONFIG_PREMPT vs. linux-next causes slub_debug=FZPUA errors

CONFIG_PREMPT vs. linux-next causes slub_debug=FZPUA errors

[Doug Anderson]

Hi,

Recently, both Andrew Bresticker (CCed) and I have noticed errors
reported by slub_debug when using linuxnext.  I'm still seeing them on
20140606.


NOTES:
* errors only seem to be present when you've got CONFIG_PREMPT enabled.

* errors seem to happen most often when rebooting, but I've seen them
during other times as well.

* a quick guesstimate is that ~50% of my reboots show this problem.

* I'm testing on an exynos5250-based board and Andrew on an ARM board
with a different SoC.

* I don't see issues on a current fetch of "linux", AKA (5b174fd Merge
branch 'for-3.16' of git://linux-nfs.org/~bfields/linux)


I didn't do a full bisect, but I don't seem to see issues on 20140528.
On 20140530 I see almost constant slub errors (seems much worse than
20140606)

An example error is below:


[   13.521842] =============================================================================
[   13.528566] BUG bio-1 (Not tainted): Poison overwritten
[   13.533765] -----------------------------------------------------------------------------
[   13.533765]
[   13.543394] Disabling lock debugging due to kernel taint
[   13.548685] INFO: 0xec495848-0xec49584f. First byte 0x80 instead of 0x6b
[   13.555378] INFO: Allocated in mempool_alloc_slab+0x24/0x28 age=28
cpu=5 pid=220
[   13.562741]  kmem_cache_alloc+0x90/0x1d0
[   13.566640]  mempool_alloc_slab+0x24/0x28
[   13.570631]  mempool_alloc+0x4c/0x144
[   13.574279]  bio_alloc_bioset+0xa4/0x1b0
[   13.578182]  __split_and_process_bio+0x33c/0x3f0
[   13.582778]  dm_request+0x1f0/0x204
[   13.586248]  generic_make_request+0xa4/0xdc
[   13.590411]  submit_bio+0x114/0x144
[   13.593882]  _submit_bh+0x22c/0x250
[   13.597351]  submit_bh+0x1c/0x20
[   13.600566]  jbd2_journal_commit_transaction+0xb24/0x1778
[   13.605940]  kjournald2+0xc0/0x264
[   13.609325]  kthread+0xf8/0x10c
[   13.612447]  ret_from_fork+0x14/0x20
[   13.616006] INFO: Freed in mempool_free_slab+0x24/0x28 age=40 cpu=1 pid=196
[   13.622944]  kmem_cache_free+0x1a4/0x234
[   13.626845]  mempool_free_slab+0x24/0x28
[   13.630748]  mempool_free+0x98/0x9c
[   13.634220]  bio_put+0x8c/0x98
[   13.637259]  clone_endio+0xd0/0xec
[   13.640640]  bio_endio+0xc0/0xcc
[   13.643852]  crypt_end_io+0x50/0x54
[   13.647320]  crypt_endio+0xb8/0xc0
[   13.650703]  bio_endio+0xc0/0xcc
[   13.653914]  loop_thread+0x61c/0x660
[   13.657470]  kthread+0xf8/0x10c
[   13.660592]  ret_from_fork+0x14/0x20
[   13.664150] INFO: Slab 0xef57c200 objects=39 used=0 fp=0xec490000
flags=0x4080
[   13.671347] INFO: Object 0xec4957c0 @offset=22464 fp=0x  (null)
[   13.671347]
[   13.678726] Bytes b4 ec4957b0: c4 00 00 00 eb 1e ff ff 5a 5a 5a 5a
5a 5a 5a 5a  ........ZZZZZZZZ
[   13.687484] Object ec4957c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.696072] Object ec4957d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.704660] Object ec4957e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.713249] Object ec4957f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.721837] Object ec495800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.730425] Object ec495810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.739013] Object ec495820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.747601] Object ec495830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.756190] Object ec495840: 6b 6b 6b 6b 6b 6b 6b 6b 80 15 00 00 00
00 00 00  kkkkkkkk........
[   13.764778] Object ec495850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.773366] Object ec495860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.781954] Object ec495870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.790543] Object ec495880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.799131] Object ec495890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.807719] Object ec4958a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.816307] Object ec4958b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.824896] Object ec4958c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.833484] Object ec4958d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.842072] Object ec4958e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.850660] Object ec4958f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.859248] Object ec495900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.867837] Object ec495910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.876425] Object ec495920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.885014] Object ec495930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.893602] Object ec495940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.902190] Object ec495950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.910778] Object ec495960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.919366] Object ec495970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.927954] Object ec495980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.936543] Object ec495990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.945131] Object ec4959a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.953719] Object ec4959b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.962308] Object ec4959c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.970896] Object ec4959d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.979484] Object ec4959e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.988072] Object ec4959f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   13.996661] Object ec495a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   14.005249] Object ec495a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   14.013837] Object ec495a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   14.022425] Object ec495a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   14.031014] Object ec495a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b a5  kkkkkkkkkkkkkkk.
[   14.039602] Redzone ec495a50: bb bb bb bb
           ....
[   14.047244] Padding ec495af8: 5a 5a 5a 5a 5a 5a 5a 5a
           ZZZZZZZZ
[   14.055222] CPU: 4 PID: 1072 Comm: dmsetup Tainted: G    B
3.15.0-rc8-next-20140606-00004-g73d55cf #113
[   14.065300] [<80015c2c>] (unwind_backtrace) from [<80012378>]
(show_stack+0x20/0x24)
[   14.073015] [<80012378>] (show_stack) from [<804f3ca4>]
(dump_stack+0x78/0xc4)
[   14.080213] [<804f3ca4>] (dump_stack) from [<80109b6c>]
(print_trailer+0x11c/0x13c)
[   14.087849] [<80109b6c>] (print_trailer) from [<80109c2c>]
(check_bytes_and_report+0xa0/0x100)
[   14.096436] [<80109c2c>] (check_bytes_and_report) from [<80109dc0>]
(check_object+0x134/0x234)
[   14.105024] [<80109dc0>] (check_object) from [<80109fd0>]
(__free_slab+0x110/0x124)
[   14.112658] [<80109fd0>] (__free_slab) from [<8010a060>]
(discard_slab+0x7c/0x84)
[   14.120120] [<8010a060>] (discard_slab) from [<8010d308>]
(free_partial+0x94/0x24c)
[   14.127754] [<8010d308>] (free_partial) from [<8010d4ec>]
(__kmem_cache_shutdown+0x2c/0x68)
[   14.136084] [<8010d4ec>] (__kmem_cache_shutdown) from [<800edec4>]
(kmem_cache_destroy+0x40/0xc0)
[   14.144931] [<800edec4>] (kmem_cache_destroy) from [<80224d58>]
(bioset_free+0xd4/0xfc)
[   14.152912] [<80224d58>] (bioset_free) from [<80386a24>]
(__dm_destroy+0x144/0x1c4)
[   14.160545] [<80386a24>] (__dm_destroy) from [<8038797c>]
(dm_destroy+0x1c/0x20)
[   14.167917] [<8038797c>] (dm_destroy) from [<8038d318>]
(dev_remove+0x104/0x110)
[   14.175289] [<8038d318>] (dev_remove) from [<8038c030>]
(ctl_ioctl+0x3ec/0x4a4)
[   14.182575] [<8038c030>] (ctl_ioctl) from [<8038c108>]
(dm_ctl_ioctl+0x20/0x24)
[   14.189865] [<8038c108>] (dm_ctl_ioctl) from [<801209ec>]
(do_vfs_ioctl+0x4e4/0x5a0)
[   14.197585] [<801209ec>] (do_vfs_ioctl) from [<80120b04>]
(SyS_ioctl+0x5c/0x84)
[   14.204873] [<80120b04>] (SyS_ioctl) from [<8000e500>]
(ret_fast_syscall+0x0/0x30)
[   14.212411] FIX bio-1: Restoring 0xec495848-0xec49584f=0x6b
[   14.212411]
[   14.975183] reboot: Res

Re: CONFIG_PREMPT vs. linux-next causes slub_debug=FZPUA errors

[Andrew Morton]

On Tue, 10 Jun 2014 15:38:00 -0700 Doug Anderson <dianders@chromium.org> wrote:

> Hi,
> 
> Recently, both Andrew Bresticker (CCed) and I have noticed errors
> reported by slub_debug when using linuxnext.  I'm still seeing them on
> 20140606.
> 
> 
> NOTES:
> * errors only seem to be present when you've got CONFIG_PREMPT enabled.
> 
> * errors seem to happen most often when rebooting, but I've seen them
> during other times as well.
> 
> * a quick guesstimate is that ~50% of my reboots show this problem.
> 
> * I'm testing on an exynos5250-based board and Andrew on an ARM board
> with a different SoC.
> 
> * I don't see issues on a current fetch of "linux", AKA (5b174fd Merge
> branch 'for-3.16' of git://linux-nfs.org/~bfields/linux)
> 
> 
> I didn't do a full bisect, but I don't seem to see issues on 20140528.
> On 20140530 I see almost constant slub errors (seems much worse than
> 20140606)
> 
> An example error is below:
> 
> 
> [   13.521842] =============================================================================
> [   13.528566] BUG bio-1 (Not tainted): Poison overwritten
> [   13.533765] -----------------------------------------------------------------------------
> [   13.533765]
> [   13.543394] Disabling lock debugging due to kernel taint
> [   13.548685] INFO: 0xec495848-0xec49584f. First byte 0x80 instead of 0x6b
> [   13.555378] INFO: Allocated in mempool_alloc_slab+0x24/0x28 age=28
> cpu=5 pid=220
> [   13.562741]  kmem_cache_alloc+0x90/0x1d0
> [   13.566640]  mempool_alloc_slab+0x24/0x28
> [   13.570631]  mempool_alloc+0x4c/0x144
> [   13.574279]  bio_alloc_bioset+0xa4/0x1b0
> [   13.578182]  __split_and_process_bio+0x33c/0x3f0
> [   13.582778]  dm_request+0x1f0/0x204
> [   13.586248]  generic_make_request+0xa4/0xdc
> [   13.590411]  submit_bio+0x114/0x144

This one is possibly a bug in DM or the BIO layer.

Do you have more examples?  I couldn't locate a report from Andrew.

Re: CONFIG_PREMPT vs. linux-next causes slub_debug=FZPUA errors

[Andrew Bresticker]

> Do you have more examples?  I couldn't locate a report from Andrew.

Here's a dump I've gotten with next-20140610 on a Jetson TK1 board:

[   10.488994] =============================================================================
[   10.497177] BUG bio-1 (Not tainted): Poison overwritten
[   10.502403] -----------------------------------------------------------------------------
[   10.502403]
[   10.512052] Disabling lock debugging due to kernel taint
[   10.517368] INFO: 0xe9b1b078-0xe9b1b07f. First byte 0x20 instead of 0x6b
[   10.524079] INFO: Allocated in mempool_alloc+0x54/0x168 age=4 cpu=0 pid=876
[   10.531050] INFO: Freed in mempool_free+0x34/0x90 age=5 cpu=1 pid=176
[   10.537483] INFO: Slab 0xead0d300 objects=21 used=21 fp=0x  (null)
flags=0x4080
[   10.544788] INFO: Object 0xe9b1b000 @offset=12288 fp=0xe9b19500
[   10.544788]
[   10.552188] Bytes b4 e9b1aff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[   10.560974] Object e9b1b000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.569575] Object e9b1b010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.578183] Object e9b1b020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.586792] Object e9b1b030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.595401] Object e9b1b040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.604011] Object e9b1b050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.612620] Object e9b1b060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.621230] Object e9b1b070: 6b 6b 6b 6b 6b 6b 6b 6b 20 30 01 00 00
00 00 00  kkkkkkkk 0......
[   10.629829] Object e9b1b080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.638437] Object e9b1b090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.647046] Object e9b1b0a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.655656] Object e9b1b0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.664265] Object e9b1b0c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.672875] Object e9b1b0d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.681485] Object e9b1b0e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.690086] Object e9b1b0f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.698687] Object e9b1b100: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.707295] Object e9b1b110: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.715904] Object e9b1b120: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.724515] Object e9b1b130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.733126] Object e9b1b140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.741736] Object e9b1b150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.750349] Object e9b1b160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.758947] Object e9b1b170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.767554] Object e9b1b180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.776164] Object e9b1b190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.784775] Object e9b1b1a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.793384] Object e9b1b1b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.801995] Object e9b1b1c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.810604] Object e9b1b1d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.819203] Object e9b1b1e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.827810] Object e9b1b1f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.836420] Object e9b1b200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.845029] Object e9b1b210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.853639] Object e9b1b220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.862249] Object e9b1b230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.870858] Object e9b1b240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.879457] Object e9b1b250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.888064] Object e9b1b260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.896674] Object e9b1b270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.905284] Object e9b1b280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b  kkkkkkkkkkkkkkkk
[   10.913892] Object e9b1b290: 6b 6b 6b 6b 6b 6b 6b a5
          kkkkkkk.
[   10.921810] Redzone e9b1b298: bb bb bb bb
           ....
[   10.929456] Padding e9b1b2c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[   10.938151] Padding e9b1b2d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[   10.946848] Padding e9b1b2e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[   10.955545] Padding e9b1b2f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[   10.964244] CPU: 2 PID: 204 Comm: jbd2/dm-0-8 Tainted: G    B
  3.15.0-next-20140610-00010-gd4a46d4 #194
[   10.974260] [<c0016240>] (unwind_backtrace) from [<c0011b60>]
(show_stack+0x10/0x14)
[   10.982019] [<c0011b60>] (show_stack) from [<c0634c8c>]
(dump_stack+0x84/0xc4)
[   10.989241] [<c0634c8c>] (dump_stack) from [<c00dcb6c>]
(check_bytes_and_report+0xc4/0x100)
[   10.997602] [<c00dcb6c>] (check_bytes_and_report) from [<c00dcd58>]
(check_object+0x1b0/0x23c)
[   11.006225] [<c00dcd58>] (check_object) from [<c00dec30>]
(alloc_debug_processing+0x78/0x160)
[   11.014763] [<c00dec30>] (alloc_debug_processing) from [<c00df3f4>]
(__slab_alloc.isra.66.constprop.73+0x6dc/0x7b0)
[   11.025207] [<c00df3f4>] (__slab_alloc.isra.66.constprop.73) from
[<c00df748>] (kmem_cache_alloc+0x110/0x140)
[   11.035127] [<c00df748>] (kmem_cache_alloc) from [<c00a5ea8>]
(mempool_alloc+0x54/0x168)
[   11.043232] [<c00a5ea8>] (mempool_alloc) from [<c02103d0>]
(bio_alloc_bioset+0x14c/0x1cc)
[   11.051424] [<c02103d0>] (bio_alloc_bioset) from [<c04227dc>]
(__split_and_process_bio+0x1d8/0x400)
[   11.060477] [<c04227dc>] (__split_and_process_bio) from
[<c0422b20>] (dm_request+0x11c/0x1ec)
[   11.068996] [<c0422b20>] (dm_request) from [<c02146f8>]
(generic_make_request+0xa8/0xd4)
[   11.077094] [<c02146f8>] (generic_make_request) from [<c02147a8>]
(submit_bio+0x84/0x140)
[   11.085281] [<c02147a8>] (submit_bio) from [<c010fc90>]
(_submit_bh+0x1a4/0x250)
[   11.092690] [<c010fc90>] (_submit_bh) from [<c01a5c40>]
(jbd2_journal_commit_transaction+0x958/0x15b8)
[   11.102006] [<c01a5c40>] (jbd2_journal_commit_transaction) from
[<c01a9c60>] (kjournald2+0xc4/0x24c)
[   11.111149] [<c01a9c60>] (kjournald2) from [<c0042034>] (kthread+0xd8/0xf0)
[   11.118108] [<c0042034>] (kthread) from [<c000e7f8>]
(ret_from_fork+0x14/0x3c)