kernel BUG at mm/slub.c:3338! RIP [<ffffffff8117912c>] kfree+0x20c/0x210

mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed

* kernel BUG at mm/slub.c:3338!  RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
@ 2013-10-20 18:05 Sander Eikelenboom
  2013-10-22 15:12 ` Christoph Lameter
  0 siblings, 1 reply; 5+ messages in thread
From: Sander Eikelenboom @ 2013-10-20 18:05 UTC (permalink / raw)
  To: Christoph Lameter, Pekka Enberg; +Cc: linux-kernel

Hi,

With i 3.12.0-rc6 kernel i encounterd this splat:

[19424.151472] ------------[ cut here ]------------
[19424.151496] kernel BUG at mm/slub.c:3338!
[19424.151506] invalid opcode: 0000 [#1] PREEMPT SMP
[19424.151520] Modules linked in:
[19424.151532] CPU: 1 PID: 2824 Comm: glusterfs Not tainted 3.12.0-rc6-20131020 #1
[19424.151549] task: ffff88001d140000 ti: ffff88001df4c000 task.ti: ffff88001df4c000
[19424.151561] RIP: e030:[<ffffffff8117912c>]  [<ffffffff8117912c>] kfree+0x20c/0x210
[19424.151580] RSP: e02b:ffff88001df4dde8  EFLAGS: 00010246
[19424.151589] RAX: 0040000000000000 RBX: ffff88001df4de68 RCX: 0000000000000000
[19424.151600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88001df4de68
[19424.151611] RBP: ffff88001df4de28 R08: ffff88001e0f7930 R09: 0000000000000058
[19424.151622] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88001cf4de68
[19424.151633] R13: ffffea000077d340 R14: ffff88001e317a00 R15: 0000000000000058
[19424.151649] FS:  00007f549af8c700(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000
[19424.151661] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[19424.151671] CR2: 00007f549f026000 CR3: 000000001c891000 CR4: 0000000000000660
[19424.151683] Stack:
[19424.151689]  0000000000020050 0000000000000000 0000000000000000 0000000000000000
[19424.151708]  ffff88001cf4de68 0000000000000002 ffff88001e317a00 0000000000000058
[19424.151726]  ffff88001df4df18 ffffffff81186b01 ffff88001df4ded8 ffffffff810f0edf
[19424.151744] Call Trace:
[19424.151756]  [<ffffffff81186b01>] do_readv_writev+0x111/0x250
[19424.151771]  [<ffffffff810f0edf>] ? lock_release_non_nested+0xaf/0x320
[19424.151784]  [<ffffffff810f127f>] ? lock_release+0x12f/0x250
[19424.151796]  [<ffffffff81186ccd>] vfs_readv+0x2d/0x50
[19424.151807]  [<ffffffff81186d44>] SyS_readv+0x54/0xd0
[19424.151820]  [<ffffffff81a560b9>] system_call_fastpath+0x16/0x1b
[19424.151830] Code: 89 4d 8b 6d 30 e9 7e fe ff ff 48 8b 4d c0 48 89 da 4c 89 ee 4c 89 e7 e8 54 e4 8c 00 e9 69 ff ff ff e8 19 a2 8d 00 e9 2f ff ff ff <0f> 0b 66 90 55 48 89 e5 48 83 ec 30 48 89 5d d8 4c 89 75 f0 48
[19424.151954] RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
[19424.151970]  RSP <ffff88001df4dde8>
[19424.151983] ---[ end trace 225a70bf87bd3a82 ]---


--
Sander


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: kernel BUG at mm/slub.c:3338!  RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
  2013-10-20 18:05 kernel BUG at mm/slub.c:3338! RIP [<ffffffff8117912c>] kfree+0x20c/0x210 Sander Eikelenboom
@ 2013-10-22 15:12 ` Christoph Lameter
  2013-10-22 15:29   ` Sander Eikelenboom
  0 siblings, 1 reply; 5+ messages in thread
From: Christoph Lameter @ 2013-10-22 15:12 UTC (permalink / raw)
  To: Sander Eikelenboom; +Cc: Pekka Enberg, linux-kernel

On Sun, 20 Oct 2013, Sander Eikelenboom wrote:

> With i 3.12.0-rc6 kernel i encounterd this splat:

Likely some slab object got corrupted.

Please enable debugging by either specifying slub_debug on the kernel
command line or enabling CONFIG_SLUB_DEBUG_ON


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: kernel BUG at mm/slub.c:3338!  RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
  2013-10-22 15:12 ` Christoph Lameter
@ 2013-10-22 15:29   ` Sander Eikelenboom
  2013-11-09 18:18     ` Sander Eikelenboom
  0 siblings, 1 reply; 5+ messages in thread
From: Sander Eikelenboom @ 2013-10-22 15:29 UTC (permalink / raw)
  To: Christoph Lameter; +Cc: Pekka Enberg, linux-kernel


Tuesday, October 22, 2013, 5:12:35 PM, you wrote:

> On Sun, 20 Oct 2013, Sander Eikelenboom wrote:

>> With i 3.12.0-rc6 kernel i encounterd this splat:

> Likely some slab object got corrupted.

> Please enable debugging by either specifying slub_debug on the kernel
> command line or enabling CONFIG_SLUB_DEBUG_ON


Well since it seems a "one timer" so i'm not very sure if and when i would encounter it again.
It seems "slub_debug" on the commandline would do the same, so i will run my VM with that for some time, see if triggers again.

--
Sander


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: kernel BUG at mm/slub.c:3338!  RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
  2013-10-22 15:29   ` Sander Eikelenboom
@ 2013-11-09 18:18     ` Sander Eikelenboom
  2013-11-11 14:57       ` Christoph Lameter
  0 siblings, 1 reply; 5+ messages in thread
From: Sander Eikelenboom @ 2013-11-09 18:18 UTC (permalink / raw)
  To: Christoph Lameter, Pekka Enberg; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

Hello Sander,

Tuesday, October 22, 2013, 5:29:05 PM, you wrote:


> Tuesday, October 22, 2013, 5:12:35 PM, you wrote:

>> On Sun, 20 Oct 2013, Sander Eikelenboom wrote:

>>> With i 3.12.0-rc6 kernel i encounterd this splat:

>> Likely some slab object got corrupted.

>> Please enable debugging by either specifying slub_debug on the kernel
>> command line or enabling CONFIG_SLUB_DEBUG_ON


> Well since it seems a "one timer" so i'm not very sure if and when i would encounter it again.
> It seems "slub_debug" on the commandline would do the same, so i will run my VM with that for some time, see if triggers again.

> --
> Sander

Something triggered another oops on this VM, although it looks a bit different.

This VM got quite some I/O and network load, it got a lot of large files transferred to Samba and the files are stored on a glusterfs volume (by tcp/ip).


[34856.970314] =============================================================================
[34856.970332] BUG kmalloc-2048 (Not tainted): Poison overwritten
[34856.970339] -----------------------------------------------------------------------------
[34856.970339] 
[34856.970350] Disabling lock debugging due to kernel taint
[34856.970357] INFO: 0xffff88001b36099b-0xffff88001b36099b. First byte 0x6a instead of 0x6b
[34856.970371] INFO: Allocated in __alloc_skb+0x82/0x290 age=2 cpu=0 pid=2818
[34856.970381]  __slab_alloc.isra.57+0x61e/0x6d6
[34856.970389]  __kmalloc_node_track_caller+0xc9/0x1b0
[34856.970396]  __kmalloc_reserve.isra.49+0x37/0xa0
[34856.970402]  __alloc_skb+0x82/0x290
[34856.970408]  sk_stream_alloc_skb+0x3f/0x120
[34856.970414]  tcp_sendmsg+0x615/0xe20
[34856.970419]  inet_sendmsg+0xa6/0x100
[34856.970426]  sock_aio_write+0xc2/0xe0
[34856.970433]  do_sync_readv_writev+0x4e/0x80
[34856.970439]  do_readv_writev+0xe8/0x270
[34856.970444]  vfs_writev+0x30/0x60
[34856.970449]  SyS_writev+0x54/0xd0
[34856.970454]  system_call_fastpath+0x16/0x1b
[34856.970461] INFO: Freed in skb_free_head+0x64/0x70 age=2 cpu=0 pid=0
[34856.970468]  __slab_free+0x30/0x561
[34856.970474]  kfree+0x1fd/0x210
[34856.970479]  skb_free_head+0x64/0x70
[34856.970484]  skb_release_data+0xf3/0x100
[34856.970492]  skb_release_all+0x23/0x30
[34856.970497]  __kfree_skb+0x11/0xa0
[34856.970503]  tcp_ack+0x64a/0xf20
[34856.970509]  tcp_rcv_established+0xe2/0x5c0
[34856.970515]  tcp_v4_do_rcv+0x14e/0x4a0
[34856.970520]  tcp_v4_rcv+0x8f2/0x9c0
[34856.970527]  ip_local_deliver_finish+0xd8/0x1d0
[34856.970533]  ip_local_deliver+0x34/0x90
[34856.970539]  ip_rcv_finish+0x150/0x660
[34856.970544]  ip_rcv+0x22b/0x370
[34856.970551]  __netif_receive_skb_core+0x6e2/0x8b0
[34856.970557]  __netif_receive_skb+0x1c/0x70
[34856.970563] INFO: Slab 0xffffea00006cd800 objects=13 used=13 fp=0x          (null) flags=0x40000000004080
[34856.970572] INFO: Object 0xffff88001b360948 @offset=2376 fp=0xffff88001b3637b0
[34856.970572]

The rest is in the attached dmesg.

attached: dmesg-oops-3.12.0-20131104.txt
--
Sander

[-- Attachment #2: dmesg-oops-3.12.0-20131104.txt --]
[-- Type: text/plain, Size: 47098 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.12.0-20131104 (root@serveerstertje) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP PREEMPT Mon Nov 4 09:34:18 CET 2013
[    0.000000] Command line: root=/dev/xvda1 ro slub_debug
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] MPS support code is not built-in.
[    0.000000] Using acpi=off or acpi=noirq or pci=noacpi may have problem
[    0.000000] DMI not present or invalid.
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x1fe00000-0x1fffffff]
[    0.000000]  [mem 0x1fe00000-0x1fffffff] page 4k
[    0.000000] BRK [0x02d72000, 0x02d72fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x1c000000-0x1fdfffff]
[    0.000000]  [mem 0x1c000000-0x1fdfffff] page 4k
[    0.000000] BRK [0x02d73000, 0x02d73fff] PGTABLE
[    0.000000] BRK [0x02d74000, 0x02d74fff] PGTABLE
[    0.000000] BRK [0x02d75000, 0x02d75fff] PGTABLE
[    0.000000] BRK [0x02d76000, 0x02d76fff] PGTABLE
[    0.000000] BRK [0x02d77000, 0x02d77fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x00100000-0x1bffffff]
[    0.000000]  [mem 0x00100000-0x1bffffff] page 4k
[    0.000000] RAMDISK: [mem 0x03188000-0x03f9cfff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1fffffff]
[    0.000000]   NODE_DATA [mem 0x1ff1d000-0x1ff27fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000] On node 0 totalpages: 130975
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1984 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] e820: [mem 0x20000000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.4-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88001fc00000 s83200 r8192 d23296 u1048576
[    0.000000] pcpu-alloc: s83200 r8192 d23296 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128906
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/xvda1 ro slub_debug
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 468620K/523900K available (10729K kernel code, 978K rwdata, 4200K rodata, 1068K init, 9580K bss, 55280K reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Additional per-CPU info printed with stalls.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
[    0.000000] NR_IRQS:4352 nr_irqs:288 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 5855 kB
[    0.000000]  per task-struct memory footprint: 1920 bytes
[    0.000000] kmemleak: Kernel memory leak detector disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3200.180 MHz processor
[    0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 6402.04 BogoMIPS (lpj=10667266)
[    0.003333] pid_max: default: 32768 minimum: 301
[    0.003333] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.003333] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.003333] Mount-cache hash table entries: 256
[    0.003333] Initializing cgroup subsys freezer
[    0.003333] Initializing cgroup subsys blkio
[    0.003333] tseg: 0000000000
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
[    0.003333] Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64
[    0.003333] tlb_flushall_shift: 4
[    0.044828] cpu 0 spinlock event irq 17
[    0.045830] Performance Events: 
[    0.045848] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.045855] no hardware sampling interrupt available.
[    0.045880] Broken PMU hardware detected, using software events only.
[    0.045882] Failed to access perfctr msr (MSR c0010004 is 0)
[    0.063598] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.070288] installing Xen timer for CPU 1
[    0.070377] cpu 1 spinlock event irq 24
[    0.070456] SMP alternatives: lockdep: fixing up alternatives
[    0.070469] SMP alternatives: switching to SMP code
[    0.112838] Brought up 2 CPUs
[    0.113810] devtmpfs: initialized
[    0.115040] xor: measuring software checksum speed
[    0.144411]    prefetch64-sse:  5412.000 MB/sec
[    0.178307]    generic_sse:  5330.400 MB/sec
[    0.178318] xor: using function: prefetch64-sse (5412.000 MB/sec)
[    0.178361] xen:grant_table: Grant tables using version 2 layout
[    0.178420] Grant table initialized
[    0.178917] NET: Registered protocol family 16
[    0.183776] PCI: setting up Xen PCI frontend stub
[    0.183800] PCI: pci_cache_line_size set to 64 bytes
[    0.320441] bio: create slab <bio-0> at 0
[    0.376754] raid6: sse2x1    3860 MB/s
[    0.433368] raid6: sse2x2    5362 MB/s
[    0.490003] raid6: sse2x4    5857 MB/s
[    0.490005] raid6: using algorithm sse2x4 (5857 MB/s)
[    0.490006] raid6: using intx1 recovery algorithm
[    0.490306] ACPI: Interpreter disabled.
[    0.490306] xen:balloon: Initialising balloon driver
[    0.493520] xen_balloon: Initialising balloon driver
[    0.496781] vgaarb: loaded
[    0.497942] SCSI subsystem initialized
[    0.497942] libata version 3.00 loaded.
[    0.500538] usbcore: registered new interface driver usbfs
[    0.500538] usbcore: registered new interface driver hub
[    0.500538] usbcore: registered new device driver usb
[    0.500538] Linux video capture interface: v2.00
[    0.503529] pps_core: LinuxPPS API ver. 1 registered
[    0.503542] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.503612] PTP clock support registered
[    0.503625] Advanced Linux Sound Architecture Driver Initialized.
[    0.503648] PCI: System does not support PCI
[    0.503650] PCI: System does not support PCI
[    0.506907] Bluetooth: Core ver 2.16
[    0.506997] NET: Registered protocol family 31
[    0.506997] Bluetooth: HCI device and connection manager initialized
[    0.506997] Bluetooth: HCI socket layer initialized
[    0.506997] Bluetooth: L2CAP socket layer initialized
[    0.506997] Bluetooth: SCO socket layer initialized
[    0.507279] Switched to clocksource xen
[    0.507396] FS-Cache: Loaded
[    0.507636] pnp: PnP ACPI: disabled
[    0.539117] NET: Registered protocol family 2
[    0.539897] TCP established hash table entries: 4096 (order: 4, 65536 bytes)
[    0.540019] TCP bind hash table entries: 4096 (order: 6, 262144 bytes)
[    0.540277] TCP: Hash tables configured (established 4096 bind 4096)
[    0.540362] TCP: reno registered
[    0.540401] UDP hash table entries: 256 (order: 3, 40960 bytes)
[    0.540458] UDP-Lite hash table entries: 256 (order: 3, 40960 bytes)
[    0.540871] NET: Registered protocol family 1
[    0.540925] PCI: CLS 0 bytes, default 64
[    0.541330] Trying to unpack rootfs image as initramfs...
[    0.604671] Freeing initrd memory: 14420K (ffff880003188000 - ffff880003f9d000)
[    0.810210] DMA-API: preallocated 65536 debug entries
[    0.810232] DMA-API: debugging enabled by kernel config
[    0.810899] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.813974] Scanning for low memory corruption every 60 seconds
[    0.817883] sha1_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817897] sha256_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817902] sha512_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817908] AVX or AES-NI instructions are not detected.
[    0.817912] AVX instructions are not detected.
[    0.817916] AVX instructions are not detected.
[    0.820512] audit: initializing netlink socket (disabled)
[    0.820601] type=2000 audit(1383778218.007:1): initialized
[    0.822275] bounce pool size: 64 pages
[    0.822293] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.878200] VFS: Disk quotas dquot_6.5.2
[    0.878818] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.897415] NTFS driver 2.1.30 [Flags: R/W].
[    0.900483] fuse init (API version 7.22)
[    0.905703] bio: create slab <bio-1> at 1
[    0.910323] Btrfs loaded
[    0.914633] GFS2 installed
[    0.916780] ceph: loaded (mds proto 32)
[    0.916827] msgmni has been set to 943
[    0.920847] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    0.920891] io scheduler noop registered
[    0.920896] io scheduler deadline registered
[    0.921766] io scheduler cfq registered (default)
[    0.923034] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[    0.923046] crc32: self tests passed, processed 225944 bytes in 109410 nsec
[    0.923163] crc32c: CRC_LE_BITS = 64
[    0.923168] crc32c: self tests passed, processed 225944 bytes in 54579 nsec
[    0.924335] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.925132] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.925447] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.925456] acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
[    0.926534] usbcore: registered new interface driver udlfb
[    0.927691] xen:xen_evtchn: Event-channel device installed
[    0.932426] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.936695] Linux agpgart interface v0.103
[    0.938516] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[    0.938518] Hangcheck: Using getrawmonotonic().
[    0.938784] [drm] Initialized drm 1.1.0 20060810
[    0.939112] [drm] radeon kernel modesetting enabled.
[    0.957684] brd: module loaded
[    0.966412] loop: module loaded
[    0.975940] tun: Universal TUN/TAP device driver, 1.6
[    0.975955] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.976391] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    0.976403] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.976818] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[    0.976831] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
[    0.977343] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k
[    0.977353] igb: Copyright (c) 2007-2013 Intel Corporation.
[    0.977696] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.2-k
[    0.977709] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.978502] xen_netfront: Initialising Xen virtual ethernet driver
[    0.983343] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.983353] ehci-pci: EHCI PCI platform driver
[    0.984249] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.984258] ohci-pci: OHCI PCI platform driver
[    0.984564] uhci_hcd: USB Universal Host Controller Interface driver
[    0.986205] usbcore: registered new interface driver usblp
[    0.986958] usbcore: registered new interface driver usb-storage
[    0.987932] usbcore: registered new interface driver usbserial
[    0.988274] usbcore: registered new interface driver cp210x
[    0.988703] usbserial: USB Serial support registered for cp210x
[    0.988952] usbcore: registered new interface driver cypress_m8
[    0.989238] usbserial: USB Serial support registered for DeLorme Earthmate USB
[    0.989573] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[    0.989870] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[    0.990111] usbcore: registered new interface driver mos7720
[    0.990417] usbserial: USB Serial support registered for Moschip 2 port adapter
[    0.990654] usbcore: registered new interface driver mos7840
[    0.993506] usbserial: USB Serial support registered for Moschip 7840/7820 USB Serial Driver
[    0.994349] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.009583] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.002564] i8042: No controller found
[    2.004200] mousedev: PS/2 mouse device common for all mice
[    2.066703] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    2.068031] rtc_cmos: probe of rtc_cmos failed with error -38
[    2.071810] lirc_dev: IR Remote Control driver registered, major 248 
[    2.071854] IR NEC protocol handler initialized
[    2.071860] IR RC5(x) protocol handler initialized
[    2.071864] IR RC6 protocol handler initialized
[    2.071868] IR JVC protocol handler initialized
[    2.071873] IR Sony protocol handler initialized
[    2.071877] IR RC5 (streamzap) protocol handler initialized
[    2.071882] IR SANYO protocol handler initialized
[    2.071886] IR MCE Keyboard/mouse protocol handler initialized
[    2.071891] IR LIRC bridge handler initialized
[    2.071896] cx25821: driver version 0.0.106 loaded
[    2.072860] usbcore: registered new interface driver pvrusb2
[    2.072890] pvrusb2: V4L in-tree version:Hauppauge WinTV-PVR-USB2 MPEG2 Encoder/Tuner
[    2.072897] pvrusb2: Debug mask is 31 (0x1f)
[    2.073903] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05
[    2.074490] xen_wdt: Xen WatchDog Timer Driver v0.01
[    2.075432] xen_wdt: initialized (timeout=60s, nowayout=0)
[    2.079512] device-mapper: ioctl: 4.26.0-ioctl (2013-08-15) initialised: dm-devel@redhat.com
[    2.082630] device-mapper: cache-policy-mq: version 1.0.0 loaded
[    2.082660] device-mapper: cache cleaner: version 1.0.0 loaded
[    2.082666] Bluetooth: Virtual HCI driver ver 1.3
[    2.083086] Bluetooth: HCI UART driver ver 2.2
[    2.083098] Bluetooth: HCI H4 protocol initialized
[    2.083103] Bluetooth: HCI BCSP protocol initialized
[    2.083107] Bluetooth: HCILL protocol initialized
[    2.083111] Bluetooth: HCIATH3K protocol initialized
[    2.083115] Bluetooth: HCI Three-wire UART (H5) protocol initialized
[    2.083439] usbcore: registered new interface driver bcm203x
[    2.084033] usbcore: registered new interface driver bpa10x
[    2.084374] usbcore: registered new interface driver bfusb
[    2.084716] usbcore: registered new interface driver btusb
[    2.085210] usbcore: registered new interface driver ath3k
[    2.085814] hidraw: raw HID events driver (C) Jiri Kosina
[    2.088974] usbcore: registered new interface driver usbhid
[    2.088983] usbhid: USB HID core driver
[    2.097677] usbcore: registered new interface driver snd-usb-audio
[    2.097975] usbcore: registered new interface driver snd-ua101
[    2.098252] usbcore: registered new interface driver snd-usb-usx2y
[    2.098526] usbcore: registered new interface driver snd-usb-caiaq
[    2.098797] usbcore: registered new interface driver snd-usb-6fire
[    2.098882] Netfilter messages via NETLINK v0.30.
[    2.098913] nfnl_acct: registering with nfnetlink.
[    2.099088] nf_conntrack version 0.5.0 (3773 buckets, 15092 max)
[    2.100344] ctnetlink v0.93: registering with nfnetlink.
[    2.101476] xt_time: kernel timezone is -0000
[    2.101498]  xvda: xvda1 xvda2
[    2.101509] ip_set: protocol 6
[    2.101550] IPVS: Registered protocols ()
[    2.102027] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[    2.102231] IPVS: Creating netns size=1904 id=0
[    2.102595] IPVS: ipvs loaded.
[    2.103069] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.103473] TCP: cubic registered
[    2.103480] NET: Registered protocol family 17
[    2.104492] Bridge firewalling registered
[    2.104524] Ebtables v2.0 registered
[    2.105453] Bluetooth: RFCOMM TTY layer initialized
[    2.105561] Bluetooth: RFCOMM socket layer initialized
[    2.105567] Bluetooth: RFCOMM ver 1.11
[    2.105571] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    2.105576] Bluetooth: BNEP filters: protocol multicast
[    2.105598] Bluetooth: BNEP socket layer initialized
[    2.105603] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    2.105626] Bluetooth: HIDP socket layer initialized
[    2.105668] Key type ceph registered
[    2.107631] libceph: loaded (mon/osd proto 15/24)
[    2.109119] registered taskstats version 1
[    2.109623] console [netcon0] enabled
[    2.109634] netconsole: network logging started
[    2.109640] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    2.109837] ALSA device list:
[    2.109844]   No soundcards found.
[    2.111258] Freeing unused kernel memory: 1068K (ffffffff822f6000 - ffffffff82401000)
[    2.111275] Write protecting the kernel read-only data: 18432k
[    2.120228] Freeing unused kernel memory: 1548K (ffff880001a7d000 - ffff880001c00000)
[    2.121537] Freeing unused kernel memory: 1944K (ffff88000201a000 - ffff880002200000)
[    2.174907] udevd[1207]: starting version 175
[    4.324325] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[    8.933317] udevd[1429]: starting version 175
[   15.570258] Adding 523260k swap on /dev/xvda2.  Priority:-1 extents:1 across:523260k SS
[   15.637450] EXT4-fs (xvda1): re-mounted. Opts: (null)
[   15.896610] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro
[34856.970314] =============================================================================
[34856.970332] BUG kmalloc-2048 (Not tainted): Poison overwritten
[34856.970339] -----------------------------------------------------------------------------
[34856.970339] 
[34856.970350] Disabling lock debugging due to kernel taint
[34856.970357] INFO: 0xffff88001b36099b-0xffff88001b36099b. First byte 0x6a instead of 0x6b
[34856.970371] INFO: Allocated in __alloc_skb+0x82/0x290 age=2 cpu=0 pid=2818
[34856.970381] 	__slab_alloc.isra.57+0x61e/0x6d6
[34856.970389] 	__kmalloc_node_track_caller+0xc9/0x1b0
[34856.970396] 	__kmalloc_reserve.isra.49+0x37/0xa0
[34856.970402] 	__alloc_skb+0x82/0x290
[34856.970408] 	sk_stream_alloc_skb+0x3f/0x120
[34856.970414] 	tcp_sendmsg+0x615/0xe20
[34856.970419] 	inet_sendmsg+0xa6/0x100
[34856.970426] 	sock_aio_write+0xc2/0xe0
[34856.970433] 	do_sync_readv_writev+0x4e/0x80
[34856.970439] 	do_readv_writev+0xe8/0x270
[34856.970444] 	vfs_writev+0x30/0x60
[34856.970449] 	SyS_writev+0x54/0xd0
[34856.970454] 	system_call_fastpath+0x16/0x1b
[34856.970461] INFO: Freed in skb_free_head+0x64/0x70 age=2 cpu=0 pid=0
[34856.970468] 	__slab_free+0x30/0x561
[34856.970474] 	kfree+0x1fd/0x210
[34856.970479] 	skb_free_head+0x64/0x70
[34856.970484] 	skb_release_data+0xf3/0x100
[34856.970492] 	skb_release_all+0x23/0x30
[34856.970497] 	__kfree_skb+0x11/0xa0
[34856.970503] 	tcp_ack+0x64a/0xf20
[34856.970509] 	tcp_rcv_established+0xe2/0x5c0
[34856.970515] 	tcp_v4_do_rcv+0x14e/0x4a0
[34856.970520] 	tcp_v4_rcv+0x8f2/0x9c0
[34856.970527] 	ip_local_deliver_finish+0xd8/0x1d0
[34856.970533] 	ip_local_deliver+0x34/0x90
[34856.970539] 	ip_rcv_finish+0x150/0x660
[34856.970544] 	ip_rcv+0x22b/0x370
[34856.970551] 	__netif_receive_skb_core+0x6e2/0x8b0
[34856.970557] 	__netif_receive_skb+0x1c/0x70
[34856.970563] INFO: Slab 0xffffea00006cd800 objects=13 used=13 fp=0x          (null) flags=0x40000000004080
[34856.970572] INFO: Object 0xffff88001b360948 @offset=2376 fp=0xffff88001b3637b0
[34856.970572] 
[34856.973176] Bytes b4 ffff88001b360938: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[34856.973176] Object ffff88001b360948: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360958: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360968: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360978: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360988: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360998: 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkjkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361008: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361018: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361028: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361038: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361048: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361058: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361068: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361078: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361088: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361098: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361108: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361118: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361128: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361138: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
[34856.973176] Redzone ffff88001b361148: bb bb bb bb bb bb bb bb                          ........
[34856.973176] Padding ffff88001b361288: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[34856.973176] CPU: 1 PID: 2818 Comm: glusterfs Tainted: G    B        3.12.0-20131104 #1
[34856.973176]  ffff88001b360948 ffff88001ad218e8 ffffffff81a6a8ba ffff88001eb36b40
[34856.973176]  ffff88001e802d80 ffff88001ad21918 ffffffff8117af19 ffff88001b36099c
[34856.973176]  ffff88001e802d80 000000000000006b ffff88001b360948 ffff88001ad21968
[34856.973176] Call Trace:
[34856.973176]  [<ffffffff81a6a8ba>] dump_stack+0x4f/0x84
[34856.973176]  [<ffffffff8117af19>] print_trailer+0xf9/0x160
[34856.973176]  [<ffffffff8117b0b2>] check_bytes_and_report+0xe2/0x120
[34856.973176]  [<ffffffff8117b2b7>] check_object+0x1c7/0x240
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff81a66cc2>] alloc_debug_processing+0x153/0x168
[34856.973176]  [<ffffffff81a672f5>] __slab_alloc.isra.57+0x61e/0x6d6
[34856.973176]  [<ffffffff810ef7bb>] ? trace_hardirqs_on_caller+0xfb/0x240
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff810ef624>] ? mark_held_locks+0x94/0x130
[34856.973176]  [<ffffffff8117ef49>] __kmalloc_node_track_caller+0xc9/0x1b0
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff818ff9d7>] __kmalloc_reserve.isra.49+0x37/0xa0
[34856.973176]  [<ffffffff818ffa92>] ? __alloc_skb+0x52/0x290
[34856.973176]  [<ffffffff818ffac2>] __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff810f372a>] ? lock_release+0x12a/0x250
[34856.973176]  [<ffffffff819af1af>] sk_stream_alloc_skb+0x3f/0x120
[34856.973176]  [<ffffffff819aff85>] tcp_sendmsg+0x615/0xe20
[34856.973176]  [<ffffffff819d7e16>] inet_sendmsg+0xa6/0x100
[34856.973176]  [<ffffffff819d7d70>] ? inet_autobind+0x60/0x60
[34856.973176]  [<ffffffff818f57b2>] sock_aio_write+0xc2/0xe0
[34856.973176]  [<ffffffff811899be>] do_sync_readv_writev+0x4e/0x80
[34856.973176]  [<ffffffff8118b0e8>] do_readv_writev+0xe8/0x270
[34856.973176]  [<ffffffff810f372a>] ? lock_release+0x12a/0x250
[34856.973176]  [<ffffffff8118b2a0>] vfs_writev+0x30/0x60
[34856.973176]  [<ffffffff8118b444>] SyS_writev+0x54/0xd0
[34856.973176]  [<ffffffff81a76379>] system_call_fastpath+0x16/0x1b
[34856.973176] FIX kmalloc-2048: Restoring 0xffff88001b36099b-0xffff88001b36099b=0x6b
[34856.973176] 
[34856.973176] FIX kmalloc-2048: Marking all objects used
[81155.636434] =============================================================================
[81155.636459] BUG skbuff_head_cache (Tainted: G    B       ): Poison overwritten
[81155.636469] -----------------------------------------------------------------------------
[81155.636469] 
[81155.636481] INFO: 0xffff88001e1e7b1b-0xffff88001e1e7b1b. First byte 0x6a instead of 0x6b
[81155.636495] INFO: Allocated in build_skb+0x2f/0x1c0 age=3 cpu=0 pid=0
[81155.636506] 	__slab_alloc.isra.57+0x61e/0x6d6
[81155.636513] 	kmem_cache_alloc+0x10a/0x140
[81155.636519] 	build_skb+0x2f/0x1c0
[81155.636524] 	__netdev_alloc_skb+0x91/0xf0
[81155.636532] 	xennet_alloc_rx_buffers+0xb6/0x340
[81155.636538] 	xennet_poll+0x6c7/0xdb0
[81155.636545] 	net_rx_action+0x159/0x250
[81155.636551] 	__do_softirq+0xea/0x1e0
[81155.636557] 	call_softirq+0x1c/0x30
[81155.636564] 	do_softirq+0x95/0x100
[81155.636569] 	irq_exit+0xa2/0xd0
[81155.636576] 	xen_evtchn_do_upcall+0x2f/0x40
[81155.636582] 	xen_do_hypervisor_callback+0x1e/0x30
[81155.636595] INFO: Freed in __kfree_skb+0x3a/0xa0 age=2 cpu=0 pid=0
[81155.636603] 	__slab_free+0x30/0x561
[81155.636608] 	kmem_cache_free+0x212/0x230
[81155.636614] 	__kfree_skb+0x3a/0xa0
[81155.636620] 	tcp_rcv_established+0x33b/0x5c0
[81155.636627] 	tcp_v4_do_rcv+0x14e/0x4a0
[81155.636633] 	tcp_v4_rcv+0x8f2/0x9c0
[81155.636639] 	ip_local_deliver_finish+0xd8/0x1d0
[81155.636647] 	ip_local_deliver+0x34/0x90
[81155.636653] 	ip_rcv_finish+0x150/0x660
[81155.636659] 	ip_rcv+0x22b/0x370
[81155.636664] 	__netif_receive_skb_core+0x6e2/0x8b0
[81155.636670] 	__netif_receive_skb+0x1c/0x70
[81155.636676] 	netif_receive_skb+0x1e/0x100
[81155.636681] 	xennet_poll+0x8a4/0xdb0
[81155.636687] 	net_rx_action+0x159/0x250
[81155.636692] 	__do_softirq+0xea/0x1e0
[81155.636698] INFO: Slab 0xffffea0000787980 objects=14 used=14 fp=0x          (null) flags=0x40000000004080
[81155.636707] INFO: Object 0xffff88001e1e7b00 @offset=6912 fp=0xffff88001e1e6b40
[81155.636707] 
[81155.636719] Bytes b4 ffff88001e1e7af0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[81155.636728] Object ffff88001e1e7b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636738] Object ffff88001e1e7b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b  kkkkkkkkkkkjkkkk
[81155.636748] Object ffff88001e1e7b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636757] Object ffff88001e1e7b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636766] Object ffff88001e1e7b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636775] Object ffff88001e1e7b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636785] Object ffff88001e1e7b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636794] Object ffff88001e1e7b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636803] Object ffff88001e1e7b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636812] Object ffff88001e1e7b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636821] Object ffff88001e1e7ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636830] Object ffff88001e1e7bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636839] Object ffff88001e1e7bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636848] Object ffff88001e1e7bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636857] Object ffff88001e1e7be0: 6b 6b 6b 6b 6b 6b 6b a5                          kkkkkkk.
[81155.636869] Redzone ffff88001e1e7be8: bb bb bb bb bb bb bb bb                          ........
[81155.636881] Padding ffff88001e1e7d28: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[81155.636890] Padding ffff88001e1e7d38: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[81155.636912] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G    B        3.12.0-20131104 #1
[81155.636914]  ffff88001e1e7b00 ffff88001fc039d0 ffffffff81a6a8ba ffffffff822134e0
[81155.636920]  ffff88001e93e6c0 ffff88001fc03a00 ffffffff8117af19 ffff88001e1e7b1c
[81155.636924]  ffff88001e93e6c0 000000000000006b ffff88001e1e7b00 ffff88001fc03a50
[81155.636928] Call Trace:
[81155.636930]  <IRQ>  [<ffffffff81a6a8ba>] dump_stack+0x4f/0x84
[81155.636936]  [<ffffffff8117af19>] print_trailer+0xf9/0x160
[81155.636940]  [<ffffffff8117b0b2>] check_bytes_and_report+0xe2/0x120
[81155.636943]  [<ffffffff8117b2b7>] check_object+0x1c7/0x240
[81155.636947]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636950]  [<ffffffff81a66cc2>] alloc_debug_processing+0x153/0x168
[81155.636954]  [<ffffffff81a672f5>] __slab_alloc.isra.57+0x61e/0x6d6
[81155.636957]  [<ffffffff819a058d>] ? ip_local_deliver_finish+0x17d/0x1d0
[81155.636961]  [<ffffffff819a044c>] ? ip_local_deliver_finish+0x3c/0x1d0
[81155.636964]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636968]  [<ffffffff819a0754>] ? ip_local_deliver+0x34/0x90
[81155.636971]  [<ffffffff8199ff00>] ? ip_rcv_finish+0x150/0x660
[81155.636975]  [<ffffffff8117ce7a>] kmem_cache_alloc+0x10a/0x140
[81155.636978]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636981]  [<ffffffff818fe78f>] build_skb+0x2f/0x1c0
[81155.636985]  [<ffffffff819004b1>] __netdev_alloc_skb+0x91/0xf0
[81155.636988]  [<ffffffff816d50a6>] xennet_alloc_rx_buffers+0xb6/0x340
[81155.636991]  [<ffffffff8190d1be>] ? netif_receive_skb+0x1e/0x100
[81155.636995]  [<ffffffff816d6fb7>] xennet_poll+0x6c7/0xdb0
[81155.636999]  [<ffffffff8190d669>] net_rx_action+0x159/0x250
[81155.637003]  [<ffffffff8109fa3a>] __do_softirq+0xea/0x1e0
[81155.637006]  [<ffffffff81a777bc>] call_softirq+0x1c/0x30
[81155.637009]  [<ffffffff81010dc5>] do_softirq+0x95/0x100
[81155.637013]  [<ffffffff8109fce2>] irq_exit+0xa2/0xd0
[81155.637016]  [<ffffffff814bb2af>] xen_evtchn_do_upcall+0x2f/0x40
[81155.637019]  [<ffffffff81a7781e>] xen_do_hypervisor_callback+0x1e/0x30
[81155.637021]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[81155.637027]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[81155.637031]  [<ffffffff81008da0>] ? xen_safe_halt+0x10/0x20
[81155.637034]  [<ffffffff810185cd>] ? default_idle+0x1d/0x30
[81155.637038]  [<ffffffff81018f0e>] ? arch_cpu_idle+0x2e/0x40
[81155.637042]  [<ffffffff810db410>] ? cpu_startup_entry+0x80/0x1d0
[81155.637046]  [<ffffffff81a5ebf1>] ? rest_init+0xc1/0xd0
[81155.637049]  [<ffffffff81a5eb30>] ? csum_partial_copy_generic+0x170/0x170
[81155.637054]  [<ffffffff8230bee6>] ? start_kernel+0x3d3/0x3e0
[81155.637058]  [<ffffffff8230b912>] ? repair_env_string+0x5e/0x5e
[81155.637061]  [<ffffffff810d89f7>] ? __add_preferred_console.constprop.18+0x87/0xb0
[81155.637065]  [<ffffffff8230b5f8>] ? x86_64_start_reservations+0x2a/0x2c
[81155.637069]  [<ffffffff8230ecd9>] ? xen_start_kernel+0x563/0x56d
[81155.637072] FIX skbuff_head_cache: Restoring 0xffff88001e1e7b1b-0xffff88001e1e7b1b=0x6b
[81155.637072] 
[81155.637084] FIX skbuff_head_cache: Marking all objects used
[81234.562076] BUG: unable to handle kernel paging request at ffffffff80014120
[81234.562131] IP: [<ffffffff80014120>] 0xffffffff8001411f
[81234.562159] PGD 220f067 PUD 2210067 PMD 0 
[81234.562186] Oops: 0010 [#1] PREEMPT SMP 
[81234.562192] Modules linked in:
[81234.562198] CPU: 1 PID: 7992 Comm: smbd Tainted: G    B        3.12.0-20131104 #1
[81234.562206] task: ffff88001d934780 ti: ffff88000eb28000 task.ti: ffff88000eb28000
[81234.562212] RIP: e030:[<ffffffff80014120>]  [<ffffffff80014120>] 0xffffffff8001411f
[81234.562219] RSP: e02b:ffff88000eb29e10  EFLAGS: 00010246
[81234.562224] RAX: ffffffff80014120 RBX: fffffffffffffff4 RCX: 0000000000000000
[81234.562230] RDX: 000000000007e000 RSI: 0000000000000000 RDI: 0000000000000000
[81234.562235] RBP: ffff88000eb29e28 R08: 0000000000000022 R09: 00007ffffffff000
[81234.562241] R10: 000000000000007e R11: fffffffffffffff4 R12: 000000000007e000
[81234.562246] R13: 0000000000000022 R14: ffff88001b25cac0 R15: 000000000007e000
[81234.562254] FS:  00007fdc5cc42720(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000
[81234.562260] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[81234.562265] CR2: ffffffff80014120 CR3: 0000000004d1d000 CR4: 0000000000000660
[81234.562271] Stack:
[81234.562274]  ffffffff8115ce9d 0000000000000003 0000000000000000 ffff88000eb29e88
[81234.562283]  ffffffff8115fb4a 0000000000000000 0000000000000000 000000000000007e
[81234.562292]  0000000000000000 ffff88000eb29e88 ffff88001b25cb50 0000000000000000
[81234.562300] Call Trace:
[81234.562308]  [<ffffffff8115ce9d>] ? get_unmapped_area+0x9d/0x150
[81234.562314]  [<ffffffff8115fb4a>] do_mmap_pgoff+0x10a/0x3e0
[81234.562321]  [<ffffffff8114ba10>] vm_mmap_pgoff+0x70/0xb0
[81234.562326]  [<ffffffff8115e38c>] SyS_mmap_pgoff+0x1cc/0x260
[81234.562334]  [<ffffffff814209ee>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[81234.562342]  [<ffffffff81013f1d>] SyS_mmap+0x1d/0x20
[81234.562349]  [<ffffffff81a76379>] system_call_fastpath+0x16/0x1b
[81234.562354] Code:  Bad RIP value.
[81234.562363] RIP  [<ffffffff80014120>] 0xffffffff8001411f
[81234.562368]  RSP <ffff88000eb29e10>
[81234.562372] CR2: ffffffff80014120
[81234.562379] ---[ end trace 4d2d192c2478dd2a ]---

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: kernel BUG at mm/slub.c:3338!  RIP  [<ffffffff8117912c>] kfree+0x20c/0x210
  2013-11-09 18:18     ` Sander Eikelenboom
@ 2013-11-11 14:57       ` Christoph Lameter
  0 siblings, 0 replies; 5+ messages in thread
From: Christoph Lameter @ 2013-11-11 14:57 UTC (permalink / raw)
  To: Sander Eikelenboom; +Cc: Pekka Enberg, linux-kernel, netdev, eric.dumazet

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2466 bytes --]

On Sat, 9 Nov 2013, Sander Eikelenboom wrote:

> This VM got quite some I/O and network load, it got a lot of large files
> transferred to Samba and the files are stored on a glusterfs volume (by
> tcp/ip).

That looks like its related to networking. CCing netdev.
>
>
> [34856.970314] =============================================================================
> [34856.970332] BUG kmalloc-2048 (Not tainted): Poison overwritten
> [34856.970339] -----------------------------------------------------------------------------
> [34856.970339]
> [34856.970350] Disabling lock debugging due to kernel taint
> [34856.970357] INFO: 0xffff88001b36099b-0xffff88001b36099b. First byte 0x6a instead of 0x6b
> [34856.970371] INFO: Allocated in __alloc_skb+0x82/0x290 age=2 cpu=0 pid=2818
> [34856.970381]  __slab_alloc.isra.57+0x61e/0x6d6
> [34856.970389]  __kmalloc_node_track_caller+0xc9/0x1b0
> [34856.970396]  __kmalloc_reserve.isra.49+0x37/0xa0
> [34856.970402]  __alloc_skb+0x82/0x290
> [34856.970408]  sk_stream_alloc_skb+0x3f/0x120
> [34856.970414]  tcp_sendmsg+0x615/0xe20
> [34856.970419]  inet_sendmsg+0xa6/0x100
> [34856.970426]  sock_aio_write+0xc2/0xe0
> [34856.970433]  do_sync_readv_writev+0x4e/0x80
> [34856.970439]  do_readv_writev+0xe8/0x270
> [34856.970444]  vfs_writev+0x30/0x60
> [34856.970449]  SyS_writev+0x54/0xd0
> [34856.970454]  system_call_fastpath+0x16/0x1b
> [34856.970461] INFO: Freed in skb_free_head+0x64/0x70 age=2 cpu=0 pid=0
> [34856.970468]  __slab_free+0x30/0x561
> [34856.970474]  kfree+0x1fd/0x210
> [34856.970479]  skb_free_head+0x64/0x70
> [34856.970484]  skb_release_data+0xf3/0x100
> [34856.970492]  skb_release_all+0x23/0x30
> [34856.970497]  __kfree_skb+0x11/0xa0
> [34856.970503]  tcp_ack+0x64a/0xf20
> [34856.970509]  tcp_rcv_established+0xe2/0x5c0
> [34856.970515]  tcp_v4_do_rcv+0x14e/0x4a0
> [34856.970520]  tcp_v4_rcv+0x8f2/0x9c0
> [34856.970527]  ip_local_deliver_finish+0xd8/0x1d0
> [34856.970533]  ip_local_deliver+0x34/0x90
> [34856.970539]  ip_rcv_finish+0x150/0x660
> [34856.970544]  ip_rcv+0x22b/0x370
> [34856.970551]  __netif_receive_skb_core+0x6e2/0x8b0
> [34856.970557]  __netif_receive_skb+0x1c/0x70
> [34856.970563] INFO: Slab 0xffffea00006cd800 objects=13 used=13 fp=0x          (null) flags=0x40000000004080
> [34856.970572] INFO: Object 0xffff88001b360948 @offset=2376 fp=0xffff88001b3637b0
> [34856.970572]
>
> The rest is in the attached dmesg.
>
> attached: dmesg-oops-3.12.0-20131104.txt
> --
> Sander

[-- Attachment #2: Type: TEXT/PLAIN, Size: 47098 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.12.0-20131104 (root@serveerstertje) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP PREEMPT Mon Nov 4 09:34:18 CET 2013
[    0.000000] Command line: root=/dev/xvda1 ro slub_debug
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] MPS support code is not built-in.
[    0.000000] Using acpi=off or acpi=noirq or pci=noacpi may have problem
[    0.000000] DMI not present or invalid.
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x1fe00000-0x1fffffff]
[    0.000000]  [mem 0x1fe00000-0x1fffffff] page 4k
[    0.000000] BRK [0x02d72000, 0x02d72fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x1c000000-0x1fdfffff]
[    0.000000]  [mem 0x1c000000-0x1fdfffff] page 4k
[    0.000000] BRK [0x02d73000, 0x02d73fff] PGTABLE
[    0.000000] BRK [0x02d74000, 0x02d74fff] PGTABLE
[    0.000000] BRK [0x02d75000, 0x02d75fff] PGTABLE
[    0.000000] BRK [0x02d76000, 0x02d76fff] PGTABLE
[    0.000000] BRK [0x02d77000, 0x02d77fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x00100000-0x1bffffff]
[    0.000000]  [mem 0x00100000-0x1bffffff] page 4k
[    0.000000] RAMDISK: [mem 0x03188000-0x03f9cfff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1fffffff]
[    0.000000]   NODE_DATA [mem 0x1ff1d000-0x1ff27fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000] On node 0 totalpages: 130975
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1984 pages used for memmap
[    0.000000]   DMA32 zone: 126976 pages, LIFO batch:31
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] e820: [mem 0x20000000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.4-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88001fc00000 s83200 r8192 d23296 u1048576
[    0.000000] pcpu-alloc: s83200 r8192 d23296 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128906
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/xvda1 ro slub_debug
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 468620K/523900K available (10729K kernel code, 978K rwdata, 4200K rodata, 1068K init, 9580K bss, 55280K reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Additional per-CPU info printed with stalls.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
[    0.000000] NR_IRQS:4352 nr_irqs:288 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 5855 kB
[    0.000000]  per task-struct memory footprint: 1920 bytes
[    0.000000] kmemleak: Kernel memory leak detector disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3200.180 MHz processor
[    0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 6402.04 BogoMIPS (lpj=10667266)
[    0.003333] pid_max: default: 32768 minimum: 301
[    0.003333] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.003333] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.003333] Mount-cache hash table entries: 256
[    0.003333] Initializing cgroup subsys freezer
[    0.003333] Initializing cgroup subsys blkio
[    0.003333] tseg: 0000000000
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
[    0.003333] Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64
[    0.003333] tlb_flushall_shift: 4
[    0.044828] cpu 0 spinlock event irq 17
[    0.045830] Performance Events: 
[    0.045848] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.045855] no hardware sampling interrupt available.
[    0.045880] Broken PMU hardware detected, using software events only.
[    0.045882] Failed to access perfctr msr (MSR c0010004 is 0)
[    0.063598] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.070288] installing Xen timer for CPU 1
[    0.070377] cpu 1 spinlock event irq 24
[    0.070456] SMP alternatives: lockdep: fixing up alternatives
[    0.070469] SMP alternatives: switching to SMP code
[    0.112838] Brought up 2 CPUs
[    0.113810] devtmpfs: initialized
[    0.115040] xor: measuring software checksum speed
[    0.144411]    prefetch64-sse:  5412.000 MB/sec
[    0.178307]    generic_sse:  5330.400 MB/sec
[    0.178318] xor: using function: prefetch64-sse (5412.000 MB/sec)
[    0.178361] xen:grant_table: Grant tables using version 2 layout
[    0.178420] Grant table initialized
[    0.178917] NET: Registered protocol family 16
[    0.183776] PCI: setting up Xen PCI frontend stub
[    0.183800] PCI: pci_cache_line_size set to 64 bytes
[    0.320441] bio: create slab <bio-0> at 0
[    0.376754] raid6: sse2x1    3860 MB/s
[    0.433368] raid6: sse2x2    5362 MB/s
[    0.490003] raid6: sse2x4    5857 MB/s
[    0.490005] raid6: using algorithm sse2x4 (5857 MB/s)
[    0.490006] raid6: using intx1 recovery algorithm
[    0.490306] ACPI: Interpreter disabled.
[    0.490306] xen:balloon: Initialising balloon driver
[    0.493520] xen_balloon: Initialising balloon driver
[    0.496781] vgaarb: loaded
[    0.497942] SCSI subsystem initialized
[    0.497942] libata version 3.00 loaded.
[    0.500538] usbcore: registered new interface driver usbfs
[    0.500538] usbcore: registered new interface driver hub
[    0.500538] usbcore: registered new device driver usb
[    0.500538] Linux video capture interface: v2.00
[    0.503529] pps_core: LinuxPPS API ver. 1 registered
[    0.503542] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.503612] PTP clock support registered
[    0.503625] Advanced Linux Sound Architecture Driver Initialized.
[    0.503648] PCI: System does not support PCI
[    0.503650] PCI: System does not support PCI
[    0.506907] Bluetooth: Core ver 2.16
[    0.506997] NET: Registered protocol family 31
[    0.506997] Bluetooth: HCI device and connection manager initialized
[    0.506997] Bluetooth: HCI socket layer initialized
[    0.506997] Bluetooth: L2CAP socket layer initialized
[    0.506997] Bluetooth: SCO socket layer initialized
[    0.507279] Switched to clocksource xen
[    0.507396] FS-Cache: Loaded
[    0.507636] pnp: PnP ACPI: disabled
[    0.539117] NET: Registered protocol family 2
[    0.539897] TCP established hash table entries: 4096 (order: 4, 65536 bytes)
[    0.540019] TCP bind hash table entries: 4096 (order: 6, 262144 bytes)
[    0.540277] TCP: Hash tables configured (established 4096 bind 4096)
[    0.540362] TCP: reno registered
[    0.540401] UDP hash table entries: 256 (order: 3, 40960 bytes)
[    0.540458] UDP-Lite hash table entries: 256 (order: 3, 40960 bytes)
[    0.540871] NET: Registered protocol family 1
[    0.540925] PCI: CLS 0 bytes, default 64
[    0.541330] Trying to unpack rootfs image as initramfs...
[    0.604671] Freeing initrd memory: 14420K (ffff880003188000 - ffff880003f9d000)
[    0.810210] DMA-API: preallocated 65536 debug entries
[    0.810232] DMA-API: debugging enabled by kernel config
[    0.810899] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.813974] Scanning for low memory corruption every 60 seconds
[    0.817883] sha1_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817897] sha256_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817902] sha512_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.817908] AVX or AES-NI instructions are not detected.
[    0.817912] AVX instructions are not detected.
[    0.817916] AVX instructions are not detected.
[    0.820512] audit: initializing netlink socket (disabled)
[    0.820601] type=2000 audit(1383778218.007:1): initialized
[    0.822275] bounce pool size: 64 pages
[    0.822293] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.878200] VFS: Disk quotas dquot_6.5.2
[    0.878818] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.897415] NTFS driver 2.1.30 [Flags: R/W].
[    0.900483] fuse init (API version 7.22)
[    0.905703] bio: create slab <bio-1> at 1
[    0.910323] Btrfs loaded
[    0.914633] GFS2 installed
[    0.916780] ceph: loaded (mds proto 32)
[    0.916827] msgmni has been set to 943
[    0.920847] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    0.920891] io scheduler noop registered
[    0.920896] io scheduler deadline registered
[    0.921766] io scheduler cfq registered (default)
[    0.923034] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[    0.923046] crc32: self tests passed, processed 225944 bytes in 109410 nsec
[    0.923163] crc32c: CRC_LE_BITS = 64
[    0.923168] crc32c: self tests passed, processed 225944 bytes in 54579 nsec
[    0.924335] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.925132] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.925447] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.925456] acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
[    0.926534] usbcore: registered new interface driver udlfb
[    0.927691] xen:xen_evtchn: Event-channel device installed
[    0.932426] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.936695] Linux agpgart interface v0.103
[    0.938516] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[    0.938518] Hangcheck: Using getrawmonotonic().
[    0.938784] [drm] Initialized drm 1.1.0 20060810
[    0.939112] [drm] radeon kernel modesetting enabled.
[    0.957684] brd: module loaded
[    0.966412] loop: module loaded
[    0.975940] tun: Universal TUN/TAP device driver, 1.6
[    0.975955] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.976391] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    0.976403] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.976818] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[    0.976831] e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
[    0.977343] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k
[    0.977353] igb: Copyright (c) 2007-2013 Intel Corporation.
[    0.977696] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.2-k
[    0.977709] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.978502] xen_netfront: Initialising Xen virtual ethernet driver
[    0.983343] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.983353] ehci-pci: EHCI PCI platform driver
[    0.984249] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.984258] ohci-pci: OHCI PCI platform driver
[    0.984564] uhci_hcd: USB Universal Host Controller Interface driver
[    0.986205] usbcore: registered new interface driver usblp
[    0.986958] usbcore: registered new interface driver usb-storage
[    0.987932] usbcore: registered new interface driver usbserial
[    0.988274] usbcore: registered new interface driver cp210x
[    0.988703] usbserial: USB Serial support registered for cp210x
[    0.988952] usbcore: registered new interface driver cypress_m8
[    0.989238] usbserial: USB Serial support registered for DeLorme Earthmate USB
[    0.989573] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[    0.989870] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[    0.990111] usbcore: registered new interface driver mos7720
[    0.990417] usbserial: USB Serial support registered for Moschip 2 port adapter
[    0.990654] usbcore: registered new interface driver mos7840
[    0.993506] usbserial: USB Serial support registered for Moschip 7840/7820 USB Serial Driver
[    0.994349] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.009583] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.002564] i8042: No controller found
[    2.004200] mousedev: PS/2 mouse device common for all mice
[    2.066703] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    2.068031] rtc_cmos: probe of rtc_cmos failed with error -38
[    2.071810] lirc_dev: IR Remote Control driver registered, major 248 
[    2.071854] IR NEC protocol handler initialized
[    2.071860] IR RC5(x) protocol handler initialized
[    2.071864] IR RC6 protocol handler initialized
[    2.071868] IR JVC protocol handler initialized
[    2.071873] IR Sony protocol handler initialized
[    2.071877] IR RC5 (streamzap) protocol handler initialized
[    2.071882] IR SANYO protocol handler initialized
[    2.071886] IR MCE Keyboard/mouse protocol handler initialized
[    2.071891] IR LIRC bridge handler initialized
[    2.071896] cx25821: driver version 0.0.106 loaded
[    2.072860] usbcore: registered new interface driver pvrusb2
[    2.072890] pvrusb2: V4L in-tree version:Hauppauge WinTV-PVR-USB2 MPEG2 Encoder/Tuner
[    2.072897] pvrusb2: Debug mask is 31 (0x1f)
[    2.073903] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05
[    2.074490] xen_wdt: Xen WatchDog Timer Driver v0.01
[    2.075432] xen_wdt: initialized (timeout=60s, nowayout=0)
[    2.079512] device-mapper: ioctl: 4.26.0-ioctl (2013-08-15) initialised: dm-devel@redhat.com
[    2.082630] device-mapper: cache-policy-mq: version 1.0.0 loaded
[    2.082660] device-mapper: cache cleaner: version 1.0.0 loaded
[    2.082666] Bluetooth: Virtual HCI driver ver 1.3
[    2.083086] Bluetooth: HCI UART driver ver 2.2
[    2.083098] Bluetooth: HCI H4 protocol initialized
[    2.083103] Bluetooth: HCI BCSP protocol initialized
[    2.083107] Bluetooth: HCILL protocol initialized
[    2.083111] Bluetooth: HCIATH3K protocol initialized
[    2.083115] Bluetooth: HCI Three-wire UART (H5) protocol initialized
[    2.083439] usbcore: registered new interface driver bcm203x
[    2.084033] usbcore: registered new interface driver bpa10x
[    2.084374] usbcore: registered new interface driver bfusb
[    2.084716] usbcore: registered new interface driver btusb
[    2.085210] usbcore: registered new interface driver ath3k
[    2.085814] hidraw: raw HID events driver (C) Jiri Kosina
[    2.088974] usbcore: registered new interface driver usbhid
[    2.088983] usbhid: USB HID core driver
[    2.097677] usbcore: registered new interface driver snd-usb-audio
[    2.097975] usbcore: registered new interface driver snd-ua101
[    2.098252] usbcore: registered new interface driver snd-usb-usx2y
[    2.098526] usbcore: registered new interface driver snd-usb-caiaq
[    2.098797] usbcore: registered new interface driver snd-usb-6fire
[    2.098882] Netfilter messages via NETLINK v0.30.
[    2.098913] nfnl_acct: registering with nfnetlink.
[    2.099088] nf_conntrack version 0.5.0 (3773 buckets, 15092 max)
[    2.100344] ctnetlink v0.93: registering with nfnetlink.
[    2.101476] xt_time: kernel timezone is -0000
[    2.101498]  xvda: xvda1 xvda2
[    2.101509] ip_set: protocol 6
[    2.101550] IPVS: Registered protocols ()
[    2.102027] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[    2.102231] IPVS: Creating netns size=1904 id=0
[    2.102595] IPVS: ipvs loaded.
[    2.103069] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.103473] TCP: cubic registered
[    2.103480] NET: Registered protocol family 17
[    2.104492] Bridge firewalling registered
[    2.104524] Ebtables v2.0 registered
[    2.105453] Bluetooth: RFCOMM TTY layer initialized
[    2.105561] Bluetooth: RFCOMM socket layer initialized
[    2.105567] Bluetooth: RFCOMM ver 1.11
[    2.105571] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    2.105576] Bluetooth: BNEP filters: protocol multicast
[    2.105598] Bluetooth: BNEP socket layer initialized
[    2.105603] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    2.105626] Bluetooth: HIDP socket layer initialized
[    2.105668] Key type ceph registered
[    2.107631] libceph: loaded (mon/osd proto 15/24)
[    2.109119] registered taskstats version 1
[    2.109623] console [netcon0] enabled
[    2.109634] netconsole: network logging started
[    2.109640] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    2.109837] ALSA device list:
[    2.109844]   No soundcards found.
[    2.111258] Freeing unused kernel memory: 1068K (ffffffff822f6000 - ffffffff82401000)
[    2.111275] Write protecting the kernel read-only data: 18432k
[    2.120228] Freeing unused kernel memory: 1548K (ffff880001a7d000 - ffff880001c00000)
[    2.121537] Freeing unused kernel memory: 1944K (ffff88000201a000 - ffff880002200000)
[    2.174907] udevd[1207]: starting version 175
[    4.324325] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[    8.933317] udevd[1429]: starting version 175
[   15.570258] Adding 523260k swap on /dev/xvda2.  Priority:-1 extents:1 across:523260k SS
[   15.637450] EXT4-fs (xvda1): re-mounted. Opts: (null)
[   15.896610] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro
[34856.970314] =============================================================================
[34856.970332] BUG kmalloc-2048 (Not tainted): Poison overwritten
[34856.970339] -----------------------------------------------------------------------------
[34856.970339] 
[34856.970350] Disabling lock debugging due to kernel taint
[34856.970357] INFO: 0xffff88001b36099b-0xffff88001b36099b. First byte 0x6a instead of 0x6b
[34856.970371] INFO: Allocated in __alloc_skb+0x82/0x290 age=2 cpu=0 pid=2818
[34856.970381] 	__slab_alloc.isra.57+0x61e/0x6d6
[34856.970389] 	__kmalloc_node_track_caller+0xc9/0x1b0
[34856.970396] 	__kmalloc_reserve.isra.49+0x37/0xa0
[34856.970402] 	__alloc_skb+0x82/0x290
[34856.970408] 	sk_stream_alloc_skb+0x3f/0x120
[34856.970414] 	tcp_sendmsg+0x615/0xe20
[34856.970419] 	inet_sendmsg+0xa6/0x100
[34856.970426] 	sock_aio_write+0xc2/0xe0
[34856.970433] 	do_sync_readv_writev+0x4e/0x80
[34856.970439] 	do_readv_writev+0xe8/0x270
[34856.970444] 	vfs_writev+0x30/0x60
[34856.970449] 	SyS_writev+0x54/0xd0
[34856.970454] 	system_call_fastpath+0x16/0x1b
[34856.970461] INFO: Freed in skb_free_head+0x64/0x70 age=2 cpu=0 pid=0
[34856.970468] 	__slab_free+0x30/0x561
[34856.970474] 	kfree+0x1fd/0x210
[34856.970479] 	skb_free_head+0x64/0x70
[34856.970484] 	skb_release_data+0xf3/0x100
[34856.970492] 	skb_release_all+0x23/0x30
[34856.970497] 	__kfree_skb+0x11/0xa0
[34856.970503] 	tcp_ack+0x64a/0xf20
[34856.970509] 	tcp_rcv_established+0xe2/0x5c0
[34856.970515] 	tcp_v4_do_rcv+0x14e/0x4a0
[34856.970520] 	tcp_v4_rcv+0x8f2/0x9c0
[34856.970527] 	ip_local_deliver_finish+0xd8/0x1d0
[34856.970533] 	ip_local_deliver+0x34/0x90
[34856.970539] 	ip_rcv_finish+0x150/0x660
[34856.970544] 	ip_rcv+0x22b/0x370
[34856.970551] 	__netif_receive_skb_core+0x6e2/0x8b0
[34856.970557] 	__netif_receive_skb+0x1c/0x70
[34856.970563] INFO: Slab 0xffffea00006cd800 objects=13 used=13 fp=0x          (null) flags=0x40000000004080
[34856.970572] INFO: Object 0xffff88001b360948 @offset=2376 fp=0xffff88001b3637b0
[34856.970572] 
[34856.973176] Bytes b4 ffff88001b360938: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[34856.973176] Object ffff88001b360948: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360958: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360968: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360978: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360988: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360998: 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkjkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3609f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b360ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361008: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361018: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361028: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361038: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361048: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361058: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361068: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361078: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361088: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361098: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b3610f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361108: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361118: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361128: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[34856.973176] Object ffff88001b361138: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
[34856.973176] Redzone ffff88001b361148: bb bb bb bb bb bb bb bb                          ........
[34856.973176] Padding ffff88001b361288: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[34856.973176] CPU: 1 PID: 2818 Comm: glusterfs Tainted: G    B        3.12.0-20131104 #1
[34856.973176]  ffff88001b360948 ffff88001ad218e8 ffffffff81a6a8ba ffff88001eb36b40
[34856.973176]  ffff88001e802d80 ffff88001ad21918 ffffffff8117af19 ffff88001b36099c
[34856.973176]  ffff88001e802d80 000000000000006b ffff88001b360948 ffff88001ad21968
[34856.973176] Call Trace:
[34856.973176]  [<ffffffff81a6a8ba>] dump_stack+0x4f/0x84
[34856.973176]  [<ffffffff8117af19>] print_trailer+0xf9/0x160
[34856.973176]  [<ffffffff8117b0b2>] check_bytes_and_report+0xe2/0x120
[34856.973176]  [<ffffffff8117b2b7>] check_object+0x1c7/0x240
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff81a66cc2>] alloc_debug_processing+0x153/0x168
[34856.973176]  [<ffffffff81a672f5>] __slab_alloc.isra.57+0x61e/0x6d6
[34856.973176]  [<ffffffff810ef7bb>] ? trace_hardirqs_on_caller+0xfb/0x240
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff810ef624>] ? mark_held_locks+0x94/0x130
[34856.973176]  [<ffffffff8117ef49>] __kmalloc_node_track_caller+0xc9/0x1b0
[34856.973176]  [<ffffffff818ffac2>] ? __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff818ff9d7>] __kmalloc_reserve.isra.49+0x37/0xa0
[34856.973176]  [<ffffffff818ffa92>] ? __alloc_skb+0x52/0x290
[34856.973176]  [<ffffffff818ffac2>] __alloc_skb+0x82/0x290
[34856.973176]  [<ffffffff810f372a>] ? lock_release+0x12a/0x250
[34856.973176]  [<ffffffff819af1af>] sk_stream_alloc_skb+0x3f/0x120
[34856.973176]  [<ffffffff819aff85>] tcp_sendmsg+0x615/0xe20
[34856.973176]  [<ffffffff819d7e16>] inet_sendmsg+0xa6/0x100
[34856.973176]  [<ffffffff819d7d70>] ? inet_autobind+0x60/0x60
[34856.973176]  [<ffffffff818f57b2>] sock_aio_write+0xc2/0xe0
[34856.973176]  [<ffffffff811899be>] do_sync_readv_writev+0x4e/0x80
[34856.973176]  [<ffffffff8118b0e8>] do_readv_writev+0xe8/0x270
[34856.973176]  [<ffffffff810f372a>] ? lock_release+0x12a/0x250
[34856.973176]  [<ffffffff8118b2a0>] vfs_writev+0x30/0x60
[34856.973176]  [<ffffffff8118b444>] SyS_writev+0x54/0xd0
[34856.973176]  [<ffffffff81a76379>] system_call_fastpath+0x16/0x1b
[34856.973176] FIX kmalloc-2048: Restoring 0xffff88001b36099b-0xffff88001b36099b=0x6b
[34856.973176] 
[34856.973176] FIX kmalloc-2048: Marking all objects used
[81155.636434] =============================================================================
[81155.636459] BUG skbuff_head_cache (Tainted: G    B       ): Poison overwritten
[81155.636469] -----------------------------------------------------------------------------
[81155.636469] 
[81155.636481] INFO: 0xffff88001e1e7b1b-0xffff88001e1e7b1b. First byte 0x6a instead of 0x6b
[81155.636495] INFO: Allocated in build_skb+0x2f/0x1c0 age=3 cpu=0 pid=0
[81155.636506] 	__slab_alloc.isra.57+0x61e/0x6d6
[81155.636513] 	kmem_cache_alloc+0x10a/0x140
[81155.636519] 	build_skb+0x2f/0x1c0
[81155.636524] 	__netdev_alloc_skb+0x91/0xf0
[81155.636532] 	xennet_alloc_rx_buffers+0xb6/0x340
[81155.636538] 	xennet_poll+0x6c7/0xdb0
[81155.636545] 	net_rx_action+0x159/0x250
[81155.636551] 	__do_softirq+0xea/0x1e0
[81155.636557] 	call_softirq+0x1c/0x30
[81155.636564] 	do_softirq+0x95/0x100
[81155.636569] 	irq_exit+0xa2/0xd0
[81155.636576] 	xen_evtchn_do_upcall+0x2f/0x40
[81155.636582] 	xen_do_hypervisor_callback+0x1e/0x30
[81155.636595] INFO: Freed in __kfree_skb+0x3a/0xa0 age=2 cpu=0 pid=0
[81155.636603] 	__slab_free+0x30/0x561
[81155.636608] 	kmem_cache_free+0x212/0x230
[81155.636614] 	__kfree_skb+0x3a/0xa0
[81155.636620] 	tcp_rcv_established+0x33b/0x5c0
[81155.636627] 	tcp_v4_do_rcv+0x14e/0x4a0
[81155.636633] 	tcp_v4_rcv+0x8f2/0x9c0
[81155.636639] 	ip_local_deliver_finish+0xd8/0x1d0
[81155.636647] 	ip_local_deliver+0x34/0x90
[81155.636653] 	ip_rcv_finish+0x150/0x660
[81155.636659] 	ip_rcv+0x22b/0x370
[81155.636664] 	__netif_receive_skb_core+0x6e2/0x8b0
[81155.636670] 	__netif_receive_skb+0x1c/0x70
[81155.636676] 	netif_receive_skb+0x1e/0x100
[81155.636681] 	xennet_poll+0x8a4/0xdb0
[81155.636687] 	net_rx_action+0x159/0x250
[81155.636692] 	__do_softirq+0xea/0x1e0
[81155.636698] INFO: Slab 0xffffea0000787980 objects=14 used=14 fp=0x          (null) flags=0x40000000004080
[81155.636707] INFO: Object 0xffff88001e1e7b00 @offset=6912 fp=0xffff88001e1e6b40
[81155.636707] 
[81155.636719] Bytes b4 ffff88001e1e7af0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[81155.636728] Object ffff88001e1e7b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636738] Object ffff88001e1e7b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b  kkkkkkkkkkkjkkkk
[81155.636748] Object ffff88001e1e7b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636757] Object ffff88001e1e7b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636766] Object ffff88001e1e7b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636775] Object ffff88001e1e7b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636785] Object ffff88001e1e7b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636794] Object ffff88001e1e7b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636803] Object ffff88001e1e7b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636812] Object ffff88001e1e7b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636821] Object ffff88001e1e7ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636830] Object ffff88001e1e7bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636839] Object ffff88001e1e7bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636848] Object ffff88001e1e7bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[81155.636857] Object ffff88001e1e7be0: 6b 6b 6b 6b 6b 6b 6b a5                          kkkkkkk.
[81155.636869] Redzone ffff88001e1e7be8: bb bb bb bb bb bb bb bb                          ........
[81155.636881] Padding ffff88001e1e7d28: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[81155.636890] Padding ffff88001e1e7d38: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[81155.636912] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G    B        3.12.0-20131104 #1
[81155.636914]  ffff88001e1e7b00 ffff88001fc039d0 ffffffff81a6a8ba ffffffff822134e0
[81155.636920]  ffff88001e93e6c0 ffff88001fc03a00 ffffffff8117af19 ffff88001e1e7b1c
[81155.636924]  ffff88001e93e6c0 000000000000006b ffff88001e1e7b00 ffff88001fc03a50
[81155.636928] Call Trace:
[81155.636930]  <IRQ>  [<ffffffff81a6a8ba>] dump_stack+0x4f/0x84
[81155.636936]  [<ffffffff8117af19>] print_trailer+0xf9/0x160
[81155.636940]  [<ffffffff8117b0b2>] check_bytes_and_report+0xe2/0x120
[81155.636943]  [<ffffffff8117b2b7>] check_object+0x1c7/0x240
[81155.636947]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636950]  [<ffffffff81a66cc2>] alloc_debug_processing+0x153/0x168
[81155.636954]  [<ffffffff81a672f5>] __slab_alloc.isra.57+0x61e/0x6d6
[81155.636957]  [<ffffffff819a058d>] ? ip_local_deliver_finish+0x17d/0x1d0
[81155.636961]  [<ffffffff819a044c>] ? ip_local_deliver_finish+0x3c/0x1d0
[81155.636964]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636968]  [<ffffffff819a0754>] ? ip_local_deliver+0x34/0x90
[81155.636971]  [<ffffffff8199ff00>] ? ip_rcv_finish+0x150/0x660
[81155.636975]  [<ffffffff8117ce7a>] kmem_cache_alloc+0x10a/0x140
[81155.636978]  [<ffffffff818fe78f>] ? build_skb+0x2f/0x1c0
[81155.636981]  [<ffffffff818fe78f>] build_skb+0x2f/0x1c0
[81155.636985]  [<ffffffff819004b1>] __netdev_alloc_skb+0x91/0xf0
[81155.636988]  [<ffffffff816d50a6>] xennet_alloc_rx_buffers+0xb6/0x340
[81155.636991]  [<ffffffff8190d1be>] ? netif_receive_skb+0x1e/0x100
[81155.636995]  [<ffffffff816d6fb7>] xennet_poll+0x6c7/0xdb0
[81155.636999]  [<ffffffff8190d669>] net_rx_action+0x159/0x250
[81155.637003]  [<ffffffff8109fa3a>] __do_softirq+0xea/0x1e0
[81155.637006]  [<ffffffff81a777bc>] call_softirq+0x1c/0x30
[81155.637009]  [<ffffffff81010dc5>] do_softirq+0x95/0x100
[81155.637013]  [<ffffffff8109fce2>] irq_exit+0xa2/0xd0
[81155.637016]  [<ffffffff814bb2af>] xen_evtchn_do_upcall+0x2f/0x40
[81155.637019]  [<ffffffff81a7781e>] xen_do_hypervisor_callback+0x1e/0x30
[81155.637021]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[81155.637027]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[81155.637031]  [<ffffffff81008da0>] ? xen_safe_halt+0x10/0x20
[81155.637034]  [<ffffffff810185cd>] ? default_idle+0x1d/0x30
[81155.637038]  [<ffffffff81018f0e>] ? arch_cpu_idle+0x2e/0x40
[81155.637042]  [<ffffffff810db410>] ? cpu_startup_entry+0x80/0x1d0
[81155.637046]  [<ffffffff81a5ebf1>] ? rest_init+0xc1/0xd0
[81155.637049]  [<ffffffff81a5eb30>] ? csum_partial_copy_generic+0x170/0x170
[81155.637054]  [<ffffffff8230bee6>] ? start_kernel+0x3d3/0x3e0
[81155.637058]  [<ffffffff8230b912>] ? repair_env_string+0x5e/0x5e
[81155.637061]  [<ffffffff810d89f7>] ? __add_preferred_console.constprop.18+0x87/0xb0
[81155.637065]  [<ffffffff8230b5f8>] ? x86_64_start_reservations+0x2a/0x2c
[81155.637069]  [<ffffffff8230ecd9>] ? xen_start_kernel+0x563/0x56d
[81155.637072] FIX skbuff_head_cache: Restoring 0xffff88001e1e7b1b-0xffff88001e1e7b1b=0x6b
[81155.637072] 
[81155.637084] FIX skbuff_head_cache: Marking all objects used
[81234.562076] BUG: unable to handle kernel paging request at ffffffff80014120
[81234.562131] IP: [<ffffffff80014120>] 0xffffffff8001411f
[81234.562159] PGD 220f067 PUD 2210067 PMD 0 
[81234.562186] Oops: 0010 [#1] PREEMPT SMP 
[81234.562192] Modules linked in:
[81234.562198] CPU: 1 PID: 7992 Comm: smbd Tainted: G    B        3.12.0-20131104 #1
[81234.562206] task: ffff88001d934780 ti: ffff88000eb28000 task.ti: ffff88000eb28000
[81234.562212] RIP: e030:[<ffffffff80014120>]  [<ffffffff80014120>] 0xffffffff8001411f
[81234.562219] RSP: e02b:ffff88000eb29e10  EFLAGS: 00010246
[81234.562224] RAX: ffffffff80014120 RBX: fffffffffffffff4 RCX: 0000000000000000
[81234.562230] RDX: 000000000007e000 RSI: 0000000000000000 RDI: 0000000000000000
[81234.562235] RBP: ffff88000eb29e28 R08: 0000000000000022 R09: 00007ffffffff000
[81234.562241] R10: 000000000000007e R11: fffffffffffffff4 R12: 000000000007e000
[81234.562246] R13: 0000000000000022 R14: ffff88001b25cac0 R15: 000000000007e000
[81234.562254] FS:  00007fdc5cc42720(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000
[81234.562260] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[81234.562265] CR2: ffffffff80014120 CR3: 0000000004d1d000 CR4: 0000000000000660
[81234.562271] Stack:
[81234.562274]  ffffffff8115ce9d 0000000000000003 0000000000000000 ffff88000eb29e88
[81234.562283]  ffffffff8115fb4a 0000000000000000 0000000000000000 000000000000007e
[81234.562292]  0000000000000000 ffff88000eb29e88 ffff88001b25cb50 0000000000000000
[81234.562300] Call Trace:
[81234.562308]  [<ffffffff8115ce9d>] ? get_unmapped_area+0x9d/0x150
[81234.562314]  [<ffffffff8115fb4a>] do_mmap_pgoff+0x10a/0x3e0
[81234.562321]  [<ffffffff8114ba10>] vm_mmap_pgoff+0x70/0xb0
[81234.562326]  [<ffffffff8115e38c>] SyS_mmap_pgoff+0x1cc/0x260
[81234.562334]  [<ffffffff814209ee>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[81234.562342]  [<ffffffff81013f1d>] SyS_mmap+0x1d/0x20
[81234.562349]  [<ffffffff81a76379>] system_call_fastpath+0x16/0x1b
[81234.562354] Code:  Bad RIP value.
[81234.562363] RIP  [<ffffffff80014120>] 0xffffffff8001411f
[81234.562368]  RSP <ffff88000eb29e10>
[81234.562372] CR2: ffffffff80014120
[81234.562379] ---[ end trace 4d2d192c2478dd2a ]---

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2013-11-11 14:57 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-10-20 18:05 kernel BUG at mm/slub.c:3338! RIP [<ffffffff8117912c>] kfree+0x20c/0x210 Sander Eikelenboom
2013-10-22 15:12 ` Christoph Lameter
2013-10-22 15:29   ` Sander Eikelenboom
2013-11-09 18:18     ` Sander Eikelenboom
2013-11-11 14:57       ` Christoph Lameter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®