Hello, Apologies in advance, as this is my first bug report for the kernel. I would appreciate a lot for any time spent on solving the issue. We're running Gentoo Linux x86_64 with kernel 3.9.10 as a guest OS under VMware host (VMware vCenter Server 5.0.0). Problem summary: Kernel panic occurs during nightly backup maintenance of the host system (causes heavy I/O load on all guests -- there are around 60 virtual machines in total). This issue happens randomly and always under stress I/O load. It does not happen every night, just occasionally. Attached is the call trace in the console (unfortunately, I was not able to get this trace report in the logs as the error seems to have occurred in the pvscsi). I hope it helps somehow. Additional modules used: PVSCSI VMWare Paravirtual SCSI driver, VMXNET3 VMware network driver Complete dmesg is attached (dmesg.txt) Kernel version: Linux version 3.9.10-gentoo-mrbyte (root@BEANRWEB01) (gcc version 4.7.3 (Gentoo 4.7.3 p1.0, pie-0.5.5) ) #1 SMP Sun Jul 14 10:30:58 CEST 2013 Environment: MANPATH=/usr/local/share/man:/usr/share/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/man:/usr/lib64/php5.3/man/:/usr/lib64/php5.4/man/ HOSTNAME=BEANRWEB01 crt= TERM=xterm SHELL=/bin/bash SSH_CLIENT=88.135.136.85 57765 22 SSH_TTY=/dev/pts/0 USER=root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=0 1;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: GUILE_LOAD_PATH=/usr/share/guile/1.8 LESSCHARDEF=8bcccbcc18b95.b27.b99.b PAGER=/usr/bin/less CONFIG_PROTECT_MASK=/etc/gentoo-release /etc/sandbox.d /etc/php/cli-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/apache2-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/fonts/fonts.conf /etc/terminfo /etc/ca-certificates.conf /etc/revdep-rebuild MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3:/root/scripts:/etc/php/scripts LC_COLLATE=C PWD=/root EDITOR=nano -w LANG=POSIX DEVEL_SITE=BEANRWEB01 PS1=\h:\[\033[32;1m\]\w\[\033[0m\]\$ HISTCONTROL=ignoredups SHLVL=1 HOME=/root LESS=-R -M --shift 5 LOGNAME=root GCC_SPECS= LC_CTYPE=en_US.utf8 SSH_CONNECTION=88.135.136.85 57765 192.168.99.14 22 LESSOPEN=|lesspipe %s INFOPATH=/usr/share/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/info CONFIG_PROTECT=/usr/share/gnupg/qualified.txt _=/usr/bin/env Software: Linux BEANRWEB01 3.9.10-gentoo-mrbyte #1 SMP Sun Jul 14 10:30:58 CEST 2013 x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux Gnu C 4.7.3 Gnu make 3.82 binutils 2.23.1 util-linux 2.23.1 mount assert module-init-tools 14 e2fsprogs 1.42.7 Linux C Library 2.17 Dynamic linker (ldd) 2.17 Procps 3.3.8 Net-tools 1.60_p20130513023548 Kbd 1.15.5 Sh-utils 8.21 Modules Loaded xt_multiport xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack vmwgfx cfbfillrect cfbimgblt cfbcopyarea fb fbdev ttm drm i2c_core agpgart shpchp evdev libcrc32c aes_x86_64 sha256_generic iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi tg3 ptp pps_core e1000 fuse nfs lockd sunrpc reiserfs linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq async_xor xor async_tx raid1 raid0 dm_snapshot dm_crypt megaraid_sas megaraid_mbox megaraid_mm megaraid hpsa cciss Processor information: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 1 microcode : 0x14 cpu MHz : 2666.761 cache size : 12288 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm bogomips : 5333.52 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 1 microcode : 0x14 cpu MHz : 2666.761 cache size : 12288 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm bogomips : 5333.52 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 1 microcode : 0x14 cpu MHz : 2666.761 cache size : 12288 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm bogomips : 5333.52 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 1 microcode : 0x14 cpu MHz : 2666.761 cache size : 12288 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm bogomips : 5333.52 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: Module information: xt_multiport 1468 6 - Live 0xffffffffa02e0000 xt_comment 851 1 - Live 0xffffffffa02dc000 nf_conntrack_ipv4 10607 16 - Live 0xffffffffa02d5000 nf_defrag_ipv4 1115 1 nf_conntrack_ipv4, Live 0xffffffffa02d1000 xt_conntrack 2705 16 - Live 0xffffffffa02cd000 vmwgfx 101892 0 - Live 0xffffffffa02aa000 cfbfillrect 2930 1 vmwgfx, Live 0xffffffffa02a6000 cfbimgblt 1879 1 vmwgfx, Live 0xffffffffa02a2000 cfbcopyarea 2918 1 vmwgfx, Live 0xffffffffa029e000 fb 27871 1 vmwgfx, Live 0xffffffffa0291000 fbdev 734 1 fb, Live 0xffffffffa028d000 ttm 51070 1 vmwgfx, Live 0xffffffffa0277000 drm 191627 2 vmwgfx,ttm, Live 0xffffffffa0235000 i2c_core 16307 1 drm, Live 0xffffffffa022c000 agpgart 22470 2 ttm,drm, Live 0xffffffffa0220000 shpchp 21383 0 - Live 0xffffffffa0215000 evdev 8114 0 - Live 0xffffffffa020f000 libcrc32c 890 0 - Live 0xffffffffa020b000 aes_x86_64 7191 0 - Live 0xffffffffa0206000 sha256_generic 9445 0 - Live 0xffffffffa0200000 iscsi_tcp 7540 0 - Live 0xffffffffa01fa000 libiscsi_tcp 10056 1 iscsi_tcp, Live 0xffffffffa01f3000 libiscsi 28649 2 iscsi_tcp,libiscsi_tcp, Live 0xffffffffa01e6000 scsi_transport_iscsi 32193 2 iscsi_tcp,libiscsi, Live 0xffffffffa01d6000 tg3 121160 0 - Live 0xffffffffa01b0000 ptp 6587 1 tg3, Live 0xffffffffa01aa000 pps_core 5695 1 ptp, Live 0xffffffffa01a4000 e1000 85452 0 - Live 0xffffffffa0187000 fuse 58664 0 - Live 0xffffffffa0170000 nfs 96292 0 - Live 0xffffffffa0149000 lockd 50497 1 nfs, Live 0xffffffffa0134000 sunrpc 138315 2 nfs,lockd, Live 0xffffffffa00ff000 reiserfs 175631 0 - Live 0xffffffffa00c9000 linear 3111 0 - Live 0xffffffffa00c5000 raid10 33349 0 - Live 0xffffffffa00b8000 raid456 46563 0 - Live 0xffffffffa00a7000 async_raid6_recov 1169 1 raid456, Live 0xffffffffa00a3000 async_memcpy 822 1 raid456, Live 0xffffffffa009f000 async_pq 1948 1 raid456, Live 0xffffffffa009b000 raid6_pq 89031 2 async_raid6_recov,async_pq, Live 0xffffffffa0081000 async_xor 1257 2 raid456,async_pq, Live 0xffffffffa007d000 xor 9737 1 async_xor, Live 0xffffffffa0077000 async_tx 1217 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor, Live 0xffffffffa0073000 raid1 22781 0 - Live 0xffffffffa0069000 raid0 6566 0 - Live 0xffffffffa0064000 dm_snapshot 22927 0 - Live 0xffffffffa0059000 dm_crypt 12516 0 - Live 0xffffffffa0051000 megaraid_sas 58790 0 - Live 0xffffffffa003c000 megaraid_mbox 22563 0 - Live 0xffffffffa0031000 megaraid_mm 6687 1 megaraid_mbox, Live 0xffffffffa002b000 megaraid 29643 0 - Live 0xffffffffa001f000 hpsa 39492 0 - Live 0xffffffffa0010000 cciss 39106 0 - Live 0xffffffffa0000000 Loaded driver and hardware information: 0000-0cf7 : PCI Bus 0000:00 0000-001f : dma1 0020-0021 : pic1 0040-0043 : timer0 0050-0053 : timer1 0060-0060 : keyboard 0064-0064 : keyboard 0070-0071 : rtc0 0080-008f : dma page reg 00a0-00a1 : pic2 00c0-00df : dma2 00f0-00ff : fpu 0170-0177 : 0000:00:07.1 01f0-01f7 : 0000:00:07.1 02f8-02ff : serial 0376-0376 : 0000:00:07.1 03c0-03df : vga+ 03f6-03f6 : 0000:00:07.1 03f8-03ff : serial 0cf0-0cf1 : pnp 00:00 0cf8-0cff : PCI conf1 0d00-feff : PCI Bus 0000:00 1000-103f : 0000:00:07.3 1000-103f : pnp 00:00 1000-1003 : ACPI PM1a_EVT_BLK 1004-1005 : ACPI PM1a_CNT_BLK 1008-100b : ACPI PM_TMR 100c-100f : ACPI GPE0_BLK 1010-1015 : ACPI CPU throttle 1040-104f : 0000:00:07.3 1040-104f : pnp 00:00 1060-107f : pnp 00:0b 1080-10bf : 0000:00:07.7 1080-10bf : vmw_vmci 10c0-10cf : 0000:00:07.1 10d0-10df : 0000:00:0f.0 10d0-10df : vmwgfx probe 2000-3fff : PCI Bus 0000:02 4000-4fff : PCI Bus 0000:03 4000-400f : 0000:03:00.0 5000-5fff : PCI Bus 0000:0b 5008-500f : 0000:0b:00.0 5008-500f : vmw_pvscsi 6000-6fff : PCI Bus 0000:13 7000-7fff : PCI Bus 0000:1b 8000-8fff : PCI Bus 0000:04 9000-9fff : PCI Bus 0000:0c a000-afff : PCI Bus 0000:14 b000-bfff : PCI Bus 0000:1c c000-cfff : PCI Bus 0000:05 d000-dfff : PCI Bus 0000:0d e000-efff : PCI Bus 0000:15 00000000-00000fff : reserved 00001000-0009f7ff : System RAM 0009f800-0009ffff : reserved 000a0000-000bffff : PCI Bus 0000:00 000c0000-000c7fff : Video ROM 000c8000-000c8fff : Adapter ROM 000cc000-000cffff : PCI Bus 0000:00 000d0000-000d3fff : PCI Bus 0000:00 000d4000-000d7fff : PCI Bus 0000:00 000d8000-000dbfff : PCI Bus 0000:00 000dc000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-bfedffff : System RAM 01000000-01373838 : Kernel code 01373839-0166bdbf : Kernel data 0172f000-017c5fff : Kernel bss bfee0000-bfefefff : ACPI Tables bfeff000-bfefffff : ACPI Non-volatile Storage bff00000-bfffffff : System RAM c0000000-febfffff : PCI Bus 0000:00 c0000000-c0007fff : 0000:00:0f.0 d4000000-d7ffffff : 0000:00:0f.0 d4000000-d7ffffff : vmwgfx probe d8000000-d87fffff : 0000:00:0f.0 d8000000-d87fffff : vmwgfx probe d8800000-d89fffff : pnp 00:0b d9100000-d9bfffff : PCI Bus 0000:02 d9c00000-d9cfffff : PCI Bus 0000:03 d9c02000-d9c03fff : 0000:03:00.0 d9c04000-d9c04fff : 0000:03:00.0 d9c04000-d9c04fff : vmxnet3 d9c05000-d9c05fff : 0000:03:00.0 d9c05000-d9c05fff : vmxnet3 d9d00000-d9dfffff : PCI Bus 0000:0b d9d08000-d9d0ffff : 0000:0b:00.0 d9d08000-d9d0ffff : vmw_pvscsi d9e00000-d9efffff : PCI Bus 0000:13 d9f00000-d9ffffff : PCI Bus 0000:1b da000000-da0fffff : PCI Bus 0000:04 da100000-da1fffff : PCI Bus 0000:0c da200000-da2fffff : PCI Bus 0000:14 da300000-da3fffff : PCI Bus 0000:1c da400000-da4fffff : PCI Bus 0000:05 da500000-da5fffff : PCI Bus 0000:0d da600000-da6fffff : PCI Bus 0000:15 da700000-da7fffff : PCI Bus 0000:1d da800000-da8fffff : PCI Bus 0000:06 da900000-da9fffff : PCI Bus 0000:0e daa00000-daafffff : PCI Bus 0000:16 dab00000-dabfffff : PCI Bus 0000:1e dac00000-dacfffff : PCI Bus 0000:07 dad00000-dadfffff : PCI Bus 0000:0f dae00000-daefffff : PCI Bus 0000:17 daf00000-daffffff : PCI Bus 0000:1f db000000-db0fffff : PCI Bus 0000:08 db100000-db1fffff : PCI Bus 0000:10 db200000-db2fffff : PCI Bus 0000:18 db300000-db3fffff : PCI Bus 0000:20 db400000-db4fffff : PCI Bus 0000:09 db500000-db5fffff : PCI Bus 0000:11 db600000-db6fffff : PCI Bus 0000:19 db700000-db7fffff : PCI Bus 0000:21 db800000-db8fffff : PCI Bus 0000:0a db900000-db9fffff : PCI Bus 0000:12 dba00000-dbafffff : PCI Bus 0000:1a dbb00000-dbbfffff : PCI Bus 0000:22 dc000000-dc5fffff : PCI Bus 0000:02 dc600000-dc6fffff : PCI Bus 0000:03 dc600000-dc60ffff : 0000:03:00.0 dc700000-dc7fffff : PCI Bus 0000:0b dc700000-dc70ffff : 0000:0b:00.0 dc800000-dc8fffff : PCI Bus 0000:13 dc900000-dc9fffff : PCI Bus 0000:1b dca00000-dcafffff : PCI Bus 0000:04 dcb00000-dcbfffff : PCI Bus 0000:0c dcc00000-dccfffff : PCI Bus 0000:14 dcd00000-dcdfffff : PCI Bus 0000:1c dce00000-dcefffff : PCI Bus 0000:05 dcf00000-dcffffff : PCI Bus 0000:0d dd000000-dd0fffff : PCI Bus 0000:15 dd100000-dd1fffff : PCI Bus 0000:1d dd200000-dd2fffff : PCI Bus 0000:06 dd300000-dd3fffff : PCI Bus 0000:0e dd400000-dd4fffff : PCI Bus 0000:16 dd500000-dd5fffff : PCI Bus 0000:1e dd600000-dd6fffff : PCI Bus 0000:07 dd700000-dd7fffff : PCI Bus 0000:0f dd800000-dd8fffff : PCI Bus 0000:17 dd900000-dd9fffff : PCI Bus 0000:1f dda00000-ddafffff : PCI Bus 0000:08 ddb00000-ddbfffff : PCI Bus 0000:10 ddc00000-ddcfffff : PCI Bus 0000:18 ddd00000-dddfffff : PCI Bus 0000:20 dde00000-ddefffff : PCI Bus 0000:09 ddf00000-ddffffff : PCI Bus 0000:11 de000000-de0fffff : PCI Bus 0000:19 de100000-de1fffff : PCI Bus 0000:21 de200000-de2fffff : PCI Bus 0000:0a de300000-de3fffff : PCI Bus 0000:12 de400000-de4fffff : PCI Bus 0000:1a de500000-de5fffff : PCI Bus 0000:22 e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff] e0000000-efffffff : reserved e0000000-efffffff : pnp 00:0b fec00000-fec0ffff : reserved fec00000-fec003ff : IOAPIC 0 fee00000-fee00fff : Local APIC fee00000-fee00fff : reserved fffe0000-ffffffff : reserved 100000000-43fffffff : System RAM SCSI information: Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: VMware Model: Virtual disk Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 02 Other Information of potential interest: /etc/fstab: /dev/sda1 /boot ext4 noauto,noatime 1 2 /dev/sda2 / ext4 noatime 0 1 /dev/sda3 none swap sw 0 0 /dev/mapper/ld-usr /usr ext4 noatime /dev/mapper/ld-portage /usr/portage ext4 noatime /dev/mapper/ld-var /var ext4 noatime,nosuid /dev/mapper/ld-tmp /tmp ext4 noatime,nosuid /dev/mapper/ld-home /home ext4 noatime,nosuid /dev/mapper/ld-opt /opt ext4 noatime,nosuid Kernel trace log (only available in the vSphere Client console), so I made a screen capture :