On Fri, Dec 11, 2015 at 01:15:04PM +0100, Paolo Bonzini wrote:
> My wild guess is that RSP is getting corrupted, but I guess I'll have to try
> to reproduce to figure out what happens.

Yeah, something's scribbling over stuff where it shouldn't.

> The last thing I need from you (hopefully) is a Kconfig.  If you have some

Attached.

> time, it would be great to check if you can reproduce it with an older kernel
> version---trying 4.4-rc1 and 4.3 would be great.

Just did 4.3 and got a much cleaner splat, see below. According to Code:, we're
trapping at:

  2b:*  ff 50 78                callq  *0x78(%rax)              <-- trapping instruction

in kvm_arch_vcpu_put() which is this:

	movq    kvm_x86_ops(%rip), %rax # kvm_x86_ops, kvm_x86_ops
	...
	call    *120(%rax)      # _2->vcpu_put

and RAX is 0. So ->vcpu_put() is 0?! I don't think so. So it must be
something corrupted kvm_x86_ops.

Hmmm.

...
[  133.387161] kvm: zapping shadow pages for mmio generation wraparound
[  135.998430] kvm [3717]: vcpu0 unhandled rdmsr: 0xc0011021
[  136.359458] kvm [3717]: vcpu1 unhandled rdmsr: 0xc0011021
[  136.466257] kvm [3717]: vcpu2 unhandled rdmsr: 0xc0011021
[  136.563658] kvm [3717]: vcpu3 unhandled rdmsr: 0xc0011021
[  136.663534] kvm [3717]: vcpu4 unhandled rdmsr: 0xc0011021
[  136.763708] kvm [3717]: vcpu5 unhandled rdmsr: 0xc0011021
[  136.869131] kvm [3717]: vcpu6 unhandled rdmsr: 0xc0011021
[  136.967479] kvm [3717]: vcpu7 unhandled rdmsr: 0xc0011021
[  245.585109] kvm: zapping shadow pages for mmio generation wraparound
[  247.482552] kvm [3781]: vcpu0 unhandled rdmsr: 0xc0011021
[  247.810769] kvm [3781]: vcpu1 unhandled rdmsr: 0xc0011021
[  247.909217] kvm [3781]: vcpu2 unhandled rdmsr: 0xc0011021
[  248.009079] kvm [3781]: vcpu3 unhandled rdmsr: 0xc0011021
[  248.111232] kvm [3781]: vcpu4 unhandled rdmsr: 0xc0011021
[  248.214389] kvm [3781]: vcpu5 unhandled rdmsr: 0xc0011021
[  248.318640] kvm [3781]: vcpu6 unhandled rdmsr: 0xc0011021
[  248.422611] kvm [3781]: vcpu7 unhandled rdmsr: 0xc0011021
[  266.489947] qemu-system-x86[3783]: segfault at ffffffff816c90b5 ip ffffffff816c90b5 sp 00007f1f3b50ca48 error 15
[  273.715102] kvm: zapping shadow pages for mmio generation wraparound
[  275.742117] kvm [3831]: vcpu0 unhandled rdmsr: 0xc0011021
[  276.119764] kvm [3831]: vcpu1 unhandled rdmsr: 0xc0011021
[  276.220739] kvm [3831]: vcpu2 unhandled rdmsr: 0xc0011021
[  276.325400] kvm [3831]: vcpu3 unhandled rdmsr: 0xc0011021
[  276.426287] kvm [3831]: vcpu4 unhandled rdmsr: 0xc0011021
[  276.529345] kvm [3831]: vcpu5 unhandled rdmsr: 0xc0011021
[  276.634471] kvm [3831]: vcpu6 unhandled rdmsr: 0xc0011021
[  276.733205] kvm [3831]: vcpu7 unhandled rdmsr: 0xc0011021
[  667.184231] kvm: zapping shadow pages for mmio generation wraparound
[  669.043526] kvm [3889]: vcpu0 unhandled rdmsr: 0xc0011021
[  669.363923] kvm [3889]: vcpu1 unhandled rdmsr: 0xc0011021
[  669.463309] kvm [3889]: vcpu2 unhandled rdmsr: 0xc0011021
[  669.568626] kvm [3889]: vcpu3 unhandled rdmsr: 0xc0011021
[  669.672595] kvm [3889]: vcpu4 unhandled rdmsr: 0xc0011021
[  669.771024] kvm [3889]: vcpu5 unhandled rdmsr: 0xc0011021
[  669.871012] kvm [3889]: vcpu6 unhandled rdmsr: 0xc0011021
[  669.970899] kvm [3889]: vcpu7 unhandled rdmsr: 0xc0011021
[  687.205933] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078
[  687.213883] IP: [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[  687.220484] PGD 0 
[  687.222529] Oops: 0000 [#1] PREEMPT SMP 
[  687.226526] Modules linked in: tun sha256_ssse3 sha256_generic drbg binfmt_misc ipv6 vfat fat fuse dm_crypt dm_mod kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd amd64_edac_mod fam15h_power k10temp edac_mce_amd amdkfd amd_iommu_v2 radeon acpi_cpufreq
[  687.226549] CPU: 6 PID: 3891 Comm: qemu-system-x86 Not tainted 4.3.0 #1
[  687.226550] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO R2.0, BIOS 1503 01/16/2013
[  687.226552] task: ffff88042986c740 ti: ffff880415164000 task.ti: ffff880415164000
[  687.226553] RIP: 0010:[<ffffffffa02b6b94>] 
[  687.226570]  [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[  687.226572] RSP: 0018:ffff880415167dd0  EFLAGS: 00010256
[  687.226573] RAX: 0000000000000000 RBX: ffff8804157f8000 RCX: 00000000c0000102
[  687.226574] RDX: 0000000000000000 RSI: ffffffffa029fa1b RDI: ffff8804157f8000
[  687.226575] RBP: ffff880415167dd8 R08: 0000000000000001 R09: 0000000000000000
[  687.226576] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[  687.226577] R13: ffff8804157f8000 R14: 0000000000000000 R15: 0000000000000000
[  687.226579] FS:  00007ffff6353700(0000) GS:ffff88042d200000(0000) knlGS:0000000000000000
[  687.226580] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  687.226581] CR2: 0000000000000078 CR3: 00000003f75e9000 CR4: 00000000000406e0
[  687.226582] Stack:
[  687.226583]  ffff8804157f8000
[  687.226584]  ffff880415167df0
[  687.226584]  ffffffffa02b6b97
[  687.226585]  ffff8804157f8000

[  687.226586]  ffff880415167e08
[  687.226587]  ffffffffa029f8bf
[  687.226587]  0000000000000000
[  687.226588]  ffff880415167e98

[  687.226589]  ffffffffa029fa1b
[  687.226589]  ffff880427df1a00
[  687.226590]  0000000000000070
[  687.226591]  0000000000004000

[  687.226592] Call Trace:
[  687.226610]  [<ffffffffa02b6b97>] kvm_arch_vcpu_put+0x17/0x40 [kvm]
[  687.226623]  [<ffffffffa029f8bf>] vcpu_put+0x1f/0x60 [kvm]
[  687.226636]  [<ffffffffa029fa1b>] kvm_vcpu_ioctl+0x11b/0x6f0 [kvm]
[  687.226640]  [<ffffffff811932c7>] do_vfs_ioctl+0x2d7/0x530
[  687.226643]  [<ffffffff8119f179>] ? __fget_light+0x29/0x90
[  687.226646]  [<ffffffff8119356c>] SyS_ioctl+0x4c/0x90
[  687.226650]  [<ffffffff816c905b>] entry_SYSCALL_64_fastpath+0x16/0x73
[  687.226652] Code: 83 6b de e0 e9 2a ff ff ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 05 63 75 04 00 48 89 e5 53 48 89 fb <ff> 50 78 48 89 df e8 a1 fd ff ff 0f 31 48 c1 e2 20 48 09 d0 48
[  687.226691] RIP [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[  687.226708]  RSP <ffff880415167dd0>
[  687.226710] CR2: 0000000000000078
[  687.242688] ---[ end trace 41cdf7a208af97a1 ]---
[  687.242746] note: qemu-system-x86[3891] exited with preempt_count 1

[ 687.226652] Code: 83 6b de e0 e9 2a ff ff ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 05 63 75 04 00 48 89 e5 53 48 89 fb <ff> 50 78 48 89 df e8 a1 fd ff ff 0f 31 48 c1 e2 20 48 09 d0 48
All code
========
   0:   83 6b de e0             subl   $0xffffffe0,-0x22(%rbx)
   4:   e9 2a ff ff ff          jmpq   0xffffffffffffff33
   9:   0f 1f 40 00             nopl   0x0(%rax)
   d:   66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  14:   00 00 00 
  17:   0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)
  1c:   55                      push   %rbp
  1d:   48 8b 05 63 75 04 00    mov    0x47563(%rip),%rax        # 0x47587
  24:   48 89 e5                mov    %rsp,%rbp
  27:   53                      push   %rbx
  28:   48 89 fb                mov    %rdi,%rbx
  2b:*  ff 50 78                callq  *0x78(%rax)              <-- trapping instruction
  2e:   48 89 df                mov    %rbx,%rdi
  31:   e8 a1 fd ff ff          callq  0xfffffffffffffdd7
  36:   0f 31                   rdtsc  
  38:   48 c1 e2 20             shl    $0x20,%rdx
  3c:   48 09 d0                or     %rdx,%rax
  3f:   48                      rex.W

Code starting with the faulting instruction
===========================================
   0:   ff 50 78                callq  *0x78(%rax)
   3:   48 89 df                mov    %rbx,%rdi
   6:   e8 a1 fd ff ff          callq  0xfffffffffffffdac
   b:   0f 31                   rdtsc  
   d:   48 c1 e2 20             shl    $0x20,%rdx
  11:   48 09 d0                or     %rdx,%rax
  14:   48                      rex.W

00000000000120a0 <kvm_arch_vcpu_put>:
   120a0:       e8 00 00 00 00          callq  120a5 <kvm_arch_vcpu_put+0x5>
   120a5:       55                      push   %rbp
   120a6:       48 8b 05 00 00 00 00    mov    0x0(%rip),%rax        # 120ad <kvm_arch_vcpu_put+0xd>
   120ad:       48 89 e5                mov    %rsp,%rbp
   120b0:       53                      push   %rbx
   120b1:       48 89 fb                mov    %rdi,%rbx
   120b4:       ff 50 78                callq  *0x78(%rax)
   120b7:       48 89 df                mov    %rbx,%rdi
   120ba:       e8 00 00 00 00          callq  120bf <kvm_arch_vcpu_put+0x1f>
   120bf:       0f 31                   rdtsc  
   120c1:       48 c1 e2 20             shl    $0x20,%rdx
   120c5:       48 09 d0                or     %rdx,%rax
   120c8:       48 89 83 e8 32 00 00    mov    %rax,0x32e8(%rbx)
   120cf:       5b                      pop    %rbx
   120d0:       5d                      pop    %rbp
   120d1:       c3                      retq   
   120d2:       0f 1f 40 00             nopl   0x0(%rax)
   120d6:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
   120dd:       00 00 00 

        .globl  kvm_arch_vcpu_put
        .type   kvm_arch_vcpu_put, @function
kvm_arch_vcpu_put:
.LFB3643:
        .loc 1 2742 0
        .cfi_startproc
1:      call    __fentry__
        pushq   %rbp    #
        .cfi_def_cfa_offset 16
        .cfi_offset 6, -16
        .loc 1 2743 0
        movq    kvm_x86_ops(%rip), %rax # kvm_x86_ops, kvm_x86_ops
        .loc 1 2742 0
        movq    %rsp, %rbp      #,
        .cfi_def_cfa_register 6
        pushq   %rbx    #
        .cfi_offset 3, -24
        movq    %rdi, %rbx      # vcpu, vcpu
        .loc 1 2743 0
        call    *120(%rax)      # _2->vcpu_put
        .loc 1 2744 0
        movq    %rbx, %rdi      # vcpu,
        call    kvm_put_guest_fpu       #

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.