Re: PANIC: double fault, error_code: 0x0 in 4.0.0-rc3-2, kvm related?

[Takashi Iwai <tiwai@suse.de>]

At Fri, 20 Mar 2015 19:16:53 +0100,
Denys Vlasenko wrote:
> 
> Hi,
> 
> This particular crash was hard to diagnose because of two reasons:
> 
> * CPU would happily use userspace RSP in kernel mode.
>   Crash comes only later, when we run off the stack.
>   We lose information when it started.
> 
> * Kernel's error handling code is ill prepared for RSP pointing
>   to user stack. So we take another page fault trying
>   to dump stack.
> 
> I prepared a patch which helps with both problems.
> 
> For testing, I inserted an invalid instruction right before SYSRET
> to induce a similar bug, and booted resulting kernel in qemu.
> 
> Before my patch, double fault output starts like this:
> 
> [    0.715216] PANIC: double fault, error_code: 0x0
> [    0.716033] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #7
> [    0.716033] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [    0.716033] task: ffff880007588000 ti: ffff880007590000 task.ti: ffff880007590000
> [    0.716033] RIP: 0010:[<ffffffff81017057>]  [<ffffffff81017057>] do_error_trap+0x47/0x120
> [    0.716033] RSP: 0018:00007ffd89e7ffb8  EFLAGS: 00010006
> 
> The key here is that it doesn't show at which RIP we took the first
> "bad" exception. The only useful detail visible here is bad RSP.
> "do_error_trap+0x47" is useless.
> 
> After the patch, the very moment of "bad" exception is caught:
> 
> [    0.666758] Exception on user stack 00007ffc1fd0c388: RSP: 0018:00007ffc1fd0c3b0  EFLAGS: 00010006
> [    0.667285] RIP: 0010:[<ffffffff81793688>]  [<ffffffff81793688>] ret_from_sys_call+0x5f/0x67
> [    0.667285] PANIC: double fault, error_code: 0xffffffffffffffff
> [    0.667285] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #13
> [    0.667285] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [    0.667285] task: ffff880007588000 ti: ffff880007590000 task.ti: ffff880007590000
> [    0.667285] RIP: 0010:[<ffffffff81793688>]  [<ffffffff81793688>] ret_from_sys_call+0x5f/0x67
> [    0.667285] RSP: 0018:00007ffc1fd0c3b0  EFLAGS: 00010006
> 
> The exception happened at "ret_from_sys_call+0x5f".
> We also won't take another page fault any more,
> output proceeds like this:
> 
> ...
> [    0.667285] RAX: 0000000007a00000 RBX: 00007ffc1fd0c4e0 RCX: 00000000c0000101
> [    0.667285] RDX: 00000000ffff8800 RSI: 0000000000005401 RDI: 00007ffc1fd0c388
> [    0.667285] RBP: 00007ffc1fd0c570 R08: 0000000000000010 R09: 0000000000000000
> [    0.667285] R10: 00007ffc1fd0c650 R11: 0000000000000202 R12: 0000000000000120
> [    0.667285] R13: 00000000005f7b78 R14: 0000000000000000 R15: 00000000004c9d44
> [    0.667285] FS:  0000000000000000(0000) GS:ffff880007a00000(0000) knlGS:0000000000000000
> [    0.667285] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [    0.667285] CR2: 00000000004ad1e4 CR3: 0000000000101000 CR4: 00000000000007f0
> [    0.667285] Stack:
> [    0.667285]  0000000000000018 00007ffc1fd0c490 00007ffc1fd0c3d0 0000000000000000
> [    0.667285]  0000000000000000 0000000000000000 00007ffc1fd0c490 0000000000000000
> [    0.667285]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    0.667285] Call Trace:
> [    0.667285]  <UNK>
> [    0.667285] Code: 8b 44 24 50 48 8b 54 24 60 48 8b 74 24 68 48 8b 7c 24 70 48 8b 8c 24 80 00 00 00 4c 8b 9c 24 90 00 00 00 48 8b a4 24 98 00 00 00 <0f> 0b 0f 01 f8 48 0f 07 48 c7 84 24 a0 00 00 00 2b 00 00 00 48
> [    0.667285] Kernel panic - not syncing: Machine halted.
> [    0.667285] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #13
> [    0.667285] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [    0.667285]  ffffffffffffffff ffff880007593e28 ffffffff81789625 ffff880007588000
> [    0.667285]  ffffffff81a3b181 ffff880007593ea8 ffffffff817840aa ffff880007590000
> [    0.667285]  0000000000000008 ffff880007593eb8 ffff880007593e58 0000000000000001
> [    0.667285] Call Trace:
> [    0.667285]  [<ffffffff81789625>] dump_stack+0x4c/0x65
> [    0.667285]  [<ffffffff817840aa>] panic+0xc6/0x1ff
> [    0.667285]  [<ffffffff81059ee5>] df_debug+0x35/0x40
> [    0.667285]  [<ffffffff81017e37>] do_double_fault+0x87/0x100
> [    0.667285]  [<ffffffff81017fb7>] do_userpsace_rsp_in_kernel+0x107/0x140
> [    0.667285]  [<ffffffff81793688>] ? ret_from_sys_call+0x5f/0x67
> [    0.667285]  [<ffffffff81795b49>] userpsace_rsp_in_kernel+0x39/0x40
> [    0.667285]  [<ffffffff81793688>] ? ret_from_sys_call+0x5f/0x67
> [    0.667285] Kernel Offset: disabled
> [    0.667285] Rebooting in 1 seconds..
> 
> Takashi, are you willing to reproduce the panic one more time,
> with this patch? I would like to see whether oops messages
> are more informative with it.

It can't be applied to 4.0-rc5, unfortunately.

arch/x86/kernel/entry_64.S: Assembler messages:
arch/x86/kernel/entry_64.S:1725: Error: no such instruction: `alloc_pt_gpregs_on_stack'
arch/x86/kernel/entry_64.S:1716: Error: invalid operands (*UND* and *UND* sections) for `+'
scripts/Makefile.build:294: recipe for target 'arch/x86/kernel/entry_64.o' failed


Takashi