From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752167AbbCWJC6 (ORCPT ); Mon, 23 Mar 2015 05:02:58 -0400 Received: from cantor2.suse.de ([195.135.220.15]:48912 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751873AbbCWJCz (ORCPT ); Mon, 23 Mar 2015 05:02:55 -0400 Date: Mon, 23 Mar 2015 10:02:52 +0100 Message-ID: From: Takashi Iwai To: Denys Vlasenko Cc: Andy Lutomirski , Denys Vlasenko , Jiri Kosina , Linus Torvalds , Stefan Seyfried , X86 ML , LKML , Tejun Heo Subject: Re: PANIC: double fault, error_code: 0x0 in 4.0.0-rc3-2, kvm related? In-Reply-To: <550C6415.9050402@redhat.com> References: <5505400B.8050300@message-id.googlemail.com> <5509CBF7.3040602@message-id.googlemail.com> <5509F161.3010101@redhat.com> <550AABCB.9040502@redhat.com> <550C6415.9050402@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 Emacs/24.4 (x86_64-suse-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org At Fri, 20 Mar 2015 19:16:53 +0100, Denys Vlasenko wrote: > > Hi, > > This particular crash was hard to diagnose because of two reasons: > > * CPU would happily use userspace RSP in kernel mode. > Crash comes only later, when we run off the stack. > We lose information when it started. > > * Kernel's error handling code is ill prepared for RSP pointing > to user stack. So we take another page fault trying > to dump stack. > > I prepared a patch which helps with both problems. > > For testing, I inserted an invalid instruction right before SYSRET > to induce a similar bug, and booted resulting kernel in qemu. > > Before my patch, double fault output starts like this: > > [ 0.715216] PANIC: double fault, error_code: 0x0 > [ 0.716033] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #7 > [ 0.716033] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 0.716033] task: ffff880007588000 ti: ffff880007590000 task.ti: ffff880007590000 > [ 0.716033] RIP: 0010:[] [] do_error_trap+0x47/0x120 > [ 0.716033] RSP: 0018:00007ffd89e7ffb8 EFLAGS: 00010006 > > The key here is that it doesn't show at which RIP we took the first > "bad" exception. The only useful detail visible here is bad RSP. > "do_error_trap+0x47" is useless. > > After the patch, the very moment of "bad" exception is caught: > > [ 0.666758] Exception on user stack 00007ffc1fd0c388: RSP: 0018:00007ffc1fd0c3b0 EFLAGS: 00010006 > [ 0.667285] RIP: 0010:[] [] ret_from_sys_call+0x5f/0x67 > [ 0.667285] PANIC: double fault, error_code: 0xffffffffffffffff > [ 0.667285] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #13 > [ 0.667285] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 0.667285] task: ffff880007588000 ti: ffff880007590000 task.ti: ffff880007590000 > [ 0.667285] RIP: 0010:[] [] ret_from_sys_call+0x5f/0x67 > [ 0.667285] RSP: 0018:00007ffc1fd0c3b0 EFLAGS: 00010006 > > The exception happened at "ret_from_sys_call+0x5f". > We also won't take another page fault any more, > output proceeds like this: > > ... > [ 0.667285] RAX: 0000000007a00000 RBX: 00007ffc1fd0c4e0 RCX: 00000000c0000101 > [ 0.667285] RDX: 00000000ffff8800 RSI: 0000000000005401 RDI: 00007ffc1fd0c388 > [ 0.667285] RBP: 00007ffc1fd0c570 R08: 0000000000000010 R09: 0000000000000000 > [ 0.667285] R10: 00007ffc1fd0c650 R11: 0000000000000202 R12: 0000000000000120 > [ 0.667285] R13: 00000000005f7b78 R14: 0000000000000000 R15: 00000000004c9d44 > [ 0.667285] FS: 0000000000000000(0000) GS:ffff880007a00000(0000) knlGS:0000000000000000 > [ 0.667285] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 0.667285] CR2: 00000000004ad1e4 CR3: 0000000000101000 CR4: 00000000000007f0 > [ 0.667285] Stack: > [ 0.667285] 0000000000000018 00007ffc1fd0c490 00007ffc1fd0c3d0 0000000000000000 > [ 0.667285] 0000000000000000 0000000000000000 00007ffc1fd0c490 0000000000000000 > [ 0.667285] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > [ 0.667285] Call Trace: > [ 0.667285] > [ 0.667285] Code: 8b 44 24 50 48 8b 54 24 60 48 8b 74 24 68 48 8b 7c 24 70 48 8b 8c 24 80 00 00 00 4c 8b 9c 24 90 00 00 00 48 8b a4 24 98 00 00 00 <0f> 0b 0f 01 f8 48 0f 07 48 c7 84 24 a0 00 00 00 2b 00 00 00 48 > [ 0.667285] Kernel panic - not syncing: Machine halted. > [ 0.667285] CPU: 0 PID: 1 Comm: init Not tainted 4.0.0-rc2+ #13 > [ 0.667285] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 0.667285] ffffffffffffffff ffff880007593e28 ffffffff81789625 ffff880007588000 > [ 0.667285] ffffffff81a3b181 ffff880007593ea8 ffffffff817840aa ffff880007590000 > [ 0.667285] 0000000000000008 ffff880007593eb8 ffff880007593e58 0000000000000001 > [ 0.667285] Call Trace: > [ 0.667285] [] dump_stack+0x4c/0x65 > [ 0.667285] [] panic+0xc6/0x1ff > [ 0.667285] [] df_debug+0x35/0x40 > [ 0.667285] [] do_double_fault+0x87/0x100 > [ 0.667285] [] do_userpsace_rsp_in_kernel+0x107/0x140 > [ 0.667285] [] ? ret_from_sys_call+0x5f/0x67 > [ 0.667285] [] userpsace_rsp_in_kernel+0x39/0x40 > [ 0.667285] [] ? ret_from_sys_call+0x5f/0x67 > [ 0.667285] Kernel Offset: disabled > [ 0.667285] Rebooting in 1 seconds.. > > Takashi, are you willing to reproduce the panic one more time, > with this patch? I would like to see whether oops messages > are more informative with it. It can't be applied to 4.0-rc5, unfortunately. arch/x86/kernel/entry_64.S: Assembler messages: arch/x86/kernel/entry_64.S:1725: Error: no such instruction: `alloc_pt_gpregs_on_stack' arch/x86/kernel/entry_64.S:1716: Error: invalid operands (*UND* and *UND* sections) for `+' scripts/Makefile.build:294: recipe for target 'arch/x86/kernel/entry_64.o' failed Takashi