From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752426AbbCWOpN (ORCPT ); Mon, 23 Mar 2015 10:45:13 -0400 Received: from mga09.intel.com ([134.134.136.24]:49752 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752113AbbCWOpK (ORCPT ); Mon, 23 Mar 2015 10:45:10 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.11,452,1422950400"; d="scan'208";a="669273722" Date: Mon, 23 Mar 2015 14:45:09 +0000 (UTC) From: Keith Busch X-X-Sender: vmware@localhost.lm.intel.com To: steven@uplinklabs.net cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: 4.0.0-rc4 NVMe NULL pointer dereference and hang In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (LNX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, 22 Mar 2015, Steven Noonan wrote: > This happens on boot, and then eventually results in an RCU stall. > > [ 8.047533] nvme 0000:05:00.0: Device not ready; aborting initialisation > > Note that the above is expected with this hardware (long story). > Although 3.19.x prints the above and then continues gracefully, 4.0-rc > breaks immediately after the above message: Thanks for the notice. I CC'ed the linux-nvme mailing list. Since your device failed to become ready (this is expected, you say? ok, I won't ask. :)), it triggered recovery action that assumed it's list head was initialized once before. It's a one-line fix: --- diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c index 3b43897..ab7c847 100644 --- a/drivers/block/nvme-core.c +++ b/drivers/block/nvme-core.c @@ -3007,6 +3007,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) } get_device(dev->device); + INIT_LIST_HEAD(&dev->node); INIT_WORK(&dev->probe_work, nvme_async_probe); schedule_work(&dev->probe_work); return 0; -- I'll coerce someone to merge this for rc6 today. > [ 8.054306] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 > [ 8.062155] IP: [] nvme_dev_list_remove+0x24/0xa0 [nvme] > [ 8.069043] PGD 0 > [ 8.071067] Oops: 0002 [#1] SMP > [ 8.074332] Modules linked in: ahci libahci libata ehci_pci ehci_hcd scsi_mod usbcore usb_common nvme i915 intel_gtt i2c_algo_bit video drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button > [ 8.094244] CPU: 4 PID: 632 Comm: kworker/u288:1 Not tainted 4.0.0-rc4-00347-gb87444a2 #5 > [ 8.109878] Workqueue: nvme nvme_reset_workfn [nvme] > [ 8.114852] task: ffff881f98271d70 ti: ffff881f982b8000 task.ti: ffff881f982b8000 > [ 8.122321] RIP: 0010:[] [] nvme_dev_list_remove+0x24/0xa0 [nvme] > [ 8.131624] RSP: 0000:ffff881f982bbd18 EFLAGS: 00010246 > [ 8.136930] RAX: 0000000000000000 RBX: ffff883f63f84800 RCX: ffff88bf66e6a418 > [ 8.144052] RDX: 0000000000000000 RSI: 0000000000000120 RDI: ffffffffa0269848 > [ 8.151171] RBP: ffff881f982bbd28 R08: ffff881f982b8000 R09: 0000000000000001 > [ 8.158288] R10: 0000000000000086 R11: 0000000000000020 R12: ffff883f63f84800 > [ 8.165411] R13: ffff88bf66e6a400 R14: ffff88df627ff900 R15: 0000000000001000 > [ 8.172530] FS: 0000000000000000(0000) GS:ffff883f7f880000(0000) knlGS:0000000000000000 > [ 8.180600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 8.186337] CR2: 0000000000000008 CR3: 000001007ea0c000 CR4: 00000000001406e0 > [ 8.193458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 8.200574] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 8.207693] Stack: > [ 8.209705] ffff881f982bbd28 ffff883f63f84978 ffff881f982bbdc8 ffffffffa026005e > [ 8.217150] ffff883f7f894300 00000000ffff8de0 ffff881f982bbd98 ffffffff810a65e1 > [ 8.224600] ffff881f982bbdd8 ffffffff810a9943 ffff881f982bbd98 ffff881f982bbdd0 > [ 8.232049] Call Trace: > [ 8.234500] [] nvme_dev_shutdown+0x1e/0x430 [nvme] > [ 8.240943] [] ? put_prev_entity+0x31/0x350 > [ 8.246772] [] ? pick_next_task_fair+0x103/0x4e0 > [ 8.253046] [] ? __switch_to+0x175/0x5c0 > [ 8.258607] [] nvme_reset_failed_dev+0x1e/0x100 [nvme] > [ 8.265378] [] nvme_reset_workfn+0xf/0x20 [nvme] > [ 8.271649] [] process_one_work+0x14e/0x400 > [ 8.277472] [] worker_thread+0x5b/0x530 > [ 8.282943] [] ? rescuer_thread+0x3a0/0x3a0 > [ 8.288778] [] kthread+0xc9/0xe0 > [ 8.293649] [] ? kthread_stop+0x100/0x100 > [ 8.299322] [] ret_from_fork+0x58/0x90 > [ 8.304711] [] ? kthread_stop+0x100/0x100 > [ 8.310357] Code: 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 c7 c7 48 98 26 a0 48 83 ec 08 e8 c3 23 2e e1 48 8b 13 48 8b 43 08 <48> 89 42 08 48 89 10 48 89 1b 48 81 3d 77 ae 00 00 a0 94 26 a0 > [ 8.330295] RIP [] nvme_dev_list_remove+0x24/0xa0 [nvme] > [ 8.337258] RSP > [ 8.340739] CR2: 0000000000000008 > [ 8.344056] ---[ end trace 70831a936042aa41 ]---