From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03921C04ABB for ; Tue, 11 Sep 2018 10:54:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BC62020866 for ; Tue, 11 Sep 2018 10:54:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BC62020866 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727007AbeIKPw7 (ORCPT ); Tue, 11 Sep 2018 11:52:59 -0400 Received: from cloudserver094114.home.pl ([79.96.170.134]:55034 "EHLO cloudserver094114.home.pl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726670AbeIKPw7 (ORCPT ); Tue, 11 Sep 2018 11:52:59 -0400 Received: from 79.184.255.178.ipv4.supernova.orange.pl (79.184.255.178) (HELO aspire.rjw.lan) by serwer1319399.home.pl (79.96.170.134) with SMTP (IdeaSmtpServer 0.83.123) id 4ea1e80342928395; Tue, 11 Sep 2018 12:54:10 +0200 From: "Rafael J. Wysocki" To: Greg Kroah-Hartman , Takashi Iwai Cc: "Rafael J. Wysocki" , James Wang , Borislav Petkov , linux-kernel@vger.kernel.org, Pingfan Liu Subject: Re: [REGRESSION] Errors at reboot after 722e5f2b1eec Date: Tue, 11 Sep 2018 12:51:32 +0200 Message-ID: <2580518.z31CQjiopR@aspire.rjw.lan> In-Reply-To: <20180911093324.GB10436@kroah.com> References: <20180911093324.GB10436@kroah.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tuesday, September 11, 2018 11:33:24 AM CEST Greg Kroah-Hartman wrote: > On Tue, Sep 11, 2018 at 10:17:44AM +0200, Takashi Iwai wrote: > > [ seems like my previous post didn't go out properly; if you have > > already received it, please discard this one ] > > Sorry, I got it, it's just in my large queue :( > > > Hi Rafael, Greg, > > > > James Wang reported on SUSE bugzilla that his machine spews many > > AMD-Vi errors at reboot like: > > > > [ 154.907879] systemd-shutdown[1]: Detaching loop devices. > > [ 154.954583] kvm: exiting hardware virtualization > > [ 154.999953] usb 5-2: USB disconnect, device number 2 > > [ 155.025278] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.081360] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.136778] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.191772] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.247055] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.302614] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.358996] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.392155] usb 4-2: new full-speed USB device number 2 using ohci-pci > > [ 155.413752] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.413762] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.560307] ohci-pci 0000:00:12.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.616039] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.667843] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.719497] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.772697] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.823919] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.875490] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.927258] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 155.979318] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 156.031813] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 156.084293] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.1 domain=0x0006 address=0x0000000000000080 flags=0x0020] > > [ 156.272157] reboot: Restarting system > > [ 156.290316] reboot: machine restart > > > > And, James bisected and spotted that it's introduced by the commit > > 722e5f2b1eec ("driver core: Partially revert "driver core: correct > > device's shutdown order""). Reverting the commit fixes the problem. Well, has anyone tried to understand why this is so? It looks like the probe-time reordering of the devices_kset list worked around some init-time dependency issue, but we can't reorder devices_kset then as it breaks parent-child ordering in general. > > He mentioned about Uncorrectable Machine Check Exception seen at > > shutdown, too, where it doesn't appear after the revert. (Though, > > it's not sure whether it's really relevant.) > > > > The errors are clearly related with the USB device (a KVM device, > > IIRC), and the errors are not seen if the USB device is disconnected. > > > > We experienced this at first with SLE15 kernel (4.12 with backports), > > but later the same issue was confirmed on 4.18.y and 4.19-rc2. Also, > > it's confirmed that revert works on the upstream kernels, too. > > > > Does this hit your radar? > > Ugh, no, I haven't heard of this before, Rafael? > > So the need for the revert fixes some machines, but others need the > patch, this isn't going to be fun :( We need to understand what's going on on the machines that stopped working and fix them. Calling devices_kset_move_last() from really_probe() is clearly incorrect and restoring it would be a mistake IMO. BTW, there is a series of patches from Pingfan Liu: https://patchwork.kernel.org/project/linux-pm/list/?series=9535 that may help in principle, so any chance to try them on the affected systems? Thanks, Rafael