From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754248Ab1JBTbu (ORCPT ); Sun, 2 Oct 2011 15:31:50 -0400 Received: from ogre.sisk.pl ([217.79.144.158]:43272 "EHLO ogre.sisk.pl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751070Ab1JBTbm (ORCPT ); Sun, 2 Oct 2011 15:31:42 -0400 From: "Rafael J. Wysocki" To: "Srivatsa S. Bhat" Subject: Re: [BUG] CPU hotplug, freezer: Freezing of tasks failed after 20.00 seconds Date: Sun, 2 Oct 2011 21:33:59 +0200 User-Agent: KMail/1.13.6 (Linux/3.1.0-rc8+; KDE/4.6.0; x86_64; ; ) Cc: Tejun Heo , linux-kernel@vger.kernel.org, Linux PM mailing list , oleg@redhat.com, arnd@arndb.de, Christoph Lameter , Pekka Enberg References: <1313763382-12341-1-git-send-email-tj@kernel.org> <201109060801.09210.rjw@sisk.pl> <4E88B7E6.7080402@linux.vnet.ibm.com> In-Reply-To: <4E88B7E6.7080402@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201110022134.00022.rjw@sisk.pl> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sunday, October 02, 2011, Srivatsa S. Bhat wrote: > On 09/06/2011 11:31 AM, Rafael J. Wysocki wrote: > > On Tuesday, September 06, 2011, Tejun Heo wrote: > >> Hello, again. > >> > >> On Mon, Sep 05, 2011 at 11:15:12PM +0900, Tejun Heo wrote: > >>>> Freezing of tasks failed after 20.01 seconds (2 tasks refusing to freeze, wq_busy=0): > >>>> invert_cpu_stat D 0000000000000000 5304 20435 17329 0x00000084 > >>>> ffff8801f367bab8 0000000000000046 ffff8801f367bfd8 00000000001d3a00 > >>>> ffff8801f367a010 00000000001d3a00 00000000001d3a00 00000000001d3a00 > >>>> ffff8801f367bfd8 00000000001d3a00 ffff880414cc6840 ffff8801f36783c0 > >>>> Call Trace: > >>>> [] schedule_timeout+0x235/0x320 > >>>> [] wait_for_common+0x11b/0x170 > >>>> [] wait_for_completion+0x1d/0x20 > >>>> [] _request_firmware+0x156/0x2c0 > >>>> [] request_firmware+0x16/0x20 > >>>> [] request_microcode_fw+0x70/0xf0 [microcode] > >>>> [] microcode_init_cpu+0xc0/0x100 [microcode] > >>>> [] mc_cpu_callback+0x7c/0x11f [microcode] > >>>> [] notifier_call_chain+0x94/0xd0 > >>>> [] __raw_notifier_call_chain+0xe/0x10 > >>>> [] __cpu_notify+0x20/0x40 > >>>> [] _cpu_up+0xc7/0x10e > >>>> [] cpu_up+0xd9/0xec > >>>> [] store_online+0x99/0xd0 > >>>> [] sysdev_store+0x20/0x30 > >>>> [] sysfs_write_file+0xe6/0x170 > >>>> [] vfs_write+0xd0/0x1a0 > >>>> [] sys_write+0x54/0xa0 > >>>> [] system_call_fastpath+0x16/0x1b > >>> > >>> So, this task is trying to bring a CPU up, which triggers firmware > >>> helper to load microcode. Firmware class currently sleeps > >>> non-interruptibly to wait for firmware load to complete, which is > >>> performed by another userland task. Now, the PM freezer doesn't > >>> assume that there will be non-freezable wait dependencies among > >>> userland tasks. It only knows two levels - userland and kernel tasks > >>> - and assumes that the former group may have non-freezable wait > >>> dependency on the latter but there's no such dependency among each > >>> group itself. If there's such dependency, PM freezer may fail, which > >>> is what happened here. > >>> > >>> ie. the firmware loader userland process got frozen first. > >>> invert_cpu_stat trying to bring up CPU was waiting for the firmware > >>> loader to finish in non-interruptible sleep, so the freezer couldn't > >>> proceed. > >> > >> Hmmm... I went through the code again and usermodehelper_disable() > >> seems to be there to prevent deadlocks like this. usermode helpers > >> are drained & plugged before freezing is tried. Rafael, the above > >> shouldn't be happening, right? > > > > No, it shouldn't in theory, but I'm not sure any more after the recent > > modifications of firmware loading related to the initialization. I'll have > > a closer look tomorrow. > > > > Hi, > I have posted a fix for this bug at https://lkml.org/lkml/2011/10/2/142 > With my fix, the numerous "WARNING"s at drivers/base/firmware_class.c > disappear and the task freezing failures are fixed too. > I have tested this for about 10-12 hours (much more time than what was > necessary to reproduce the bug earlier). I saw the fix, thanks for it, Rafael