From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752335AbcADT1o (ORCPT ); Mon, 4 Jan 2016 14:27:44 -0500 Received: from casper.infradead.org ([85.118.1.10]:32958 "EHLO casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751408AbcADT1l (ORCPT ); Mon, 4 Jan 2016 14:27:41 -0500 Date: Mon, 4 Jan 2016 20:27:37 +0100 From: Peter Zijlstra To: Andy Lutomirski Cc: Dominique Martinet , Thomas Gleixner , Ingo Molnar , Al Viro , "linux-kernel@vger.kernel.org" , V9FS Developers , Linux FS Devel Subject: Re: [V9fs-developer] Hang triggered by udev coldplug, looks like a race Message-ID: <20160104192737.GJ6344@twins.programming.kicks-ass.net> References: <20151208023331.GJ20997@ZenIV.linux.org.uk> <20151209062316.GA29917@nautica> <20151209064542.GW20997@ZenIV.linux.org.uk> <20151224105149.GA24863@nautica> <20160104155915.GI6344@twins.programming.kicks-ass.net> <20160104160924.GA12687@u-isr-ctg-01> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2012-12-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 04, 2016 at 11:07:43AM -0800, Andy Lutomirski wrote: > On Mon, Jan 4, 2016 at 8:09 AM, Dominique Martinet > wrote: > > Peter Zijlstra wrote on Mon, Jan 04, 2016 at 04:59:15PM +0100: > >> On Tue, Dec 29, 2015 at 10:43:26PM -0800, Andy Lutomirski wrote: > >> > [add cc's] > >> > > >> > Hi scheduler people: > >> > > >> > This is relatively easy for me to reproduce. Any hints for debugging > >> > it? Could we really have a bug in which processes that are > >> > schedulable as a result of mutex unlock aren't always reliably > >> > scheduled? > >> > >> I would expect that to cause wide-spread fail, then again, virt is known > >> to tickle timing issues that are improbable on actual hardware so > >> anything is possible. > >> > >> Does it reproduce with DEBUG_MUTEXES set? (I'm not seeing a .config > >> here). > > > > The config has CONFIG_DEBUG_MUTEXES=y > > > > It got attached a while ago, reposting it here. > > > >> If its really easy you could start by tracing events/sched/sched_switch > >> events/sched/sched_wakeup, those would be the actual scheduling events. > > > > I'm sure I've missed something in /Documentation but I'm not aware how > > to trace these? (I'm happy to save Andy some precious time as I've got a > > reproducer all set up now) > > My reproducer, at least, would make this tricky -- the system ends up > mostly hung, so I don't know how I'd read out the result. Maybe I'd > try to get something to dump the ftrace buffer to serial console after > a delay and stick all that in initramfs where it wouldn't get stuck > behind the same mutex as everything else. > > Or is there a way to tell the kernel to do that for us? If you can generate a core, I think crash knows how to read the ftrace buffers from it. http://people.redhat.com/anderson/extensions/trace_help_trace.html But yes, you can use one of the watchdog thingies to dump the buffers over 'serial' too, but I suspect that will take a little longer, even with virtual serial ports.