Re: [PATCH] tracing: use raw spinlocks instead of spinlocks

[Ingo Molnar <mingo@elte.hu>]

* Frédéric Weisbecker <fweisbec@gmail.com> wrote:

> 2008/11/3 Ingo Molnar <mingo@elte.hu>:
> > Frederic, do you have trouble finding the source of the deadlock? In
> > theory the NMI watchdog should be able to punch through it. (if not
> > then we need to improve things so that it can)
> 
> Ok...actually I still have a deadlock. Or it seems to. I launch the 
> return tracer and then...the system can block after several seconds, 
> typically after I try to write some characters on the terminal.
> 
> It happens more easily on SMP.
> 
> I enabled all I could for lock debugging, but I don't have any 
> errors printed... Lockdep uses an NMI wachdog?

lockdep works via a completely different principle: it instruments all 
the actual lock acquire/release calls and builds a graph of lock 
dependencies in the system, as it happens.

It also guarantees that all the observed locking rules are followed 
(i.e. it proves that as long as you dont get any messages from 
lockdep, all the locking patterns are mathematically safe).

So a lockdep message will most of the time occur much easier than a 
real lockup would occur - as lockdep only needs to observe 
inconsistent locking patterns to prove that a lockup _could_ occur.

The NMI watchdog just observes the system and complains if it sees 
hardirqs not progressing (i.e. a hard lockup). It will detect anything 
that causes a hard lockup. (assuming that the NMI watchdog itself is 
not locked up)

Regarding your lockup ... it's quite hard. Maybe you can get more 
output out of the system by using:

  earlyprintk=vga,keep

plus disablig regular tty output. (i.e. not passing any 'console=tty' 
line to the kernel bootup.)

this way you wont get any normal printk activities (which might lock 
up), you should only get the very simple early-printk output.

it is also probably useful to mark the early-printk code notrace in 
arch/x86/kernel/Makefile, via the patch below. (i've just applied to 
to tip/x86/debug and pushed it out into tip/master)

... but that's still not a guarantee that you'll be able to debug it 
this way :-/ Bootstrapping something this intrusive and wide is one of 
the hardest kernel development activities possible.

	Ingo

---------------->
>From 6cf87efbc7a3676e0ad7c9622ec6aec244a593bc Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@elte.hu>
Date: Tue, 4 Nov 2008 10:42:23 +0100
Subject: [PATCH] x86 debug: mark early_printk.o as notrace

Impact: do not do function-tracing in the early-printk code

this is useful when earlyprintk=vga,keep is used to debug tracer
plugins.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/Makefile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index e489ff9..943fe60 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -12,6 +12,7 @@ CFLAGS_REMOVE_tsc.o = -pg
 CFLAGS_REMOVE_rtc.o = -pg
 CFLAGS_REMOVE_paravirt-spinlocks.o = -pg
 CFLAGS_REMOVE_ftrace.o = -pg
+CFLAGS_REMOVE_early_printk.o = -pg
 endif
 
 #