From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935646AbXGWU11 (ORCPT ); Mon, 23 Jul 2007 16:27:27 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1760862AbXGWU1R (ORCPT ); Mon, 23 Jul 2007 16:27:17 -0400 Received: from gateway-1237.mvista.com ([63.81.120.158]:4969 "EHLO gateway-1237.mvista.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751345AbXGWU1Q (ORCPT ); Mon, 23 Jul 2007 16:27:16 -0400 Subject: Re: 2.6.22.1-rt4 lockups From: Daniel Walker To: Rui Nuno Capela Cc: Thomas Gleixner , Ingo Molnar , LKML , RT-Users In-Reply-To: <1185206905.2573.52.camel@imap.mvista.com> References: <1184325752.12353.312.camel@chaos> <46A283A1.6030005@rncbc.org> <1185206905.2573.52.camel@imap.mvista.com> Content-Type: text/plain Date: Mon, 23 Jul 2007 13:15:46 -0700 Message-Id: <1185221746.2573.142.camel@imap.mvista.com> Mime-Version: 1.0 X-Mailer: Evolution 2.10.3 (2.10.3-1.fc7) Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2007-07-23 at 09:08 -0700, Daniel Walker wrote: > On Sat, 2007-07-21 at 23:07 +0100, Rui Nuno Capela wrote: > > > Call Trace: > > [] show_trace_log_lvl+0x1a/0x30 > > [] show_stack_log_lvl+0xb6/0xe0 > > [] show_registers+0x201/0x330 > > [] die+0x118/0x260 > > [] do_page_fault+0x193/0x600 > > [] error_code+0x72/0x78 > > [] activate_task+0x4f/0xb0 > > [] try_to_wake_up+0x2bd/0x420 > > [] wake_up_process_mutex+0x19/0x20 > > [] wakeup_next_waiter+0xec/0x1a0 > > [] rt_spin_lock_slowunlock+0x4c/0x70 > > [] rt_spin_unlock+0x26/0x30 > > [] put_zone_pcp+0x14/0x20 > > [] get_page_from_freelist+0x145/0x380 > > [] __alloc_pages+0x54/0x2d0 > > [] __handle_mm_fault+0x7dd/0x9a0 > > [] do_page_fault+0x2f8/0x600 > > [] error_code+0x72/0x78 > > ======================= > > I was able to reproduce a similar looking hang when I combine kernbench > running with another load (I used ltpstress.sh from LTP) .. > > I'm debugging it now .. It looks like sched_class->enqueue_task() is NULL and that's why the system hangs .. The reason why that happens is because check_pgt_cache() is called from the idle thread, and with PREEMPT_RT check_pgt_cache() locks at least one mutex .. Once the idle thread is on a wait_list, as soon as it's woke by the mutex owner the system will crash in enqueue_task. Since the idle thread has a NULL sched_class->enqueue_task .. check_pgt_cache() is already getting called from the desched_thread() , so I think it could just be removed from i386 cpu_idle(). Anyone have comments on the theory above? Daniel