From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751964AbXCJCTV (ORCPT ); Fri, 9 Mar 2007 21:19:21 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752777AbXCJCTV (ORCPT ); Fri, 9 Mar 2007 21:19:21 -0500 Received: from smtp.osdl.org ([65.172.181.24]:57304 "EHLO smtp.osdl.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751964AbXCJCTT (ORCPT ); Fri, 9 Mar 2007 21:19:19 -0500 Date: Fri, 9 Mar 2007 18:18:51 -0800 From: Andrew Morton To: Michal Piotrowski Cc: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Oleg Nesterov , Gautham R Shenoy Subject: Re: 2.6.21-rc3-mm1 Message-Id: <20070309181851.e96d468a.akpm@linux-foundation.org> In-Reply-To: <45F07715.9090502@googlemail.com> References: <20070307201839.6f45735b.akpm@linux-foundation.org> <45F07715.9090502@googlemail.com> X-Mailer: Sylpheed version 2.2.4 (GTK+ 2.8.19; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org > On Thu, 08 Mar 2007 21:50:29 +0100 Michal Piotrowski wrote: > Andrew Morton napisał(a): > > Temporarily at > > > > http://userweb.kernel.org/~akpm/2.6.21-rc3-mm1/ > > > > Will appear later at > > > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc3/2.6.21-rc3-mm1/ > > > > cpu_hotplug (AutoTest) hangs at this > > ============================================= > [ INFO: possible recursive locking detected ] > 2.6.21-rc3-mm1 #2 > --------------------------------------------- > sh/7213 is trying to acquire lock: > (sched_hotcpu_mutex){--..}, at: [] mutex_lock+0x1c/0x1f > > but task is already holding lock: > (sched_hotcpu_mutex){--..}, at: [] mutex_lock+0x1c/0x1f > > other info that might help us debug this: > 4 locks held by sh/7213: > #0: (cpu_add_remove_lock){--..}, at: [] mutex_lock+0x1c/0x1f > #1: (sched_hotcpu_mutex){--..}, at: [] mutex_lock+0x1c/0x1f > #2: (cache_chain_mutex){--..}, at: [] mutex_lock+0x1c/0x1f > #3: (workqueue_mutex){--..}, at: [] mutex_lock+0x1c/0x1f That's pretty useless, isn't it? We need to know the mutex_lock() caller here. > stack backtrace > [] show_trace_log_lvl+0x1a/0x2f > [] show_trace+0x12/0x14 > [] dump_stack+0x16/0x18 > [] __lock_acquire+0x1aa/0xceb > [] lock_acquire+0x79/0x93 > [] __mutex_lock_slowpath+0x107/0x349 > [] mutex_lock+0x1c/0x1f > [] sched_getaffinity+0x14/0x91 > [] __synchronize_sched+0x11/0x5f > [] detach_destroy_domains+0x2c/0x30 > [] update_sched_domains+0x27/0x3a > [] notifier_call_chain+0x2b/0x4a > [] __raw_notifier_call_chain+0x19/0x1e > [] _cpu_down+0x70/0x282 > [] cpu_down+0x26/0x38 > [] store_online+0x27/0x5a > [] sysdev_store+0x20/0x25 > [] sysfs_write_file+0xc1/0xe9 > [] vfs_write+0xd1/0x15a > [] sys_write+0x3d/0x72 > [] syscall_call+0x7/0xb > > l *0xc033883a > 0xc033883a is in mutex_lock (/mnt/md0/devel/linux-mm/kernel/mutex.c:92). > 87 /* > 88 * The locking fastpath is the 1->0 transition from > 89 * 'unlocked' into 'locked' state. > 90 */ > 91 __mutex_fastpath_lock(&lock->count, __mutex_lock_slowpath); > 92 } > 93 > 94 EXPORT_SYMBOL(mutex_lock); > 95 > 96 static void fastcall noinline __sched > > I didn't test other -mm's with this test. > > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc3-mm1/console.log > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc3-mm1/mm-config I can't immediately spot the bug. Probably it's caused by rcu-preempt's changes to synchronize_sched(): that function now does a heap more than it used to, including taking sched_hotcpu_muex. So, what to do about this. Paul, I'm thinking that I should drop rcu-preempt for now - I don't think we ended up being able to identify any particular benefit which it brings to current mainline, and I suspect that things will become simpler if/when we start using the process freezer for CPU hotplug.