From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758171AbZEDRWT (ORCPT ); Mon, 4 May 2009 13:22:19 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756959AbZEDRWG (ORCPT ); Mon, 4 May 2009 13:22:06 -0400 Received: from xc.sipsolutions.net ([83.246.72.84]:51904 "EHLO sipsolutions.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751282AbZEDRWD (ORCPT ); Mon, 4 May 2009 13:22:03 -0400 Subject: Re: [BUG 13245] - possible circular locking dependency detected From: Johannes Berg To: LKML Cc: Miles Lane , rusty , "Rafael J. Wysocki" , Peter Zijlstra , bugme-daemon@bugzilla.kernel.org Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-i2yMeZo47TKzl4dTopUt" Date: Mon, 04 May 2009 19:21:24 +0200 Message-Id: <1241457684.8683.77.camel@johannes.local> Mime-Version: 1.0 X-Mailer: Evolution 2.26.1.1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-i2yMeZo47TKzl4dTopUt Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi all, This is effectively a copy of http://bugzilla.kernel.org/show_bug.cgi?id=3D13245 but with a few more people. I hope this works and bugme picks it up. The report is below. The problem here is effectively that something runs off the events workqueue, and ends up, with some indirection that I think doesn't matter much, using cpu_add_remove_lock, either while being on the events workqueue or in this case depending on a lock that was used from the events workqueue. It seems that the problem is in disable_nonboot_cpus, which via the notifier indirection calls cleanup_workqueue_thread() which flushes the workqueue so needs the fake "events" lock. On the other hand, via some indirection, things that run off the events end up using (depending on something that uses) cpu_add_remove_lock. Thoughts? [ 1231.558889] [ INFO: possible circular locking dependency detected ] [ 1231.558895] 2.6.30-rc4-git1 #5 [ 1231.558899] ------------------------------------------------------- [ 1231.558903] pm-suspend/4741 is trying to acquire lock: [ 1231.558908] (events){+.+.+.}, at: [] cleanup_workqueu= e_thread+0x1e/0xf8 [ 1231.558927]=20 [ 1231.558929] but task is already holding lock: [ 1231.558932] (cpu_add_remove_lock){+.+.+.}, at: [] dis= able_nonboot_cpus+0x33/0x11d [ 1231.558947]=20 [ 1231.558948] which lock already depends on the new lock. [ 1231.558951]=20 [ 1231.558954]=20 [ 1231.558955] the existing dependency chain (in reverse order) is: [ 1231.558959]=20 [ 1231.558961] -> #5 (cpu_add_remove_lock){+.+.+.}: [ 1231.558969] [] __lock_acquire+0xa88/0xc22 [ 1231.558980] [] lock_acquire+0xf6/0x11a [ 1231.558988] [] __mutex_lock_common+0x5a/0x429 [ 1231.558999] [] mutex_lock_nested+0x30/0x35 [ 1231.559008] [] cpu_maps_update_begin+0x12/0x14 [ 1231.559017] [] __create_workqueue_key+0x161/0x2= 26 [ 1231.559026] [] stop_machine_create+0x3a/0x9b [ 1231.559035] [] stop_machine+0x19/0x4b [ 1231.559044] [] 0xffffffffa0115238 [ 1231.559061] [] do_one_initcall+0x65/0x153 [ 1231.559072] [] sys_init_module+0xab/0x1d3 [ 1231.559080] [] system_call_fastpath+0x16/0x1b [ 1231.559089] [] 0xffffffffffffffff [ 1231.559097]=20 [ 1231.559099] -> #4 (setup_lock){+.+.+.}: [ 1231.559105] [] __lock_acquire+0xa88/0xc22 [ 1231.559114] [] lock_acquire+0xf6/0x11a [ 1231.559122] [] __mutex_lock_common+0x5a/0x429 [ 1231.559132] [] mutex_lock_nested+0x30/0x35 [ 1231.559141] [] stop_machine_create+0x12/0x9b [ 1231.559150] [] disable_nonboot_cpus+0x16/0x11d [ 1231.559159] [] suspend_devices_and_enter+0xd6/0= x1b1 [ 1231.559168] [] enter_state+0x163/0x1c9 [ 1231.559176] [] state_store+0xb7/0xd8 [ 1231.559184] [] kobj_attr_store+0x17/0x19 [ 1231.559194] [] sysfs_write_file+0xe4/0x119 [ 1231.559203] [] vfs_write+0xab/0x105 [ 1231.559212] [] sys_write+0x47/0x70 [ 1231.559219] [] system_call_fastpath+0x16/0x1b [ 1231.559228] [] 0xffffffffffffffff [ 1231.559236]=20 [ 1231.559238] -> #3 (dpm_list_mtx){+.+.+.}: [ 1231.559245] [] __lock_acquire+0xa88/0xc22 [ 1231.559253] [] lock_acquire+0xf6/0x11a [ 1231.559262] [] __mutex_lock_common+0x5a/0x429 [ 1231.559272] [] mutex_lock_nested+0x30/0x35 [ 1231.559281] [] device_pm_add+0x1e/0xb5 [ 1231.559292] [] device_add+0x37c/0x557 [ 1231.559301] [] wiphy_register+0x148/0x1ec [cfg8= 0211] [ 1231.559320] [] ieee80211_register_hw+0xe9/0x3aa= [mac80211] [ 1231.559348] [] iwl3945_pci_probe+0xb5c/0xcce [i= wl3945] [ 1231.559368] [] local_pci_probe+0x12/0x16 [ 1231.559378] [] pci_device_probe+0x5f/0x89 [ 1231.559388] [] driver_probe_device+0x9e/0x143 [ 1231.559398] [] __driver_attach+0x58/0x7b [ 1231.559406] [] bus_for_each_dev+0x54/0x8a [ 1231.559415] [] driver_attach+0x1c/0x1e [ 1231.559423] [] bus_add_driver+0xb2/0x1e6 [ 1231.559432] [] driver_register+0xb6/0x121 [ 1231.559439] [] __pci_register_driver+0x61/0xcf [ 1231.559448] [] 0xffffffffa018505c [ 1231.559456] [] do_one_initcall+0x65/0x153 [ 1231.559464] [] sys_init_module+0xab/0x1d3 [ 1231.559473] [] system_call_fastpath+0x16/0x1b [ 1231.559481] [] 0xffffffffffffffff [ 1231.559489]=20 [ 1231.559491] -> #2 (cfg80211_mutex){+.+.+.}: [ 1231.559498] [] __lock_acquire+0xa88/0xc22 [ 1231.559507] [] lock_acquire+0xf6/0x11a [ 1231.559515] [] __mutex_lock_common+0x5a/0x429 [ 1231.559524] [] mutex_lock_nested+0x30/0x35 [ 1231.559533] [] reg_todo+0x5d/0x4d8 [cfg80211] [ 1231.559553] [] worker_thread+0x24b/0x367 [ 1231.559561] [] kthread+0x56/0x83 [ 1231.559570] [] child_rip+0xa/0x20 [ 1231.559577] [] 0xffffffffffffffff [ 1231.559585]=20 [ 1231.559587] -> #1 (reg_work){+.+.+.}: [ 1231.559593] [] __lock_acquire+0xa88/0xc22 [ 1231.559603] [] lock_acquire+0xf6/0x11a [ 1231.559611] [] worker_thread+0x242/0x367 [ 1231.559619] [] kthread+0x56/0x83 [ 1231.559627] [] child_rip+0xa/0x20 [ 1231.559634] [] 0xffffffffffffffff [ 1231.559658]=20 [ 1231.559659] -> #0 (events){+.+.+.}: [ 1231.559666] [] __lock_acquire+0x97c/0xc22 [ 1231.559675] [] lock_acquire+0xf6/0x11a [ 1231.559683] [] cleanup_workqueue_thread+0x45/0x= f8 [ 1231.559691] [] workqueue_cpu_callback+0xc2/0x10= 5 [ 1231.559703] [] notifier_call_chain+0x5e/0x92 [ 1231.559712] [] raw_notifier_call_chain+0xf/0x11 [ 1231.559722] [] _cpu_down+0x278/0x295 [ 1231.559730] [] disable_nonboot_cpus+0x78/0x11d [ 1231.559740] [] suspend_devices_and_enter+0xd6/0= x1b1 [ 1231.559749] [] enter_state+0x163/0x1c9 [ 1231.559757] [] state_store+0xb7/0xd8 [ 1231.559765] [] kobj_attr_store+0x17/0x19 [ 1231.559773] [] sysfs_write_file+0xe4/0x119 [ 1231.559781] [] vfs_write+0xab/0x105 [ 1231.559788] [] sys_write+0x47/0x70 [ 1231.559796] [] system_call_fastpath+0x16/0x1b [ 1231.559804] [] 0xffffffffffffffff [ 1231.559813]=20 [ 1231.559814] other info that might help us debug this: [ 1231.559817]=20 [ 1231.559822] 4 locks held by pm-suspend/4741: [ 1231.559825] #0: (&buffer->mutex){+.+.+.}, at: [] sys= fs_write_file+0x38/0x119 [ 1231.559838] #1: (pm_mutex){+.+.+.}, at: [] enter_sta= te+0x1c0/0x1c9 [ 1231.559851] #2: (dpm_list_mtx){+.+.+.}, at: [] devic= e_pm_lock+0x12/0x14 [ 1231.559864] #3: (cpu_add_remove_lock){+.+.+.}, at: [= ] disable_nonboot_cpus+0x33/0x11d [ 1231.559878]=20 [ 1231.559880] stack backtrace: [ 1231.559886] Pid: 4741, comm: pm-suspend Not tainted 2.6.30-rc4-git1 #5 [ 1231.559892] Call Trace: [ 1231.559895] [] print_circular_bug_tail+0x71/0x7c [ 1231.559895] [] __lock_acquire+0x97c/0xc22 [ 1231.559895] [] lock_acquire+0xf6/0x11a [ 1231.559895] [] ? cleanup_workqueue_thread+0x1e/0xf8 [ 1231.559895] [] cleanup_workqueue_thread+0x45/0xf8 [ 1231.559895] [] ? cleanup_workqueue_thread+0x1e/0xf8 [ 1231.559895] [] workqueue_cpu_callback+0xc2/0x105 [ 1231.559895] [] notifier_call_chain+0x5e/0x92 [ 1231.559895] [] raw_notifier_call_chain+0xf/0x11 [ 1231.559895] [] _cpu_down+0x278/0x295 [ 1231.559895] [] disable_nonboot_cpus+0x78/0x11d [ 1231.559895] [] suspend_devices_and_enter+0xd6/0x1b1 [ 1231.559895] [] enter_state+0x163/0x1c9 [ 1231.559895] [] state_store+0xb7/0xd8 [ 1231.559895] [] kobj_attr_store+0x17/0x19 [ 1231.559895] [] sysfs_write_file+0xe4/0x119 [ 1231.559895] [] vfs_write+0xab/0x105 [ 1231.559895] [] sys_write+0x47/0x70 [ 1231.559895] [] system_call_fastpath+0x16/0x1b --=-i2yMeZo47TKzl4dTopUt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Comment: Johannes Berg (powerbook) iQIcBAABAgAGBQJJ/yQRAAoJEKVg1VMiehFYpWwP/RVfux8Za+5fkfTS8UOf5dQ3 igJt6CShWhL9ku9vPZ41Glvtvb+yNYJtm7g0JCWH4hKsApQ5Mfw+uFX8vZ3VvWfO dKpY4zwqYwkihuJoSXUSeT1YH8Fyvn9th6OP7DYyZ2WFfmecV6UQPhVroxc8AazK gVlJH9vRIlOR1Dl26lTlxxKcgda8Uw03V1Jq0LzlNBRfAUyanO9FOX9YP+XuImRb 6D8DxNjZmzQOJ4azH12Sy8GiiqP2bDMf6YPF34adUV1lfk42Rh02PcdkesZxNNPR U+JpY2U5fHYNPfkZFONSPRNIJ02nNXq5tB0YCMmBUSqhMmRd+sL4ZITZopySnXd6 Pc5CXPZ20KWgK+So8cQ/a6BkrMSbWsXx1oQi5/SjF8BJx3lrVcjoUQLf6dAN0n2E ggcmfm/qSy0nY6mCZBa/JQTmcS7B2/6j0l6QOEteAMdihFyrb71p+ID91QsqdKyh AsIkFBAzAqJjU/radSpBYyonvwyGJhA77S7y9DLgR0vjLGcODSydeEJfItd1ZcmU AWm8IlSoLXC3IyOsbAt3eoamKFWgs/6JPn0hBRudS4Xk/5Fk4GtCEutNPg1hY4zk hnl1XCw7AQALnrOjFE4TH7zrXDlm4ZbPyKjVFq9Q6xZxY7LP90BF7JwypE4BNEyP 4alnSOaGHHbj/2fmYzSU =lLjX -----END PGP SIGNATURE----- --=-i2yMeZo47TKzl4dTopUt--