From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755193AbdIGMQc (ORCPT ); Thu, 7 Sep 2017 08:16:32 -0400 Received: from mail.free-electrons.com ([62.4.15.54]:34072 "EHLO mail.free-electrons.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754882AbdIGMQb (ORCPT ); Thu, 7 Sep 2017 08:16:31 -0400 Date: Thu, 7 Sep 2017 14:16:19 +0200 From: Maxime Ripard To: Peter Zijlstra , Ingo Molnar Cc: linux-kernel@vger.kernel.org, Thomas Petazzoni , Quentin Schulz , Mylene Josserand Subject: mutex_lock issues during poweroff Message-ID: <20170907121619.uj245cykaxmhsuve@flea.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="shvmx3xq3uazmus4" Content-Disposition: inline User-Agent: NeoMutt/20170714 (1.8.3) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --shvmx3xq3uazmus4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, We've been investigating a bug on our kernel for the last couple monthes. The scenario is this: we have an ARM board that embed an Allwinner A33 SoC. That board is using a PMIC connected to the SoC through a proprietary bus, whose driver is in drivers/bus/sunxi-rsb.c. The poweroff is implemented by sending a shutdown command to that PMIC. http://elixir.free-electrons.com/linux/v4.9.47/source/drivers/mfd/axp20x.c#= L743 That PMIC also serves other purposes, such as controlling the regulators, but we also use it to get the various power supplies state, and report them through our power supplies driver. http://elixir.free-electrons.com/linux/v4.9.47/source/drivers/power/supply/= axp20x_usb_power.c http://elixir.free-electrons.com/linux/v4.12.11/source/drivers/power/supply= /axp20x_ac_power.c http://elixir.free-electrons.com/linux/v4.12.11/source/drivers/power/supply= /axp20x_battery.c The bug arises when we have those drivers enabled on a kernel 4.9.47 (or any 4.9 kernel. 4.8 also happens to show this). In some cases (1 out of 200-300 poweroff), the board will not poweroff. After digging through this, it turns out that in such scenario, the mutex_lock we have in the bus driver never returns. Here: http://elixir.free-electrons.com/linux/v4.9.47/source/drivers/bus/sun= xi-rsb.c#L379 Which means that we will never actually send the command, which also explains why it powered on. This gets weirder, since if we dump the return code of mutex_is_locked right before a failing case, the mutex isn't already locked, so we should not block or sleep at all. If we disable the power supplies driver that poll the PMIC status on a regular basis, it works, however we've never actually seen a concurrent usage of that bus. In our practical cases, the mutex is always unlocked. If we remove the mutex_lock / _unlock entirely, we don't stall anymore either, which seems to confirm something weird going on here. One thing worth noting is that we couldn't reproduce the issue with a 4.13. We can't bisect really easily due to the amount of patches that we still have on 4.9 and have all been merged since, but it seems like the bug was fixed (either on purpose or as a side effect), and was never sent to stable. Looking at the history of kernel/locking/mutex.c during that window didn't really show anything obvious though. If you have any ideas or spot something very wrong, I'd be happy to hear about. Thanks! Maxime --=20 Maxime Ripard, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com --shvmx3xq3uazmus4 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJZsTiTAAoJEBx+YmzsjxAgHeYQAKLS3dhpxQ3/d7OZFW+YeS+s /tlzd8kcx8lF9MvgkyUt6KSbn6odYd6ZZz19nU1JgSXI2ZQZwWysMxYqMn1tHGg7 TOdlQPHd0KixGBOLX4+xuq/XXPsA3ARx9BXLsXP4wcpAqhidF/H/2zyrloA6yKBm WbkQ26sRRpJWZMhMyXKP2KhHkskkjrSrG6heQyGrFcJXWehHxg8UjKX43LuTQgzd oiz6ESR+53pHd17PvLvxn2sKDr8VbUJIaXBI+T3EKvh883n9M9IWuWcQeDDnJWps x0r+DCxApk5LT9MfQBs944DMfjyN5ksKsKpj2baipQzcupJxJ9kArdPb4wcVnQqG 2qHKtabuS9xO23QRO+3dwqNL2WIY/ZDWSmNNfp8JyLl4dbYJIX+5rw2w7kaictAI G+i/0XxLd3DsGk0F0DLJKIx84mqvJWoxDDo+aXFYd4Z2Ob2i5IMExMQvAJyCaK70 xcrIykA8F4HZ7ua3K5V5z0pQvbo8M6R255XCFf2nCrlsqofEoCHSFIMZhngtzq3c 5C+H955ezeLuN9+/dc34U5cEQJ+2EqitUExiVYSqzIoeKfHGhXHVJ6mZfR16Ifly 2n10eKZfXupFitLR1DeJnDrl7TK37rpNZ2GjssWG0xTAReZKpd0BhKsOh0fjyuCn rP8ZFC6j46JkHQD28Cwa =cU9i -----END PGP SIGNATURE----- --shvmx3xq3uazmus4--