From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012017.outbound.protection.outlook.com [52.101.43.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1271F282F39 for ; Tue, 21 Apr 2026 03:14:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.43.17 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776741250; cv=fail; b=ucHEkn5ohU8pdMM+X6cF7JVFyVT24xbQtjY+23BbJloJrj57q9TElVl++v5jpIcYDH8CsbcmKxscBo/MhVavfsuSaIxvxKKC02amAZ5+NLUxyICSrH2n4S79+8x6rGgz0grQBvRpneP6BAaPr0kXfe+gYf4obBS/5WCa2bEDJx0= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776741250; c=relaxed/simple; bh=dy5JnwT13Jg4FIFyUedGsCUn3ChxCRTUy3cNFNIFX7s=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=lWNXruz1/GvOFs2icE6zD+KSQtxrn/4UlQy7NChR9e7W9pcxGejp1kxYi1Tl2zN+9aZQtC/DFNr66eiO2m0xTBgi7zc3NWd9MWlmYwC2KAsxk59g86DXesO1KtqgSG9kp0d6YgHCQ72a5O9ggAstFt8IWWPyRpiypGlpnG6wiTg= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=kBXkCGaG; arc=fail smtp.client-ip=52.101.43.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="kBXkCGaG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CdI7y6icXIXzNX/j+Q9I8ACPdiqr8AF/RCNI5T5/WnS8oSDBi4hXHycCiBmtRSfAmPak3cQA4IaHdV5uJl1QO4SrxMyikvMbgeMGYGrIcaKAV5GvhLi1A8Ckup04spSXL316K95Kw3XHBFkVBOn0TOzjcMtlRfDzm74NljCfH0f9pz7+Caz/JC2qA34I4C3UaNcxFBdRnlVmDkLUcqWGPWkAlenwgXevbR1Dnos69lV+OeOMZVaobRGFPrOo/wb0wMgFlKpj4r0E86a0cOHl/logUg4gIWgdWjSSJ8wkKAFIPqQQrufioRhuOmx66VoKkkOYkOrSJkherp9y0IiYHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c/LgDC1QUunwNIeWkGUB0Yi2Kh40ggoRDYQt8PFjbUQ=; b=qkHdAmtFTQmTMdBOIxpzP8OLIPozf7tDpS5X4N4CFQTzg/hRcvLXmgPKDHeQFAGsXVTmtwL55U/EGGFIY6iLZUnsR0RDlcSRdrL7SFnuZaQlohomEc8vnHcvqeYM032Ar0q18dcbJIgnLY/cMmD6Kev8oyj8eogdaKV0tRVHEE6L+WNURW6OL47tbRgNYAotWOckq4ZOP0KOLZ3Df+xT9PXM6i/xpFfqqCaur67o0ez8i91YaO92vSt58fk3mRGNjiSfZUNlnGEnuD/1MPPvuEQEPYs2fkRMK+s8JiQ4vuo2UNpfE8mT4emxTfIK8AcYrmGlshYv7m5j898HZStB2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c/LgDC1QUunwNIeWkGUB0Yi2Kh40ggoRDYQt8PFjbUQ=; b=kBXkCGaGasRj3C3iIVLP/tfZMfBObxMFhdQGBBBW1ac3uyvC/v3L++bYHD4OPTAPHEG4irRwQ23DlwURcPbW3FyrBBK8Eq2AuLqkjrQnX3qwtHEHTWKC+1+K0M5YjDKHNk7iTslJwrSDbEDFxBTQIMK2A7NvjYN4e6l7UajK2g8= Received: from SJ0PR05CA0123.namprd05.prod.outlook.com (2603:10b6:a03:33d::8) by CY8PR12MB7220.namprd12.prod.outlook.com (2603:10b6:930:58::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.16; Tue, 21 Apr 2026 03:14:00 +0000 Received: from MWH0EPF000C6192.namprd02.prod.outlook.com (2603:10b6:a03:33d:cafe::7e) by SJ0PR05CA0123.outlook.office365.com (2603:10b6:a03:33d::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9791.48 via Frontend Transport; Tue, 21 Apr 2026 03:14:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MWH0EPF000C6192.mail.protection.outlook.com (10.167.249.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9791.48 via Frontend Transport; Tue, 21 Apr 2026 03:14:00 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Mon, 20 Apr 2026 22:14:00 -0500 Received: from satlexmb08.amd.com (10.181.42.217) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 20 Apr 2026 22:13:59 -0500 Received: from [10.136.34.119] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Mon, 20 Apr 2026 22:13:58 -0500 Message-ID: <1df0e505-add4-4b7d-8e09-bc52402da617@amd.com> Date: Tue, 21 Apr 2026 08:43:57 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] sched/proxy_exec: Limit find_proxy_task() chain depth to prevent CPU hang To: John Stultz , CC: Peter Zijlstra , Ingo Molnar , , zhidao su References: <20260414053625.3582936-1-soolaugust@gmail.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Received-SPF: None (SATLEXMB04.amd.com: kprateek.nayak@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000C6192:EE_|CY8PR12MB7220:EE_ X-MS-Office365-Filtering-Correlation-Id: a8f45a81-4b53-4633-cf33-08de9f5408c2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|82310400026|1800799024|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: P4ym5c+vHskNzQUOp7WX9+Hzhwdm+O2e6czqkU94tppHdb1e4AAQc+qGfbXmVp6/SOlFmj/PERVfA2tVeEajLM5mK3O0A3udEBEEcqZeBy+3V0ZQTx+ddTICEoY4RkMYxPoKz0a6QRvMZBaURxmBve4PE5CfM0d6nL51BJND5uOZKeAa6mKmlJaY2ppTpjMT1QwDLhTSya4VDyPomyty8f3Nnj8ODos4jW4BMJrgxToQEK1RV9wbAI9WG3q/Xo7Tpr5HKfy+W/k+LNTABv0y+XiMJ4dS/So04kfK7z0HXI6Xq4B3M3avZosJGMCK/MrUw43MR2devGmzGKttSUfzt3a4PJU0+3xFvCjlfBCYZlqqAAKdmaRCw0n94sGZGa+HgrA6gK2Af//XLwlPs6+IpLgA3BVx8spsYbkbJhic41oFkx7ZEe088oMDUrZLcbn43tkEGn9SaGVhUvTbuUOnaImZ6SGKQFClsSlRt5usecGCAgscuJpTtIDkYcRpM3iBUCwe3Ua4GRrBR2lSUV2Gt5nKvCe+LruU/pZqWZ2yx8gTAo1pTqoITrInNyYUwP8TLFieJMPRV6oKEQ2ERlLzOb+I6AtuA+TUSa1jktHokjRwB1h0uu5sUKY4YnSA6xueEbtUGSd7L2yhfGRtv6Z7ggim7T7S57+F9s5JKg5vWoAPGI+RgTtXLqqQVCyq/FRze7w5IZ2opzWihXur7KJ8ptbZAcFUaS8T9trXxfrjCJG3FO14ApBbnr8+3OnNqhc0AjvdUf215pmVXhR2Gll7MQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(376014)(82310400026)(1800799024)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RDdzzOJ+a4QFzord+0ip109Jn+bvXD2rccs/VCOmgoQwZENKunBWMCq4U+D2HLifkw2ZWkiFqoACunpvWfVWltMz//1nbYFjLLc4mk/TKma5CTKSe//H+56T2//IuX+lnpnh6M3m7dJVkEim4+2dhz8f0rkDDDwX9GWsYrDywQ5aR49P7Lxhy2a5bvMAriOP3aEcOKtpaW17zpbcDjY+OFCdsV0vFR+vb8YsUdtcVWyRsG/f1R3hET39tGb+FvjIKgIdpsH5p5f5EzHcPKvQhQBC3o0YzpQBroxkggrYYqRJgXF9kcdhjWbHsiHT6sfC0FRvf/8FGiieiCLzvvJF8W1hcDqnQ7OIhskQbXRmssDj6DkcuyuymIHhXa2ltzTe9sdFDO0zaLti/XHukUYePOzRH2ft7yXeu3pQVYO5uXDA0R8xaRryyiprzZ0lnICM X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Apr 2026 03:14:00.5496 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a8f45a81-4b53-4633-cf33-08de9f5408c2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000C6192.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7220 Hello John, Zhidao, On 4/21/2026 7:57 AM, John Stultz wrote: >> With this fix: >> >> [ 111.758150] sched/pe: proxy chain depth exceeded 64, possible deadlock cycle involving pid 120 >> [ 111.758150] WARNING: CPU: 0 PID: 119 at kernel/sched/core.c:7339 __schedule+0x1e6e/0x1e80 >> ... >> [ 112.694277] pe_cycle_test: still alive after 1s (CPU not hung) >> >> Without this fix, an NMI watchdog (nmi_watchdog=1, watchdog_thresh=15) >> fires a hard LOCKUP on CPU 0 with RIP in do_raw_spin_lock, called from >> __schedule, confirming the CPU spins inside find_proxy_task() holding >> rq->lock with no forward progress: >> >> [ 109.951781] watchdog: CPU0: Watchdog detected hard LOCKUP on cpu 0 >> [ 109.951781] RIP: 0010:do_raw_spin_lock+0x3e/0xb0 >> [ 109.951781] Call Trace: >> [ 109.951781] __schedule+0x11e7/0x1e10 >> [ 109.951781] schedule_preempt_disabled+0x18/0x30 >> [ 109.951781] __mutex_lock+0x6f0/0xac0 >> [ 109.951781] pe_test_thread_a+0x9c/0xe0 > > > So, I guess I'd be curious what happens without proxy-exec. I think you hit the hung task detector in that case since the interruptible sleep has lingered for too long. > > My sense if if you have a mutex lock cycle today without proxy > execution you'll just deadlock and get a similar hard LOCKUP warning. > I assume you'd get a LOCKDEP splat as well if that was enabled in > either case, no? > > So I'm not sure if I see a whole lot of benefit to rescheduling idle > over and over to keep the system sort of alive when that cpu is not > going to make any progress. > > A few more thoughts below... > >> Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") >> Signed-off-by: zhidao su >> --- >> kernel/sched/core.c | 17 +++++++++++++++++ >> 1 file changed, 17 insertions(+) >> >> diff --git a/kernel/sched/core.c b/kernel/sched/core.c >> index 3f3425c6b2f2..bafb59432f7f 100644 >> --- a/kernel/sched/core.c >> +++ b/kernel/sched/core.c >> @@ -7310,6 +7310,17 @@ DEFINE_LOCK_GUARD_1(blocked_on_lock, struct blocked_on_lock, >> * Returns the task that is going to be used as execution context (the one >> * that is actually going to be run on cpu_of(rq)). >> */ >> +/* >> + * Limit proxy chain traversal depth to avoid infinite loops in pathological >> + * cases (e.g., A waits for B's mutex while B waits for A's mutex). The >> + * existing WARN_ON(owner == p) only catches immediate self-loops; multi-task >> + * cycles like A->B->A are not detected without a depth counter. >> + * >> + * rt-mutex uses a similar guard (max_lock_depth = 1024). We use a smaller >> + * limit since proxy chains are expected to be short in practice. >> + */ >> +#define MAX_PROXY_CHAIN_DEPTH 64 > > So while we'd hope proxy chains are short in most cases, there's no > guarantee they would be different from rt-mutexes. > In fact, with rwsem support, the chains could interleave across lock > types, so I'd probably at least match the rt-mutex value. > >> static struct task_struct * >> find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) >> __must_hold(__rq_lockp(rq)) >> @@ -7318,11 +7329,17 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) >> struct task_struct *owner = NULL; >> bool curr_in_chain = false; >> int this_cpu = cpu_of(rq); >> + int chain_depth = 0; >> struct task_struct *p; >> int owner_cpu; >> >> /* Follow blocked_on chain. */ >> for (p = donor; task_is_blocked(p); p = owner) { >> + if (++chain_depth > MAX_PROXY_CHAIN_DEPTH) { >> + WARN_ONCE(1, "sched/pe: proxy chain depth exceeded %d, possible deadlock cycle involving pid %d\n", >> + MAX_PROXY_CHAIN_DEPTH, p->pid); >> + return proxy_resched_idle(rq); > > So at this point the cpu is going to be stuck, as as soon as it > switches to idle, it will call back into __schedule(), select the same > donor task and and traverse the same chain, and then reschedule idle > and start again. > > So it seems to me like BUG() would be more appropriate here as the cpu > is effectively deadlocked. > > I guess one could deactivate the selected blocked donor task, which > would let the cpu continue to run other tasks, but the entire lock > chain would eventually get deactivated and would never be woken up, so > it would likely trip hung task warnings. So I of would lean towards > BUG() since lock cycles are a big no no (for non-ww_mutexes) and I'd > fret if you don't stop the system folks will just ignore warnings and > not really understand why things aren't working properly. > > But that's just my instinct. I would second that but I can see someone having a "creative" mutex_lock_interruptible() pattern that relies on the hung task splat to then trigger something from userspace to selectively kill tasks. (Insane? Yes! Possible? Also yes!) As an alternate approach, when traversing blocked_on links, can we start deactivating the chain if we encounter rq->donor again as owner in find_proxy_task() loop? That way we go back to triggering the hung task detector and if someone has a stack that depends on it, it'll continue to work fine while also avoiding this lockup. Thoughts? > > Anyway, thanks for the submission here! I'm excited to see more folks > working and testing with proxy-exec! +1. With the next batch of changes, when we hopefully drop the EXPERT dependency, we'll probably see even wider usage and development ;-) -- Thanks and Regards, Prateek