From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CWXP265CU010.outbound.protection.outlook.com (mail-ukwestazon11022105.outbound.protection.outlook.com [52.101.101.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B2C99463 for ; Fri, 14 Aug 2026 13:57:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.101.105 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786715848; cv=fail; b=PzB/BCly903cTYypHzfWoB3t7Bm33S8B8a/iP7rWGunsHNdzELaBs8hMGNlPeRQpeHZZRIK9Wj6qr3dRbMS3PBWAPbsSkI9w3k90fsRLVSKGReph0Zv4yZu2X+PvrtFNbjZqw7IQ5vwY+0c17ZOHOui+JrWD8Xjb+HrRmhuuhEc= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786715848; c=relaxed/simple; bh=Yriqk+P9Yjp4hExYPUiYB1EQfCH2c7LvLyAf+dDnR6s=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=JyqCNPwNmiOcN62k/C4mH623hZHHLNhseU6lxhVuXcGQp7ca1v5d7DIEMnGNdEdwr3XfWEOVZLkL7e2YZAPSjpSbHmj/ZiMt/1vyLs7AFIUhzc090DRtkl+mj3T0c0fcqY+hb1JMEmLDdtNo6wTS3vzWfYO5uYUf+SEkaspnPoY= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com; spf=pass smtp.mailfrom=atomlin.com; arc=fail smtp.client-ip=52.101.101.105 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=atomlin.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MwkD07dDh5C9P6rlOf0msUfuH8SwNRLd+uNcwC5LjxQxX/Y7Kit3O59prNgMuXiNZwNyU3j7y9Fk30YeBqxf1AfU97MoAVNOF+s4odIq8k+WXZQoYgni4moSmubPSMUNBdivLEILp/oSDUfcfJk89OqBOxCk+cx7/Xqbg3UZLPjmffAomiAQJXtYNoBR49XZHyOP0KowcpLXeY59Ii4LkGZ0+kMGRTQUSYJZvMCbO0JuNQrtVEiijeJi2gauXMHzkiH/uqMwZKwEYeXghyQ+lyXCSOg8DS9r7i+cyF9FfW3zD4JJz1aBk8GTbRVKfeFZb/i4ZdFNvrBMpTBV003orQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:MIME-Version; bh=GyJ4rmwzAgGH43Wbw2Ge5HdxIDVkArRDmf8wfuZcVyQ=; b=G5r1HzzAckqRws3CyuY5lWrQkXidTFkZ7r5D2sQSPElGe5QFHwcegCoyqGILAUt5yLLuI8toz7v4BrfiUFGfW1ND0AReslSWVg9tu2COMpX6aJaLXHx9HpLXXSgEypA+/fVSNvMVVKZ0SO/vp/lKOU3nZCEYecD67iVzB/Ge9Vih9Bv55Js1SzhZpSEdOkPdqocIWlNJr3ZmfICd1KB/+waAa+a+ex/dZ5DD9YPIbNJxtKC6As06cx7wia3r/VuJqZnwJaqZ/yx7GAmMvEh71HKHsU8MdDbAhiVInRYTxZxqYOeNtWQYBFN/bVdfIKHqlApv1865omsdg0sAaKmSVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=atomlin.com; dmarc=pass action=none header.from=atomlin.com; dkim=pass header.d=atomlin.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=atomlin.com; Received: from CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:183::5) by LO8P123MB7759.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:3d5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.16; Fri, 14 Aug 2026 13:57:21 +0000 Received: from CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM ([fe80::cec4:77ab:262e:d230]) by CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM ([fe80::cec4:77ab:262e:d230%4]) with mapi id 15.21.0315.014; Fri, 14 Aug 2026 13:57:21 +0000 From: Aaron Tomlin To: akpm@linux-foundation.org, lance.yang@linux.dev, mhiramat@kernel.org, pmladek@suse.com Cc: linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, atomlin@atomlin.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: [PATCH v9 0/2] hung_task: Improve warning budget handling and task reporting Date: Fri, 14 Aug 2026 09:57:16 -0400 Message-ID: <20260814135718.494513-1-atomlin@atomlin.com> X-Mailer: git-send-email 2.55.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MN2PR22CA0016.namprd22.prod.outlook.com (2603:10b6:208:238::21) To CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:183::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CWLP123MB6607:EE_|LO8P123MB7759:EE_ X-MS-Office365-Filtering-Correlation-Id: 0f811526-b478-4362-6ee5-08defa0bf62b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|7416014|366016|1800799024|6133799003|18002099003|56012099006|3023799007|10067099003; X-Microsoft-Antispam-Message-Info: IsKcZCmDUWCjFHg6DP7kZgszI59IGz8kXedLE5KhKAjcVVy1j3qpKPLrdh6muBWdmnaiMMU69euXChB5KV0X7BdSV+ODqr07JoXHuWdP5Jxg0nLnwOoZ/4ocYzIWwnCGnUeGmOlai2auj9kPxFugjdJPEb2wlnysXgSN24blJml1DD2QDTsXbnneCwb7rbqDqCiEjMQovkp3dAsa29/FOPS1n3MCJHvedcY0nUqh0YlCnSx5k6s6g1ODdeOGSw+5s4Me+etU5DRWtHfSQbJiyMYK/3YH+U7Tuw3LZ+wb8YV5UjFag6YvJjunWRNtGTqeum9WlTYuFxuMxkJE3GLZrBACFh0/F6dPFS/N2D4Wq1/HXrTu6hlk0PD9s5m10zzzkRtb8xfIDKu3J2rmdBzU0xba7K11MJEDK6rBIJihkbfvBdd6XxEPpab0w5Cl1Sm/gNejPTGGDxPtOePU1AwloixqyiCiJ+XM2hmonTMBfw4AngUIOLmhOqoju1rD7HOpkfu29uU+NG/Esrv4GmtUjsRCL8BhoSFxLdnq+tibXuWIZbPlgpGWpOnP0i0cpQE/JTSPPGTJq7i21orDMvAjulCuHcOW1lnGaLPv3BKAP+bfvFvfvrh4lV4egJfvJ/jlIstlevBPJx2oGklBzDku7mMpsE8rs30VAMXZDOCixOY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(7416014)(366016)(1800799024)(6133799003)(18002099003)(56012099006)(3023799007)(10067099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1rJvUZzZsfBXm3oUZwdhcdZ7CxSBCmgxE7Y4xdRT4FMqLzX9yfQJPJ85dZfh?= =?us-ascii?Q?/6hhb/eg+/zZFvzNDBMVwNyDGyih6/7KcIP2HbVj67Nk44jsykEDVy70iT0I?= =?us-ascii?Q?Jyl0vD52cSicgLo5Ww9sgRu1XqpHG1k1YpluqH4u4OyMtq1ALFO945x3r5bK?= =?us-ascii?Q?SCunB5afHm3dohjyYDTeJ1dCFHgsqoGZLURib4qJXJwmkM6igHhaHOcjHZaJ?= =?us-ascii?Q?2STHW37B9nMriD5CAxz97KCzvJmwd/605OATLCLc9lvdZKxKY5xy8wGq/OOC?= =?us-ascii?Q?5vALNhdnpM0a+27E8f3SfMPspR4aXTiSfAZmqpALf6AyKnGpGuKw501wQAF0?= =?us-ascii?Q?0eRwFlOj0h24uStoeI6nd1zjM595Ezy8bPUxi8iuKdorD9Hb67w/G/xbAjAz?= =?us-ascii?Q?6LBQ48JJkobcPrFzrFxoRwihNxWcmou2cfSGgAiFR4yK9+HQmSA5mjLnZRSZ?= =?us-ascii?Q?MsGPADljjefoy8yFSE26CXUAIUA81JlDX07yvdfXtafFqUffJCXAOWSAJGke?= =?us-ascii?Q?KZkR9QgnM8pmfKE4QZQs/Du3WzFoWiJysGAWzpZtBhtURE83s9Ar0LQhSEo1?= =?us-ascii?Q?w9qfjxw880kv1scqwwIrw8DJQgIxXZKS02P6FaUpmPeI0UAbF/T1Dq271IlC?= =?us-ascii?Q?aQWFoF5rDYxW8xernboHJyv6HOHzWMWpaRf5t1dEFV08FolzDipjkOf392p8?= =?us-ascii?Q?gzc1WOtg4vB73EbsfX3z0JsVQdpHfZiMAgl1kBlF9xVRsnr1JX4ekhetJUhY?= =?us-ascii?Q?Uy5nOaPOTrwbTzRHiWCKOUxkwxR/YXo5Nih4ZyRRwb5tJqDUysBEDzRtx16W?= =?us-ascii?Q?TwYEVfG2JmGPYSgcL82E87X/D39GbwQ8FVaSmeiBZP/r8onhNYJMNqywZ40y?= =?us-ascii?Q?Bjd5QA8rMsFsyZQ+cW20CzhBXjj9wiH0d6fH0ous25KBZcwn7Pjj6gV7RKTY?= =?us-ascii?Q?GasWg4Rto384VCdvAXjlaWRB/vL+9jzXFSFfT+qK9vxRXqprXRwhyMmpH0JI?= =?us-ascii?Q?9jNIgTlg+Pi7HOjld9qoYJiEKJrSY3sFFXS5Mnz2JUR/qJP0MwiwnDJQkm3F?= =?us-ascii?Q?Yi9PL+BhK18XtS9PLB+G5+xa4RF7gd6YTV7bzNaxmLm+3fKPxjneLCtGTnRk?= =?us-ascii?Q?9lLyuRnlpzW3YPY+aCKcg6icnoYOy1bq5IwXABdKNwTsW/9hgykhicn4ugt3?= =?us-ascii?Q?29ze70XlaNdHknAnGg9MdDGSVvVkzpX1IVFJkWRnJqBav6hZylrVUqa9cJZW?= =?us-ascii?Q?PsQaM9sPMuDO7t3/iKQtTtuGlZZtBWXI0Ux6WOfBL6taSTL9B3A5YgfMEqIk?= =?us-ascii?Q?CebxwIVhXu5jswKuQeQxmat3nguBlc1nE/3nIddIXhUcadYHHnz5+ZmB+Q0C?= =?us-ascii?Q?ELeDf4Werjl7LwBn9/qqAkyMiafFYOdq816NPHWP05mAzx7dpa/Cq5Wf4z5E?= =?us-ascii?Q?7iOKQ/YNn11rkG/J+xJ0SffH/KymBBMmYzp+JujOQa9ROUw/6cMff36H7DXt?= =?us-ascii?Q?+6aiusaf/jL2/HU+HITtUXRSw6UDc0qq/ImyJUMkWbb5tgevoH6AeJJrwbSK?= =?us-ascii?Q?RaMqR01rJVWXN0eqRY8d9ayQRUwn4eziLqCXSno3mRGlPacV7EUhiUp1jMWf?= =?us-ascii?Q?8NyFbAb9o86BVGNDAFmcJXGUGnsGD9fsvzLxRIDZfbyxVUUo8iOYI19IUvcV?= =?us-ascii?Q?zAHrn52Ny6gOI0MtGQ9wOUzWtvKH7VtfSVlIjG/agE1zpVUPX8imCny/F6Fv?= =?us-ascii?Q?8LV00Mkckg=3D=3D?= X-OriginatorOrg: atomlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0f811526-b478-4362-6ee5-08defa0bf62b X-MS-Exchange-CrossTenant-AuthSource: CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2026 13:57:21.7063 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e6a32402-7d7b-4830-9a2b-76945bbbcb57 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ei5jXbIqtoP3fayi+i2tLPJ3HkEaRmj45qZxTxZzi6bncxcmEeXFfLMZPPvTXfp0lZc01QFQgacjYavEqFv5Wg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO8P123MB7759 The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D) state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent log spam during system spikes, sysctl_hung_task_warnings enforces a budget on the number of logged warnings. However, the current implementation has two major limitations: 1. Permanent exhaustion of warning budget sysctl_hung_task_warnings is decremented directly when printing warnings. Once this budget hits zero, no further warnings are reported until an administrator manually updates the sysctl value or reboots the system. Consequently, a single temporary hang episode permanently blinds the kernel watchdog to any subsequent hung tasks after system recovery. 2. Total log suppression when budget is exhausted Once the warning budget reaches zero, hung_task_info() completely suppresses all output, including the basic single-line alert. While suppressing verbose stack dumps and lock debugging is desirable to prevent dmesg flooding, hiding basic task alerts leaves administrators entirely unaware that tasks are hanging. This patch series resolves both limitations by decoupling the configured warning limit from the active runtime budget, automatically resetting the budget upon system recovery or sysctl updates, and emitting a single aggregate summary line when hung tasks are detected under an exhausted warning budget. Patch 1 separates the configured sysctl hung_task_warnings from the runtime budget. Introduces an atomic flag so khungtaskd locklessly resets the runtime budget when a scan finds zero hung tasks or when userspace writes to the sysctl. Patch 2 prevents dmesg flooding during system-wide hangs by keeping per-task stack dumps budgeted, but provides ongoing visibility by logging a single aggregate summary line at the end of each scan iteration when the warning budget is exhausted. Changes since v8: - Resolved a concurrent sysctl write race condition on hung_task_warnings_printed by making khungtaskd the sole owner of runtime budget updates. Introduced an atomic flag set via sysctl write and picked up locklessly by khungtaskd at the start of a scan iteration (Lance Yang) - Restored per-task header logging back inside the warning budget check in hung_task_info() to prevent dmesg ring buffer saturation and console lock contention during mass hung task events (Lance Yang) - Added a single aggregate scan summary line printed at the end of each watchdog scan when hung tasks are detected and the warning budget is exhausted (Lance Yang) - Linked to v8: https://lore.kernel.org/lkml/20260804202050.262427-1-atomlin@atomlin.com/ Changes since v7: - Consolidated the commit message of each patch (Lance Yang) - Linked to v7: https://lore.kernel.org/lkml/20260804155406.254810-1-atomlin@atomlin.com/ Changes since v6: - Restructured the series in a new direction. Introduced an internal counter (hung_task_warnings_printed) decoupled from sysctl_hung_task_warnings. The warning budget automatically resets to the configured limit once a check interval completes with zero hung tasks detected, or when modified via sysctl (Petr Mladek) - Ensured the single-line blocked hung task message is always printed even after the warning budget reaches zero. Now budget enforcement is restricted solely to suppressing verbose diagnostics (Petr Mladek) - Refined the description of sysctl hung_task_warnings (Lance Yang) - Removed field hung_task_reported from struct task_struct - Removed the CONFIG_DETECT_HUNG_TASK_BLOCKER integration and hung_task_blockers[] used to track memory addresses of blocker locks across check intervals - Removed the skip_show_task logic and dmesg log suppression messages - Removed tracking variables (warnings_decremented and hung_task_has_active) and the dmesg recovery notice printed when clearing the blocker array - Linked to v6: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@atomlin.com/ Changes since v5: - Skipped hung_task_info() and sys_info() for tasks already reported in previous rounds to avoid log spam - Linked to v5: https://lore.kernel.org/lkml/20260712202100.123934-1-atomlin@atomlin.com/ Changes since v4: - Replaced the stack-local hashmap implementation with a persistent array (Petr Mladek) - Persistently track blocker addresses across scan intervals. Suppress warning reports and keep the sysctl_hung_task_warnings budget intact if the blocker is already tracked in the array (Petr Mladek) - Reset the warnings budget and clear the blocker array when the hang resolves (Petr Mladek) - Output a recovery message to the kernel ring buffer upon hang resolution - Linked to v4: https://lore.kernel.org/lkml/20260627205733.90983-1-atomlin@atomlin.com/ Changes since v3: - Deduct from the global budget if printing a full stack trace - Pivoted from heuristic Wait Channel hashing to deterministic blocker address hashing via CONFIG_DETECT_HUNG_TASK_BLOCKER - Replaced the hung_task_reported bit-field with a standalone u8 byte. Move hung_task_reported into an existing structural alignment hole within task_struct following blocked_lock, resulting in zero overall memory footprint increase and optimal cacheline grouping - Linked to v3: https://lore.kernel.org/lkml/20260621213756.43225-1-atomlin@atomlin.com/ Changes since v2: - Replaced the per-round cache flush with a task_struct bit-field for persistent cross-scan tracking, mitigating delayed budget exhaustion - Abandoned exact-stack hashing in favour of Wait Channel hashing - Transitioned from jhash() to hash_long() to optimise single-pointer hashing, and relocated the hash map to the local stack - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/ Changes since v1: - Preserve "INFO:" headers for all hung tasks; suppress only the stack dumps for duplicates (Masami Hiramatsu) - Print a clear notification when a trace is explicitly suppressed - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors - Optimise overhead by unwinding the stack only if a warning is actually going to be printed - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/ Aaron Tomlin (2): hung_task: Reset warning budget when problem gets resolved hung_task: Log summary line when warning budget is exhausted Documentation/admin-guide/sysctl/kernel.rst | 5 ++- kernel/hung_task.c | 46 +++++++++++++++++---- 2 files changed, 40 insertions(+), 11 deletions(-) -- 2.55.0