[PATCH v10 0/2] hung_task: Improve warning budget handling and task reporting

[PATCH v10 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
log spam during system spikes, sysctl_hung_task_warnings enforces a budget
on the number of logged warnings.

However, the current implementation has two major limitations:

    1. Permanent exhaustion of warning budget

       sysctl_hung_task_warnings is decremented directly when printing
       warnings. Once this budget hits zero, no further warnings are
       reported until an administrator manually updates the sysctl value or
       reboots the system. Consequently, a single temporary hang episode
       permanently blinds the kernel watchdog to any subsequent hung tasks
       after system recovery.

    2. Total log suppression when budget is exhausted

       Once the warning budget reaches zero, hung_task_info() completely
       suppresses all output, including the basic single-line alert. While
       suppressing verbose stack dumps and lock debugging is desirable to
       prevent dmesg flooding, hiding basic task alerts leaves
       administrators entirely unaware that tasks are hanging.

This patch series resolves both limitations by decoupling the configured
warning limit from the active runtime budget, automatically resetting the
budget upon system recovery or sysctl updates, and emitting a single
aggregate summary line when hung tasks are detected under an exhausted
warning budget.

Patch 1 separates the configured sysctl hung_task_warnings from the runtime
budget, making khungtaskd the sole owner of runtime budget updates. The budget
is reloaded directly when a scan finds zero hung tasks, or via an atomic reset
request published on sysctl write.

Patch 2 prevents dmesg flooding during system-wide hangs by keeping non-panic
per-task stack dumps budgeted, while providing ongoing visibility by logging a
single aggregate summary line at the end of each scan iteration when the
warning budget is exhausted.

Changes since v9:

 - Renamed hung_task_warnings_printed to hung_task_warnings_budget and
   added top-level comments distinguishing between the configured sysctl
   limit and the runtime remaining budget (Petr Mladek)

 - Used atomic_xchg_acquire() when checking reset_hung_task_warnings in
   check_hung_uninterruptible_tasks() for explicit acquire/release barrier
   symmetry with the sysctl handler (Petr Mladek, Lance Yang)

 - Refined the suppression message in hung_task_info() to explicitly
   reflect that details remain suppressed until the budget is reset or a
   panic is triggered (Lance Yang)

 - Clarified commit descriptions regarding khungtaskd single-writer budget
   ownership and panic diagnostic retention (Lance Yang)

 - Added Reviewed-by: tag from Petr Mladek

 - Linked to v9: https://lore.kernel.org/lkml/20260814135718.494513-1-atomlin@atomlin.com/

Changes since v8:

 - Resolved a concurrent sysctl write race condition on
   hung_task_warnings_printed by making khungtaskd the sole owner of
   runtime budget updates. Introduced an atomic flag set via sysctl write
   and picked up locklessly by khungtaskd at the start of a scan iteration
   (Lance Yang)

 - Restored per-task header logging back inside the warning budget check in
   hung_task_info() to prevent dmesg ring buffer saturation and console
   lock contention during mass hung task events (Lance Yang)

 - Added a single aggregate scan summary line printed at the end of each
   watchdog scan when hung tasks are detected and the warning budget is
   exhausted (Lance Yang)

 - Linked to v8: https://lore.kernel.org/lkml/20260804202050.262427-1-atomlin@atomlin.com/

Changes since v7:

 - Consolidated the commit message of each patch (Lance Yang)

 - Linked to v7: https://lore.kernel.org/lkml/20260804155406.254810-1-atomlin@atomlin.com/

Changes since v6:

 - Restructured the series in a new direction. Introduced an internal
   counter (hung_task_warnings_printed) decoupled from
   sysctl_hung_task_warnings. The warning budget automatically resets to
   the configured limit once a check interval completes with zero hung
   tasks detected, or when modified via sysctl (Petr Mladek)

 - Ensured the single-line blocked hung task message is always printed even
   after the warning budget reaches zero. Now budget enforcement is
   restricted solely to suppressing verbose diagnostics (Petr Mladek)

 - Refined the description of sysctl hung_task_warnings (Lance Yang)

 - Removed field hung_task_reported from struct task_struct

 - Removed the CONFIG_DETECT_HUNG_TASK_BLOCKER integration and
   hung_task_blockers[] used to track memory addresses of blocker locks
   across check intervals

 - Removed the skip_show_task logic and dmesg log suppression messages

 - Removed tracking variables (warnings_decremented and
   hung_task_has_active) and the dmesg recovery notice printed when
   clearing the blocker array

 - Linked to v6: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@atomlin.com/

Changes since v5:

 - Skipped hung_task_info() and sys_info() for tasks already reported in
   previous rounds to avoid log spam

 - Linked to v5: https://lore.kernel.org/lkml/20260712202100.123934-1-atomlin@atomlin.com/

Changes since v4:

 - Replaced the stack-local hashmap implementation with a persistent
   array (Petr Mladek)

 - Persistently track blocker addresses across scan intervals. Suppress
   warning reports and keep the sysctl_hung_task_warnings budget intact
   if the blocker is already tracked in the array (Petr Mladek)

 - Reset the warnings budget and clear the blocker array when the hang
   resolves (Petr Mladek)

 - Output a recovery message to the kernel ring buffer upon hang
   resolution

 - Linked to v4: https://lore.kernel.org/lkml/20260627205733.90983-1-atomlin@atomlin.com/

Changes since v3:

 - Deduct from the global budget if printing a full stack trace

 - Pivoted from heuristic Wait Channel hashing to deterministic
   blocker address hashing via CONFIG_DETECT_HUNG_TASK_BLOCKER

 - Replaced the hung_task_reported bit-field with a standalone u8 byte.
   Move hung_task_reported into an existing structural alignment hole
   within task_struct following blocked_lock, resulting in zero overall
   memory footprint increase and optimal cacheline grouping

 - Linked to v3: https://lore.kernel.org/lkml/20260621213756.43225-1-atomlin@atomlin.com/

Changes since v2:

 - Replaced the per-round cache flush with a task_struct bit-field for
   persistent cross-scan tracking, mitigating delayed budget exhaustion

 - Abandoned exact-stack hashing in favour of Wait Channel hashing

 - Transitioned from jhash() to hash_long() to optimise single-pointer
   hashing, and relocated the hash map to the local stack

 - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/

Changes since v1:

 - Preserve "INFO:" headers for all hung tasks; suppress only the stack
   dumps for duplicates (Masami Hiramatsu)

 - Print a clear notification when a trace is explicitly suppressed

 - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors

 - Optimise overhead by unwinding the stack only if a warning is
   actually going to be printed

 - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/

Aaron Tomlin (2):
  hung_task: Reset warning budget when problem gets resolved
  hung_task: Log summary line when warning budget is exhausted

 Documentation/admin-guide/sysctl/kernel.rst |  5 +-
 kernel/hung_task.c                          | 55 +++++++++++++++++----
 2 files changed, 49 insertions(+), 11 deletions(-)

-- 
2.55.0

[PATCH v10 1/2] hung_task: Reset warning budget when problem gets resolved

[Aaron Tomlin]

The sysctl hung_task_warnings currently holds both the configured warning
limit and the remaining budget. Each detailed report decrements the
sysctl, so once it reaches zero, the configured limit is lost and cannot
be restored automatically.

Keep sysctl_hung_task_warnings as the configured warning limit and make
khungtaskd the sole owner of the remaining budget. A check that finds no
hung tasks reloads the budget directly from the configured limit. A
successful sysctl write publishes an atomic reset request, which
khungtaskd consumes at the start of the next check.

Suggested-by: Petr Mladek <pmladek@suse.com>
Suggested-by: Lance Yang <lance.yang@linux.dev>
Tested-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  5 ++-
 kernel/hung_task.c                          | 49 +++++++++++++++++----
 2 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index c6994e55d141..b0f8e55efc1a 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -459,8 +459,9 @@ hung_task_warnings
 ==================
 
 The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
-When this value reaches 0, no more warnings will be reported.
+if a hung task is detected, the internal warning budget is decreased by 1.
+When this budget reaches 0, no more detailed warnings will be reported. The
+warning budget is reset to the configured limit when no hung task is found.
 This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
 
 -1: report an infinite number of warnings.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..a5043188456d 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -57,8 +57,20 @@ unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_
  */
 static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
+/*
+ * Limit the number of printed hung tasks to prevent printing
+ * the same or similar backtraces repeatedly.
+ */
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+/*
+ * The number of hung tasks which still can be reported.
+ * The budget gets restored to the original limit when
+ * the previous stall is resolved.
+ */
+static int hung_task_warnings_budget = 10;
+static atomic_t reset_hung_task_warnings = ATOMIC_INIT(0);
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -245,11 +257,11 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
-	 * accordingly
+	 * accordingly with full details if the budget is not exhausted.
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (hung_task_warnings_budget || hung_task_call_panic) {
+		if (hung_task_warnings_budget > 0)
+			hung_task_warnings_budget--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,7 +276,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!sysctl_hung_task_warnings)
+		if (!hung_task_warnings_budget)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -304,7 +316,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = sysctl_hung_task_warnings;
+	int need_warning;
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -314,6 +326,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	if (test_taint(TAINT_DIE) || did_panic)
 		return;
 
+	if (atomic_xchg_acquire(&reset_hung_task_warnings, 0))
+		hung_task_warnings_budget =
+			READ_ONCE(sysctl_hung_task_warnings);
+	need_warning = hung_task_warnings_budget;
+
 	this_round_count = 0;
 	rcu_read_lock();
 	for_each_process_thread(g, t) {
@@ -340,8 +357,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+		hung_task_warnings_budget =
+			READ_ONCE(sysctl_hung_task_warnings);
 		return;
+	}
 
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
@@ -425,6 +445,19 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
+				     void *buffer,
+				     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+	if (!ret && write)
+		atomic_set_release(&reset_hung_task_warnings, 1);
+
+	return ret;
+}
+
 /*
  * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs
  * and hung_task_check_interval_secs
@@ -480,7 +513,7 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.data		= &sysctl_hung_task_warnings,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= proc_dohung_task_warnings,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
 	{
-- 
2.55.0

[PATCH v10 2/2] hung_task: Log summary line when warning budget is exhausted

[Aaron Tomlin]

Once the warning budget is exhausted, hung_task_info() normally stops
printing per-task details. When panic is triggered, full details are
still printed so diagnostics remain available before panic.

To retain visibility without restoring per-task output after budget
exhaustion, emit a single aggregate summary line at the end of each
watchdog scan that detects hung tasks with an exhausted budget. This
keeps non-panic per-task reports budgeted during system-wide hangs.

Suggested-by: Petr Mladek <pmladek@suse.com>
Suggested-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index a5043188456d..49d47ae475ab 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -277,7 +277,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		debug_show_blocker(t, timeout);
 
 		if (!hung_task_warnings_budget)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+			pr_info("hung_task: further per-task details suppressed until warning budget is reset or panic is triggered (see sysctl kernel.hung_task_warnings)\n");
 	}
 
 	touch_nmi_watchdog();
@@ -363,6 +363,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 		return;
 	}
 
+	if (!hung_task_warnings_budget && !hung_task_call_panic)
+		pr_info("hung_task: %lu hung tasks detected, warning budget exhausted\n",
+			this_round_count);
+
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
 
-- 
2.55.0

Re: [PATCH v10 2/2] hung_task: Log summary line when warning budget is exhausted

[Bradley Morgan]

On 29 August 2026 15:53:21 BST, Aaron Tomlin <atomlin@atomlin.com> wrote:
>Once the warning budget is exhausted, hung_task_info() normally stops
>printing per-task details. When panic is triggered, full details are
>still printed so diagnostics remain available before panic.
>
>To retain visibility without restoring per-task output after budget
>exhaustion, emit a single aggregate summary line at the end of each
>watchdog scan that detects hung tasks with an exhausted budget. This
>keeps non-panic per-task reports budgeted during system-wide hangs.
>
>Suggested-by: Petr Mladek <pmladek@suse.com>
>Suggested-by: Lance Yang <lance.yang@linux.dev>
>Reviewed-by: Petr Mladek <pmladek@suse.com>
>Reviewed-by: Lance Yang <lance.yang@linux.dev>

Ok, Reviewed-by: Bradley Morgan <brads@mainlining.org>

>Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
>---
> kernel/hung_task.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
>diff --git a/kernel/hung_task.c b/kernel/hung_task.c
>index a5043188456d..49d47ae475ab 100644
>--- a/kernel/hung_task.c
>+++ b/kernel/hung_task.c
>@@ -277,7 +277,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
> 		debug_show_blocker(t, timeout);
> 
> 		if (!hung_task_warnings_budget)
>-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
>+			pr_info("hung_task: further per-task details suppressed until warning budget is reset or panic is triggered (see sysctl kernel.hung_task_warnings)\n");

Cool.

> 	}
> 
> 	touch_nmi_watchdog();
>@@ -363,6 +363,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> 		return;
> 	}
> 
>+	if (!hung_task_warnings_budget && !hung_task_call_panic)
>+		pr_info("hung_task: %lu hung tasks detected, warning budget exhausted\n",
>+			this_round_count);
>+

Nice!

> 	if (need_warning || hung_task_call_panic) {
> 		si_mask |= SYS_INFO_LOCKS;
> 
>


--- Thanks!
https://lore.kernel.org/all/EE579805-42F2-4C58-B752-F28779EEB717@grrlz.net/