From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00E1D40D59E for ; Mon, 22 Jun 2026 00:18:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782087534; cv=none; b=cZ8HVix1kTJqri+doN43R9iNWVjNdjqROT2t6YOE4F4a3LUeEI/g3Ta+7yvb084aJao1EK6js4AIoUBHYrTwlmEMKywRqpuyF6y/CBhIv9nAsoiblmRPyVPUHu49ohSGiEq+22ARPeVEFV7bHKdFy50U9L8I/HE0mokTFS1mjZw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782087534; c=relaxed/simple; bh=Hm5KYwF5crVO58YTlhQXjz28/s20XrsrCT+ZelaHi4o=; h=Date:From:To:Cc:Subject:Message-Id:In-Reply-To:References: Mime-Version:Content-Type; b=o/+5KYMDJl6/AWG42WYwbdS7zdNXpX+hA37nEYU1N9lqhhP/afnC0QdbGrArXCj56hnv1g/I8j/X7luTAKdtn1lVtWfrHC0OSFwkYnq8WOCBhLy3KZ+Ap2URmdvxuxXjFgfMj+lUljFWfg6areU42Rt3tjYEjiOAWc+3iNSuhQY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jCT+wvBY; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jCT+wvBY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 15BD81F000E9; Mon, 22 Jun 2026 00:18:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1782087532; bh=ZWxVS39WlYkb4hPpDmMNVJPE2RNQpEjTYakRsNJ7L0E=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=jCT+wvBY0bd8+6aWviOx1fAygz3NFkeYTqs6tVddlv6HQmzjhdjLqdcY4lARvaglB 3m5ms+7g3QLzI9R4aIdiR4GFwYLI0R/ocBOwVckANgsNiXDav8ZQ+5+OLyz0dm7k6B Z73jvqZKD43pOoOb2BRfDGuuXcufikFSLEY9AGdUcmCbrqoLKG3dJXq4g651F4Va+2 3d/8kHzAFra07dC39IngOyVpENjCFxcRJqlivV5XX7Wxn9NdMncdeF1CRPcy1ib1BS CgQ+spN4BRptVMJOLsIC4iqgnJXzRO4zheCR0iZAdAHQ+T79vFuP00L4zkLVJEvYU4 3r0tLptkqGvBQ== Date: Mon, 22 Jun 2026 09:18:47 +0900 From: Masami Hiramatsu (Google) To: Aaron Tomlin Cc: akpm@linux-foundation.org, lance.yang@linux.dev, pmladek@suse.com, linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: Re: [PATCH v3] hung_task: deduplicate identical hang reports Message-Id: <20260622091847.45d881387f8e84478758af01@kernel.org> In-Reply-To: <20260621213756.43225-1-atomlin@atomlin.com> References: <20260621213756.43225-1-atomlin@atomlin.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Sun, 21 Jun 2026 17:37:56 -0400 Aaron Tomlin wrote: > Currently, during severe lock contention, multiple tasks can hang while > waiting on the exact same resource. The khungtaskd kthread > indiscriminately reports every single instance with a stack trace. > This can roll the kernel ring buffer and prematurely exhaust the > kernel.hung_task_warnings budget. Consequently, the kernel is left > entirely blind to subsequent, unrelated deadlocks. > > To preserve the warning budget and ring buffer without sacrificing > observability, introduce a Wait Channel (wchan) and task-state based > deduplicator: > > 1. Implement a lightweight, stack-allocated 64-slot Wait Channel > (wchan) hash map. Tasks blocked on the exact same wchan during a > single scan are recognised as sharing the same bottleneck, > successfully deduplicating contentions even when the callers > possess entirely disparate call stacks. Hmm, wouldn't this essentially erase everything that's typically expected in a standard lock? Ideally, we'd like to sort by the time the lock was first blocked and display only the oldest stack. > > 2. Introduce a hung_task_reported bit-field in task_struct. If a task > remains hung across multiple intervals, khungtaskd recognises it > has already been reported. The bit is safely cleared without > locks or atomics the moment the task's context switch counter > increments. > > 3. For duplicate tasks, we still print the single-line > "INFO: task ..." message and trigger tracepoint > trace_sched_process_hang(). It merely skips calling > sched_show_task() and debug_show_blocker(), printing a concise > suppression notice instead. Ah, OK. So if we need more information, we can record it on trace ring buffer. > > Signed-off-by: Aaron Tomlin > -- > Changes since v2: > > - Replaced the per-round cache flush with a task_struct bit-field for > persistent cross-scan tracking, mitigating delayed budget exhaustion > > - Abandoned exact-stack hashing in favour of Wait Channel hashing > > - Transitioned from jhash() to hash_long() to optimise single-pointer > hashing, and relocated the hash map to the local stack > > - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/ > > Changes since v1: > > - Preserve "INFO:" headers for all hung tasks; suppress only the stack > dumps for duplicates (Masami Hiramatsu) > > - Print a clear notification when a trace is explicitly suppressed > > - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors > > - Optimise overhead by unwinding the stack only if a warning is > actually going to be printed > > - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/ > --- > include/linux/sched.h | 3 +++ > kernel/hung_task.c | 32 ++++++++++++++++++++++++++++---- > 2 files changed, 31 insertions(+), 4 deletions(-) > > diff --git a/include/linux/sched.h b/include/linux/sched.h > index b3204a15d512..e76cf221cc78 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -1046,6 +1046,9 @@ struct task_struct { > /* Used by page_owner=on to detect recursion in page tracking. */ > unsigned in_page_owner:1; > #endif > +#ifdef CONFIG_DETECT_HUNG_TASK > + unsigned hung_task_reported:1; > +#endif > #ifdef CONFIG_EVENTFD > /* Recursion prevention for eventfd_signal() */ > unsigned in_eventfd:1; > diff --git a/kernel/hung_task.c b/kernel/hung_task.c > index 6fcc94ce4ca9..5dcce0e7041b 100644 > --- a/kernel/hung_task.c > +++ b/kernel/hung_task.c > @@ -25,6 +25,7 @@ > #include > #include > #include > +#include > > #include > > @@ -125,6 +126,7 @@ static bool task_is_hung(struct task_struct *t, unsigned long timeout) > if (switch_count != t->last_switch_count) { > t->last_switch_count = switch_count; > t->last_switch_time = jiffies; > + t->hung_task_reported = 0; > return false; > } > if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) > @@ -228,12 +230,14 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti > * @t: Pointer to the detected hung task. > * @timeout: Timeout threshold for detecting hung tasks > * @this_round_count: Count of hung tasks detected in the current iteration > + * @skip_show_task: Indicating if stack trace should be skipped > * > * Print structured information about the specified hung task, if warnings > * are enabled or if the panic batch threshold is exceeded. > */ > static void hung_task_info(struct task_struct *t, unsigned long timeout, > - unsigned long this_round_count) > + unsigned long this_round_count, > + unsigned int skip_show_task) > { > trace_sched_process_hang(t); > > @@ -261,8 +265,12 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, > pr_err(" Blocked by coredump.\n"); > pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\"" > " disables this message.\n"); > - sched_show_task(t); > - debug_show_blocker(t, timeout); > + if (!skip_show_task) { > + sched_show_task(t); > + debug_show_blocker(t, timeout); > + } else { > + pr_err(" Stack trace suppressed. Already reported or duplicate wchan\n"); Can we show the wchan hash for each task, so that we can see which tasks are waiting on the same wchan? Thanks, > + } > > if (!sysctl_hung_task_warnings) > pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); > @@ -306,6 +314,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > unsigned long this_round_count; > int need_warning = sysctl_hung_task_warnings; > unsigned long si_mask = hung_task_si_mask; > + unsigned long wchan, wchan_hash[64] = { 0 }; > + unsigned int hash; > + unsigned int skip_show_task; > > /* > * If the system crashed already then all bets are off, > @@ -326,6 +337,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > } > > if (task_is_hung(t, timeout)) { > + skip_show_task = t->hung_task_reported; > /* > * Increment the global counter so that userspace could > * start migrating tasks ASAP. But count the current > @@ -334,7 +346,19 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > */ > atomic_long_inc(&sysctl_hung_task_detect_count); > this_round_count++; > - hung_task_info(t, timeout, this_round_count); > + > + wchan = get_wchan(t); > + if (wchan) { > + hash = hash_long(wchan, 6); > + if (wchan_hash[hash] == wchan) > + skip_show_task = 1; > + else > + wchan_hash[hash] = wchan; > + } > + > + hung_task_info(t, timeout, this_round_count, > + skip_show_task); > + t->hung_task_reported = 1; > } > } > unlock: > -- > 2.51.0 > -- Masami Hiramatsu (Google)