[PATCH v3] hung_task: Explicitly report I/O wait state in log output

[PATCH v3] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

Currently, the hung task reporting mechanism indiscriminately labels all
TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
are awaiting I/O completion or kernel locking primitives. This ambiguity
compels system administrators to manually inspect stack traces to discern
whether the delay stems from an I/O wait (typically indicative of
hardware or filesystem anomalies) or software contention. Such detailed
analysis is not always immediately accessible to system administrators
or support engineers.

To address this, this patch utilises the existing in_iowait field within
struct task_struct to augment the failure report. If the task is blocked
due to I/O (e.g., via io_schedule_prepare()), the log message is updated
to explicitly state "blocked in I/O wait".

Examples:
        - Standard Block: "INFO: task bash:123 blocked for more than 120
          seconds".

        - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
          120 seconds".

Accessing in_iowait is safe in this context. The detector holds
rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
task structure remains valid in memory. Furthermore, as the task is
confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
modify its own in_iowait flag, rendering the read operation stable and
free from data races.

Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8bc043fbe89c..6fcc94ce4ca9 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
-- 
2.51.0

Re: [PATCH v3] hung_task: Explicitly report I/O wait state in log output

[Petr Mladek]

On Sat 2026-02-07 19:24:58, Aaron Tomlin wrote:
> Currently, the hung task reporting mechanism indiscriminately labels all
> TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
> are awaiting I/O completion or kernel locking primitives. This ambiguity
> compels system administrators to manually inspect stack traces to discern
> whether the delay stems from an I/O wait (typically indicative of
> hardware or filesystem anomalies) or software contention. Such detailed
> analysis is not always immediately accessible to system administrators
> or support engineers.
> 
> To address this, this patch utilises the existing in_iowait field within
> struct task_struct to augment the failure report. If the task is blocked
> due to I/O (e.g., via io_schedule_prepare()), the log message is updated
> to explicitly state "blocked in I/O wait".
> 
> Examples:
>         - Standard Block: "INFO: task bash:123 blocked for more than 120
>           seconds".
> 
>         - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
>           120 seconds".
> 
> Accessing in_iowait is safe in this context. The detector holds
> rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
> task structure remains valid in memory.

This is true.

> Furthermore, as the task is
> confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> modify its own in_iowait flag, rendering the read operation stable and
> free from data races.

IMHO, this is not true. The blocked tasks might wake up at any time.
There is a small chance to print an inconsistent information.
But we could live with it. The entire hung task report is racy.

The race would actually be a lucky moment where the task get
unblocked. In this case, it won't be reported in the next
round...

I suggest to omit the entire paragraph.

> Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>

Othrewise, it looks good.

With the removed paragraph:

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

Re: [PATCH v3] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]

On Mon, Feb 09, 2026 at 11:14:27AM +0100, Petr Mladek wrote:
> > Accessing in_iowait is safe in this context. The detector holds
> > rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
> > task structure remains valid in memory.
> 
> This is true.
> 
> > Furthermore, as the task is
> > confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> > modify its own in_iowait flag, rendering the read operation stable and
> > free from data races.
> 
> IMHO, this is not true. The blocked tasks might wake up at any time.
> There is a small chance to print an inconsistent information.
> But we could live with it. The entire hung task report is racy.
> 
> The race would actually be a lucky moment where the task get
> unblocked. In this case, it won't be reported in the next
> round...
> 
> I suggest to omit the entire paragraph.
> 
> > Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> > Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> 
> Othrewise, it looks good.
> 
> With the removed paragraph:
> 
> Reviewed-by: Petr Mladek <pmladek@suse.com>

Hi Petr, Lance,

Apologies. The old commit message was erroneously used.

I would prefer to replace the last paragraph with the following to reflect
that this is a diagnostic snapshot only:

    Theoretically, io_schedule_finish() could be called immediately after
    khungtaskd checks the flag, rendering the output stale. However,
    strictly preventing this would require blocking the waking task (e.g.,
    within the mutex unlock code path - see mutex_lock_io()) to inhibit the
    state change during the scan. This seems entirely disproportionate for
    a "best effort" diagnostic tool, especially given the probability of
    such a race is negligible after a long hang, by default.

Please let me know your thoughts.


Kind regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] hung_task: Explicitly report I/O wait state in log output

[Petr Mladek]

On Mon 2026-02-09 09:30:34, Aaron Tomlin wrote:
> On Mon, Feb 09, 2026 at 11:14:27AM +0100, Petr Mladek wrote:
> > > Accessing in_iowait is safe in this context. The detector holds
> > > rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
> > > task structure remains valid in memory.
> > 
> > This is true.
> > 
> > > Furthermore, as the task is
> > > confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> > > modify its own in_iowait flag, rendering the read operation stable and
> > > free from data races.
> > 
> > IMHO, this is not true. The blocked tasks might wake up at any time.
> > There is a small chance to print an inconsistent information.
> > But we could live with it. The entire hung task report is racy.
> > 
> > The race would actually be a lucky moment where the task get
> > unblocked. In this case, it won't be reported in the next
> > round...
> > 
> > I suggest to omit the entire paragraph.
> > 
> > > Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> > > Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> > 
> > Othrewise, it looks good.
> > 
> > With the removed paragraph:
> > 
> > Reviewed-by: Petr Mladek <pmladek@suse.com>
> 
> Hi Petr, Lance,
> 
> Apologies. The old commit message was erroneously used.
> 
> I would prefer to replace the last paragraph with the following to reflect
> that this is a diagnostic snapshot only:
> 
>     Theoretically, io_schedule_finish() could be called immediately after
>     khungtaskd checks the flag, rendering the output stale. However,
>     strictly preventing this would require blocking the waking task (e.g.,
>     within the mutex unlock code path - see mutex_lock_io()) to inhibit the
>     state change during the scan. This seems entirely disproportionate for
>     a "best effort" diagnostic tool, especially given the probability of
>     such a race is negligible after a long hang, by default.
> 
> Please let me know your thoughts.

Strictly speaking, io_schedule_finish() might be called even before
khungtaskd checks the flag. It can be called in parallel at any time.
The entire khungtaskd out is racy. But it is good enough in practice,
especially because it reports stalls. The chance that a long stall
gets finished while the report is being printed is very small.
And you are right. Any attempt to serialize the output seems
disproportionate here. It might make the stall even worse.

If you really want to write something about the possible race
then I would write something like:

  Theoretically, io_schedule_finish() could be called in parallel
  and the read flag need not match the later printed backtrace.
  It should be acceptable in practice. The entire report is
  racy. And it works most of the time, especially because
  long stalls are being reported. And adding any extra
  synchronization seems entirely disproportionate in this
  scenario.

Best Regards,
Petr

Re: [PATCH v3] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Tue, Feb 10, 2026 at 10:40:40AM +0100, Petr Mladek wrote:
> If you really want to write something about the possible race
> then I would write something like:
> 
>   Theoretically, io_schedule_finish() could be called in parallel
>   and the read flag need not match the later printed backtrace.
>   It should be acceptable in practice. The entire report is
>   racy. And it works most of the time, especially because
>   long stalls are being reported. And adding any extra
>   synchronization seems entirely disproportionate in this
>   scenario.
> 
Hi Petr,

Thank you. This is acceptable.

Hi Andrew,

Would you mind replacing the last paragraph of the current commit message
with the version above? I believe it would be pertinent to include the race
condition, notwithstanding khungtaskd by design.

Kind regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]