From: Mayank Rungta via B4 Relay <devnull+mrungta.google.com@kernel.org>
To: Petr Mladek <pmladek@suse.com>,
Jinchao Wang <wangjinchao600@gmail.com>,
Yunhui Cui <cuiyunhui@bytedance.com>,
Stephane Eranian <eranian@google.com>,
Ian Rogers <irogers@google.com>,
Li Huafei <lihuafei1@huawei.com>,
Feng Tang <feng.tang@linux.alibaba.com>,
Max Kellermann <max.kellermann@ionos.com>,
Jonathan Corbet <corbet@lwn.net>,
Douglas Anderson <dianders@chromium.org>,
Andrew Morton <akpm@linux-foundation.org>,
Florian Delizy <fdelizy@google.com>,
Shuah Khan <skhan@linuxfoundation.org>
Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
Mayank Rungta <mrungta@google.com>
Subject: [PATCH v2 2/5] watchdog: Update saved interrupts during check
Date: Thu, 12 Mar 2026 16:22:03 -0700 [thread overview]
Message-ID: <20260312-hardlockup-watchdog-fixes-v2-2-45bd8a0cc7ed@google.com> (raw)
In-Reply-To: <20260312-hardlockup-watchdog-fixes-v2-0-45bd8a0cc7ed@google.com>
From: Mayank Rungta <mrungta@google.com>
Currently, arch_touch_nmi_watchdog() causes an early return that
skips updating hrtimer_interrupts_saved. This leads to stale
comparisons and delayed lockup detection.
I found this issue because in our system the serial console is fairly
chatty. For example, the 8250 console driver frequently calls
touch_nmi_watchdog() via console_write(). If a CPU locks up after a
timer interrupt but before next watchdog check, we see the following
sequence:
* watchdog_hardlockup_check() saves counter (e.g., 1000)
* Timer runs and updates the counter (1001)
* touch_nmi_watchdog() is called
* CPU locks up
* 10s pass: check() notices touch, returns early, skips update
* 10s pass: check() saves counter (1001)
* 10s pass: check() finally detects lockup
This delays detection to 30 seconds. With this fix, we detect the
lockup in 20 seconds.
Reviewed-by: Douglas Anderson <dianders@chromium.org>
Signed-off-by: Mayank Rungta <mrungta@google.com>
---
kernel/watchdog.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 4c5b47495745..431c540bd035 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -159,21 +159,28 @@ void watchdog_hardlockup_touch_cpu(unsigned int cpu)
per_cpu(watchdog_hardlockup_touched, cpu) = true;
}
-static bool is_hardlockup(unsigned int cpu)
+static void watchdog_hardlockup_update(unsigned int cpu)
{
int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu));
- if (per_cpu(hrtimer_interrupts_saved, cpu) == hrint)
- return true;
-
/*
* NOTE: we don't need any fancy atomic_t or READ_ONCE/WRITE_ONCE
* for hrtimer_interrupts_saved. hrtimer_interrupts_saved is
* written/read by a single CPU.
*/
per_cpu(hrtimer_interrupts_saved, cpu) = hrint;
+}
+
+static bool is_hardlockup(unsigned int cpu)
+{
+ int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu));
+
+ if (per_cpu(hrtimer_interrupts_saved, cpu) != hrint) {
+ watchdog_hardlockup_update(cpu);
+ return false;
+ }
- return false;
+ return true;
}
static void watchdog_hardlockup_kick(void)
@@ -191,6 +198,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
unsigned long flags;
if (per_cpu(watchdog_hardlockup_touched, cpu)) {
+ watchdog_hardlockup_update(cpu);
per_cpu(watchdog_hardlockup_touched, cpu) = false;
return;
}
--
2.53.0.851.ga537e3e6e9-goog
next prev parent reply other threads:[~2026-03-12 23:22 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-12 23:22 [PATCH v2 0/5] watchdog/hardlockup: Improvements to hardlockup detection and documentation Mayank Rungta via B4 Relay
2026-03-12 23:22 ` [PATCH v2 1/5] watchdog: Return early in watchdog_hardlockup_check() Mayank Rungta via B4 Relay
2026-03-13 15:27 ` Doug Anderson
2026-03-23 15:47 ` Petr Mladek
2026-03-12 23:22 ` Mayank Rungta via B4 Relay [this message]
2026-03-13 15:27 ` [PATCH v2 2/5] watchdog: Update saved interrupts during check Doug Anderson
2026-03-23 15:58 ` Petr Mladek
2026-03-12 23:22 ` [PATCH v2 3/5] doc: watchdog: Clarify hardlockup detection timing Mayank Rungta via B4 Relay
2026-03-12 23:22 ` [PATCH v2 4/5] watchdog/hardlockup: improve buddy system detection timeliness Mayank Rungta via B4 Relay
2026-03-23 16:26 ` Petr Mladek
2026-03-12 23:22 ` [PATCH v2 5/5] doc: watchdog: Document buddy detector Mayank Rungta via B4 Relay
2026-03-23 17:26 ` Petr Mladek
2026-03-23 22:45 ` Doug Anderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260312-hardlockup-watchdog-fixes-v2-2-45bd8a0cc7ed@google.com \
--to=devnull+mrungta.google.com@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=corbet@lwn.net \
--cc=cuiyunhui@bytedance.com \
--cc=dianders@chromium.org \
--cc=eranian@google.com \
--cc=fdelizy@google.com \
--cc=feng.tang@linux.alibaba.com \
--cc=irogers@google.com \
--cc=lihuafei1@huawei.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=max.kellermann@ionos.com \
--cc=mrungta@google.com \
--cc=pmladek@suse.com \
--cc=skhan@linuxfoundation.org \
--cc=wangjinchao600@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®