mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Tony Camuso <tcamuso@redhat.com>
To: corey@minyard.net, Matt Fleming <matt@readmodwrite.com>
Cc: openipmi-developer@lists.sourceforge.net,
	linux-kernel@vger.kernel.org, kernel-team@cloudflare.com,
	Matt Fleming <mfleming@cloudflare.com>
Subject: Re: [PATCH] ipmi: Add timeout to unconditional wait in __get_device_id()
Date: Wed, 15 Apr 2026 11:46:27 -0400	[thread overview]
Message-ID: <9b6af9ab-79f9-4f87-ab7c-8ad6efeb18ed@redhat.com> (raw)
In-Reply-To: <ad-BtS5b3qiowqb7@mail.minyard.net>

On Wed, Apr 15, 2026 at 12:59:30PM +0100, Matt Fleming wrote:
> From: Matt Fleming <mfl...@cl...>
> 
> When the BMC does not respond to a "Get Device ID" command, the
> wait_event() in __get_device_id() blocks forever in
> TASK_UNINTERRUPTIBLE while holding bmc->dyn_mutex. Every subsequent
> sysfs reader then piles up in D state. Replace with
> wait_event_timeout() to return -EIO after 1 second.

On Wed, Apr 15, 2026 at 12:17:04PM, Corey Minyard wrote:
> This is the second report I have of something like this.  So
> something is up.  I'm adding Tony, who reported something like this
> dealing with the watchdog.
>
> The lower level driver should never not return an answer, it is
> supposed to guarantee that it returns an error if the BMC doesn't
> respond. So the bug is not here, the bug is elsewhere.

I've been tracking a related issue (RHEL customer case) where BMC
reset while the IPMI watchdog is active causes D-state hangs. This
appears to be the same root cause Matt is hitting.

I backported the recent upstream KCS/SI fixes to a RHEL 9 test kernel
(54 patches bringing it to mainline parity) and tested today on a
Dell R640.

Test: Trigger `ipmitool mc reset cold` while watchdog daemon is
running.

Results with backported fixes:

   [  245.376402] IPMI Watchdog: heartbeat completion received
   [  246.376392] IPMI Watchdog: heartbeat send failure: -16
   [  247.377560] IPMI Watchdog: heartbeat send failure: -16
   ...
   [  252.413240] IPMI Watchdog: set timeout error: -16

The watchdog daemon received error 16 (EBUSY) and eventually
initiated orderly shutdown:

   write watchdog device gave error 16 = 'Device or resource busy'
   shutting down the system because of error 16

Key finding: With the upstream fixes, the driver returns -EBUSY
instead of blocking forever. No D-state hang. The watchdog daemon
handles the error and initiates orderly reboot.

Note: There was still a delay of several minutes before the daemon
timed out and triggered shutdown. The driver returned errors
promptly, but the watchdog daemon's retry logic (error retry
time-out = 120 seconds) extended the overall recovery time. This
may warrant a separate look at whether the daemon's retry behavior
is appropriate when the BMC is completely unresponsive.

This confirms Corey's assessment - the bug is in the lower-level
driver not returning errors, not in __get_device_id(). Matt's
timeout patch would be a defensive fallback, but the real fix is
ensuring KCS/SI properly returns errors when the BMC is
unresponsive.

Tony


  reply	other threads:[~2026-04-15 15:46 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-04-15 11:59 Matt Fleming
2026-04-15 12:16 ` Corey Minyard
2026-04-15 15:46   ` Tony Camuso [this message]
2026-04-15 21:22     ` Frederick Lawler
2026-04-16 14:28       ` Tony Camuso
2026-04-17 16:01         ` Matt Fleming
2026-04-17 15:41   ` Matt Fleming
2026-04-17 22:23   ` Matt Fleming
2026-04-17 23:53     ` Corey Minyard
2026-04-19 20:50       ` Matt Fleming
2026-04-20 16:33         ` Corey Minyard
2026-04-20 18:11           ` Corey Minyard

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9b6af9ab-79f9-4f87-ab7c-8ad6efeb18ed@redhat.com \
    --to=tcamuso@redhat.com \
    --cc=corey@minyard.net \
    --cc=kernel-team@cloudflare.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=matt@readmodwrite.com \
    --cc=mfleming@cloudflare.com \
    --cc=openipmi-developer@lists.sourceforge.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®