From: Tony Camuso <tcamuso@redhat.com>
To: Frederick Lawler <fred@cloudflare.com>
Cc: corey@minyard.net, Matt Fleming <matt@readmodwrite.com>,
openipmi-developer@lists.sourceforge.net,
linux-kernel@vger.kernel.org, kernel-team@cloudflare.com,
Matt Fleming <mfleming@cloudflare.com>
Subject: Re: [PATCH] ipmi: Add timeout to unconditional wait in __get_device_id()
Date: Thu, 16 Apr 2026 10:28:50 -0400 [thread overview]
Message-ID: <cb6c3b2d-8c22-4f62-9157-a5918c1e1ea0@redhat.com> (raw)
In-Reply-To: <aeABpewNzo4MURpO@CMGLRV3>
On 4/15/26 5:22 PM, Frederick Lawler wrote:
> Hi Corey & Tony,
>
> On Wed, Apr 15, 2026 at 11:46:27AM -0400, 'Tony Camuso' via kernel-team wrote:
>> On Wed, Apr 15, 2026 at 12:59:30PM +0100, Matt Fleming wrote:
>>> From: Matt Fleming <mfl...@cl...>
>>>
>>> When the BMC does not respond to a "Get Device ID" command, the
>>> wait_event() in __get_device_id() blocks forever in
>>> TASK_UNINTERRUPTIBLE while holding bmc->dyn_mutex. Every subsequent
>>> sysfs reader then piles up in D state. Replace with
>>> wait_event_timeout() to return -EIO after 1 second.
>>
>> On Wed, Apr 15, 2026 at 12:17:04PM, Corey Minyard wrote:
>>> This is the second report I have of something like this. So
>>> something is up. I'm adding Tony, who reported something like this
>>> dealing with the watchdog.
>>>
>>> The lower level driver should never not return an answer, it is
>>> supposed to guarantee that it returns an error if the BMC doesn't
>>> respond. So the bug is not here, the bug is elsewhere.
>
> This is a bit of a throwback to our previous discussions around [1].
>
> I did end up applying [2] based on that discussion, and had limited
> success, but we still have external resets that cause us to enter
> this undesirable state :(
In my testing with updates from the Linus tree, after a BMC cold reset:
1. The KCS driver returned -EBUSY to callers (good)
2. The watchdog daemon received the error and initiated shutdown
3. No D-state hang
My tests, conducted on a Dell PER640, verified that Corey's upstream fixes
cause the driver to properly return errors instead of blocking.
At least on that platform.
Which hich low-level driver are you using (KCS, BT, SSIF)?
The PER640 uses KCS.
# cat /sys/class/ipmi/ipmi0/device/params 2>/dev/null
kcs,i/o,0xca8,rsp=4,rsi=1,rsh=0,irq=10,ipmb=32
> [1]: https://lore.kernel.org/all/aJUMlAG17c6lCgFq@mail.minyard.net/
> [2]: https://lore.kernel.org/all/20250807230648.1112569-2-corey@minyard.net/
>>
>> I've been tracking a related issue (RHEL customer case) where BMC
>> reset while the IPMI watchdog is active causes D-state hangs. This
>> appears to be the same root cause Matt is hitting.
>>
>> I backported the recent upstream KCS/SI fixes to a RHEL 9 test kernel
>> (54 patches bringing it to mainline parity) and tested today on a
>> Dell R640.
>
> I assume this patch series: "ipmi:watchdog: Fix panic, D-state hang, and
> lost protection on BMC reset" [3]?
>
> [3]: https://lore.kernel.org/all/20260407175134.3367345-1-tcamuso@redhat.com/
>
Actually, no. The 54 commits I backported simply bring my RHEL-9 test kernel
to parity with the Linus tree, which includes [2] and ...
cae66f1a1dcd 2026-02-13 corey@minyard.net ipmi:si: Fix check for a misbehaving BMC
next prev parent reply other threads:[~2026-04-16 14:28 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-15 11:59 Matt Fleming
2026-04-15 12:16 ` Corey Minyard
2026-04-15 15:46 ` Tony Camuso
2026-04-15 21:22 ` Frederick Lawler
2026-04-16 14:28 ` Tony Camuso [this message]
2026-04-17 16:01 ` Matt Fleming
2026-04-17 15:41 ` Matt Fleming
2026-04-17 22:23 ` Matt Fleming
2026-04-17 23:53 ` Corey Minyard
2026-04-19 20:50 ` Matt Fleming
2026-04-20 16:33 ` Corey Minyard
2026-04-20 18:11 ` Corey Minyard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cb6c3b2d-8c22-4f62-9157-a5918c1e1ea0@redhat.com \
--to=tcamuso@redhat.com \
--cc=corey@minyard.net \
--cc=fred@cloudflare.com \
--cc=kernel-team@cloudflare.com \
--cc=linux-kernel@vger.kernel.org \
--cc=matt@readmodwrite.com \
--cc=mfleming@cloudflare.com \
--cc=openipmi-developer@lists.sourceforge.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®