From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64F6C32692B for ; Thu, 9 Apr 2026 14:33:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775745189; cv=none; b=KF2js5yNEdH23x+1fnwZya54mGc/fVnp7Q0FNo5l+/HBI8oFwNtX5KJV7ZY3yErBLdhYay1OSLST1dQnkEtlsirr2IYIpiTSjFCQIl6hLpoBhDCuSxuvr3aYEIGlmbgiep30aIOkRKyjwu3OcY37fLtXtT93hf5Xu5sNTx/xiIM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775745189; c=relaxed/simple; bh=Wo5OjN1z8IpYLECFL1X/Dh5bjjFG55kQsSj7DW89MzI=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=P8aVlVeuXOztFpjjsA11mQBKPf2LQ4ic8Pz2YCHS454NJPSi2olmMgoBdn+i3dc2gnEosQBFckIDLgxKiKl1TF6+Vi5Dz9QbIwIZ3kPuXgyEdr5IgEw08MJLWC1WvVItcZbX2BvjgMzDjYhOomwsdIk5kiW8cbAZQeiMkgjsUQk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=djCNmic9; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=hrIhwm3y; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="djCNmic9"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="hrIhwm3y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1775745187; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hobH8xIAdm6sShBt8ymeYZ5V7PM2aejeN8dxMghifZ8=; b=djCNmic9o1skaA7OiT4ucxyM8rG7lswjAR6hj0UeSwovKu+AwzO9Vt2tryT/kEJ8ygbWIo +Y2UT4McRS9cU+cGTgEgM8R4ddJewfx6jb5K0H3ZkrGeJcyjIqKkM2O0lhYwfRJK5ygcld HyO5IdJ8z4KNkILyfMD2mDfmmV7OB4c= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-609-j-d33Q2TOmKR8h0VQKNaxQ-1; Thu, 09 Apr 2026 10:33:06 -0400 X-MC-Unique: j-d33Q2TOmKR8h0VQKNaxQ-1 X-Mimecast-MFC-AGG-ID: j-d33Q2TOmKR8h0VQKNaxQ_1775745185 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-50da529ff48so28650481cf.3 for ; Thu, 09 Apr 2026 07:33:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1775745185; x=1776349985; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=hobH8xIAdm6sShBt8ymeYZ5V7PM2aejeN8dxMghifZ8=; b=hrIhwm3yFtRxT4dngBpenDynPqQM57Ulh/0Y86y2ABOEiPTYCD2jb7oSiuJjMG7XjG CyXfjkEjIDFzFMro5hdiitNHHxV+iPWlhCExWwXxPik6savKR75nhMRpePvQxIVg7ZWr Y9gmO+PuAljDn8r5SEM7/MtqO6N3luEZecXWO8h8N4tRW2pzHkfo5XiYzkqoVMtfoBV6 kn838LB9t17l1rb4oNbcOfxZAdmABVNUyQHUE6bsZTd/9Fvotshdmyuw96jbg1sfANXg EWEkE/bgRPKXQ5j+Yz2wPMTlpOp0DyrBfmkYkHf+RoBJIZ2wfb5Lci+DNoWSHHwj3ljc /b1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775745185; x=1776349985; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hobH8xIAdm6sShBt8ymeYZ5V7PM2aejeN8dxMghifZ8=; b=H7Ski897cmxrIFCwPu5BJ8qX04UOLZqI1CrL1pdQlw9nQ1UwIpDfKG3uKYkRZtH3Tu mRdOU+3UQhzmXP9wOTqhQu2uLMf5eVEuBxhKETVN4ZEfxd5xZhTn/pcWV4wdPiR6n0mQ S9rl/FcWmsjggnPb8fouHwLY1W7exdpGM9WEpnauZV7QKfoTHIeToELYPqijb5W2VG7D Kacf87s9IavAnY/FIpmzUo58u7FpYAm0BQRf7tb9LFfoJxs7j1/40bXrwAHDbS698Ubt qo0N91XUU0DpRKLxH8XTiAO2MGAVUvmo92xbD1njYowf8sQJVvAWA6ajS61WiD5CGdz9 H8/Q== X-Forwarded-Encrypted: i=1; AJvYcCW3R/sHmDIo2AwYn5/jUdc4ch5HqoB1HyNSuwiNWcCXC9JdnjmeKSGRlMzSlVCkKGfXAgmv2s/JuGKI4Qk=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2TmRVXOhEQ3aJvaG6sVd2NA4ZwiWb/BdgvXw2zr1W4osIEeMJ cYcoc62lVXhrs9VzPm8pXDcMRoEoxfaFS+bX/rHO5gB3OVFmroJoM08O7KXyrtm5+tdGolfKW9z ZaqWyvrju76tLd5U2+axb6wthbaLOQPAUl2yebsUwwPMKjwxSmH8Rey2w/w8Y1vfSNFU4B85IBA == X-Gm-Gg: AeBDietwayCTEPMdkvZ4yEq883pnz2x1Fnp0iXVa3LptC84WJ9YN5EzSQrTkMTxtED1 Et6n9sOLbuOH14AwaTGEsbccMApc9VZPJVf/17McvadkVs9F6Oms3H+aUQs8khBIPXPhQ6CDU0S RaH4rUN1mu2TecFyUhEjwhMxuts6T1FXA4mV3Aa5cLevP+dhCG59OgLP2Vq5CgOe0/leg/B0Oyq INW5mUiBC+29wa+H8FaV/kFAGIoqCtUGRvTjKvgbGHzOH/Sxg69SHGlaXqTPYYsD5tXpyddtJEJ WTwldIkTICpC/tzeKwRJRozUDczTl3L5GlVEsGEUBaK6b0voEZrgOm+9kVAse97HBDX4n/4KQA7 yko4BIMCsv/XEbJ+KPjkJhHaYIh2Th4HQUydZ X-Received: by 2002:ac8:7d8a:0:b0:50d:7f91:6bd8 with SMTP id d75a77b69052e-50d7f91764dmr295543541cf.28.1775745184948; Thu, 09 Apr 2026 07:33:04 -0700 (PDT) X-Received: by 2002:ac8:7d8a:0:b0:50d:7f91:6bd8 with SMTP id d75a77b69052e-50d7f91764dmr295542581cf.28.1775745183984; Thu, 09 Apr 2026 07:33:03 -0700 (PDT) Received: from [192.168.3.252] ([74.75.144.57]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-50d4aed181asm211837171cf.0.2026.04.09.07.33.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 09 Apr 2026 07:33:03 -0700 (PDT) Message-ID: Date: Thu, 9 Apr 2026 10:33:02 -0400 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/2] ipmi:watchdog: Fix panic, D-state hang, and lost protection on BMC reset To: corey@minyard.net Cc: openipmi-developer@lists.sourceforge.net, linux-kernel@vger.kernel.org, minyard@acm.org References: <20260407175134.3367345-1-tcamuso@redhat.com> Content-Language: en-US From: Tony Camuso In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 4/7/2026 5:54 PM, Corey Minyard wrote: > On Tue, Apr 07, 2026 at 01:51:32PM -0400, Tony Camuso wrote: >> When the BMC resets while the IPMI watchdog is active, the driver has >> three failure modes depending on timing: >> >> 1. list_add double add panic -- the watchdog daemon retries while the >> static smi_msg/recv_msg structures are still queued in the IPMI >> layer from the previous (unanswered) request. > > I'm trying to make sense of this. Are you sure this didn't start > happening after you added a timeout on the wait_for_completion()? > Otherwise it would never return, the mutex would be held, and no new > message could be added. > > Just timing out in wait_for_completion() there could cause all kinds of > bad things to happen. > You're right. This work was done on a RHEL 9 kernel that did not yet have your recent upstream KCS/SI fixes applied, so some of the behavior I observed may have been caused/influenced by bugs you've more recently addressed. >> >> 2. D-state hang -- wait_for_completion() blocks indefinitely because >> the BMC never delivers a response. > > This is an issue. The lower level driver is *always* supposed to return > a failure. Something else needs to be fixed. > > I have seen several creative ways in which BMCs "fail to respond" that > have confused the lower level drivers. If my guess is correct, there's > a bug in the low level driver that's causing it to not time out the > message. > > If we don't fix this, it will cause other issues outside the watchdog. > Agreed -- the D-state hang is a symptom, not the root cause. If the KCS driver correctly transitions through error recovery to SI_SM_HOSED, and the SI layer returns an error completion to the caller, then wait_for_completion() should never block indefinitely. To get to the bottom of this, I've instrumented three layers: - ipmi_kcs_sm.c: trace entry into start_error_recovery() and the transition to KCS_HOSED after MAX_ERROR_RETRIES - ipmi_si_intf.c: trace return_hosed_msg(), the SI_SM_HOSED handler in smi_event_handler(), and HOSED recovery in smi_timeout() - ipmi_watchdog.c: trace message send/completion in _ipmi_set_timeout() and __ipmi_heartbeat(), and the completion code received in ipmi_wdog_msg_handler() I've applied your recent upstream patches to my test kernel, so the KCS/SI code is congruent with current mainline. The traces will show whether the error recovery chain works correctly with your fixes in place, or whether the BMC is doing something that still confuses the low-level driver. I'll collect the data and follow up. >> >> 3. Silent loss of watchdog protection -- the BMC returns a non-zero >> completion code, the driver's internal state becomes inconsistent, >> writes to /dev/watchdog return -EINVAL, and the daemon gives up. >> The system continues running without hardware watchdog coverage. > > Again, are you sure this didn't start happening after you added the > timeout? > I think this one is pre-existing, independent of any timeout changes. When the BMC comes back after a reset and returns a non-zero completion code (e.g. 0xD5 or 0xFF), the watchdog handler treats this as a permanent failure. The userspace daemon sees -EINVAL on subsequent writes to /dev/watchdog and stops retrying. The system continues running without hardware watchdog coverage, with no indication to the administrator. But I need to confirm this with the instrumented traces on the the patched kernel. I should have traces collected within the next week or so. Tony >> >> All three stem from the same root cause: the static message structures >> and unbounded completion waits were never designed for a BMC that >> disappears mid-transaction. > > All that is supposed to be protected by a mutex. That mutex is claimed > on all IPMI watchdog operations, and it shouldn't be released until all > resources have been freed. Anything that violates that is asking for > trouble. > > You don't mention the lower level interface (KCS, BT, SMIC, SSIF) but I > think we need to start looking there. > > It may be that the timeouts on the watchdog messages need to be > adjusted. The whole IPMI driver was designed on the presumption that > the BMC would go away for only a short period of time (5-10 seconds) and > not permanantly. That has slowly been fixed over time, but things might > need to be adjusted in the watchdog. > > -corey > >>>> This has been independently reported by Kenta Akagi on a Dell PowerEdge >> R640 running 6.18.7, also triggered by a BMC reset with the watchdog >> active: >> >> https://sourceforge.net/p/openipmi/mailman/message/59292850/ >> >> The fix takes a simple, deterministic approach: detect the failure via >> BMC error response, guard against structure reuse (msg_in_flight) and >> indefinite waits (completion timeout), then initiate orderly_reboot() >> when the watchdog is active. This produces the same outcome the >> hardware watchdog would have -- a system reset -- but through a >> controlled path with clear logging and no panics or hangs. >> >> If the watchdog is stopped when the BMC resets, no reboot occurs and >> the system continues normally. >> >> Tested on Dell PowerEdge R640 with kernel 5.14 (RHEL 9) and verified >> against mainline (both patches apply cleanly). >> >> Corey Minyard's recent fix for list corruption in smi_work() >> (ipmi_msghandler.c) addresses a related but separate code path. The >> watchdog driver's own static structure reuse requires this fix. >> >> Tony Camuso (2): >> ipmi:watchdog: Reboot cleanly on BMC reset >> Documentation: ipmi: Update BMC reset behavior for watchdog >> >> Documentation/driver-api/ipmi.rst | 61 ++++++++++++++++++ >> drivers/char/ipmi/ipmi_watchdog.c | 101 ++++++++++++++++++++++++------ >> 2 files changed, 144 insertions(+), 18 deletions(-) >> >> -- >> 2.53.0 >> >