From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C113D37C936 for ; Tue, 7 Apr 2026 21:54:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775598891; cv=none; b=hsAhQVIdS4Tb9XXpHe6o9dJqsO1u9U4glxJi4jK5/lFiUEvEMNGBC5bv2cHMl6mV/RnK35rUAZW/aqyyYWsXB66DNrdMQ6mXE42AB+NTBhvEJLEM9hcZhPcblFbCV3hpfauCv80TH01T1fJolAhxnVXPvKOeqr2OI6fKr8ChYtA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775598891; c=relaxed/simple; bh=3YYP3dT8Gkq2fYkvO87SXgf+quljjFSfCoNeXT6prig=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=HdbRtQxu13uRYc4eTH2CwsSnab6GOEohfXj7aGwj3qTkd0i9omFNVAGu82IqcD8TcUnpP8DmPN6ONTLhDaiozmnGMOqR6alxjkXRh//SPDPLJvcd+DkeSxHBFll4mWMbdhQN0pCIhxNWenjLdsINsPuTfCZ7lnUBwpsLgk5Knks= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=minyard.net; spf=pass smtp.mailfrom=minyard.net; dkim=pass (2048-bit key) header.d=minyard.net header.i=@minyard.net header.b=LSyDTxEX; arc=none smtp.client-ip=209.85.210.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=minyard.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=minyard.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=minyard.net header.i=@minyard.net header.b="LSyDTxEX" Received: by mail-ot1-f52.google.com with SMTP id 46e09a7af769-7dbd08144deso2754856a34.0 for ; Tue, 07 Apr 2026 14:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minyard.net; s=google; t=1775598889; x=1776203689; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=TI4j+R7vZ7D2UHMNo8VEOjHd1KyU4QozG4yX67r0d0Y=; b=LSyDTxEX8/K0OvtgiJFtXXRyMZ0VMEgbId72wBD83LeYNV7IuIHKC298fYDNA8hmKa UYAEVa6v+Ws3y8wAivYTXCGBoawoVGS0z3dPX23LO9ujj14QuAjwMJ5ImbDNqptti07j 5eERnvU2y8JY24ZlBRGBDkg8A4sNa+ZGzVQTNPCZLIo4bdNzCAIPv7+YOH4Yj4zJxAt/ T0fkhHnWmSZvuwzve1US46PZPjw2y9mMCIsELF28W2BexahmpXlC3saN3pun/fY94mDd wA9VN6CoGiKwToxCXOnhTVxrAE3RVOcEjMdwKNGQKJvscPwcuvUGOvW0fd/uJ6anmm1D TI2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775598889; x=1776203689; h=in-reply-to:content-disposition:mime-version:references:reply-to :message-id:subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TI4j+R7vZ7D2UHMNo8VEOjHd1KyU4QozG4yX67r0d0Y=; b=DwpNbgI2MhpyrpXvw7rnY6Z6nk3tlnhCb9vT7Q91n/4k75tTtdkvuer27X1S6QuraG t40Z7PNdu8XvOlrZFBH0T0exWxTgDnc6BVeQe5ZJAwzLD/NTfNf/STMJZ+/5I+fg76CX ezOmAsZdBbGpKSz3SrNGoVjZll11ElxzMzscABJaI4+MOvLqQwp5TsAGGb7uMNsVP44E 4sjiQ092frYoFs+7t1i4CZpKGyUGIreR4+ed1C+HiJxZJxkhZe/wHSMlIZKNejsV5CeB 4phRIIPSgMNllxHBpRx9gHNeKtulltTAQAYLBEiSk2q9rXzxOYZBGNHkHtFDufUMVBgs SOBA== X-Forwarded-Encrypted: i=1; AJvYcCWSYTyg1oESmsJroa5yHqYlmgxag+/cnHaEZlXfem2Xt6nOvKsa6Z/7VFkL8vulv7g20QOZXZl9zzp1EpY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy6OAoyZOxDLyUOk+sbRhgtAnYD5YuybJ4rH0JWJs9XgKBsucch GDJvLf9Jko1W3X2fhNhhOe5rKO4WwxEnbx+hTTNcsJzHnSYmVhYFH679G3AjrpxaWys= X-Gm-Gg: AeBDiet3mQOqW/AOod87FN3B/Xk9J2rByNnpES6xzbs2pbGIQXh59Kz4Nc0JZhm5Vfx iIqfgMyaPA+IWIAhhHrzWAJePYTQLWY68a6nuxTvTnVFl1YjggIOFKjyuQP3QXSddJmkKZ/9ZVe jdeMAG5XFJXwrigd18EgbnaUGSBigzbvf05azd+iN8XtNayUDkUsVURujk1Io1RvOFnlTI3YrFA zx4J1IpwI/owe67FnN02z4hysWw1P5EAl9jG6Wezq2vcJzVuf9oNblbYaGB3G80nazxbaL8t2Du HpaOZCaCB6qxEkiSfqyETQlpMvs84U366AuC2waUjiQBZhfXpr18ifOErGAEhlgy0wA/yjpkWut RFaDxFPXPKLe1M+E6NRHLZSJVHlthR9H/cckfT+xcbvxuc+HotYINkK7xCJPQ2YwxDHC8PXCaDm RTE1jU9wwA3glw0ntMlc+Qt5yWZ35wrCM9sAZO0pnDVrEQxuNI+SsUHWnjFt8SEwcTRL2U4V8Zz j/676O9616jdlE= X-Received: by 2002:a05:6830:d19:b0:7d7:cea3:6d89 with SMTP id 46e09a7af769-7dbb6f248ecmr13998650a34.4.1775598888596; Tue, 07 Apr 2026 14:54:48 -0700 (PDT) Received: from mail.minyard.net ([2001:470:b8f6:1b:b3d9:38af:c282:fdde]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7dbfc1cb79esm1633373a34.15.2026.04.07.14.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Apr 2026 14:54:48 -0700 (PDT) Date: Tue, 7 Apr 2026 16:54:44 -0500 From: Corey Minyard To: Tony Camuso Cc: openipmi-developer@lists.sourceforge.net, linux-kernel@vger.kernel.org, minyard@acm.org Subject: Re: [PATCH 0/2] ipmi:watchdog: Fix panic, D-state hang, and lost protection on BMC reset Message-ID: Reply-To: corey@minyard.net References: <20260407175134.3367345-1-tcamuso@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260407175134.3367345-1-tcamuso@redhat.com> On Tue, Apr 07, 2026 at 01:51:32PM -0400, Tony Camuso wrote: > When the BMC resets while the IPMI watchdog is active, the driver has > three failure modes depending on timing: > > 1. list_add double add panic -- the watchdog daemon retries while the > static smi_msg/recv_msg structures are still queued in the IPMI > layer from the previous (unanswered) request. I'm trying to make sense of this. Are you sure this didn't start happening after you added a timeout on the wait_for_completion()? Otherwise it would never return, the mutex would be held, and no new message could be added. Just timing out in wait_for_completion() there could cause all kinds of bad things to happen. > > 2. D-state hang -- wait_for_completion() blocks indefinitely because > the BMC never delivers a response. This is an issue. The lower level driver is *always* supposed to return a failure. Something else needs to be fixed. I have seen several creative ways in which BMCs "fail to respond" that have confused the lower level drivers. If my guess is correct, there's a bug in the low level driver that's causing it to not time out the message. If we don't fix this, it will cause other issues outside the watchdog. > > 3. Silent loss of watchdog protection -- the BMC returns a non-zero > completion code, the driver's internal state becomes inconsistent, > writes to /dev/watchdog return -EINVAL, and the daemon gives up. > The system continues running without hardware watchdog coverage. Again, are you sure this didn't start happening after you added the timeout? > > All three stem from the same root cause: the static message structures > and unbounded completion waits were never designed for a BMC that > disappears mid-transaction. All that is supposed to be protected by a mutex. That mutex is claimed on all IPMI watchdog operations, and it shouldn't be released until all resources have been freed. Anything that violates that is asking for trouble. You don't mention the lower level interface (KCS, BT, SMIC, SSIF) but I think we need to start looking there. It may be that the timeouts on the watchdog messages need to be adjusted. The whole IPMI driver was designed on the presumption that the BMC would go away for only a short period of time (5-10 seconds) and not permanantly. That has slowly been fixed over time, but things might need to be adjusted in the watchdog. -corey > > This has been independently reported by Kenta Akagi on a Dell PowerEdge > R640 running 6.18.7, also triggered by a BMC reset with the watchdog > active: > > https://sourceforge.net/p/openipmi/mailman/message/59292850/ > > The fix takes a simple, deterministic approach: detect the failure via > BMC error response, guard against structure reuse (msg_in_flight) and > indefinite waits (completion timeout), then initiate orderly_reboot() > when the watchdog is active. This produces the same outcome the > hardware watchdog would have -- a system reset -- but through a > controlled path with clear logging and no panics or hangs. > > If the watchdog is stopped when the BMC resets, no reboot occurs and > the system continues normally. > > Tested on Dell PowerEdge R640 with kernel 5.14 (RHEL 9) and verified > against mainline (both patches apply cleanly). > > Corey Minyard's recent fix for list corruption in smi_work() > (ipmi_msghandler.c) addresses a related but separate code path. The > watchdog driver's own static structure reuse requires this fix. > > Tony Camuso (2): > ipmi:watchdog: Reboot cleanly on BMC reset > Documentation: ipmi: Update BMC reset behavior for watchdog > > Documentation/driver-api/ipmi.rst | 61 ++++++++++++++++++ > drivers/char/ipmi/ipmi_watchdog.c | 101 ++++++++++++++++++++++++------ > 2 files changed, 144 insertions(+), 18 deletions(-) > > -- > 2.53.0 >