mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Corey Minyard <corey@minyard.net>
To: Shakeel Butt <shakeel.butt@linux.dev>
Cc: openipmi-developer@lists.sourceforge.net,
	Meta kernel team <kernel-team@meta.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] ipmi:msghandler: Free queued requests of a destroyed user
Date: Sat, 21 Nov 2026 19:59:01 -0600	[thread overview]
Message-ID: <awJMZfksZDfGO7k4@mail.minyard.net> (raw)
In-Reply-To: <20261121031549.3599823-1-shakeel.butt@linux.dev>

On Fri, Nov 20, 2026 at 07:15:49PM -0800, Shakeel Butt wrote:
> In the Meta fleet we found hosts whose BMC had stopped answering IPMI
> requests while a power management daemon kept polling it:
> 
> - the daemon runs "timeout 2 ipmitool raw 0x30 0x7e" every two seconds;
> - every run leaves one request and one closed IPMI user behind;
> - one host held over a million closed users after 25 days, using about
>   33 GiB and growing by 1.3 GiB a day.
> 
> ipmitool opens /dev/ipmi0, sends a request and waits for the answer.
> When timeout kills it, the file is closed and ipmi_destroy_user() runs.
> A request that is still waiting on intf->xmit_msgs or intf->hp_xmit_msgs
> stays there, and its receive message keeps a reference on the user.
> With a BMC that does not answer, the lower layer is slow to fail each
> request: about 15 seconds of retries for SSIF, and up to a minute for
> KCS on kernels without the hosed BMC handling from
> commit bc3a9d217755 ("ipmi:si: Gracefully handle if the BMC is non-functional").

Oops, that's not good.  This is in my next queue for the next release
and I added a stable backport.

Thanks,

-corey

> Requests arrive faster than they leave, so the queue, the users and
> their messages grow until the BMC answers again.
> 
> Nobody can receive these responses once the user is destroyed. Free the
> user's requests that were not handed to the lower layer yet. Such a
> request is then never sent to the BMC. This only matters for a program
> that sends a request and closes the device without waiting for the
> answer, and with a working BMC a request waits in these lists only
> while another request is being handled.
> 
> The request the lower layer is working on (intf->curr_msg) is left
> alone. IPMB and LAN commands do not point to their receive message, so
> they are not touched here. Their receive messages are freed by the
> sequence table loop just above, as before.
> 
> Tested in QEMU with a fake BMC that stops answering and four loops of
> "timeout 2 ipmitool raw 0x30 0x7e", with KASAN and lockdep enabled:
> 
> - SSIF on linux-next: 465 closed users were held after 4 minutes,
>   growing by about 116 a minute. With this patch, 5 are held (the four
>   live callers and the request in flight), and the number stays flat.
> - KCS on v6.16, which lacks the hosed BMC handling, with this change
>   adapted: 476 held after 4 minutes without it, 5 with it.
> - KCS on linux-next: at most 5 held instead of up to 28.
> - With a working BMC, 1600 requests from 8 parallel ipmitool loops all
>   succeed over KCS and SSIF, and nothing is left held afterwards.
> 
> QEMU needed small changes to behave like the failing BMC: ipmi-bmc-extern
> makes up a timeout reply after 4 seconds, its KCS keeps IBF set while
> the BMC has not answered, and smbus-ipmi returns the previous reply
> instead of NAKing the read.
> 
> Fixes: 7ea0ed2b5be8 ("ipmi: Make the message handler easier to use for SMI interfaces")
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> ---
>  drivers/char/ipmi/ipmi_msghandler.c | 41 +++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
> 
> diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
> index 152dd6b5bb27..52e95e1949b9 100644
> --- a/drivers/char/ipmi/ipmi_msghandler.c
> +++ b/drivers/char/ipmi/ipmi_msghandler.c
> @@ -1339,6 +1339,45 @@ int ipmi_get_smi_info(int if_num, struct ipmi_smi_info *data)
>  }
>  EXPORT_SYMBOL(ipmi_get_smi_info);
>  
> +static void move_user_xmit_msgs(struct list_head *from, struct list_head *to,
> +				struct ipmi_user *user)
> +{
> +	struct ipmi_smi_msg *msg, *msg2;
> +
> +	list_for_each_entry_safe(msg, msg2, from, link) {
> +		if (msg->recv_msg && msg->recv_msg->user == user)
> +			list_move_tail(&msg->link, to);
> +	}
> +}
> +
> +/*
> + * Free the requests of a user that are still waiting to be handed to the
> + * lower layer.  Once the user is destroyed, nobody can receive their
> + * responses.  If the BMC is slow or does not answer, they could otherwise
> + * stay queued for a long time, each one keeping the user and its own
> + * messages allocated.  A request that was already handed to the lower
> + * layer (intf->curr_msg) is not on these lists and is left alone.
> + */
> +static void remove_user_xmit_msgs(struct ipmi_smi *intf,
> +				  struct ipmi_user *user)
> +{
> +	int run_to_completion = READ_ONCE(intf->run_to_completion);
> +	struct ipmi_smi_msg *msg, *msg2;
> +	unsigned long flags = 0;
> +	LIST_HEAD(msgs);
> +
> +	ipmi_lock_xmit_msgs(intf, run_to_completion, &flags);
> +	move_user_xmit_msgs(&intf->hp_xmit_msgs, &msgs, user);
> +	move_user_xmit_msgs(&intf->xmit_msgs, &msgs, user);
> +	ipmi_unlock_xmit_msgs(intf, run_to_completion, &flags);
> +
> +	list_for_each_entry_safe(msg, msg2, &msgs, link) {
> +		list_del(&msg->link);
> +		ipmi_free_recv_msg(msg->recv_msg);
> +		ipmi_free_smi_msg(msg);
> +	}
> +}
> +
>  /* Must be called with intf->users_mutex held. */
>  static void _ipmi_destroy_user(struct ipmi_user *user)
>  {
> @@ -1375,6 +1414,8 @@ static void _ipmi_destroy_user(struct ipmi_user *user)
>  	}
>  	mutex_unlock(&intf->seq_lock);
>  
> +	remove_user_xmit_msgs(intf, user);
> +
>  	/*
>  	 * Remove the user from the command receiver's table.  First
>  	 * we build a list of everything (not using the standard link,
> -- 
> 2.53.1-Meta
> 

      reply	other threads:[~2026-11-22  1:59 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-21  3:15 Shakeel Butt
2026-11-22  1:59 ` Corey Minyard [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=awJMZfksZDfGO7k4@mail.minyard.net \
    --to=corey@minyard.net \
    --cc=kernel-team@meta.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=openipmi-developer@lists.sourceforge.net \
    --cc=shakeel.butt@linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®