From: Waiman Long <longman@redhat.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
"Rafael J. Wysocki" <rafael@kernel.org>
Cc: linux-kernel@vger.kernel.org,
Vishal Agrawal <vagrawal@redhat.com>,
Joe Mario <jmario@redhat.com>, Waiman Long <longman@redhat.com>
Subject: [PATCH] driver/base/cpu: Retry online operation if -EBUSY
Date: Mon, 24 Jul 2023 10:38:26 -0400 [thread overview]
Message-ID: <20230724143826.3996163-1-longman@redhat.com> (raw)
Booting the kernel with "maxcpus=1" is a common technique for CPU
partitioning and isolation. It delays the CPU bringup process until
when the bootup scripts are ready to bring CPUs online by writing 1 to
/sys/device/system/cpu/cpu<X>/online. However, it was found that not
all the CPUs were online after bootup. The collection of offline CPUs
are different after every reboot.
Further investigation reveals that some "online" write operations
fail with an -EBUSY error. This error is returned when CPU hotplug is
temporiarly disabled when cpu_hotplug_disable() is called.
During bootup, the main caller of cpu_hotplug_disable() is
pci_call_probe() for PCI device initialization. By measuring the times
spent with cpu_hotplug_disabled set in a typical 2-socket server, most
of them last less than 10ms. However, there are a few that can last
hundreds of ms. Note that the cpu_hotplug_disabled period of different
devices can overlap leading to longer cpu_hotplug_disabled hold time.
Since the CPU hotplug disable condition is transient and it is not
that easy to modify all the existing bootup scripts to handle this
condition, the kernel can help by retrying the online operation when
an -EBUSY error is returned. This patch retries the online operation
in cpu_subsys_online() when an -EBUSY error is returned for up to 5
times after an exponentially increasing delay that can last a total of
at least 620ms of waiting time by calling msleep().
With this patch in place, booting up the patched kernel with "maxcpus=1"
does not leave any CPU in an offline state in 10 reboot attempts.
Reported-by: Vishal Agrawal <vagrawal@redhat.com>
Signed-off-by: Waiman Long <longman@redhat.com>
---
drivers/base/cpu.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index c1815b9dae68..4b828f54f9f4 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -19,6 +19,7 @@
#include <linux/cpufeature.h>
#include <linux/tick.h>
#include <linux/pm_qos.h>
+#include <linux/delay.h>
#include <linux/sched/isolation.h>
#include "base.h"
@@ -50,12 +51,30 @@ static int cpu_subsys_online(struct device *dev)
int cpuid = dev->id;
int from_nid, to_nid;
int ret;
+ int retries = 0;
from_nid = cpu_to_node(cpuid);
if (from_nid == NUMA_NO_NODE)
return -ENODEV;
+retry:
ret = cpu_device_up(dev);
+
+ /*
+ * If -EBUSY is returned, it is likely that hotplug is temporarily
+ * disabled when cpu_hotplug_disable() was called. This condition is
+ * transient. So we retry after waiting for an exponentially
+ * increasing delay up to a total of at least 620ms as some PCI
+ * device initialization can take quite a while.
+ */
+ if (ret == -EBUSY) {
+ retries++;
+ if (retries > 5)
+ return ret;
+ msleep(10 * (1 << retries));
+ goto retry;
+ }
+
/*
* When hot adding memory to memoryless node and enabling a cpu
* on the node, node number of the cpu may internally change.
--
2.31.1
reply other threads:[~2023-07-24 14:40 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230724143826.3996163-1-longman@redhat.com \
--to=longman@redhat.com \
--cc=gregkh@linuxfoundation.org \
--cc=jmario@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=rafael@kernel.org \
--cc=vagrawal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®