From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, bpf@vger.kernel.org
Cc: arnd@arndb.de, catalin.marinas@arm.com, will@kernel.org,
peterz@infradead.org, akpm@linux-foundation.org,
mark.rutland@arm.com, harisokn@amazon.com, cl@gentwo.org,
ast@kernel.org, memxor@gmail.com, zhenglifeng1@huawei.com,
xueshuai@linux.alibaba.com, joao.m.martins@oracle.com,
boris.ostrovsky@oracle.com, konrad.wilk@oracle.com
Subject: [PATCH v2 4/7] arm64: barrier: add coarse wait for smp_cond_load_relaxed_timewait()
Date: Fri, 2 May 2025 01:52:20 -0700 [thread overview]
Message-ID: <20250502085223.1316925-5-ankur.a.arora@oracle.com> (raw)
In-Reply-To: <20250502085223.1316925-1-ankur.a.arora@oracle.com>
smp_cond_load_relaxed_timewait() waits on a conditional variable
until a timeout expires. This waiting is via some mix of looping
around, dereferencing an address, or waiting in a WFE until the CPU
gets an event due to a store to the address, or because of periodic
events from the event-stream.
Define __smp_cond_timewait_coarse() for usecases where the caller can
tolerate a relatively large overshoot. This allows us to minimize the
time spent spinning at the cost of spending extra time in the WFE
state.
This would result in a worst case delay of ARCH_TIMER_EVT_STREAM_PERIOD_US
and a spin period of no more than SMP_TIMEWAIT_CHECK_US.
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
arch/arm64/include/asm/barrier.h | 66 ++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index eaeb78dd48c0..f4a184a96933 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -10,6 +10,7 @@
#ifndef __ASSEMBLY__
#include <linux/kasan-checks.h>
+#include <linux/minmax.h>
#include <asm/alternative-macros.h>
@@ -219,6 +220,71 @@ do { \
#define __smp_timewait_store(ptr, val) \
__cmpwait_relaxed(ptr, val)
+/*
+ * Redefine ARCH_TIMER_EVT_STREAM_PERIOD_US locally to avoid include hell.
+ */
+#define __ARCH_TIMER_EVT_STREAM_PERIOD_US 100UL
+extern bool arch_timer_evtstrm_available(void);
+
+/*
+ * For coarse grained waits, allow overshoot by the event-stream period.
+ * Defined without reference to ARCH_TIMER_EVT_STREAM_PERIOD_US to avoid
+ * include hell.
+ */
+#define SMP_TIMEWAIT_SLACK_COARSE_US __ARCH_TIMER_EVT_STREAM_PERIOD_US
+
+#define SMP_TIMEWAIT_SPIN_BASE 16
+#define SMP_TIMEWAIT_CHECK_US 2UL
+
+static inline u64 ___cond_timewait(u64 now, u64 prev, u64 end,
+ u32 *spin, bool *wait, u64 slack)
+{
+ bool wfet = alternative_has_cap_unlikely(ARM64_HAS_WFXT);
+ bool wfe, ev = arch_timer_evtstrm_available();
+ u64 evt_period = __ARCH_TIMER_EVT_STREAM_PERIOD_US;
+ u64 remaining = end - now;
+
+ if (now >= end)
+ return 0;
+
+ /*
+ * Use WFE if there's enough slack to get an event-stream wakeup even
+ * if we don't come out of the WFE due to natural causes.
+ */
+ wfe = ev && ((remaining + slack) > evt_period);
+
+ if (wfe || wfet) {
+ *wait = true;
+ *spin = 0;
+ return now;
+ }
+
+ /*
+ * Our wait period is shorter than our best granularity. Spin.
+ *
+ * A time-check is expensive but not too expensive. Scale the
+ * spin-count so we stay close to the fine-grained slack period.
+ */
+ *wait = false;
+ if ((now - prev) < SMP_TIMEWAIT_CHECK_US)
+ *spin <<= 1;
+ else
+ *spin = max((*spin >> 1) + (*spin >> 2), SMP_TIMEWAIT_SPIN_BASE);
+ return now;
+}
+
+/*
+ * Coarse wait_policy: minimizes the duration spent spinning at the cost of
+ * potentially spending the available slack in a WFE wait state.
+ *
+ * The resultant worst case timeout delay is SMP_TIMEWAIT_SLACK_COARSE_US
+ * (same as ARCH_TIMER_EVT_STREAM_PERIOD_US) and a spin period of no more
+ * than SMP_TIMEWAIT_CHECK_US.
+ */
+#define __smp_cond_timewait_coarse(now, prev, end, spin, wait) \
+ ___cond_timewait(now, prev, end, spin, wait, \
+ SMP_TIMEWAIT_SLACK_COARSE_US)
+
#include <asm-generic/barrier.h>
#endif /* __ASSEMBLY__ */
--
2.43.5
next prev parent reply other threads:[~2025-05-02 8:52 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-02 8:52 [PATCH v2 0/7] barrier: introduce smp_cond_load_*_timewait() Ankur Arora
2025-05-02 8:52 ` [PATCH v2 1/7] asm-generic: barrier: add smp_cond_load_relaxed_timewait() Ankur Arora
2025-05-21 18:37 ` Catalin Marinas
2025-05-24 3:22 ` Ankur Arora
2025-05-02 8:52 ` [PATCH v2 2/7] asm-generic: barrier: add wait_policy handlers Ankur Arora
2025-05-02 8:52 ` [PATCH v2 3/7] arm64: barrier: enable waiting in smp_cond_load_relaxed_timewait() Ankur Arora
2025-05-02 8:52 ` Ankur Arora [this message]
2025-05-02 8:52 ` [PATCH v2 5/7] arm64: barrier: add fine wait for smp_cond_load_relaxed_timewait() Ankur Arora
2025-05-02 8:52 ` [PATCH v2 6/7] asm-generic: barrier: add smp_cond_load_acquire_timewait() Ankur Arora
2025-05-02 8:52 ` [PATCH v2 7/7] bpf: rqspinlock: add rqspinlock policy handler for arm64 Ankur Arora
2025-05-02 16:42 ` [PATCH v2 0/7] barrier: introduce smp_cond_load_*_timewait() Christoph Lameter (Ampere)
2025-05-02 20:05 ` Ankur Arora
2025-05-05 16:13 ` Christoph Lameter (Ampere)
2025-05-05 17:08 ` Ankur Arora
2025-05-16 22:50 ` Okanovic, Haris
2025-05-17 1:16 ` Ankur Arora
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250502085223.1316925-5-ankur.a.arora@oracle.com \
--to=ankur.a.arora@oracle.com \
--cc=akpm@linux-foundation.org \
--cc=arnd@arndb.de \
--cc=ast@kernel.org \
--cc=boris.ostrovsky@oracle.com \
--cc=bpf@vger.kernel.org \
--cc=catalin.marinas@arm.com \
--cc=cl@gentwo.org \
--cc=harisokn@amazon.com \
--cc=joao.m.martins@oracle.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=memxor@gmail.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
--cc=xueshuai@linux.alibaba.com \
--cc=zhenglifeng1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®