From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-pm@vger.kernel.org,
bpf@vger.kernel.org
Cc: arnd@arndb.de, catalin.marinas@arm.com, will@kernel.org,
peterz@infradead.org, akpm@linux-foundation.org,
mark.rutland@arm.com, harisokn@amazon.com, cl@gentwo.org,
ast@kernel.org, rafael@kernel.org, daniel.lezcano@linaro.org,
memxor@gmail.com, zhenglifeng1@huawei.com,
xueshuai@linux.alibaba.com, joao.m.martins@oracle.com,
boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH v8 10/12] bpf/rqspinlock: Use smp_cond_load_acquire_timeout()
Date: Sun, 14 Dec 2025 20:49:17 -0800 [thread overview]
Message-ID: <20251215044919.460086-11-ankur.a.arora@oracle.com> (raw)
In-Reply-To: <20251215044919.460086-1-ankur.a.arora@oracle.com>
Switch out the conditional load interfaces used by rqspinlock
to smp_cond_read_acquire_timeout() and its wrapper,
atomic_cond_read_acquire_timeout().
Both these handle the timeout and amortize as needed, so use
clock_deadlock() directly instead of going through RES_CHECK_TIMEOUT().
For correctness, however, we need to ensure that the timeout case in
smp_cond_read_acquire_timeout() always agrees with that in
clock_deadlock(), which returns with -ETIMEDOUT.
For the most part, this is fine because smp_cond_load_acquire_timeout()
does not have an independent clock and does not do double reads from
clock_deadlock() which could cause its internal state to go out of
sync from that of clock_deadlock().
There is, however, an edge case where clock_deadlock() checks for:
if (time > ts->timeout_end)
return -ETIMEDOUT;
while smp_cond_load_acquire_timeout() checks for:
__time_now = (time_expr_ns);
if (__time_now <= 0 || __time_now >= __time_end) {
VAL = READ_ONCE(*__PTR);
break;
}
This runs into a problem when (__time_now == __time_end) since
clock_deadlock() does not treat it as a timeout condition but
the second clause in the conditional above does.
So, add an equality check in clock_deadlock().
Finally, redefine SMP_TIMEOUT_POLL_COUNT to be 16k to be similar to the
spin-count used in RES_CHECK_TIMEOUT(). We only do this for non-arm64
as that uses a waiting implementation.
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: bpf@vger.kernel.org
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
- change the check in clock_deadlock()
kernel/bpf/rqspinlock.c | 37 ++++++++++++++++++++-----------------
1 file changed, 20 insertions(+), 17 deletions(-)
diff --git a/kernel/bpf/rqspinlock.c b/kernel/bpf/rqspinlock.c
index ac9b3572e42f..2a361c4c7393 100644
--- a/kernel/bpf/rqspinlock.c
+++ b/kernel/bpf/rqspinlock.c
@@ -215,7 +215,7 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
}
time = ktime_get_mono_fast_ns();
- if (time > ts->timeout_end)
+ if (time >= ts->timeout_end)
return -ETIMEDOUT;
/*
@@ -235,20 +235,14 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
}
/*
- * Do not amortize with spins when res_smp_cond_load_acquire is defined,
- * as the macro does internal amortization for us.
+ * Amortize timeout check for busy-wait loops.
*/
-#ifndef res_smp_cond_load_acquire
#define RES_CHECK_TIMEOUT(ts, ret, mask) \
({ \
if (!(ts).spin++) \
(ret) = clock_deadlock((lock), (mask), &(ts));\
(ret); \
})
-#else
-#define RES_CHECK_TIMEOUT(ts, ret, mask) \
- ({ (ret) = clock_deadlock((lock), (mask), &(ts)); })
-#endif
/*
* Initialize the 'spin' member.
@@ -262,6 +256,18 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
*/
#define RES_RESET_TIMEOUT(ts, _duration) ({ (ts).timeout_end = 0; (ts).duration = _duration; })
+/*
+ * Limit how often we invoke clock_deadlock() while spin-waiting in
+ * smp_cond_load_acquire_timeout() or atomic_cond_read_acquire_timeout().
+ *
+ * (ARM64 generally uses a waited implementation so we use the default
+ * value there.)
+ */
+#ifndef CONFIG_ARM64
+#undef SMP_TIMEOUT_POLL_COUNT
+#define SMP_TIMEOUT_POLL_COUNT (16*1024)
+#endif
+
/*
* Provide a test-and-set fallback for cases when queued spin lock support is
* absent from the architecture.
@@ -312,12 +318,6 @@ EXPORT_SYMBOL_GPL(resilient_tas_spin_lock);
*/
static DEFINE_PER_CPU_ALIGNED(struct qnode, rqnodes[_Q_MAX_NODES]);
-#ifndef res_smp_cond_load_acquire
-#define res_smp_cond_load_acquire(v, c) smp_cond_load_acquire(v, c)
-#endif
-
-#define res_atomic_cond_read_acquire(v, c) res_smp_cond_load_acquire(&(v)->counter, (c))
-
/**
* resilient_queued_spin_lock_slowpath - acquire the queued spinlock
* @lock: Pointer to queued spinlock structure
@@ -415,7 +415,9 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
*/
if (val & _Q_LOCKED_MASK) {
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT);
- res_smp_cond_load_acquire(&lock->locked, !VAL || RES_CHECK_TIMEOUT(ts, timeout_err, _Q_LOCKED_MASK) < 0);
+ smp_cond_load_acquire_timeout(&lock->locked, !VAL,
+ (timeout_err = clock_deadlock(lock, _Q_LOCKED_MASK, &ts)),
+ ts.duration);
}
if (timeout_err < 0) {
@@ -577,8 +579,9 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
* us.
*/
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT * 2);
- val = res_atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK) ||
- RES_CHECK_TIMEOUT(ts, timeout_err, _Q_LOCKED_PENDING_MASK) < 0);
+ val = atomic_cond_read_acquire_timeout(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK),
+ (timeout_err = clock_deadlock(lock, _Q_LOCKED_PENDING_MASK, &ts)),
+ ts.duration);
/* Disable queue destruction when we detect deadlocks. */
if (timeout_err == -EDEADLK) {
--
2.31.1
next prev parent reply other threads:[~2025-12-15 4:51 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-15 4:49 [PATCH v8 00/12] barrier: Add smp_cond_load_{relaxed,acquire}_timeout() Ankur Arora
2025-12-15 4:49 ` [PATCH v8 01/12] asm-generic: barrier: Add smp_cond_load_relaxed_timeout() Ankur Arora
2025-12-22 7:47 ` Ankur Arora
2025-12-15 4:49 ` [PATCH v8 02/12] arm64: barrier: Support smp_cond_load_relaxed_timeout() Ankur Arora
2026-01-08 21:36 ` Will Deacon
2026-01-09 9:06 ` Ankur Arora
2025-12-15 4:49 ` [PATCH v8 03/12] arm64/delay: move some constants out to a separate header Ankur Arora
2025-12-15 17:59 ` kernel test robot
2025-12-19 4:52 ` kernel test robot
2025-12-15 4:49 ` [PATCH v8 04/12] arm64: support WFET in smp_cond_relaxed_timeout() Ankur Arora
2026-01-08 21:35 ` Will Deacon
2026-01-09 9:05 ` Ankur Arora
2026-01-09 14:16 ` Will Deacon
2026-01-09 18:55 ` Ankur Arora
2026-01-09 14:17 ` Will Deacon
2026-01-09 19:05 ` Ankur Arora
2026-01-20 13:58 ` Will Deacon
2026-01-20 22:49 ` Ankur Arora
2025-12-15 4:49 ` [PATCH v8 05/12] arm64: rqspinlock: Remove private copy of smp_cond_load_acquire_timewait() Ankur Arora
2025-12-15 5:14 ` bot+bpf-ci
2025-12-15 4:49 ` [PATCH v8 06/12] asm-generic: barrier: Add smp_cond_load_acquire_timeout() Ankur Arora
2025-12-15 4:49 ` [PATCH v8 07/12] atomic: Add atomic_cond_read_*_timeout() Ankur Arora
2026-01-09 14:11 ` Will Deacon
2026-01-09 19:06 ` Ankur Arora
2025-12-15 4:49 ` [PATCH v8 08/12] locking/atomic: scripts: build atomic_long_cond_read_*_timeout() Ankur Arora
2025-12-15 4:49 ` [PATCH v8 09/12] bpf/rqspinlock: switch check_timeout() to a clock interface Ankur Arora
2025-12-15 4:49 ` Ankur Arora [this message]
2025-12-15 21:40 ` [PATCH v8 10/12] bpf/rqspinlock: Use smp_cond_load_acquire_timeout() Alexei Starovoitov
2025-12-16 7:34 ` Ankur Arora
2025-12-15 4:49 ` [PATCH v8 11/12] sched: add need-resched timed wait interface Ankur Arora
2025-12-15 4:49 ` [PATCH v8 12/12] cpuidle/poll_state: Wait for need-resched via tif_need_resched_relaxed_wait() Ankur Arora
2025-12-15 11:11 ` Rafael J. Wysocki
2025-12-16 6:55 ` Ankur Arora
2025-12-15 21:23 ` kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251215044919.460086-11-ankur.a.arora@oracle.com \
--to=ankur.a.arora@oracle.com \
--cc=akpm@linux-foundation.org \
--cc=arnd@arndb.de \
--cc=ast@kernel.org \
--cc=boris.ostrovsky@oracle.com \
--cc=bpf@vger.kernel.org \
--cc=catalin.marinas@arm.com \
--cc=cl@gentwo.org \
--cc=daniel.lezcano@linaro.org \
--cc=harisokn@amazon.com \
--cc=joao.m.martins@oracle.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=memxor@gmail.com \
--cc=peterz@infradead.org \
--cc=rafael@kernel.org \
--cc=will@kernel.org \
--cc=xueshuai@linux.alibaba.com \
--cc=zhenglifeng1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®