* [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters
@ 2026-11-19 22:49 Peng Huang
2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
Waiman Long, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang
Hi,
High-priority interactive threads (such as UI/render threads) can stall
for tens to hundreds of milliseconds on shared rw_semaphores like
mm->mmap_lock, inode->i_rwsem, or mapping->i_mmap_rwsem/invalidate_lock
when lower-priority (or SCHED_IDLE) read-owners are preempted on busy
CPUs and block an incoming higher-weight writer. Upstream
CONFIG_SCHED_PROXY_EXEC currently covers mutexes, and out-of-tree rwsem
proxy efforts only track a single write-owner, leaving multi-reader
inversions unaddressed.
This RFC proposes a lightweight priority-inheritance mechanism for CFS
rw_semaphore holders that accelerates stalled readers and writers in
parallel without requiring lock-side reader tracking.
Design rationale
----------------
1. Per-task hold tracking (avoiding lock-side reader lists):
Because rw_semaphore allows unbounded concurrent readers, storing a
reader list inside struct rw_semaphore would require dynamic
allocation or extra locking in the down_read()/up_read() fastpath.
Instead, each task records up to CONFIG_RWSEM_SCHED_BOOST_SLOTS
(default 4) rwsem pointers in task_struct under current-only access.
This supports an unlimited number of concurrent readers per rwsem;
the slot count only caps simultaneous nested rwsems held by one task.
2. Contended waiter demand (sem->max_waiter_prio):
Waiters in rwsem_down_{read,write}_slowpath() publish the lowest
normal prio (i.e. highest nice weight, capped at MAX_RT_PRIO so CFS
holders never escalate into RT/DL) into sem->max_waiter_prio,
recalculating without redundant atomic writes as waiters wake/leave.
3. Per-rq runqueue list and parallel boost:
Runnable CFS tasks holding at least one rwsem are linked onto a
per-CPU list (rq->held_rwsem_tasks) during enqueue/dequeue/migrate
under rq_lock, keeping lock/unlock fastpaths list-free. Normal short
holds never touch rq->held_rwsem_tasks; tasks are only linked if
preempted, enqueued, or crossing a tick while holding a lock, so tick
and idle-pull typically exit immediately on an empty list with zero
iteration or max_waiter_prio lookups. Across wakeup, preemption, tick,
and idle-balance pull paths, the scheduler checks held locks'
sem->max_waiter_prio and elevates the holder's effective priority and
EEVDF weight (preserving static_prio/nice and handling SCHED_IDLE
separately) down to min(p->prio, max_waiter_prio), restoring cleanly
on unlock. Runtime/build disablement is provided via
rwsem_sched_boost=0, CONFIG_RWSEM_SCHED_BOOST_SLOTS=0, or PREEMPT_RT.
4. Model trade-off (Parallel priority spray vs. Proxy Execution):
Proxy Execution follows task->blocked_on to boost one owner at a
time on the owner's CPU (entailing cross-CPU proxy migrations and
requiring a lock-side owner list for readers), while naturally
propagating transitively across mutex/futex/rwsem chains. By
contrast, this RFC avoids lock-side reader tracking and lets all
preempted readers on any CPU boost and drain in parallel once
contended, at the intentional cost of not propagating transitively
through mutex/futex chains (e.g. RT -> mutex -> rwsem).
Validation & known trade-offs
-----------------------------
- Included KUnit suite (Patch 3/3, CONFIG_RWSEM_SCHED_BOOST_KUNIT_TEST):
Reproducible under ARCH=um via kunit.py (10/10 pass incl. SCHED_IDLE
cap, interleaved unlock, and slot overflow):
* Preemption inversion (n=3, 4 nice=0 hogs):
Control mean 1939 ms (max 1970 ms) -> Treatment mean 49 ms (-97.5%)
* Uncontended fastpath (1,000,000 ops):
read +2.75 ns/op (21.43 -> 24.18),
write +1.99 ns/op (20.78 -> 22.77),
nested2 read +4.31 ns/op (21.20 -> 25.51)
- ARM64 mobile hardware (6.18 backport, asymmetric SMP, user-space
madvise/mprotect stress reproducer):
* mmap_lock inversion (nice -10 madvise vs 7 nice 0 mprotect hogs):
- Unpinned 5-round AB (100 runs): mean 120.22 -> 7.26 ms (-94.0%),
missed 16ms deadlines 46/100 -> 8/100
- Mid-cluster pinned 4-round AB/BA (80 runs): mean 281.11 ->
7.34 ms (-97.4%), max 736.14 -> 50.76 ms, missed 16ms 80/80 -> 4/80
* Big-core pinned syscall fastpath (big core, 4 rounds x 100k iters):
madvise 615.8 -> 604.8 ns/op (-1.8%),
mprotect 857.5 -> 848.5 ns/op (-1.0%)
* Contended equal-priority non-inversion (4 threads, Mid pinned):
145.5 -> 162.5 ms (+11.7% throughput trade-off under contention;
also +10.8% in x86_64 QEMU -smp 4 TCG)
* Pinned hackbench / pipe ping-pong (Mid+Big):
hackbench 263.7 -> 243.3 ms (-7.7%),
pipe 10.81 -> 10.33 us/op (-4.4%)
Request for feedback
--------------------
Before iterating further, feedback is especially welcome on:
- Parallel priority spray (task-slot tracking) vs. extending Proxy
Execution with lock-side reader tracking, vs. a slowpath/owner-only
heuristic;
- Using normal prio / nice weight as the demand currency vs. EEVDF
virtual deadline / lag considerations;
- Struct size impact of adding max_waiter_prio to struct rw_semaphore
(+8 bytes aligned on 64-bit, 32 -> 40 bytes, embedded in inode/mm);
- Intentional RFC boundaries: 4-slot nested limit per task, boosting
all concurrent readers equally, idle-pull inspecting only the
busiest rq (no global CPU scan/cpumask), no cgroup/task_group share
propagation, and no transitive inheritance across mutex/futex chains.
Per Documentation/process/generated-content.rst: Jetski AI coding
assistance was used for drafting, refactoring discussions, and KUnit /
benchmark verification; all patches and tests have been reviewed and
verified by the author.
Peng Huang (3):
locking/rwsem, sched: track held rwsems and waiter priorities
sched: boost rwsem holders with contended high-priority waiters
locking/rwsem: add KUnit tests for scheduler priority boost
.../admin-guide/kernel-parameters.txt | 5 +
include/linux/rwsem.h | 28 +
include/linux/sched.h | 16 +
init/init_task.c | 4 +
kernel/Kconfig.locks | 15 +
kernel/locking/Makefile | 1 +
kernel/locking/rwsem.c | 209 +++++++-
kernel/locking/rwsem_sched_boost_test.c | 504 ++++++++++++++++++
kernel/sched/core.c | 211 +++++++-
kernel/sched/fair.c | 51 ++
kernel/sched/sched.h | 70 +++
kernel/sched/syscalls.c | 2 +-
lib/Kconfig.debug | 10 +
13 files changed, 1101 insertions(+), 25 deletions(-)
create mode 100644 kernel/locking/rwsem_sched_boost_test.c
--
2.56.1.750.gf70dc682cf-goog
^ permalink raw reply [flat|nested] 4+ messages in thread
* [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities
2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang
2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
Waiman Long, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang
Track per-task held rw_semaphore pointers in a small fixed-size slot
array (CONFIG_RWSEM_SCHED_BOOST_SLOTS) and track the highest-priority
contending waiter on each rw_semaphore (max_waiter_prio, capped at
MAX_RT_PRIO).
Blocking acquisitions skip irqs/atomic guards to avoid pushf on the
fastpath, while trylock paths keep IRQ/atomic context filtering for
non-owner unlockers such as BPF helpers. Held slots are cleared under
preempt_disable() before releasing sem->count so another CPU cannot
acquire and free the semaphore while a tick/preempt check still sees
it in held_rwsems[].
Also wire up core priority calculation, runtime toggle
(rwsem_sched_boost=), and post-unlock priority restoration.
Signed-off-by: Peng Huang <penghuang@google.com>
---
.../admin-guide/kernel-parameters.txt | 5 +
include/linux/rwsem.h | 28 +++
include/linux/sched.h | 16 ++
init/init_task.c | 4 +
kernel/Kconfig.locks | 15 ++
kernel/locking/rwsem.c | 211 ++++++++++++++++--
kernel/sched/core.c | 102 ++++++++-
kernel/sched/sched.h | 43 ++++
kernel/sched/syscalls.c | 2 +-
9 files changed, 402 insertions(+), 24 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec..2c5207dd9c53 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6778,6 +6778,11 @@ Kernel parameters
rw [KNL] Mount root device read-write on boot
+ rwsem_sched_boost=
+ [KNL] Enable or disable rwsem scheduler priority
+ boosting when CONFIG_RWSEM_SCHED_BOOST=y.
+ Defaults to 1 (enabled). Format: <bool>
+
S [KNL] Run init in single mode
s390_iommu= [HW,S390]
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 6a1a7bae5f81..b9188de31881 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -16,6 +16,8 @@
#include <linux/atomic.h>
#include <linux/err.h>
#include <linux/cleanup.h>
+#include <linux/jump_label.h>
+#include <linux/sched/prio.h>
#ifdef CONFIG_DEBUG_LOCK_ALLOC
# define __RWSEM_DEP_MAP_INIT(lockname) \
@@ -58,6 +60,9 @@ context_lock_struct(rw_semaphore) {
#endif
raw_spinlock_t wait_lock;
struct rwsem_waiter *first_waiter __guarded_by(&wait_lock);
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+ atomic_t max_waiter_prio;
+#endif
#ifdef CONFIG_DEBUG_RWSEMS
void *magic;
#endif
@@ -101,12 +106,19 @@ static inline void rwsem_assert_held_write_nolockdep(const struct rw_semaphore *
#define __RWSEM_OPT_INIT(lockname)
#endif
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+# define __RWSEM_SCHED_BOOST_INIT .max_waiter_prio = ATOMIC_INIT(MAX_PRIO),
+#else
+# define __RWSEM_SCHED_BOOST_INIT
+#endif
+
#define __RWSEM_INITIALIZER(name) \
{ __RWSEM_COUNT_INIT(name), \
.owner = ATOMIC_LONG_INIT(0), \
__RWSEM_OPT_INIT(name) \
.wait_lock = __RAW_SPIN_LOCK_UNLOCKED(name.wait_lock),\
.first_waiter = NULL, \
+ __RWSEM_SCHED_BOOST_INIT \
__RWSEM_DEBUG_INIT(name) \
__RWSEM_DEP_MAP_INIT(name) }
@@ -327,8 +339,24 @@ extern void up_read_non_owner(struct rw_semaphore *sem) __releases_shared(sem);
# define down_write_nest_lock(sem, nest_lock) down_write(sem)
# define down_write_nested(sem, subclass) down_write(sem)
# define down_write_killable_nested(sem, subclass) down_write_killable(sem)
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+void down_read_non_owner(struct rw_semaphore *sem) __acquires_shared(sem);
+void up_read_non_owner(struct rw_semaphore *sem) __releases_shared(sem);
+#else
# define down_read_non_owner(sem) down_read(sem)
# define up_read_non_owner(sem) up_read(sem)
#endif
+#endif
+
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+extern struct static_key_true rwsem_sched_boost_key;
+static inline void rwsem_sched_boost_set(bool enable)
+{
+ if (enable)
+ static_branch_enable(&rwsem_sched_boost_key);
+ else
+ static_branch_disable(&rwsem_sched_boost_key);
+}
+#endif
#endif /* _LINUX_RWSEM_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..9ce2c11aa500 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -78,6 +78,7 @@ struct rcu_node;
struct reclaim_state;
struct root_domain;
struct rq;
+struct rw_semaphore;
struct sched_attr;
struct sched_dl_entity;
struct seq_file;
@@ -1266,6 +1267,13 @@ struct task_struct {
struct rt_mutex_waiter *pi_blocked_on;
#endif
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+ struct rw_semaphore *held_rwsems[CONFIG_RWSEM_SCHED_BOOST_SLOTS];
+ u8 nr_held_rwsems;
+ int rwsem_boost_prio;
+ struct list_head rwsem_rq_node;
+#endif
+
struct mutex *blocked_on; /* lock we're blocked on */
raw_spinlock_t blocked_lock;
@@ -2250,6 +2258,14 @@ static inline void clear_task_blocked_on(struct task_struct *p, struct rt_mutex
}
#endif /* !CONFIG_PREEMPT_RT */
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+void sched_rwsem_recalc_boost(struct task_struct *p);
+#else
+static inline void sched_rwsem_recalc_boost(struct task_struct *p)
+{
+}
+#endif
+
static __always_inline bool need_resched(void)
{
return unlikely(tif_need_resched());
diff --git a/init/init_task.c b/init/init_task.c
index ce7c2b07d855..a52a08f11ffd 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -134,6 +134,10 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
.time_slice = RR_TIMESLICE,
},
.tasks = LIST_HEAD_INIT(init_task.tasks),
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+ .rwsem_boost_prio = MAX_PRIO,
+ .rwsem_rq_node = LIST_HEAD_INIT(init_task.rwsem_rq_node),
+#endif
#ifdef CONFIG_SMP
.pushable_tasks = PLIST_NODE_INIT(init_task.pushable_tasks, MAX_PRIO),
#endif
diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks
index 1c6423aafcd4..7487cdab4e59 100644
--- a/kernel/Kconfig.locks
+++ b/kernel/Kconfig.locks
@@ -279,3 +279,18 @@ config ARCH_HAS_MMIOWB
config MMIOWB
def_bool y if ARCH_HAS_MMIOWB
depends on SMP
+
+config RWSEM_SCHED_BOOST_SLOTS
+ int "Maximum held rw_semaphores tracked per task for priority boost"
+ range 0 16
+ default 4
+ depends on !PREEMPT_RT
+ help
+ Number of rw_semaphore pointers recorded per task_struct so the
+ scheduler can detect and boost holders blocking higher-priority
+ waiters across wakeup, preemption, tick, and idle-balance hooks.
+ Set to 0 to disable.
+
+config RWSEM_SCHED_BOOST
+ def_bool y
+ depends on RWSEM_SCHED_BOOST_SLOTS > 0
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
index b9c180ac1eee..5289bb4bf5eb 100644
--- a/kernel/locking/rwsem.c
+++ b/kernel/locking/rwsem.c
@@ -30,6 +30,80 @@
#include <linux/hung_task.h>
#include <trace/events/lock.h>
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+/*
+ * Record @sem in a free per-task slot for scheduler boost inspection.
+ * Trylock callers can run in IRQ/atomic contexts (e.g. BPF helpers whose
+ * deferred irq_work uses up_read_non_owner() when irqs_disabled() was true),
+ * while process-context non-owner callers use down_read_non_owner(). Guard
+ * trylocks with @check_atomic so skipped contexts never strand @sem in slots.
+ */
+static inline void rwsem_record_hold(struct rw_semaphore *sem,
+ bool check_atomic)
+{
+ struct task_struct *curr = current;
+ u8 nr;
+ int i;
+
+ if (!static_branch_likely(&rwsem_sched_boost_key))
+ return;
+ if (unlikely(!in_task() ||
+ (check_atomic && (irqs_disabled() || in_atomic()))))
+ return;
+
+ nr = READ_ONCE(curr->nr_held_rwsems);
+ for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+ if (!curr->held_rwsems[i]) {
+ WRITE_ONCE(curr->held_rwsems[i], sem);
+ WRITE_ONCE(curr->nr_held_rwsems, nr + 1);
+ return;
+ }
+ }
+}
+
+/*
+ * Clear @sem from held_rwsems[] under preempt_disable() BEFORE
+ * releasing sem->count so another CPU cannot acquire/free @sem after
+ * unlock while scheduler inspection still sees it in held_rwsems[].
+ */
+static inline void rwsem_record_release(struct rw_semaphore *sem)
+{
+ struct task_struct *curr = current;
+ u8 nr = READ_ONCE(curr->nr_held_rwsems);
+ int i;
+
+ if (unlikely(!in_task() || !nr))
+ return;
+
+ for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+ if (curr->held_rwsems[i] == sem) {
+ WRITE_ONCE(curr->held_rwsems[i], NULL);
+ WRITE_ONCE(curr->nr_held_rwsems, nr - 1);
+ return;
+ }
+ }
+}
+
+static inline void rwsem_sched_unboost_if_needed(void)
+{
+ if (unlikely(in_task() && current->rwsem_boost_prio < MAX_PRIO))
+ sched_rwsem_recalc_boost(current);
+}
+#else
+static inline void rwsem_record_hold(struct rw_semaphore *sem,
+ bool check_atomic)
+{
+}
+
+static inline void rwsem_record_release(struct rw_semaphore *sem)
+{
+}
+
+static inline void rwsem_sched_unboost_if_needed(void)
+{
+}
+#endif
+
#ifndef CONFIG_PREEMPT_RT
#include "lock_events.h"
@@ -324,6 +398,9 @@ void __init_rwsem(struct rw_semaphore *sem, const char *name,
scoped_guard (raw_spinlock_init, &sem->wait_lock) {
sem->first_waiter = NULL;
}
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+ atomic_set(&sem->max_waiter_prio, MAX_PRIO);
+#endif
#ifdef CONFIG_RWSEM_SPIN_ON_OWNER
osq_lock_init(&sem->osq);
#endif
@@ -349,6 +426,50 @@ enum rwsem_wake_type {
RWSEM_WAKE_READ_OWNED /* Waker thread holds the read lock */
};
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+static inline int rwsem_clamp_waiter_prio(struct task_struct *tsk)
+{
+ /* Cap boost at CFS highest priority (nice -20) to protect RT tasks. */
+ return max_t(int, READ_ONCE(tsk->prio), MAX_RT_PRIO);
+}
+
+static inline void rwsem_update_max_waiter_prio(struct rw_semaphore *sem,
+ struct task_struct *waiter_task)
+{
+ int prio = rwsem_clamp_waiter_prio(waiter_task);
+
+ lockdep_assert_held(&sem->wait_lock);
+ if (prio < atomic_read(&sem->max_waiter_prio))
+ atomic_set(&sem->max_waiter_prio, prio);
+}
+
+static void rwsem_recalc_max_waiter_prio(struct rw_semaphore *sem)
+{
+ struct rwsem_waiter *first = sem->first_waiter;
+ struct rwsem_waiter *w;
+ int prio = MAX_PRIO;
+
+ lockdep_assert_held(&sem->wait_lock);
+ if (first) {
+ prio = min(prio, rwsem_clamp_waiter_prio(first->task));
+ list_for_each_entry(w, &first->list, list)
+ prio = min(prio, rwsem_clamp_waiter_prio(w->task));
+ }
+
+ if (atomic_read(&sem->max_waiter_prio) != prio)
+ atomic_set(&sem->max_waiter_prio, prio);
+}
+#else
+static inline void rwsem_update_max_waiter_prio(struct rw_semaphore *sem,
+ struct task_struct *waiter_task)
+{
+}
+
+static inline void rwsem_recalc_max_waiter_prio(struct rw_semaphore *sem)
+{
+}
+#endif
+
/*
* The typical HZ value is either 250 or 1000. So set the minimum waiting
* time to at least 4ms or 1 jiffy (if it is higher than 4ms) in the wait
@@ -394,9 +515,12 @@ static inline bool
rwsem_del_waiter(struct rw_semaphore *sem, struct rwsem_waiter *waiter)
{
lockdep_assert_held(&sem->wait_lock);
- if (__rwsem_del_waiter(sem, waiter))
+ if (__rwsem_del_waiter(sem, waiter)) {
+ rwsem_recalc_max_waiter_prio(sem);
return true;
+ }
atomic_long_andnot(RWSEM_FLAG_HANDOFF | RWSEM_FLAG_WAITERS, &sem->count);
+ rwsem_recalc_max_waiter_prio(sem);
return false;
}
@@ -542,6 +666,8 @@ static void rwsem_mark_wake(struct rw_semaphore *sem,
break;
} while ((waiter = next) != NULL);
+ rwsem_recalc_max_waiter_prio(sem);
+
adjustment = woken * RWSEM_READER_BIAS - adjustment;
lockevent_cond_inc(rwsem_wake_reader, woken);
@@ -683,6 +809,7 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,
* success.
*/
__rwsem_del_waiter(sem, waiter);
+ rwsem_recalc_max_waiter_prio(sem);
rwsem_set_owner(sem);
return true;
@@ -1081,6 +1208,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
} else {
list_add_tail(&waiter.list, &first->list);
}
+ rwsem_update_max_waiter_prio(sem, current);
/* we're now waiting on the lock, but no longer actively locking */
count = atomic_long_add_return(adjustment, &sem->count);
@@ -1177,6 +1305,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
sem->first_waiter = &waiter;
atomic_long_or(RWSEM_FLAG_WAITERS, &sem->count);
}
+ rwsem_update_max_waiter_prio(sem, current);
/* wait until we successfully acquire the lock */
set_current_state(state);
@@ -1376,7 +1505,7 @@ static inline int __down_write_trylock(struct rw_semaphore *sem)
/*
* unlock after reading
*/
-static inline void __up_read(struct rw_semaphore *sem)
+static inline void __up_read_common(struct rw_semaphore *sem, bool track_owner)
{
long tmp;
@@ -1384,6 +1513,8 @@ static inline void __up_read(struct rw_semaphore *sem)
DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem);
preempt_disable();
+ if (track_owner)
+ rwsem_record_release(sem);
rwsem_clear_reader_owned(sem);
tmp = atomic_long_add_return_release(-RWSEM_READER_BIAS, &sem->count);
DEBUG_RWSEMS_WARN_ON(tmp < 0, sem);
@@ -1395,6 +1526,20 @@ static inline void __up_read(struct rw_semaphore *sem)
rwsem_wake(sem);
}
preempt_enable();
+
+ /* Drop boost after waking waiters and re-enabling preemption. */
+ if (track_owner)
+ rwsem_sched_unboost_if_needed();
+}
+
+static inline void __up_read(struct rw_semaphore *sem)
+{
+ __up_read_common(sem, true);
+}
+
+static inline void __up_read_non_owner(struct rw_semaphore *sem)
+{
+ __up_read_common(sem, false);
}
/*
@@ -1413,6 +1558,7 @@ static inline void __up_write(struct rw_semaphore *sem)
!rwsem_test_oflags(sem, RWSEM_NONSPINNABLE), sem);
preempt_disable();
+ rwsem_record_release(sem);
rwsem_clear_owner(sem);
tmp = atomic_long_fetch_add_release(-RWSEM_WRITER_LOCKED, &sem->count);
if (unlikely(tmp & RWSEM_FLAG_WAITERS)) {
@@ -1420,6 +1566,7 @@ static inline void __up_write(struct rw_semaphore *sem)
rwsem_wake(sem);
}
preempt_enable();
+ rwsem_sched_unboost_if_needed();
}
/*
@@ -1517,11 +1664,16 @@ static inline int __down_read_trylock(struct rw_semaphore *sem)
return rwbase_read_trylock(&sem->rwbase);
}
-static inline void __up_read(struct rw_semaphore *sem)
+static inline void __up_read_non_owner(struct rw_semaphore *sem)
{
rwbase_read_unlock(&sem->rwbase, TASK_NORMAL);
}
+static inline void __up_read(struct rw_semaphore *sem)
+{
+ __up_read_non_owner(sem);
+}
+
static inline void __sched __down_write(struct rw_semaphore *sem)
{
rwbase_write_lock(&sem->rwbase, TASK_UNINTERRUPTIBLE);
@@ -1574,6 +1726,7 @@ void __sched down_read(struct rw_semaphore *sem)
rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_);
LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+ rwsem_record_hold(sem, false);
}
EXPORT_SYMBOL(down_read);
@@ -1588,6 +1741,7 @@ int __sched down_read_interruptible(struct rw_semaphore *sem)
return -EINTR;
}
+ rwsem_record_hold(sem, false);
return 0;
}
EXPORT_SYMBOL(down_read_interruptible);
@@ -1603,6 +1757,7 @@ int __sched down_read_killable(struct rw_semaphore *sem)
return -EINTR;
}
+ rwsem_record_hold(sem, false);
return 0;
}
EXPORT_SYMBOL(down_read_killable);
@@ -1615,8 +1770,10 @@ int down_read_trylock(struct rw_semaphore *sem)
{
int ret = __down_read_trylock(sem);
- if (ret == 1)
+ if (ret == 1) {
rwsem_acquire_read(&sem->dep_map, 0, 1, _RET_IP_);
+ rwsem_record_hold(sem, true);
+ }
return ret;
}
EXPORT_SYMBOL(down_read_trylock);
@@ -1630,6 +1787,7 @@ void __sched down_write(struct rw_semaphore *sem)
might_sleep();
rwsem_acquire(&sem->dep_map, 0, 0, _RET_IP_);
LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+ rwsem_record_hold(sem, false);
}
EXPORT_SYMBOL(down_write);
@@ -1648,6 +1806,7 @@ int __sched down_write_killable(struct rw_semaphore *sem)
return -EINTR;
}
+ rwsem_record_hold(sem, false);
return 0;
}
EXPORT_SYMBOL(down_write_killable);
@@ -1660,8 +1819,10 @@ int down_write_trylock(struct rw_semaphore *sem)
{
int ret = __down_write_trylock(sem);
- if (ret == 1)
+ if (ret == 1) {
rwsem_acquire(&sem->dep_map, 0, 1, _RET_IP_);
+ rwsem_record_hold(sem, true);
+ }
return ret;
}
@@ -1708,6 +1869,7 @@ void down_read_nested(struct rw_semaphore *sem, int subclass)
might_sleep();
rwsem_acquire_read(&sem->dep_map, subclass, 0, _RET_IP_);
LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+ rwsem_record_hold(sem, false);
}
EXPORT_SYMBOL(down_read_nested);
@@ -1722,6 +1884,7 @@ int down_read_killable_nested(struct rw_semaphore *sem, int subclass)
return -EINTR;
}
+ rwsem_record_hold(sem, false);
return 0;
}
EXPORT_SYMBOL(down_read_killable_nested);
@@ -1732,30 +1895,17 @@ void _down_write_nest_lock(struct rw_semaphore *sem, struct lockdep_map *nest)
might_sleep();
rwsem_acquire_nest(&sem->dep_map, 0, 0, nest, _RET_IP_);
LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+ rwsem_record_hold(sem, false);
}
EXPORT_SYMBOL(_down_write_nest_lock);
-void down_read_non_owner(struct rw_semaphore *sem)
- __no_context_analysis
-{
- might_sleep();
- __down_read(sem);
- /*
- * The owner value for a reader-owned lock is mostly for debugging
- * purpose only and is not critical to the correct functioning of
- * rwsem. So it is perfectly fine to set it in a preempt-enabled
- * context here.
- */
- __rwsem_set_reader_owned(sem, NULL);
-}
-EXPORT_SYMBOL(down_read_non_owner);
-
void down_write_nested(struct rw_semaphore *sem, int subclass)
__no_context_analysis
{
might_sleep();
rwsem_acquire(&sem->dep_map, subclass, 0, _RET_IP_);
LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+ rwsem_record_hold(sem, false);
}
EXPORT_SYMBOL(down_write_nested);
@@ -1771,15 +1921,34 @@ int __sched down_write_killable_nested(struct rw_semaphore *sem, int subclass)
return -EINTR;
}
+ rwsem_record_hold(sem, false);
return 0;
}
EXPORT_SYMBOL(down_write_killable_nested);
+#endif
+
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RWSEM_SCHED_BOOST)
+void down_read_non_owner(struct rw_semaphore *sem)
+ __no_context_analysis
+{
+ might_sleep();
+ __down_read(sem);
+ /*
+ * The owner value for a reader-owned lock is mostly for debugging
+ * purpose only and is not critical to the correct functioning of
+ * rwsem. So it is perfectly fine to set it in a preempt-enabled
+ * context here.
+ */
+ __rwsem_set_reader_owned(sem, NULL);
+}
+EXPORT_SYMBOL(down_read_non_owner);
+
void up_read_non_owner(struct rw_semaphore *sem)
__no_context_analysis
{
DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem);
- __up_read(sem);
+ __up_read_non_owner(sem);
}
EXPORT_SYMBOL(up_read_non_owner);
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5ed660f099a5..5737fe55d52c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -39,6 +39,7 @@
#include <linux/sched/nohz.h>
#include <linux/sched/rseq_api.h>
#include <linux/sched/rt.h>
+#include <linux/rwsem.h>
#include <linux/context_tracking.h>
#include <linux/cpuset.h>
@@ -1534,12 +1535,108 @@ int tg_nop(struct task_group *tg, void *data)
}
#endif
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+DEFINE_STATIC_KEY_TRUE(rwsem_sched_boost_key);
+
+static int __init rwsem_sched_boost_cmdline(char *str)
+{
+ bool val;
+
+ if (str && !kstrtobool(str, &val) && !val)
+ static_branch_disable(&rwsem_sched_boost_key);
+ return 1;
+}
+__setup("rwsem_sched_boost=", rwsem_sched_boost_cmdline);
+
+static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
+{
+ return p->rwsem_boost_prio < MAX_PRIO;
+}
+
+static inline void sched_rwsem_fork_init(struct task_struct *p)
+{
+ memset(p->held_rwsems, 0, sizeof(p->held_rwsems));
+ p->nr_held_rwsems = 0;
+ p->rwsem_boost_prio = MAX_PRIO;
+ INIT_LIST_HEAD(&p->rwsem_rq_node);
+}
+
+static int sched_rwsem_compute_boost_prio(struct task_struct *p)
+{
+ int best = MAX_PRIO;
+ int i;
+
+ if (!READ_ONCE(p->nr_held_rwsems))
+ return MAX_PRIO;
+
+ for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+ struct rw_semaphore *sem = READ_ONCE(p->held_rwsems[i]);
+ int wprio;
+
+ if (!sem)
+ continue;
+ wprio = atomic_read(&sem->max_waiter_prio);
+ if (wprio < best)
+ best = wprio;
+ }
+
+ if (best >= MAX_PRIO)
+ return MAX_PRIO;
+ if (best < MAX_RT_PRIO)
+ best = MAX_RT_PRIO;
+ if (best >= p->static_prio && !task_has_idle_policy(p))
+ return MAX_PRIO;
+
+ return best;
+}
+
+/*
+ * Recompute @p's effective priority and EEVDF weight from its held rwsems.
+ * Callers must hold @p's rq->lock (or pi_lock while @p is WAKING) and ensure
+ * @p cannot concurrently execute up_read() + kfree(sem) on another CPU.
+ */
+bool sched_rwsem_check_boost(struct task_struct *p, bool update_load)
+{
+ int new_boost;
+
+ if (p->sched_class != &fair_sched_class)
+ return false;
+
+ new_boost = sched_rwsem_compute_boost_prio(p);
+ if (new_boost == p->rwsem_boost_prio)
+ return false;
+
+ p->rwsem_boost_prio = new_boost;
+ if (!rt_or_dl_prio(p->prio))
+ p->prio = sched_rwsem_effective_prio(p);
+ set_load_weight(p, update_load);
+ return true;
+}
+
+void sched_rwsem_recalc_boost(struct task_struct *p)
+{
+ guard(task_rq_lock)(p);
+
+ if (p->sched_class == &fair_sched_class) {
+ update_rq_clock(task_rq(p));
+ sched_rwsem_check_boost(p, !p->se.sched_delayed);
+ }
+}
+#else
+static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
+{
+ return false;
+}
+
+static inline void sched_rwsem_fork_init(struct task_struct *p) { }
+#endif
+
void set_load_weight(struct task_struct *p, bool update_load)
{
- int prio = p->static_prio - MAX_RT_PRIO;
+ int prio = sched_rwsem_cfs_prio(p) - MAX_RT_PRIO;
struct load_weight lw;
- if (task_has_idle_policy(p)) {
+ if (task_has_idle_policy(p) && !sched_rwsem_is_boosted(p)) {
lw.weight = scale_load(WEIGHT_IDLEPRIO);
lw.inv_weight = WMULT_IDLEPRIO;
} else {
@@ -4888,6 +4985,7 @@ int sched_fork(u64 clone_flags, struct task_struct *p)
* Make sure we do not leak PI boosting priority to the child.
*/
p->prio = current->normal_prio;
+ sched_rwsem_fork_init(p);
uclamp_fork(p);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 220927453a88..98bc9029f9ae 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4344,4 +4344,47 @@ void sched_push_current_non_preferred_cpu(struct rq *rq);
static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { }
#endif
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+static inline int sched_rwsem_prio(struct task_struct *p, int base_prio)
+{
+ /*
+ * SCHED_IDLE tasks carry static_prio == NICE_TO_PRIO(0) despite
+ * WEIGHT_IDLEPRIO; use rwsem_boost_prio directly so low-priority
+ * waiters do not boost a SCHED_IDLE holder up to nice 0.
+ */
+ if (task_has_idle_policy(p))
+ return p->rwsem_boost_prio < MAX_PRIO ? p->rwsem_boost_prio
+ : base_prio;
+ return min(base_prio, p->rwsem_boost_prio);
+}
+
+static inline int sched_rwsem_cfs_prio(struct task_struct *p)
+{
+ return sched_rwsem_prio(p, p->static_prio);
+}
+
+static inline int sched_rwsem_effective_prio(struct task_struct *p)
+{
+ return sched_rwsem_prio(p, p->normal_prio);
+}
+
+bool sched_rwsem_check_boost(struct task_struct *p, bool update_load);
+#else
+static inline int sched_rwsem_cfs_prio(const struct task_struct *p)
+{
+ return p->static_prio;
+}
+
+static inline int sched_rwsem_effective_prio(const struct task_struct *p)
+{
+ return p->normal_prio;
+}
+
+static inline bool sched_rwsem_check_boost(struct task_struct *p,
+ bool update_load)
+{
+ return false;
+}
+#endif
+
#endif /* _KERNEL_SCHED_SCHED_H */
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 0ec401af80d1..d9e3c61fcfba 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -58,7 +58,7 @@ static int effective_prio(struct task_struct *p)
* to the normal priority:
*/
if (!rt_or_dl_prio(p->prio))
- return p->normal_prio;
+ return sched_rwsem_effective_prio(p);
return p->prio;
}
--
2.56.1.750.gf70dc682cf-goog
^ permalink raw reply [flat|nested] 4+ messages in thread
* [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters
2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang
2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
Waiman Long, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang
Keep queued fair tasks holding rwsems on rq->held_rwsem_tasks and check
waiter demand across scheduler wakeup, preemption, tick, and idle
load-balance pull paths.
Uncontended lock/unlock paths never touch rq->held_rwsem_tasks; a task
is linked only if preempted, enqueued, or crossing a tick while holding
an rwsem. Thus rq->held_rwsem_tasks stays empty for the vast majority of
ticks and idle pulls, exiting immediately with zero iteration or
max_waiter_prio lookups.
When a fair task holds a contended rwsem (max_waiter_prio < task prio),
temporarily elevate its effective priority and EEVDF weight without
altering user-visible static_prio/nice, and restore it on unlock or
lazily unlink uncontended unlocks on tick.
Signed-off-by: Peng Huang <penghuang@google.com>
---
kernel/sched/core.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
kernel/sched/fair.c | 51 ++++++++++++++++++++
kernel/sched/sched.h | 27 +++++++++++
3 files changed, 185 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5737fe55d52c..2ead771c703a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1561,7 +1561,12 @@ static inline void sched_rwsem_fork_init(struct task_struct *p)
INIT_LIST_HEAD(&p->rwsem_rq_node);
}
-static int sched_rwsem_compute_boost_prio(struct task_struct *p)
+static inline void sched_rwsem_rq_init(struct rq *rq)
+{
+ INIT_LIST_HEAD(&rq->held_rwsem_tasks);
+}
+
+int sched_rwsem_compute_boost_prio(struct task_struct *p)
{
int best = MAX_PRIO;
int i;
@@ -1613,10 +1618,88 @@ bool sched_rwsem_check_boost(struct task_struct *p, bool update_load)
return true;
}
+static inline void sched_rwsem_track_on_rq(struct rq *rq, struct task_struct *p)
+{
+ if (list_empty(&p->rwsem_rq_node))
+ list_add(&p->rwsem_rq_node, &rq->held_rwsem_tasks);
+}
+
+/*
+ * Safe to inspect p->held_rwsems[] here because ttwu() holds p->pi_lock
+ * after smp_cond_load_acquire(&p->on_cpu, !VAL), so @p is off-CPU and
+ * cannot concurrently run up_read()/kfree(sem).
+ */
+static inline void sched_rwsem_wakeup_boost(struct task_struct *p)
+{
+ if (READ_ONCE(p->nr_held_rwsems))
+ sched_rwsem_check_boost(p, false);
+}
+
+static inline void sched_rwsem_enqueue_locked(struct rq *rq,
+ struct task_struct *p)
+{
+ if (!READ_ONCE(p->nr_held_rwsems) ||
+ p->sched_class != &fair_sched_class)
+ return;
+
+ sched_rwsem_check_boost(p, false);
+ sched_rwsem_track_on_rq(rq, p);
+}
+
+static inline void sched_rwsem_on_preempt_locked(struct rq *rq,
+ struct task_struct *prev)
+{
+ if (!READ_ONCE(prev->nr_held_rwsems) || task_rq(prev) != rq ||
+ !task_on_rq_queued(prev) || prev->se.sched_delayed ||
+ prev->sched_class != &fair_sched_class)
+ return;
+
+ sched_rwsem_track_on_rq(rq, prev);
+ sched_rwsem_check_boost(prev, true);
+}
+
+static inline void sched_rwsem_tick_locked(struct rq *rq)
+{
+ struct task_struct *curr = rq->curr;
+ struct task_struct *p, *tmp;
+ bool need_resched = false;
+
+ if (READ_ONCE(curr->nr_held_rwsems) &&
+ curr->sched_class == &fair_sched_class &&
+ task_rq(curr) == rq && task_on_rq_queued(curr) &&
+ !curr->se.sched_delayed) {
+ sched_rwsem_track_on_rq(rq, curr);
+ sched_rwsem_check_boost(curr, true);
+ }
+
+ /*
+ * Lazily unlink tasks whose unlock fastpath skipped rq->lock because
+ * they were never boosted (!nr_held_rwsems short-circuits sem access),
+ * or if proxy/delayed-dequeue movement left a stale rq link.
+ */
+ list_for_each_entry_safe(p, tmp, &rq->held_rwsem_tasks, rwsem_rq_node) {
+ if (!READ_ONCE(p->nr_held_rwsems) ||
+ task_rq(p) != rq || !task_on_rq_queued(p)) {
+ sched_rwsem_dequeue_locked(p);
+ continue;
+ }
+ if (p == curr || task_on_cpu(rq, p) || p->se.sched_delayed)
+ continue;
+ sched_rwsem_check_boost(p, true);
+ if (p->rwsem_boost_prio < curr->prio)
+ need_resched = true;
+ }
+
+ if (need_resched)
+ resched_curr(rq);
+}
+
void sched_rwsem_recalc_boost(struct task_struct *p)
{
guard(task_rq_lock)(p);
+ if (!READ_ONCE(p->nr_held_rwsems))
+ sched_rwsem_dequeue_locked(p);
if (p->sched_class == &fair_sched_class) {
update_rq_clock(task_rq(p));
sched_rwsem_check_boost(p, !p->se.sched_delayed);
@@ -1629,6 +1712,16 @@ static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
}
static inline void sched_rwsem_fork_init(struct task_struct *p) { }
+static inline void sched_rwsem_rq_init(struct rq *rq) { }
+static inline void sched_rwsem_wakeup_boost(struct task_struct *p) { }
+
+static inline void sched_rwsem_enqueue_locked(struct rq *rq,
+ struct task_struct *p) { }
+
+static inline void sched_rwsem_on_preempt_locked(struct rq *rq,
+ struct task_struct *prev) { }
+
+static inline void sched_rwsem_tick_locked(struct rq *rq) { }
#endif
void set_load_weight(struct task_struct *p, bool update_load)
@@ -2284,6 +2377,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
if (!(flags & ENQUEUE_NOCLOCK))
update_rq_clock(rq);
+ sched_rwsem_enqueue_locked(rq, p);
+
/*
* Can be before ->enqueue_task() because uclamp considers the
* ENQUEUE_DELAYED task before its ->sched_delayed gets cleared
@@ -2323,7 +2418,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
* and mark the task ->sched_delayed.
*/
uclamp_rq_dec(rq, p);
- return p->sched_class->dequeue_task(rq, p, flags);
+ if (p->sched_class->dequeue_task(rq, p, flags)) {
+ sched_rwsem_dequeue_locked(p);
+ return true;
+ }
+ return false;
}
void activate_task(struct rq *rq, struct task_struct *p, int flags)
@@ -4566,6 +4665,8 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
*/
smp_cond_load_acquire(&p->on_cpu, !VAL);
+ sched_rwsem_wakeup_boost(p);
+
cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
if (task_cpu(p) != cpu) {
wake_flags |= WF_MIGRATED;
@@ -5957,6 +6058,7 @@ void sched_tick(void)
resched_curr(rq);
donor->sched_class->task_tick(rq, donor, 0);
+ sched_rwsem_tick_locked(rq);
if (sched_feat(LATENCY_WARN))
resched_latency = cpu_resched_latency(rq);
calc_global_load_tick(rq);
@@ -7391,6 +7493,8 @@ static void __sched notrace __schedule(int sched_mode)
switch_count = &prev->nvcsw;
}
+ sched_rwsem_on_preempt_locked(rq, prev);
+
pick_again:
assert_balance_callbacks_empty(rq);
next = pick_next_task(rq, &rf);
@@ -9125,6 +9229,7 @@ void __init sched_init(void)
rq->max_idle_balance_cost = sysctl_sched_migration_cost;
INIT_LIST_HEAD(&rq->cfs_tasks);
+ sched_rwsem_rq_init(rq);
rq_attach_root(rq, &def_root_domain);
#ifdef CONFIG_NO_HZ_COMMON
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0c8af1cac2d5..59ce6321a265 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11312,6 +11312,53 @@ static struct task_struct *detach_one_task(struct lb_env *env)
return NULL;
}
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+/*
+ * On idle pull, prioritize migrating a fair task holding a contended rwsem
+ * so a blocked high-priority waiter is not starved on a busy little core.
+ */
+static int detach_boosted_rwsem_task(struct lb_env *env)
+{
+ struct task_struct *p, *tmp;
+
+ if (env->idle != CPU_IDLE && env->idle != CPU_NEWLY_IDLE)
+ return 0;
+
+ list_for_each_entry_safe(p, tmp, &env->src_rq->held_rwsem_tasks,
+ rwsem_rq_node) {
+ if (env->src_rq->nr_running <= 1)
+ break;
+ if (task_rq(p) != env->src_rq || !task_on_rq_queued(p)) {
+ sched_rwsem_dequeue_locked(p);
+ continue;
+ }
+ if (task_on_cpu(env->src_rq, p) || p->se.sched_delayed)
+ continue;
+ if (sched_rwsem_compute_boost_prio(p) >= MAX_PRIO)
+ continue;
+ if (!can_migrate_task(p, env))
+ continue;
+
+ /*
+ * enqueue_task() on dst_rq updates the boosted weight
+ * via sched_rwsem_enqueue_locked(), avoiding a redundant
+ * reweight_eevdf() on src_rq before detaching.
+ */
+ detach_task(p, env);
+ list_add(&p->se.group_node, &env->tasks);
+ env->imbalance = 0;
+ return 1;
+ }
+
+ return 0;
+}
+#else
+static inline int detach_boosted_rwsem_task(struct lb_env *env)
+{
+ return 0;
+}
+#endif
+
/*
* detach_tasks() -- tries to detach up to imbalance load/util/tasks from
* busiest_rq, as part of a balancing operation within domain "sd".
@@ -11339,6 +11386,10 @@ static int detach_tasks(struct lb_env *env)
if (env->imbalance <= 0)
return 0;
+ detached = detach_boosted_rwsem_task(env);
+ if (detached)
+ return detached;
+
while (!list_empty(tasks)) {
/*
* We don't want to steal all, otherwise we may be treated likewise,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 98bc9029f9ae..58e53980e71b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1317,6 +1317,13 @@ struct rq {
int online;
struct list_head cfs_tasks;
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+ /*
+ * Queued fair tasks holding an rwsem; protected by rq->lock and
+ * unlinked on dequeue_task() (incl. migration and TASK_DEAD).
+ */
+ struct list_head held_rwsem_tasks;
+#endif
struct sched_avg avg_rt;
struct sched_avg avg_dl;
@@ -4368,7 +4375,18 @@ static inline int sched_rwsem_effective_prio(struct task_struct *p)
return sched_rwsem_prio(p, p->normal_prio);
}
+int sched_rwsem_compute_boost_prio(struct task_struct *p);
bool sched_rwsem_check_boost(struct task_struct *p, bool update_load);
+
+/*
+ * Remove @p when it leaves an rq (sleep, migration, class change,
+ * TASK_DEAD via DEQUEUE_SPECIAL, or stale cross-rq cleanup).
+ */
+static inline void sched_rwsem_dequeue_locked(struct task_struct *p)
+{
+ if (!list_empty(&p->rwsem_rq_node))
+ list_del_init(&p->rwsem_rq_node);
+}
#else
static inline int sched_rwsem_cfs_prio(const struct task_struct *p)
{
@@ -4380,11 +4398,20 @@ static inline int sched_rwsem_effective_prio(const struct task_struct *p)
return p->normal_prio;
}
+static inline int sched_rwsem_compute_boost_prio(struct task_struct *p)
+{
+ return MAX_PRIO;
+}
+
static inline bool sched_rwsem_check_boost(struct task_struct *p,
bool update_load)
{
return false;
}
+
+static inline void sched_rwsem_dequeue_locked(struct task_struct *p)
+{
+}
#endif
#endif /* _KERNEL_SCHED_SCHED_H */
--
2.56.1.750.gf70dc682cf-goog
^ permalink raw reply [flat|nested] 4+ messages in thread
* [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost
2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
Waiman Long, Juri Lelli, Vincent Guittot
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang
Add KUnit tests covering per-task rwsem slot tracking, interleaved
unlock order, slot overflow fallback, write-lock downgrade, wakeup
boost for both reader and writer holders, preemption priority-inversion
A/B latency, false-promotion checks, and fastpath overhead measurement.
Signed-off-by: Peng Huang <penghuang@google.com>
---
kernel/locking/Makefile | 1 +
kernel/locking/rwsem_sched_boost_test.c | 504 ++++++++++++++++++++++++
lib/Kconfig.debug | 10 +
3 files changed, 515 insertions(+)
create mode 100644 kernel/locking/rwsem_sched_boost_test.c
diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile
index cee1901d4cff..3df6ae1f1707 100644
--- a/kernel/locking/Makefile
+++ b/kernel/locking/Makefile
@@ -37,4 +37,5 @@ obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o
obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o
obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o
obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
+obj-$(CONFIG_RWSEM_SCHED_BOOST_KUNIT_TEST) += rwsem_sched_boost_test.o
obj-$(CONFIG_LOCK_EVENT_COUNTS) += lock_events.o
diff --git a/kernel/locking/rwsem_sched_boost_test.c b/kernel/locking/rwsem_sched_boost_test.c
new file mode 100644
index 000000000000..d60d2a87237a
--- /dev/null
+++ b/kernel/locking/rwsem_sched_boost_test.c
@@ -0,0 +1,504 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit tests for rw_semaphore per-task hold tracking and scheduler
+ * priority boosting.
+ */
+
+#include <kunit/test.h>
+#include <linux/atomic.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
+#include <linux/ktime.h>
+#include <linux/rwsem.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <uapi/linux/sched/types.h>
+
+static bool task_holds_rwsem(const struct task_struct *p,
+ const struct rw_semaphore *sem)
+{
+ int i;
+
+ for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+ if (READ_ONCE(p->held_rwsems[i]) == sem)
+ return true;
+ }
+ return false;
+}
+
+static void test_basic_hold_and_release(struct kunit *test)
+{
+ DECLARE_RWSEM(sem);
+
+ KUNIT_ASSERT_EQ(test, current->nr_held_rwsems, 0);
+ KUNIT_ASSERT_EQ(test, atomic_read(&sem.max_waiter_prio), MAX_PRIO);
+
+ down_read(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+ up_read(&sem);
+
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+ KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem));
+
+ KUNIT_ASSERT_EQ(test, down_read_trylock(&sem), 1);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ up_read(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+
+ down_write(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+ up_write(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+
+ KUNIT_ASSERT_EQ(test, down_write_trylock(&sem), 1);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ up_write(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+static void test_interleaved_unlock(struct kunit *test)
+{
+ DECLARE_RWSEM(sem_a);
+ DECLARE_RWSEM(sem_b);
+ DECLARE_RWSEM(sem_c);
+
+ down_read(&sem_a);
+ down_read(&sem_b);
+ down_read(&sem_c);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems,
+ min(3, CONFIG_RWSEM_SCHED_BOOST_SLOTS));
+
+ /* Release out of acquisition order: B, A, C. */
+ up_read(&sem_b);
+ KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem_b));
+ up_read(&sem_a);
+ KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem_a));
+ up_read(&sem_c);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+static void test_overflow_graceful(struct kunit *test)
+{
+ const int total = CONFIG_RWSEM_SCHED_BOOST_SLOTS + 3;
+ struct rw_semaphore *sems;
+ int i;
+
+ sems = kunit_kcalloc(test, total, sizeof(*sems), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, sems);
+
+ for (i = 0; i < total; i++)
+ init_rwsem(&sems[i]);
+
+ for (i = 0; i < total; i++)
+ down_read(&sems[i]);
+
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems,
+ CONFIG_RWSEM_SCHED_BOOST_SLOTS);
+
+ for (i = 0; i < total; i++)
+ up_read(&sems[i]);
+
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+ for (i = 0; i < total; i++)
+ KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sems[i]));
+}
+
+static void test_downgrade_write(struct kunit *test)
+{
+ DECLARE_RWSEM(sem);
+
+ down_write(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+
+ downgrade_write(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+ KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+
+ up_read(&sem);
+ KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+struct boost_ctx {
+ struct rw_semaphore sem;
+ struct completion reader_locked;
+ struct completion reader_done;
+ bool holder_is_writer;
+ bool holder_is_idle;
+ int waiter_nice;
+ int observed_boost_prio;
+ int observed_prio;
+ int observed_nice;
+ int post_unlock_boost_prio;
+ int post_unlock_prio;
+ u64 reader_work_ns;
+};
+
+static int wakeup_holder_fn(void *arg)
+{
+ struct boost_ctx *ctx = arg;
+
+ if (ctx->holder_is_idle) {
+ struct sched_param sp = { .sched_priority = 0 };
+
+ sched_setscheduler_nocheck(current, SCHED_IDLE, &sp);
+ } else {
+ set_user_nice(current, 19);
+ }
+ if (ctx->holder_is_writer)
+ down_write(&ctx->sem);
+ else
+ down_read(&ctx->sem);
+ complete(&ctx->reader_locked);
+
+ /* Sleep so the higher-priority contender queues on ctx->sem first. */
+ msleep(25);
+
+ ctx->observed_boost_prio = current->rwsem_boost_prio;
+ ctx->observed_prio = current->prio;
+ ctx->observed_nice = task_nice(current);
+
+ if (ctx->holder_is_writer)
+ up_write(&ctx->sem);
+ else
+ up_read(&ctx->sem);
+ ctx->post_unlock_boost_prio = current->rwsem_boost_prio;
+ ctx->post_unlock_prio = current->prio;
+ complete(&ctx->reader_done);
+ return 0;
+}
+
+static void run_wakeup_boost_case(struct kunit *test, bool holder_is_writer,
+ bool holder_is_idle, int waiter_nice)
+{
+ struct boost_ctx *ctx;
+ struct task_struct *holder;
+ int saved_nice = task_nice(current);
+ int exp_prio = NICE_TO_PRIO(waiter_nice);
+ int exp_post_prio = holder_is_idle ? DEFAULT_PRIO : NICE_TO_PRIO(19);
+
+ ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, ctx);
+ ctx->holder_is_writer = holder_is_writer;
+ ctx->holder_is_idle = holder_is_idle;
+ ctx->waiter_nice = waiter_nice;
+ init_rwsem(&ctx->sem);
+ init_completion(&ctx->reader_locked);
+ init_completion(&ctx->reader_done);
+
+ holder = kthread_run(wakeup_holder_fn, ctx,
+ holder_is_writer ? "rwsem_wr_hld" : "rwsem_rd_hld");
+ KUNIT_ASSERT_FALSE(test, IS_ERR(holder));
+
+ wait_for_completion(&ctx->reader_locked);
+ set_user_nice(current, waiter_nice);
+ if (holder_is_writer) {
+ down_read(&ctx->sem);
+ up_read(&ctx->sem);
+ } else {
+ down_write(&ctx->sem);
+ up_write(&ctx->sem);
+ }
+ set_user_nice(current, saved_nice);
+
+ wait_for_completion(&ctx->reader_done);
+
+ KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, exp_prio);
+ KUNIT_EXPECT_EQ(test, ctx->observed_prio, exp_prio);
+ if (!holder_is_idle)
+ KUNIT_EXPECT_EQ(test, ctx->observed_nice, 19);
+ KUNIT_EXPECT_EQ(test, ctx->post_unlock_boost_prio, MAX_PRIO);
+ KUNIT_EXPECT_EQ(test, ctx->post_unlock_prio, exp_post_prio);
+ KUNIT_EXPECT_EQ(test, atomic_read(&ctx->sem.max_waiter_prio), MAX_PRIO);
+}
+
+static void test_contention_wakeup_boost(struct kunit *test)
+{
+ run_wakeup_boost_case(test, false, false, -10);
+}
+
+static void test_writer_holder_wakeup_boost(struct kunit *test)
+{
+ run_wakeup_boost_case(test, true, false, -10);
+}
+
+static void test_sched_idle_holder_boost_cap(struct kunit *test)
+{
+ run_wakeup_boost_case(test, false, true, 10);
+}
+
+static int hog_fn(void *arg)
+{
+ set_user_nice(current, 0);
+ while (!kthread_should_stop()) {
+ cpu_relax();
+ cond_resched();
+ }
+ return 0;
+}
+
+static int busy_reader_fn(void *arg)
+{
+ struct boost_ctx *ctx = arg;
+ u64 start, end;
+ int i;
+
+ set_user_nice(current, 19);
+ down_read(&ctx->sem);
+ start = ktime_get_ns();
+ complete(&ctx->reader_locked);
+
+ /*
+ * Busy-loop for ~10ms of CPU work while competing against nice=0
+ * hogs; once the nice=-10 writer blocks on ctx->sem, tick/preempt
+ * hooks should boost this reader to prio 110.
+ */
+ for (i = 0; i < 20; i++) {
+ u64 t0 = ktime_get_ns();
+
+ while (ktime_get_ns() - t0 < 500 * NSEC_PER_USEC)
+ cpu_relax();
+ cond_resched();
+ }
+ end = ktime_get_ns();
+
+ ctx->observed_boost_prio = current->rwsem_boost_prio;
+ ctx->observed_prio = current->prio;
+ ctx->observed_nice = task_nice(current);
+ ctx->reader_work_ns = end - start;
+
+ up_read(&ctx->sem);
+ ctx->post_unlock_boost_prio = current->rwsem_boost_prio;
+ ctx->post_unlock_prio = current->prio;
+ complete(&ctx->reader_done);
+ return 0;
+}
+
+static u64 run_one_inversion_iter(struct kunit *test, struct boost_ctx *ctx)
+{
+ struct task_struct *reader;
+ int saved_nice = task_nice(current);
+ u64 wait_ns;
+
+ reinit_completion(&ctx->reader_locked);
+ reinit_completion(&ctx->reader_done);
+
+ reader = kthread_run(busy_reader_fn, ctx, "rwsem_busy_rdr");
+ KUNIT_ASSERT_FALSE(test, IS_ERR(reader));
+
+ wait_for_completion(&ctx->reader_locked);
+ set_user_nice(current, -10);
+ wait_ns = ktime_get_ns();
+ down_write(&ctx->sem);
+ wait_ns = ktime_get_ns() - wait_ns;
+ up_write(&ctx->sem);
+ set_user_nice(current, saved_nice);
+
+ wait_for_completion(&ctx->reader_done);
+ return wait_ns;
+}
+
+static void start_hogs(struct kunit *test, struct task_struct **hogs, int count)
+{
+ int i;
+
+ for (i = 0; i < count; i++) {
+ hogs[i] = kthread_run(hog_fn, NULL, "rwsem_hog%d", i);
+ KUNIT_ASSERT_FALSE(test, IS_ERR(hogs[i]));
+ }
+}
+
+static void stop_hogs(struct task_struct **hogs, int count)
+{
+ int i;
+
+ for (i = 0; i < count; i++)
+ kthread_stop(hogs[i]);
+}
+
+static void test_preemption_inversion_latency(struct kunit *test)
+{
+ struct boost_ctx *ctx;
+ struct task_struct *hogs[4];
+ const int iters = 3;
+ u64 ctrl_sum = 0, treat_sum = 0;
+ u64 ctrl_max = 0, treat_max = 0;
+ int i;
+
+ ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, ctx);
+ init_rwsem(&ctx->sem);
+ init_completion(&ctx->reader_locked);
+ init_completion(&ctx->reader_done);
+
+ start_hogs(test, hogs, ARRAY_SIZE(hogs));
+
+ rwsem_sched_boost_set(false);
+ for (i = 0; i < iters; i++) {
+ u64 w = run_one_inversion_iter(test, ctx);
+
+ ctrl_sum += w;
+ ctrl_max = max(ctrl_max, w);
+ KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, MAX_PRIO);
+ }
+
+ rwsem_sched_boost_set(true);
+ for (i = 0; i < iters; i++) {
+ u64 w = run_one_inversion_iter(test, ctx);
+
+ treat_sum += w;
+ treat_max = max(treat_max, w);
+ KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio,
+ NICE_TO_PRIO(-10));
+ KUNIT_EXPECT_EQ(test, ctx->observed_prio, NICE_TO_PRIO(-10));
+ KUNIT_EXPECT_EQ(test, ctx->post_unlock_boost_prio, MAX_PRIO);
+ }
+
+ stop_hogs(hogs, ARRAY_SIZE(hogs));
+
+ kunit_info(test,
+ "A/B (n=%d, 4 hogs): Control mean=%llu ms max=%llu ms | Treatment mean=%llu ms max=%llu ms\n",
+ iters,
+ (ctrl_sum / iters) / NSEC_PER_MSEC,
+ ctrl_max / NSEC_PER_MSEC,
+ (treat_sum / iters) / NSEC_PER_MSEC,
+ treat_max / NSEC_PER_MSEC);
+
+ KUNIT_EXPECT_LT(test, treat_sum, ctrl_sum);
+}
+
+static void test_no_false_promotion_uncontended(struct kunit *test)
+{
+ struct boost_ctx *ctx;
+ struct task_struct *hogs[2];
+ struct task_struct *reader;
+
+ ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, ctx);
+ init_rwsem(&ctx->sem);
+ init_completion(&ctx->reader_locked);
+ init_completion(&ctx->reader_done);
+
+ start_hogs(test, hogs, ARRAY_SIZE(hogs));
+
+ rwsem_sched_boost_set(true);
+ reader = kthread_run(busy_reader_fn, ctx, "rwsem_fp_rdr");
+ KUNIT_ASSERT_FALSE(test, IS_ERR(reader));
+
+ wait_for_completion(&ctx->reader_locked);
+ /* No waiter blocks on ctx->sem. */
+ wait_for_completion(&ctx->reader_done);
+
+ stop_hogs(hogs, ARRAY_SIZE(hogs));
+
+ KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, MAX_PRIO);
+ KUNIT_EXPECT_EQ(test, ctx->observed_prio, NICE_TO_PRIO(19));
+ KUNIT_EXPECT_EQ(test, ctx->observed_nice, 19);
+}
+
+static noinline u64 bench_read_fastpath(struct rw_semaphore *sem, int iters)
+{
+ u64 start = ktime_get_ns();
+ int i;
+
+ for (i = 0; i < iters; i++) {
+ down_read(sem);
+ barrier();
+ up_read(sem);
+ }
+ return ktime_get_ns() - start;
+}
+
+static noinline u64 bench_write_fastpath(struct rw_semaphore *sem, int iters)
+{
+ u64 start = ktime_get_ns();
+ int i;
+
+ for (i = 0; i < iters; i++) {
+ down_write(sem);
+ barrier();
+ up_write(sem);
+ }
+ return ktime_get_ns() - start;
+}
+
+static void test_fastpath_overhead(struct kunit *test)
+{
+ DECLARE_RWSEM(outer1);
+ DECLARE_RWSEM(outer2);
+ DECLARE_RWSEM(sem);
+ const int rounds = 5;
+ const int per_round = 200000;
+ const u64 total_ops = (u64)rounds * per_round;
+ u64 r_off = 0, r_on = 0;
+ u64 w_off = 0, w_on = 0;
+ u64 n_off = 0, n_on = 0;
+ int i;
+
+ /* Warmup instruction and branch caches. */
+ bench_read_fastpath(&sem, 10000);
+ bench_write_fastpath(&sem, 10000);
+
+ for (i = 0; i < rounds; i++) {
+ rwsem_sched_boost_set(false);
+ r_off += bench_read_fastpath(&sem, per_round);
+ w_off += bench_write_fastpath(&sem, per_round);
+
+ rwsem_sched_boost_set(true);
+ r_on += bench_read_fastpath(&sem, per_round);
+ w_on += bench_write_fastpath(&sem, per_round);
+ }
+
+ /* Measure nested read fastpath with 2 outer slots already occupied. */
+ for (i = 0; i < rounds; i++) {
+ rwsem_sched_boost_set(false);
+ down_read(&outer1);
+ down_read(&outer2);
+ n_off += bench_read_fastpath(&sem, per_round);
+ up_read(&outer2);
+ up_read(&outer1);
+
+ rwsem_sched_boost_set(true);
+ down_read(&outer1);
+ down_read(&outer2);
+ n_on += bench_read_fastpath(&sem, per_round);
+ up_read(&outer2);
+ up_read(&outer1);
+ }
+
+ kunit_info(test,
+ "Fastpath (%llu ops): read off=%llu.%02llu on=%llu.%02llu ns/op | write off=%llu.%02llu on=%llu.%02llu ns/op | nested2 read off=%llu.%02llu on=%llu.%02llu ns/op\n",
+ total_ops,
+ (r_off / total_ops), (r_off * 100 / total_ops) % 100,
+ (r_on / total_ops), (r_on * 100 / total_ops) % 100,
+ (w_off / total_ops), (w_off * 100 / total_ops) % 100,
+ (w_on / total_ops), (w_on * 100 / total_ops) % 100,
+ (n_off / total_ops), (n_off * 100 / total_ops) % 100,
+ (n_on / total_ops), (n_on * 100 / total_ops) % 100);
+}
+
+static struct kunit_case rwsem_sched_boost_test_cases[] = {
+ KUNIT_CASE(test_basic_hold_and_release),
+ KUNIT_CASE(test_interleaved_unlock),
+ KUNIT_CASE(test_overflow_graceful),
+ KUNIT_CASE(test_downgrade_write),
+ KUNIT_CASE(test_contention_wakeup_boost),
+ KUNIT_CASE(test_writer_holder_wakeup_boost),
+ KUNIT_CASE(test_sched_idle_holder_boost_cap),
+ KUNIT_CASE(test_preemption_inversion_latency),
+ KUNIT_CASE(test_no_false_promotion_uncontended),
+ KUNIT_CASE(test_fastpath_overhead),
+ {}
+};
+
+static struct kunit_suite rwsem_sched_boost_test_suite = {
+ .name = "rwsem_sched_boost",
+ .test_cases = rwsem_sched_boost_test_cases,
+};
+
+kunit_test_suite(rwsem_sched_boost_test_suite);
+
+MODULE_LICENSE("GPL");
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2d22638ec13a..b3f2b8d6633e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1692,6 +1692,16 @@ config WW_MUTEX_SELFTEST
Say M if you want these self tests to build as a module.
Say N if you are unsure.
+config RWSEM_SCHED_BOOST_KUNIT_TEST
+ bool "KUnit tests for rw_semaphore scheduler priority boost"
+ depends on KUNIT && RWSEM_SCHED_BOOST
+ default KUNIT_ALL_TESTS
+ help
+ Enable KUnit tests verifying per-task rw_semaphore hold tracking,
+ atomic max_waiter_prio updates, contention-driven priority boosting,
+ and zero false promotion when uncontended.
+ If unsure, say N.
+
config SCF_TORTURE_TEST
tristate "torture tests for smp_call_function*()"
depends on DEBUG_KERNEL
--
2.56.1.750.gf70dc682cf-goog
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-19 22:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®