mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters
@ 2026-11-19 22:49 Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
	Waiman Long, Juri Lelli, Vincent Guittot
  Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang

Hi,

High-priority interactive threads (such as UI/render threads) can stall
for tens to hundreds of milliseconds on shared rw_semaphores like
mm->mmap_lock, inode->i_rwsem, or mapping->i_mmap_rwsem/invalidate_lock
when lower-priority (or SCHED_IDLE) read-owners are preempted on busy
CPUs and block an incoming higher-weight writer. Upstream
CONFIG_SCHED_PROXY_EXEC currently covers mutexes, and out-of-tree rwsem
proxy efforts only track a single write-owner, leaving multi-reader
inversions unaddressed.

This RFC proposes a lightweight priority-inheritance mechanism for CFS
rw_semaphore holders that accelerates stalled readers and writers in
parallel without requiring lock-side reader tracking.

Design rationale
----------------
1. Per-task hold tracking (avoiding lock-side reader lists):
   Because rw_semaphore allows unbounded concurrent readers, storing a
   reader list inside struct rw_semaphore would require dynamic
   allocation or extra locking in the down_read()/up_read() fastpath.
   Instead, each task records up to CONFIG_RWSEM_SCHED_BOOST_SLOTS
   (default 4) rwsem pointers in task_struct under current-only access.
   This supports an unlimited number of concurrent readers per rwsem;
   the slot count only caps simultaneous nested rwsems held by one task.

2. Contended waiter demand (sem->max_waiter_prio):
   Waiters in rwsem_down_{read,write}_slowpath() publish the lowest
   normal prio (i.e. highest nice weight, capped at MAX_RT_PRIO so CFS
   holders never escalate into RT/DL) into sem->max_waiter_prio,
   recalculating without redundant atomic writes as waiters wake/leave.

3. Per-rq runqueue list and parallel boost:
   Runnable CFS tasks holding at least one rwsem are linked onto a
   per-CPU list (rq->held_rwsem_tasks) during enqueue/dequeue/migrate
   under rq_lock, keeping lock/unlock fastpaths list-free. Normal short
   holds never touch rq->held_rwsem_tasks; tasks are only linked if
   preempted, enqueued, or crossing a tick while holding a lock, so tick
   and idle-pull typically exit immediately on an empty list with zero
   iteration or max_waiter_prio lookups. Across wakeup, preemption, tick,
   and idle-balance pull paths, the scheduler checks held locks'
   sem->max_waiter_prio and elevates the holder's effective priority and
   EEVDF weight (preserving static_prio/nice and handling SCHED_IDLE
   separately) down to min(p->prio, max_waiter_prio), restoring cleanly
   on unlock. Runtime/build disablement is provided via
   rwsem_sched_boost=0, CONFIG_RWSEM_SCHED_BOOST_SLOTS=0, or PREEMPT_RT.

4. Model trade-off (Parallel priority spray vs. Proxy Execution):
   Proxy Execution follows task->blocked_on to boost one owner at a
   time on the owner's CPU (entailing cross-CPU proxy migrations and
   requiring a lock-side owner list for readers), while naturally
   propagating transitively across mutex/futex/rwsem chains. By
   contrast, this RFC avoids lock-side reader tracking and lets all
   preempted readers on any CPU boost and drain in parallel once
   contended, at the intentional cost of not propagating transitively
   through mutex/futex chains (e.g. RT -> mutex -> rwsem).

Validation & known trade-offs
-----------------------------
- Included KUnit suite (Patch 3/3, CONFIG_RWSEM_SCHED_BOOST_KUNIT_TEST):
  Reproducible under ARCH=um via kunit.py (10/10 pass incl. SCHED_IDLE
  cap, interleaved unlock, and slot overflow):
  * Preemption inversion (n=3, 4 nice=0 hogs):
    Control mean 1939 ms (max 1970 ms) -> Treatment mean 49 ms (-97.5%)
  * Uncontended fastpath (1,000,000 ops):
    read +2.75 ns/op (21.43 -> 24.18),
    write +1.99 ns/op (20.78 -> 22.77),
    nested2 read +4.31 ns/op (21.20 -> 25.51)
- ARM64 mobile hardware (6.18 backport, asymmetric SMP, user-space
  madvise/mprotect stress reproducer):
  * mmap_lock inversion (nice -10 madvise vs 7 nice 0 mprotect hogs):
    - Unpinned 5-round AB (100 runs): mean 120.22 -> 7.26 ms (-94.0%),
      missed 16ms deadlines 46/100 -> 8/100
    - Mid-cluster pinned 4-round AB/BA (80 runs): mean 281.11 ->
      7.34 ms (-97.4%), max 736.14 -> 50.76 ms, missed 16ms 80/80 -> 4/80
  * Big-core pinned syscall fastpath (big core, 4 rounds x 100k iters):
    madvise 615.8 -> 604.8 ns/op (-1.8%),
    mprotect 857.5 -> 848.5 ns/op (-1.0%)
  * Contended equal-priority non-inversion (4 threads, Mid pinned):
    145.5 -> 162.5 ms (+11.7% throughput trade-off under contention;
    also +10.8% in x86_64 QEMU -smp 4 TCG)
  * Pinned hackbench / pipe ping-pong (Mid+Big):
    hackbench 263.7 -> 243.3 ms (-7.7%),
    pipe 10.81 -> 10.33 us/op (-4.4%)

Request for feedback
--------------------
Before iterating further, feedback is especially welcome on:
- Parallel priority spray (task-slot tracking) vs. extending Proxy
  Execution with lock-side reader tracking, vs. a slowpath/owner-only
  heuristic;
- Using normal prio / nice weight as the demand currency vs. EEVDF
  virtual deadline / lag considerations;
- Struct size impact of adding max_waiter_prio to struct rw_semaphore
  (+8 bytes aligned on 64-bit, 32 -> 40 bytes, embedded in inode/mm);
- Intentional RFC boundaries: 4-slot nested limit per task, boosting
  all concurrent readers equally, idle-pull inspecting only the
  busiest rq (no global CPU scan/cpumask), no cgroup/task_group share
  propagation, and no transitive inheritance across mutex/futex chains.

Per Documentation/process/generated-content.rst: Jetski AI coding
assistance was used for drafting, refactoring discussions, and KUnit /
benchmark verification; all patches and tests have been reviewed and
verified by the author.

Peng Huang (3):
  locking/rwsem, sched: track held rwsems and waiter priorities
  sched: boost rwsem holders with contended high-priority waiters
  locking/rwsem: add KUnit tests for scheduler priority boost

 .../admin-guide/kernel-parameters.txt         |   5 +
 include/linux/rwsem.h                         |  28 +
 include/linux/sched.h                         |  16 +
 init/init_task.c                              |   4 +
 kernel/Kconfig.locks                          |  15 +
 kernel/locking/Makefile                       |   1 +
 kernel/locking/rwsem.c                        | 209 +++++++-
 kernel/locking/rwsem_sched_boost_test.c       | 504 ++++++++++++++++++
 kernel/sched/core.c                           | 211 +++++++-
 kernel/sched/fair.c                           |  51 ++
 kernel/sched/sched.h                          |  70 +++
 kernel/sched/syscalls.c                       |   2 +-
 lib/Kconfig.debug                             |  10 +
 13 files changed, 1101 insertions(+), 25 deletions(-)
 create mode 100644 kernel/locking/rwsem_sched_boost_test.c

-- 
2.56.1.750.gf70dc682cf-goog


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities
  2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang
  2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
	Waiman Long, Juri Lelli, Vincent Guittot
  Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang

Track per-task held rw_semaphore pointers in a small fixed-size slot
array (CONFIG_RWSEM_SCHED_BOOST_SLOTS) and track the highest-priority
contending waiter on each rw_semaphore (max_waiter_prio, capped at
MAX_RT_PRIO).

Blocking acquisitions skip irqs/atomic guards to avoid pushf on the
fastpath, while trylock paths keep IRQ/atomic context filtering for
non-owner unlockers such as BPF helpers. Held slots are cleared under
preempt_disable() before releasing sem->count so another CPU cannot
acquire and free the semaphore while a tick/preempt check still sees
it in held_rwsems[].

Also wire up core priority calculation, runtime toggle
(rwsem_sched_boost=), and post-unlock priority restoration.

Signed-off-by: Peng Huang <penghuang@google.com>
---
 .../admin-guide/kernel-parameters.txt         |   5 +
 include/linux/rwsem.h                         |  28 +++
 include/linux/sched.h                         |  16 ++
 init/init_task.c                              |   4 +
 kernel/Kconfig.locks                          |  15 ++
 kernel/locking/rwsem.c                        | 211 ++++++++++++++++--
 kernel/sched/core.c                           | 102 ++++++++-
 kernel/sched/sched.h                          |  43 ++++
 kernel/sched/syscalls.c                       |   2 +-
 9 files changed, 402 insertions(+), 24 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec..2c5207dd9c53 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6778,6 +6778,11 @@ Kernel parameters
 
 	rw		[KNL] Mount root device read-write on boot
 
+	rwsem_sched_boost=
+			[KNL] Enable or disable rwsem scheduler priority
+			boosting when CONFIG_RWSEM_SCHED_BOOST=y.
+			Defaults to 1 (enabled). Format: <bool>
+
 	S		[KNL] Run init in single mode
 
 	s390_iommu=	[HW,S390]
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 6a1a7bae5f81..b9188de31881 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -16,6 +16,8 @@
 #include <linux/atomic.h>
 #include <linux/err.h>
 #include <linux/cleanup.h>
+#include <linux/jump_label.h>
+#include <linux/sched/prio.h>
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 # define __RWSEM_DEP_MAP_INIT(lockname)			\
@@ -58,6 +60,9 @@ context_lock_struct(rw_semaphore) {
 #endif
 	raw_spinlock_t wait_lock;
 	struct rwsem_waiter *first_waiter __guarded_by(&wait_lock);
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+	atomic_t max_waiter_prio;
+#endif
 #ifdef CONFIG_DEBUG_RWSEMS
 	void *magic;
 #endif
@@ -101,12 +106,19 @@ static inline void rwsem_assert_held_write_nolockdep(const struct rw_semaphore *
 #define __RWSEM_OPT_INIT(lockname)
 #endif
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+# define __RWSEM_SCHED_BOOST_INIT .max_waiter_prio = ATOMIC_INIT(MAX_PRIO),
+#else
+# define __RWSEM_SCHED_BOOST_INIT
+#endif
+
 #define __RWSEM_INITIALIZER(name)				\
 	{ __RWSEM_COUNT_INIT(name),				\
 	  .owner = ATOMIC_LONG_INIT(0),				\
 	  __RWSEM_OPT_INIT(name)				\
 	  .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(name.wait_lock),\
 	  .first_waiter = NULL,					\
+	  __RWSEM_SCHED_BOOST_INIT				\
 	  __RWSEM_DEBUG_INIT(name)				\
 	  __RWSEM_DEP_MAP_INIT(name) }
 
@@ -327,8 +339,24 @@ extern void up_read_non_owner(struct rw_semaphore *sem) __releases_shared(sem);
 # define down_write_nest_lock(sem, nest_lock)	down_write(sem)
 # define down_write_nested(sem, subclass)	down_write(sem)
 # define down_write_killable_nested(sem, subclass)	down_write_killable(sem)
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+void down_read_non_owner(struct rw_semaphore *sem) __acquires_shared(sem);
+void up_read_non_owner(struct rw_semaphore *sem) __releases_shared(sem);
+#else
 # define down_read_non_owner(sem)		down_read(sem)
 # define up_read_non_owner(sem)			up_read(sem)
 #endif
+#endif
+
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+extern struct static_key_true rwsem_sched_boost_key;
+static inline void rwsem_sched_boost_set(bool enable)
+{
+	if (enable)
+		static_branch_enable(&rwsem_sched_boost_key);
+	else
+		static_branch_disable(&rwsem_sched_boost_key);
+}
+#endif
 
 #endif /* _LINUX_RWSEM_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..9ce2c11aa500 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -78,6 +78,7 @@ struct rcu_node;
 struct reclaim_state;
 struct root_domain;
 struct rq;
+struct rw_semaphore;
 struct sched_attr;
 struct sched_dl_entity;
 struct seq_file;
@@ -1266,6 +1267,13 @@ struct task_struct {
 	struct rt_mutex_waiter		*pi_blocked_on;
 #endif
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+	struct rw_semaphore		*held_rwsems[CONFIG_RWSEM_SCHED_BOOST_SLOTS];
+	u8				nr_held_rwsems;
+	int				rwsem_boost_prio;
+	struct list_head		rwsem_rq_node;
+#endif
+
 	struct mutex			*blocked_on;	/* lock we're blocked on */
 	raw_spinlock_t			blocked_lock;
 
@@ -2250,6 +2258,14 @@ static inline void clear_task_blocked_on(struct task_struct *p, struct rt_mutex
 }
 #endif /* !CONFIG_PREEMPT_RT */
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+void sched_rwsem_recalc_boost(struct task_struct *p);
+#else
+static inline void sched_rwsem_recalc_boost(struct task_struct *p)
+{
+}
+#endif
+
 static __always_inline bool need_resched(void)
 {
 	return unlikely(tif_need_resched());
diff --git a/init/init_task.c b/init/init_task.c
index ce7c2b07d855..a52a08f11ffd 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -134,6 +134,10 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 		.time_slice	= RR_TIMESLICE,
 	},
 	.tasks		= LIST_HEAD_INIT(init_task.tasks),
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+	.rwsem_boost_prio = MAX_PRIO,
+	.rwsem_rq_node	= LIST_HEAD_INIT(init_task.rwsem_rq_node),
+#endif
 #ifdef CONFIG_SMP
 	.pushable_tasks	= PLIST_NODE_INIT(init_task.pushable_tasks, MAX_PRIO),
 #endif
diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks
index 1c6423aafcd4..7487cdab4e59 100644
--- a/kernel/Kconfig.locks
+++ b/kernel/Kconfig.locks
@@ -279,3 +279,18 @@ config ARCH_HAS_MMIOWB
 config MMIOWB
 	def_bool y if ARCH_HAS_MMIOWB
 	depends on SMP
+
+config RWSEM_SCHED_BOOST_SLOTS
+	int "Maximum held rw_semaphores tracked per task for priority boost"
+	range 0 16
+	default 4
+	depends on !PREEMPT_RT
+	help
+	  Number of rw_semaphore pointers recorded per task_struct so the
+	  scheduler can detect and boost holders blocking higher-priority
+	  waiters across wakeup, preemption, tick, and idle-balance hooks.
+	  Set to 0 to disable.
+
+config RWSEM_SCHED_BOOST
+	def_bool y
+	depends on RWSEM_SCHED_BOOST_SLOTS > 0
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
index b9c180ac1eee..5289bb4bf5eb 100644
--- a/kernel/locking/rwsem.c
+++ b/kernel/locking/rwsem.c
@@ -30,6 +30,80 @@
 #include <linux/hung_task.h>
 #include <trace/events/lock.h>
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+/*
+ * Record @sem in a free per-task slot for scheduler boost inspection.
+ * Trylock callers can run in IRQ/atomic contexts (e.g. BPF helpers whose
+ * deferred irq_work uses up_read_non_owner() when irqs_disabled() was true),
+ * while process-context non-owner callers use down_read_non_owner(). Guard
+ * trylocks with @check_atomic so skipped contexts never strand @sem in slots.
+ */
+static inline void rwsem_record_hold(struct rw_semaphore *sem,
+				     bool check_atomic)
+{
+	struct task_struct *curr = current;
+	u8 nr;
+	int i;
+
+	if (!static_branch_likely(&rwsem_sched_boost_key))
+		return;
+	if (unlikely(!in_task() ||
+		     (check_atomic && (irqs_disabled() || in_atomic()))))
+		return;
+
+	nr = READ_ONCE(curr->nr_held_rwsems);
+	for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+		if (!curr->held_rwsems[i]) {
+			WRITE_ONCE(curr->held_rwsems[i], sem);
+			WRITE_ONCE(curr->nr_held_rwsems, nr + 1);
+			return;
+		}
+	}
+}
+
+/*
+ * Clear @sem from held_rwsems[] under preempt_disable() BEFORE
+ * releasing sem->count so another CPU cannot acquire/free @sem after
+ * unlock while scheduler inspection still sees it in held_rwsems[].
+ */
+static inline void rwsem_record_release(struct rw_semaphore *sem)
+{
+	struct task_struct *curr = current;
+	u8 nr = READ_ONCE(curr->nr_held_rwsems);
+	int i;
+
+	if (unlikely(!in_task() || !nr))
+		return;
+
+	for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+		if (curr->held_rwsems[i] == sem) {
+			WRITE_ONCE(curr->held_rwsems[i], NULL);
+			WRITE_ONCE(curr->nr_held_rwsems, nr - 1);
+			return;
+		}
+	}
+}
+
+static inline void rwsem_sched_unboost_if_needed(void)
+{
+	if (unlikely(in_task() && current->rwsem_boost_prio < MAX_PRIO))
+		sched_rwsem_recalc_boost(current);
+}
+#else
+static inline void rwsem_record_hold(struct rw_semaphore *sem,
+				     bool check_atomic)
+{
+}
+
+static inline void rwsem_record_release(struct rw_semaphore *sem)
+{
+}
+
+static inline void rwsem_sched_unboost_if_needed(void)
+{
+}
+#endif
+
 #ifndef CONFIG_PREEMPT_RT
 #include "lock_events.h"
 
@@ -324,6 +398,9 @@ void __init_rwsem(struct rw_semaphore *sem, const char *name,
 	scoped_guard (raw_spinlock_init, &sem->wait_lock) {
 		sem->first_waiter = NULL;
 	}
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+	atomic_set(&sem->max_waiter_prio, MAX_PRIO);
+#endif
 #ifdef CONFIG_RWSEM_SPIN_ON_OWNER
 	osq_lock_init(&sem->osq);
 #endif
@@ -349,6 +426,50 @@ enum rwsem_wake_type {
 	RWSEM_WAKE_READ_OWNED	/* Waker thread holds the read lock */
 };
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+static inline int rwsem_clamp_waiter_prio(struct task_struct *tsk)
+{
+	/* Cap boost at CFS highest priority (nice -20) to protect RT tasks. */
+	return max_t(int, READ_ONCE(tsk->prio), MAX_RT_PRIO);
+}
+
+static inline void rwsem_update_max_waiter_prio(struct rw_semaphore *sem,
+						struct task_struct *waiter_task)
+{
+	int prio = rwsem_clamp_waiter_prio(waiter_task);
+
+	lockdep_assert_held(&sem->wait_lock);
+	if (prio < atomic_read(&sem->max_waiter_prio))
+		atomic_set(&sem->max_waiter_prio, prio);
+}
+
+static void rwsem_recalc_max_waiter_prio(struct rw_semaphore *sem)
+{
+	struct rwsem_waiter *first = sem->first_waiter;
+	struct rwsem_waiter *w;
+	int prio = MAX_PRIO;
+
+	lockdep_assert_held(&sem->wait_lock);
+	if (first) {
+		prio = min(prio, rwsem_clamp_waiter_prio(first->task));
+		list_for_each_entry(w, &first->list, list)
+			prio = min(prio, rwsem_clamp_waiter_prio(w->task));
+	}
+
+	if (atomic_read(&sem->max_waiter_prio) != prio)
+		atomic_set(&sem->max_waiter_prio, prio);
+}
+#else
+static inline void rwsem_update_max_waiter_prio(struct rw_semaphore *sem,
+						struct task_struct *waiter_task)
+{
+}
+
+static inline void rwsem_recalc_max_waiter_prio(struct rw_semaphore *sem)
+{
+}
+#endif
+
 /*
  * The typical HZ value is either 250 or 1000. So set the minimum waiting
  * time to at least 4ms or 1 jiffy (if it is higher than 4ms) in the wait
@@ -394,9 +515,12 @@ static inline bool
 rwsem_del_waiter(struct rw_semaphore *sem, struct rwsem_waiter *waiter)
 {
 	lockdep_assert_held(&sem->wait_lock);
-	if (__rwsem_del_waiter(sem, waiter))
+	if (__rwsem_del_waiter(sem, waiter)) {
+		rwsem_recalc_max_waiter_prio(sem);
 		return true;
+	}
 	atomic_long_andnot(RWSEM_FLAG_HANDOFF | RWSEM_FLAG_WAITERS, &sem->count);
+	rwsem_recalc_max_waiter_prio(sem);
 	return false;
 }
 
@@ -542,6 +666,8 @@ static void rwsem_mark_wake(struct rw_semaphore *sem,
 			break;
 	} while ((waiter = next) != NULL);
 
+	rwsem_recalc_max_waiter_prio(sem);
+
 	adjustment = woken * RWSEM_READER_BIAS - adjustment;
 	lockevent_cond_inc(rwsem_wake_reader, woken);
 
@@ -683,6 +809,7 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,
 	 * success.
 	 */
 	__rwsem_del_waiter(sem, waiter);
+	rwsem_recalc_max_waiter_prio(sem);
 
 	rwsem_set_owner(sem);
 	return true;
@@ -1081,6 +1208,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
 	} else {
 		list_add_tail(&waiter.list, &first->list);
 	}
+	rwsem_update_max_waiter_prio(sem, current);
 
 	/* we're now waiting on the lock, but no longer actively locking */
 	count = atomic_long_add_return(adjustment, &sem->count);
@@ -1177,6 +1305,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
 		sem->first_waiter = &waiter;
 		atomic_long_or(RWSEM_FLAG_WAITERS, &sem->count);
 	}
+	rwsem_update_max_waiter_prio(sem, current);
 
 	/* wait until we successfully acquire the lock */
 	set_current_state(state);
@@ -1376,7 +1505,7 @@ static inline int __down_write_trylock(struct rw_semaphore *sem)
 /*
  * unlock after reading
  */
-static inline void __up_read(struct rw_semaphore *sem)
+static inline void __up_read_common(struct rw_semaphore *sem, bool track_owner)
 {
 	long tmp;
 
@@ -1384,6 +1513,8 @@ static inline void __up_read(struct rw_semaphore *sem)
 	DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem);
 
 	preempt_disable();
+	if (track_owner)
+		rwsem_record_release(sem);
 	rwsem_clear_reader_owned(sem);
 	tmp = atomic_long_add_return_release(-RWSEM_READER_BIAS, &sem->count);
 	DEBUG_RWSEMS_WARN_ON(tmp < 0, sem);
@@ -1395,6 +1526,20 @@ static inline void __up_read(struct rw_semaphore *sem)
 		rwsem_wake(sem);
 	}
 	preempt_enable();
+
+	/* Drop boost after waking waiters and re-enabling preemption. */
+	if (track_owner)
+		rwsem_sched_unboost_if_needed();
+}
+
+static inline void __up_read(struct rw_semaphore *sem)
+{
+	__up_read_common(sem, true);
+}
+
+static inline void __up_read_non_owner(struct rw_semaphore *sem)
+{
+	__up_read_common(sem, false);
 }
 
 /*
@@ -1413,6 +1558,7 @@ static inline void __up_write(struct rw_semaphore *sem)
 			    !rwsem_test_oflags(sem, RWSEM_NONSPINNABLE), sem);
 
 	preempt_disable();
+	rwsem_record_release(sem);
 	rwsem_clear_owner(sem);
 	tmp = atomic_long_fetch_add_release(-RWSEM_WRITER_LOCKED, &sem->count);
 	if (unlikely(tmp & RWSEM_FLAG_WAITERS)) {
@@ -1420,6 +1566,7 @@ static inline void __up_write(struct rw_semaphore *sem)
 		rwsem_wake(sem);
 	}
 	preempt_enable();
+	rwsem_sched_unboost_if_needed();
 }
 
 /*
@@ -1517,11 +1664,16 @@ static inline int __down_read_trylock(struct rw_semaphore *sem)
 	return rwbase_read_trylock(&sem->rwbase);
 }
 
-static inline void __up_read(struct rw_semaphore *sem)
+static inline void __up_read_non_owner(struct rw_semaphore *sem)
 {
 	rwbase_read_unlock(&sem->rwbase, TASK_NORMAL);
 }
 
+static inline void __up_read(struct rw_semaphore *sem)
+{
+	__up_read_non_owner(sem);
+}
+
 static inline void __sched __down_write(struct rw_semaphore *sem)
 {
 	rwbase_write_lock(&sem->rwbase, TASK_UNINTERRUPTIBLE);
@@ -1574,6 +1726,7 @@ void __sched down_read(struct rw_semaphore *sem)
 	rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_);
 
 	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	rwsem_record_hold(sem, false);
 }
 EXPORT_SYMBOL(down_read);
 
@@ -1588,6 +1741,7 @@ int __sched down_read_interruptible(struct rw_semaphore *sem)
 		return -EINTR;
 	}
 
+	rwsem_record_hold(sem, false);
 	return 0;
 }
 EXPORT_SYMBOL(down_read_interruptible);
@@ -1603,6 +1757,7 @@ int __sched down_read_killable(struct rw_semaphore *sem)
 		return -EINTR;
 	}
 
+	rwsem_record_hold(sem, false);
 	return 0;
 }
 EXPORT_SYMBOL(down_read_killable);
@@ -1615,8 +1770,10 @@ int down_read_trylock(struct rw_semaphore *sem)
 {
 	int ret = __down_read_trylock(sem);
 
-	if (ret == 1)
+	if (ret == 1) {
 		rwsem_acquire_read(&sem->dep_map, 0, 1, _RET_IP_);
+		rwsem_record_hold(sem, true);
+	}
 	return ret;
 }
 EXPORT_SYMBOL(down_read_trylock);
@@ -1630,6 +1787,7 @@ void __sched down_write(struct rw_semaphore *sem)
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, 0, 0, _RET_IP_);
 	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	rwsem_record_hold(sem, false);
 }
 EXPORT_SYMBOL(down_write);
 
@@ -1648,6 +1806,7 @@ int __sched down_write_killable(struct rw_semaphore *sem)
 		return -EINTR;
 	}
 
+	rwsem_record_hold(sem, false);
 	return 0;
 }
 EXPORT_SYMBOL(down_write_killable);
@@ -1660,8 +1819,10 @@ int down_write_trylock(struct rw_semaphore *sem)
 {
 	int ret = __down_write_trylock(sem);
 
-	if (ret == 1)
+	if (ret == 1) {
 		rwsem_acquire(&sem->dep_map, 0, 1, _RET_IP_);
+		rwsem_record_hold(sem, true);
+	}
 
 	return ret;
 }
@@ -1708,6 +1869,7 @@ void down_read_nested(struct rw_semaphore *sem, int subclass)
 	might_sleep();
 	rwsem_acquire_read(&sem->dep_map, subclass, 0, _RET_IP_);
 	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	rwsem_record_hold(sem, false);
 }
 EXPORT_SYMBOL(down_read_nested);
 
@@ -1722,6 +1884,7 @@ int down_read_killable_nested(struct rw_semaphore *sem, int subclass)
 		return -EINTR;
 	}
 
+	rwsem_record_hold(sem, false);
 	return 0;
 }
 EXPORT_SYMBOL(down_read_killable_nested);
@@ -1732,30 +1895,17 @@ void _down_write_nest_lock(struct rw_semaphore *sem, struct lockdep_map *nest)
 	might_sleep();
 	rwsem_acquire_nest(&sem->dep_map, 0, 0, nest, _RET_IP_);
 	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	rwsem_record_hold(sem, false);
 }
 EXPORT_SYMBOL(_down_write_nest_lock);
 
-void down_read_non_owner(struct rw_semaphore *sem)
-	__no_context_analysis
-{
-	might_sleep();
-	__down_read(sem);
-	/*
-	 * The owner value for a reader-owned lock is mostly for debugging
-	 * purpose only and is not critical to the correct functioning of
-	 * rwsem. So it is perfectly fine to set it in a preempt-enabled
-	 * context here.
-	 */
-	__rwsem_set_reader_owned(sem, NULL);
-}
-EXPORT_SYMBOL(down_read_non_owner);
-
 void down_write_nested(struct rw_semaphore *sem, int subclass)
 	__no_context_analysis
 {
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, subclass, 0, _RET_IP_);
 	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	rwsem_record_hold(sem, false);
 }
 EXPORT_SYMBOL(down_write_nested);
 
@@ -1771,15 +1921,34 @@ int __sched down_write_killable_nested(struct rw_semaphore *sem, int subclass)
 		return -EINTR;
 	}
 
+	rwsem_record_hold(sem, false);
 	return 0;
 }
 EXPORT_SYMBOL(down_write_killable_nested);
 
+#endif
+
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RWSEM_SCHED_BOOST)
+void down_read_non_owner(struct rw_semaphore *sem)
+	__no_context_analysis
+{
+	might_sleep();
+	__down_read(sem);
+	/*
+	 * The owner value for a reader-owned lock is mostly for debugging
+	 * purpose only and is not critical to the correct functioning of
+	 * rwsem. So it is perfectly fine to set it in a preempt-enabled
+	 * context here.
+	 */
+	__rwsem_set_reader_owned(sem, NULL);
+}
+EXPORT_SYMBOL(down_read_non_owner);
+
 void up_read_non_owner(struct rw_semaphore *sem)
 	__no_context_analysis
 {
 	DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem);
-	__up_read(sem);
+	__up_read_non_owner(sem);
 }
 EXPORT_SYMBOL(up_read_non_owner);
 
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5ed660f099a5..5737fe55d52c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -39,6 +39,7 @@
 #include <linux/sched/nohz.h>
 #include <linux/sched/rseq_api.h>
 #include <linux/sched/rt.h>
+#include <linux/rwsem.h>
 
 #include <linux/context_tracking.h>
 #include <linux/cpuset.h>
@@ -1534,12 +1535,108 @@ int tg_nop(struct task_group *tg, void *data)
 }
 #endif
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+DEFINE_STATIC_KEY_TRUE(rwsem_sched_boost_key);
+
+static int __init rwsem_sched_boost_cmdline(char *str)
+{
+	bool val;
+
+	if (str && !kstrtobool(str, &val) && !val)
+		static_branch_disable(&rwsem_sched_boost_key);
+	return 1;
+}
+__setup("rwsem_sched_boost=", rwsem_sched_boost_cmdline);
+
+static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
+{
+	return p->rwsem_boost_prio < MAX_PRIO;
+}
+
+static inline void sched_rwsem_fork_init(struct task_struct *p)
+{
+	memset(p->held_rwsems, 0, sizeof(p->held_rwsems));
+	p->nr_held_rwsems = 0;
+	p->rwsem_boost_prio = MAX_PRIO;
+	INIT_LIST_HEAD(&p->rwsem_rq_node);
+}
+
+static int sched_rwsem_compute_boost_prio(struct task_struct *p)
+{
+	int best = MAX_PRIO;
+	int i;
+
+	if (!READ_ONCE(p->nr_held_rwsems))
+		return MAX_PRIO;
+
+	for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+		struct rw_semaphore *sem = READ_ONCE(p->held_rwsems[i]);
+		int wprio;
+
+		if (!sem)
+			continue;
+		wprio = atomic_read(&sem->max_waiter_prio);
+		if (wprio < best)
+			best = wprio;
+	}
+
+	if (best >= MAX_PRIO)
+		return MAX_PRIO;
+	if (best < MAX_RT_PRIO)
+		best = MAX_RT_PRIO;
+	if (best >= p->static_prio && !task_has_idle_policy(p))
+		return MAX_PRIO;
+
+	return best;
+}
+
+/*
+ * Recompute @p's effective priority and EEVDF weight from its held rwsems.
+ * Callers must hold @p's rq->lock (or pi_lock while @p is WAKING) and ensure
+ * @p cannot concurrently execute up_read() + kfree(sem) on another CPU.
+ */
+bool sched_rwsem_check_boost(struct task_struct *p, bool update_load)
+{
+	int new_boost;
+
+	if (p->sched_class != &fair_sched_class)
+		return false;
+
+	new_boost = sched_rwsem_compute_boost_prio(p);
+	if (new_boost == p->rwsem_boost_prio)
+		return false;
+
+	p->rwsem_boost_prio = new_boost;
+	if (!rt_or_dl_prio(p->prio))
+		p->prio = sched_rwsem_effective_prio(p);
+	set_load_weight(p, update_load);
+	return true;
+}
+
+void sched_rwsem_recalc_boost(struct task_struct *p)
+{
+	guard(task_rq_lock)(p);
+
+	if (p->sched_class == &fair_sched_class) {
+		update_rq_clock(task_rq(p));
+		sched_rwsem_check_boost(p, !p->se.sched_delayed);
+	}
+}
+#else
+static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
+{
+	return false;
+}
+
+static inline void sched_rwsem_fork_init(struct task_struct *p) { }
+#endif
+
 void set_load_weight(struct task_struct *p, bool update_load)
 {
-	int prio = p->static_prio - MAX_RT_PRIO;
+	int prio = sched_rwsem_cfs_prio(p) - MAX_RT_PRIO;
 	struct load_weight lw;
 
-	if (task_has_idle_policy(p)) {
+	if (task_has_idle_policy(p) && !sched_rwsem_is_boosted(p)) {
 		lw.weight = scale_load(WEIGHT_IDLEPRIO);
 		lw.inv_weight = WMULT_IDLEPRIO;
 	} else {
@@ -4888,6 +4985,7 @@ int sched_fork(u64 clone_flags, struct task_struct *p)
 	 * Make sure we do not leak PI boosting priority to the child.
 	 */
 	p->prio = current->normal_prio;
+	sched_rwsem_fork_init(p);
 
 	uclamp_fork(p);
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 220927453a88..98bc9029f9ae 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4344,4 +4344,47 @@ void sched_push_current_non_preferred_cpu(struct rq *rq);
 static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { }
 #endif
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+static inline int sched_rwsem_prio(struct task_struct *p, int base_prio)
+{
+	/*
+	 * SCHED_IDLE tasks carry static_prio == NICE_TO_PRIO(0) despite
+	 * WEIGHT_IDLEPRIO; use rwsem_boost_prio directly so low-priority
+	 * waiters do not boost a SCHED_IDLE holder up to nice 0.
+	 */
+	if (task_has_idle_policy(p))
+		return p->rwsem_boost_prio < MAX_PRIO ? p->rwsem_boost_prio
+						      : base_prio;
+	return min(base_prio, p->rwsem_boost_prio);
+}
+
+static inline int sched_rwsem_cfs_prio(struct task_struct *p)
+{
+	return sched_rwsem_prio(p, p->static_prio);
+}
+
+static inline int sched_rwsem_effective_prio(struct task_struct *p)
+{
+	return sched_rwsem_prio(p, p->normal_prio);
+}
+
+bool sched_rwsem_check_boost(struct task_struct *p, bool update_load);
+#else
+static inline int sched_rwsem_cfs_prio(const struct task_struct *p)
+{
+	return p->static_prio;
+}
+
+static inline int sched_rwsem_effective_prio(const struct task_struct *p)
+{
+	return p->normal_prio;
+}
+
+static inline bool sched_rwsem_check_boost(struct task_struct *p,
+					   bool update_load)
+{
+	return false;
+}
+#endif
+
 #endif /* _KERNEL_SCHED_SCHED_H */
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 0ec401af80d1..d9e3c61fcfba 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -58,7 +58,7 @@ static int effective_prio(struct task_struct *p)
 	 * to the normal priority:
 	 */
 	if (!rt_or_dl_prio(p->prio))
-		return p->normal_prio;
+		return sched_rwsem_effective_prio(p);
 	return p->prio;
 }
 
-- 
2.56.1.750.gf70dc682cf-goog


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters
  2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang
  2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
	Waiman Long, Juri Lelli, Vincent Guittot
  Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang

Keep queued fair tasks holding rwsems on rq->held_rwsem_tasks and check
waiter demand across scheduler wakeup, preemption, tick, and idle
load-balance pull paths.

Uncontended lock/unlock paths never touch rq->held_rwsem_tasks; a task
is linked only if preempted, enqueued, or crossing a tick while holding
an rwsem. Thus rq->held_rwsem_tasks stays empty for the vast majority of
ticks and idle pulls, exiting immediately with zero iteration or
max_waiter_prio lookups.

When a fair task holds a contended rwsem (max_waiter_prio < task prio),
temporarily elevate its effective priority and EEVDF weight without
altering user-visible static_prio/nice, and restore it on unlock or
lazily unlink uncontended unlocks on tick.

Signed-off-by: Peng Huang <penghuang@google.com>
---
 kernel/sched/core.c  | 109 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/fair.c  |  51 ++++++++++++++++++++
 kernel/sched/sched.h |  27 +++++++++++
 3 files changed, 185 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5737fe55d52c..2ead771c703a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1561,7 +1561,12 @@ static inline void sched_rwsem_fork_init(struct task_struct *p)
 	INIT_LIST_HEAD(&p->rwsem_rq_node);
 }
 
-static int sched_rwsem_compute_boost_prio(struct task_struct *p)
+static inline void sched_rwsem_rq_init(struct rq *rq)
+{
+	INIT_LIST_HEAD(&rq->held_rwsem_tasks);
+}
+
+int sched_rwsem_compute_boost_prio(struct task_struct *p)
 {
 	int best = MAX_PRIO;
 	int i;
@@ -1613,10 +1618,88 @@ bool sched_rwsem_check_boost(struct task_struct *p, bool update_load)
 	return true;
 }
 
+static inline void sched_rwsem_track_on_rq(struct rq *rq, struct task_struct *p)
+{
+	if (list_empty(&p->rwsem_rq_node))
+		list_add(&p->rwsem_rq_node, &rq->held_rwsem_tasks);
+}
+
+/*
+ * Safe to inspect p->held_rwsems[] here because ttwu() holds p->pi_lock
+ * after smp_cond_load_acquire(&p->on_cpu, !VAL), so @p is off-CPU and
+ * cannot concurrently run up_read()/kfree(sem).
+ */
+static inline void sched_rwsem_wakeup_boost(struct task_struct *p)
+{
+	if (READ_ONCE(p->nr_held_rwsems))
+		sched_rwsem_check_boost(p, false);
+}
+
+static inline void sched_rwsem_enqueue_locked(struct rq *rq,
+					      struct task_struct *p)
+{
+	if (!READ_ONCE(p->nr_held_rwsems) ||
+	    p->sched_class != &fair_sched_class)
+		return;
+
+	sched_rwsem_check_boost(p, false);
+	sched_rwsem_track_on_rq(rq, p);
+}
+
+static inline void sched_rwsem_on_preempt_locked(struct rq *rq,
+						 struct task_struct *prev)
+{
+	if (!READ_ONCE(prev->nr_held_rwsems) || task_rq(prev) != rq ||
+	    !task_on_rq_queued(prev) || prev->se.sched_delayed ||
+	    prev->sched_class != &fair_sched_class)
+		return;
+
+	sched_rwsem_track_on_rq(rq, prev);
+	sched_rwsem_check_boost(prev, true);
+}
+
+static inline void sched_rwsem_tick_locked(struct rq *rq)
+{
+	struct task_struct *curr = rq->curr;
+	struct task_struct *p, *tmp;
+	bool need_resched = false;
+
+	if (READ_ONCE(curr->nr_held_rwsems) &&
+	    curr->sched_class == &fair_sched_class &&
+	    task_rq(curr) == rq && task_on_rq_queued(curr) &&
+	    !curr->se.sched_delayed) {
+		sched_rwsem_track_on_rq(rq, curr);
+		sched_rwsem_check_boost(curr, true);
+	}
+
+	/*
+	 * Lazily unlink tasks whose unlock fastpath skipped rq->lock because
+	 * they were never boosted (!nr_held_rwsems short-circuits sem access),
+	 * or if proxy/delayed-dequeue movement left a stale rq link.
+	 */
+	list_for_each_entry_safe(p, tmp, &rq->held_rwsem_tasks, rwsem_rq_node) {
+		if (!READ_ONCE(p->nr_held_rwsems) ||
+		    task_rq(p) != rq || !task_on_rq_queued(p)) {
+			sched_rwsem_dequeue_locked(p);
+			continue;
+		}
+		if (p == curr || task_on_cpu(rq, p) || p->se.sched_delayed)
+			continue;
+		sched_rwsem_check_boost(p, true);
+		if (p->rwsem_boost_prio < curr->prio)
+			need_resched = true;
+	}
+
+	if (need_resched)
+		resched_curr(rq);
+}
+
 void sched_rwsem_recalc_boost(struct task_struct *p)
 {
 	guard(task_rq_lock)(p);
 
+	if (!READ_ONCE(p->nr_held_rwsems))
+		sched_rwsem_dequeue_locked(p);
 	if (p->sched_class == &fair_sched_class) {
 		update_rq_clock(task_rq(p));
 		sched_rwsem_check_boost(p, !p->se.sched_delayed);
@@ -1629,6 +1712,16 @@ static inline bool sched_rwsem_is_boosted(const struct task_struct *p)
 }
 
 static inline void sched_rwsem_fork_init(struct task_struct *p) { }
+static inline void sched_rwsem_rq_init(struct rq *rq) { }
+static inline void sched_rwsem_wakeup_boost(struct task_struct *p) { }
+
+static inline void sched_rwsem_enqueue_locked(struct rq *rq,
+					      struct task_struct *p) { }
+
+static inline void sched_rwsem_on_preempt_locked(struct rq *rq,
+						 struct task_struct *prev) { }
+
+static inline void sched_rwsem_tick_locked(struct rq *rq) { }
 #endif
 
 void set_load_weight(struct task_struct *p, bool update_load)
@@ -2284,6 +2377,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
+	sched_rwsem_enqueue_locked(rq, p);
+
 	/*
 	 * Can be before ->enqueue_task() because uclamp considers the
 	 * ENQUEUE_DELAYED task before its ->sched_delayed gets cleared
@@ -2323,7 +2418,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	 * and mark the task ->sched_delayed.
 	 */
 	uclamp_rq_dec(rq, p);
-	return p->sched_class->dequeue_task(rq, p, flags);
+	if (p->sched_class->dequeue_task(rq, p, flags)) {
+		sched_rwsem_dequeue_locked(p);
+		return true;
+	}
+	return false;
 }
 
 void activate_task(struct rq *rq, struct task_struct *p, int flags)
@@ -4566,6 +4665,8 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		 */
 		smp_cond_load_acquire(&p->on_cpu, !VAL);
 
+		sched_rwsem_wakeup_boost(p);
+
 		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
 		if (task_cpu(p) != cpu) {
 			wake_flags |= WF_MIGRATED;
@@ -5957,6 +6058,7 @@ void sched_tick(void)
 		resched_curr(rq);
 
 	donor->sched_class->task_tick(rq, donor, 0);
+	sched_rwsem_tick_locked(rq);
 	if (sched_feat(LATENCY_WARN))
 		resched_latency = cpu_resched_latency(rq);
 	calc_global_load_tick(rq);
@@ -7391,6 +7493,8 @@ static void __sched notrace __schedule(int sched_mode)
 		switch_count = &prev->nvcsw;
 	}
 
+	sched_rwsem_on_preempt_locked(rq, prev);
+
 pick_again:
 	assert_balance_callbacks_empty(rq);
 	next = pick_next_task(rq, &rf);
@@ -9125,6 +9229,7 @@ void __init sched_init(void)
 		rq->max_idle_balance_cost = sysctl_sched_migration_cost;
 
 		INIT_LIST_HEAD(&rq->cfs_tasks);
+		sched_rwsem_rq_init(rq);
 
 		rq_attach_root(rq, &def_root_domain);
 #ifdef CONFIG_NO_HZ_COMMON
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0c8af1cac2d5..59ce6321a265 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11312,6 +11312,53 @@ static struct task_struct *detach_one_task(struct lb_env *env)
 	return NULL;
 }
 
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+/*
+ * On idle pull, prioritize migrating a fair task holding a contended rwsem
+ * so a blocked high-priority waiter is not starved on a busy little core.
+ */
+static int detach_boosted_rwsem_task(struct lb_env *env)
+{
+	struct task_struct *p, *tmp;
+
+	if (env->idle != CPU_IDLE && env->idle != CPU_NEWLY_IDLE)
+		return 0;
+
+	list_for_each_entry_safe(p, tmp, &env->src_rq->held_rwsem_tasks,
+				 rwsem_rq_node) {
+		if (env->src_rq->nr_running <= 1)
+			break;
+		if (task_rq(p) != env->src_rq || !task_on_rq_queued(p)) {
+			sched_rwsem_dequeue_locked(p);
+			continue;
+		}
+		if (task_on_cpu(env->src_rq, p) || p->se.sched_delayed)
+			continue;
+		if (sched_rwsem_compute_boost_prio(p) >= MAX_PRIO)
+			continue;
+		if (!can_migrate_task(p, env))
+			continue;
+
+		/*
+		 * enqueue_task() on dst_rq updates the boosted weight
+		 * via sched_rwsem_enqueue_locked(), avoiding a redundant
+		 * reweight_eevdf() on src_rq before detaching.
+		 */
+		detach_task(p, env);
+		list_add(&p->se.group_node, &env->tasks);
+		env->imbalance = 0;
+		return 1;
+	}
+
+	return 0;
+}
+#else
+static inline int detach_boosted_rwsem_task(struct lb_env *env)
+{
+	return 0;
+}
+#endif
+
 /*
  * detach_tasks() -- tries to detach up to imbalance load/util/tasks from
  * busiest_rq, as part of a balancing operation within domain "sd".
@@ -11339,6 +11386,10 @@ static int detach_tasks(struct lb_env *env)
 	if (env->imbalance <= 0)
 		return 0;
 
+	detached = detach_boosted_rwsem_task(env);
+	if (detached)
+		return detached;
+
 	while (!list_empty(tasks)) {
 		/*
 		 * We don't want to steal all, otherwise we may be treated likewise,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 98bc9029f9ae..58e53980e71b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1317,6 +1317,13 @@ struct rq {
 	int			online;
 
 	struct list_head cfs_tasks;
+#ifdef CONFIG_RWSEM_SCHED_BOOST
+	/*
+	 * Queued fair tasks holding an rwsem; protected by rq->lock and
+	 * unlinked on dequeue_task() (incl. migration and TASK_DEAD).
+	 */
+	struct list_head	held_rwsem_tasks;
+#endif
 
 	struct sched_avg	avg_rt;
 	struct sched_avg	avg_dl;
@@ -4368,7 +4375,18 @@ static inline int sched_rwsem_effective_prio(struct task_struct *p)
 	return sched_rwsem_prio(p, p->normal_prio);
 }
 
+int sched_rwsem_compute_boost_prio(struct task_struct *p);
 bool sched_rwsem_check_boost(struct task_struct *p, bool update_load);
+
+/*
+ * Remove @p when it leaves an rq (sleep, migration, class change,
+ * TASK_DEAD via DEQUEUE_SPECIAL, or stale cross-rq cleanup).
+ */
+static inline void sched_rwsem_dequeue_locked(struct task_struct *p)
+{
+	if (!list_empty(&p->rwsem_rq_node))
+		list_del_init(&p->rwsem_rq_node);
+}
 #else
 static inline int sched_rwsem_cfs_prio(const struct task_struct *p)
 {
@@ -4380,11 +4398,20 @@ static inline int sched_rwsem_effective_prio(const struct task_struct *p)
 	return p->normal_prio;
 }
 
+static inline int sched_rwsem_compute_boost_prio(struct task_struct *p)
+{
+	return MAX_PRIO;
+}
+
 static inline bool sched_rwsem_check_boost(struct task_struct *p,
 					   bool update_load)
 {
 	return false;
 }
+
+static inline void sched_rwsem_dequeue_locked(struct task_struct *p)
+{
+}
 #endif
 
 #endif /* _KERNEL_SCHED_SCHED_H */
-- 
2.56.1.750.gf70dc682cf-goog


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost
  2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
  2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
@ 2026-11-19 22:49 ` Peng Huang
  2 siblings, 0 replies; 4+ messages in thread
From: Peng Huang @ 2026-11-19 22:49 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Will Deacon, Boqun Feng,
	Waiman Long, Juri Lelli, Vincent Guittot
  Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak, linux-kernel, Peng Huang

Add KUnit tests covering per-task rwsem slot tracking, interleaved
unlock order, slot overflow fallback, write-lock downgrade, wakeup
boost for both reader and writer holders, preemption priority-inversion
A/B latency, false-promotion checks, and fastpath overhead measurement.

Signed-off-by: Peng Huang <penghuang@google.com>
---
 kernel/locking/Makefile                 |   1 +
 kernel/locking/rwsem_sched_boost_test.c | 504 ++++++++++++++++++++++++
 lib/Kconfig.debug                       |  10 +
 3 files changed, 515 insertions(+)
 create mode 100644 kernel/locking/rwsem_sched_boost_test.c

diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile
index cee1901d4cff..3df6ae1f1707 100644
--- a/kernel/locking/Makefile
+++ b/kernel/locking/Makefile
@@ -37,4 +37,5 @@ obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o
 obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o
 obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o
 obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
+obj-$(CONFIG_RWSEM_SCHED_BOOST_KUNIT_TEST) += rwsem_sched_boost_test.o
 obj-$(CONFIG_LOCK_EVENT_COUNTS) += lock_events.o
diff --git a/kernel/locking/rwsem_sched_boost_test.c b/kernel/locking/rwsem_sched_boost_test.c
new file mode 100644
index 000000000000..d60d2a87237a
--- /dev/null
+++ b/kernel/locking/rwsem_sched_boost_test.c
@@ -0,0 +1,504 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit tests for rw_semaphore per-task hold tracking and scheduler
+ * priority boosting.
+ */
+
+#include <kunit/test.h>
+#include <linux/atomic.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
+#include <linux/ktime.h>
+#include <linux/rwsem.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <uapi/linux/sched/types.h>
+
+static bool task_holds_rwsem(const struct task_struct *p,
+			     const struct rw_semaphore *sem)
+{
+	int i;
+
+	for (i = 0; i < CONFIG_RWSEM_SCHED_BOOST_SLOTS; i++) {
+		if (READ_ONCE(p->held_rwsems[i]) == sem)
+			return true;
+	}
+	return false;
+}
+
+static void test_basic_hold_and_release(struct kunit *test)
+{
+	DECLARE_RWSEM(sem);
+
+	KUNIT_ASSERT_EQ(test, current->nr_held_rwsems, 0);
+	KUNIT_ASSERT_EQ(test, atomic_read(&sem.max_waiter_prio), MAX_PRIO);
+
+	down_read(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+	up_read(&sem);
+
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+	KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem));
+
+	KUNIT_ASSERT_EQ(test, down_read_trylock(&sem), 1);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	up_read(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+
+	down_write(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+	up_write(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+
+	KUNIT_ASSERT_EQ(test, down_write_trylock(&sem), 1);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	up_write(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+static void test_interleaved_unlock(struct kunit *test)
+{
+	DECLARE_RWSEM(sem_a);
+	DECLARE_RWSEM(sem_b);
+	DECLARE_RWSEM(sem_c);
+
+	down_read(&sem_a);
+	down_read(&sem_b);
+	down_read(&sem_c);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems,
+			min(3, CONFIG_RWSEM_SCHED_BOOST_SLOTS));
+
+	/* Release out of acquisition order: B, A, C. */
+	up_read(&sem_b);
+	KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem_b));
+	up_read(&sem_a);
+	KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sem_a));
+	up_read(&sem_c);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+static void test_overflow_graceful(struct kunit *test)
+{
+	const int total = CONFIG_RWSEM_SCHED_BOOST_SLOTS + 3;
+	struct rw_semaphore *sems;
+	int i;
+
+	sems = kunit_kcalloc(test, total, sizeof(*sems), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, sems);
+
+	for (i = 0; i < total; i++)
+		init_rwsem(&sems[i]);
+
+	for (i = 0; i < total; i++)
+		down_read(&sems[i]);
+
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems,
+			CONFIG_RWSEM_SCHED_BOOST_SLOTS);
+
+	for (i = 0; i < total; i++)
+		up_read(&sems[i]);
+
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+	for (i = 0; i < total; i++)
+		KUNIT_EXPECT_FALSE(test, task_holds_rwsem(current, &sems[i]));
+}
+
+static void test_downgrade_write(struct kunit *test)
+{
+	DECLARE_RWSEM(sem);
+
+	down_write(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+
+	downgrade_write(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 1);
+	KUNIT_EXPECT_TRUE(test, task_holds_rwsem(current, &sem));
+
+	up_read(&sem);
+	KUNIT_EXPECT_EQ(test, current->nr_held_rwsems, 0);
+}
+
+struct boost_ctx {
+	struct rw_semaphore sem;
+	struct completion reader_locked;
+	struct completion reader_done;
+	bool holder_is_writer;
+	bool holder_is_idle;
+	int waiter_nice;
+	int observed_boost_prio;
+	int observed_prio;
+	int observed_nice;
+	int post_unlock_boost_prio;
+	int post_unlock_prio;
+	u64 reader_work_ns;
+};
+
+static int wakeup_holder_fn(void *arg)
+{
+	struct boost_ctx *ctx = arg;
+
+	if (ctx->holder_is_idle) {
+		struct sched_param sp = { .sched_priority = 0 };
+
+		sched_setscheduler_nocheck(current, SCHED_IDLE, &sp);
+	} else {
+		set_user_nice(current, 19);
+	}
+	if (ctx->holder_is_writer)
+		down_write(&ctx->sem);
+	else
+		down_read(&ctx->sem);
+	complete(&ctx->reader_locked);
+
+	/* Sleep so the higher-priority contender queues on ctx->sem first. */
+	msleep(25);
+
+	ctx->observed_boost_prio = current->rwsem_boost_prio;
+	ctx->observed_prio = current->prio;
+	ctx->observed_nice = task_nice(current);
+
+	if (ctx->holder_is_writer)
+		up_write(&ctx->sem);
+	else
+		up_read(&ctx->sem);
+	ctx->post_unlock_boost_prio = current->rwsem_boost_prio;
+	ctx->post_unlock_prio = current->prio;
+	complete(&ctx->reader_done);
+	return 0;
+}
+
+static void run_wakeup_boost_case(struct kunit *test, bool holder_is_writer,
+				  bool holder_is_idle, int waiter_nice)
+{
+	struct boost_ctx *ctx;
+	struct task_struct *holder;
+	int saved_nice = task_nice(current);
+	int exp_prio = NICE_TO_PRIO(waiter_nice);
+	int exp_post_prio = holder_is_idle ? DEFAULT_PRIO : NICE_TO_PRIO(19);
+
+	ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, ctx);
+	ctx->holder_is_writer = holder_is_writer;
+	ctx->holder_is_idle = holder_is_idle;
+	ctx->waiter_nice = waiter_nice;
+	init_rwsem(&ctx->sem);
+	init_completion(&ctx->reader_locked);
+	init_completion(&ctx->reader_done);
+
+	holder = kthread_run(wakeup_holder_fn, ctx,
+			     holder_is_writer ? "rwsem_wr_hld" : "rwsem_rd_hld");
+	KUNIT_ASSERT_FALSE(test, IS_ERR(holder));
+
+	wait_for_completion(&ctx->reader_locked);
+	set_user_nice(current, waiter_nice);
+	if (holder_is_writer) {
+		down_read(&ctx->sem);
+		up_read(&ctx->sem);
+	} else {
+		down_write(&ctx->sem);
+		up_write(&ctx->sem);
+	}
+	set_user_nice(current, saved_nice);
+
+	wait_for_completion(&ctx->reader_done);
+
+	KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, exp_prio);
+	KUNIT_EXPECT_EQ(test, ctx->observed_prio, exp_prio);
+	if (!holder_is_idle)
+		KUNIT_EXPECT_EQ(test, ctx->observed_nice, 19);
+	KUNIT_EXPECT_EQ(test, ctx->post_unlock_boost_prio, MAX_PRIO);
+	KUNIT_EXPECT_EQ(test, ctx->post_unlock_prio, exp_post_prio);
+	KUNIT_EXPECT_EQ(test, atomic_read(&ctx->sem.max_waiter_prio), MAX_PRIO);
+}
+
+static void test_contention_wakeup_boost(struct kunit *test)
+{
+	run_wakeup_boost_case(test, false, false, -10);
+}
+
+static void test_writer_holder_wakeup_boost(struct kunit *test)
+{
+	run_wakeup_boost_case(test, true, false, -10);
+}
+
+static void test_sched_idle_holder_boost_cap(struct kunit *test)
+{
+	run_wakeup_boost_case(test, false, true, 10);
+}
+
+static int hog_fn(void *arg)
+{
+	set_user_nice(current, 0);
+	while (!kthread_should_stop()) {
+		cpu_relax();
+		cond_resched();
+	}
+	return 0;
+}
+
+static int busy_reader_fn(void *arg)
+{
+	struct boost_ctx *ctx = arg;
+	u64 start, end;
+	int i;
+
+	set_user_nice(current, 19);
+	down_read(&ctx->sem);
+	start = ktime_get_ns();
+	complete(&ctx->reader_locked);
+
+	/*
+	 * Busy-loop for ~10ms of CPU work while competing against nice=0
+	 * hogs; once the nice=-10 writer blocks on ctx->sem, tick/preempt
+	 * hooks should boost this reader to prio 110.
+	 */
+	for (i = 0; i < 20; i++) {
+		u64 t0 = ktime_get_ns();
+
+		while (ktime_get_ns() - t0 < 500 * NSEC_PER_USEC)
+			cpu_relax();
+		cond_resched();
+	}
+	end = ktime_get_ns();
+
+	ctx->observed_boost_prio = current->rwsem_boost_prio;
+	ctx->observed_prio = current->prio;
+	ctx->observed_nice = task_nice(current);
+	ctx->reader_work_ns = end - start;
+
+	up_read(&ctx->sem);
+	ctx->post_unlock_boost_prio = current->rwsem_boost_prio;
+	ctx->post_unlock_prio = current->prio;
+	complete(&ctx->reader_done);
+	return 0;
+}
+
+static u64 run_one_inversion_iter(struct kunit *test, struct boost_ctx *ctx)
+{
+	struct task_struct *reader;
+	int saved_nice = task_nice(current);
+	u64 wait_ns;
+
+	reinit_completion(&ctx->reader_locked);
+	reinit_completion(&ctx->reader_done);
+
+	reader = kthread_run(busy_reader_fn, ctx, "rwsem_busy_rdr");
+	KUNIT_ASSERT_FALSE(test, IS_ERR(reader));
+
+	wait_for_completion(&ctx->reader_locked);
+	set_user_nice(current, -10);
+	wait_ns = ktime_get_ns();
+	down_write(&ctx->sem);
+	wait_ns = ktime_get_ns() - wait_ns;
+	up_write(&ctx->sem);
+	set_user_nice(current, saved_nice);
+
+	wait_for_completion(&ctx->reader_done);
+	return wait_ns;
+}
+
+static void start_hogs(struct kunit *test, struct task_struct **hogs, int count)
+{
+	int i;
+
+	for (i = 0; i < count; i++) {
+		hogs[i] = kthread_run(hog_fn, NULL, "rwsem_hog%d", i);
+		KUNIT_ASSERT_FALSE(test, IS_ERR(hogs[i]));
+	}
+}
+
+static void stop_hogs(struct task_struct **hogs, int count)
+{
+	int i;
+
+	for (i = 0; i < count; i++)
+		kthread_stop(hogs[i]);
+}
+
+static void test_preemption_inversion_latency(struct kunit *test)
+{
+	struct boost_ctx *ctx;
+	struct task_struct *hogs[4];
+	const int iters = 3;
+	u64 ctrl_sum = 0, treat_sum = 0;
+	u64 ctrl_max = 0, treat_max = 0;
+	int i;
+
+	ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, ctx);
+	init_rwsem(&ctx->sem);
+	init_completion(&ctx->reader_locked);
+	init_completion(&ctx->reader_done);
+
+	start_hogs(test, hogs, ARRAY_SIZE(hogs));
+
+	rwsem_sched_boost_set(false);
+	for (i = 0; i < iters; i++) {
+		u64 w = run_one_inversion_iter(test, ctx);
+
+		ctrl_sum += w;
+		ctrl_max = max(ctrl_max, w);
+		KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, MAX_PRIO);
+	}
+
+	rwsem_sched_boost_set(true);
+	for (i = 0; i < iters; i++) {
+		u64 w = run_one_inversion_iter(test, ctx);
+
+		treat_sum += w;
+		treat_max = max(treat_max, w);
+		KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio,
+				NICE_TO_PRIO(-10));
+		KUNIT_EXPECT_EQ(test, ctx->observed_prio, NICE_TO_PRIO(-10));
+		KUNIT_EXPECT_EQ(test, ctx->post_unlock_boost_prio, MAX_PRIO);
+	}
+
+	stop_hogs(hogs, ARRAY_SIZE(hogs));
+
+	kunit_info(test,
+		   "A/B (n=%d, 4 hogs): Control mean=%llu ms max=%llu ms | Treatment mean=%llu ms max=%llu ms\n",
+		   iters,
+		   (ctrl_sum / iters) / NSEC_PER_MSEC,
+		   ctrl_max / NSEC_PER_MSEC,
+		   (treat_sum / iters) / NSEC_PER_MSEC,
+		   treat_max / NSEC_PER_MSEC);
+
+	KUNIT_EXPECT_LT(test, treat_sum, ctrl_sum);
+}
+
+static void test_no_false_promotion_uncontended(struct kunit *test)
+{
+	struct boost_ctx *ctx;
+	struct task_struct *hogs[2];
+	struct task_struct *reader;
+
+	ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, ctx);
+	init_rwsem(&ctx->sem);
+	init_completion(&ctx->reader_locked);
+	init_completion(&ctx->reader_done);
+
+	start_hogs(test, hogs, ARRAY_SIZE(hogs));
+
+	rwsem_sched_boost_set(true);
+	reader = kthread_run(busy_reader_fn, ctx, "rwsem_fp_rdr");
+	KUNIT_ASSERT_FALSE(test, IS_ERR(reader));
+
+	wait_for_completion(&ctx->reader_locked);
+	/* No waiter blocks on ctx->sem. */
+	wait_for_completion(&ctx->reader_done);
+
+	stop_hogs(hogs, ARRAY_SIZE(hogs));
+
+	KUNIT_EXPECT_EQ(test, ctx->observed_boost_prio, MAX_PRIO);
+	KUNIT_EXPECT_EQ(test, ctx->observed_prio, NICE_TO_PRIO(19));
+	KUNIT_EXPECT_EQ(test, ctx->observed_nice, 19);
+}
+
+static noinline u64 bench_read_fastpath(struct rw_semaphore *sem, int iters)
+{
+	u64 start = ktime_get_ns();
+	int i;
+
+	for (i = 0; i < iters; i++) {
+		down_read(sem);
+		barrier();
+		up_read(sem);
+	}
+	return ktime_get_ns() - start;
+}
+
+static noinline u64 bench_write_fastpath(struct rw_semaphore *sem, int iters)
+{
+	u64 start = ktime_get_ns();
+	int i;
+
+	for (i = 0; i < iters; i++) {
+		down_write(sem);
+		barrier();
+		up_write(sem);
+	}
+	return ktime_get_ns() - start;
+}
+
+static void test_fastpath_overhead(struct kunit *test)
+{
+	DECLARE_RWSEM(outer1);
+	DECLARE_RWSEM(outer2);
+	DECLARE_RWSEM(sem);
+	const int rounds = 5;
+	const int per_round = 200000;
+	const u64 total_ops = (u64)rounds * per_round;
+	u64 r_off = 0, r_on = 0;
+	u64 w_off = 0, w_on = 0;
+	u64 n_off = 0, n_on = 0;
+	int i;
+
+	/* Warmup instruction and branch caches. */
+	bench_read_fastpath(&sem, 10000);
+	bench_write_fastpath(&sem, 10000);
+
+	for (i = 0; i < rounds; i++) {
+		rwsem_sched_boost_set(false);
+		r_off += bench_read_fastpath(&sem, per_round);
+		w_off += bench_write_fastpath(&sem, per_round);
+
+		rwsem_sched_boost_set(true);
+		r_on += bench_read_fastpath(&sem, per_round);
+		w_on += bench_write_fastpath(&sem, per_round);
+	}
+
+	/* Measure nested read fastpath with 2 outer slots already occupied. */
+	for (i = 0; i < rounds; i++) {
+		rwsem_sched_boost_set(false);
+		down_read(&outer1);
+		down_read(&outer2);
+		n_off += bench_read_fastpath(&sem, per_round);
+		up_read(&outer2);
+		up_read(&outer1);
+
+		rwsem_sched_boost_set(true);
+		down_read(&outer1);
+		down_read(&outer2);
+		n_on += bench_read_fastpath(&sem, per_round);
+		up_read(&outer2);
+		up_read(&outer1);
+	}
+
+	kunit_info(test,
+		   "Fastpath (%llu ops): read off=%llu.%02llu on=%llu.%02llu ns/op | write off=%llu.%02llu on=%llu.%02llu ns/op | nested2 read off=%llu.%02llu on=%llu.%02llu ns/op\n",
+		   total_ops,
+		   (r_off / total_ops), (r_off * 100 / total_ops) % 100,
+		   (r_on / total_ops), (r_on * 100 / total_ops) % 100,
+		   (w_off / total_ops), (w_off * 100 / total_ops) % 100,
+		   (w_on / total_ops), (w_on * 100 / total_ops) % 100,
+		   (n_off / total_ops), (n_off * 100 / total_ops) % 100,
+		   (n_on / total_ops), (n_on * 100 / total_ops) % 100);
+}
+
+static struct kunit_case rwsem_sched_boost_test_cases[] = {
+	KUNIT_CASE(test_basic_hold_and_release),
+	KUNIT_CASE(test_interleaved_unlock),
+	KUNIT_CASE(test_overflow_graceful),
+	KUNIT_CASE(test_downgrade_write),
+	KUNIT_CASE(test_contention_wakeup_boost),
+	KUNIT_CASE(test_writer_holder_wakeup_boost),
+	KUNIT_CASE(test_sched_idle_holder_boost_cap),
+	KUNIT_CASE(test_preemption_inversion_latency),
+	KUNIT_CASE(test_no_false_promotion_uncontended),
+	KUNIT_CASE(test_fastpath_overhead),
+	{}
+};
+
+static struct kunit_suite rwsem_sched_boost_test_suite = {
+	.name = "rwsem_sched_boost",
+	.test_cases = rwsem_sched_boost_test_cases,
+};
+
+kunit_test_suite(rwsem_sched_boost_test_suite);
+
+MODULE_LICENSE("GPL");
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2d22638ec13a..b3f2b8d6633e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1692,6 +1692,16 @@ config WW_MUTEX_SELFTEST
 	  Say M if you want these self tests to build as a module.
 	  Say N if you are unsure.
 
+config RWSEM_SCHED_BOOST_KUNIT_TEST
+	bool "KUnit tests for rw_semaphore scheduler priority boost"
+	depends on KUNIT && RWSEM_SCHED_BOOST
+	default KUNIT_ALL_TESTS
+	help
+	  Enable KUnit tests verifying per-task rw_semaphore hold tracking,
+	  atomic max_waiter_prio updates, contention-driven priority boosting,
+	  and zero false promotion when uncontended.
+	  If unsure, say N.
+
 config SCF_TORTURE_TEST
 	tristate "torture tests for smp_call_function*()"
 	depends on DEBUG_KERNEL
-- 
2.56.1.750.gf70dc682cf-goog


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-19 22:50 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 22:49 [RFC PATCH 0/3] rwsem: priority boost for holders blocking high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 1/3] locking/rwsem, sched: track held rwsems and waiter priorities Peng Huang
2026-11-19 22:49 ` [RFC PATCH 2/3] sched: boost rwsem holders with contended high-priority waiters Peng Huang
2026-11-19 22:49 ` [RFC PATCH 3/3] locking/rwsem: add KUnit tests for scheduler priority boost Peng Huang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®