mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,  Will Deacon <will@kernel.org>,
	Boqun Feng <boqun@kernel.org>, Waiman Long <longman@redhat.com>
Cc: linux-kernel@vger.kernel.org,
	 "David Hildenbrand (Arm)" <david@kernel.org>
Subject: Re: [PATCH] locking/rwsem: don't let a read lock steal from a queued writer
Date: Thu, 19 Nov 2026 14:14:45 +0000	[thread overview]
Message-ID: <av8DKviO1fVURLcJ@gremlin> (raw)
In-Reply-To: <20261105-rwsem-dont-steal-from-queued-writer-v1-1-275e070f426f@kernel.org>

I know the cycle is grim on workload (equally so for us), but ping on this, as
we're relying on this behaviour for a series [0].

I could re-implement as an optional mode and respin [0] if necessary, but Peter
objected [1] so hoisted the rwsem change to here :)

Thanks!

[0]: https://lore.kernel.org/linux-mm/20261110-khugepaged-latency-fix-v3-0-6eea743cf9a3@kernel.org/
[1]: https://lore.kernel.org/all/20261103115159.GK1813469@noisy.programming.kicks-ass.net/

On Thu, Nov 05, 2026 at 08:24:27PM +0000, Lorenzo Stoakes (ARM) wrote:
> Since commit 1a728dff855a ("locking/rwsem: Enable reader optimistic lock
> stealing"), a racing read lock can steal an rwsem from under a queued
> writer whose handoff timeout hasn't expired yet.
>
> This is most problematic when an rwsem read lock is released then
> reacquired in order to relieve lock contention, as instead it will simply
> have the lock handed right back to it ahead of the intended beneficiary.
>
> Naturally, this behaviour rather defeats the object of doing this kind of
> thing, so this is less than ideal.
>
> Fix this by updating the optimistic read lock stealing code in
> rwsem_down_read_slowpath() to explicitly check whether the first queued
> lock is a waiting writer, and to not allow the theft if so.
>
> When a steal is disallowed, keep the existing sem->wait_lock held when
> jumping to the queue code to avoid useless lock acquisition/release.
>
> In order to perform this check, sem->wait_lock is acquired whenever there
> are queued waiters in the steal code, rather than the previous behaviour of
> only doing so when the reader count is 1.
>
> Since failing to steal a read lock unconditionally results in acquisition
> of the sem->wait_lock, this doesn't seem likely to be a significant
> overhead.
>
> Also update the comment describing optimistic read lock acquisition
> conditions to include this newly introduced behaviour.
>
> Care is taken to ensure correct ordering of lock acquisition, lock waking
> and marking the lock reader-owned.
>
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> ---
>  kernel/locking/rwsem.c | 30 ++++++++++++++++++------------
>  1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
> index b9c180ac1eee..cdb3dd6260bf 100644
> --- a/kernel/locking/rwsem.c
> +++ b/kernel/locking/rwsem.c
> @@ -59,7 +59,8 @@
>   * A fast path reader optimistic lock stealing is supported when the rwsem
>   * is previously owned by a writer and the following conditions are met:
>   *  - rwsem is not currently writer owned
> - *  - the handoff isn't set.
> + *  - the handoff isn't set
> + *  - a writer isn't at the head of the wait queue.
>   */
>  #define RWSEM_READER_OWNED	(1UL << 0)
>  #define RWSEM_NONSPINNABLE	(1UL << 1)
> @@ -1034,31 +1035,36 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
>  	 * Reader optimistic lock stealing.
>  	 */
>  	if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) {
> -		rwsem_set_reader_owned(sem);
> -		lockevent_inc(rwsem_rlock_steal);
> -
> -		/*
> -		 * Wake up other readers in the wait queue if it is
> -		 * the first reader.
> -		 */
> -		if ((rcnt == 1) && (count & RWSEM_FLAG_WAITERS)) {
> +		if (count & RWSEM_FLAG_WAITERS) {
>  			raw_spin_lock_irq(&sem->wait_lock);
> -			if (sem->first_waiter)
> +			first = sem->first_waiter;
> +			/* A writer at the head of queue takes precedence. */
> +			if (first && first->type == RWSEM_WAITING_FOR_WRITE)
> +				goto queue_locked;
> +			rwsem_set_reader_owned(sem);
> +			/* If first reader, wake other readers in queue. */
> +			if (first && rcnt == 1)
>  				rwsem_mark_wake(sem, RWSEM_WAKE_READ_OWNED,
>  						&wake_q);
>  			raw_spin_unlock_irq(&sem->wait_lock);
> -			wake_up_q(&wake_q);
> +		} else {
> +			rwsem_set_reader_owned(sem);
>  		}
> +
> +		lockevent_inc(rwsem_rlock_steal);
> +		if (!wake_q_empty(&wake_q))
> +			wake_up_q(&wake_q);
>  		return sem;
>  	}
>
>  queue:
> +	raw_spin_lock_irq(&sem->wait_lock);
> +queue_locked:
>  	waiter.task = current;
>  	waiter.type = RWSEM_WAITING_FOR_READ;
>  	waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT;
>  	waiter.handoff_set = false;
>
> -	raw_spin_lock_irq(&sem->wait_lock);
>  	first = sem->first_waiter;
>  	if (!first) {
>  		/*
>
> ---
> base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
> change-id: 20261105-rwsem-dont-steal-from-queued-writer-04b11d9d406b
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@kernel.org>
>

--
Cheers, Lorenzo

  parent reply	other threads:[~2026-11-19 14:14 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-05 20:24 Lorenzo Stoakes (ARM)
2026-11-06  4:55 ` Waiman Long
2026-11-07 19:28   ` Lorenzo Stoakes (ARM)
2026-11-19 14:14 ` Lorenzo Stoakes (ARM) [this message]
2026-11-19 15:56 ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av8DKviO1fVURLcJ@gremlin \
    --to=ljs@kernel.org \
    --cc=boqun@kernel.org \
    --cc=david@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®