[PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Waiman Long]

Make the code more readable by replacing the atomic_cmpxchg_acquire()
by an equivalent atomic_try_cmpxchg_acquire() and change atomic_add()
to atomic_or().

For architectures that use qrwlock, I do not find one that has an
atomic_add() defined but not an atomic_or().  I guess it should be fine
by changing atomic_add() to atomic_or(). I add a comment to state that
we can change it back to atomic_add() if there is an architecture that
has a more performant atomic_add() than an atomic_or().

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/locking/qrwlock.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c
index b94f3831e963..a1fa0f636b07 100644
--- a/kernel/locking/qrwlock.c
+++ b/kernel/locking/qrwlock.c
@@ -60,18 +60,26 @@ EXPORT_SYMBOL(queued_read_lock_slowpath);
  */
 void queued_write_lock_slowpath(struct qrwlock *lock)
 {
-	int cnts;
+	int cnts = 0;
 
 	/* Put the writer into the wait queue */
 	arch_spin_lock(&lock->wait_lock);
 
 	/* Try to acquire the lock directly if no reader is present */
 	if (!atomic_read(&lock->cnts) &&
-	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
+	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
 		goto unlock;
 
-	/* Set the waiting flag to notify readers that a writer is pending */
-	atomic_add(_QW_WAITING, &lock->cnts);
+	/*
+	 * Set the waiting flag to notify readers that a writer is pending
+	 *
+	 * As only one writer who is the wait_lock owner can set the waiting
+	 * flag which will be cleared later on when acquiring the write lock,
+	 * we can easily replace atomic_or() by an atomic_add() if there is
+	 * an architecture where an atomic_add() performs better than an
+	 * atomic_or().
+	 */
+	atomic_or(_QW_WAITING, &lock->cnts);
 
 	/* When no more readers or writers, set the locked flag */
 	do {
-- 
2.18.1

Re: [PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Peter Zijlstra]

On Sun, Apr 25, 2021 at 04:06:37PM -0400, Waiman Long wrote:

>  void queued_write_lock_slowpath(struct qrwlock *lock)
>  {
> -	int cnts;
> +	int cnts = 0;
>  
>  	/* Put the writer into the wait queue */
>  	arch_spin_lock(&lock->wait_lock);
>  
>  	/* Try to acquire the lock directly if no reader is present */
>  	if (!atomic_read(&lock->cnts) &&
> -	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
> +	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
>  		goto unlock;

Would not something like:

	if (!(cnts = atomic_read(&lock->cnts)) &&
	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED)
		goto unlock;

Be clearer?

>  
> -	/* Set the waiting flag to notify readers that a writer is pending */
> -	atomic_add(_QW_WAITING, &lock->cnts);
> +	/*
> +	 * Set the waiting flag to notify readers that a writer is pending
> +	 *
> +	 * As only one writer who is the wait_lock owner can set the waiting
> +	 * flag which will be cleared later on when acquiring the write lock,
> +	 * we can easily replace atomic_or() by an atomic_add() if there is
> +	 * an architecture where an atomic_add() performs better than an
> +	 * atomic_or().

That might be a little overboard on the comment, but sure :-) I don't
think there's any arch that doesn't have atomic_or(), like I wrote
elsewhere, the one that's often an issue is atomic_fetch_or().

> +	 */
> +	atomic_or(_QW_WAITING, &lock->cnts);

Re: [PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Waiman Long]

On 4/26/21 4:08 AM, Peter Zijlstra wrote:
> On Sun, Apr 25, 2021 at 04:06:37PM -0400, Waiman Long wrote:
>
>>   void queued_write_lock_slowpath(struct qrwlock *lock)
>>   {
>> -	int cnts;
>> +	int cnts = 0;
>>   
>>   	/* Put the writer into the wait queue */
>>   	arch_spin_lock(&lock->wait_lock);
>>   
>>   	/* Try to acquire the lock directly if no reader is present */
>>   	if (!atomic_read(&lock->cnts) &&
>> -	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
>> +	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
>>   		goto unlock;
> Would not something like:
>
> 	if (!(cnts = atomic_read(&lock->cnts)) &&
> 	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED)
> 		goto unlock;
>
> Be clearer?
That works for me too. It is equivalent anyway.
>
>>   
>> -	/* Set the waiting flag to notify readers that a writer is pending */
>> -	atomic_add(_QW_WAITING, &lock->cnts);
>> +	/*
>> +	 * Set the waiting flag to notify readers that a writer is pending
>> +	 *
>> +	 * As only one writer who is the wait_lock owner can set the waiting
>> +	 * flag which will be cleared later on when acquiring the write lock,
>> +	 * we can easily replace atomic_or() by an atomic_add() if there is
>> +	 * an architecture where an atomic_add() performs better than an
>> +	 * atomic_or().
> That might be a little overboard on the comment, but sure :-) I don't
> think there's any arch that doesn't have atomic_or(), like I wrote
> elsewhere, the one that's often an issue is atomic_fetch_or().
>
I was not sure as I didn't look at other archs that hadn't used qrwlock 
yet. Given what you said, I will remove the comment.

Cheers,
Longman

[PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Waiman Long]

Make the code more readable by replacing the atomic_cmpxchg_acquire()
by an equivalent atomic_try_cmpxchg_acquire() and change atomic_add()
to atomic_or().

For architectures that use qrwlock, I do not find one that has an
atomic_add() defined but not an atomic_or().  I guess it should be fine
by changing atomic_add() to atomic_or().

Note that the previous use of atomic_add() isn't wrong as only one
writer that is the wait_lock owner can set the waiting flag and the
flag will be cleared later on when acquiring the write lock.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/locking/qrwlock.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c
index b94f3831e963..ec36b73f4733 100644
--- a/kernel/locking/qrwlock.c
+++ b/kernel/locking/qrwlock.c
@@ -66,12 +66,12 @@ void queued_write_lock_slowpath(struct qrwlock *lock)
 	arch_spin_lock(&lock->wait_lock);
 
 	/* Try to acquire the lock directly if no reader is present */
-	if (!atomic_read(&lock->cnts) &&
-	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
+	if (!(cnts = atomic_read(&lock->cnts)) &&
+	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
 		goto unlock;
 
 	/* Set the waiting flag to notify readers that a writer is pending */
-	atomic_add(_QW_WAITING, &lock->cnts);
+	atomic_or(_QW_WAITING, &lock->cnts);
 
 	/* When no more readers or writers, set the locked flag */
 	do {
-- 
2.18.1

Re: [PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Waiman Long]

On 4/26/21 2:50 PM, Waiman Long wrote:
> Make the code more readable by replacing the atomic_cmpxchg_acquire()
> by an equivalent atomic_try_cmpxchg_acquire() and change atomic_add()
> to atomic_or().
>
> For architectures that use qrwlock, I do not find one that has an
> atomic_add() defined but not an atomic_or().  I guess it should be fine
> by changing atomic_add() to atomic_or().
>
> Note that the previous use of atomic_add() isn't wrong as only one
> writer that is the wait_lock owner can set the waiting flag and the
> flag will be cleared later on when acquiring the write lock.
>
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Waiman Long <longman@redhat.com>
> ---
>   kernel/locking/qrwlock.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c
> index b94f3831e963..ec36b73f4733 100644
> --- a/kernel/locking/qrwlock.c
> +++ b/kernel/locking/qrwlock.c
> @@ -66,12 +66,12 @@ void queued_write_lock_slowpath(struct qrwlock *lock)
>   	arch_spin_lock(&lock->wait_lock);
>   
>   	/* Try to acquire the lock directly if no reader is present */
> -	if (!atomic_read(&lock->cnts) &&
> -	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
> +	if (!(cnts = atomic_read(&lock->cnts)) &&
> +	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
>   		goto unlock;
>   
>   	/* Set the waiting flag to notify readers that a writer is pending */
> -	atomic_add(_QW_WAITING, &lock->cnts);
> +	atomic_or(_QW_WAITING, &lock->cnts);
>   
>   	/* When no more readers or writers, set the locked flag */
>   	do {

Sorry, I missed the "v2" in the title.

Cheers,
Longman

Re: [PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Will Deacon]

On Mon, Apr 26, 2021 at 02:50:17PM -0400, Waiman Long wrote:
> Make the code more readable by replacing the atomic_cmpxchg_acquire()
> by an equivalent atomic_try_cmpxchg_acquire() and change atomic_add()
> to atomic_or().
> 
> For architectures that use qrwlock, I do not find one that has an
> atomic_add() defined but not an atomic_or().  I guess it should be fine
> by changing atomic_add() to atomic_or().
> 
> Note that the previous use of atomic_add() isn't wrong as only one
> writer that is the wait_lock owner can set the waiting flag and the
> flag will be cleared later on when acquiring the write lock.

Right, there's no functional change here.

> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Waiman Long <longman@redhat.com>
> ---
>  kernel/locking/qrwlock.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c
> index b94f3831e963..ec36b73f4733 100644
> --- a/kernel/locking/qrwlock.c
> +++ b/kernel/locking/qrwlock.c
> @@ -66,12 +66,12 @@ void queued_write_lock_slowpath(struct qrwlock *lock)
>  	arch_spin_lock(&lock->wait_lock);
>  
>  	/* Try to acquire the lock directly if no reader is present */
> -	if (!atomic_read(&lock->cnts) &&
> -	    (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0))
> +	if (!(cnts = atomic_read(&lock->cnts)) &&
> +	    atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED))
>  		goto unlock;
>  
>  	/* Set the waiting flag to notify readers that a writer is pending */
> -	atomic_add(_QW_WAITING, &lock->cnts);
> +	atomic_or(_QW_WAITING, &lock->cnts);

Acked-by: Will Deacon <will@kernel.org>

Will

Re: [PATCH] locking/qrwlock: queued_write_lock_slowpath() cleanup

[Peter Zijlstra]

On Tue, Apr 27, 2021 at 08:56:59AM +0100, Will Deacon wrote:
> On Mon, Apr 26, 2021 at 02:50:17PM -0400, Waiman Long wrote:
> > Make the code more readable by replacing the atomic_cmpxchg_acquire()
> > by an equivalent atomic_try_cmpxchg_acquire() and change atomic_add()
> > to atomic_or().
> > 
> > For architectures that use qrwlock, I do not find one that has an
> > atomic_add() defined but not an atomic_or().  I guess it should be fine
> > by changing atomic_add() to atomic_or().
> > 
> > Note that the previous use of atomic_add() isn't wrong as only one
> > writer that is the wait_lock owner can set the waiting flag and the
> > flag will be cleared later on when acquiring the write lock.
> 
> Right, there's no functional change here.
> 
> > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> > Signed-off-by: Waiman Long <longman@redhat.com>

> Acked-by: Will Deacon <will@kernel.org>

Thanks!