Re: [PATCH-tip v3 04/14] locking/rwsem: Implement lock handoff to prevent lock starvation

[Peter Zijlstra <peterz@infradead.org>]

Hurph, I was still looking at v2.. I suppose I'll go stare at this
verison, I don't think you said there were many changes, right?

This version seems to still suffer that HANDOFF issue I found on v2.

On Wed, Apr 10, 2019 at 02:42:21PM -0400, Waiman Long wrote:
> Because of writer lock stealing, it is possible that a constant
> stream of incoming writers will cause a waiting writer or reader to
> wait indefinitely leading to lock starvation.
> 
> The mutex code has a lock handoff mechanism to prevent lock starvation.
> This patch implements a similar lock handoff mechanism to disable
> lock stealing and force lock handoff to the first waiter in the queue
> after at least a 4ms waiting period unless it is a RT writer task which
> doesn't need to wait. The waiting period is used to avoid discouraging
> lock stealing too much to affect performance.
> 
> A rwsem microbenchmark was run for 5 seconds on a 2-socket 40-core
> 80-thread Skylake system with a v5.1 based kernel and 240 write_lock
> threads with 5us sleep critical section.
> 
> Before the patch, the min/mean/max numbers of locking operations for
> the locking threads were 1/7,792/173,696. After the patch, the figures
> became 5,842/6,542/7,458.  It can be seen that the rwsem became much
> more fair, though there was a drop of about 16% in the mean locking
> operations done which was a tradeoff of having better fairness.
> 
> Making the waiter set the handoff bit right after the first wakeup can

What does 'right after the first wakeup' mean? If that the top-waiter
setting it if it fails to acquire the lock due to steals?

> impact performance especially with a mixed reader/writer workload. With
> the same microbenchmark with short critical section and equal number of
> reader and writer threads (40/40), the reader/writer locking operation
> counts with the current patch were:
> 
>   40 readers, Iterations Min/Mean/Max = 1,793/1,794/1,796
>   40 writers, Iterations Min/Mean/Max = 1,793/34,956/86,081
> 
> By making waiter set handoff bit immediately after wakeup:
> 
>   40 readers, Iterations Min/Mean/Max = 43/44/46
>   40 writers, Iterations Min/Mean/Max = 43/1,263/3,191