From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752787AbcFCK5E (ORCPT ); Fri, 3 Jun 2016 06:57:04 -0400 Received: from terminus.zytor.com ([198.137.202.10]:56620 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932777AbcFCK47 (ORCPT ); Fri, 3 Jun 2016 06:56:59 -0400 Date: Fri, 3 Jun 2016 03:55:45 -0700 From: tip-bot for Jason Low Message-ID: Cc: heiko.carstens@de.ibm.com, tony.luck@intel.com, tim.c.chen@linux.intel.com, paulmck@linux.vnet.ibm.com, dave@stgolabs.net, akpm@linux-foundation.org, torvalds@linux-foundation.org, ink@jurassic.park.msu.ru, hpa@zytor.com, arnd@arndb.de, schwidefsky@de.ibm.com, peterz@infradead.org, cl@linux.com, linux-kernel@vger.kernel.org, jason.low2@hpe.com, tglx@linutronix.de, fenghua.yu@intel.com, peter@hurleysoftware.com, jason.low2@hp.com, rth@twiddle.net, mattst88@gmail.com, Waiman.Long@hpe.com, terry.rudd@hpe.com, mingo@kernel.org Reply-To: tim.c.chen@linux.intel.com, heiko.carstens@de.ibm.com, tony.luck@intel.com, akpm@linux-foundation.org, dave@stgolabs.net, paulmck@linux.vnet.ibm.com, torvalds@linux-foundation.org, hpa@zytor.com, ink@jurassic.park.msu.ru, peterz@infradead.org, cl@linux.com, arnd@arndb.de, schwidefsky@de.ibm.com, linux-kernel@vger.kernel.org, jason.low2@hpe.com, peter@hurleysoftware.com, jason.low2@hp.com, tglx@linutronix.de, fenghua.yu@intel.com, mingo@kernel.org, rth@twiddle.net, mattst88@gmail.com, Waiman.Long@hpe.com, terry.rudd@hpe.com In-Reply-To: <1463445486-16078-2-git-send-email-jason.low2@hpe.com> References: <1463445486-16078-2-git-send-email-jason.low2@hpe.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/rwsem: Optimize write lock by reducing operations in slowpath Git-Commit-ID: c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8 Gitweb: http://git.kernel.org/tip/c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8 Author: Jason Low AuthorDate: Mon, 16 May 2016 17:38:00 -0700 Committer: Ingo Molnar CommitDate: Fri, 3 Jun 2016 09:47:13 +0200 locking/rwsem: Optimize write lock by reducing operations in slowpath When acquiring the rwsem write lock in the slowpath, we first try to set count to RWSEM_WAITING_BIAS. When that is successful, we then atomically add the RWSEM_WAITING_BIAS in cases where there are other tasks on the wait list. This causes write lock operations to often issue multiple atomic operations. We can instead make the list_is_singular() check first, and then set the count accordingly, so that we issue at most 1 atomic operation when acquiring the write lock and reduce unnecessary cacheline contention. Signed-off-by: Jason Low Signed-off-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Acked-by: Davidlohr Bueso Cc: Andrew Morton Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Fenghua Yu Cc: Heiko Carstens Cc: Ivan Kokshaysky Cc: Jason Low Cc: Linus Torvalds Cc: Martin Schwidefsky Cc: Matt Turner Cc: Paul E. McKenney Cc: Peter Hurley Cc: Peter Zijlstra Cc: Richard Henderson Cc: Terry Rudd Cc: Thomas Gleixner Cc: Tim Chen Cc: Tony Luck Link: http://lkml.kernel.org/r/1463445486-16078-2-git-send-email-jason.low2@hpe.com Signed-off-by: Ingo Molnar --- kernel/locking/rwsem-xadd.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index fcbf75a..b957da7 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -261,17 +261,28 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem) } EXPORT_SYMBOL(rwsem_down_read_failed); +/* + * This function must be called with the sem->wait_lock held to prevent + * race conditions between checking the rwsem wait list and setting the + * sem->count accordingly. + */ static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem) { /* - * Try acquiring the write lock. Check count first in order - * to reduce unnecessary expensive cmpxchg() operations. + * Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS. */ - if (count == RWSEM_WAITING_BIAS && - cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, - RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) { - if (!list_is_singular(&sem->wait_list)) - rwsem_atomic_update(RWSEM_WAITING_BIAS, sem); + if (count != RWSEM_WAITING_BIAS) + return false; + + /* + * Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there + * are other tasks on the wait list, we need to add on WAITING_BIAS. + */ + count = list_is_singular(&sem->wait_list) ? + RWSEM_ACTIVE_WRITE_BIAS : + RWSEM_ACTIVE_WRITE_BIAS + RWSEM_WAITING_BIAS; + + if (cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, count) == RWSEM_WAITING_BIAS) { rwsem_set_owner(sem); return true; }