From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932362AbdCFTGZ (ORCPT ); Mon, 6 Mar 2017 14:06:25 -0500 Received: from mx1.redhat.com ([209.132.183.28]:42352 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754143AbdCFTF4 (ORCPT ); Mon, 6 Mar 2017 14:05:56 -0500 From: Waiman Long To: Ingo Molnar , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Davidlohr Bueso , Waiman Long Subject: [PATCH-tip v2 1/3] locking/rwsem: Check wait_list without lock if spinner present Date: Mon, 6 Mar 2017 14:04:41 -0500 Message-Id: <1488827083-30078-2-git-send-email-longman@redhat.com> In-Reply-To: <1488827083-30078-1-git-send-email-longman@redhat.com> References: <1488827083-30078-1-git-send-email-longman@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 06 Mar 2017 19:04:58 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In rwsem_wake(), we can safely check the wait_list to see if waiters are present without lock when there are spinners to fall back on in case we miss a waiter. The advantage is that we can save a pair of spin_lock/unlock calls when the wait_list is empty. This translates to a reduction in latency and hence slightly better performance. The raw_spin_trylock_irqsave() call is also being replaced by a raw_spin_is_locked() to reduce the overhead of irqsave/irqrestore especially when the trylock fails. On a 2-socket 36-core x86-64 E5-2699 v3 system, a rwsem microbenchmark was run with 36 locking threads doing 1 million writer lock/unlock operations each, the resulting locking rates (avg of 3 runs) on a 4.11-rc1 based kernel were 4,923 Mop/s and 5,136 Mop/s without and with the patch respectively. That was an increase of about 4%. On the same system, a 36-thread fio direct IO random write test to the same 2GB file on a XFS formatted ramdisk was run. The aggregated bandwidth was 1564.4 MB/s and 1593.0 MB/s before and after the patch. That was an increase of about 2%. Signed-off-by: Waiman Long --- kernel/locking/rwsem-xadd.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 34e727f..f31dd61b 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -587,7 +587,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) /* * If a spinner is present, it is not necessary to do the wakeup. - * Try to do wakeup only if the trylock succeeds to minimize + * Try to do wakeup only if the wait_lock is free to minimize * spinlock contention which may introduce too much delay in the * unlock operation. * @@ -595,7 +595,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) * --------------- ----------------------- * [S] osq_unlock() [L] osq * MB RMB - * [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock) + * [RmW] rwsem_try_write_lock() [RmW] spin_is_locked(wait_lock) * * Here, it is important to make sure that there won't be a missed * wakeup while the rwsem is free and the only spinning writer goes @@ -611,12 +611,18 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) * state is consulted before reading the wait_lock. */ smp_rmb(); - if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags)) + + /* + * Normally checking wait_list without wait_lock isn't safe + * as we may miss an incoming waiter. With spinners present, + * however, we have someone to fall back on in case that + * happens. + */ + if (list_empty(&sem->wait_list) || + raw_spin_is_locked(&sem->wait_lock)) return sem; - goto locked; } raw_spin_lock_irqsave(&sem->wait_lock, flags); -locked: if (!list_empty(&sem->wait_list)) __rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q); -- 1.8.3.1