From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933139AbdBVSFo (ORCPT ); Wed, 22 Feb 2017 13:05:44 -0500 Received: from mx1.redhat.com ([209.132.183.28]:35211 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932804AbdBVSF1 (ORCPT ); Wed, 22 Feb 2017 13:05:27 -0500 From: Waiman Long To: Ingo Molnar , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Davidlohr Bueso , Waiman Long Subject: [PATCH-tip 1/3] locking/rwsem: Check wait_list without lock if spinner present Date: Wed, 22 Feb 2017 13:03:52 -0500 Message-Id: <1487786634-22641-2-git-send-email-longman@redhat.com> In-Reply-To: <1487786634-22641-1-git-send-email-longman@redhat.com> References: <1487786634-22641-1-git-send-email-longman@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 22 Feb 2017 18:05:27 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We can safely check the wait_list to see if waiters are present without lock when there are spinners to fall back on in case we miss a waiter. The advantage is that we can save a pair of spin_lock/unlock calls when the wait_list is empty. This translates to a reduction in latency and hence slightly better performance. On a 2-socket 36-core 72-thread x86-64 E5-2699 v3 system, a rwsem microbenchmark was run with 36 locking threads (one/core) doing 1 million writer lock/unlock operations each, the resulting locking rates (avg of 4 runs) on a 4.10 kernel were 7,755 Mop/s and 8,276 Mop/s without and with the patch respectively. That was an increase of about 7%. Signed-off-by: Waiman Long --- kernel/locking/rwsem-xadd.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 34e727f..b5d7055 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -611,6 +611,17 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) * state is consulted before reading the wait_lock. */ smp_rmb(); + + /* + * Normally checking wait_list without wait_lock isn't safe + * as we may miss an incoming waiter. With spinners present, + * however, we have someone to fall back on in case that + * happens. This can save a pair of spin_lock/unlock calls + * when there is no waiter. + */ + if (list_empty(&sem->wait_list)) + return sem; + if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags)) return sem; goto locked; -- 1.8.3.1