From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755688Ab3IYWKw (ORCPT ); Wed, 25 Sep 2013 18:10:52 -0400 Received: from mga02.intel.com ([134.134.136.20]:5797 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755629Ab3IYWKu (ORCPT ); Wed, 25 Sep 2013 18:10:50 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.90,981,1371106800"; d="scan'208";a="409401454" Subject: [PATCH v6 4/6] rwsem/wake: check lock before do atomic update From: Tim Chen To: Ingo Molnar , Andrew Morton Cc: Andrea Arcangeli , Alex Shi , Andi Kleen , Michel Lespinasse , Davidlohr Bueso , Matthew R Wilcox , Dave Hansen , Peter Zijlstra , Rik van Riel , Peter Hurley , Tim Chen , linux-kernel@vger.kernel.org, linux-mm In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Wed, 25 Sep 2013 15:10:46 -0700 Message-ID: <1380147046.3467.66.camel@schen9-DESK> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 (2.32.3-1.fc14) Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Atomic update lock and roll back will cause cache bouncing in large machine. A lock status pre-read can relieve this problem Suggested-by: Davidlohr bueso Suggested-by: Tim Chen Signed-off-by: Alex Shi --- lib/rwsem.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/lib/rwsem.c b/lib/rwsem.c index a8055cf..1d6e6e8 100644 --- a/lib/rwsem.c +++ b/lib/rwsem.c @@ -64,7 +64,7 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) struct rwsem_waiter *waiter; struct task_struct *tsk; struct list_head *next; - long oldcount, woken, loop, adjustment; + long woken, loop, adjustment; waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); if (waiter->type == RWSEM_WAITING_FOR_WRITE) { @@ -86,6 +86,12 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) if (wake_type != RWSEM_WAKE_READ_OWNED) { adjustment = RWSEM_ACTIVE_READ_BIAS; while (1) { + long oldcount; + + /* A writer stole the lock. */ + if (unlikely(sem->count < RWSEM_WAITING_BIAS)) + return sem; + oldcount = rwsem_atomic_update(adjustment, sem) - adjustment; if (likely(oldcount >= RWSEM_WAITING_BIAS)) -- 1.7.4.4