From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753475Ab3FZW2E (ORCPT ); Wed, 26 Jun 2013 18:28:04 -0400 Received: from mga09.intel.com ([134.134.136.24]:60715 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753417Ab3FZW2A (ORCPT ); Wed, 26 Jun 2013 18:28:00 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.87,947,1363158000"; d="scan'208";a="360144384" Subject: [PATCH v4 4/5] rwsem/wake: check lock before do atomic update From: Tim Chen To: Ingo Molnar , Andrew Morton Cc: Andrea Arcangeli , Alex Shi , Andi Kleen , Michel Lespinasse , Davidlohr Bueso , Matthew R Wilcox , Dave Hansen , Peter Zijlstra , Rik van Riel , Peter Hurley , Tim Chen , linux-kernel@vger.kernel.org, linux-mm In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Wed, 26 Jun 2013 15:28:03 -0700 Message-ID: <1372285683.22432.144.camel@schen9-DESK> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 (2.32.3-1.fc14) Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Atomic update lock and roll back will cause cache bouncing in large machine. A lock status pre-read can relieve this problem Suggested-by: Davidlohr bueso Suggested-by: Tim Chen Signed-off-by: Alex Shi --- lib/rwsem.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/lib/rwsem.c b/lib/rwsem.c index a8055cf..1d6e6e8 100644 --- a/lib/rwsem.c +++ b/lib/rwsem.c @@ -64,7 +64,7 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) struct rwsem_waiter *waiter; struct task_struct *tsk; struct list_head *next; - long oldcount, woken, loop, adjustment; + long woken, loop, adjustment; waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); if (waiter->type == RWSEM_WAITING_FOR_WRITE) { @@ -86,6 +86,12 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) if (wake_type != RWSEM_WAKE_READ_OWNED) { adjustment = RWSEM_ACTIVE_READ_BIAS; while (1) { + long oldcount; + + /* A writer stole the lock. */ + if (unlikely(sem->count < RWSEM_WAITING_BIAS)) + return sem; + oldcount = rwsem_atomic_update(adjustment, sem) - adjustment; if (likely(oldcount >= RWSEM_WAITING_BIAS)) -- 1.7.4.4