From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756046AbcIEMou (ORCPT ); Mon, 5 Sep 2016 08:44:50 -0400 Received: from merlin.infradead.org ([205.233.59.134]:58938 "EHLO merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754802AbcIEMoD (ORCPT ); Mon, 5 Sep 2016 08:44:03 -0400 Message-Id: <20160905124027.619747346@infradead.org> User-Agent: quilt/0.61-1 Date: Mon, 05 Sep 2016 14:36:43 +0200 From: Peter Zijlstra To: Linus Torvalds , Waiman Long , Jason Low , Ding Tianhong , Thomas Gleixner , Will Deacon , Ingo Molnar , Imre Deak , Linux Kernel Mailing List , Davidlohr Bueso , Tim Chen , Terry Rudd , "Paul E. McKenney" , Jason Low , Peter Zijlstra Subject: [PATCH -v3 07/10] mutex: Restructure wait loop References: <20160905123636.450529224@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline; filename=peterz-locking-mutex-wait_lock-opt.patch Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Doesn't really matter yet, but pull the HANDOFF and trylock out from under the wait_lock. The intention is to add an optimistic spin loop here, which requires we do not hold the wait_lock, so shuffle code around in preparation. Also clarify the purpose of taking the wait_lock in the wait loop, its tempting to want to avoid it altogether, but the cancellation cases need to to avoid loosing wakeups. Suggested-by: Waiman Long Signed-off-by: Peter Zijlstra (Intel) --- kernel/locking/mutex.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -627,10 +627,12 @@ __mutex_lock_common(struct mutex *lock, lock_contended(&lock->dep_map, ip); + set_task_state(task, state); for (;;) { /* - * got a signal? (This code gets eliminated in the - * TASK_UNINTERRUPTIBLE case.) + * Check for signals and wound conditions while holding + * wait_lock. This ensures the lock cancellation is ordered + * against mutex_unlock() and wake-ups do not go missing. */ if (unlikely(signal_pending_state(state, task))) { ret = -EINTR; @@ -643,20 +645,27 @@ __mutex_lock_common(struct mutex *lock, goto err; } - __set_task_state(task, state); spin_unlock_mutex(&lock->wait_lock, flags); schedule_preempt_disabled(); - spin_lock_mutex(&lock->wait_lock, flags); if (__mutex_waiter_is_first(lock, &waiter)) { first = true; __mutex_set_flag(lock, MUTEX_FLAG_HANDOFF); } + set_task_state(task, state); + /* + * Here we order against unlock; we must either see it change + * state back to RUNNING and fall through the next schedule(), + * or we must see its unlock and acquire. + */ if (__mutex_trylock(lock, true)) break; + + spin_lock_mutex(&lock->wait_lock, flags); } __set_task_state(task, TASK_RUNNING); + spin_lock_mutex(&lock->wait_lock, flags); remove_waiter: mutex_remove_waiter(lock, &waiter, task); @@ -681,6 +690,7 @@ __mutex_lock_common(struct mutex *lock, return 0; err: + __set_task_state(task, TASK_RUNNING); mutex_remove_waiter(lock, &waiter, task); spin_unlock_mutex(&lock->wait_lock, flags); debug_mutex_free_waiter(&waiter);