From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753266AbcL2QP4 (ORCPT ); Thu, 29 Dec 2016 11:15:56 -0500 Received: from mx1.redhat.com ([209.132.183.28]:36658 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753148AbcL2QOg (ORCPT ); Thu, 29 Dec 2016 11:14:36 -0500 From: Waiman Long To: Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Jonathan Corbet Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Arnaldo Carvalho de Melo , Davidlohr Bueso , Mike Galbraith , Scott J Norton , Waiman Long Subject: [PATCH v4 15/20] TP-futex: Enable kernel reader lock stealing Date: Thu, 29 Dec 2016 11:13:41 -0500 Message-Id: <1483028026-10305-16-git-send-email-longman@redhat.com> In-Reply-To: <1483028026-10305-1-git-send-email-longman@redhat.com> References: <1483028026-10305-1-git-send-email-longman@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 29 Dec 2016 16:14:36 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The TP-futexes prefer writers as the code path is much simpler for them. This may not be good for overall system performance especially if there is a fair amount of reader activities. This patch enables kernel reader to steal the lock when the futex is currently reader-owned and the lock handoff mechanism hasn't been enabled yet. A new field locksteal_disabled is added to the futex state object for controlling reader lock stealing. So the waiting reader must retrieve the futex state object first before doing it. Signed-off-by: Waiman Long --- kernel/futex.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/futex.c b/kernel/futex.c index ee35264..b690a79 100644 --- a/kernel/futex.c +++ b/kernel/futex.c @@ -237,6 +237,7 @@ struct futex_state { atomic_t refcount; u32 handoff_pid; /* For TP futexes only */ + int locksteal_disabled; /* For TP futexes only */ enum futex_type type; union futex_key key; @@ -3436,6 +3437,7 @@ void exit_robust_list(struct task_struct *curr) state = alloc_futex_state(); state->type = TYPE_TP; state->key = *key; + state->locksteal_disabled = false; list_add(&state->fs_list, &hb->fs_head); WARN_ON(atomic_read(&state->refcount) != 1); @@ -3781,9 +3783,10 @@ static int futex_spin_on_owner(u32 __user *uaddr, const u32 vpid, WRITE_ONCE(state->handoff_pid, vpid); /* - * Disable handoff check. + * Disable handoff check and reader lock + * stealing. */ - handoff_set = true; + state->locksteal_disabled = handoff_set = true; } } @@ -3886,6 +3889,7 @@ static int futex_spin_on_owner(u32 __user *uaddr, const u32 vpid, */ WRITE_ONCE(state->mutex_owner, NULL); WRITE_ONCE(state->handoff_pid, 0); + state->locksteal_disabled = false; preempt_enable(); return (ret < 0) ? ret : TP_STATUS_SLEEP(ret, nsleep); @@ -3966,6 +3970,21 @@ static noinline int futex_lock(u32 __user *uaddr, unsigned int flags, goto out_put_state_key; } + /* + * For reader, we will try to steal the lock here as if it is the + * top waiter without taking the serialization mutex if reader + * lock stealing isn't disabled even though the FUTEX_WAITERS bit + * is set. + */ + if (shared && !state->locksteal_disabled) { + ret = futex_trylock(uaddr, vpid, &uval, true); + if (ret) { + if (ret > 0) + ret = TP_LOCK_STOLEN; + goto out_put_state_key; + } + } + if (to) hrtimer_start_expires(&to->timer, HRTIMER_MODE_ABS); -- 1.8.3.1