From: Gregory Haskins <ghaskins@novell.com>
To: mingo@elte.hu, a.p.zijlstra@chello.nl, tglx@linutronix.de,
rostedt@goodmis.org, linux-rt-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, bill.huey@gmail.com,
kevin@hilman.org, cminyard@mvista.com, dsingleton@mvista.com,
dwalker@mvista.com, npiggin@suse.de, dsaxena@plexity.net,
ak@suse.de, gregkh@suse.de, sdietrich@novell.com,
pmorreale@novell.com, mkohari@novell.com, ghaskins@novell.com
Subject: [PATCH [RT] 13/14] allow rt-mutex lock-stealing to include lateral priority
Date: Thu, 21 Feb 2008 10:27:32 -0500 [thread overview]
Message-ID: <20080221152732.4804.44134.stgit@novell1.haskins.net> (raw)
In-Reply-To: <20080221152504.4804.8724.stgit@novell1.haskins.net>
The current logic only allows lock stealing to occur if the current task
is of higher priority than the pending owner. We can gain signficant
throughput improvements (200%+) by allowing the lock-stealing code to
include tasks of equal priority. The theory is that the system will make
faster progress by allowing the task already on the CPU to take the lock
rather than waiting for the system to wake-up a different task.
This does add a degree of unfairness, yes. But also note that the users
of these locks under non -rt environments have already been using unfair
raw spinlocks anyway so the tradeoff is probably worth it.
The way I like to think of this is that higher priority tasks should
clearly preempt, and lower priority tasks should clearly block. However,
if tasks have an identical priority value, then we can think of the
scheduler decisions as the tie-breaking parameter. (e.g. tasks that the
scheduler picked to run first have a logically higher priority amoung tasks
of the same prio). This helps to keep the system "primed" with tasks doing
useful work, and the end result is higher throughput.
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
---
kernel/Kconfig.preempt | 10 ++++++++++
kernel/rtmutex.c | 31 +++++++++++++++++++++++--------
2 files changed, 33 insertions(+), 8 deletions(-)
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index d2b0daa..343b93c 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -273,3 +273,13 @@ config SPINLOCK_BKL
Say Y here if you are building a kernel for a desktop system.
Say N if you are unsure.
+config RTLOCK_LATERAL_STEAL
+ bool "Allow equal-priority rtlock stealing"
+ default y
+ depends on PREEMPT_RT
+ help
+ This option alters the rtlock lock-stealing logic to allow
+ equal priority tasks to preempt a pending owner in addition
+ to higher priority tasks. This allows for a significant
+ boost in throughput under certain circumstances at the expense
+ of strict FIFO lock access.
diff --git a/kernel/rtmutex.c b/kernel/rtmutex.c
index 95c3644..da077e5 100644
--- a/kernel/rtmutex.c
+++ b/kernel/rtmutex.c
@@ -323,12 +323,27 @@ static int rt_mutex_adjust_prio_chain(struct task_struct *task,
return ret;
}
+static inline int lock_is_stealable(struct task_struct *pendowner, int unfair)
+{
+#ifndef CONFIG_RTLOCK_LATERAL_STEAL
+ if (current->prio >= pendowner->prio)
+#else
+ if (current->prio > pendowner->prio)
+ return 0;
+
+ if (!unfair && (current->prio == pendowner->prio))
+#endif
+ return 0;
+
+ return 1;
+}
+
/*
* Optimization: check if we can steal the lock from the
* assigned pending owner [which might not have taken the
* lock yet]:
*/
-static inline int try_to_steal_lock(struct rt_mutex *lock)
+static inline int try_to_steal_lock(struct rt_mutex *lock, int unfair)
{
struct task_struct *pendowner = rt_mutex_owner(lock);
struct rt_mutex_waiter *next;
@@ -340,7 +355,7 @@ static inline int try_to_steal_lock(struct rt_mutex *lock)
return 1;
spin_lock(&pendowner->pi_lock);
- if (current->prio >= pendowner->prio) {
+ if (!lock_is_stealable(pendowner, unfair)) {
spin_unlock(&pendowner->pi_lock);
return 0;
}
@@ -393,7 +408,7 @@ static inline int try_to_steal_lock(struct rt_mutex *lock)
*
* Must be called with lock->wait_lock held.
*/
-static int try_to_take_rt_mutex(struct rt_mutex *lock)
+static int try_to_take_rt_mutex(struct rt_mutex *lock, int unfair)
{
/*
* We have to be careful here if the atomic speedups are
@@ -416,7 +431,7 @@ static int try_to_take_rt_mutex(struct rt_mutex *lock)
*/
mark_rt_mutex_waiters(lock);
- if (rt_mutex_owner(lock) && !try_to_steal_lock(lock))
+ if (rt_mutex_owner(lock) && !try_to_steal_lock(lock, unfair))
return 0;
/* We got the lock. */
@@ -737,7 +752,7 @@ rt_spin_lock_slowlock(struct rt_mutex *lock)
int saved_lock_depth = current->lock_depth;
/* Try to acquire the lock */
- if (try_to_take_rt_mutex(lock))
+ if (try_to_take_rt_mutex(lock, 1))
break;
/*
* waiter.task is NULL the first time we come here and
@@ -985,7 +1000,7 @@ rt_mutex_slowlock(struct rt_mutex *lock, int state,
init_lists(lock);
/* Try to acquire the lock again: */
- if (try_to_take_rt_mutex(lock)) {
+ if (try_to_take_rt_mutex(lock, 0)) {
spin_unlock_irqrestore(&lock->wait_lock, flags);
return 0;
}
@@ -1006,7 +1021,7 @@ rt_mutex_slowlock(struct rt_mutex *lock, int state,
unsigned long saved_flags;
/* Try to acquire the lock: */
- if (try_to_take_rt_mutex(lock))
+ if (try_to_take_rt_mutex(lock, 0))
break;
/*
@@ -1120,7 +1135,7 @@ rt_mutex_slowtrylock(struct rt_mutex *lock)
init_lists(lock);
- ret = try_to_take_rt_mutex(lock);
+ ret = try_to_take_rt_mutex(lock, 0);
/*
* try_to_take_rt_mutex() sets the lock waiters
* bit unconditionally. Clean this up.
next prev parent reply other threads:[~2008-02-21 16:01 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-02-21 15:26 [PATCH [RT] 00/14] RFC - adaptive real-time locks Gregory Haskins
2008-02-21 15:26 ` [PATCH [RT] 01/14] spinlocks: fix preemption feature when PREEMPT_RT is enabled Gregory Haskins
2008-02-21 15:26 ` [PATCH [RT] 02/14] spinlock: make preemptible-waiter feature a specific config option Gregory Haskins
2008-02-22 19:09 ` Pavel Machek
2008-02-21 15:26 ` [PATCH [RT] 03/14] x86: FIFO ticket spinlocks Gregory Haskins
2008-02-21 15:26 ` [PATCH [RT] 04/14] disable PREEMPT_SPINLOCK_WAITERS when x86 ticket/fifo spins are in use Gregory Haskins
2008-02-21 15:26 ` [PATCH [RT] 05/14] rearrange rt_spin_lock sleep Gregory Haskins
2008-02-22 13:29 ` Gregory Haskins
2008-02-22 13:35 ` Steven Rostedt
2008-02-22 13:40 ` Peter Zijlstra
2008-02-22 13:35 ` Ingo Molnar
2008-02-22 13:43 ` Steven Rostedt
2008-02-22 13:46 ` Steven Rostedt
2008-02-21 15:26 ` [PATCH [RT] 06/14] optimize rt lock wakeup Gregory Haskins
2008-02-21 15:27 ` [PATCH [RT] 07/14] adaptive real-time lock support Gregory Haskins
2008-02-22 19:14 ` Pavel Machek
2008-02-21 15:27 ` [PATCH [RT] 08/14] add a loop counter based timeout mechanism Gregory Haskins
2008-02-21 16:41 ` Andi Kleen
2008-02-21 17:02 ` Gregory Haskins
2008-02-21 17:04 ` Peter W. Morreale
2008-02-21 17:06 ` Sven-Thorsten Dietrich
2008-02-22 19:08 ` Paul E. McKenney
2008-02-22 19:19 ` Bill Huey (hui)
2008-02-22 19:21 ` Bill Huey (hui)
2008-02-22 19:43 ` Paul E. McKenney
2008-02-22 19:55 ` Sven-Thorsten Dietrich
2008-02-22 20:23 ` Paul E. McKenney
2008-02-22 22:03 ` Gregory Haskins
2008-02-23 12:31 ` Andi Kleen
2008-02-23 16:32 ` Paul E. McKenney
2008-02-25 23:52 ` Sven-Thorsten Dietrich
2008-02-22 20:36 ` Peter W. Morreale
2008-02-23 7:36 ` Sven-Thorsten Dietrich
2008-02-22 20:15 ` Peter W. Morreale
2008-02-21 15:27 ` [PATCH [RT] 09/14] adaptive mutexes Gregory Haskins
2008-02-21 15:27 ` [PATCH [RT] 10/14] adjust pi_lock usage in wakeup Gregory Haskins
2008-02-21 16:48 ` Steven Rostedt
2008-02-21 17:09 ` Peter W. Morreale
2008-02-21 15:27 ` [PATCH [RT] 11/14] optimize the !printk fastpath through the lock acquisition Gregory Haskins
2008-02-21 16:36 ` Andi Kleen
2008-02-21 16:47 ` Gregory Haskins
2008-02-22 19:18 ` Pavel Machek
2008-02-22 22:20 ` Gregory Haskins
2008-02-23 0:43 ` Bill Huey (hui)
2008-02-25 5:20 ` Gregory Haskins
2008-02-25 6:21 ` Bill Huey (hui)
2008-02-25 9:02 ` Bill Huey (hui)
2008-02-21 15:27 ` [PATCH [RT] 12/14] remove the extra call to try_to_take_lock Gregory Haskins
2008-02-21 15:27 ` Gregory Haskins [this message]
2008-02-21 15:27 ` [PATCH [RT] 14/14] sysctl for runtime-control of lateral mutex stealing Gregory Haskins
2008-02-21 16:05 ` [PATCH [RT] 00/14] RFC - adaptive real-time locks Gregory Haskins
2008-02-21 21:24 ` Ingo Molnar
2008-02-21 21:33 ` Bill Huey (hui)
[not found] ` <20080221214219.GA27209@elte.hu>
2008-02-21 21:56 ` Gregory Haskins
2008-02-21 22:53 ` Bill Huey (hui)
2008-02-21 21:40 ` Gregory Haskins
2008-02-21 22:12 ` Peter W. Morreale
2008-02-21 22:42 ` Peter W. Morreale
2008-02-23 8:03 ` Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080221152732.4804.44134.stgit@novell1.haskins.net \
--to=ghaskins@novell.com \
--cc=a.p.zijlstra@chello.nl \
--cc=ak@suse.de \
--cc=bill.huey@gmail.com \
--cc=cminyard@mvista.com \
--cc=dsaxena@plexity.net \
--cc=dsingleton@mvista.com \
--cc=dwalker@mvista.com \
--cc=gregkh@suse.de \
--cc=kevin@hilman.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=mkohari@novell.com \
--cc=npiggin@suse.de \
--cc=pmorreale@novell.com \
--cc=rostedt@goodmis.org \
--cc=sdietrich@novell.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®