From: "Nicolai Hähnle" <nhaehnle@gmail.com>
To: linux-kernel@vger.kernel.org
Cc: "Nicolai Hähnle" <Nicolai.Haehnle@amd.com>,
"Peter Zijlstra" <peterz@infradead.org>,
"Ingo Molnar" <mingo@redhat.com>,
"Maarten Lankhorst" <dev@mblankhorst.nl>,
"Daniel Vetter" <daniel@ffwll.ch>,
"Chris Wilson" <chris@chris-wilson.co.uk>,
dri-devel@lists.freedesktop.org
Subject: [PATCH 11/11] [rfc] locking/ww_mutex: Always spin optimistically for the first waiter
Date: Mon, 28 Nov 2016 13:20:12 +0100 [thread overview]
Message-ID: <1480335612-12069-12-git-send-email-nhaehnle@gmail.com> (raw)
In-Reply-To: <1480335612-12069-1-git-send-email-nhaehnle@gmail.com>
From: Nicolai Hähnle <Nicolai.Haehnle@amd.com>
Check the current owner's context once against our stamp. If our stamp is
lower, we continue to spin optimistically instead of backing off.
This is correct with respect to deadlock detection because while the
(owner, ww_ctx) pair may re-appear if the owner task manages to unlock
and re-acquire the lock while we're spinning, the context could only have
been re-initialized with an even higher stamp. We also still detect when
we have to back off for other waiters that join the list while we're
spinning.
But taking the wait_lock in mutex_spin_on_owner feels iffy, even if it is
done only once.
Median timings taken of a contention-heavy GPU workload:
Before:
real 0m53.086s
user 0m7.360s
sys 1m46.204s
After:
real 0m52.577s
user 0m7.544s
sys 1m49.200s
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Maarten Lankhorst <dev@mblankhorst.nl>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Nicolai Hähnle <Nicolai.Haehnle@amd.com>
---
kernel/locking/mutex.c | 35 ++++++++++++++++++++++++++++++++---
1 file changed, 32 insertions(+), 3 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index ee84007..e7d5fac 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -378,6 +378,28 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner,
struct mutex_waiter *waiter)
{
bool ret = true;
+ struct ww_acquire_ctx *owner_ww_ctx = NULL;
+
+ if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) {
+ struct ww_mutex *ww;
+ unsigned long flags;
+
+ ww = container_of(lock, struct ww_mutex, base);
+
+ /*
+ * Check the stamp of the current owner once. This allows us
+ * to spin optimistically in the case where the current owner
+ * has a higher stamp than us.
+ */
+ spin_lock_mutex(&lock->wait_lock, flags);
+ owner_ww_ctx = ww->ctx;
+ if (owner_ww_ctx &&
+ __ww_mutex_stamp_after(ww_ctx, owner_ww_ctx)) {
+ spin_unlock_mutex(&lock->wait_lock, flags);
+ return false;
+ }
+ spin_unlock_mutex(&lock->wait_lock, flags);
+ }
rcu_read_lock();
while (__mutex_owner(lock) == owner) {
@@ -414,9 +436,16 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner,
* Check this in every inner iteration because we may
* be racing against another thread's ww_mutex_lock.
*/
- if (ww_ctx->acquired > 0 && READ_ONCE(ww->ctx)) {
- ret = false;
- break;
+ if (ww_ctx->acquired > 0) {
+ struct ww_acquire_ctx *current_ctx;
+
+ current_ctx = READ_ONCE(ww->ctx);
+
+ if (current_ctx &&
+ current_ctx != owner_ww_ctx) {
+ ret = false;
+ break;
+ }
}
/*
--
2.7.4
next prev parent reply other threads:[~2016-11-28 12:21 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-28 12:20 [PATCH 00/11] locking/ww_mutex: Keep sorted wait list to avoid stampedes Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 01/11] drm/vgem: Use ww_mutex_(un)lock even with a NULL context Nicolai Hähnle
2016-11-28 12:42 ` Maarten Lankhorst
2016-11-28 12:50 ` Peter Zijlstra
2016-11-28 12:58 ` Christian König
2016-11-28 12:20 ` [PATCH 02/11] locking/ww_mutex: Re-check ww->ctx in the inner optimistic spin loop Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 03/11] locking/ww_mutex: Extract stamp comparison to __ww_mutex_stamp_after Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 04/11] locking/ww_mutex: Set use_ww_ctx even when locking without a context Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 05/11] locking/ww_mutex: Add waiters in stamp order Nicolai Hähnle
2016-11-30 14:10 ` Chris Wilson
2016-11-28 12:20 ` [PATCH 06/11] locking/ww_mutex: Notify waiters that have to back off while adding tasks to wait list Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 07/11] locking/ww_mutex: Wake at most one waiter for back off when acquiring the lock Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 08/11] locking/ww_mutex: Yield to other waiters from optimistic spin Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 09/11] locking/mutex: Initialize mutex_waiter::ww_ctx with poison when debugging Nicolai Hähnle
2016-11-28 12:20 ` [PATCH 10/11] Documentation/locking/ww_mutex: Update the design document Nicolai Hähnle
2016-11-28 12:20 ` Nicolai Hähnle [this message]
2016-11-30 0:35 ` [PATCH 1/4] locking: Begin kselftests for ww_mutex Chris Wilson
2016-11-30 0:35 ` [PATCH 2/4] locking: Add kselftests for ww_mutex AA deadlock detection Chris Wilson
2016-11-30 0:35 ` [PATCH 3/4] locking: Add kselftests for ww_mutex ABBA " Chris Wilson
2016-11-30 0:35 ` [PATCH 4/4] locking: Add kselftests for ww_mutex stress Chris Wilson
2016-11-30 12:29 ` Maarten Lankhorst
2016-11-30 12:52 ` Chris Wilson
2016-11-30 8:00 ` [PATCH 1/4] locking: Begin kselftests for ww_mutex Nicolai Hähnle
2016-11-30 19:08 ` kbuild test robot
2016-11-30 9:40 ` [PATCH 00/11] locking/ww_mutex: Keep sorted wait list to avoid stampedes Chris Wilson
2016-11-30 11:52 ` Nicolai Hähnle
2016-11-30 12:20 ` Chris Wilson
2016-11-30 13:40 ` Nicolai Hähnle
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1480335612-12069-12-git-send-email-nhaehnle@gmail.com \
--to=nhaehnle@gmail.com \
--cc=Nicolai.Haehnle@amd.com \
--cc=chris@chris-wilson.co.uk \
--cc=daniel@ffwll.ch \
--cc=dev@mblankhorst.nl \
--cc=dri-devel@lists.freedesktop.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®