[PATCH RT 0/6] lockstat measurement extensions

[PATCH RT 0/6] lockstat measurement extensions

[Bill Huey (hui)]

Hello,

I'd like to announce extensions to the lockstat/lockdep framework to
measure the possibility of whether or not adaptive spins and/or lock
steals can happen within the rtmutex implementation's slow path. This
extends the common rtmutex functions to pass the depmap and friends so
that the lock_note_contention function can determine whether a
rtmutex->owner is live on another run queue. If so, then it logs it in
per cpu storage with the peterz's lockstat framework.

I had to extend a lot of function headers using some preprocessor
definitions to minimize the ifdef complexity, but it's still rather
complex even with some of the reductions. With that said and done, I'd
like suggest that a better method would be to add fields in the struct
rtmutex to pass values down to the lock_note_contention() function and
to contain state/events that can be post processed by
LOCK_CONTENDED*() macros instead. This was originally proposed by
Peter Zijlstra, but I had already decided to complete this track just
to see where it would take me.

This is a reimplementation of my own lockstat work into Peter's stuff.
I hope to pass it over to the Novell folks so that they can maintain
and take over development of this feature which will shine a light on
whether adaptive locks and lateral steals are useful in -rt.

Some results here:
----

lock_stat version 0.2
spinnables_total = 320571
contentions_total = 1670097
stolen_total = 1161888
cpu range error = 0
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                              class name    con-bounces contentions
[adapt,steals]   waittime-min   waittime-max waittime-total
acq-bounces   acquisitions   holdtime-min   holdtime-max
holdtime-total
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                &type->i_mutex_dir_key#2:        273392         659581
[172517, 658744] 18446744073709      580800.58   524881532.33
315374        1050378 18446744073709      605616.37   107058004.06
                ------------------------
                &type->i_mutex_dir_key#2         659581
[<ffffffff802a785b>] vfs_readdir+0x52/0xaf
                &type->i_mutex_dir_key#2              0
[<ffffffff802a2917>] do_lookup+0x84/0x1b4

...............................................................................................................................................................................................

                              irq_desc#1:        305009         305009
[1, 0]           0.51          13.55     1120626.91         620872
   1783324           0.23          36.62     8316839.04
                              ----------
                              irq_desc#1         304949
[<ffffffff8026c3bf>] do_irqd+0x86/0x2c7
                              irq_desc#1             39
[<ffffffff8026cfc8>] handle_fasteoi_irq+0x2a/0x10a

...............................................................................................................................................................................................

    (raw_spinlock_t *)(&lock->wait_lock):         77005          77561
[623, 0]           0.32          44.14       93979.97         743444
    10998216           0.24          53.89     4278155.20
    ------------------------------------
    (raw_spinlock_t *)(&lock->wait_lock)          14706
[<ffffffff804ff9bd>] rt_spin_lock_slowunlock+0xf/0x5c
    (raw_spinlock_t *)(&lock->wait_lock)          15631
[<ffffffff804ffa19>] rt_mutex_slowunlock+0xf/0x59
    (raw_spinlock_t *)(&lock->wait_lock)             61
[<ffffffff804ffe8e>] rt_mutex_slowlock+0x1f7/0x2d5
    (raw_spinlock_t *)(&lock->wait_lock)              9
[<ffffffff804ffbf0>] rt_spin_lock_slowlock+0x128/0x1cf

...............................................................................................................................................................................................

                   dcache_lock.wait_lock:         69559          72424
[34806, 0]           0.36          13.90       87260.09         379404
       1234745           0.29          27.01     1749680.84
                   ---------------------
                   dcache_lock.wait_lock          12618
[<ffffffff804ff9bd>] rt_spin_lock_slowunlock+0xf/0x5c
                   dcache_lock.wait_lock          51591
[<ffffffff804ffbf0>] rt_spin_lock_slowlock+0x128/0x1cf
                   dcache_lock.wait_lock           8072
[<ffffffff804ffb02>] rt_spin_lock_slowlock+0x3a/0x1cf
                   dcache_lock.wait_lock            143
[<ffffffff802623b7>] task_blocks_on_rt_mutex+0x1aa/0x1bf

----

I measure the total number of contention events printed out at the
very top of the listing and spin/steals within "[]" to the right of
the contention number for that lock. It's interesting how many steals
actually happen in that the percentage is quite substantial. This was
against a load of "find /" commands which hit inode locks pretty
heavily. irq_desc is interesting as well.

Patches can be found here:
http://mmlinux.sourceforge.net/public/lockstat/patch?.diff

More of the output can be found here
http://mmlinux.sourceforge.net/public/lockstat/output

bill

Re: [PATCH RT 1/6] lockstat measurement extensions

[Bill Huey (hui)]

[-- Attachment #1: Type: text/plain, Size: 80 bytes --]

Function parameter extension macros and function prototype redefinitions.

bill

[-- Attachment #2: patch0.diff --]
[-- Type: application/octet-stream, Size: 7020 bytes --]

--- linux-2.6.24/include/linux/lockdep.h	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/include/linux/lockdep.h	2008-03-13 14:02:43.000000000 -0700
@@ -19,6 +19,8 @@
 #include <linux/debug_locks.h>
 #include <linux/stacktrace.h>
 
+//#include <linux/rtmutex_common.h>
+
 /*
  * Lock-class usage-state bits:
  */
@@ -143,6 +145,10 @@
 
 struct lock_class_stats {
 	unsigned long			contention_point[4];
+#ifdef CONFIG_LOCK_STAT
+	unsigned long			spinnable;
+	unsigned long			stolen;
+#endif
 	struct lock_time		read_waittime;
 	struct lock_time		write_waittime;
 	struct lock_time		read_holdtime;
@@ -347,57 +353,190 @@
 
 #endif /* !LOCKDEP */
 
-#ifdef CONFIG_LOCK_STAT
+/*
+ * lockstat contention tracking is pushed down into the rtmutex implementation
+ * CONFIG_PREEMPT_RT instead.
+ */
+#if !defined(CONFIG_PREEMPT_RT) && defined(CONFIG_LOCK_STAT)
+
+extern void lock_note_contended(struct lockdep_map *lock, unsigned long ip);
+extern void lock_note_acquired(struct lockdep_map *lock);
+extern void lock_note_stolen(struct lockdep_map *lock);
+
+#define RET_IP_DECL	unsigned long _ip
+#define __RET_IP_DECL	, RET_IP_DECL
+#define __RET_IP_PARAM	, _ip
 
-extern void lock_contended(struct lockdep_map *lock, unsigned long ip);
-extern void lock_acquired(struct lockdep_map *lock);
+#define DEP_MAP_DECL	struct lockdep_map *_dep_map
+#define __DEP_MAP_DECL	, DEP_MAP_DECL
+#define DEP_MAP_PARAM	_dep_map
+#define __DEP_MAP_PARAM , DEP_MAP_PARAM
+
+#define LOCKDEP_TS_DECL
+#define __LOCKDEP_TS_DECL
+#define __LOCKDEP_TS_PARAM
+#define __LOCKDEP_TS_NULL
+
+#define LOCKDEP_PARAMS DEP_MAP_PARAM __RET_IP_PARAM
+#define __LOCKDEP_PARAMS , LOCKDEP_PARAMS
+#define __LOCKDEP_DECLS , DEP_MAP_DECL __RET_IP_DECL
+
+#define LOCKDEP_PARAMS_L0	NULL, _RET_IP_
+#define __LOCKDEP_PARAMS_L0	, LOCKDEP_PARAMS_L0
+
+#define LOCKDEP_PARAMS_SEM(lock)
+#define __LOCKDEP_PARAMS_SEM(lock)
+
+#define LOCKDEP_PARAMS_RWSEM(lock)
+#define __LOCKDEP_PARAMS_RWSEM(lock)
+
+#define LOCKDEP_PARAMS_MUTEX(lock)
+#define __LOCKDEP_PARAMS_MUTEX(lock)
 
 #define LOCK_CONTENDED(_lock, try, lock)			\
 do {								\
 	if (!try(_lock)) {					\
-		lock_contended(&(_lock)->dep_map, _RET_IP_);	\
+		lock_note_contended(&(_lock)->dep_map, _RET_IP_);	\
 		lock(_lock);					\
 	}							\
-	lock_acquired(&(_lock)->dep_map);			\
+	lock_note_acquired(&(_lock)->dep_map);			\
 } while (0)
 
-#define LOCK_CONTENDED_RT(_lock, f_try, f_lock)			\
+#define LOCK_CONTENDED_COMPAT(_lock, try, lock)			\
+do {								\
+	if (!try(_lock)) {					\
+		lock_note_contended(&(_lock)->dep_map, _RET_IP_);	\
+		lock(_lock);					\
+	}							\
+	lock_note_acquired(&(_lock)->dep_map);			\
+} while (0)
+
+#define LOCK_CONTENDED_RT(_lock, f_try, f_lock, ip)		\
 do {								\
 	if (!f_try(&(_lock)->lock)) {				\
-		lock_contended(&(_lock)->dep_map, _RET_IP_);	\
+		lock_note_contended(&(_lock)->dep_map, _RET_IP_);	\
 		f_lock(&(_lock)->lock);				\
 	}							\
-	lock_acquired(&(_lock)->dep_map);			\
+	lock_note_acquired(&(_lock)->dep_map);			\
 } while (0)
 
 
-#define LOCK_CONTENDED_RT_RET(_lock, f_try, f_lock)		\
+#define LOCK_CONTENDED_RT_RET(_lock, f_try, f_lock, ip)		\
 ({								\
  	int ret = 0;						\
 	if (!f_try(&(_lock)->lock)) {				\
-		lock_contended(&(_lock)->dep_map, _RET_IP_);	\
+		lock_note_contended(&(_lock)->dep_map, _RET_IP_);	\
 		ret = f_lock(&(_lock)->lock);			\
 	}							\
 	if (!ret)						\
-		lock_acquired(&(_lock)->dep_map);		\
+		lock_note_acquired(&(_lock)->dep_map);		\
  	ret;							\
 })
 
-#else /* CONFIG_LOCK_STAT */
+#else /* !defined(CONFIG_PREEMPT_RT) && defined(CONFIG_LOCK_STAT) */
+
+#if defined(CONFIG_PREEMPT_RT) && defined(CONFIG_LOCK_STAT)
+
+extern void lock_note_contended(struct lockdep_map *_dep_map, unsigned long _ip, struct task_struct *_owner);
+extern void lock_note_acquired(struct lockdep_map *lock);
+extern void lock_note_stolen(struct lockdep_map *lock);
+
+#define RET_IP_DECL	unsigned long _ip
+#define __RET_IP_DECL	, RET_IP_DECL
+#define __RET_IP_PARAM	, ip
+
+#define DEP_MAP_DECL	struct lockdep_map *_dep_map
+#define __DEP_MAP_DECL	, DEP_MAP_DECL
+#define DEP_MAP_PARAM	_dep_map
+#define __DEP_MAP_PARAM , DEP_MAP_PARAM
+
+#define LOCKDEP_TS_DECL		struct task_struct *_owner
+#define __LOCKDEP_TS_DECL	, LOCKDEP_TS_DECL
+#define __LOCKDEP_TS_PARAM	, _owner
+#define __LOCKDEP_TS_NULL	, NULL
+
+//#define LOCKDEP_PARAMS	 (struct task_struct *) ((unsigned long)_owner & (unsigned long) ~0x3), _dep_map, _ip
+#define LOCKDEP_PARAMS	 rt_mutex_real_owner(lock), _dep_map, _ip
+
+#define __LOCKDEP_PARAMS , LOCKDEP_PARAMS
+//#define __LOCKDEP_DECLS  , DEP_MAP_DECL, RET_IP_DECL
+#define __LOCKDEP_DECLS  , struct task_struct *_owner, DEP_MAP_DECL, RET_IP_DECL
+
+#define LOCKDEP_PARAMS_L0		current, NULL, _RET_IP_
+#define __LOCKDEP_PARAMS_L0		, LOCKDEP_PARAMS_L0
+
+#define LOCKDEP_PARAMS_SEM(lock)	LOCKDEP_PARAMS_L0
+#define __LOCKDEP_PARAMS_SEM(lock)	, LOCKDEP_PARAMS_L0
 
-#define lock_contended(lockdep_map, ip) do {} while (0)
-#define lock_acquired(lockdep_map) do {} while (0)
+#define LOCKDEP_PARAMS_RWSEM(lock)	LOCKDEP_PARAMS_L0
+#define __LOCKDEP_PARAMS_RWSEM(lock)	, LOCKDEP_PARAMS_L0
+
+#define LOCKDEP_PARAMS_MUTEX(lock)	LOCKDEP_PARAMS_L0
+#define __LOCKDEP_PARAMS_MUTEX(lock)	, LOCKDEP_PARAMS_L0
+
+#define LOCK_CONTENDED(_lock, try, lock)			\
+do {								\
+	if (!try(_lock)) {					\
+		lock_note_contended((struct task_struct *) &(_lock)->owner, \
+						&(_lock)->dep_map, \
+						_RET_IP_); \
+		lock(_lock);					\
+	}							\
+	lock_note_acquired(&(_lock)->dep_map);			\
+} while (0)
+
+#define LOCK_CONTENDED_COMPAT(_lock, try, lock)			\
+do {								\
+	if (!try(_lock)) {					\
+		lock_note_contended(NULL, &(_lock)->dep_map, _RET_IP_); \
+		lock(_lock);					\
+	}							\
+	lock_note_acquired(&(_lock)->dep_map);			\
+} while (0)
+
+#define LOCK_CONTENDED_RT(_lock, f_try, f_lock) \
+	f_lock(&(_lock)->lock, NULL, &(_lock)->dep_map, _RET_IP_)
+
+#define LOCK_CONTENDED_RT_RET(_lock, f_try, f_lock) \
+	f_lock(&(_lock)->lock, NULL, NULL, _RET_IP_)
+
+#elif /*defined(CONFIG_PREEMPT_RT) && */!defined(CONFIG_LOCK_STAT)
+/*#elif !defined(CONFIG_PREEMPT_RT) && !defined(CONFIG_LOCK_STAT) */
+
+#define lock_note_contended(lockdep_map, ip) do {} while (0)
+#define lock_note_acquired(lockdep_map) do {} while (0)
+
+#define __RET_IP_DECL
+#define __RET_IP_PARAM
+
+#define __TASK_STRUCT_LOCK_DECL
+#define __TASK_STRUCT_LOCK_PARAM
+#define __TASK_STRUCT_LOCK_PARAM2 lock,
 
 #define LOCK_CONTENDED(_lock, try, lock) \
 	lock(_lock)
 
+#define LOCK_CONTENDED_COMPAT(_lock, try, lock) \
+	lock(_lock)
+
 #define LOCK_CONTENDED_RT(_lock, f_try, f_lock) \
 	f_lock(&(_lock)->lock)
 
 #define LOCK_CONTENDED_RT_RET(_lock, f_try, f_lock) \
 	f_lock(&(_lock)->lock)
 
-#endif /* CONFIG_LOCK_STAT */
+#else
+#error
+#endif
+
+/*
+ * ac	(c1)
+ * a	(c2)
+ * ab	(c2)
+ * abc	(c2)
+ */
+
+#endif /* !defined(CONFIG_PREEMPT_RT) && defined(CONFIG_LOCK_STAT) */
 
 #if defined(CONFIG_TRACE_IRQFLAGS) && defined(CONFIG_GENERIC_HARDIRQS)
 extern void early_init_irq_lock_class(void);

Re: [PATCH RT 2/6] lockstat measurement extensions

[Bill Huey (hui)]

[-- Attachment #1: Type: text/plain, Size: 40 bytes --]

Function prototype redefinitions.

bill

[-- Attachment #2: patch1.diff --]
[-- Type: application/octet-stream, Size: 1974 bytes --]

--- linux-2.6.24/include/linux/rtmutex.h	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/include/linux/rtmutex.h	2008-03-04 01:01:06.000000000 -0800
@@ -85,14 +85,15 @@
 extern void __rt_mutex_init(struct rt_mutex *lock, const char *name);
 extern void rt_mutex_destroy(struct rt_mutex *lock);
 
-extern void rt_mutex_lock(struct rt_mutex *lock);
+extern void rt_mutex_lock(struct rt_mutex *lock __LOCKDEP_DECLS);
 extern int rt_mutex_lock_interruptible(struct rt_mutex *lock,
-						int detect_deadlock);
+						int detect_deadlock __LOCKDEP_DECLS);
 extern int rt_mutex_timed_lock(struct rt_mutex *lock,
 					struct hrtimer_sleeper *timeout,
 					int detect_deadlock);
 
 extern int rt_mutex_trylock(struct rt_mutex *lock);
+extern int __rt_mutex_trylock(struct rt_mutex *lock __LOCKDEP_DECLS);
 
 extern void rt_mutex_unlock(struct rt_mutex *lock);
 
--- linux-2.6.24/include/linux/sched.h	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/include/linux/sched.h	2008-03-13 00:40:02.000000000 -0700
@@ -2093,10 +2093,17 @@
 extern void signal_wake_up(struct task_struct *t, int resume_stopped);
 
 /*
+ * lockstat support
+ */
+extern int task_spinnable(struct task_struct *p);
+
+/*
  * Wrappers for p->thread_info->cpu access. No-op on UP.
  */
 #ifdef CONFIG_SMP
 
+extern atomic_t cpu_range_error;
+
 static inline unsigned int task_cpu(const struct task_struct *p)
 {
 	return task_thread_info(p)->cpu;
--- linux-2.6.24/include/linux/spinlock.h	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/include/linux/spinlock.h	2008-03-03 14:01:48.000000000 -0800
@@ -169,7 +169,7 @@
  * lockdep-less calls, for derived types like rwlock:
  * (for trylock they can use rt_mutex_trylock() directly.
  */
-extern void __lockfunc __rt_spin_lock(struct rt_mutex *lock);
+extern void __lockfunc __rt_spin_lock(struct rt_mutex *lock __LOCKDEP_DECLS);
 extern void __lockfunc __rt_spin_unlock(struct rt_mutex *lock);
 
 #ifdef CONFIG_PREEMPT_RT

Re: [PATCH RT 3/6] lockstat measurement extensions

[Bill Huey (hui)]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

lock_contention/lock_acquire() refactor to put common code searching
for an hlock into a common function. A function pointer is pass to ths
common function which can do lock_note* specific functionality. gcc
4.3 will inline this so this will not effect codegen quality for that
compiler version (it's now hitting ubuntu and other distributions)

bill

[-- Attachment #2: patch2.diff --]
[-- Type: application/octet-stream, Size: 8490 bytes --]

--- linux-2.6.24/kernel/lockdep.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/lockdep.c	2008-03-13 14:02:14.000000000 -0700
@@ -40,6 +40,7 @@
 #include <linux/utsname.h>
 #include <linux/hash.h>
 #include <linux/ftrace.h>
+#include <linux/sched.h>
 
 #include <asm/sections.h>
 
@@ -179,6 +180,9 @@
 		lock_time_add(&pcs->read_holdtime, &stats.read_holdtime);
 		lock_time_add(&pcs->write_holdtime, &stats.write_holdtime);
 
+		stats.stolen += pcs->stolen;
+		stats.spinnable += pcs->spinnable;
+
 		for (i = 0; i < ARRAY_SIZE(stats.bounces); i++)
 			stats.bounces[i] += pcs->bounces[i];
 	}
@@ -2854,13 +2858,21 @@
 }
 
 static void
-__lock_contended(struct lockdep_map *lock, unsigned long ip)
+__lock_note_common(struct lockdep_map *dep_map,
+		   void (*blockfn) (struct lockdep_map *dep_map,
+				    struct held_lock *hlock,
+				    unsigned long ip, int i
+				    __LOCKDEP_TS_DECL),
+		    unsigned long ip
+		    __LOCKDEP_TS_DECL)
 {
 	struct task_struct *curr = current;
 	struct held_lock *hlock, *prev_hlock;
-	struct lock_class_stats *stats;
 	unsigned int depth;
-	int i, point;
+	int i;
+
+	if (!dep_map)
+		return;
 
 	depth = curr->lockdep_depth;
 	if (DEBUG_LOCKS_WARN_ON(!depth))
@@ -2874,14 +2886,30 @@
 		 */
 		if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
 			break;
-		if (hlock->instance == lock)
+		if (hlock->instance == dep_map)
 			goto found_it;
 		prev_hlock = hlock;
 	}
-	print_lock_contention_bug(curr, lock, ip);
+	print_lock_contention_bug(curr, dep_map, ip);
 	return;
 
 found_it:
+	blockfn(dep_map, hlock, ip, i __LOCKDEP_TS_PARAM);
+}
+
+atomic_t spinnables_total = {.counter = 0},
+	contentions_total = {.counter = 0},
+	stolen_total = {.counter = 0};
+
+static void
+___lock_note_contended(struct lockdep_map *dep_map,
+		       struct held_lock *hlock,
+		       unsigned long ip, int i
+		       __LOCKDEP_TS_DECL)
+{
+	struct lock_class_stats *stats;
+	int point;
+
 	hlock->waittime_stamp = sched_clock();
 
 	point = lock_contention_point(hlock->class, ip);
@@ -2889,42 +2917,38 @@
 	stats = get_lock_stats(hlock->class);
 	if (point < ARRAY_SIZE(stats->contention_point))
 		stats->contention_point[i]++;
-	if (lock->cpu != smp_processor_id())
+	if (dep_map->cpu != smp_processor_id())
 		stats->bounces[bounce_contended + !!hlock->read]++;
+#ifdef CONFIG_PREEMPT_RT
+	if (likely(_owner) && task_spinnable(_owner)) {
+		stats->spinnable++;
+		atomic_inc(&spinnables_total);
+	}
+
+	atomic_inc(&contentions_total);
+#endif
 	put_lock_stats(stats);
 }
 
 static void
-__lock_acquired(struct lockdep_map *lock)
+__lock_note_contended(struct lockdep_map *dep_map,
+		      unsigned long ip
+		      __LOCKDEP_TS_DECL)
+{
+	__lock_note_common(dep_map, ___lock_note_contended, ip __LOCKDEP_TS_PARAM);
+}
+
+static void
+___lock_note_acquired(struct lockdep_map *lock,
+		      struct held_lock *hlock,
+		      unsigned long ip, int i
+		      __LOCKDEP_TS_DECL)
 {
-	struct task_struct *curr = current;
-	struct held_lock *hlock, *prev_hlock;
 	struct lock_class_stats *stats;
-	unsigned int depth;
 	u64 now;
 	s64 waittime = 0;
-	int i, cpu;
-
-	depth = curr->lockdep_depth;
-	if (DEBUG_LOCKS_WARN_ON(!depth))
-		return;
-
-	prev_hlock = NULL;
-	for (i = depth-1; i >= 0; i--) {
-		hlock = curr->held_locks + i;
-		/*
-		 * We must not cross into another context:
-		 */
-		if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
-			break;
-		if (hlock->instance == lock)
-			goto found_it;
-		prev_hlock = hlock;
-	}
-	print_lock_contention_bug(curr, lock, _RET_IP_);
-	return;
+	int cpu;
 
-found_it:
 	cpu = smp_processor_id();
 	if (hlock->waittime_stamp) {
 		now = sched_clock();
@@ -2946,7 +2970,58 @@
 	lock->cpu = cpu;
 }
 
-void lock_contended(struct lockdep_map *lock, unsigned long ip)
+static void
+__lock_note_acquired(struct lockdep_map *lock,
+		     unsigned long ip
+		     __LOCKDEP_TS_DECL)
+{
+	__lock_note_common(lock, ___lock_note_acquired, ip __LOCKDEP_TS_PARAM);
+	//__lock_note_common(lock, ___lock_note_acquired, _RET_IP_ __LOCKDEP_TS_PARAM);
+}
+
+static void
+___lock_note_stolen(struct lockdep_map *lock,
+		      struct held_lock *hlock,
+		      unsigned long ip, int i
+		      __LOCKDEP_TS_DECL)
+{
+	struct lock_class_stats *stats;
+
+	stats = get_lock_stats(hlock->class);
+	stats->stolen++;
+	atomic_inc(&stolen_total);
+	put_lock_stats(stats);
+}
+
+static void
+__lock_note_stolen(struct lockdep_map *lock,
+		     unsigned long ip
+		     __LOCKDEP_TS_DECL)
+{
+	__lock_note_common(lock, ___lock_note_stolen, _RET_IP_ __LOCKDEP_TS_PARAM);
+}
+
+void lock_note_stolen(struct lockdep_map *dep_map)
+{
+	unsigned long flags;
+
+	if (unlikely(!lock_stat))
+		return;
+
+	if (unlikely(current->lockdep_recursion))
+		return;
+
+	raw_local_irq_save(flags);
+	check_flags(flags);
+	current->lockdep_recursion = 1;
+	__lock_note_stolen(dep_map, 0 __LOCKDEP_TS_NULL);
+	current->lockdep_recursion = 0;
+	raw_local_irq_restore(flags);
+}
+EXPORT_SYMBOL_GPL(lock_note_stolen);
+
+void lock_note_contended( struct lockdep_map *dep_map,
+			unsigned long ip __LOCKDEP_TS_DECL)
 {
 	unsigned long flags;
 
@@ -2959,13 +3034,13 @@
 	raw_local_irq_save(flags);
 	check_flags(flags);
 	current->lockdep_recursion = 1;
-	__lock_contended(lock, ip);
+	__lock_note_contended(dep_map, ip __LOCKDEP_TS_PARAM);
 	current->lockdep_recursion = 0;
 	raw_local_irq_restore(flags);
 }
-EXPORT_SYMBOL_GPL(lock_contended);
+EXPORT_SYMBOL_GPL(lock_note_contended);
 
-void lock_acquired(struct lockdep_map *lock)
+void lock_note_acquired(struct lockdep_map *lock)
 {
 	unsigned long flags;
 
@@ -2978,11 +3053,11 @@
 	raw_local_irq_save(flags);
 	check_flags(flags);
 	current->lockdep_recursion = 1;
-	__lock_acquired(lock);
+	__lock_note_acquired(lock, 0 __LOCKDEP_TS_NULL);
 	current->lockdep_recursion = 0;
 	raw_local_irq_restore(flags);
 }
-EXPORT_SYMBOL_GPL(lock_acquired);
+EXPORT_SYMBOL_GPL(lock_note_acquired);
 #endif
 
 /*
--- linux-2.6.24/kernel/lockdep_proc.c	2008-01-24 14:58:37.000000000 -0800
+++ linux-2.6.24.working/kernel/lockdep_proc.c	2008-03-13 00:08:32.000000000 -0700
@@ -418,9 +418,12 @@
 	seq_printf(m, " %14s", num);
 }
 
-static void seq_lock_time(struct seq_file *m, struct lock_time *lt)
+static void seq_lock_time(struct seq_file *m, struct lock_time *lt, int header, unsigned long spinnables, unsigned long steals)
 {
 	seq_printf(m, "%14lu", lt->nr);
+	if (header) {
+		seq_printf(m, " [%lu, %lu]", spinnables, steals);
+	}
 	seq_time(m, lt->min);
 	seq_time(m, lt->max);
 	seq_time(m, lt->total);
@@ -432,6 +435,7 @@
 	struct lock_class *class;
 	struct lock_class_stats *stats;
 	int i, namelen;
+	int spinnables = 1;
 
 	class = data->class;
 	stats = &data->stats;
@@ -468,18 +472,21 @@
 			seq_printf(m, "%40s:", name);
 
 		seq_printf(m, "%14lu ", stats->bounces[bounce_contended_write]);
-		seq_lock_time(m, &stats->write_waittime);
+		seq_lock_time(m, &stats->write_waittime, spinnables, stats->spinnable, stats->stolen);
+		if (spinnables) {
+			spinnables = 0;
+		}
 		seq_printf(m, " %14lu ", stats->bounces[bounce_acquired_write]);
-		seq_lock_time(m, &stats->write_holdtime);
+		seq_lock_time(m, &stats->write_holdtime, 0, 0, 0);
 		seq_puts(m, "\n");
 	}
 
 	if (stats->read_holdtime.nr) {
 		seq_printf(m, "%38s-R:", name);
 		seq_printf(m, "%14lu ", stats->bounces[bounce_contended_read]);
-		seq_lock_time(m, &stats->read_waittime);
+		seq_lock_time(m, &stats->read_waittime, spinnables, stats->spinnable, stats->stolen);
 		seq_printf(m, " %14lu ", stats->bounces[bounce_acquired_read]);
-		seq_lock_time(m, &stats->read_holdtime);
+		seq_lock_time(m, &stats->read_holdtime, 0, 0, 0);
 		seq_puts(m, "\n");
 	}
 
@@ -513,15 +520,22 @@
 	}
 }
 
+extern atomic_t spinnables_total, contentions_total, stolen_total;
+atomic_t cpu_range_error = {.counter = 0};
+
 static void seq_header(struct seq_file *m)
 {
 	seq_printf(m, "lock_stat version 0.2\n");
+	seq_printf(m, "spinnables_total = %u\n", atomic_read(&spinnables_total));
+	seq_printf(m, "contentions_total = %u\n", atomic_read(&contentions_total));
+	seq_printf(m, "stolen_total = %u\n", atomic_read(&stolen_total));
+	seq_printf(m, "cpu range error = %u\n", atomic_read(&cpu_range_error));
 	seq_line(m, '-', 0, 40 + 1 + 10 * (14 + 1));
 	seq_printf(m, "%40s %14s %14s %14s %14s %14s %14s %14s %14s "
 			"%14s %14s\n",
 			"class name",
 			"con-bounces",
-			"contentions",
+			"contentions [adapt,steals]",
 			"waittime-min",
 			"waittime-max",
 			"waittime-total",

Re: [PATCH RT 4/6] lockstat measurement extensions

[Bill Huey (hui)]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

rtmutex.c push down of lock_note_contention()/lock_acquire() and new
functions lock_note_stolen() into the rtmutex.c common code itself
from LOCK_CONTENTION() top level macros. This is so that I can note
events at the specific points it occurs.

bill

[-- Attachment #2: patch3.diff --]
[-- Type: application/octet-stream, Size: 8804 bytes --]

--- linux-2.6.24/kernel/rtmutex.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rtmutex.c	2008-03-12 01:07:45.000000000 -0700
@@ -318,7 +318,7 @@
  * assigned pending owner [which might not have taken the
  * lock yet]:
  */
-static inline int try_to_steal_lock(struct rt_mutex *lock)
+static inline int try_to_steal_lock(struct rt_mutex *lock __DEP_MAP_DECL)
 {
 	struct task_struct *pendowner = rt_mutex_owner(lock);
 	struct rt_mutex_waiter *next;
@@ -327,7 +327,7 @@
 		return 0;
 
 	if (pendowner == current)
-		return 1;
+		goto stolen;
 
 	spin_lock(&pendowner->pi_lock);
 	if (current->prio >= pendowner->prio) {
@@ -342,7 +342,7 @@
 	 */
 	if (likely(!rt_mutex_has_waiters(lock))) {
 		spin_unlock(&pendowner->pi_lock);
-		return 1;
+		goto stolen;
 	}
 
 	/* No chain handling, pending owner is not blocked on anything: */
@@ -356,7 +356,7 @@
 	 * enqueued on the pending owners pi_waiters queue. So
 	 * we have to enqueue this waiter into
 	 * current->pi_waiters list. This covers the case,
-	 * where current is boosted because it holds another
+	 * where current is priority boosted because it holds another
 	 * lock and gets unboosted because the booster is
 	 * interrupted, so we would delay a waiter with higher
 	 * priority as current->normal_prio.
@@ -371,6 +371,9 @@
 		__rt_mutex_adjust_prio(current);
 		spin_unlock(&current->pi_lock);
 	}
+
+stolen:
+	lock_note_stolen(DEP_MAP_PARAM);
 	return 1;
 }
 
@@ -383,7 +386,7 @@
  *
  * Must be called with lock->wait_lock held.
  */
-static int try_to_take_rt_mutex(struct rt_mutex *lock)
+static int try_to_take_rt_mutex(struct rt_mutex *lock __DEP_MAP_DECL)
 {
 	/*
 	 * We have to be careful here if the atomic speedups are
@@ -406,7 +409,7 @@
 	 */
 	mark_rt_mutex_waiters(lock);
 
-	if (rt_mutex_owner(lock) && !try_to_steal_lock(lock))
+	if (rt_mutex_owner(lock) && !try_to_steal_lock(lock __DEP_MAP_PARAM))
 		return 0;
 
 	/* We got the lock. */
@@ -631,7 +634,7 @@
 
 static inline void
 rt_spin_lock_fastlock(struct rt_mutex *lock,
-		void fastcall (*slowfn)(struct rt_mutex *lock))
+		void fastcall (*slowfn)(struct rt_mutex *lock __LOCKDEP_DECLS) __LOCKDEP_DECLS)
 {
 	/* Temporary HACK! */
 	if (!current->in_printk)
@@ -643,7 +646,7 @@
 	if (likely(rt_mutex_cmpxchg(lock, NULL, current)))
 		rt_mutex_deadlock_account_lock(lock, current);
 	else
-		slowfn(lock);
+		slowfn(lock __LOCKDEP_PARAMS);
 }
 
 static inline void
@@ -672,7 +675,7 @@
  * sleep/wakeup event loops.
  */
 static void fastcall noinline __sched
-rt_spin_lock_slowlock(struct rt_mutex *lock)
+rt_spin_lock_slowlock(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
 	struct rt_mutex_waiter waiter;
 	unsigned long saved_state, state, flags;
@@ -680,13 +683,15 @@
 	debug_rt_mutex_init_waiter(&waiter);
 	waiter.task = NULL;
 
+	lock_note_contended(LOCKDEP_PARAMS);
+
 	spin_lock_irqsave(&lock->wait_lock, flags);
 	init_lists(lock);
 
 	/* Try to acquire the lock again: */
-	if (try_to_take_rt_mutex(lock)) {
+	if (try_to_take_rt_mutex(lock __DEP_MAP_PARAM)) {
 		spin_unlock_irqrestore(&lock->wait_lock, flags);
-		return;
+		goto acquired;
 	}
 
 	BUG_ON(rt_mutex_owner(lock) == current);
@@ -707,8 +712,9 @@
 		int saved_lock_depth = current->lock_depth;
 
 		/* Try to acquire the lock */
-		if (try_to_take_rt_mutex(lock))
+		if (try_to_take_rt_mutex(lock __DEP_MAP_PARAM))
 			break;
+
 		/*
 		 * waiter.task is NULL the first time we come here and
 		 * when we have been woken up by the previous owner
@@ -763,6 +769,9 @@
 	spin_unlock_irqrestore(&lock->wait_lock, flags);
 
 	debug_rt_mutex_free_waiter(&waiter);
+
+acquired:
+	lock_note_acquired(_dep_map);
 }
 
 /*
@@ -800,9 +809,9 @@
 }
 EXPORT_SYMBOL(rt_spin_lock);
 
-void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
+void __lockfunc __rt_spin_lock(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
-	rt_spin_lock_fastlock(lock, rt_spin_lock_slowlock);
+	rt_spin_lock_fastlock(lock, rt_spin_lock_slowlock __LOCKDEP_PARAMS);
 }
 EXPORT_SYMBOL(__rt_spin_lock);
 
@@ -934,7 +943,8 @@
 static int __sched
 rt_mutex_slowlock(struct rt_mutex *lock, int state,
 		  struct hrtimer_sleeper *timeout,
-		  int detect_deadlock)
+		  int detect_deadlock
+		  __LOCKDEP_DECLS)
 {
 	int ret = 0, saved_lock_depth = -1;
 	struct rt_mutex_waiter waiter;
@@ -943,13 +953,16 @@
 	debug_rt_mutex_init_waiter(&waiter);
 	waiter.task = NULL;
 
+	lock_note_contended(LOCKDEP_PARAMS);
+
 	spin_lock_irqsave(&lock->wait_lock, flags);
 	init_lists(lock);
 
 	/* Try to acquire the lock again: */
-	if (try_to_take_rt_mutex(lock)) {
+	if (try_to_take_rt_mutex(lock __DEP_MAP_PARAM)) {
 		spin_unlock_irqrestore(&lock->wait_lock, flags);
-		return 0;
+		ret = 0;
+		goto acquired;
 	}
 
 	/*
@@ -970,8 +983,9 @@
 		unsigned long saved_flags;
 
 		/* Try to acquire the lock: */
-		if (try_to_take_rt_mutex(lock))
+		if (try_to_take_rt_mutex(lock __DEP_MAP_PARAM)) {
 			break;
+		}
 
 		/*
 		 * TASK_INTERRUPTIBLE checks for signals and
@@ -1060,6 +1074,9 @@
 
 	debug_rt_mutex_free_waiter(&waiter);
 
+acquired:
+	lock_note_acquired(_dep_map);
+
 	return ret;
 }
 
@@ -1067,7 +1084,7 @@
  * Slow path try-lock function:
  */
 static inline int
-rt_mutex_slowtrylock(struct rt_mutex *lock)
+rt_mutex_slowtrylock(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
 	unsigned long flags;
 	int ret = 0;
@@ -1078,7 +1095,10 @@
 
 		init_lists(lock);
 
-		ret = try_to_take_rt_mutex(lock);
+		lock_note_contended(LOCKDEP_PARAMS);
+
+		ret = try_to_take_rt_mutex(lock __DEP_MAP_PARAM);
+
 		/*
 		 * try_to_take_rt_mutex() sets the lock waiters
 		 * bit unconditionally. Clean this up.
@@ -1088,6 +1108,9 @@
 
 	spin_unlock_irqrestore(&lock->wait_lock, flags);
 
+	if (!ret)
+		lock_note_acquired(_dep_map);
+
 	return ret;
 }
 
@@ -1130,13 +1153,13 @@
 		  int detect_deadlock,
 		  int (*slowfn)(struct rt_mutex *lock, int state,
 				struct hrtimer_sleeper *timeout,
-				int detect_deadlock))
+				int detect_deadlock __LOCKDEP_DECLS) __LOCKDEP_DECLS)
 {
 	if (!detect_deadlock && likely(rt_mutex_cmpxchg(lock, NULL, current))) {
 		rt_mutex_deadlock_account_lock(lock, current);
 		return 0;
 	} else
-		return slowfn(lock, state, NULL, detect_deadlock);
+		return slowfn(lock, state, NULL, detect_deadlock __LOCKDEP_PARAMS);
 }
 
 static inline int
@@ -1144,24 +1167,24 @@
 			struct hrtimer_sleeper *timeout, int detect_deadlock,
 			int (*slowfn)(struct rt_mutex *lock, int state,
 				      struct hrtimer_sleeper *timeout,
-				      int detect_deadlock))
+				      int detect_deadlock __LOCKDEP_DECLS) __LOCKDEP_DECLS)
 {
 	if (!detect_deadlock && likely(rt_mutex_cmpxchg(lock, NULL, current))) {
 		rt_mutex_deadlock_account_lock(lock, current);
 		return 0;
 	} else
-		return slowfn(lock, state, timeout, detect_deadlock);
+		return slowfn(lock, state, timeout, detect_deadlock __LOCKDEP_PARAMS);
 }
 
 static inline int
 rt_mutex_fasttrylock(struct rt_mutex *lock,
-		     int (*slowfn)(struct rt_mutex *lock))
+		     int (*slowfn)(struct rt_mutex *lock __LOCKDEP_DECLS) __LOCKDEP_DECLS)
 {
 	if (likely(rt_mutex_cmpxchg(lock, NULL, current))) {
 		rt_mutex_deadlock_account_lock(lock, current);
 		return 1;
 	}
-	return slowfn(lock);
+	return slowfn(lock __LOCKDEP_PARAMS);
 }
 
 static inline void
@@ -1179,11 +1202,11 @@
  *
  * @lock: the rt_mutex to be locked
  */
-void __sched rt_mutex_lock(struct rt_mutex *lock)
+void __sched rt_mutex_lock(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
 	might_sleep();
 
-	rt_mutex_fastlock(lock, TASK_UNINTERRUPTIBLE, 0, rt_mutex_slowlock);
+	rt_mutex_fastlock(lock, TASK_UNINTERRUPTIBLE, 0, rt_mutex_slowlock __LOCKDEP_PARAMS);
 }
 EXPORT_SYMBOL_GPL(rt_mutex_lock);
 
@@ -1199,12 +1222,12 @@
  * -EDEADLK	when the lock would deadlock (when deadlock detection is on)
  */
 int __sched rt_mutex_lock_interruptible(struct rt_mutex *lock,
-						 int detect_deadlock)
+						 int detect_deadlock __LOCKDEP_DECLS)
 {
 	might_sleep();
 
 	return rt_mutex_fastlock(lock, TASK_INTERRUPTIBLE,
-				 detect_deadlock, rt_mutex_slowlock);
+				 detect_deadlock, rt_mutex_slowlock __LOCKDEP_PARAMS);
 }
 EXPORT_SYMBOL_GPL(rt_mutex_lock_interruptible);
 
@@ -1230,7 +1253,7 @@
 	might_sleep();
 
 	return rt_mutex_timed_fastlock(lock, TASK_INTERRUPTIBLE, timeout,
-				       detect_deadlock, rt_mutex_slowlock);
+				       detect_deadlock, rt_mutex_slowlock __LOCKDEP_PARAMS_L0);
 }
 EXPORT_SYMBOL_GPL(rt_mutex_timed_lock);
 
@@ -1241,9 +1264,14 @@
  *
  * Returns 1 on success and 0 on contention
  */
+int __rt_mutex_trylock(struct rt_mutex *lock __LOCKDEP_DECLS)
+{
+	return rt_mutex_fasttrylock(lock, rt_mutex_slowtrylock __LOCKDEP_PARAMS);
+}
+
 int __sched rt_mutex_trylock(struct rt_mutex *lock)
 {
-	return rt_mutex_fasttrylock(lock, rt_mutex_slowtrylock);
+	return __rt_mutex_trylock(lock __LOCKDEP_PARAMS_L0);
 }
 EXPORT_SYMBOL_GPL(rt_mutex_trylock);
 

Re: [PATCH RT 5/6] lockstat measurement extensions

[Bill Huey (hui)]

[-- Attachment #1: Type: text/plain, Size: 110 bytes --]

Lock function bodies are extended to pass down a struct lockdep_map
and friends for contention logging.

bill

[-- Attachment #2: patch4.diff --]
[-- Type: application/octet-stream, Size: 10929 bytes --]

--- linux-2.6.24/kernel/rt.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rt.c	2008-03-12 22:40:09.000000000 -0700
@@ -102,9 +102,9 @@
 }
 EXPORT_SYMBOL(_mutex_lock);
 
-static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock)
+static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
-	return rt_mutex_lock_interruptible(lock, 0);
+	return rt_mutex_lock_interruptible(lock, 0 __LOCKDEP_PARAMS);
 }
 
 int __lockfunc _mutex_lock_interruptible(struct mutex *lock)
@@ -144,7 +144,7 @@
 
 int __lockfunc _mutex_trylock(struct mutex *lock)
 {
-	int ret = rt_mutex_trylock(&lock->lock);
+	int ret = __rt_mutex_trylock(&lock->lock __LOCKDEP_PARAMS_L0);
 
 	if (ret)
 		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
@@ -163,21 +163,28 @@
 /*
  * rwlock_t functions
  */
-int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+static
+int __lockfunc __rt_write_trylock(rwlock_t *rwlock __LOCKDEP_DECLS)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	//int ret = __rt_mutex_trylock(&rwlock->lock __LOCKDEP_PARAMS);
+	int ret = __rt_mutex_trylock(&rwlock->lock, rt_mutex_real_owner(&rwlock->lock), _dep_map, _ip);
 
 	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
 
 	return ret;
 }
+
+int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+{
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
+}
 EXPORT_SYMBOL(rt_write_trylock);
 
 int __lockfunc rt_write_trylock_irqsave(rwlock_t *rwlock, unsigned long *flags)
 {
 	*flags = 0;
-	return rt_write_trylock(rwlock);
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
 }
 EXPORT_SYMBOL(rt_write_trylock_irqsave);
 
@@ -453,7 +460,7 @@
 
 void fastcall rt_down(struct semaphore *sem)
 {
-	rt_mutex_lock(&sem->lock);
+	rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 	__down_complete(sem);
 }
 EXPORT_SYMBOL(rt_down);
@@ -462,7 +469,7 @@
 {
 	int ret;
 
-	ret = rt_mutex_lock_interruptible(&sem->lock, 0);
+	ret = rt_mutex_lock_interruptible(&sem->lock, 0 __LOCKDEP_PARAMS_SEM(sem));
 	if (ret)
 		return ret;
 	__down_complete(sem);
@@ -516,7 +523,7 @@
 	switch (val) {
 	case 0:
 		__rt_mutex_init(&sem->lock, name);
-		rt_mutex_lock(&sem->lock);
+		rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 		break;
 	default:
 		__rt_mutex_init(&sem->lock, name);
--- linux-2.6.24/kernel/mutex.c	2008-01-24 14:58:37.000000000 -0800
+++ linux-2.6.24.working/kernel/mutex.c	2008-03-13 13:03:06.000000000 -0700
@@ -146,7 +146,7 @@
 	if (old_val == 1)
 		goto done;
 
-	lock_contended(&lock->dep_map, ip);
+//	lock_note_contended(LOCKDEP_PARAMS);
 
 	for (;;) {
 		/*
@@ -184,7 +184,7 @@
 	}
 
 done:
-	lock_acquired(&lock->dep_map);
+//	lock_acquired(&lock->dep_map);
 	/* got the lock - rejoice! */
 	mutex_remove_waiter(lock, &waiter, task_thread_info(task));
 	debug_mutex_set_owner(lock, task_thread_info(task));
--- linux-2.6.24/kernel/rwsem.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rwsem.c	2008-03-11 02:45:20.000000000 -0700
@@ -21,7 +21,7 @@
 	might_sleep();
 	rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	LOCK_CONTENDED_COMPAT(sem, __down_read_trylock, __down_read);
 }
 
 EXPORT_SYMBOL(compat_down_read);
@@ -48,7 +48,7 @@
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, 0, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	LOCK_CONTENDED_COMPAT(sem, __down_write_trylock, __down_write);
 }
 
 EXPORT_SYMBOL(compat_down_write);
@@ -112,7 +112,7 @@
 	might_sleep();
 	rwsem_acquire_read(&sem->dep_map, subclass, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	LOCK_CONTENDED_COMPAT(sem, __down_read_trylock, __down_read);
 }
 
 EXPORT_SYMBOL(compat_down_read_nested);
@@ -131,7 +131,7 @@
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, subclass, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	LOCK_CONTENDED_COMPAT(sem, __down_write_trylock, __down_write);
 }
 
 EXPORT_SYMBOL(compat_down_write_nested);
--- linux-2.6.24/kernel/rt.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rt.c	2008-03-12 22:40:09.000000000 -0700
@@ -102,9 +102,9 @@
 }
 EXPORT_SYMBOL(_mutex_lock);
 
-static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock)
+static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
-	return rt_mutex_lock_interruptible(lock, 0);
+	return rt_mutex_lock_interruptible(lock, 0 __LOCKDEP_PARAMS);
 }
 
 int __lockfunc _mutex_lock_interruptible(struct mutex *lock)
@@ -144,7 +144,7 @@
 
 int __lockfunc _mutex_trylock(struct mutex *lock)
 {
-	int ret = rt_mutex_trylock(&lock->lock);
+	int ret = __rt_mutex_trylock(&lock->lock __LOCKDEP_PARAMS_L0);
 
 	if (ret)
 		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
@@ -163,21 +163,28 @@
 /*
  * rwlock_t functions
  */
-int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+static
+int __lockfunc __rt_write_trylock(rwlock_t *rwlock __LOCKDEP_DECLS)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	//int ret = __rt_mutex_trylock(&rwlock->lock __LOCKDEP_PARAMS);
+	int ret = __rt_mutex_trylock(&rwlock->lock, rt_mutex_real_owner(&rwlock->lock), _dep_map, _ip);
 
 	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
 
 	return ret;
 }
+
+int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+{
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
+}
 EXPORT_SYMBOL(rt_write_trylock);
 
 int __lockfunc rt_write_trylock_irqsave(rwlock_t *rwlock, unsigned long *flags)
 {
 	*flags = 0;
-	return rt_write_trylock(rwlock);
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
 }
 EXPORT_SYMBOL(rt_write_trylock_irqsave);
 
@@ -453,7 +460,7 @@
 
 void fastcall rt_down(struct semaphore *sem)
 {
-	rt_mutex_lock(&sem->lock);
+	rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 	__down_complete(sem);
 }
 EXPORT_SYMBOL(rt_down);
@@ -462,7 +469,7 @@
 {
 	int ret;
 
-	ret = rt_mutex_lock_interruptible(&sem->lock, 0);
+	ret = rt_mutex_lock_interruptible(&sem->lock, 0 __LOCKDEP_PARAMS_SEM(sem));
 	if (ret)
 		return ret;
 	__down_complete(sem);
@@ -516,7 +523,7 @@
 	switch (val) {
 	case 0:
 		__rt_mutex_init(&sem->lock, name);
-		rt_mutex_lock(&sem->lock);
+		rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 		break;
 	default:
 		__rt_mutex_init(&sem->lock, name);
--- linux-2.6.24/kernel/mutex.c	2008-01-24 14:58:37.000000000 -0800
+++ linux-2.6.24.working/kernel/mutex.c	2008-03-13 13:03:06.000000000 -0700
@@ -146,7 +146,7 @@
 	if (old_val == 1)
 		goto done;
 
-	lock_contended(&lock->dep_map, ip);
+//	lock_note_contended(LOCKDEP_PARAMS);
 
 	for (;;) {
 		/*
@@ -184,7 +184,7 @@
 	}
 
 done:
-	lock_acquired(&lock->dep_map);
+//	lock_acquired(&lock->dep_map);
 	/* got the lock - rejoice! */
 	mutex_remove_waiter(lock, &waiter, task_thread_info(task));
 	debug_mutex_set_owner(lock, task_thread_info(task));
--- linux-2.6.24/kernel/rwsem.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rwsem.c	2008-03-11 02:45:20.000000000 -0700
@@ -21,7 +21,7 @@
 	might_sleep();
 	rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	LOCK_CONTENDED_COMPAT(sem, __down_read_trylock, __down_read);
 }
 
 EXPORT_SYMBOL(compat_down_read);
@@ -48,7 +48,7 @@
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, 0, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	LOCK_CONTENDED_COMPAT(sem, __down_write_trylock, __down_write);
 }
 
 EXPORT_SYMBOL(compat_down_write);
@@ -112,7 +112,7 @@
 	might_sleep();
 	rwsem_acquire_read(&sem->dep_map, subclass, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
+	LOCK_CONTENDED_COMPAT(sem, __down_read_trylock, __down_read);
 }
 
 EXPORT_SYMBOL(compat_down_read_nested);
@@ -131,7 +131,7 @@
 	might_sleep();
 	rwsem_acquire(&sem->dep_map, subclass, 0, _RET_IP_);
 
-	LOCK_CONTENDED(sem, __down_write_trylock, __down_write);
+	LOCK_CONTENDED_COMPAT(sem, __down_write_trylock, __down_write);
 }
 
 EXPORT_SYMBOL(compat_down_write_nested);
--- linux-2.6.24/kernel/rt.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/rt.c	2008-03-12 22:40:09.000000000 -0700
@@ -102,9 +102,9 @@
 }
 EXPORT_SYMBOL(_mutex_lock);
 
-static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock)
+static int __lockfunc __rt_mutex_lock_interruptible(struct rt_mutex *lock __LOCKDEP_DECLS)
 {
-	return rt_mutex_lock_interruptible(lock, 0);
+	return rt_mutex_lock_interruptible(lock, 0 __LOCKDEP_PARAMS);
 }
 
 int __lockfunc _mutex_lock_interruptible(struct mutex *lock)
@@ -144,7 +144,7 @@
 
 int __lockfunc _mutex_trylock(struct mutex *lock)
 {
-	int ret = rt_mutex_trylock(&lock->lock);
+	int ret = __rt_mutex_trylock(&lock->lock __LOCKDEP_PARAMS_L0);
 
 	if (ret)
 		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
@@ -163,21 +163,28 @@
 /*
  * rwlock_t functions
  */
-int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+static
+int __lockfunc __rt_write_trylock(rwlock_t *rwlock __LOCKDEP_DECLS)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	//int ret = __rt_mutex_trylock(&rwlock->lock __LOCKDEP_PARAMS);
+	int ret = __rt_mutex_trylock(&rwlock->lock, rt_mutex_real_owner(&rwlock->lock), _dep_map, _ip);
 
 	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
 
 	return ret;
 }
+
+int __lockfunc rt_write_trylock(rwlock_t *rwlock)
+{
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
+}
 EXPORT_SYMBOL(rt_write_trylock);
 
 int __lockfunc rt_write_trylock_irqsave(rwlock_t *rwlock, unsigned long *flags)
 {
 	*flags = 0;
-	return rt_write_trylock(rwlock);
+	return __rt_write_trylock(rwlock __LOCKDEP_PARAMS_L0);
 }
 EXPORT_SYMBOL(rt_write_trylock_irqsave);
 
@@ -453,7 +460,7 @@
 
 void fastcall rt_down(struct semaphore *sem)
 {
-	rt_mutex_lock(&sem->lock);
+	rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 	__down_complete(sem);
 }
 EXPORT_SYMBOL(rt_down);
@@ -462,7 +469,7 @@
 {
 	int ret;
 
-	ret = rt_mutex_lock_interruptible(&sem->lock, 0);
+	ret = rt_mutex_lock_interruptible(&sem->lock, 0 __LOCKDEP_PARAMS_SEM(sem));
 	if (ret)
 		return ret;
 	__down_complete(sem);
@@ -516,7 +523,7 @@
 	switch (val) {
 	case 0:
 		__rt_mutex_init(&sem->lock, name);
-		rt_mutex_lock(&sem->lock);
+		rt_mutex_lock(&sem->lock __LOCKDEP_PARAMS_SEM(sem));
 		break;
 	default:
 		__rt_mutex_init(&sem->lock, name);
--- linux-2.6.24/kernel/mutex.c	2008-01-24 14:58:37.000000000 -0800
+++ linux-2.6.24.working/kernel/mutex.c	2008-03-13 13:03:06.000000000 -0700
@@ -146,7 +146,7 @@
 	if (old_val == 1)
 		goto done;
 
-	lock_contended(&lock->dep_map, ip);
+//	lock_note_contended(LOCKDEP_PARAMS);
 
 	for (;;) {
 		/*
@@ -184,7 +184,7 @@
 	}
 
 done:
-	lock_acquired(&lock->dep_map);
+//	lock_acquired(&lock->dep_map);
 	/* got the lock - rejoice! */
 	mutex_remove_waiter(lock, &waiter, task_thread_info(task));
 	debug_mutex_set_owner(lock, task_thread_info(task));

Re: [PATCH RT 6/6] lockstat measurement extensions

[Bill Huey (hui)]

Last one. Spin checking against the rtmutex->owner's run queue to
determine if an adaptive spin or not is useful.

bill

Re: [PATCH RT 0/6] lockstat measurement extensions

[Bill Huey (hui)]

Also, this is not meant for inclusion in it's current revision. It'll
need to have various compilation modes
(CONFIG_PREEMPT_RT/CONFIG_LOCK_STAT) combination enabled and such. I'm
handing this over to Peter Morreale and it'll be up to him to decided
what to do next.

However, it does work and should get interesting numbers for the
discussion of whether or not adaptive spins and friends are useful or
not.

bill

Re: [PATCH RT 0/6] lockstat measurement extensions

[Bill Huey (hui)]

Sorry missing the patch. At Thomas's urging, this is inlined:

--- linux-2.6.24/kernel/sched.c	2008-02-25 15:32:05.000000000 -0800
+++ linux-2.6.24.working/kernel/sched.c	2008-03-13 13:53:24.000000000 -0700
@@ -1175,6 +1175,20 @@
 	return cpu_curr(task_cpu(p)) == p;
 }

+int task_spinnable(struct task_struct *p)
+{
+/*
+ * The use of task_curr can crash the system since the struct
thread_info seems
+ * to disappear when dereferenced arbitrarily, so becareful.
+ */
+#ifdef CONFIG_SMP
+	if (p && p->se.on_rq && task_curr(p))
+		return 1;
+#else
+	return 0;
+#endif
+}
+
 /* Used instead of source_load when we know the type == 0 */
 unsigned long weighted_cpuload(const int cpu)
 {
@@ -1239,6 +1253,11 @@
 		      *new_cfsrq = cpu_cfs_rq(old_cfsrq, new_cpu);
 	u64 clock_offset;

+//--billh
+//	if (old_cpu >= NR_CPUS)
+//		panic("bogus cpu id %u\n", old_cpu);
+//
+//
 	clock_offset = old_rq->clock - new_rq->clock;

 #ifdef CONFIG_SCHEDSTATS