Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Norval Watson]

I have tried to build 2.6.12-rc3 kernel with Ingo's
realtime-preempt-2.6.12-rc3-V0.7.46-02 on amd64

When I enabled option 4, Complete Realtime, make bzImage failed as
described in Lee Revell's Apr 12 lkml post:
http://lkml.org/lkml/2005/4/12/523

"HOSTCC scripts/bin2c 
CC arch/x86_64/kernel/asm-offsets.s 
CHK include/asm-x86_64/offset.h 
UPD include/asm-x86_64/offset.h 
CC init/main.o 
In file included from include/linux/rwsem.h:38, 
from include/linux/kobject.h:24, 
from include/linux/module.h:19, 
from init/main.c:16: 
include/asm/rwsem.h:55: error: redefinition of `struct rw_semaphore' 
etc etc......"

When I started again with option 3, Low Latency Desktop, the build got a
bit further before hanging: (Full ouput:
http://www.longforest.com/index.php?option=com_content&task=view&id=133&Itemid=2 )

"
  CC      arch/x86_64/kernel/head64.o
  CC      arch/x86_64/kernel/init_task.o
arch/x86_64/kernel/init_task.c:17: warning: implicit declaration of
function `__RWSEM_INITIALIZER'
arch/x86_64/kernel/init_task.c:17: warning: missing braces around
initializer

...etc etc etc...

 constant
arch/x86_64/kernel/init_task.c:17: error: (near initialization for
`init_mm.default_kioctx')
make[1]: *** [arch/x86_64/kernel/init_task.o] Error 1
make: *** [arch/x86_64/kernel] Error 2"

Please CC any advice (or abuse!),
I can supply further info if required
Cheers,
Norv

[patch] Real-Time Preemption, -RT-2.6.12-rc1-V0.7.41-10

[Ingo Molnar]

i have released the -V0.7.41-10 Real-Time Preemption patch, which can be 
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this release fixes two bugs:

 - one affecting SMP systems & RCU (the missing smp_mb()s)

 - the other one in net/xfrm/xfrm_policy.c, affecting systems where 
   network interfaces (or pseudo-interfaces) are frequently 
   created/destroyed

i've also added a new debugging feature which is activated if 
RT_DEADLOCK_DETECT is enabled: the checking of active locks in freed 
memory.

This catches a dangerous category of bugs which the upstream kernel can 
silently ignore (because there locks are quite 'passive'), while the -RT 
kernel will often go down in flames sometime later (it has lists within 
the lock, etc.). This mechanism found the xfrm_policy.c bug:

 BUG: events/0/4, active lock [e94a8cdc(e94a8cd0-e94a90dc)] freed!
  [<c0103e31>] dump_stack+0x1e/0x20 (20)
  [<c0136fa2>] check_no_locks_freed+0x158/0x210 (60)
  [<c01477e9>] kfree+0x59/0x15a (48)
  [<c03638db>] xfrm_policy_gc_task+0x6f/0x7e (28)
  [<c012f3c7>] worker_thread+0x1c1/0x26c (132)
  [<c01333e9>] kthread+0x95/0xbd (48)
  [<c01012c9>] kernel_thread_helper+0x5/0xb (1039269908)

so i'd expect more such bugs to be found. If you had stability problems 
under PREEMPT_RT (hangs, crashes), please enable RT_DEADLOCK_DETECT and 
try to reproduce the problem and send me the resulting log messages.

to create a -V0.7.41-10 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc1-V0.7.41-10

	Ingo

[patch] Real-Time Preemption, -RT-2.6.12-rc1-V0.7.41-25

[Ingo Molnar]

i have released the -V0.7.41-25 Real-Time Preemption patch, which can be 
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this release tries to stabilize things some more. In particular i've 
changed 'nocheck' semaphores to not be included in any PI or debugging 
logic. This makes them pretty much stateless and compatible. XFS seems 
to be working much better with this approach, and maybe some of the 
other problematic subsystems (USB/firewire) will improve too.

there's also a latency tracer fix which should cure some of the 
'truncated traces' problems.

i've also included Trond Myklebust's NFS client patch which should 
reduce latencies in that area (on PREEMPT_DESKTOP/VOLUNTARY kernels - 
under PREEMPT_RT the whole NFS code was fully preemptable already).  
It's experimental so be careful if you are using NFS.

to create a -V0.7.41-25 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc1-V0.7.41-25

	Ingo

[patch] Real-Time Preemption, -RT-2.6.12-rc1-V0.7.43-00

[Ingo Molnar]

i have released the -V0.7.43-00 Real-Time Preemption patch, which can be 
downloaded from the usual place:

  http://redhat.com/~mingo/realtime-preempt/

this release too is a step towards more robustness. I found a bug that
caused an infinite recursion and subsequent spontaneous reboot. The bug
was once again related to lock->debug locks, so i decided to get rid of
them altogether: from now on every lock in the -RT domain is debugged.

To be able to use code that relies on incompatible properties of stock
Linux semaphores (and rwsems), i've added a new compile-time
semaphore-type mechanism that enables the easy switching from RT
semaphores to stock semaphores. I've done this conversion for all
subsystems that needed it - e.g. XFS, firewire, USB and SCSI. XFS seems
to be working much better with this approach - BYMMV.

but an unavoidable side-effect is that the whole codebase got turned 
upside down once again, so be careful and expect a few rough edges.  In 
particular keep an eye on new compile-time warnings related to 
semaphores - code that gives a warning might build but it will almost 
certainly not work.

to create a -V0.7.43-00 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc1-V0.7.43-00

	Ingo

[patch] Real-Time Preemption, -RT-2.6.12-rc2-V0.7.44-00

[Ingo Molnar]

i have released the -V0.7.44-00 Real-Time Preemption patch, which can be 
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this is a merge of -43-08 to 2.6.12-rc2.

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc2.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc2-V0.7.44-00

	Ingo

[patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-00

[Ingo Molnar]

i have released the -V0.7.46-00 Real-Time Preemption patch, which can be 
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this is a merge to 2.6.12-rc3, plus the 'ping localhost' fix from 
yang.yi@bmrtech.com.

there are still some unsolved slowdowns probably related to the recent 
plist.h changes.

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc3-V0.7.46-00

	Ingo

[patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this includes fixes from Daniel Walker, which could fix the plist 
related slowdown bugs:

 - fix plist_del_init() argument order bug

 - fix plist related priority calculation bug in __up_mutex()

to build a -V0.7.46-01 tree, the following patches have to be applied:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc3-V0.7.46-01

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

> this includes fixes from Daniel Walker, which could fix the plist 
> related slowdown bugs:

there are still some problems remaining: i just ran Esben Nielsen's 
priority-inheritance validation testsuite, and the plist code gives a 
worst-case latency of 9.0 msecs.

I've reverted the plist changes for now and have uploaded -46-02 - this 
gives the expected 1.0 msec worst-case latencies. Diffing -01 against 
-02 should give you the latest plist snapshot.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

Is this the PriorityInversionTest (pi_test.tgz) or was there another one?


Daniel


On Fri, 22 Apr 2005, Ingo Molnar wrote:

> 
> > this includes fixes from Daniel Walker, which could fix the plist 
> > related slowdown bugs:
> 
> there are still some problems remaining: i just ran Esben Nielsen's 
> priority-inheritance validation testsuite, and the plist code gives a 
> worst-case latency of 9.0 msecs.
> 
> I've reverted the plist changes for now and have uploaded -46-02 - this 
> gives the expected 1.0 msec worst-case latencies. Diffing -01 against 
> -02 should give you the latest plist snapshot.
> 
> 	Ingo
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> Is this the PriorityInversionTest (pi_test.tgz) or was there another one?

yes, it's pi_test.tgz.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

What command line did you use ?

./test --samples 10000 --hertz 128 --tasks 0


Daniel

On Fri, 22 Apr 2005, Ingo Molnar wrote:

> 
> * Daniel Walker <dwalker@mvista.com> wrote:
> 
> > Is this the PriorityInversionTest (pi_test.tgz) or was there another one?
> 
> yes, it's pi_test.tgz.
> 
> 	Ingo
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> What command line did you use ?
> 
> ./test --samples 10000 --hertz 128 --tasks 0

i used:

  ./test --tasks 10 file.hist

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Daniel Walker <dwalker@mvista.com> wrote:
> 
> > What command line did you use ?
> > 
> > ./test --samples 10000 --hertz 128 --tasks 0
> 
> i used:
> 
>   ./test --tasks 10 file.hist

but first i did:

chrt -f 98 -p `pidof 'IRQ 8'`

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

[-- Attachment #1: Type: text/plain, Size: 211 bytes --]

On Fri, 2005-04-22 at 08:55, Ingo Molnar wrote:

> i used:
> 
>   ./test --tasks 10 file.hist


This patch cleanup the delays on increased numbers of tasks. It goes on
to of the current plist snapshot. 

Daniel

[-- Attachment #2: fix_pi_list_init.patch --]
[-- Type: text/x-patch, Size: 1772 bytes --]

Index: linux-2.6.11/include/linux/plist.h
===================================================================
--- linux-2.6.11.orig/include/linux/plist.h	2005-04-22 19:36:54.000000000 +0000
+++ linux-2.6.11/include/linux/plist.h	2005-04-22 19:38:29.000000000 +0000
@@ -251,7 +251,7 @@
 static inline void plist_del_init(struct plist *pl, struct plist *plist)
 {
         plist_del (pl, plist);
-        plist_init(pl, 0);
+        plist_init(pl, INT_MAX);
 }
 
 /* Return the priority a pl node */
Index: linux-2.6.11/include/linux/rt_lock.h
===================================================================
--- linux-2.6.11.orig/include/linux/rt_lock.h	2005-04-21 17:49:26.000000000 +0000
+++ linux-2.6.11/include/linux/rt_lock.h	2005-04-22 22:35:37.000000000 +0000
@@ -95,12 +95,12 @@
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 # define __RT_MUTEX_INITIALIZER(lockname) \
 	{ .wait_lock = RAW_SPIN_LOCK_UNLOCKED, \
-	.wait_list = PLIST_INIT((lockname).wait_list, 0),  \
+	.wait_list = PLIST_INIT((lockname).wait_list, MAX_PRIO),  \
 	.name = #lockname, .file = __FILE__, .line = __LINE__ }
 #else
 # define __RT_MUTEX_INITIALIZER(lockname) \
 	{ .wait_lock = RAW_SPIN_LOCK_UNLOCKED, \
-	PLIST_INIT((lockname).wait_list, 0) }
+	PLIST_INIT((lockname).wait_list, MAX_PRIO) }
 #endif
 /*
  * RW-semaphores are an RT mutex plus a reader-depth count.
Index: linux-2.6.11/kernel/rt.c
===================================================================
--- linux-2.6.11.orig/kernel/rt.c	2005-04-22 15:05:33.000000000 +0000
+++ linux-2.6.11/kernel/rt.c	2005-04-22 22:34:22.000000000 +0000
@@ -939,6 +939,7 @@
 
 	set_task_state(task, TASK_UNINTERRUPTIBLE);
 
+	plist_init (&waiter.list, task->prio);
 	task_blocks_on_lock(&waiter, task, lock, eip);
 
 	TRACE_BUG_ON(!irqs_disabled());

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> On Fri, 2005-04-22 at 08:55, Ingo Molnar wrote:
> 
> > i used:
> > 
> >   ./test --tasks 10 file.hist
> 
> This patch cleanup the delays on increased numbers of tasks. It goes 
> on to of the current plist snapshot.

ok, these fixes for priority init appear to have solved the pi-test 
latencies. I've uploaded the -rc4-RT-V0.7.47-00 patch with the plist 
code re-added again. It's looking good so far on my testboxes.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Inaky Perez-Gonzalez]

>>>>> Ingo Molnar <mingo@elte.hu> writes:

>> this includes fixes from Daniel Walker, which could fix the plist
>> related slowdown bugs:

> there are still some problems remaining: i just ran Esben Nielsen's
> priority-inheritance validation testsuite, and the plist code gives
> a worst-case latency of 9.0 msecs.

With which machine is this? 

I tried to reproduce with V0.7.45-01 in my 2xPIII 0.9 Ghz I cannot get
more than 1.9ms when doing 2000 samples repeatedly (60 iterations and
counting). 

Did you use other parameters to 'test'?

# cnt=0 
# while true
> do echo -n "$cnt "
> ./test --samples 2000 file.hist 2>&1 | grep maximum; cnt=$(($cnt+1))
> done

With 2.6.12-rc3-V0.7.46-02 I get:

...
2 maximum cycle time: 0.003ms
3 maximum cycle time: 1.706ms
4 maximum cycle time: 1.538ms
5 maximum cycle time: 1.168ms
6 maximum cycle time: 0.003ms
7 maximum cycle time: 0.008ms
8 maximum cycle time: 1.821ms
9 maximum cycle time: 1.013ms
10 maximum cycle time: 1.043ms
11 maximum cycle time: 1.787ms
12 maximum cycle time: 1.519ms
13 ...

roughly the same as with 0.7.45-01:

0 maximum cycle time: 1.872ms
1 maximum cycle time: 0.008ms
2 maximum cycle time: 1.947ms
3 maximum cycle time: 1.902ms
4 maximum cycle time: 1.925ms
5 maximum cycle time: 0.002ms
6 maximum cycle time: 1.950ms
7 maximum cycle time: 0.008ms
8 maximum cycle time: 0.008ms
9 maximum cycle time: 0.008ms
10 maximum cycle time: 1.943ms
11 maximum cycle time: 1.173ms
12 maximum cycle time: 0.008ms
13 maximum cycle time: 0.008ms
...


-- 

Inaky

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1025 bytes --]

t
On Fri, 22 Apr 2005, Inaky Perez-Gonzalez wrote:

> >>>>> Ingo Molnar <mingo@elte.hu> writes:
> 
> >> this includes fixes from Daniel Walker, which could fix the plist
> >> related slowdown bugs:
> 
> > there are still some problems remaining: i just ran Esben Nielsen's
> > priority-inheritance validation testsuite, and the plist code gives
> > a worst-case latency of 9.0 msecs.
> 
> With which machine is this? 
> 
> I tried to reproduce with V0.7.45-01 in my 2xPIII 0.9 Ghz I cannot get
> more than 1.9ms when doing 2000 samples repeatedly (60 iterations and
> counting). 
> 
> Did you use other parameters to 'test'?
> 
> # cnt=0 
> # while true
> > do echo -n "$cnt "
> > ./test --samples 2000 file.hist 2>&1 | grep maximum; cnt=$(($cnt+1))
> > done
> 
> With 2.6.12-rc3-V0.7.46-02 I get:

It's still to high , it should be under a millisecond .. 

 
I'm still testing but the times get better with this patch . I was 
initializing the lists to 0, instead of MAX_INT . Let me know if it 
changes anything.


Daniel


[-- Attachment #2: Type: TEXT/PLAIN, Size: 954 bytes --]

Index: linux-2.6.11/include/linux/plist.h
===================================================================
--- linux-2.6.11.orig/include/linux/plist.h	2005-04-22 19:36:54.000000000 +0000
+++ linux-2.6.11/include/linux/plist.h	2005-04-22 19:38:29.000000000 +0000
@@ -251,7 +251,7 @@
 static inline void plist_del_init(struct plist *pl, struct plist *plist)
 {
         plist_del (pl, plist);
-        plist_init(pl, 0);
+        plist_init(pl, INT_MAX);
 }
 
 /* Return the priority a pl node */
Index: linux-2.6.11/kernel/rt.c
===================================================================
--- linux-2.6.11.orig/kernel/rt.c	2005-04-22 15:05:33.000000000 +0000
+++ linux-2.6.11/kernel/rt.c	2005-04-22 19:36:29.000000000 +0000
@@ -939,6 +939,7 @@
 
 	set_task_state(task, TASK_UNINTERRUPTIBLE);
 
+	plist_init (&waiter.list, task->prio);
 	task_blocks_on_lock(&waiter, task, lock, eip);
 
 	TRACE_BUG_ON(!irqs_disabled());

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> > With 2.6.12-rc3-V0.7.46-02 I get:
> 
> It's still to high , it should be under a millisecond ..
>  
> I'm still testing but the times get better with this patch . I was 
> initializing the lists to 0, instead of MAX_INT . Let me know if it 
> changes anything.

could you send me a single patch that adds the plist stuff to the 
current tree? (that way i'll have your latest and i'll be able to 
add/remove it quicker)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

Oops, I missed that you tested a non-plist kernel .. Maybe there are some 
lingering problems in current stuff..

Daniel


On Fri, 22 Apr 2005, Inaky Perez-Gonzalez wrote:


> With 2.6.12-rc3-V0.7.46-02 I get:
> 
> ...
> 2 maximum cycle time: 0.003ms
> 3 maximum cycle time: 1.706ms
> 4 maximum cycle time: 1.538ms
> 5 maximum cycle time: 1.168ms
> 6 maximum cycle time: 0.003ms
> 7 maximum cycle time: 0.008ms
> 8 maximum cycle time: 1.821ms
> 9 maximum cycle time: 1.013ms
> 10 maximum cycle time: 1.043ms
> 11 maximum cycle time: 1.787ms
> 12 maximum cycle time: 1.519ms
> 13 ...
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Lee Revell]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Fri, 2005-04-22 at 08:27 +0200, Ingo Molnar wrote:
> i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
> downloaded from the usual place:
> 
>     http://redhat.com/~mingo/realtime-preempt/

Ingo,

With Mingming's patch to resolve the ext3 allocate-with-reservation
latency, it looks like we are finally getting close to the lower limit
that can be achieved with PREEMPT_DESKTOP.  I've attached the trace of
the lowest latency over several days of testing with
2.6.12-rc3-RT-V0.7.46-02 + Mingming's patch.  It's only 127 usecs, and
IIRC you mentioned previously that this code path can't be made
preemptible in !PREEMPT_RT.

Since Mingming's patch will have to live in -mm for a while, can it be
added to the RT patchset as well?

http://lkml.org/lkml/2005/4/22/127

Lee

[-- Attachment #2: signal-delivery-trace.bz2 --]
[-- Type: application/x-bzip, Size: 1957 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Fri, 2005-04-22 at 08:27 +0200, Ingo Molnar wrote:
> > i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
> > downloaded from the usual place:
> > 
> >     http://redhat.com/~mingo/realtime-preempt/
> 
> Ingo,
> 
> With Mingming's patch to resolve the ext3 allocate-with-reservation 
> latency, it looks like we are finally getting close to the lower limit 
> that can be achieved with PREEMPT_DESKTOP.  I've attached the trace of 
> the lowest latency over several days of testing with 
> 2.6.12-rc3-RT-V0.7.46-02 + Mingming's patch.  It's only 127 usecs, and 
> IIRC you mentioned previously that this code path can't be made 
> preemptible in !PREEMPT_RT.

yeah, signal delivery will have to stay atomic in !PREEMPT_RT kernels.

> Since Mingming's patch will have to live in -mm for a while, can it be 
> added to the RT patchset as well?

i think so - i'll add it to the next patch.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Lee Revell]

On Wed, 2005-05-04 at 10:22 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> > it looks like we are finally getting close to the lower limit 
> > that can be achieved with PREEMPT_DESKTOP ... It's only 127 usecs, and 
> > IIRC you mentioned previously that this code path can't be made 
> > preemptible in !PREEMPT_RT.
> 
> yeah, signal delivery will have to stay atomic in !PREEMPT_RT kernels.
> 

OK.  I found a few more.

When umounting, shrink_dcache_sb() will produce a 3-4 ms. bump.  However
it's not clear this can be made preemptible.  AFAICT the whole thing
needs to be under the dcache_lock.  This one is pretty obvious so I'm
not attaching a trace.

There's also still one path in the VM related to page freeing that can
produce ~500 usec latencies but I don't have a trace handy now.

> > Since Mingming's patch will have to live in -mm for a while, can it be 
> > added to the RT patchset as well?
> 
> i think so - i'll add it to the next patch.

OK, great.

Lee