Well, after turning off hrtimers, I keep getting one bug. A possible soft lockup with the ext3 code. But this didn't seem to be caused by the changes I made. So just to be sure, I ran my test on the vanilla 2.6.13-rc6-rt11 and it gave the same bug too. So, it looks like my changes are now at par with what is out there with the rt11 release. They both give the same bug! ;-) Attached is the test I ran. I did a while : ; do ./test3a_rt ; done Where test3a_rt is a C program that does adding, deleting and reading of files, by different tasks that are each at a different priority. Here's the soft lockup I'm getting: NFSD: starting 90-second grace period BUG: test3a_rt:6138, possible softlockup detected on CPU#0! [] dump_stack+0x23/0x30 (20) [] softlockup_detected+0x41/0x60 (24) [] softlockup_tick+0xf0/0x100 (24) [] update_process_times+0x5b/0x80 (28) [] smp_apic_timer_interrupt+0xee/0x100 (36) [] apic_timer_interrupt+0x21/0x28 (124) [] ext3_add_entry+0xd1/0x210 (60) [] ext3_add_nondir+0x2d/0x80 (32) [] ext3_create+0xbb/0x110 (40) [] vfs_create+0xf1/0x170 (44) [] open_namei+0x61b/0x6e0 (72) [] filp_open+0x3f/0x70 (92) [] sys_open+0x53/0x100 (44) [] syscall_call+0x7/0xb (-8116) --------------------------- | preempt count: 00010000 ] | 0-level deep critical section nesting: ---------------------------------------- ------------------------------ | showing all locks held by: | (test3a_rt/6138 [f7ef61a0, 89]): ------------------------------ #001: [f2eaddac] {(struct semaphore *)(&inode->i_sem)} ... acquired at: open_namei+0xf9/0x6e0 NMI show regs on CPU#0: Pid: 6138, comm: test3a_rt EIP: 0060:[] CPU: 0 EIP is at nmi_show_all_regs+0x89/0xd0 EFLAGS: 00000046 Not tainted (2.6.13-rc6-rt11-real) EAX: 00000000 EBX: 000017fa ECX: 00000060 EDX: 00000080 ESI: 00000000 EDI: 0006c692 EBP: f07dbd48 DS: 007b ES: 007b CR0: 8005003b CR2: b7e37ae0 CR3: 30850000 CR4: 000006d0 [] show_regs+0x164/0x16c (36) [] nmi_watchdog_tick+0x128/0x2b0 (56) [] default_do_nmi+0x7f/0x190 (52) [] do_nmi+0x50/0x60 (24) [] nmi_stack_correct+0x1d/0x22 (72) [] softlockup_detected+0x46/0x60 (24) [] softlockup_tick+0xf0/0x100 (24) [] update_process_times+0x5b/0x80 (28) [] smp_apic_timer_interrupt+0xee/0x100 (36) [] apic_timer_interrupt+0x21/0x28 (124) [] ext3_add_entry+0xd1/0x210 (60) [] ext3_add_nondir+0x2d/0x80 (32) [] ext3_create+0xbb/0x110 (40) [] vfs_create+0xf1/0x170 (44) [] open_namei+0x61b/0x6e0 (72) [] filp_open+0x3f/0x70 (92) [] sys_open+0x53/0x100 (44) [] syscall_call+0x7/0xb (-8116) --------------------------- | preempt count: 00010001 ] | 1-level deep critical section nesting: ---------------------------------------- .. [] .... add_preempt_count+0x1c/0x20 .....[] .. ( <= nmi_watchdog_tick+0x10d/0x2b0) ------------------------------ | showing all locks held by: | (test3a_rt/6138 [f7ef61a0, 89]): ------------------------------ #001: [f2eaddac] {(struct semaphore *)(&inode->i_sem)} ... acquired at: open_namei+0xf9/0x6e0 NMI show regs on CPU#1: Pid: 6137, comm: test3a_rt EIP: 0060:[] CPU: 1 EIP is at do_get_write_access+0x83/0x560 EFLAGS: 00000206 Not tainted (2.6.13-rc6-rt11-real) EAX: 000e201d EBX: f7ffe000 ECX: f71e49bc EDX: f71e49bc ESI: f7f6b3bc EDI: f71e49bc EBP: f1409e94 DS: 007b ES: 007b CR0: 8005003b CR2: b7fdb000 CR3: 326fb000 CR4: 000006d0 [] show_regs+0x164/0x16c (36) [] nmi_watchdog_tick+0x128/0x2b0 (56) [] default_do_nmi+0x7f/0x190 (52) [] do_nmi+0x50/0x60 (24) [] nmi_stack_correct+0x1d/0x22 (132) [] journal_get_write_access+0x35/0x50 (28) [] ext3_orphan_del+0x1a2/0x230 (80) [] ext3_delete_inode+0x73/0x110 (24) [] generic_delete_inode+0x99/0x160 (28) [] generic_drop_inode+0x1d/0x30 (12) [] iput+0x70/0x80 (20) [] sys_unlink+0x110/0x140 (96) [] syscall_call+0x7/0xb (-8116) --------------------------- | preempt count: 00000001 ] | 1-level deep critical section nesting: ---------------------------------------- .. [] .... add_preempt_count+0x1c/0x20 .....[] .. ( <= nmi_watchdog_tick+0x10d/0x2b0) ------------------------------ | showing all locks held by: | (test3a_rt/6137 [f5d437f0, 90]): ------------------------------ #001: [f709cc88] {(struct semaphore *)(&s->s_lock)} ... acquired at: ext3_orphan_del+0x3a/0x230 Well, looking at this further, it seems to release somehow, and continue, since I've got multiple "possible soft lockups", by different threads. But It doesn't seem to go any further. Somethings spinning quite a bit, and something else must be starved. I'll look further into this. So, Ingo, what do you think of the changes so far? Do you feel that it is stable enough to send you an actual real patch. That way we can work together in cleaning it up and get all the other kinks out. Also, let me know if you are getting this. The last couple of messages I sent, never reached me on the LKML, and I sent the last one 4 hours ago. :-( Thomas, I did get your reply. -- Steve