From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966079AbXGSTZl (ORCPT ); Thu, 19 Jul 2007 15:25:41 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S965630AbXGSTZQ (ORCPT ); Thu, 19 Jul 2007 15:25:16 -0400 Received: from mail.windriver.com ([147.11.1.11]:49088 "EHLO mail.wrs.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965500AbXGSTZM (ORCPT ); Thu, 19 Jul 2007 15:25:12 -0400 Date: Thu, 19 Jul 2007 12:24:02 -0700 (PDT) From: Jeremy Katz X-X-Sender: jkatz@localhost.localdomain To: Thomas Gleixner cc: linux-kernel@vger.kernel.org, Andrew Morton , Ingo Molnar , Oleg Nesterov , Stable Team Subject: Re: [PATCH] posix-timer: fix deletion race In-Reply-To: <1184824221.12353.526.camel@chaos> Message-ID: References: <1184662429.12353.426.camel@chaos> <1184703427.12353.476.camel@chaos> <1184738755.12353.481.camel@chaos> <1184741938.12353.484.camel@chaos> <1184824221.12353.526.camel@chaos> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-OriginalArrivalTime: 19 Jul 2007 19:24:03.0162 (UTC) FILETIME=[5DA903A0:01C7CA3A] Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 19 Jul 2007, Thomas Gleixner wrote: > On Wed, 2007-07-18 at 16:43 -0700, Jeremy Katz wrote: >> On Wed, 18 Jul 2007, Jeremy Katz wrote: >> >>> On Wed, 18 Jul 2007, Thomas Gleixner wrote: >>> >>>>> Also can you please enable CONFIG_PROVE_LOCKING, which might catch any >>>>> locking problem, which might be related to this. >>>> >>>> Another test: Can you please disable CONFIG_SCHED_SMT to narrow it down >>>> further ? >>> >>> I'll try both of these. >> >> I'm still seeing the sys_timer_delete version with your patch, and >> sys_timer_delete and posix_timer_event without. The itimer_delete version >> is currently missing in action, but getting any particular one to perform >> on demand is currently not in my power. > > Ok, let me summarize: > > 2.6.22 + hrt6 > > Both problems are there whether CONFIG_SCHED_SMT is on or not If both means BUG triggered in sigqueue_free and send_sigqueue, then yes. > > 2.6.22 + hrt6 + posixtimer patch > > Both problems are there whether CONFIG_SCHED_SMT is on > The timer callback problem is gone when CONFIG_SCHED_SMT is off It turns out that the callback problem just took longer to show up with CONFIG_SCHED_SMT off. I saw this when I came in this morning: ------------[ cut here ]------------ Kernel BUG at c012682d [verbose debug info unavailable] invalid opcode: 0000 [#6] SMP Modules linked in: CPU: 3 EIP: 0060:[] Not tainted VLI EFLAGS: 00010046 (2.6.22.1-WR1.4aq_cgl #5) EIP is at send_sigqueue+0xe0/0xe8 eax: 00000020 ebx: f735e7d8 ecx: f73ea030 edx: f735e7d8 esi: f73ea030 edi: 00000020 ebp: c050cf84 esp: c050cf70 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 Process hrtm_test (pid: 11102, ti=c050c000 task=f6a2d550 task.ti=f6fb7000) Stack: 00000000 f722c89c f722c894 f722c8e0 f735e864 c050cf94 c012d519 f722c894 00000025 c050cfc0 c012d595 c2a41990 c2a41980 c050cfb4 00000046 00000000 00000282 c012d54d f722c8e0 c2a41980 c050cfdc c0131c20 00000000 c2a41a18 Call Trace: [] show_trace_log_lvl+0x1a/0x30 [] show_stack_log_lvl+0x8d/0xaa [] show_registers+0x1cd/0x2cb [] die+0x113/0x207 [] do_trap+0x8f/0xc6 [] do_invalid_op+0x88/0x92 [] error_code+0x72/0x78 [] posix_timer_event+0x71/0xa5 [] posix_timer_fn+0x48/0xdd [] run_hrtimer_softirq+0x5a/0xba [] __do_softirq+0x7e/0xf9 [] do_softirq+0x8c/0x101 [] irq_exit+0x4b/0x4d [] smp_apic_timer_interrupt+0x2f/0x39 [] apic_timer_interrupt+0x33/0x38 [] __sigqueue_free+0x24/0x33 [] sigqueue_free+0x1f/0x75 [] release_posix_timer+0x1b/0x7a [] sys_timer_delete+0xda/0x10f [] syscall_call+0x7/0xb ======================= Code: 45 ec 01 00 00 00 eb c8 89 fa 89 f0 e8 ac 99 06 00 eb 93 81 7b 14 fe ff 01 00 75 13 83 43 1c 01 eb ae c7 45 ec ff ff ff ff eb b8 <0f> 0b eb fe 0f 0b eb fe 55 89 e5 57 89 c7 56 89 ce 53 89 d3 83 EIP: [] send_sigqueue+0xe0/0xe8 SS:ESP 0068:c050cf70 Kernel panic - not syncing: Fatal exception in interrupt BUG: spinlock lockup on CPU#2, hrtm_test/1355, f722c89c [] show_trace_log_lvl+0x1a/0x30 [] show_trace+0x12/0x14 [] dump_stack+0x16/0x18 [] __spin_lock_debug+0xb3/0xc5 [] _raw_spin_lock+0x55/0x73 [] _spin_lock+0x32/0x38 [] lock_timer+0x3d/0xab [] sys_timer_delete+0x2b/0x10f [] syscall_call+0x7/0xb ======================= BUG: spinlock lockup on CPU#0, hrtm_test/1374, c04918d0 BUG: spinlock lockup on CPU#1, hrtm_test/1387, c04918d0 [] show_trace_log_lvl+0x1a/0x30 [] show_trace+0x12/0x14 [] dump_stack+0x16/0x18 [] __spin_lock_debug+0xb3/0xc5 [] _raw_spin_lock+0x55/0x73 [] _spin_lock_irqsave+0x3f/0x4b [] lock_timer+0x1e/0xab [] sys_timer_delete+0x2b/0x10f [] syscall_call+0x7/0xb ======================= [] show_trace_log_lvl+0x1a/0x30 [] show_trace+0x12/0x14 [] dump_stack+0x16/0x18 [] __spin_lock_debug+0xb3/0xc5 [] _raw_spin_lock+0x55/0x73 [] _spin_lock_irqsave+0x3f/0x4b [] lock_timer+0x1e/0xab [] sys_timer_delete+0x2b/0x10f [] syscall_call+0x7/0xb ======================= Jeremy