From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752947Ab3KOO76 (ORCPT ); Fri, 15 Nov 2013 09:59:58 -0500 Received: from merlin.infradead.org ([205.233.59.134]:54346 "EHLO merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751185Ab3KOO7v (ORCPT ); Fri, 15 Nov 2013 09:59:51 -0500 Date: Fri, 15 Nov 2013 15:59:44 +0100 From: Peter Zijlstra To: Vince Weaver Cc: LKML , Ingo Molnar , Paul Mackerras , Arnaldo Carvalho de Melo Subject: Re: perf sw_event related lockup Message-ID: <20131115145944.GA3694@twins.programming.kicks-ass.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2012-12-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Nov 13, 2013 at 05:45:59PM -0500, Vince Weaver wrote: > Hello > > so with the perf_fuzzer modified to avoid the tracepoint issues, I've > triggered this software-event related soft lockup. > > From what I can gather from the backtraces, they all map to the loop > in do_perf_sw_event() in kernel/events/core.c > > hlist_for_each_entry_rcu(event, head, hlist_entry) { > if (perf_swevent_match(event, type, event_id, data, regs)) > perf_swevent_event(event, nr, data, regs); > } > > is it possible to get stuck in that as an infinite loop? > > below is the dmesg from the lockup, I eventually had to reboot to clear > the problem: > [ 416.755310] NOHZ: local_softirq_pending 100 > [ 452.232000] BUG: soft lockup - CPU#1 stuck for 23s! [perf_fuzzer:7211] > [ 452.232000] RIP: 0010:[] [] __perf_sw_event+0x9a/0x1a5 > [ 452.232000] Call Trace: > [ 452.232000] [] ? __do_page_fault+0x191/0x3f5 > [ 452.232000] [] ? sched_clock_local+0x13/0x76 > [ 452.232000] [] ? sched_clock_local+0x13/0x76 > [ 452.232000] [] ? page_fault+0x22/0x30 > [ 452.232000] [] ? __put_user_4+0x20/0x30 > [ 452.232000] [] ? schedule_tail+0x5c/0x60 > [ 452.232000] [] ? ret_from_fork+0xf/0xb0 > [ 480.232000] RIP: 0010:[] [] __perf_sw_event+0x15a/0x1a5 > [ 480.232000] Call Trace: > [ 480.232000] [] ? __do_page_fault+0x191/0x3f5 > [ 480.232000] [] ? sched_clock_local+0x13/0x76 > [ 480.232000] [] ? sched_clock_local+0x13/0x76 > [ 480.232000] [] ? page_fault+0x22/0x30 > [ 480.232000] [] ? __put_user_4+0x20/0x30 > [ 480.232000] [] ? schedule_tail+0x5c/0x60 > [ 480.232000] [] ? ret_from_fork+0xf/0xb0 > [ 486.528000] RIP: 0010:[] [] delay_tsc+0x23/0x50 > [ 486.528000] Call Trace: > [ 486.528000] [] ? __perf_sw_event+0x186/0x1a5 > [ 486.528000] [] ? __perf_sw_event+0x18e/0x1a5 > [ 486.528000] [] ? __do_page_fault+0x191/0x3f5 > [ 486.528000] [] ? sched_clock_local+0x13/0x76 > [ 486.528000] [] ? sched_clock_local+0x13/0x76 > [ 486.528000] [] ? page_fault+0x22/0x30 > [ 486.528000] [] ? __put_user_4+0x20/0x30 > [ 486.528000] [] ? schedule_tail+0x5c/0x60 > [ 486.528000] [] ? ret_from_fork+0xf/0xb0 > [ 486.528000] Call Trace: > [ 486.528000] [] ? __perf_sw_event+0x186/0x1a5 > [ 486.528000] [] ? __perf_sw_event+0x18e/0x1a5 > [ 486.528000] [] ? __do_page_fault+0x191/0x3f5 > [ 486.528000] [] ? sched_clock_local+0x13/0x76 > [ 486.528000] [] ? sched_clock_local+0x13/0x76 > [ 486.528000] [] ? page_fault+0x22/0x30 > [ 486.528000] [] ? __put_user_4+0x20/0x30 > [ 486.528000] [] ? schedule_tail+0x5c/0x60 > [ 486.528000] [] ? ret_from_fork+0xf/0xb0 Please enable CONFIG_FRAME_POINTER to get better backtraces, but the above suggests the pagefault swevent, will have a look.