From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752380AbcCARtO (ORCPT <rfc822;w@1wt.eu>);
	Tue, 1 Mar 2016 12:49:14 -0500
Received: from casper.infradead.org ([85.118.1.10]:39477 "EHLO
	casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1750967AbcCARtN (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 1 Mar 2016 12:49:13 -0500
Date: Tue, 1 Mar 2016 18:49:03 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>, "Liang, Kan" <kan.liang@intel.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Ingo Molnar <mingo@kernel.org>, Stephane Eranian <eranian@google.com>,
        Wang Nan <wangnan0@huawei.com>,
        "zheng.z.yan@intel.com" <zheng.z.yan@intel.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [BUG] Core2 cpu triggers hard lockup with perf test
Message-ID: <20160301174903.GX6356@twins.programming.kicks-ass.net>
References: <20160227123636.GB30858@krava.redhat.com>
 <37D7C6CF3E00A74B8858931C1DB2F0770589EC94@SHSMSX103.ccr.corp.intel.com>
 <20160301091703.GN6356@twins.programming.kicks-ass.net>
 <20160301110651.GA15260@krava.redhat.com>
 <20160301145105.GQ5083@two.firstfloor.org>
 <20160301145909.GS6356@twins.programming.kicks-ass.net>
 <20160301171722.GA2666@krava.redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160301171722.GA2666@krava.redhat.com>
User-Agent: Mutt/1.5.21 (2012-12-30)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 01, 2016 at 06:17:22PM +0100, Jiri Olsa wrote:

> [  125.982977]  [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M
> [  125.982977]  [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M
> [  125.982977]  [<ffffffff8100ae7b>] ? __intel_pmu_enable_all.isra.11+0x4b/0xd0^M
> [  125.982977]  <<EOE>>  [<ffffffff8100af10>] intel_pmu_enable_all+0x10/0x20^M
> [  125.982977]  [<ffffffff81006283>] x86_pmu_enable+0x263/0x2f0^M
> [  125.982977]  [<ffffffff811632d2>] perf_pmu_enable+0x22/0x30^M
> [  125.982977]  [<ffffffff81163f51>] ctx_resched+0x51/0x60^M
> [  125.982977]  [<ffffffff81164b09>] perf_event_exec+0x109/0x150^M
> [  125.982977]  [<ffffffff811fff7d>] setup_new_exec+0x6d/0x1a0^M
> [  125.982977]  [<ffffffff8125104a>] load_elf_binary+0x37a/0x10e0^M
> [  125.982977]  [<ffffffff811a06c2>] ? get_user_pages+0x52/0x60^M
> [  125.982977]  [<ffffffff811fe32e>] search_binary_handler+0x9e/0x1e0^M
> [  125.982977]  [<ffffffff811ffccd>] do_execveat_common.isra.37+0x54d/0x6e0^M
> [  125.982977]  [<ffffffff812000ea>] SyS_execve+0x3a/0x50^M
> [  125.982977]  [<ffffffff81679065>] stub_execve+0x5/0x5^M
> [  125.982977]  [<ffffffff81678dd7>] ? entry_SYSCALL_64_fastpath+0x12/0x6a^M

> the exception addr is on wrmsr:
> 
> ffffffff8100ae30 <__intel_pmu_enable_all.isra.11>:
> ffffffff8100ae30:       e8 bb 02 67 00          callq  ffffffff8167b0f0 <__fentry__>
> ffffffff8100ae35:       55                      push   %rbp
> ffffffff8100ae36:       48 89 e5                mov    %rsp,%rbp
> ffffffff8100ae39:       41 54                   push   %r12
> ffffffff8100ae3b:       41 89 fc                mov    %edi,%r12d
> ffffffff8100ae3e:       53                      push   %rbx
> ffffffff8100ae3f:       48 c7 c3 80 a3 00 00    mov    $0xa380,%rbx
> ffffffff8100ae46:       65 48 03 1d d2 f2 ff    add    %gs:0x7efff2d2(%rip),%rbx        # a120 <this_cpu_off>
> ffffffff8100ae4d:       7e
> ffffffff8100ae4e:       e8 6d 49 00 00          callq  ffffffff8100f7c0 <intel_pmu_pebs_enable_all>
> ffffffff8100ae53:       41 0f b6 fc             movzbl %r12b,%edi
> ffffffff8100ae57:       e8 94 58 00 00          callq  ffffffff810106f0 <intel_pmu_lbr_enable_all>
> ffffffff8100ae5c:       48 8b 83 68 0c 00 00    mov    0xc68(%rbx),%rax
> ffffffff8100ae63:       b9 8f 03 00 00          mov    $0x38f,%ecx
> ffffffff8100ae68:       48 f7 d0                not    %rax
> ffffffff8100ae6b:       48 23 05 26 80 ad 00    and    0xad8026(%rip),%rax        # ffffffff81ae2e98 <x86_pmu+0x138>
> ffffffff8100ae72:       48 89 c2                mov    %rax,%rdx
> ffffffff8100ae75:       48 c1 ea 20             shr    $0x20,%rdx
> ffffffff8100ae79:       0f 30                   wrmsr
> 

That's the PERF_GLOBAL_CTRL, right? But it must have succeeded,
otherwise the NMI watchdog would never have fired.

Something is hosed alright.

I think I've seen my IVB-EP do something similar. But mostly that
machine gets stuck in intel_bts_enable_local().