From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754474AbdKAJ1l (ORCPT ); Wed, 1 Nov 2017 05:27:41 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:54970 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754461AbdKAJ1h (ORCPT ); Wed, 1 Nov 2017 05:27:37 -0400 Subject: Re: [PATCH] powerpc/perf: Fix core-imc hotplug callback failure during imc initialization To: Michael Ellerman , Anju T Sudhakar Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org References: <1509443398-26539-1-git-send-email-anju@linux.vnet.ibm.com> <87efpi95wb.fsf@concordia.ellerman.id.au> From: Madhavan Srinivasan Date: Wed, 1 Nov 2017 14:57:00 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <87efpi95wb.fsf@concordia.ellerman.id.au> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 17110109-0016-0000-0000-000004FC2FC8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17110109-0017-0000-0000-00002837BBBC Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-11-01_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1711010133 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wednesday 01 November 2017 06:22 AM, Michael Ellerman wrote: > Anju T Sudhakar writes: > >> Call trace observed during boot: > What's the actual oops? I could recreate this in mambo with CPUS=2 and THREAD=2 Here is the complete stack trace. [    0.045367] core_imc memory allocation for cpu 2 failed [    0.045408] Unable to handle kernel paging request for data at address 0x7d20e2a6f92d03b8 [    0.045443] Faulting instruction address: 0xc0000000000dde18 cpu 0x0: Vector: 380 (Data Access Out of Range) at [c0000000fd1cb890]     pc: c0000000000dde18: event_function_call+0x28/0x14c     lr: c0000000000dde00: event_function_call+0x10/0x14c     sp: c0000000fd1cbb10    msr: 9000000000009033    dar: 7d20e2a6f92d03b8   current = 0xc0000000fd15da00   paca    = 0xc00000000fff0000   softe: 0        irq_happened: 0x01     pid   = 11, comm = cpuhp/0 Linux version 4.14.0-rc7-00014-g0a08377b127b (maddy@SrihariSrinidhi) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.1)) #5 SMP Wed Nov 1 14:12:27 IST 2017 enter ? for help [c0000000fd1cbb10] 0000000000000000 (unreliable) [c0000000fd1cbba0] c0000000000de180 perf_remove_from_context+0x30/0x9c [c0000000fd1cbbe0] c0000000000e9108 perf_pmu_migrate_context+0x9c/0x224 [c0000000fd1cbc60] c0000000000682e0 ppc_core_imc_cpu_offline+0xdc/0x144 [c0000000fd1cbcb0] c000000000070568 cpuhp_invoke_callback+0xe4/0x244 [c0000000fd1cbd10] c000000000070824 cpuhp_thread_fun+0x15c/0x1b0 [c0000000fd1cbd60] c00000000008e8cc smpboot_thread_fn+0x1e0/0x200 [c0000000fd1cbdc0] c00000000008ae58 kthread+0x150/0x158 [c0000000fd1cbe30] c00000000000b464 ret_from_kernel_thread+0x5c/0x78 > >> [c000000ff38ffb80] c0000000002ddfac perf_pmu_migrate_context+0xac/0x470 >> [c000000ff38ffc40] c00000000011385c ppc_core_imc_cpu_offline+0x1ac/0x1e0 >> [c000000ff38ffc90] c000000000125758 cpuhp_invoke_callback+0x198/0x5d0 >> [c000000ff38ffd00] c00000000012782c cpuhp_thread_fun+0x8c/0x3d0 >> [c000000ff38ffd60] c0000000001678d0 smpboot_thread_fn+0x290/0x2a0 >> [c000000ff38ffdc0] c00000000015ee78 kthread+0x168/0x1b0 >> [c000000ff38ffe30] c00000000000b368 ret_from_kernel_thread+0x5c/0x74 >> >> While registering the cpuhoplug callbacks for core-imc, if we fails >> in the cpuhotplug online path for any random core (either because opal call to >> initialize the core-imc counters fails or because memory allocation fails for >> that core), ppc_core_imc_cpu_offline() will get invoked for other cpus who >> successfully returned from cpuhotplug online path. >> >> But in the ppc_core_imc_cpu_offline() path we are trying to migrate the event >> context, when core-imc counters are not even initialized. Thus creating the >> above stack dump. >> >> Add a check to see if core-imc counters are enabled or not in the cpuhotplug >> offline path before migrating the context to handle this failing scenario. > Why do we need a bool to track this? Can't we just check the data > structure we're deinitialising has been initialised? My bad. yes we could do that. Something like this will work? @@ -606,6 +608,20 @@ static int ppc_core_imc_cpu_offline(unsigned int cpu)         if (!cpumask_test_and_clear_cpu(cpu, &core_imc_cpumask))                 return 0; +       /* +        * Check whether core_imc is registered. We could end up here +        * if the cpuhotplug callback registration fails. i.e, callback +        * invokes the offline path for all sucessfully registered cpus. +        * At this stage, core_imc pmu will not be registered and we +        * should return here. +        * +        * We return with a zero since this is not a offline failure. +        * And cpuhp_setup_state() returns the actual failure reason +        * to the caller, which inturn will call the cleanup routine. +        */ +       if (!core_imc_pmu->pmu.event_init) +               return 0; +         /* Find any online cpu in that core except the current "cpu" */         ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu); > > Doesn't this also mean we won't cleanup the initialisation for any CPUs > that have been initialised? No, we do. cpuhp_setup_state() returns the actual failure reason which in-turn calls the cleanup routine. Thanks for review comments Maddy > > cheers > >> diff --git a/arch/powerpc/perf/imc-pmu.c b/arch/powerpc/perf/imc-pmu.c >> index 8812624..08139f9 100644 >> --- a/arch/powerpc/perf/imc-pmu.c >> +++ b/arch/powerpc/perf/imc-pmu.c >> @@ -30,6 +30,7 @@ static struct imc_pmu *per_nest_pmu_arr[IMC_MAX_PMUS]; >> static cpumask_t nest_imc_cpumask; >> struct imc_pmu_ref *nest_imc_refc; >> static int nest_pmus; >> +static bool core_imc_enabled; >> >> /* Core IMC data structures and variables */ >> >> @@ -607,6 +608,19 @@ static int ppc_core_imc_cpu_offline(unsigned int cpu) >> if (!cpumask_test_and_clear_cpu(cpu, &core_imc_cpumask)) >> return 0; >> >> + /* >> + * See if core imc counters are enabled or not. >> + * >> + * Suppose we reach here from core_imc_cpumask_init(), >> + * since we failed at the cpuhotplug online path for any random >> + * core (either because opal call to initialize the core-imc counters >> + * failed or because memory allocation failed). >> + * We need to check whether core imc counters are enabled or not before >> + * migrating the event context from cpus in the other cores. >> + */ >> + if (!core_imc_enabled) >> + return 0; >> + >> /* Find any online cpu in that core except the current "cpu" */ >> ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu); >> >> @@ -1299,6 +1313,7 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id >> return ret; >> } >> >> + core_imc_enabled = true; >> break; >> case IMC_DOMAIN_THREAD: >> ret = thread_imc_cpu_init(); >> -- >> 2.7.4