From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BA1D84908D2 for ; Fri, 24 Jul 2026 08:02:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784880129; cv=none; b=d5IRcJri4KMty/I0mvS2lSMgJj3V1xoHLTGbjIfwF4qTorRGezwT2Gv3+5C1vY0mXPYW9UOEf8HwcmF30+eWVgTERQGM1WwG+xojp7Ww2USjg1Tw7zjaHr8TCk+dmg1xnpDEzs0KRKVSAJgvb3hvXSA8xSvNGFwW5545m8pAdjg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784880129; c=relaxed/simple; bh=WQ4xFyt5t+SCsEfKHGUvjPGAiCV+QA9PcHJ02/DqaEI=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=JrOcO1NuH4DsSfngkxs/hynFjNW1MBgnFLq7tlxZnUz/uuAH+dZmBYwYp1dC/lxJuUw1Z+ZFqCcdoVBmy7+TYtsJA/evnu9suEQs8XLd506FKbF6IEjoPk8gKzwevZqyXsnb4ZVtCxf1tvdmbZdGleosnr/0Y0ZjaigWqbBsBsU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=McSP4NbP; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="McSP4NbP" Received: from [192.168.0.105] (unknown [4.194.122.170]) by linux.microsoft.com (Postfix) with ESMTPSA id 983A520B7167; Fri, 24 Jul 2026 01:01:49 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 983A520B7167 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1784880113; bh=2rw6y/sxs9sPzKNdYkOwUr8B9CRiLfuTdglKA4rmF4E=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=McSP4NbPaH1W9QrlrA4uRWhgzCvqGUOr0rwwS0HZSABmzWZa3g9jN/sch1Vr6vR39 7R2wd6DQNvaiGpMES2vE7YANQCD5R+7AHLTpQRG5mUJV6XVziw9WtKB1KWCeTxGXNK 7g0Qi8Db1YgWlgymJYltA2dTxU+qq3ONay0RSaA8= Message-ID: <8ddb4f1e-eb14-478e-a76e-8483b75bfa48@linux.microsoft.com> Date: Fri, 24 Jul 2026 13:31:59 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S To: Anshuman Khandual , Jinjie Ruan , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Marc Zyngier , Thomas Huth , Fuad Tabba , Thomas Gleixner , Pengjie Zhang , mrigendrachaubey , Saurabh Sengar References: <20260722113044.1835365-1-namjain@linux.microsoft.com> <6e13c10f-ccc2-48c3-bf8a-d65133a33e17@huawei.com> <70e8596f-7316-4cc9-90e8-ff06a94c4616@arm.com> Content-Language: en-US From: Naman Jain In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 7/24/2026 10:53 AM, Anshuman Khandual wrote: > > > On 24/07/26 10:16 AM, Naman Jain wrote: >> >> >> On 7/23/2026 11:26 AM, Anshuman Khandual wrote: >>> >>> >>> On 23/07/26 8:19 AM, Jinjie Ruan wrote: >>>> >>>> 在 2026/7/22 19:30, Naman Jain 写道: >>>>> When a secondary CPU fails to come online, __cpu_up() falls back to >>>>> __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot >>>>> distinguish a CPU that never executed head.S (firmware/hypervisor never >>>>> dispatched it, so it never ran a single instruction) from one that >>>>> entered head.S, started executing, and then got stuck somewhere in kernel >>>>> bring-up. Add a change to let us tell those two cases apart, which >>>>> narrows down where to look when a CPU goes missing during boot. >>>> I previously encountered this issue when debugging the parallel startup >>>> of ARM64 secondary cores. It is difficult for the kernel to determine >>>> whether the secondary core is hung in the firmware or whether it has not >>>> executed a single instruction. So I think this motive is reasonable. >>> >>> Why should kernel determine the difference here ? Would not the firmware >>> know if it has started any secondary CPU for the kernel which must have >>> come inside head.S ? If the cpu gets hung inside firmware while starting >>> up then the debug responsibilities belong there instead. >>> >>> Still wondering what's the rationale for this change. >> >> Hello Anshuman, >> This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with 0x0 status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start of head.S and marking itself online at the end of secondary_start_kernel(). > > Why that information is useful ? If firmware is sure to have dispatched > given CPU then should not the early kernel boot failure be debugged via > generally available methods. Is this trying create an alternative ? I am trying to cover the cases, where the kernel does not know that it is stuck, and can thus cannot handle that scenario. Existing CPU_STUCK_IN_KERNEL status with CPU_STUCK_REASON_* are applied only after the kernel detects that something failed. Generally available methods, like adding breakpoints, debug logs and traces, is not always possible due to production system limitations and limited reproducibility of issues. >> There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues. > But wondering if the intent is to further classify failures including > entered ASM, but failed else where during boot for ease in debugging > why not add more into CPU_STUCK_REASON_* ? If we agree on the requirement for this, I can then reuse CPU_STUCK_IN_KERNEL with new CPU_STUCK_REASON_* variables, to proactively mark the CPU as STUCK first and then making checkpoints based on the boot-up progress, and finally changing the status to CPU_BOOT_SUCCESS as usual. I would be adding these checkpoints only on a few logical blocks of secondary CPU boot flow to narrow down the problem. Please let me know your thoughts on it. Regards, Naman