From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7A8A53B6BF3 for ; Tue, 7 Apr 2026 13:17:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567878; cv=none; b=PTi/880UFYYfSnYbCqneFzIqrAmRCXdKvTqgyIIYAw3V5k6ML1mK+MyjP4/9OdAVaTrMPiA/LHKv5FLQpRgFiPJJIud2b3iQFVEzxPeYBZr+rxmN83sSdssTWKr9O9uicCcAQ8wrQr98chtF5KAnHzN6VMPYaJbV6290dVHO7v8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567878; c=relaxed/simple; bh=tUnltGjcHlvxDk6vVHLGfWgevDbKvhXeMPsvKLPUGi0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LOH2mdpnKH9ym3UppNuw6H13eFI+4/LferjzN7g9i9XSteexww0zziOO+d54tcoqrpr+ujKwEPhYp2MpbE/nIDJPE02zdVhnDsjWZi1OqKfAj8vRPtvRYiGHoKYWuVH2iH9Rn9tlEPo1KrAExWounvnNK7Aa3NMyLzg2J0P0Wj0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=i1GCfrQy; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="i1GCfrQy" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0EACC1BB2; Tue, 7 Apr 2026 06:17:50 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DFFA93F7D8; Tue, 7 Apr 2026 06:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567875; bh=tUnltGjcHlvxDk6vVHLGfWgevDbKvhXeMPsvKLPUGi0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i1GCfrQypMEjFouN11jFhy570D5SApBrIpY3oo1ghHbwpi+3FQik0WHbrIoeg4PIy oEy/KcYEqwQtdxl6bbDdhdZ7wsLZvdU4F1CPmrkySts9U6HeKxlfprLXKBbwmurcy+ QYODOJQ13glCj+K16sN6NsPbo3CZqejEPGl8FIig= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 05/10] entry: Split preemption from irqentry_exit_to_kernel_mode() Date: Tue, 7 Apr 2026 14:16:45 +0100 Message-Id: <20260407131650.3813777-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Some architecture-specific work needs to be performed between the state management for exception entry/exit and the "real" work to handle the exception. For example, arm64 needs to manipulate a number of exception masking bits, with different exceptions requiring different masking. Generally this can all be hidden in the architecture code, but for arm64 the current structure of irqentry_exit_to_kernel_mode() makes this particularly difficult to handle in a way that is correct, maintainable, and efficient. The gory details are described in the thread surrounding: https://lore.kernel.org/lkml/acPAzdtjK5w-rNqC@J2N7QTR9R3/ The summary is: * Currently, irqentry_exit_to_kernel_mode() handles both involuntary preemption AND state management necessary for exception return. * When scheduling (including involuntary preemption), arm64 needs to have all arm64-specific exceptions unmasked, though regular interrupts must be masked. * Prior to the state management for exception return, arm64 needs to mask a number of arm64-specific exceptions, and perform some work with these exceptions masked (with RCU watching, etc). While in theory it is possible to handle this with a new arch_*() hook called somewhere under irqentry_exit_to_kernel_mode(), this is fragile and complicated, and doesn't match the flow used for exception return to user mode, which has a separate 'prepare' step (where preemption can occur) prior to the state management. To solve this, refactor irqentry_exit_to_kernel_mode() to match the style of {irqentry,syscall}_exit_to_user_mode(), moving preemption logic into a new irqentry_exit_to_kernel_mode_preempt() function, and moving state management in a new irqentry_exit_to_kernel_mode_after_preempt() function. The existing irqentry_exit_to_kernel_mode() is left as a caller of both of these, avoiding the need to modify existing callers. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon --- include/linux/irq-entry-common.h | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) Thomas/Peter/Andy, as mentioned on IRC, I haven't created kerneldoc comments for these new functions because the existing comments don't seem all that consistent (e.g. for user mode vs kernel mode), and I suspect we want to rewrite them all in one go for wider consistency. I'm happy to respin this, or to follow-up with that as per your preference. Mark. diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-common.h index 2206150e526d8..24830baa539c6 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -421,10 +421,18 @@ static __always_inline irqentry_state_t irqentry_enter_from_kernel_mode(struct p return ret; } -static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs, irqentry_state_t state) +static inline void irqentry_exit_to_kernel_mode_preempt(struct pt_regs *regs, irqentry_state_t state) { - lockdep_assert_irqs_disabled(); + if (regs_irqs_disabled(regs) || state.exit_rcu) + return; + + if (IS_ENABLED(CONFIG_PREEMPTION)) + irqentry_exit_cond_resched(); +} +static __always_inline void +irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_state_t state) +{ if (!regs_irqs_disabled(regs)) { /* * If RCU was not watching on entry this needs to be done @@ -443,9 +451,6 @@ static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs, i } instrumentation_begin(); - if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); - /* Covers both tracing and lockdep */ trace_hardirqs_on(); instrumentation_end(); @@ -459,6 +464,17 @@ static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs, i } } +static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs, irqentry_state_t state) +{ + lockdep_assert_irqs_disabled(); + + instrumentation_begin(); + irqentry_exit_to_kernel_mode_preempt(regs, state); + instrumentation_end(); + + irqentry_exit_to_kernel_mode_after_preempt(regs, state); +} + /** * irqentry_enter - Handle state tracking on ordinary interrupt entries * @regs: Pointer to pt_regs of interrupted context -- 2.30.2