From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABA844418FA for ; Thu, 26 Feb 2026 16:37:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=216.40.44.10 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772123866; cv=none; b=O2SvXx9cmChSubO0ceqSkPmz3KbkZtWYWzmXCAACfU8kZHdEUvFJmmZ4yqCyt4NBSKzrgS5P6AzoGZcBEwA+Ed4KOGdVdOv172iE1BOCwdb/V0m5KrdFtAILj3HIYK8cLJ7LIZMUHZe823tUC5CDTi2TKFoYsaekCDJ/gBIzpgw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772123866; c=relaxed/simple; bh=XpBSmgpMsaMoutF/YCCxrxztWj7XXQ0lgLi9u3Jwd7E=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=l4/Lro31KDoxIWCEDqyW7oTAH53qMEbx0PHhVf2oziAsLNJGhsMbdUmJOxELkhwl73dBkFCjJJc+vUA/0EUScmR7IxXKqZVATDyHaK3JaaADl8oHblsajCZVQW8Gc6UDYSe2VKv7ybccBWPowH1uBs+pxh6WWMxAB7xEOF7OkEQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org; spf=pass smtp.mailfrom=goodmis.org; arc=none smtp.client-ip=216.40.44.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=goodmis.org Received: from omf02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BC5A9534EC; Thu, 26 Feb 2026 16:37:38 +0000 (UTC) Received: from [HIDDEN] (Authenticated sender: rostedt@goodmis.org) by omf02.hostedemail.com (Postfix) with ESMTPA id 8394880012; Thu, 26 Feb 2026 16:37:36 +0000 (UTC) Date: Thu, 26 Feb 2026 11:37:58 -0500 From: Steven Rostedt To: Bert Karwatzki Cc: Sebastian Andrzej Siewior , Tejun Heo , Thomas Gleixner , calvin@wbinvd.org, dschatzberg@meta.com, peterz@infradead.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev Subject: Re: ~90s reboot delay with v6.19 and PREEMPT_RT Message-ID: <20260226113758.6ff33946@gandalf.local.home> In-Reply-To: References: <20260219164648.3014-1-spasswolf@web.de> <20260225154341.32AjXoVi@linutronix.de> <6abee0b4bca811680a7f7fb6ab70bf27c6805cf5.camel@web.de> <20260225165954.180aRkaV@linutronix.de> <20260225223135.vjfaFo0c@linutronix.de> X-Mailer: Claws Mail 3.20.0git84 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 8394880012 X-Stat-Signature: d45o7qyzcnx6hq575jujp9oi6mjt3yf9 X-Rspamd-Server: rspamout08 X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Session-ID: U2FsdGVkX1+vuCumGQGyTlK/Iq8XbFJTgTB97ZA30LQ= X-HE-Tag: 1772123856-826326 X-HE-Meta: U2FsdGVkX18vnu2wI3HCpGkpE5xdG+3XWJ6wGKVyOc2XF+WhboEBsJVyM30Y3S/xv8TJ8bXs8G9bVhnR/9P3zUkNzy8n42MHQfVrQvk2TcoLduTS3btzUfzwZ57pyfQExTN/neuE1k14ymlYlDdyKb3AZsSoELRxE+6rl9u09dQfxpT8K/24j4RrfFu5CqpeBSDLjPlrl3t1agLqm0ZINrEV4QeRl9sjh+GMGGdVlXxbZzh5EKgqCv3WVPKHJLPsP6Jmd49HpZ8ssuDZcxKhTXUqo7kdV5wev6XplLvvyHf2gAOBOdQ8uuwQ1e4afLQW On Thu, 26 Feb 2026 14:24:34 +0100 Bert Karwatzki wrote: > I think I've found the reason for the stall. I was looking at commit 9311e6c29b34 > ("cgroup: Fix sleeping from invalid context warning on PREEMPT_RT"), > and noticed this: > > +static void __init cgroup_rt_init(void) > +{ > + int cpu; > + > + for_each_possible_cpu(cpu) { > + init_llist_head(per_cpu_ptr(&cgrp_dead_tasks, cpu)); > + per_cpu(cgrp_dead_tasks_iwork, cpu) = > + IRQ_WORK_INIT_LAZY(cgrp_dead_tasks_iwork_fn); > + } > > IRQ_WORK_INIT_LAZY() expands to __IRQ_WORK_INIT(_func, IRQ_WORK_LAZY): > (in include/linux/irq_work.h) > IRQ_WORK_LAZY is declare in include/linux/smp-types.h: > IRQ_WORK_LAZY = 0x04, /* No IPI, wait for tick */ > > The "wait for tick" gave me an idea as I'm also using > CONFIG_NO_HZ_FULL=y If you disable NO_HZ_FULL, does the problem also go away? I wonder if you add irq events and trace a good and bad boot to see if it definitely shows the delayed tick being an issue. echo 1 > /sys/kernel/tracing/instances/boot_map/events/irq/enable -- Steve