From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mout.web.de (mout.web.de [217.72.192.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66975324B17 for ; Thu, 26 Feb 2026 13:25:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.72.192.78 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772112308; cv=none; b=ZcSxD3MNXHv963SSIBiUMQyFr5MnGhdL+VrosD5oI5RUAuftNXqgMZDjLwTn1rcAeOie5j5IgOy4ZSay/Ejn3Kdc7K3s63Nx/nXql2gr4DFv50jMkC5EgRBnwCGuEv1bZg/XqJ2Z+UKtQOPI9ZZ+FlYQ46Vjzzu8rg7OMvJ5KdY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772112308; c=relaxed/simple; bh=1JJFPASpIIjQR26HiaZE86XKPtyZGDjamzUt6TFCAeQ=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=G+rvQRkWNpNeO+HtpGCskQyHratFPeDzsjIJZnR93NSYag7CDDCWlwHIFlEREbQWTfgReHUEBpaBfcsapFopK7CngZsBw95bo7x8IRsvIU2vMc76gXCQ0XuvarSpITq9P5aAI2m9rrYeLN9n1pkdcYBpKXZaKpHUUaRRDcA1mLs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=web.de; spf=pass smtp.mailfrom=web.de; dkim=pass (2048-bit key) header.d=web.de header.i=spasswolf@web.de header.b=Gu/itmoU; arc=none smtp.client-ip=217.72.192.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=web.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=web.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=web.de header.i=spasswolf@web.de header.b="Gu/itmoU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=web.de; s=s29768273; t=1772112276; x=1772717076; i=spasswolf@web.de; bh=4lw6JK1LbdqgNtXX572wRAIGDxuhI0Nq+F2ZSBrf36Y=; h=X-UI-Sender-Class:Message-ID:Subject:From:To:Cc:Date:In-Reply-To: References:Content-Type:MIME-Version:Content-Transfer-Encoding:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=Gu/itmoUEAovgNrM8bM85BtdkJD4WqE98x8TaHzVaVywrfRES7k2KNIQdXzkJNm6 53RGyGeqATN9iEhPuPjqP83gFbAGZmtEdoDZChpnVesp6pZOcyfiFiZA2MtV838jL ABT6z61zTo3WDauBoVjxUDPCU2ZB1Wr1i9+TGsKRRFkb3pOrvlzun8CnxzVMe2z1c tAG6WteP+s24yE2oO7xbWl0QjtpdZflRexpnLPDQHMCUZbH4MwzBQIotebO8YOmRU Az2MUUtFpecFyW1vILSsQwffprstnLNLrZEE5Hd9lj6ENrf35DJPPSxeZh2gTdKv/ 2/AilltujIKMB9bwpw== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from client.hidden.invalid by smtp.web.de (mrweb105 [213.165.67.124]) with ESMTPSA (Nemesis) id 1N4vNe-1vWPVQ21au-015Tdb; Thu, 26 Feb 2026 14:24:36 +0100 Message-ID: Subject: Re: ~90s reboot delay with v6.19 and PREEMPT_RT From: Bert Karwatzki To: Sebastian Andrzej Siewior Cc: Tejun Heo , Thomas Gleixner , calvin@wbinvd.org, dschatzberg@meta.com, peterz@infradead.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, Steven Rostedt , spasswolf@web.de Date: Thu, 26 Feb 2026 14:24:34 +0100 In-Reply-To: <20260225223135.vjfaFo0c@linutronix.de> References: <20260219164648.3014-1-spasswolf@web.de> <20260225154341.32AjXoVi@linutronix.de> <6abee0b4bca811680a7f7fb6ab70bf27c6805cf5.camel@web.de> <20260225165954.180aRkaV@linutronix.de> <20260225223135.vjfaFo0c@linutronix.de> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.56.2-0+deb13u1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:0FG2oamYUeoAgbei93vfIEM39OHsjbrwJe4zzELiY+a2KFO6tSg 746gK7hxP3nEtIcikoDdmD8gfNGapd/mfRZnC/WgS/BbIHlejIfa0fdEMesguevFTw83Ydb Ni4YXo8ewCMR1yZyshRuMznkwDGPljXSuuXefF1sp15+ybUztfxkE8K5e0VzyUtwsH0wxu8 KsDySJ1BvMTD7kR9OYwBA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:MFvcWN0uC4g=;Zj10nVef3twCER9s68Q7eGxft93 VUKmZygob4ELo4Vdatco3OIDQRkGxEupldyL6+qtUD2byFXYThHud/rrwBKdV4Bfy+20so6pZ AO/NukH3XZ20XbF/U7SxpPSJGVVru6VJ4Ovyqq0esi5LNULa6OMhibscpqTX+wknWtoJ7DAvy FyarO+JEOH0s8n19qswg+/rZdMqI5/yTHB+c/4piHGWaCOwZGVaAXbSrjqQZlD7c3s/iVAFnl qnHyoB8OHz4yiEboyEMPbsQGiKeMswaHYvw1IQAgMChvRpuSjfNjxkF/diADDRsWtFYN08vJZ DjGEt7k+dEWYm/XVJT1YIbsMnYO4npLvV51oTxFZeJoJvZzCeqnIGvE5OgH4XPLgoU9Fkb+80 KdmUkoM7eKHt3oxn5BbxYScy/QqKI9rydcBXKZksBLPjgYztLgHzGykXaWuodQcFTOR8ZvssT nY2EGXyyFvxvW5oPREIf+y3d6PPprfSuR8qgc60gX0ZcaMyC8H8kSRs+bWd4XzXD/K6U0GYsb Zkdah8m87zjJc92bG91zMf353KNZFEzY9SAAH2tMbHvkCbzM61RKFYL3ZOb1y0v1DAoFxMtIA iEUy1smrLhEU0C9fSOaYxlfXKMU8f/zNpsj+uJaooi9RFVPZ+7HhgfOXCOGdUeIrz1PP895rw FseUZk7xCOp8JUJ3WN51dLcC+ZhRLE8NrkM79MYGtJbFeDPu7VDJXDCqxtMRS5sdIWILpjBX0 te9k82/D3lrgX54FygYKEw1OcKZZxO0ov0xxRJV2wX9OlHI8Y4koTuSRhgJL9J/9uThJBkAoV rnrJsTOQ47cSXOmpXIJxm+KaDEpy4vH7c21SIREOJzWdWL9JXkW+X9szRFdiGOvgEEfIjmf5+ 6OD+r/ZDmFWWfJvc3xlqMkIv+jyunU2sj5X9z6VvceLwP7iXs0oMjXzxEPuK9lh06RdCedvDa gFsXBLpQXAWnoIgRU21PVQqe0egJAg0QZHnswwowNJf9nvYS/EhSXIx185Ja50coylXPL3Lcx Q7GBOFYjmPL/tlOJlOZd4iOFdIzcqJXwvMo3z5orqxz27eS3gXDjkp64r6UMaQUM7jnUM72dW bMnBB0Av+XstOd4GMqL7WEpNPLUnXEISOBQKNs2MEq14/wBTuZUf1SrrMO0qaUJM3GECHHWYA j5037x0rNdQ/AknbznGQktrx5QkXuoAO2FXk6LmtBDBLkBcrCitQyHfxlv/TS2QZDYXhVwzx0 GxN3A8z4u4QD275Xtily8D7Ymsu5QQ0ITxwKmBBtPhkp/iouUQxFvOBpbfOJwuxGh/fHwVUW1 ubS6Ge7qLTze8Zk8CnLvMiBOlNGHe2NlLXGjJxXdqYO7qrbJvU3LcwKr+h1LhVb3JcwNA6e9U I+FgWAfoNIEtgVFhZKdg++GAtFG1DSExSjYOGQZ3Bwr0ZBbiTOCbkFmiKQipXOvPDq/GlpSTa NsnRq0J+hy1SpMXY9ePZQYzrJrNOUWcECJEFhpAkkziKsjSeJoo+Si7vgfC20b36J/hbQWcf3 6fmr4I3A26EnD0z4donyHFIFbgszKhRG13vt1FF/IoUEdSoW03jSmXBQceT7sZKy4KOMh1UDV lWYPEDEQjE+Ss6e7oXBLRW8J/PO53vB3IN/RTe9Svtb+/YEMreZmBGbP1SBj1+3G5XgeXYDKf yOXuTfEsqKcFCy1UX+WDkPw7TcuJDh1J5XisDL5icR81M/5mGHnOEQP3vYKMIIdw7oXRvH5Dg tLrCe7uNx+MjFD+Wl05404zTW6WYWhaH+4bmJ0wMpv/P1AgR936juT2sqWsvTNKwV3tKuUKt/ touF/TI+hfohMuWES989Yh6J0Ck+Zop75ZV9YPJap5VS7Ci6644amY9Qsqi97uX8wNG45KdsI dJRNllfDhtAuC+tUpGvD9YGuqAK0/lTrz7fqJsK8dm2uFMC59lqSQaeVulrz0C3sZRq0D8XkM 2MEV9wDTaGAK0NtklcBBl5ukImil7Kty8XJwPyyWTlitpuxpxGExlRtRKApySHQiCVFf+anEV nplhWKiP+Zj2Pq53VX/eIiiOmr6NKhbQbBc3+Fp4VLBV7Tvc6qK6LLCrAAHwkLt8UsKC0wr3L WQ3NjFL7++94JcS5aqU/18+GZCJRoU8zOt/uIYisZIJmV0pZJ3AqLo8EkYGQUmD4xvccgT7ZZ LM3Z4L9RQrWej+iV1qFdNwwFTPgOBTYgsIyYJ8HttN29rz6JRXJQf1BNxuDkB2D2nccmp+o+Z sus9ZmX2gDv17sgd5RuKzyMoDe9uw89jmH4xwzKyYl9os+7/6+N6Z8AkjYbrFZ2ZHParXNxay MV3UdNEXd1/zq+uuNj+Ad/DWyqXABXlKgx5ROCwJGWk9+3GYz/7KwD9LqVyOkt7DWcZLYl7Xs aAOyxgQmKd0oVKHuRLKaRwQvKp93bothsgXRok9GO21q5P5t+hQVbAKbmrt0kMSNJXyn4Pl/L peNBDydDwaQVUJzVf+uT6ApeKOfkXqdMLJ8WSDMlulYw4KQZmhcud6vgzdp/g/BSpXi7wF055 /K3F3FjPj+cmCCRLqTjyzeOErRVc8wbG84TAZRfofcte+D517C7y8TQW8C1qOQjGRU2+3Ng5O 0pyyMXlcp9mo0IP/10+ZVpZFBb7plqF+FQUoMH7X+kyI4URmhZ/SpdP9Gdq/gGeNMdvpPw3+T MfTobTAJTJeZn6iHyVaQXjHoFvpp3jSNc9NKBryOkGwvTswjWLr4RLDLphw4Gw5Dm44G4Vh1s 8CfC6d/hOpFyRbTl8u9lDbfbJ0nsV4K8o8EIjeWPgundyK74Eoci/oMRqo3YkNp602nfoOjyS 6OirSWWVpUmtatVkseWbHrrHhz8XtG1OqGzA1CJP05fsvTQseC9s/mx/K9Q9agVY6X6zprCdK sKHMh8FG0na9CuJ8i3TcUAlVdH0p7tuBmK/2bEXBEiNjZzNYKO87ntYa8D470hlZIXa/7AxxF qAGQmv2XYXDr/P/pcXonQkNKifKt3/Q6R0Jqva/9Lh5V9Oz/pIOiT//CLZsmFdcZOKR/79DQb EPrNzVFcB+0vqjVJVLLQSJhZgOfzFUeJXBQfnefbUcd30pSXP+9fPXwVhIa6kWrlCA7YEC5nO kTm1SzaaLiFrDty0m8dZ17eLYI8exY2i0umEBsIyAbEmEG0jE5PbKK2MQmIn8Z9RbqOkmORvX xyG4QBC63i6LKyVO+BNr/yYngkKQfK3/ly51BYDfLYfuIwNzyFaHUxqeKUlLkyrzeISLMF/+h 68P73/yvCqXpHyNM4JKYuQo7DROvAyK9PP/Xb0tUHrAUIHhafl7Mxb+rEy+6tnlh1wRS8hJul cd2klsmn3Gj+TcB+wKCrNLku3hMgMlGbxiNv37Dm0nfC/ERPh7UzyFOmduoCkMmhrwlhbXszd bWu2wtLlqmhihPIzNh3F/ofCRUmHrYwTqJIgfs+1yERFcXDd8cvMVh0WdshN0KYC/qNrLPm6L 2QH/Wb4T8kVLYI46SATRdESUOfIgfJ8CZhEXwC4A/FaXv0wpLlEXxWlYXAb5cCB1OMnPfJpCK q5pnQNVpj4KzllJ0hP7OT4w35MCCaQoRT+YxB4+4QcUWIOKjsSs4DteL/wFhl8ZJH1dQX3O8e XpzGtU1e6nF/VWdbhcv79q0VWl0uR/87sSOIwnMVo7QhOQjqe6R774msE6IXWDeuWs3QThiES 2sgkCx6JaYT4BnmX9c3AMdA5sU1RsqGqtPuYrHSwOrtV8fo2jCZebWPG5kHQGPlqtQLzGKlvT LMdy2Nrw35yyBZ17S3TeELZ1LC64LYIq+cEYoJRPdx3F0leDTEVhhz8zDhLwQVpTXd5goaKnG G1MHXN1NdEib3Vv7piIBBEW4EDUU1EofjI172ok8AwgFE98yezOk+b/bcjCOemBtRnDTrQ8YS 7SIafBcrv6OaN44oooB6oLvgnoovJTMsD/D2r1xL1wQP0dCwJl4LIh/EMfkVqgzNdoriOG5xW bkekDXGv219KPipc+7XcKfF7hu5towQ6FWKG5KkZDbgc9FC76LLzlCWYFGnFzQ6B/7QMlgdNw NPqyagQCxUcMU+mHVHGQE+dF3V58BTNZkZ7dm5Pq7NRjo2Uz9kGTytpMOlhuS3oPucnZ468ZA VA/DV1efqw0/VFW0ukr4gvqSwb0w3pQ1F+WK+XLmRkgPpQ6t+P2VPbD4pahZDwpHi4kQuNjn8 WhA3rT6cA8Bsi1ym6nIvkvBdp32CKpVGrrBPJusnOzB4px+UMF39zGhKPZPIUu9F6TAugvBnF XH6h+wdWHuHlDCFsCNo24t+L8ZH1QnaBR++e9y5Xo8YXBE166vl3wolG6X3yLr+xM88igaq8W mxQm6aQa3cVtQWrGU8JmjGeWZFyEVAqSLzlUJrIp0IF57pIPsD+u1fsE8GDuA1YRhKwonmjeo 9tFIC3AIeq1Nq7kU8P7xNom6GlUjVzrl0Lh60aWv2BYSrpOSjLu01IfKXe8UroDYQp6nNUlgO dlU0TzVeA1yEbWeKoudj+e7d05UuFKbqFSRFwkq/i3OjcbZx7Ouj+fGFao3Qv8RoxvmG2YxTP kXW9MBtzC1vIOy4YxfGs5FjRmsRSXlxGsfCOEq1j9Mcdo9biQRroHnPj7UnmEO6xsVmipb3II Ko2VkdDPobRkaqdpdjnMIjIcgS2+FtWkcWI5CCfNzVME4TpdfE+2UDSkkFB76cJ/IH5xb0U6U qIDPEIHRP+iOzZlSWTwHn+BG+0LwiSAi3ziRFcVBFZSq71wMFedekMWwWDHC7pSU38qADMopn aR7Ud86Wl1aXuXr/5ru3GWB+YvsZeVHUOTbPwlhqYCz4aYistItnZ0qMnq+EHm3K/uLSuFKkF yeBgsJdvBBwkNncqXFNCLuy2YJM5Gx9zbVzu/sI15a3TNZvoyTbz8R5gg1NRpEaWnJA+jO5qp Cn8+DNspdon90SMLAf8VvE1bmePOfnyoaRv/Kug/RfzlK1+FvH/4GK7biyBukCn9cc5N7CEIE WZB5PKTaPWn6ky0zFReVxFazQDO6Uy968gYjv5zS5IMCnAsgNNjD3Tzl9tP77W5kYbq4pomDc ubffADoCWbet55sf0KQje0vNDZOCeVlIAOJGYn1gV3TVlQKOGsJMnKzV3tg90D5l9YSvnn/cm VnJFCgd6EzKYamUxGrUQVzERIk/Ou7nhw1GjQ90vYaVFWB06gNogI4A+28XTKs8MamNmpSGZV KiCQuvYI= I think I've found the reason for the stall. I was looking at commit 9311e= 6c29b34 ("cgroup: Fix sleeping from invalid context warning on PREEMPT_RT"), and noticed this: +static void __init cgroup_rt_init(void) +{ + int cpu; + + for_each_possible_cpu(cpu) { + init_llist_head(per_cpu_ptr(&cgrp_dead_tasks, cpu)); + per_cpu(cgrp_dead_tasks_iwork, cpu) =3D + IRQ_WORK_INIT_LAZY(cgrp_dead_tasks_iwork_fn); + } IRQ_WORK_INIT_LAZY() expands to __IRQ_WORK_INIT(_func, IRQ_WORK_LAZY): (in include/linux/irq_work.h) IRQ_WORK_LAZY is declare in include/linux/smp-types.h: IRQ_WORK_LAZY =3D 0x04, /* No IPI, wait for tick */ The "wait for tick" gave me an idea as I'm also using CONFIG_NO_HZ_FULL=3Dy so I tried commit b7453da1c7de288235234eb9265c3a2d661c1a2d (HEAD -> reboot_delay_debu= g_0) Author: Bert Karwatzki Date: Thu Feb 26 14:01:41 2026 +0100 cgroup: use IRQ_WORK_INIT in cgroup_rt_init() =20 Signed-off-by: Bert Karwatzki diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index b750aa284b89..ed22842eb5d3 100644 =2D-- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -7047,7 +7047,7 @@ static void __init cgroup_rt_init(void) for_each_possible_cpu(cpu) { init_llist_head(per_cpu_ptr(&cgrp_dead_tasks, cpu)); per_cpu(cgrp_dead_tasks_iwork, cpu) =3D - IRQ_WORK_INIT_LAZY(cgrp_dead_tasks_iwork_fn); + IRQ_WORK_INIT(cgrp_dead_tasks_iwork_fn); } } =20 and this seems to have fixes the issue for me. (Five successful reboots in a row, needs more testing). Bert Karwatzki