From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 706AB2BDC0F for ; Mon, 20 Apr 2026 03:53:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=212.227.17.20 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776657230; cv=none; b=Fyz0Z2bNOcmrm3QIZrHQzUuEpc+g9NElBxKI4sCDwW9KcKRsrKvjCEcuQZ0X1c38Toc1QJNHPGFQVMdEIp6eIQSsyezwjpscKLzcy0D7dEQ8qXyfWpx6YCwNkTxh7dFcQiP4QoaWYGPRtZYGRSnbUS6kWy8Zn6wQ4Nz04KO6kFw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776657230; c=relaxed/simple; bh=arW2Z7sV8k1unUfo3vTqCEGknNHnichkHHHlTlwHxbg=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=TZtfuZia51n/adIL6PQX3A8j/+gqgcPB1ZJea3X+cJpyqzmtuGQDnTFLiAN9MxZgZW78vZmG78AUJdSIwL54HWLAHai3myF3dL9yRs2jSO2nzt9uS0HAzezMC/dx3Bx9bbeRimr/BKxQGg9OPlL/El9JJrALRWX1TZoeuvx4wJI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gmx.de; spf=pass smtp.mailfrom=gmx.de; dkim=pass (2048-bit key) header.d=gmx.de header.i=efault@gmx.de header.b=X3cTZ7ZS; arc=none smtp.client-ip=212.227.17.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gmx.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmx.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmx.de header.i=efault@gmx.de header.b="X3cTZ7ZS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmx.de; s=s31663417; t=1776657218; x=1777262018; i=efault@gmx.de; bh=v37mBwIdeKM9Fy51M6eoKoHnhcfFmNKXwkEinCLvFMw=; h=X-UI-Sender-Class:Message-ID:Subject:From:To:Cc:Date:In-Reply-To: References:Content-Type:Content-Transfer-Encoding:MIME-Version:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=X3cTZ7ZSrURxYblQVu8kpJi4reGHAbWJzr4CisTI/7MiOWdCHhrPG+f9Ty/SpnsX Qt23WFS70lvSBc2GVi6uAaWrD17zAr17uKJiGHIki0bUIs3n/UtgZvqpswejCTQ/o Qm7nUHyxvO2pqAS+wDtIvJJEJwSj3iJdM6YjVgY6H7AvyAy6DzJzSNRpKQ9mlbxFs MzXqZXsTq3AJPq9n5eKLLS3Gz+eX0wu2RpCDCG50tewLMUIg9wqqwJqwb6ZOQmkw5 QMWiUm/OGOju3zWTuoqPO+E1gq4W3VB7yL5ZTg/yvhHwGX2Tfjm6MbMYo1FkpSZ05 htVu5xy5XmWg71Qvhw== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from client.hidden.invalid by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M9Fnj-1wIGTW3bdf-009xEZ; Mon, 20 Apr 2026 05:53:37 +0200 Message-ID: <2877d886d8470f4ea3393e8c2e570a059c2b8b1b.camel@gmx.de> Subject: Re: sched/fair: DELAY_DEQUEUE causes ~25% pipe IPC regression on Raspberry Pi 5 From: Mike Galbraith To: Tom Gebhardt , peterz@infradead.org Cc: mingo@redhat.com, linux-kernel@vger.kernel.org Date: Mon, 20 Apr 2026 05:53:37 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.2 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Provags-ID: V03:K1:Mgzfc8FL0zn8rDIGkny8wLlhDlBzYsjSqSuCoOyiTD4FmTlns1M iIyTidXeqp50m8lOAH7/b5FxRglDJLdwUBaerZ4vgxqs0BNqLIHMG0kxVk2abGER4tHycmT elWYLSjM21NZhIdfpzLYMWnFxPDsdTMYh5qbRyGH4TcPXyowamM++BTfz+iJ2fqvOotv5CJ deizaDyhHNJMqdvJ6Uulw== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:/+sxguMLNlo=;jbADL7VtKT02BL6ft8Ru77gmibl c81uG6ZNaWQ7dwjOYdf+SYt8K4VYW7LChfiTOdLNWwexIY2QnQtFDuwcH6i7TRqr7u2+MT9ZG eSkFdnMiYpF1B8tlVYCLMU4y9L2RYpGF9aVk3gI0myFwJAAzwzX3xtDpPwQFTeHcQfMJpUUqx DSXD0Y+GEegeSPDo6wt/uQGbFVOlS8hQoP91D0kRdDMg9O4veC0PzLqs+BtuWXrD6i5UXu/Ek SE/bb20rMsPDzHM4nicDehzWFScWCupIXDnxmyFo9bE0Yk1isVxzVBpfn5iPbFizUIhNQRIgY HPOVWMGT3uU1E+53cCOihVjfOJEy6yDhK+ABWsxCMi0tDi2uy8Gd0/0AClE8V8/Ic1GCJq8AI njgYYOUDXRWax/bjlGfHfStug8fpEXvjOsrHnVxxpkUUyUDKVyyKAbQ/cFmE6ysWRsgs3gZDf DKPu8YHFJh8KJudc46yqxAbDXaQP17xG93QFWA663iblA3RgDI2C0RurwjVS3gJ/i5nJX+3a9 Vyai/5hwWDR9m2yvSMik9x7nAGB9vyhSVYZffxUJ/ZhzvM20NySHymj+vN3rPCT+vxRDb64zD 0JYxrrRX96uYgbcMseoUZynC1IW9iDBRCCZkG9m6JlIO4aSoL+omCWKBiWo1ki2pdliIhG5wR xLirEt489cHfeG7Ap2rq3/1dUip/0jsO0sXaoxcm+RgDyKIGzQGV7zlTN1Aa0MMYAXtw53H6z 3da05IiY1injdCT9xxW5pEtfnOcndkU4pbB9bhYJ2LSH+Lt7Gc6YfI59CsVDZsXRWuDvhV5vw Y6Q/3eGOpI7+garKmieQ0dODnhakl25EZotMmaWLKF4EvVodmF1RSuHsYyj49WB/OousZS5um 4PFw5sqmkXU8qKXbVEUbPLCsyM+LUPeNS42c2yTV7auWhA9XjcpuceEhsPwjfjE+iAtrlYnP6 U41g337qeYZ+3BurfMf35+6UDzts4/HeFbaIi11zIjdy2BaE3FTdEhYUthOKUCW38Lztmr74/ lrxXUxuRsHzS7agh/V4krvUxKt3TvuRaRKzAAdspDsKNuOvoySN34I4Ls856wh9HtmhUgxFrY PSp/Rige1/n+KAOni1KD36nVZ1lEWY65xMeCkWpsxdOS/t10QI4BuLEygxnGrywQwA7tXWa39 Q0wAgrvcdVrDEpWOtP/+ChHMMFrfON9PoXUhkgVSU6kOiQRNrJiQed1ZELyoFcPXB9Q3H5gH6 YJLU/MFF1bhGw4yVT5aaLRpueXQ2srfZlj/Y0I7r+1Uey++Ycii2bIMD8G0ei/7ICTC6C3TOh 85xuTtkD+VRastgagIdoZ42tJ9FgJn7DENJADg5wqwoaVnZcjiw49v4/AgbW74aO3vSLBiSEY a3D9kpMdWcZr5sA9iflYd/1pCsFvPSQHsTJRVrcwVz+vtcPhlM87pGLELtCdig2MGm7k0PRs2 tX84435bOYvwoYY16Fc3nWn5hdpKVq5R7V0tJ2HS35I0HFl6pxJs7BaCXO9K0RNdgNjI/ravh DYPNvcjIVtx/EIEfkGprsR62mRpJXcs3J+krYQySW/ba05/bu6nY0maBhLpeu8vggCXRHSzUf nd9pbgwXPiy7T0nScGCGPAFaUgNjdY8jgI13MWvGRNsXY+E1HIEOwFWoBtHyCoewkdrvpXPS6 hpakI9WNPxY+HcSfajaSsls4ZMbfQuPQpUawY4TQcnCtYkDJPQloW3HH752RFjaVP6Onl6PZR 3WeE2viwdJLvRcsVDSqEl5c8xdW3iQr/pWmCLsFXRrKucTTfgHexJlVEJNR7pomPJJzobvGU6 zXuIA6CJLDwUdFeFWWxToTWqugZgwheZfUam1xQsuWIdMM3D/pRoLRAbDRwgGQwv/Aj3C2owp 8ZUINslIUr+NY4+9EZ0WZ1Fts8IlbkP8RpyMi5Jco6+p7C/BLgp8g2rHki2cSbTqb+G8BHTOT kvQDRLsom/uDGra8ZplUidfHHEh5wtyMoZO6ua3mfQmu+dBR5eeUnrlYYUfzTJnK+dFRFJ9ns WuFi+kQC8fhLLWDxEhMSnQvkb6mSkWHcJWFXvXUZQTDu3qoQCiuoFi2lmaqbn3vhiBa9X/nsj BeYdAoEBap1qYwCRwi+pb0sQkGSw2ZIvgyCCiVAyARU8cTQzYr+8CQROsQ9RiELAJWpWREe+v MNbBhWHwVX8DN/R2Ap0LluQ+tUaX0i6PMDBGSufnI+5OvtvMucp+fzXIf13X0gu06pdqnXFnq GooWhoGgmRuwWm6kZd3Vta46D+g2lpguQrYQJPP8yZsY2nM+Jg2yzWwCwOGgDJdJFWa0/LhAT TFcc1xATuYjfyh/Ekskn67IsJ7fn5YFDTD4XljPlby6SNZIglnNUNQeUmbsiA7nSd4rCtnCrj GQVFnWZPPgOH6RWygFdmwkUB9yD0RnrBSw9rQfWU08ecRnDxhrRYp9P1QiU6tluQkB+XzlSEp glP3QnnB3nGCH34duaA/05FpnsnTBl5BAn4IxkIGIh9mypi25vlvzChg7MgjuTuiY3v60y9e6 c2K04rt2QeC5waV4G1AfMY8qQa9Psbsxs3UWrNBDE4PhJu7Ef0pGzMLo1I0E4VvicC/9UtCeN FpOarPnFFnsxbq7YZyIh+N+WoPvMhf3N4mcjbn/6x00wZLlpi2b8rHnlvcAUErco6y/DzLn8l yXvzhP49Yo23SX/2Ml+eTWmEhVkHC315+nVWqm43O1PVcyWXppJM92nKcj2M8qKRu/wXKUgGs fJuexMDSxlcbJlMy03KeEndJ6Yp9LDQpvvad0lIvBBOUVHydWmzC1bBYUQZO21AiqTkm8eisH JtIvi5KTkPmqqYBr1N9JabIva+jl/cyNhLWzQQCVT4+PsA+QQcGzfxuOcv5KsAn+52hLgSuId PI70qy0C/43gT0i76GrmK7NQTSL3alPUFiEhJtedOJdn5emaESij3JccSW7/lzaph14S+9kR2 Q6dFO5v+VVMOmZhqqiUIqzLUU7Ahu7aROtcodpeEZHxwZ3cBZ8QRef2dSXPF6nWxmsDFfc/Mb FZhcQDm5deKqSAAdsICzITi1lir5s46fd49p1hBXB45mGe+4giYF0X829qoIqvwEm8Jb4Yxyk qALuzp6tydU1n/zZk4JtqvESlPNyVlScp48kWZWrcqJmL4qHhbyU/jFWvGUWb8kwZtfhSqYSk Cj0JWupbMpoxtxo77Rlyd1NIhBGjpWLBPnvLZmedudbAzPhQQ/WYcTrENKGcy5lyvC7iYoY7Z uS/AwMvEWzWbt37KPMXTCWrBBZwjqY0g+jeJPBZotou84hFTBwRfFlAZwB1GoYr2ZgOg7S9vY /Y8/Z6EC1O9T3qf9am4Icd+0ifl6/8PlE8CfOXZswb8JeD9zA7pTbRoyGsNc0+Sg27cAcawvK c1PvSMWVChILcbsm5Kz0ZSExKEzh7SjnZg8sbLj2PCNnmadnj/fQ3VQJKphTy5baU7o3FwEJ+ R2RsESRV/XdZ47PM+w0lAI+c7xUikOhYOBjo+g4Y0anGRLJ/tlwuYZ3Q2+mGnMZOOEq184Vi6 r05yU/3YO90qzdEnweGbnTeFzFadLVNPd69FJr+XFgkevbQ+5cngAykuQFkl78NsvhuIl7T2f AaajzNDc4kD/Dne2Q/iLIxXojE/BYj5W9rzeqreSZDPMQuyIvFFlQsJJoH6aGpat8Nga80J2L 5nTKpUBt+69q3l5OHiD5wWdVO+IymB3Q6wEJBZUgIKoNfZu0tmsKHphbZSKxrRRpMHorXR1DO nYhDxRrsBoQIxVv7H75shfBjbBDJt1qSaqX813tmr7HOyPjNueY7+2VS9eeEZ3rIQhpt959ZL K8wJvXkWhsWawgtr74PvBfFiClQDNxzDXN2jBvvbj04l6jW1jIGK6pdYORdeVrA7kBZSHZOap sEpUAoD6jd0fvRsawrQUe9B5TMXxnxHK1kCZTx1Zpy8TI+pmGJ4PWISpJO9SXf4k6tgtAXO0y qun1nO45DI9TduvXbtMgg2/Tlt90ehTwcDi6eR/Ud+GBULcK1WLY6tlMFUdAphYLVCUHF+MI/ 3j7JaCoVFINkufWI0oaL0C7qln0yC74s3Ogo66JrNl6V9ScueQWo60CdJSkGe1QGrbeBczufA lkolcrip15v6a9dFN3xqCQCjo8MU+rzQqoldnr+iVnVfwt+QwUEkAhLcxGvx72+wkvMPwYINA l6BXoyDLbT79bD1nyP5Td4jD0oOvIL96gTbXgXeIibW9ChsGAvbDN0wfZ6Ovn0xzUL82ikT+A z40Qlp9WUdtPQk0yOUA6SNlU6vrpxhUsZqs12dELsgOTlz4qyS5/pocIeT4c0QtzLWvQhw4pr FZP1vgDNNPJWz10zo8OqEf73iJqvc4gcRF+THhMrKnYPFcvE0zByXn+iYZSTAXXdh2lIf3eG2 q79Kpz+OBgVjL18zUozOm/a2P5jOHh4YqR6pyen3e6IoGymacEbyx3bisMarVBlJFBeXLvNko m2OOz9gJPiAvYoQZGTEvN0eZhsiLXsXr6XeBF5MNSnA4PcqHStFZtFpzWjui8ErKLNTei+U7u /rcdYspihXj/E8J1y3Z+rd4o/gRgGvr8XcvCQGK5ska/jlo//t7knLe4rFyApsZvONO2wFBmk otiREpj1Ef3wjqk3aidO4ncrLqZzxhp1MFMJPat7SQ7OF6+VKHQUBoc10Z+Ml+ZXjKydU8Od1 RL7rh1DdTqJycBjkij8kXgKLa5L2l3au7OKLJ1Z24ddKuDYprNtRBNTkAO9qQJRZum42XMYUw ox7yGECveoVlBKM4W1S5KdFpXwfZM+M04FGm+wt7+mKUSUW7U0G/x2+dr+uIAyyn3VrBUcu5G nr0Wp4f6e4N7kEimXJrssGuqwITAy1CvX+E6wKU5djekA/g+/xvl6AyPgd89ViQs6ol3DAYRt MoIxLvfYjUJ6Z3TVbqnRCX/20K+K5pW9+2kA/quS72q4axbPRrlMd+BW0qshDOkL/F6wK4rof h8Ola+rDPkRKExXngGIB+JLhvfLkKC0/jGz8rK5eJtdD/13o3CqexpMomlRUSSmsbIeuEeQ1h emCHEjNAg57KczslTXevRWHxtVNEQCXcnkm8QMQgLc95/v+dQWxO/LmHSKpudyPCp6CuRHEaF N8sAWBirIAcNTcBZtmp5Covvv9/lv/tPr2/M8GjmKSYups/2SF/ryaTRW0PB7ZyB+oLMt3IKf yG6Y/fQXk9rC+7hgs1q0hOaCZMcmjuUfzZZlNOF1kBLfF0mT4lJNzUFTCzfNAQO4PllXtDfiJ At5XtMVxDjyvlX31NJXIMMEIlke7JohhAG1X790xMghlWk4VInB2ZDm8TGzXrRQB92lE6Ov65 Cno5BOMexpDhtgWDixtUtGsIrYDsZ3wJejIDtwzYMfIxZRNxinx0C1za6QdaZAiTHeOcg8hac odUsVxqUxfqgMcQKDMfRsr3JoxjH91vB0yE3nqZmoNL6LvklZa23vJ3tt8dIjaZb+1ktYJeX+ Iohzh1/ukIMywGq6Zkq1+ritGICPdTomUjNnnbkuTs9cTnNnNr5UPfcxU Greetings, On Thu, 2026-04-16 at 14:22 +0200, Tom Gebhardt wrote: > Hi Peter, >=20 > I would like to report a measurable pipe IPC throughput regression introd= uced by > commit 152e11f ("sched/fair: Implement delayed dequeue"), first > present in v6.12-rc1. >=20 > This has been independently confirmed on the official Raspberry Pi > Linux issue tracker > (raspberrypi/linux #7308), where the RPi kernel team directed the > issue upstream. >=20 >=20 > Hardware / Software > ------------------- > - Raspberry Pi 5 Model B, BCM2712 (C1 stepping), 8 GB RAM > - Raspberry Pi OS Bookworm (arm64) > - Kernels tested: 6.6.78-v8-16k+ (rpi-6.6.y), 6.12.75+rpt-rpi-2712, > 6.12.81-v8-16k+ (custom) > - Benchmark: stress-ng 0.15.06, --pipe 4 --timeout 20s --metrics-brief >=20 >=20 > Observed regression > ------------------- > Comparing pipe IPC throughput across kernels (overclocked, arm_freq=3D280= 0): >=20 > =C2=A0 Kernel=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pipe bogo ops/s= =C2=A0=C2=A0 vs. 6.6 > =C2=A0 6.6.78=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 2 487 746=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 100% > =C2=A0 6.12.75=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 1 651 427=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -34% > =C2=A0 6.18.21=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 2 049 701=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -18% >=20 > This regression pattern is consistent across two separate Raspberry Pi > 5 units and has > been independently reproduced by the RPi kernel team with 20-run averages= : > 6.6=3D2065 Kops/s, 6.12=3D1662, 6.18=3D1805, 7.0=3D1570 (lowest). Does reproduction require more than the posted stress-ng args? Raw data from my pi5+bookworm+6.12.75+rpt-rpi-2712 (no overclock) rpi5:..debug/sched # stress-ng --version stress-ng, version 0.21.00 (gcc 12.2.0, aarch64 Linux 6.12.75+rpt-rpi-2712)= =F0=9F=92=BB=F0=9F=94=A5 rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief stress-ng: info: [2965] setting to a 20 secs run per stressor stress-ng: info: [2965] dispatching hogs: 4 pipe stress-ng: metrc: [2965] stressor bogo ops real time usr time sys t= ime bogo ops/s bogo ops/s stress-ng: metrc: [2965] (secs) (secs) (sec= s) (real time) (usr+sys time) stress-ng: metrc: [2965] pipe 57731375 20.00 6.69 73= .24 2886616.84 722246.62 stress-ng: info: [2965] skipped: 0 stress-ng: info: [2965] passed: 4: pipe (4) stress-ng: info: [2965] failed: 0 stress-ng: info: [2965] metrics untrustworthy: 0 stress-ng: info: [2965] successful run completed in 20.00 secs rpi5:..debug/sched # echo NO_DELAY_DEQUEUE > features rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief stress-ng: info: [3048] setting to a 20 secs run per stressor stress-ng: info: [3048] dispatching hogs: 4 pipe stress-ng: metrc: [3048] stressor bogo ops real time usr time sys t= ime bogo ops/s bogo ops/s stress-ng: metrc: [3048] (secs) (secs) (sec= s) (real time) (usr+sys time) stress-ng: metrc: [3048] pipe 58814508 20.00 6.78 73= .17 2940713.90 735679.81 stress-ng: info: [3048] skipped: 0 stress-ng: info: [3048] passed: 4: pipe (4) stress-ng: info: [3048] failed: 0 stress-ng: info: [3048] metrics untrustworthy: 0 stress-ng: info: [3048] successful run completed in 20.00 secs rpi5:..debug/sched # echo DELAY_DEQUEUE > features rpi5:..debug/sched # stress-ng --pipe 4 --timeout 20s --metrics-brief stress-ng: info: [3349] setting to a 20 secs run per stressor stress-ng: info: [3349] dispatching hogs: 4 pipe stress-ng: metrc: [3349] stressor bogo ops real time usr time sys t= ime bogo ops/s bogo ops/s stress-ng: metrc: [3349] (secs) (secs) (sec= s) (real time) (usr+sys time) stress-ng: metrc: [3349] pipe 58058262 20.00 6.62 73= .32 2902974.63 726327.83 stress-ng: info: [3349] skipped: 0 stress-ng: info: [3349] passed: 4: pipe (4) stress-ng: info: [3349] failed: 0 stress-ng: info: [3349] metrics untrustworthy: 0 stress-ng: info: [3349] successful run completed in 20.00 secs