From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 971AE1CEAC2 for ; Sun, 31 May 2026 02:35:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.42 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780194952; cv=none; b=KYznwUsTMXGF8/QHQqoIgvTIPa65DpTiIRDJ+zz+WA4pBMBl3DUhP7eaFkun9pzF4nNOCpqT/XhrJOGFEjYwotOS1N85jhDjhkG1VcUQAorAmvAspwIVK3bVNvzd5jytMwT+5AO7rO5PSrb0ebsCF5EUMSVQOZ5W9gNiMcFliZM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780194952; c=relaxed/simple; bh=Dv3n2FXMQ9LAS09z8Jgf1jR2JX1IMRjgkKIai7PKzU4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=AZzod5NxRZ8BFMKhDzk1gc6iPB79dFS3o/xsz8MC4TbicZhoWFvdn2ist//JoObhtb28YoO13FEGHKehno+ptjfrWNl27abSPltNTML0HlKfCK+sH0On7uupHrUWHAUwlOrCxivfkBJ5Ydfo73k0KAZu8eCaRgMetI+3Um0wZT0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io; spf=pass smtp.mailfrom=layalina.io; dkim=pass (2048-bit key) header.d=layalina-io.20251104.gappssmtp.com header.i=@layalina-io.20251104.gappssmtp.com header.b=UY7dlUBL; arc=none smtp.client-ip=209.85.221.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=layalina.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=layalina-io.20251104.gappssmtp.com header.i=@layalina-io.20251104.gappssmtp.com header.b="UY7dlUBL" Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-45ef4223be7so1004480f8f.2 for ; Sat, 30 May 2026 19:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=layalina-io.20251104.gappssmtp.com; s=20251104; t=1780194949; x=1780799749; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=6c3QKGtcC5uSNa3hMNajgdTtcl83gn+MVCbN/kNxIbI=; b=UY7dlUBLCXxIpBOUCMC6hEv4DArBARtPiXwEa8f9iUJ1NkKfhWDxWsCbYwxlBkyGqd FVfail7O2eUiPqlqBy6HlJo0Tnjq2Vgg6245V1DpIEosPlxt2a5xBhWtUqTxPf+3rfmm irNLomSz9BcKouEDcotAUKQ/D0hA81nSs5xv9aYfy81NfWZAx/7xsbp9naZI/JHvk//P IglcBQhAgP9YTIEfqKdDbYeDMyaJLdmvoBq6wScZ3B6oxFCBSZCdBv6oXo0MUh3yWVha xEa08aJB+gkmTjkpyilMitsuhndKfEApubkdJXsdGK5gw8SK5F+o++o7zm9HxO9a2Vsy X69g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780194949; x=1780799749; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6c3QKGtcC5uSNa3hMNajgdTtcl83gn+MVCbN/kNxIbI=; b=JhRCqg1sztFb1W/c6LNNuil1XKsitQ65o1XoupFC48QGyr/Wnslkk6KGEdggYtz3F7 KnDVoUhpK0cNHOTNF61UVASY+9Xg9VhzG4zThUE/R8mT/ZrY9FLCJ3iZN7c+DteCLmos SvD+UVdOOw9P2JhKDVprN20YC+hScxeAse6KDUMAbuZFwg60yklcmjeqlBiGb+Q5DEii CsPwoQQdEti5TZ8kCN8kJUQIorFG0O+Le5OvyHYztvdFH1YKKat9FJeq6kS1wjiBqR+a 5QRnbZNQULEFbf2K+Z1ilSDQU5dsc3kseJsT8j8PLbC/6Q/l5pkyx7M6gX9JEFa/Y5H+ yf4g== X-Gm-Message-State: AOJu0YxrHy/S3YIa8ZRZCLdV99akO8N/krjxPqWvJjmhTfOMdHBySTGS SbOPjKMykp9e1DPH2s1bixzvahpcqDvvAH9I4rUQj1fa6qPmYJo/XzySnY1mw+vpL90= X-Gm-Gg: Acq92OHyTvHKnorvCQY+0Y2sN7ZmtNytZviT0EC+WVSSdyMc+4CYQAQZuJ9wODEEVFd hdzbOF9dj8ky/d8+fH12w79mA1WDw+rSrSAhyUmWYdkOFNJDMyOqZlxmXggvcP6nXLoHmRUE4pq t+gEh7JWXE/ZvctciiLNbQlw/2d1VTArz/t6Kw0bO8mRpunMC0Aw9ysN0jeqg/CcW/p49hUhFKv EbGanOlMTcAeWrF6Qi0xEP238CLCRFu82phTkURIOLdbIals6xR6JTCnP6cjCE6P3KnoQCjP+1r KhXmAZ7PHw8let6IbbgYwRrDSbJ27EdNWaZ1TX+fS/1wemN9HDPu4NrI1A6Ueo5xA1daj2wDBdF a74td7/c1+lkTLXRAj2RHiO6QrwhMeNANfx0jkALO/0Zu6HUDSS13o2dTrqeOBtnsZ8EUtkrFop B9ijWTMoyQgxTMOyjbT6ZjwaDsIHkFTOQ5hmaX X-Received: by 2002:a05:6000:4610:b0:45e:779a:302c with SMTP id ffacd0b85a97d-45ef6b67b17mr10880984f8f.29.1780194948480; Sat, 30 May 2026 19:35:48 -0700 (PDT) Received: from airbuntu ([149.40.48.91]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-45ef3595ef4sm14189022f8f.37.2026.05.30.19.35.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 May 2026 19:35:47 -0700 (PDT) Date: Sun, 31 May 2026 03:35:45 +0100 From: Qais Yousef To: Tom Gebhardt Cc: linux-kernel@vger.kernel.org, Peter Zijlstra , Ingo Molnar , Vincent Guittot Subject: Re: [PATCH v2 00/13] sched/fair/schedutil: Better manage system response time Message-ID: <20260531023545.2mohhzwkl5oqsxh3@airbuntu> References: <20260529014333.w3tqvkirgo7jij6l@airbuntu> <65ef80bbf22867d1ea59642a3e03e3ef.tomge68@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <65ef80bbf22867d1ea59642a3e03e3ef.tomge68@gmail.com> On 05/29/26 09:53, Tom Gebhardt wrote: > On 05/29/26 02:43, Qais Yousef wrote: > > 7.0+ttwu+vincent is the best, right? > > Yes. > > > Have you verified your actual workload is seeing benefit? I think when > > I scanned the github bug you references the original report was observing > > a regression in some real setup, not this stressng tests. > > I am the reporter. The original issue (#7308) was observed as a drop in Yes, I realized, thanks for taking the time to chase all of this :) > camera frame rate when running two parallel IMX477 streams via libcamera on > RPi5 under kernel 6.12+. The camera pipeline is pipe-IPC-heavy (GStreamer / > libcamera internal queues), so the regression surfaced there first in a real > workload. To isolate the cause I moved to synthetic pipe benchmarks > (stress-ng), which confirmed and quantified the regression cleanly. > A Raspberry Pi developer (popcornmix) also posted IPC benchmark results on > the issue, independently confirming the trend across kernel versions > (6.6=2065 > 6.18=1805 > 6.12=1662 > 7.0=1570 Kops/s). I was wondering if the real workload is as sensitive > > The stress-ng pipe stressor is therefore not an artificial worst-case -- it > directly exercises the code path that causes the real-world camera regression. > That said, I agree stress-ng amplifies the effect, and I cannot give you an > exact frame-rate number yet with the ttwu+vincent patches applied. No worries, don't want to ask you to do more work ;-) > > > IPC drops 14% on 7.0 stock. Due to stalling you reckon? > > Yes. The branch misprediction rate explains most of it. On Cortex-A76 a > branch mispredict costs ~13 cycles. Normalised by instruction count: > > Kernel branch-miss rate vs 6.6 > ----------------- ----------------- ------ > 6.6.78 0.178% ref > 7.0.0 stock 0.427% +140% > 7.0.0+ttwu+vincent 0.271% +52% This could potentially be due to the higher ctx switches > > The raw counts I reported yesterday were misleading because the instruction > counts differ between kernels (different amounts of useful work). Apologies > for not normalising upfront. The rate tells a cleaner story: stock EEVDF > causes 2.4× more mispredictions per instruction than CFS, and ttwu+vincent > brings that down to 1.5× -- significant improvement but not full recovery. Note 6.6 kernels are EEVDF too. > > > Do you have the full output? It would be interesting to use perf diff. > > A proper perf diff with resolved kernel symbols requires running against the > matching kernel. I ran `perf report --no-children -s symbol` on each .data > file while booted on the corresponding kernel. Key findings: > > 7.0.0 stock (flat, self-overhead): > > 12.98% finish_task_switch.isra.0 > -> __schedule -> schedule > -> anon_pipe_read 5.72% > -> anon_pipe_write 1.38% > > 7.0.0+ttwu+vincent (flat, self-overhead): > > 19.62% finish_task_switch.isra.0 > -> __schedule -> schedule > -> anon_pipe_read 8.22% > -> anon_pipe_write 4.34% > > The striking difference is in the pipe_write -> schedule() path: 1.38% on > stock vs 4.34% with ttwu+vincent. The ttwu patches make pipe writers yield > the CPU far more aggressively after each write, allowing the reader to run > immediately. Stock EEVDF leaves this to the scheduler's own timing, which > results in more latency and lower throughput. I collected a trace for stress-ng --pipe 2 on a 2 CPU system (6.8 kernel) and I can see it ends up with 4 tasks, 2 almost always running and 2 that sleep and wake up, rather rapidly. stress-ng 1: 58% RUNNING, 41% RUNNABLE, ~1% sleeping stress-ng 2: 41.5% RUNNING, 58.5% RUNNABLE, ~1% sleeping stress-ng 3: 40.1% RUNNING, 22.2% RUNNABLE, ~37.6% sleeping stress-ng 4: 59.9% RUNNING, 21.7% RUNNABLE, ~18.5% sleeping The avg RUNNING time of these tasks is few 10s of us and min is 100s of ns.. It seems the tasks are pinned too, 2 per cpu. I hope your real workload doesn't behave this way, this is very inefficient :) > > The higher absolute percentage in finish_task_switch for vincent is expected: > vincent completes ~24% more pipe operations in the same wall time, so there > are proportionally more context switches completing. > > On 6.6 (from the call-graph profile recorded separately), finish_task_switch > is not visible as a top-level hotspot at all -- consistent with CFS handling > this path much more efficiently. It could also be about the wakeup preemption pattern. The pattern I see is that one task wakes up runs for a bit before the other tasks wakes up rapidly for 4 times. The first 3 it preempts with ~0.5us but the last one it waits behind the original task until it sleeps which takes ~9us. If I do echo NO_WAKEUP_PREEMPTION | sudo tee /sys/kernel/debug/features I can see the bogo ops/s jump by 16%. The two tasks now interleave equally and the tasks that had ~1% sleeping time now go up to 7 and 10% of sleeping time. You can achieve the same outcome by running as SCHED_BATCH chrt -b 0 stress-ng --pipe 4 --timeout 20s --metrics-brief > > > Maybe there's higher rq lock contention. But this finish_task_switch and > > __raw_spin_unlock_irqrestore are common to see, especially when there's > > high context switch rate. > > Agreed -- I cannot rule out rq lock contention without perf diff with > matched build-IDs. The pattern I see (finish_task_switch dominant, driven > by pipe_read/write) is consistent with high context switch rate rather than > a pathological lock. But your point about a 'hot variable' like rq->clock > is noted -- I cannot confirm or deny that from flat profiles alone. > > > I hope perfetto trace will help visualize the pattern that led to this > > higher context switching. > > I will work on getting a perfetto trace. Expecting to have that in a > follow-up. > > Tom