From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 78E1033290F for ; Mon, 5 Jan 2026 11:45:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767613562; cv=none; b=IQ577U8O4MQjPEKJQ+G1xjNBAT6nKLRX6a5CxKqIxvVGNVaC2nOKDH6tYjfkF7oDhKgaeDZjo9SVe2wEFjJpCbK2TGatrCbfrZdPPJfT9Lej80zcJkI9MYRvJCcu9V27Bq5gOnS4sX8zLKamjTN9L4DZSX3lgxyoHeXgb9rs7io= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767613562; c=relaxed/simple; bh=RaBaI+kC228n+pz1Di4i7pqnwPtj3Tu7icZ1P3fVpFg=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=kpA5NaTCMwGB46m5hAU3JeveIJ6C7TjhMD4lGiZKJTmFKy6ysktNdgYzWdXY6+ch2OJhlquJOX2Z7ZdY5cNRtoSCziK8t6nf7O1nqG8DoIRF46zv8iCSrWOEn9pZlOc3yT68YU9Bd5K8kNuAj+eaEtKSAv29RYQ03QUesH6JlME= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B684A339; Mon, 5 Jan 2026 03:45:51 -0800 (PST) Received: from [10.1.38.150] (XHFQ2J9959.cambridge.arm.com [10.1.38.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D45C93F5A1; Mon, 5 Jan 2026 03:45:57 -0800 (PST) Message-ID: Date: Mon, 5 Jan 2026 11:45:56 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [REGRESSION] sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals Content-Language: en-GB To: Dietmar Eggemann , Mel Gorman , "Peter Zijlstra (Intel)" Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Aishwarya TCV References: <20251112122521.1331238-3-mgorman@techsingularity.net> <176339661525.498.7070393041762616565.tip-bot2@tip-bot2> <4b96909a-f1ac-49eb-b814-97b8adda6229@arm.com> <756efd17-682f-4ffc-b8d9-dbb2517bc152@arm.com> From: Ryan Roberts In-Reply-To: <756efd17-682f-4ffc-b8d9-dbb2517bc152@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 02/01/2026 15:52, Dietmar Eggemann wrote: > On 02.01.26 13:38, Ryan Roberts wrote: >> Hi, I appreciate I sent this report just before Xmas so most likely you haven't >> had a chance to look, but wanted to bring it back to the top of your mailbox in >> case it was missed. >> >> Happy new year! >> >> Thanks, >> Ryan >> >> On 22/12/2025 10:57, Ryan Roberts wrote: >>> Hi Mel, Peter, >>> >>> We are building out a kernel performance regression monitoring lab at Arm, and >>> I've noticed some fairly large perofrmance regressions in real-world workloads, >>> for which bisection has fingered this patch. >>> >>> We are looking at performance changes between v6.18 and v6.19-rc1, and by >>> reverting this patch on top of v6.19-rc1 many regressions are resolved. (We plan >>> to move the testing to linux-next over the next couple of quarters so hopefully >>> we will be able to deliver this sort of news prior to merging in future). >>> >>> All testing is done on AWS Graviton3 (arm64) bare metal systems. (R)/(I) mean >>> statistically significant regression/improvement, where "statistically >>> significant" means the 95% confidence intervals do not overlap". > > You mentioned that you reverted this patch 'patch 2/2 'sched/fair: > Reimplement NEXT_BUDDY to align with EEVDF goals'. > > Does this mean NEXT_BUDDY is still enabled, i.e. you haven't reverted > patch 1/2 'sched/fair: Enable scheduler feature NEXT_BUDDY' as well? Yes that's correct; patch 1 is still present. I could revert that as well and rerun if useful? > > --- > > Mel mentioned that he tested on a 2-socket machine. So I guess something > like my Intel Xeon Silver 4314: > > cpu0 0 0 > domain0 SMT 00000001,00000001 > domain1 MC 55555555,55555555 > domain2 NUMA ffffffff,ffffffff > > node distances: > node 0 1 > 0: 10 20 > 1: 20 10 > > Whereas I assume the Graviton3 has 64 CPUs (cores) flat in a single MC > domain? I guess topology has influence in benchmark numbers here as well. I can't easily enable scheduler debugging right now (which I think is needed to get this info directly?). But that's what I'd expect, yes. lscpu confirms there is a single NUMA node and topology for cpu0 gives this if it helps: /sys/devices/system/cpu/cpu0/topology$ grep "" -r . ./cluster_cpus:ffffffff,ffffffff ./cluster_cpus_list:0-63 ./physical_package_id:0 ./core_cpus_list:0 ./core_siblings:ffffffff,ffffffff ./cluster_id:0 ./core_siblings_list:0-63 ./package_cpus:ffffffff,ffffffff ./package_cpus_list:0-63 ./thread_siblings_list:0 ./core_id:0 ./core_cpus:00000000,00000001 ./thread_siblings:00000000,00000001 > > --- > > There was also a lot of improvement on schbench (wakeup latency) on > higher percentiles (>= 99.0th) on the 2-socket machine with those 2 > patches. I guess you haven't seen those on Grav3? > I don't have schbench results for 6.18 but I do have them for 6.19-rc1 and for revert-next-buddy. The means have moved a bit but there are only a couple of cases that we consisder statistically significant (marked (R)egression / (I)mprovement): +----------------------------+------------------------------------------------------+-------------+-------------------+ | Benchmark | Result Class | 6-19-0-rc1 | revert-next-buddy | +============================+======================================================+=============+===================+ | schbench/thread-contention | -m 16 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 1263.97 | -6.43% | | | -m 16 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | -0.28% | | | -m 16 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.00 | 0.00% | | | -m 16 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 6433.07 | -10.99% | | | -m 16 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | -0.39% | | | -m 16 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 4.17 | (R) -16.67% | | | -m 16 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 1458.33 | -1.57% | | | -m 16 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 813056.00 | 15.46% | | | -m 16 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 14240.00 | -5.97% | | | -m 16 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 434.22 | 3.21% | | | -m 16 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 11354112.00 | 2.92% | | | -m 16 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 63168.00 | -2.87% | | | -m 32 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 2828.63 | 2.58% | | | -m 32 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | 0.00% | | | -m 32 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.00 | 0.00% | | | -m 32 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 3182.15 | 5.18% | | | -m 32 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 116266.67 | 8.22% | | | -m 32 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 6186.67 | (R) -5.34% | | | -m 32 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 749.20 | 2.91% | | | -m 32 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 3702784.00 | (I) 13.76% | | | -m 32 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 33514.67 | 0.24% | | | -m 32 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 392.23 | 3.42% | | | -m 32 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 16695296.00 | (I) 5.82% | | | -m 32 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 120618.67 | -3.22% | | | -m 64 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 5951.15 | 5.02% | | | -m 64 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15157.33 | 0.42% | | | -m 64 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.67 | -4.35% | | | -m 64 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 1510.23 | -1.38% | | | -m 64 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 802816.00 | 13.73% | | | -m 64 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 14890.67 | -10.44% | | | -m 64 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 458.87 | 4.60% | | | -m 64 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 11348650.67 | (I) 2.67% | | | -m 64 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 63445.33 | (R) -5.48% | | | -m 64 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 541.33 | 2.65% | | | -m 64 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 36743850.67 | (I) 10.95% | | | -m 64 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 211370.67 | -1.94% | +----------------------------+------------------------------------------------------+-------------+-------------------+ I could get the results for 6.18 if useful, but I think what I have probably shows enough of the picture: This patch has not impacted schbench much on this HW. Thanks, Ryan