From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F00DB4F05CA for ; Thu, 8 Jan 2026 13:15:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767878127; cv=none; b=fcuuOg7WxYGcgwGxN5Fh88RAcxA9MquvoRUrLNBejuKof0+D5XfPmFM9oJBSjR0fdFltgeWeh+QU5X7BQ/FbCMGl4IJ57fh5IqBdZPfcnSKxOL61DAwkPKrZpPyPf/1G29lMHGOwlpketaXWE4BDo3ju+rPQf29cVq8CT1DwQ8w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767878127; c=relaxed/simple; bh=PpX97cSWX2l3qWj7qYP4I+HS3rVOrYptcMeb/H4HF1c=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=QV/X9OKl17kpvINxus11WOqL4NrPjUb4znjqU8cWt8H+v6rpS/hcmYKkf5zP9Cej8Dtj2YPhSBn53SytiQefAt9TUf+wt7z81lnKwS8SgIUonDaUVk3zr2XXMFcGaLXUMxjZuVt/soLolPnlfvLoI8/YEvJnR/bQSvylnpDPks4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7274E497; Thu, 8 Jan 2026 05:15:16 -0800 (PST) Received: from [10.57.95.107] (unknown [10.57.95.107]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8A77E3F6A8; Thu, 8 Jan 2026 05:15:21 -0800 (PST) Message-ID: <6c137b30-d5f7-4dd9-9699-d7e22c174285@arm.com> Date: Thu, 8 Jan 2026 13:15:18 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [REGRESSION] sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals Content-Language: en-GB To: Mel Gorman , Dietmar Eggemann Cc: "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org, Aishwarya TCV References: <20251112122521.1331238-3-mgorman@techsingularity.net> <176339661525.498.7070393041762616565.tip-bot2@tip-bot2> <4b96909a-f1ac-49eb-b814-97b8adda6229@arm.com> <756efd17-682f-4ffc-b8d9-dbb2517bc152@arm.com> <051285c7-56c0-455d-ab3e-9959edb5d13f@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 08/01/2026 08:50, Mel Gorman wrote: > On Wed, Jan 07, 2026 at 04:30:09PM +0100, Dietmar Eggemann wrote: >> On 05.01.26 12:45, Ryan Roberts wrote: >>> On 02/01/2026 15:52, Dietmar Eggemann wrote: >>>> On 02.01.26 13:38, Ryan Roberts wrote: >> >> [...] >> > > Sorry for slow responses. I'm still back from holidays yet and unfortunately > do not have access to test machines right now cannot revalidate any of > the results against 6.19-rc*. No problem, thanks for getting back to me! > >>>>>> All testing is done on AWS Graviton3 (arm64) bare metal systems. (R)/(I) mean >>>>>> statistically significant regression/improvement, where "statistically >>>>>> significant" means the 95% confidence intervals do not overlap". >>>> >>>> You mentioned that you reverted this patch 'patch 2/2 'sched/fair: >>>> Reimplement NEXT_BUDDY to align with EEVDF goals'. >>>> >>>> Does this mean NEXT_BUDDY is still enabled, i.e. you haven't reverted >>>> patch 1/2 'sched/fair: Enable scheduler feature NEXT_BUDDY' as well? >>> >>> Yes that's correct; patch 1 is still present. I could revert that as well and rerun if useful? >> >> Well, I assume this would be more valuable. > > Agreed because we need to know if it's NEXT_BUDDY that is conceptually > an issue with EEVDF in these cases or the specific implementation. The > comparison between > > 6.18A (baseline) > 6.19-rcN vanilla (New NEXT_BUDDY implementation enabled) > 6.19-rcN revert patches 1+2 (NEXT_BUDDY disabled) > 6.19-rcN revert patch 2 only (Old NEXT_BUDDY implementation enabled) OK, I've already got 1, 2 and 4. Let me grab 3 and come back to you - hopefully tomorrow. Then we can take it from there. I appreciate your time on this! Thanks, Ryan > > It was known that NEXT_BUDDY was always a tradeoff but one that is workload, > architecture and specific arch implementation dependent. If it cannot be > sanely reconciled then it may be best to completely remove NEXT_BUDDY from > EEVDF or disable it by default again for now. I don't think NEXT_BUDDY as > it existed in CFS can be sanely implemented against EEVDF so it'll never > be equivalent. Related to that I doubt anyone has good data on NEXT_BUDDY > vs !NEXT_BUDDY even on CFS as it was enabled for so long. > >>>> Whereas I assume the Graviton3 has 64 CPUs (cores) flat in a single MC >>>> domain? I guess topology has influence in benchmark numbers here as well. >>> >>> I can't easily enable scheduler debugging right now (which I think is needed to >>> get this info directly?). But that's what I'd expect, yes. lscpu confirms there >>> is a single NUMA node and topology for cpu0 gives this if it helps: >>> >>> /sys/devices/system/cpu/cpu0/topology$ grep "" -r . >>> ./cluster_cpus:ffffffff,ffffffff >>> ./cluster_cpus_list:0-63 >>> ./physical_package_id:0 >>> ./core_cpus_list:0 >>> ./core_siblings:ffffffff,ffffffff >>> ./cluster_id:0 >>> ./core_siblings_list:0-63 >>> ./package_cpus:ffffffff,ffffffff >>> ./package_cpus_list:0-63 >> >> [...] >> >> OK, so single (flat) MC domain with 64 CPUs. >> > > That is what the OS sees but does it reflect reality? e.g. does Graviton3 > have multiple caches that are simply not advertised to the OS? > >>>> There was also a lot of improvement on schbench (wakeup latency) on >>>> higher percentiles (>= 99.0th) on the 2-socket machine with those 2 >>>> patches. I guess you haven't seen those on Grav3? >>>> >>> >>> I don't have schbench results for 6.18 but I do have them for 6.19-rc1 and for >>> revert-next-buddy. The means have moved a bit but there are only a couple of >>> cases that we consisder statistically significant (marked (R)egression / >>> (I)mprovement): >>> >>> +----------------------------+------------------------------------------------------+-------------+-------------------+ >>> | Benchmark | Result Class | 6-19-0-rc1 | revert-next-buddy | >>> +============================+======================================================+=============+===================+ >>> | schbench/thread-contention | -m 16 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 1263.97 | -6.43% | >>> | | -m 16 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | -0.28% | >>> | | -m 16 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.00 | 0.00% | >>> | | -m 16 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 6433.07 | -10.99% | >>> | | -m 16 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | -0.39% | >>> | | -m 16 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 4.17 | (R) -16.67% | >>> | | -m 16 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 1458.33 | -1.57% | >>> | | -m 16 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 813056.00 | 15.46% | >>> | | -m 16 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 14240.00 | -5.97% | >>> | | -m 16 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 434.22 | 3.21% | >>> | | -m 16 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 11354112.00 | 2.92% | >>> | | -m 16 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 63168.00 | -2.87% | >>> | | -m 32 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 2828.63 | 2.58% | >>> | | -m 32 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15088.00 | 0.00% | >>> | | -m 32 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.00 | 0.00% | >>> | | -m 32 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 3182.15 | 5.18% | >>> | | -m 32 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 116266.67 | 8.22% | >>> | | -m 32 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 6186.67 | (R) -5.34% | >>> | | -m 32 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 749.20 | 2.91% | >>> | | -m 32 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 3702784.00 | (I) 13.76% | >>> | | -m 32 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 33514.67 | 0.24% | >>> | | -m 32 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 392.23 | 3.42% | >>> | | -m 32 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 16695296.00 | (I) 5.82% | >>> | | -m 32 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 120618.67 | -3.22% | >>> | | -m 64 -t 1 -r 10 -s 1000, avg_rps (req/sec) | 5951.15 | 5.02% | >>> | | -m 64 -t 1 -r 10 -s 1000, req_latency_p99 (usec) | 15157.33 | 0.42% | >>> | | -m 64 -t 1 -r 10 -s 1000, wakeup_latency_p99 (usec) | 3.67 | -4.35% | >>> | | -m 64 -t 4 -r 10 -s 1000, avg_rps (req/sec) | 1510.23 | -1.38% | >>> | | -m 64 -t 4 -r 10 -s 1000, req_latency_p99 (usec) | 802816.00 | 13.73% | >>> | | -m 64 -t 4 -r 10 -s 1000, wakeup_latency_p99 (usec) | 14890.67 | -10.44% | >>> | | -m 64 -t 16 -r 10 -s 1000, avg_rps (req/sec) | 458.87 | 4.60% | >>> | | -m 64 -t 16 -r 10 -s 1000, req_latency_p99 (usec) | 11348650.67 | (I) 2.67% | >>> | | -m 64 -t 16 -r 10 -s 1000, wakeup_latency_p99 (usec) | 63445.33 | (R) -5.48% | >>> | | -m 64 -t 64 -r 10 -s 1000, avg_rps (req/sec) | 541.33 | 2.65% | >>> | | -m 64 -t 64 -r 10 -s 1000, req_latency_p99 (usec) | 36743850.67 | (I) 10.95% | >>> | | -m 64 -t 64 -r 10 -s 1000, wakeup_latency_p99 (usec) | 211370.67 | -1.94% | >>> +----------------------------+------------------------------------------------------+-------------+-------------------+ >>> >>> I could get the results for 6.18 if useful, but I think what I have probably >>> shows enough of the picture: This patch has not impacted schbench much on >>> this HW. >> >> I see. IMHO, task scheduler tests are all about putting the right about >> of stress onto the system, not too little and not too much. >> >> I guess, these ~ `-m 64 -t 4` tests should be fine for a 64 CPUs system. > > Agreed. It's not the full picture but it's a valuable part. > >> Not sure which parameter set Mel was using on his 2 socket machine. And >> I still assume he tested w/o (base) against with these 2 patches. >> > > He did. The most comparable test I used was NUM_CPUS so 64 for Graviton > is ok. > >> The other test Mel was using is this modified dbench4 (prefers >> throughput (less preemption)). Not sure if this is part of the MmTests >> suite? >> > > It is. The modifications are not extensive. dbench by default reports overall > throughput over time which masks actual throughput at a point in time. The > new metric tracks time taken to process "loadfiles" over time which is > more sensible to analyse. Other metrics such as loadfiles processed per > client would be easily extracted but isn't at the moment as dbench itself > is not designed for measuring fairness of forward progress as such. > >> It would be nice to be able to run the same tests on different machines >> (with a parameter set adapted to the number of CPUs), so we have only >> the arch and the topology as variables). But there is definitely more >> variety (e.g. used filesystem, etc) ... so this is not trivial. >> > > From a topology perspective it is fairly trivial though. For example, > MMTESTS has a schbench configuration that runs one message thread per NUMA > node communicating with nr_cpus/nr_nodes to evaluate placement. A similar > configuration could use (nr_cpus/nr_nodes)-1 to ensure tasks are packed > properly or nr_llcs could also be used fairly trivially. You're right that > once filesystems are involved then it all gets more interesting. ext4 and > xfs use kernel threads differently (jbd vs kworkers), the underlying storage > is a factor, workset size vs RAM impacts dirty throttling and reclaim and > NUMA sizes all play a part. dbench is useful in this regard because while > it interacts with the filesystem a wakeups between userspace and kernel > threads get exercised, the amount of IO is relatively small. > > Lets start with getting figures for 6.18, new-NEXT, old-NEXT and > no-NEXT side-by-side. Ideally I'd do the same across a range of x86-64 > machines but I can't start that yet. >