From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 04B5A28DB54 for ; Fri, 9 Jan 2026 10:15:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767953752; cv=none; b=DWKf22s+tztlSQ00K/238EQszXhiOg08DuAu/uFFqC0/opfVimCmMHl96UPDwvYSSFtq08v4LabZFEbAe36ItWyqV1yPEtuU/+AvmjnlzDOEAj+ZKFXE0NSgqMPl+Mv40z+okBNd5F1lf44IVJtJctfpeo7TGS8hrEui0rAugA8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767953752; c=relaxed/simple; bh=RIAJbKcXBXUhxPNVx4UQ4SbY8K0sd/2GsNJ3hOuttzo=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=JSb5GDACnnbcPWNrvmaCpKe4D85eLiJO7jJ+ohxeDX6wXUibkww5irag5Flvmixyr7UhSvabDLRZTOYdqyLDWJuCTvJ0FxbJZPQN/orTuA1BzthBDKqZuTR9CLp6kJp6TFHxVKr1DXAlyYJuAP6rkAfLRaPY9Inq2hIaMJ3k69Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B3072497; Fri, 9 Jan 2026 02:15:43 -0800 (PST) Received: from [10.57.95.107] (unknown [10.57.95.107]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C95D73F5A1; Fri, 9 Jan 2026 02:15:48 -0800 (PST) Message-ID: <63d22eb9-b309-4d11-aa56-3f1e7e12edb1@arm.com> Date: Fri, 9 Jan 2026 10:15:46 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [REGRESSION] sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals Content-Language: en-GB From: Ryan Roberts To: Mel Gorman , Dietmar Eggemann Cc: "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org, Aishwarya TCV References: <20251112122521.1331238-3-mgorman@techsingularity.net> <176339661525.498.7070393041762616565.tip-bot2@tip-bot2> <4b96909a-f1ac-49eb-b814-97b8adda6229@arm.com> <756efd17-682f-4ffc-b8d9-dbb2517bc152@arm.com> <051285c7-56c0-455d-ab3e-9959edb5d13f@arm.com> <6c137b30-d5f7-4dd9-9699-d7e22c174285@arm.com> In-Reply-To: <6c137b30-d5f7-4dd9-9699-d7e22c174285@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 08/01/2026 13:15, Ryan Roberts wrote: > On 08/01/2026 08:50, Mel Gorman wrote: >> On Wed, Jan 07, 2026 at 04:30:09PM +0100, Dietmar Eggemann wrote: >>> On 05.01.26 12:45, Ryan Roberts wrote: >>>> On 02/01/2026 15:52, Dietmar Eggemann wrote: >>>>> On 02.01.26 13:38, Ryan Roberts wrote: >>> >>> [...] >>> >> >> Sorry for slow responses. I'm still back from holidays yet and unfortunately >> do not have access to test machines right now cannot revalidate any of >> the results against 6.19-rc*. > > No problem, thanks for getting back to me! > >> >>>>>>> All testing is done on AWS Graviton3 (arm64) bare metal systems. (R)/(I) mean >>>>>>> statistically significant regression/improvement, where "statistically >>>>>>> significant" means the 95% confidence intervals do not overlap". >>>>> >>>>> You mentioned that you reverted this patch 'patch 2/2 'sched/fair: >>>>> Reimplement NEXT_BUDDY to align with EEVDF goals'. >>>>> >>>>> Does this mean NEXT_BUDDY is still enabled, i.e. you haven't reverted >>>>> patch 1/2 'sched/fair: Enable scheduler feature NEXT_BUDDY' as well? >>>> >>>> Yes that's correct; patch 1 is still present. I could revert that as well and rerun if useful? >>> >>> Well, I assume this would be more valuable. >> >> Agreed because we need to know if it's NEXT_BUDDY that is conceptually >> an issue with EEVDF in these cases or the specific implementation. The >> comparison between >> >> 6.18A (baseline) >> 6.19-rcN vanilla (New NEXT_BUDDY implementation enabled) >> 6.19-rcN revert patches 1+2 (NEXT_BUDDY disabled) >> 6.19-rcN revert patch 2 only (Old NEXT_BUDDY implementation enabled) > > OK, I've already got 1, 2 and 4. Let me grab 3 and come back to you - hopefully > tomorrow. Then we can take it from there. Hi Mel, Dietmar, Here are the updated results, now including column for "revert #1 & #2". 6-18-0 (base) (baseline) 6-19-0-rc1 (New NEXT_BUDDY implementation enabled) revert #1 & #2 (NEXT_BUDDY disabled) revert #2 (Old NEXT_BUDDY implementation enabled) The regressions that are fixed by "revert #2" (as originally reported) are still fixed in "revert #1 & #2". Interestingly, performance actually improves further for the latter in the multi-node mysql benchmark (which is our VIP workload). There are a couple of hackbench cases (sockets with high thread counts) that showed an improvement with "revert #2" but which is gone with "revert #1 & #2". Let me know if I can usefully do anything else. Multi-node SUT (workload running across 2 machines): +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | Benchmark | Result Class | 6-18-0 (base) | 6-19-0-rc1 | revert #2 | revert #1 & #2 | +=================================+====================================================+===============+=============+============+================+ | repro-collection/mysql-workload | db transaction rate (transactions/min) | 646267.33 | (R) -1.33% | (I) 5.87% | (I) 7.63% | | | new order rate (orders/min) | 213256.50 | (R) -1.32% | (I) 5.87% | (I) 7.64% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ Single-node SUT (workload running on single machine): +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | Benchmark | Result Class | 6-18-0 (base) | 6-19-0-rc1 | revert #2 | revert #1 & #2 | +=================================+====================================================+===============+=============+============+================+ | specjbb/composite | critical-jOPS (jOPS) | 94700.00 | (R) -5.10% | -0.90% | -0.37% | | | max-jOPS (jOPS) | 113984.50 | (R) -3.90% | -0.65% | 0.65% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | repro-collection/mysql-workload | db transaction rate (transactions/min) | 245438.25 | (R) -3.88% | -0.13% | 0.24% | | | new order rate (orders/min) | 80985.75 | (R) -3.78% | -0.07% | 0.29% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | pts/pgbench | Scale: 1 Clients: 1 Read Only (TPS) | 63124.00 | (I) 2.90% | 0.74% | 0.85% | | | Scale: 1 Clients: 1 Read Only - Latency (ms) | 0.016 | (I) 5.49% | 1.05% | 1.05% | | | Scale: 1 Clients: 1 Read Write (TPS) | 974.92 | 0.11% | -0.08% | -0.03% | | | Scale: 1 Clients: 1 Read Write - Latency (ms) | 1.03 | 0.12% | -0.06% | -0.06% | | | Scale: 1 Clients: 250 Read Only (TPS) | 1915931.58 | (R) -2.25% | (I) 2.12% | 1.62% | | | Scale: 1 Clients: 250 Read Only - Latency (ms) | 0.13 | (R) -2.37% | (I) 2.09% | 1.69% | | | Scale: 1 Clients: 250 Read Write (TPS) | 855.67 | -1.36% | -0.14% | -0.12% | | | Scale: 1 Clients: 250 Read Write - Latency (ms) | 292.39 | -1.31% | -0.08% | -0.08% | | | Scale: 1 Clients: 1000 Read Only (TPS) | 1534130.08 | (R) -11.37% | 0.08% | 0.48% | | | Scale: 1 Clients: 1000 Read Only - Latency (ms) | 0.65 | (R) -11.38% | 0.08% | 0.44% | | | Scale: 1 Clients: 1000 Read Write (TPS) | 578.75 | -1.11% | 2.15% | -0.96% | | | Scale: 1 Clients: 1000 Read Write - Latency (ms) | 1736.98 | -1.26% | 2.47% | -0.90% | | | Scale: 100 Clients: 1 Read Only (TPS) | 57170.33 | 1.68% | 0.10% | 0.22% | | | Scale: 100 Clients: 1 Read Only - Latency (ms) | 0.018 | 1.94% | 0.00% | 0.96% | | | Scale: 100 Clients: 1 Read Write (TPS) | 836.58 | -0.37% | -0.41% | 0.07% | | | Scale: 100 Clients: 1 Read Write - Latency (ms) | 1.20 | -0.37% | -0.40% | 0.06% | | | Scale: 100 Clients: 250 Read Only (TPS) | 1773440.67 | -1.61% | 1.67% | 1.34% | | | Scale: 100 Clients: 250 Read Only - Latency (ms) | 0.14 | -1.40% | 1.56% | 1.20% | | | Scale: 100 Clients: 250 Read Write (TPS) | 5505.50 | -0.17% | -0.86% | -1.66% | | | Scale: 100 Clients: 250 Read Write - Latency (ms) | 45.42 | -0.17% | -0.85% | -1.67% | | | Scale: 100 Clients: 1000 Read Only (TPS) | 1393037.50 | (R) -10.31% | -0.19% | 0.53% | | | Scale: 100 Clients: 1000 Read Only - Latency (ms) | 0.72 | (R) -10.30% | -0.17% | 0.53% | | | Scale: 100 Clients: 1000 Read Write (TPS) | 5085.92 | 0.27% | 0.07% | -0.79% | | | Scale: 100 Clients: 1000 Read Write - Latency (ms) | 196.79 | 0.23% | 0.05% | -0.81% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | mmtests/hackbench | hackbench-process-pipes-1 (seconds) | 0.14 | -1.51% | -1.05% | -1.51% | | | hackbench-process-pipes-4 (seconds) | 0.44 | (I) 6.49% | (I) 5.42% | (I) 6.06% | | | hackbench-process-pipes-7 (seconds) | 0.68 | (R) -18.36% | (I) 3.40% | -0.41% | | | hackbench-process-pipes-12 (seconds) | 1.24 | (R) -19.89% | -0.45% | (R) -2.23% | | | hackbench-process-pipes-21 (seconds) | 1.81 | (R) -8.41% | -1.22% | (R) -2.46% | | | hackbench-process-pipes-30 (seconds) | 2.39 | (R) -9.06% | (R) -2.95% | -1.62% | | | hackbench-process-pipes-48 (seconds) | 3.18 | (R) -11.68% | (R) -4.10% | -0.26% | | | hackbench-process-pipes-79 (seconds) | 3.84 | (R) -9.74% | (R) -3.25% | (R) -2.45% | | | hackbench-process-pipes-110 (seconds) | 4.68 | (R) -6.57% | (R) -2.12% | (R) -2.25% | | | hackbench-process-pipes-141 (seconds) | 5.75 | (R) -5.86% | (R) -3.44% | (R) -2.89% | | | hackbench-process-pipes-172 (seconds) | 6.80 | (R) -4.28% | (R) -2.81% | (R) -2.44% | | | hackbench-process-pipes-203 (seconds) | 7.94 | (R) -4.01% | (R) -3.00% | (R) -2.17% | | | hackbench-process-pipes-234 (seconds) | 9.02 | (R) -3.52% | (R) -2.81% | (R) -2.20% | | | hackbench-process-pipes-256 (seconds) | 9.78 | (R) -3.24% | (R) -2.81% | (R) -2.74% | | | hackbench-process-sockets-1 (seconds) | 0.29 | 0.50% | 0.26% | 0.03% | | | hackbench-process-sockets-4 (seconds) | 0.76 | (I) 17.44% | (I) 16.31% | (I) 19.09% | | | hackbench-process-sockets-7 (seconds) | 1.16 | (I) 12.10% | (I) 9.78% | (I) 11.83% | | | hackbench-process-sockets-12 (seconds) | 1.86 | (I) 10.19% | (I) 9.83% | (I) 11.21% | | | hackbench-process-sockets-21 (seconds) | 3.12 | (I) 9.38% | (I) 9.20% | (I) 10.30% | | | hackbench-process-sockets-30 (seconds) | 4.30 | (I) 6.43% | (I) 6.11% | (I) 7.22% | | | hackbench-process-sockets-48 (seconds) | 6.58 | (I) 3.00% | (I) 2.19% | (I) 2.85% | | | hackbench-process-sockets-79 (seconds) | 10.56 | (I) 2.87% | (I) 3.31% | 3.10% | | | hackbench-process-sockets-110 (seconds) | 13.85 | -1.15% | (I) 2.33% | 0.22% | | | hackbench-process-sockets-141 (seconds) | 19.23 | -1.40% | (I) 14.53% | 2.64% | | | hackbench-process-sockets-172 (seconds) | 26.33 | (I) 3.52% | (I) 30.37% | (I) 4.32% | | | hackbench-process-sockets-203 (seconds) | 30.27 | 1.10% | (I) 27.20% | 0.32% | | | hackbench-process-sockets-234 (seconds) | 35.12 | 1.60% | (I) 28.24% | 1.28% | | | hackbench-process-sockets-256 (seconds) | 38.74 | 0.70% | (I) 28.74% | 0.53% | | | hackbench-thread-pipes-1 (seconds) | 0.17 | -1.32% | -0.76% | -0.67% | | | hackbench-thread-pipes-4 (seconds) | 0.45 | (I) 6.91% | (I) 7.64% | (I) 9.08% | | | hackbench-thread-pipes-7 (seconds) | 0.74 | (R) -7.51% | (I) 5.26% | (I) 2.82% | | | hackbench-thread-pipes-12 (seconds) | 1.32 | (R) -8.40% | (I) 2.32% | -0.53% | | | hackbench-thread-pipes-21 (seconds) | 1.95 | (R) -2.95% | 0.91% | (R) -2.00% | | | hackbench-thread-pipes-30 (seconds) | 2.50 | (R) -4.61% | 1.47% | -1.63% | | | hackbench-thread-pipes-48 (seconds) | 3.32 | (R) -5.45% | (I) 2.15% | 0.81% | | | hackbench-thread-pipes-79 (seconds) | 4.04 | (R) -5.53% | 1.85% | -0.53% | | | hackbench-thread-pipes-110 (seconds) | 4.94 | (R) -2.33% | 1.51% | 0.59% | | | hackbench-thread-pipes-141 (seconds) | 6.04 | (R) -2.47% | 1.15% | 0.24% | | | hackbench-thread-pipes-172 (seconds) | 7.15 | -0.91% | 1.48% | 0.45% | | | hackbench-thread-pipes-203 (seconds) | 8.31 | -1.29% | 0.77% | 0.40% | | | hackbench-thread-pipes-234 (seconds) | 9.49 | -1.03% | 0.77% | 0.65% | | | hackbench-thread-pipes-256 (seconds) | 10.30 | -0.80% | 0.42% | 0.30% | | | hackbench-thread-sockets-1 (seconds) | 0.31 | 0.05% | -0.05% | -0.43% | | | hackbench-thread-sockets-4 (seconds) | 0.79 | (I) 18.91% | (I) 16.82% | (I) 19.79% | | | hackbench-thread-sockets-7 (seconds) | 1.16 | (I) 12.57% | (I) 10.63% | (I) 12.95% | | | hackbench-thread-sockets-12 (seconds) | 1.87 | (I) 12.65% | (I) 12.26% | (I) 13.90% | | | hackbench-thread-sockets-21 (seconds) | 3.16 | (I) 11.62% | (I) 12.74% | (I) 13.89% | | | hackbench-thread-sockets-30 (seconds) | 4.32 | (I) 7.35% | (I) 8.89% | (I) 9.51% | | | hackbench-thread-sockets-48 (seconds) | 6.45 | (I) 2.69% | (I) 3.06% | (I) 3.74% | | | hackbench-thread-sockets-79 (seconds) | 10.15 | (I) 3.30% | 1.98% | (I) 2.76% | | | hackbench-thread-sockets-110 (seconds) | 13.45 | -0.25% | (I) 3.68% | 0.44% | | | hackbench-thread-sockets-141 (seconds) | 17.87 | (R) -2.18% | (I) 8.46% | 1.51% | | | hackbench-thread-sockets-172 (seconds) | 24.38 | 1.02% | (I) 24.33% | 1.38% | | | hackbench-thread-sockets-203 (seconds) | 28.38 | -0.99% | (I) 24.20% | 0.57% | | | hackbench-thread-sockets-234 (seconds) | 32.75 | -0.42% | (I) 24.35% | 0.72% | | | hackbench-thread-sockets-256 (seconds) | 36.49 | -1.30% | (I) 26.22% | 0.81% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ | pts/nginx | Connections: 200 (Requests Per Second) | 252332.60 | (I) 17.54% | -0.53% | -0.61% | | | Connections: 1000 (Requests Per Second) | 248591.29 | (I) 20.41% | 0.10% | 0.57% | +---------------------------------+----------------------------------------------------+---------------+-------------+------------+----------------+ Thanks, Ryan