From: Chen Yu <yu.c.chen@intel.com>
To: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>, <mingo@kernel.org>,
<longman@redhat.com>, <chenridong@huaweicloud.com>,
<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
<dietmar.eggemann@arm.com>, <rostedt@goodmis.org>,
<bsegall@google.com>, <mgorman@suse.de>, <vschneid@redhat.com>,
<tj@kernel.org>, <hannes@cmpxchg.org>, <mkoutny@suse.com>,
<cgroups@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<jstultz@google.com>, <kprateek.nayak@amd.com>,
<qyousef@layalina.io>, Ryan Roberts <ryan.roberts@arm.com>,
<chen.yu@linux.dev>
Subject: Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Date: Thu, 24 Sep 2026 00:48:10 +0800 [thread overview]
Message-ID: <arQCykkkCnADu5ek@chenyu-dev> (raw)
In-Reply-To: <bd81bbec-ee59-401a-bec6-116196ad3a34@arm.com>
On Wed, Sep 23, 2026 at 07:51:35PM +0530, Aishwarya Rambhadran wrote:
> Hi Peter,
>
> We have observed some schbench p99 request latency regressions while
> comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal)
> using Fastpath (Linux kernel performance benchmarking tool).
>
> The schbench suite as a whole does not show a significant regression,
> and the results appear to be a mix of improvements and regressions
> depending on the thread configuration. However, a few configurations
> show fairly large p99 request latency regressions. For example,
> comparing v7.2 against v7.3-rc4:
>
> message-thread:16, worker-thread:16
> v7.2 = 662869 us
> v7.3-rc4 = 969557 us
> change% = -31.63%
>
> message-thread:64, worker-thread:4
> v7.2 = 730453 us
> v7.3-rc4 = 942251 us
> change% = -22.48%
>
> message-thread:32, worker-thread:4
> v7.2 = 74251 us
> v7.3-rc4 = 56725 us
> change% = +30.90%
>
>
> So this does not appear to be a general latency degradation; some
> configs improve while others regress.
>
> I also ran a Fastpath automated git bisection for the m 16, t 16 p99
> request latency regression between v7.2 and v7.3-rc1, which identified
> this commit as the first bad commit. The original regression was ~30.9%,
> & the bisection reproducibility check showed ~27.8% regression with 20
> repeats across 2 boot sessions.
>
> This commit looks relevant since it changes EEVDF runqueue organization
> and is intended, in part, to address latency issues with hierarchical fair/
> cgroup scheduling. I also noticed that there have been fixes on top of this
> change, including 68e37487810a ("sched/fair: Fix flat hierarchy"), which
> fixes vruntime handling and has a Fixes: 85570f10a4c6 tag.
>
> Given that the overall schbench result is mostly neutral while some p99
> configurations move considerably in both directions, I wanted to check
> whether this could be an expected trade-off from the single-runqueue
> change, or whether the p99 regressions are worth investigating further.
>
> I can run additional tests or provide the full results/bisection data if
> useful.
>
>
Would disabling WA_WEIGHT help? I happened to find that this workaround works
for me to restore netperf performance (though it is a throughput score rather
than a latency one), it seems that task_h_load() becomes much
larger(also on sched/task_h_load):
echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/
thanks,
Chenyu
next prev parent reply other threads:[~2026-09-23 17:01 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-05 12:40 [PATCH v3 0/7] sched: Flatten the pick Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 1/7] sched/fair: Add cgroup_mode switch Peter Zijlstra
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 2/7] sched/fair: Add cgroup_mode: up Peter Zijlstra
2026-06-05 15:07 ` Peter Zijlstra
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 3/7] sched/fair: Add cgroup_mode: max Peter Zijlstra
2026-06-10 15:09 ` Waiman Long
2026-06-10 15:42 ` Waiman Long
2026-06-11 13:49 ` Peter Zijlstra
2026-06-11 13:47 ` Peter Zijlstra
2026-06-11 20:57 ` Waiman Long
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 4/7] sched/fair: Add cgroup_mode: concur Peter Zijlstra
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks Peter Zijlstra
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur Peter Zijlstra
2026-06-30 9:03 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 7/7] sched/eevdf: Move to a single runqueue Peter Zijlstra
2026-06-20 3:54 ` Chen, Yu C
2026-06-26 11:40 ` Peter Zijlstra
2026-06-29 14:02 ` Vincent Guittot
2026-06-30 9:03 ` [tip: sched/core] sched/fair: Fix overflow in update_tg_cfs_runnable() tip-bot2 for Chen, Yu C
2026-06-30 9:03 ` [tip: sched/core] sched/eevdf: Move to a single runqueue tip-bot2 for Peter Zijlstra (Intel)
2026-09-23 14:21 ` [REGRESSION] [PATCH v3 7/7] " Aishwarya Rambhadran
2026-09-23 16:48 ` Chen Yu [this message]
2026-09-24 7:05 ` K Prateek Nayak
2026-09-24 8:19 ` Mike Galbraith
2026-06-09 5:37 ` [PATCH v3 0/7] sched: Flatten the pick K Prateek Nayak
2026-06-12 2:29 ` Shubhang Kaushik
2026-08-17 16:05 ` Szabina Korbai
2026-08-17 16:35 ` K Prateek Nayak
2026-08-18 9:04 ` Szabina Korbai
2026-08-18 9:16 ` Peter Zijlstra
2026-08-21 10:37 ` Szabina Korbai
2026-08-24 14:51 ` Chen Yu
2026-08-24 13:53 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=arQCykkkCnADu5ek@chenyu-dev \
--to=yu.c.chen@intel.com \
--cc=aishwarya.rambhadran@arm.com \
--cc=bsegall@google.com \
--cc=cgroups@vger.kernel.org \
--cc=chen.yu@linux.dev \
--cc=chenridong@huaweicloud.com \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=jstultz@google.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=mkoutny@suse.com \
--cc=peterz@infradead.org \
--cc=qyousef@layalina.io \
--cc=rostedt@goodmis.org \
--cc=ryan.roberts@arm.com \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®