mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Mihail Mihov <mihovmihailp@gmail.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: mingo@kernel.org, longman@redhat.com, chenridong@huaweicloud.com,
	 juri.lelli@redhat.com, vincent.guittot@linaro.org,
	dietmar.eggemann@arm.com,  rostedt@goodmis.org,
	bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
	 tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com,
	cgroups@vger.kernel.org,  linux-kernel@vger.kernel.org,
	jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io,
	 regressions@lists.linux.dev
Subject: [REGRESSION] core scheduling latency with idle load
Date: Mon, 16 Nov 2026 23:22:07 +0100	[thread overview]
Message-ID: <avt_mdiPQ5gwAisa@lima-fedora> (raw)
In-Reply-To: <20260605124052.227463677@infradead.org>

Hi Peter,

I was running some experiments that co-locate a latency-sensitive
service with SCHED_IDLE batch work in a cpu.idle=1 cgroup, with the
service in its own core-scheduling cookie. After updating to 7.3 its
tail latency got much worse. With the help of an LLM I bisected it to
85570f10a4c6 ("sched/eevdf: Move to a single runqueue") and reduced it
to the script below. It is still present in v7.4-rc3.

Reproducer (root, SMT machine, CONFIG_SCHED_CORE, cyclictest):

#!/bin/sh
set -e
mkdir -p /sys/fs/cgroup/be
echo 1 > /sys/fs/cgroup/be/cpu.idle
trap 'echo 1 > /sys/fs/cgroup/be/cgroup.kill; sleep 1; rmdir /sys/fs/cgroup/be' EXIT
# one SCHED_IDLE spinner per CPU in the cpu.idle=1 cgroup
chrt -i 0 sh -c 'echo $$ > /sys/fs/cgroup/be/cgroup.procs; for i in $(seq $(nproc)); do while :; do :; done & done'
sleep 1
# p99 from cyclictest's 1 us histogram, overflows (> 1 s) included
p99() { awk '/^[0-9]/ { h[$1+0] = $2; n += $2 } /Overflows/ { o = $4 } END { n += o; for (i = 0; c < n * 0.99; i++) c += h[i]; print "p99", i - 1, "us" }'; }
printf 'no cookie: '; cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99
printf 'cookie:    '; coresched new -- cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99

i7-1185G7 (4C/8T), vanilla kernels built with the same config (Gentoo's
distribution kernel config, HZ=300, PREEMPT_LAZY). p99 in us:

  kernel            no cookie  cookie
  85570f10a4c6^     54         59
  85570f10a4c6      53         3695
  v7.3.4            54         3276
  v7.4-rc3          53         3290

I also tried the two cpu.idle preemption fixes under discussion [1][2]
on v7.3.4; the cookie p99 stayed around 3.1 ms with both.

Changing cgroup_mode (up, smp, concur, max, tasks) on v7.4-rc3 did not
fix it.

On Fri, Jun 05, 2026 at 02:40:20PM +0200, Peter Zijlstra wrote:
> @@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
[...]
> -     se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
> -     se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);

In case it helps: adding these two calls back on v7.3.4 brought the
cookie p99 back to 55 us. I don't know whether that is the right thing
to do, only that it changes this result.

Thanks,
Mihail

[1] https://lore.kernel.org/all/20261019182622.1584454-1-vincent.guittot@linaro.org/
[2] https://lore.kernel.org/all/20261105165525.011876858@infradead.org/

#regzbot introduced: 85570f10a4c6

  parent reply	other threads:[~2026-11-16 22:22 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-05 12:40 [PATCH v3 0/7] sched: Flatten the pick Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 1/7] sched/fair: Add cgroup_mode switch Peter Zijlstra
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 2/7] sched/fair: Add cgroup_mode: up Peter Zijlstra
2026-06-05 15:07   ` Peter Zijlstra
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 3/7] sched/fair: Add cgroup_mode: max Peter Zijlstra
2026-06-10 15:09   ` Waiman Long
2026-06-10 15:42     ` Waiman Long
2026-06-11 13:49       ` Peter Zijlstra
2026-06-11 13:47     ` Peter Zijlstra
2026-06-11 20:57       ` Waiman Long
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 4/7] sched/fair: Add cgroup_mode: concur Peter Zijlstra
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks Peter Zijlstra
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur Peter Zijlstra
2026-06-30  9:03   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 7/7] sched/eevdf: Move to a single runqueue Peter Zijlstra
2026-06-20  3:54   ` Chen, Yu C
2026-06-26 11:40     ` Peter Zijlstra
2026-06-29 14:02       ` Vincent Guittot
2026-06-30  9:03     ` [tip: sched/core] sched/fair: Fix overflow in update_tg_cfs_runnable() tip-bot2 for Chen, Yu C
2026-06-30  9:03   ` [tip: sched/core] sched/eevdf: Move to a single runqueue tip-bot2 for Peter Zijlstra (Intel)
2026-09-23 14:21   ` [REGRESSION] [PATCH v3 7/7] " Aishwarya Rambhadran
2026-09-23 16:48     ` Chen Yu
2026-09-24  7:05       ` K Prateek Nayak
2026-09-24  8:19       ` Mike Galbraith
2026-09-28  3:12         ` Chen Yu
2026-09-28 13:00           ` Mike Galbraith
2026-10-07 12:05             ` Aishwarya Rambhadran
2026-10-12 11:27               ` Mike Galbraith
2026-10-16 13:18   ` Jakub Okoński
2026-10-18  6:49     ` Mike Galbraith
2026-10-19 17:01       ` Vincent Guittot
2026-10-19 17:35         ` Jakub Okoński
2026-10-19 18:04           ` Vincent Guittot
2026-10-19 18:24             ` Jakub Okoński
2026-10-19 18:26               ` Vincent Guittot
2026-10-21 11:55                 ` Vincent Guittot
2026-10-22 12:05                   ` Jakub Okoński
2026-10-30 15:48       ` Chengfeng Lin
2026-10-22  2:12     ` Mike Galbraith
2026-11-16 22:22   ` Mihail Mihov [this message]
2026-06-09  5:37 ` [PATCH v3 0/7] sched: Flatten the pick K Prateek Nayak
2026-06-12  2:29 ` Shubhang Kaushik
2026-08-17 16:05 ` Szabina Korbai
2026-08-17 16:35   ` K Prateek Nayak
2026-08-18  9:04     ` Szabina Korbai
2026-08-18  9:16       ` Peter Zijlstra
2026-08-21 10:37         ` Szabina Korbai
2026-08-24 14:51         ` Chen Yu
2026-08-24 13:53           ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avt_mdiPQ5gwAisa@lima-fedora \
    --to=mihovmihailp@gmail.com \
    --cc=bsegall@google.com \
    --cc=cgroups@vger.kernel.org \
    --cc=chenridong@huaweicloud.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=hannes@cmpxchg.org \
    --cc=jstultz@google.com \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=mkoutny@suse.com \
    --cc=peterz@infradead.org \
    --cc=qyousef@layalina.io \
    --cc=regressions@lists.linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=tj@kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®