From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0143B3655CB; Tue, 30 Jun 2026 09:03:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782810242; cv=none; b=k8YlsEj8O94or3ieLPylm7LC9Gzfa7eO2C5IlpnSTb03P/cIaIDamhlAZHVhn4AzVHrkcSlQ7JB3d35f+QUOvdxmkgPhvRSvMH4vMe9qI3TNh+wQtXt8heReAN3sIvsH1kCTK5gw79uuUTUVywIu7uJWXOKkkkkxI2a/3sECqs4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782810242; c=relaxed/simple; bh=Ng/78l06Pmi3GnjQ4i2pjYk43O2unr60Cd/qxZcEJqo=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=Ts8TYNYicpXoOnEWNvY2WrOsY1BbJNVbEO1QJI0daYNT+y08RmFpyKmb1cWPgG2ry7bOuUUuIeQWVnBsMOnzbfw1Dido9zJ1G1QULT7/k/IFOIlFD9Uom1iU+F9IejrLuc27fLJpZpjA8/PU2tzZpfyHHPH24n5jaVdol4VjNFk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=U8INhLsE; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=RZyVFYCZ; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="U8INhLsE"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="RZyVFYCZ" Date: Tue, 30 Jun 2026 09:03:52 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1782810234; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lR/ZULIzvJGA0whpHijYCFHo178WcjR2wc/PYTS2neg=; b=U8INhLsEsBzR1TD5AGstVtU83BhcaYUOtRzfe93gAYZ2PyajxrgYNAeUsPqhSbWqgJRGnn q5siZWtbt9oBQnelCUcRo1IXjO9B8fHzxPoUDFX0tduY/8DfXeCBK2pIA7exhdwAypu+gU a/hBdjjxQsFi94pmf8V6pqa93IfrtyqW0eEa7FrPnJ2UUbhDrGEViBEgV3v9u0X3Xhi83M V1TSwOqaZxzojZpRV088ptcNpwQPD60IFDHZLIigfu6LM7E8E+SNTQfpWRUsBsNkKjnpxe NWxhfSAf9EgbAM+g9y6J9BemE5ruCRETyTRPGJEZVN2LoFgqeGm9u1nK9qlkjA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1782810234; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lR/ZULIzvJGA0whpHijYCFHo178WcjR2wc/PYTS2neg=; b=RZyVFYCZ4oHE3R05cy6yZanOIzJ9Fl5nBez+5Oed2+VunddbWR2dcOOvFi/Sy6i2/lj5Kb s9qmu8FBBDAhVKBQ== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/fair: Add cgroup_mode: concur Cc: "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260605124051.740585993@infradead.org> References: <20260605124051.740585993@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <178281023260.3843924.17401667014596472967.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: 5c0b58bd1c9f09c4e9363665702c82694c6ee7a0 Gitweb: https://git.kernel.org/tip/5c0b58bd1c9f09c4e9363665702c82694c6= ee7a0 Author: Peter Zijlstra AuthorDate: Thu, 12 Mar 2026 16:04:18 +01:00 Committer: Peter Zijlstra CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00 sched/fair: Add cgroup_mode: concur Improve upon the previous scheme ("max") by no longer assuming maximal concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully: F_g_n' =3D min(M, N) * F_g_n Notably this is the first mode where: avg(F_g_n) =3D 1 In the single task case it reduces to ("smp") and then it nicely scales up until it hits N, where it behaves like ("max"). This is no longer clipped at nice -20. Strictly speaking it isn't different from the normal SMP scenario where all tasks are extremely unbalanced. There are no unnatural inflations in this scheme. The meaning of "cpu.weight" would be: weight per active CPU. NOTE: Compute the group wide number of tasks by extending the tg->load_avg computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on cfs_rq->h_nr_running. Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260605124051.740585993%40infradead.org --- kernel/sched/debug.c | 1 +- kernel/sched/fair.c | 43 ++++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 3 +++- 3 files changed, 40 insertions(+), 7 deletions(-) diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 84e0ac4..a5e3f8a 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -640,6 +640,7 @@ static int cgroup_mode =3D 1; static const char *cgroup_mode_str[] =3D { "up", "smp", + "concur", "max", }; =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b556d53..64181e4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4846,6 +4846,11 @@ static int tg_cpus(struct task_group *tg) return nr; } =20 +static inline int tg_tasks(struct task_group *tg) +{ + return max(1, atomic_long_read(&tg->runnable_avg) >> SCHED_CAPACITY_SHIFT); +} + /* * Func: min(fraction(nr_cpus * tg->shares), nice -20) * @@ -4863,6 +4868,20 @@ static long calc_max_shares(struct cfs_rq *cfs_rq) } =20 /* + * Func: fraction(nr * tg->shares); nr =3D min(nr_tasks, nr_cpus) + * + * Scales between "smp" and "max" in a natural way. No longer needs clipping + * since there are no unnatural inflations like with "max". + */ +static long calc_concur_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + int nr =3D min(tg_tasks(tg), tg_cpus(tg)); + long tg_shares =3D READ_ONCE(tg->shares); + return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares); +} + +/* * Func: fraction(tg->shares) * * This infamously results in tiny shares when you have many CPUs. @@ -4897,6 +4916,9 @@ void __sched_cgroup_mode_update(int mode) func =3D &calc_smp_shares; break; case 2: + func =3D &calc_concur_shares; + break; + case 3: func =3D &calc_max_shares; break; } @@ -5043,7 +5065,7 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs= _rq) */ static inline void update_tg_load_avg(struct cfs_rq *cfs_rq) { - long delta; + long dl, dr; u64 now; =20 /* @@ -5064,17 +5086,21 @@ static inline void update_tg_load_avg(struct cfs_rq *= cfs_rq) if (now - cfs_rq->last_update_tg_load_avg < NSEC_PER_MSEC) return; =20 - delta =3D cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib; - if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) { - atomic_long_add(delta, &cfs_rq->tg->load_avg); + dl =3D cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib; + dr =3D cfs_rq->avg.runnable_avg - cfs_rq->tg_runnable_avg_contrib; + if (abs(dl) > cfs_rq->tg_load_avg_contrib / 64 || + abs(dr) > cfs_rq->tg_runnable_avg_contrib / 64) { + atomic_long_add(dl, &cfs_rq->tg->load_avg); + atomic_long_add(dr, &cfs_rq->tg->runnable_avg); cfs_rq->tg_load_avg_contrib =3D cfs_rq->avg.load_avg; + cfs_rq->tg_runnable_avg_contrib =3D cfs_rq->avg.runnable_avg; cfs_rq->last_update_tg_load_avg =3D now; } } =20 static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq) { - long delta; + long dl, dr; u64 now; =20 /* @@ -5084,9 +5110,12 @@ static inline void clear_tg_load_avg(struct cfs_rq *cf= s_rq) return; =20 now =3D rq_clock(rq_of(cfs_rq)); - delta =3D 0 - cfs_rq->tg_load_avg_contrib; - atomic_long_add(delta, &cfs_rq->tg->load_avg); + dl =3D 0 - cfs_rq->tg_load_avg_contrib; + dr =3D 0 - cfs_rq->tg_runnable_avg_contrib; + atomic_long_add(dl, &cfs_rq->tg->load_avg); + atomic_long_add(dr, &cfs_rq->tg->runnable_avg); cfs_rq->tg_load_avg_contrib =3D 0; + cfs_rq->tg_runnable_avg_contrib =3D 0; cfs_rq->last_update_tg_load_avg =3D now; } =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 922170b..fd267bf 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -493,6 +493,8 @@ struct task_group { * will also be accessed at each tick. */ atomic_long_t load_avg ____cacheline_aligned; + atomic_long_t runnable_avg; + #endif /* CONFIG_FAIR_GROUP_SCHED */ =20 #ifdef CONFIG_RT_GROUP_SCHED @@ -722,6 +724,7 @@ struct cfs_rq { #ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg; unsigned long tg_load_avg_contrib; + unsigned long tg_runnable_avg_contrib; long propagate; long prop_runnable_sum; =20