From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97A872D9EC2 for ; Fri, 30 Jan 2026 09:47:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769766445; cv=none; b=eiLN67NpMSTutOCDCpUw0SWaT54luXpqrdCjN/ttIXHqeZn/K0fVsFQxg4XUsCNqpn1bgoDBvgkrfOxigwf6Oc4b4e1tZViNt8LnO7P3M6ttihg4c6dhphlLXp2CVrGS+OxL+h1vVWNC7dEEgOuhRzyjcutyc+TDyFAeXn2x50I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769766445; c=relaxed/simple; bh=iSlTT1I0NuIhQRv2468yXZwTuwx4cZvUP25ewWENwnw=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IO6CFp11bfJueoTZKDm07ALYgkEAJy1Kze9OVapclwxBTJvVvsAId1xnzLATsN1DcmVZ+aFaQ89FbnCzEom7hM0Q0+RCDU5kIcq7zDJ0YaVSiwnvEKpr0X+esMbdjRX1pGJ0hFw06lKBc9IKkignnwBFIR4qDS5gWVz5knAFiuM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=CmHCgrlL; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CmHCgrlL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=pIs0NMRsyfAwIcaR5r6HVvUwYdHyo1fldgrnn8JRq5I=; b=CmHCgrlLarBMO5Ph7rl3ki2t9u Yixr34pF0PFjJ777FdgMYABXSe56/N6/pVpUb/vLayK3N/VxuGIdDpnscWAGiSC/Ir4S0DiO3sgs7 cCvo6RFqtZ147w8cH4/EIX7PCyxvQOkj5cz+rwK2TkdP7bHnQy2UiGgOnQZ9mmfQVve2BoFOLplJI qX7UzLxe4ZlL45PX9syHQH4GGCHTsA+LPkGOT4jVBmA5MyYlV5B5p+gUmnuRFBag90/FFusOoxUCm GRmfve4oHkYYxzc/7L5ssWSaRcwnx3dMC5JgxzK18dnNMnzeQSZawPNJLfHNThVaNF72/fHkfLK5e NrO4UpWQ==; Received: from 2001-1c00-8d85-5700-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:5700:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1vll5n-0000000BZrD-3GIk; Fri, 30 Jan 2026 09:47:08 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 35D55303324; Fri, 30 Jan 2026 10:47:06 +0100 (CET) Message-ID: <20260130094608.190020747@infradead.org> User-Agent: quilt/0.68 Date: Fri, 30 Jan 2026 10:34:42 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, wangtao554@huawei.com, quzicheng@huawei.com, kprateek.nayak@amd.com, wuyun.abel@bytedance.com, dsmythies@telus.net Subject: [PATCH 3/4] sched/fair: Increase weight bits for avg_vruntime References: <20260130093439.803225718@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Due to the zero_vruntime patch, the deltas are now a lot smaller and measurement with kernel-build and hackbench runs show about 54 bits used. This ensures avg_vruntime() tracks the full weight range, reducing numerical artifacts in reweight and the like. Also, lets keep the paranoid debug code around this time. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 14 ++++++++ kernel/sched/fair.c | 77 +++++++++++++++++++++++++++++++++++++++++------- kernel/sched/features.h | 2 + kernel/sched/sched.h | 3 + 4 files changed, 83 insertions(+), 13 deletions(-) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -8,6 +8,7 @@ */ #include #include +#include #include "sched.h" /* @@ -790,10 +791,13 @@ static void print_rq(struct seq_file *m, void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) { - s64 left_vruntime = -1, zero_vruntime, right_vruntime = -1, left_deadline = -1, spread; + s64 left_vruntime = -1, right_vruntime = -1, left_deadline = -1, spread; + s64 zero_vruntime = -1, sum_w_vruntime = -1; struct sched_entity *last, *first, *root; struct rq *rq = cpu_rq(cpu); + unsigned int sum_shift; unsigned long flags; + u64 sum_weight; #ifdef CONFIG_FAIR_GROUP_SCHED SEQ_printf(m, "\n"); @@ -814,6 +818,9 @@ void print_cfs_rq(struct seq_file *m, in if (last) right_vruntime = last->vruntime; zero_vruntime = cfs_rq->zero_vruntime; + sum_w_vruntime = cfs_rq->sum_w_vruntime; + sum_weight = cfs_rq->sum_weight; + sum_shift = cfs_rq->sum_shift; raw_spin_rq_unlock_irqrestore(rq, flags); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "left_deadline", @@ -822,6 +829,11 @@ void print_cfs_rq(struct seq_file *m, in SPLIT_NS(left_vruntime)); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "zero_vruntime", SPLIT_NS(zero_vruntime)); + SEQ_printf(m, " .%-30s: %Ld (%d bits)\n", "sum_w_vruntime", + sum_w_vruntime, ilog2(abs(sum_w_vruntime))); + SEQ_printf(m, " .%-30s: %Lu\n", "sum_weight", + sum_weight); + SEQ_printf(m, " .%-30s: %u\n", "sum_shift", sum_shift); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "avg_vruntime", SPLIT_NS(avg_vruntime(cfs_rq))); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "right_vruntime", --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -650,15 +650,20 @@ static inline s64 entity_key(struct cfs_ * Since zero_vruntime closely tracks the per-task service, these * deltas: (v_i - v0), will be in the order of the maximal (virtual) lag * induced in the system due to quantisation. - * - * Also, we use scale_load_down() to reduce the size. - * - * As measured, the max (key * weight) value was ~44 bits for a kernel build. */ -static void -sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se) +static inline unsigned long avg_vruntime_weight(struct cfs_rq *cfs_rq, unsigned long w) +{ +#ifdef CONFIG_64BIT + if (cfs_rq->sum_shift) + w = max(2UL, w >> cfs_rq->sum_shift); +#endif + return w; +} + +static inline void +__sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se) { - unsigned long weight = scale_load_down(se->load.weight); + unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight); s64 key = entity_key(cfs_rq, se); cfs_rq->sum_w_vruntime += key * weight; @@ -666,9 +671,59 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq } static void +sum_w_vruntime_add_paranoid(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + unsigned long weight; + s64 key, tmp; + +again: + weight = avg_vruntime_weight(cfs_rq, se->load.weight); + key = entity_key(cfs_rq, se); + + if (check_mul_overflow(key, weight, &key)) + goto overflow; + + if (check_add_overflow(cfs_rq->sum_w_vruntime, key, &tmp)) + goto overflow; + + cfs_rq->sum_w_vruntime = tmp; + cfs_rq->sum_weight += weight; + return; + +overflow: + /* + * There's gotta be a limit -- if we're still failing at this point + * there's really nothing much to be done about things. + */ + BUG_ON(cfs_rq->sum_shift >= 10); + cfs_rq->sum_shift++; + + /* + * Note: \Sum (k_i * (w_i >> 1)) != (\Sum (k_i * w_i)) >> 1 + */ + cfs_rq->sum_w_vruntime = 0; + cfs_rq->sum_weight = 0; + + for (struct rb_node *node = cfs_rq->tasks_timeline.rb_leftmost; + node; node = rb_next(node)) + __sum_w_vruntime_add(cfs_rq, __node_2_se(node)); + + goto again; +} + +static void +sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (sched_feat(PARANOID_AVG)) + return sum_w_vruntime_add_paranoid(cfs_rq, se); + + __sum_w_vruntime_add(cfs_rq, se); +} + +static void sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se) { - unsigned long weight = scale_load_down(se->load.weight); + unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight); s64 key = entity_key(cfs_rq, se); cfs_rq->sum_w_vruntime -= key * weight; @@ -695,7 +750,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq) long load = cfs_rq->sum_weight; if (curr && curr->on_rq) { - unsigned long weight = scale_load_down(curr->load.weight); + unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight); avg += entity_key(cfs_rq, curr) * weight; load += weight; @@ -5170,9 +5225,9 @@ place_entity(struct cfs_rq *cfs_rq, stru */ load = cfs_rq->sum_weight; if (curr && curr->on_rq) - load += scale_load_down(curr->load.weight); + load += avg_vruntime_weight(cfs_rq, curr->load.weight); - lag *= load + scale_load_down(se->load.weight); + lag *= load + avg_vruntime_weight(cfs_rq, se->load.weight); if (WARN_ON_ONCE(!load)) load = 1; lag = div_s64(lag, load); --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -58,6 +58,8 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true) SCHED_FEAT(DELAY_DEQUEUE, true) SCHED_FEAT(DELAY_ZERO, true) +SCHED_FEAT(PARANOID_AVG, false) + /* * Allow wakeup-time preemption of the current task: */ --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -679,8 +679,9 @@ struct cfs_rq { s64 sum_w_vruntime; u64 sum_weight; - u64 zero_vruntime; + unsigned int sum_shift; + #ifdef CONFIG_SCHED_CORE unsigned int forceidle_seq; u64 zero_vruntime_fi;