From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1EB4514A82 for ; Wed, 8 Jan 2025 13:12:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736341929; cv=none; b=Y/3HCJ7G25JBWsq0OdvAUtNgL6aBvRShRrrCVKoXmKmkDZantj9hrqQZoJ+8zk72xpewzdQgr7HUpry3IRu1j6A7nspKbxgXqvq54OIX3ATtkC+ahLiZVpFEF+/KLeGFAooKxup4BgbufmsmoFBbGQkDsFXCozy6pK6R/C1u7Bw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736341929; c=relaxed/simple; bh=GlRf7SHn5YdYIrCDXB8W9xZoRXUN1SfBl5xH07iGKcM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=M+bMjRn47duaOiSXyCwOxXlsPslzQWrcAsmklU5ga2J7tqIgLR5+zmJ55QEZE6wLRHw6KOG3RpIvADF+eEX31vbgt6Bv25pfEGkeX8D0hXCElNnoDwA4YqmU9iWLpp2ii8rJgJ4krsB2HsaHqMeL/IAkpdgOZNFq8OIFqnxfYu4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Uzf61wki; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Uzf61wki" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=1gwBytc2Uw1knlGVMN8Hwj0rTYRZflK0KorhIHaL/j0=; b=Uzf61wkic7F4DhMOuxILz3b74W j3EVCUTeVT/KmpLqiHXZFz3U6ux9hNtN5dI0xfc2v5lH7vJDmSJ5atKIITbE3ZB+Co37Iu3igcqP4 QQF20OHHqY2TDK4Q6OGFlXc2cdw7AMn6wEHEwIBckEScMKxfo/bPhH3NXFTkdiwFA8Fl8v9MwV6zQ hiq0nXt/jsV0vcBDEGPo4MwCtT7lgvSk9iDoR7lhq5ZtCiJZkjRMTs1TX68T+0M3KqXGpuVgpVg+f DEtb354SCiJJywOiAPq3uYXX9xpumDhFs1lI1qTdfeIF3mVEuQ5v4CXGIOuoqnol1xKK6FIPY/jDv gS5SUvkA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tVVqv-0000000HOGW-3nDz; Wed, 08 Jan 2025 13:12:05 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 7D00A3005D6; Wed, 8 Jan 2025 14:12:05 +0100 (CET) Date: Wed, 8 Jan 2025 14:12:05 +0100 From: Peter Zijlstra To: Doug Smythies Cc: linux-kernel@vger.kernel.org, vincent.guittot@linaro.org Subject: Re: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Message-ID: <20250108131205.GO20870@noisy.programming.kicks-ass.net> References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> <20250107112606.GN20870@noisy.programming.kicks-ass.net> <20250107192340.GB36003@noisy.programming.kicks-ass.net> <001501db618c$67bd8170$37388450$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <001501db618c$67bd8170$37388450$@telus.net> I failed to realize the follow up email was private, so duplicating that here again, but also new content :-) On Tue, Jan 07, 2025 at 09:15:59PM -0800, Doug Smythies wrote: > On 2025.07.11:24 Peter Zijlstra wrote: > > What exact cgroup config are you having? /sys/kernel/debug/sched/debug > > should be able to tell you. > > I do not know. > I'll capture the above output, compress it, and send it to you. > > I did also boot with systemd.unified_cgroup_hierarchy=0 > and it made no difference. I think you need: "cgroup_disable=cpu noautogroup" to fully disable all the cpu-cgroup muck. Anyway: $ zcat cgroup2.txt.gz | grep -e yes -e turbo | awk '{print $2 "\t" $16}' yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope turbostat /autogroup-286 yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope yes /user.slice/user-1000.slice/session-1.scope turbostat /autogroup-286 That matches the scenario where I could reproduce, two competing groups. I'm seeing wild vruntime divergence when this happens -- this is definitely wonky. Basically the turbostat groups gets starved for a while while the yes group catches up. It looks like reweight_entity() is shooting out the cgroup entity to the right. So it builds up some negative lag (received surplus service) and then because turbostat goes sleep for a second, it's cgroup's share gets truncated to 2 and it shoots the cgroup entity out waaaaaaaay far. Thing is, waking up *should* fix that up again, but that doesn't appear to happen, leaving us up a creek. /me noodles a bit.... Does this help? diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c0e58e51801f..daa62cfa3092 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7000,6 +7063,13 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (flags & ENQUEUE_DELAYED) { requeue_delayed_entity(se); + se = se->parent; + for_each_sched_entity(se) { + cfs_rq = cfs_rq_of(se); + update_load_avg(cfs_rq, se, UPDATE_TG); + se_update_runnable(se); + update_cfs_group(se); + } return; }