From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACA431FC7C2 for ; Wed, 8 Jan 2025 15:48:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736351311; cv=none; b=NdIBHLC1IwsiJS0oyHuPNwIrlUDHrWq7mknCtBfG9uj4GfzyOySfgcDQ7vzgIAxN9bhxyyhY46eIAkzaJR/FMMYEbRJ6thQIpgTbLFkytAdKSYhYpPIVjOBm5g32SRKWN/3S/So/f9uLT3UEjMjNH4hxheDidzbYM527h/0EBWs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736351311; c=relaxed/simple; bh=xJ2NUnE2OIPZxn306WHOI+sk7Ewav/5bzbfn/HF0Ye4=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=lN7Q07yIP9Dtb8cSoNJ98ONVX1EcKJrfuaRR002hIoUaPw43HeJvm4uuvH+0Ljpf98TABBOJIQw7RlRumeyIucrcMCNkGGlWX/ismZMQJv2E5gMd/eXIT/bC8GYSO1YweEg7HgcNWNzsN/d99fpsNER6+S/gEiZTMyX1srze2L8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=D+Qh+5EO; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="D+Qh+5EO" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2f43da61ba9so20282114a91.2 for ; Wed, 08 Jan 2025 07:48:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736351309; x=1736956109; darn=vger.kernel.org; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=iLKWca6uIJFWoWVCX9UFI8+pcPx97MLr4FZniWAyeUQ=; b=D+Qh+5EOMx1UPchng31qNoXJcVMMpSs247RSLQonraapQzfKKfMWe9Ez06F0KEnlqQ t35qf4hE8Fja7O6puLeDvqRsVGdbRx8dJAibS1fSjfwjUTIhpCBglIyYA+IMMfiNW7fA dmpAm4qptVEAf3bwVW06a6WnUb70qf9it2Ov39w4YjxIrYVtkLr3xCEEuaIRFMOuI00Z jqWWMICT3EmDeaavWlUbzsigNGU8aJiGN1StR5A7fmgCwK/gPVFe+MGNO3poQ6IhZUtM 1rXftKWTHu8xeboc4Jf9XskNzExlXvk8amenmopujuoKhPjBVLEsjD4Sj6U2K7SzLJAC l+dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736351309; x=1736956109; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iLKWca6uIJFWoWVCX9UFI8+pcPx97MLr4FZniWAyeUQ=; b=ugKbiODr5yZoUNY3bYvC7+BbSYFENT+fq4Q1pn3tu0lRVZ1sBJnPRx1bxm0fTpU7pH PyjooyXoJiswb5Tm82Iha1DscPNhKlTZT76svDeZMQfaZCuezyltqruU+a+L1HoLjb7Z 9fCMprDuogS7gkKINP+u2hYmUXd6CL+62KSiYPL1TBpG4GO6MBAfA1DEROHZ23FJpE8Q rVIEQ4trIuV1H6/OK0m+2i4VoQho99E1s3nz8dfg9SAUgy9kx9WMtFOYrh4i5HkX5D7v 8AJzuuEEaiJM/iuBJ53juWCSG+jW4Lhi74lz05KnkvB260HfeBvB8UyCurnemC2OMxMI uF5w== X-Gm-Message-State: AOJu0YzNkRCRxLJVz/D8REq/9hJ9ZMO9hira+X0dyC4/AXfUK8HWw452 gAVYEFZpOQOqEF0y3gSvI0w+pG4NDr0I6Z9x1Kv4Sgbbn53mZ4QpWbbfAjr6vx8= X-Gm-Gg: ASbGncvebMHJ109OQ+VloGTpSmIVfpFOtqtB8Mjws2k4qVFYcR/1/QAWGqUItHqGmuW qvy2ssZY4kwsHMUGTwxxQXh69Hh2oIMU53MwBZUVV89A4WIAAWCMMwNCDjT9/qXPqp/53KHRR9X x3Ry8gnPWaYhemWQoA7ek3vw10PLftxuOGgQAKp2euyMNgOlZKZe5alpOMvs6JxzfCfYL914ReP a1FdfYiS/zUlBf7+tKTLuQFvD+tqU/Wn13H1ezVu9JXza2bYCPtex/025or8n+VXiQP07+leIh5 4VyZVksRbMKFnnc3ci8VVA== X-Google-Smtp-Source: AGHT+IF0DnfsqVR4dQBlNXdZi75KC9wwHXCMelIBb0IA2W8WAN/V2cD3rzTZGpCpyrK/AfFLmCsMSQ== X-Received: by 2002:a17:90b:4d0c:b0:2ee:8619:210b with SMTP id 98e67ed59e1d1-2f5490bd21dmr4902124a91.29.1736351308887; Wed, 08 Jan 2025 07:48:28 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f54a2ad284sm1694765a91.27.2025.01.08.07.48.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Jan 2025 07:48:28 -0800 (PST) From: "Doug Smythies" To: "'Peter Zijlstra'" Cc: , , "Doug Smythies" References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> <20250107112606.GN20870@noisy.programming.kicks-ass.net> <20250107192340.GB36003@noisy.programming.kicks-ass.net> <001501db618c$67bd8170$37388450$@telus.net> <20250108131205.GO20870@noisy.programming.kicks-ass.net> In-Reply-To: <20250108131205.GO20870@noisy.programming.kicks-ass.net> Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Wed, 8 Jan 2025 07:48:29 -0800 Message-ID: <001b01db61e4$c3ce5a40$4b6b0ec0$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQJdvfVp9nqa9troPEySggPg+fzySgGTHC4vAVKdTGwBPBBbIAH1WJEKAgKs/R8BZUmAdQHtSwJ2AyWxwjwBnjptfbGGyZaw Content-Language: en-ca On 2025.01.08 05:12 Peter Zijlstra wrote: > On Tue, Jan 07, 2025 at 09:15:59PM -0800, Doug Smythies wrote: >> On 2025.07.11:24 Peter Zijlstra wrote: > >>> What exact cgroup config are you having? /sys/kernel/debug/sched/debug >>> should be able to tell you. >> >> I do not know. >> I'll capture the above output, compress it, and send it to you. >> >> I did also boot with systemd.unified_cgroup_hierarchy=0 >> and it made no difference. > > I think you need: "cgroup_disable=cpu noautogroup" to fully disable all > the cpu-cgroup muck. Anyway: > > $ zcat cgroup2.txt.gz | grep -e yes -e turbo | awk '{print $2 "\t" $16}' > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > turbostat /autogroup-286 > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > yes /user.slice/user-1000.slice/session-1.scope > turbostat /autogroup-286 > > That matches the scenario where I could reproduce, two competing groups. > > I'm seeing wild vruntime divergence when this happens -- this is > definitely wonky. Basically the turbostat groups gets starved for a > while while the yes group catches up. > > It looks like reweight_entity() is shooting out the cgroup entity to the > right. > > So it builds up some negative lag (received surplus service) and then > because turbostat goes sleep for a second, it's cgroup's share gets > truncated to 2 and it shoots the cgroup entity out waaaaaaaay far. > > Thing is, waking up *should* fix that up again, but that doesn't appear > to happen, leaving us up a creek. > > /me noodles a bit.... > > Does this help? Sorry, but no it did not help. > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index c0e58e51801f..daa62cfa3092 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -7000,6 +7063,13 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) > > if (flags & ENQUEUE_DELAYED) { > requeue_delayed_entity(se); > + se = se->parent; > + for_each_sched_entity(se) { > + cfs_rq = cfs_rq_of(se); > + update_load_avg(cfs_rq, se, UPDATE_TG); > + se_update_runnable(se); > + update_cfs_group(se); > + } > return; > }