From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F8D425948E for ; Wed, 8 Jan 2025 05:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736313365; cv=none; b=Byv4dgki6QG6K/f72QHC+GITW0ekBpPPUEyzBGQ47+7OK5dyVELCNIaYO9m6EW7EtT0gsk03E16PKBYca7+O/uHbrYSxOrMjhOe9n7zg/Yy7XBHmgDkxrBPxT+yniLlOpQ6leaVHziJiVKelHy9kPrTWFgO4Z3txA9KUtJowCPY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736313365; c=relaxed/simple; bh=l2MOTZTxYDswbkpeNiPWFLrJXV+6QewERWK7hmE5Udg=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=QBcfyQWVZqU5Q5dbNjnTxm5kkRxu+uMZZl8iIVJKT6LgJ0o6dEOlB5ID7/oR65wc805Ylc/EnVWLv5WV9umdkGrhORDQKbKgUA19jZeZ8Enb8xQwxeHJWmdui+8XmO1ryObsQh8eNp/ttQwXypZV8g2tE+OqhqNbIonb9l5ZODM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=L8TAEA2H; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="L8TAEA2H" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2ef72924e53so24009782a91.3 for ; Tue, 07 Jan 2025 21:16:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736313361; x=1736918161; darn=vger.kernel.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=qfifYMRclRsFGxPpRF2ADQcpKEAziZDiuZWNz8YnOOw=; b=L8TAEA2HxOZ1ed0UoJbViGl61b4ted2fCUbU74jgsJHWtdDAeDchtHUWvEYsa2IfBo gk9IOaucBsINOsrOWxZATbZ6H/50Wmvp92Xl3AjDKQgB2EB+Mrlhpn2qC2Kv/kwP7Bto YNL6RYJJxZ00VQvuTDY1FdLe0Le0gibQ3GkOsFl8PibwQf/L9jsRihw3v/8KZNHHaZQZ 2JGyBOumDgn4yXfUcZ7pwl1vzpUQyyJ5I0FKou3r2wpHV/EFsDH0zRF626VSotV8nXYw 1uXKT5MXU+HMyaSzD8cEbyuls6g6lRmTak6wMvzZ294u25yKMqqYDySYlmUi2ElN5AIP EdgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736313361; x=1736918161; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qfifYMRclRsFGxPpRF2ADQcpKEAziZDiuZWNz8YnOOw=; b=N7E920LzTkWaxMsTBb9e/yw3OsI/SMiihm4y78aa1W9zQK5FKJIU8JrerFmJEjbYOp jsTfZ6vLsp2/STXEUqABiEfjdn++MQaZpWmX5Ivk71Eyg2wBNaWWREvPUMzSfvEYbfak fc+QgQcR3ktLjSz46EiGxFjA6dvbLCpSYBpSSmniergrAP8zBH88CXoXPXDNphKz3drp v1m/St2jdn0+eOvdqsli/FoLpb5O1Bq5Er46k38BE3Pz6pCldjovOBQyShDhxTtcRs7h QrY4xFGeo+9QC37kWCWKcwyfK+WxrmJxa6tW0tM49p71bQahEzde9PQbqtVQMqPqR/jT XgPw== X-Gm-Message-State: AOJu0YxxlIImQzHf5dBGzYMdZgrs7I10dIZiP2JIDpXTYH+ma6zA8GxX l07eGghEKY8hZBhVyLA1ZDpckCTN62Usq3IgitR4RaoQxFIamgbuYvMhTI7wwPY= X-Gm-Gg: ASbGnctJCVM8s9P/cgzFbjD9qhXd5iTG+/RiMihujcQmvjGWxxOUdT7eaP2CNMzIzqS Z3Nh50VRRVlaUz6SznBCr+mWJ/CE34QdV0j1GoNXLGatL0VXOYNK9m+lN59qV2xhi2P3N0kjt16 HooebtSeujgUTgJ5gvacM/poOo2g0FvNEFyvHkToHxrqepVI4zMAuSzsCjaaK2Am4HeyWke3iq5 GD8Q3m+sIJoTwxMxzyNFu9sAoIiPjN9Q96wNFkZLj0VlSvq1ZPtoax9Zoc3pZMT6r80e5TCPPpR cOek3q/8tjwL5QPWKc91Fw== X-Google-Smtp-Source: AGHT+IGim0cVg7tppE4xYwVX0YG2iBS72evBi8s+3d4LRKd9sVX85ngwkfyWxyZaA+vMSrEHxYYbXg== X-Received: by 2002:a05:6a00:2e97:b0:729:49a:2da6 with SMTP id d2e1a72fcca58-72d21f1ac4emr2369354b3a.3.1736313360744; Tue, 07 Jan 2025 21:16:00 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-8c016b92c6asm22729267a12.56.2025.01.07.21.15.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Jan 2025 21:16:00 -0800 (PST) From: "Doug Smythies" To: "'Peter Zijlstra'" Cc: , , "Doug Smythies" References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> <20250107112606.GN20870@noisy.programming.kicks-ass.net> <20250107192340.GB36003@noisy.programming.kicks-ass.net> In-Reply-To: <20250107192340.GB36003@noisy.programming.kicks-ass.net> Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Tue, 7 Jan 2025 21:15:59 -0800 Message-ID: <001501db618c$67bd8170$37388450$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AQJdvfVp9nqa9troPEySggPg+fzySgGTHC4vAVKdTGwBPBBbIAH1WJEKAgKs/R8BZUmAdQHtSwJ2saw4cFA= On 2025.07.11:24 Peter Zijlstra wrote: > On Tue, Jan 07, 2025 at 12:26:07PM +0100, Peter Zijlstra wrote: >> So I've taken the second node offline, running with 10 cores (20 >> threads) now. >> >> usec Time_Of_Day_Seconds CPU Busy% IRQ >> 106783 1736248404.951438 - 100.00 20119 >> 46 1736248404.844701 0 100.00 1005 >> 41 1736248404.844742 20 100.00 1007 >> 42 1736248404.844784 1 100.00 1005 >> 40 1736248404.844824 21 100.00 1006 >> 41 1736248404.844865 2 100.00 1005 >> 40 1736248404.844905 22 100.00 1006 >> 41 1736248404.844946 3 100.00 1006 >> 40 1736248404.844986 23 100.00 1005 >> 41 1736248404.845027 4 100.00 1005 >> 40 1736248404.845067 24 100.00 1006 >> 41 1736248404.845108 5 100.00 1011 >> 40 1736248404.845149 25 100.00 1005 >> 41 1736248404.845190 6 100.00 1005 >> 40 1736248404.845230 26 100.00 1005 >> 42 1736248404.845272 7 100.00 1007 >> 41 1736248404.845313 27 100.00 1005 >> 41 1736248404.845355 8 100.00 1005 >> 42 1736248404.845397 28 100.00 1006 >> 46 1736248404.845443 9 100.00 1009 >> 105995 1736248404.951438 29 100.00 1005 >> >> Is by far the worst I've had in the past few minutes playing with this. > > Much also depends on how much (or if at all) cgroups are used. > > What exact cgroup config are you having? /sys/kernel/debug/sched/debug > should be able to tell you. I do not know. I'll capture the above output, compress it, and send it to you. I did also boot with systemd.unified_cgroup_hierarchy=0 and it made no difference.