From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E267F19067A for ; Wed, 15 Jan 2025 16:47:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736959632; cv=none; b=kubogWbS3794imaGQhOWeNHFtDEKiSnA8IpOQvckAsClIBhuqcYBSXEJOw+DaEqMmMW8TUA0iVaR5lu2xk+EgV9VZI7sVAovh1ReITtE4DzVPl0OnJ4QixBmoPi/BqJHRmTZUX4nha3AoFBvnSvLTiIDEnmXd+b1eg8Dp9YewpI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736959632; c=relaxed/simple; bh=JYZZ4lvG20N1nwkkU3kvE3SuvdXZiJWptEJ2yWCmmE4=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=kDC+LudzT9ViOMwFb8Pz18VXcPuZUzHORZ+cTqMXg5g+6fdvRCi9Q532YhEtW8btNqgzQOtZz/Vs3Xj7EA+lwrlYiElGhXQPNeuCycDOKuYiuu/BKWgQnJo/v0/KXpDUhVTcsEo0bkj1xYdDYpLfXH7p/y0y5lXwpzMazvX2KGc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=eoVWmXV6; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="eoVWmXV6" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21649a7bcdcso121492565ad.1 for ; Wed, 15 Jan 2025 08:47:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736959629; x=1737564429; darn=vger.kernel.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=GqaackKd3T3BDf0rqCuEaFapLeaySADmTZr7KUiCBW0=; b=eoVWmXV6KRiKJ40UTmDQpZzcOBb3hN13VuuI8/L8OC7dMPoSlZNh5wOTj5i93Hk3cy VSoRnuglbFfA2kVpXyHrhGATP9TstMC1TVckBVs9KWudh2zaMKosneIinUq9slSaReHE YsPUnK+438zwGhZslRQlCSxRqmoKfxc6OSGVk93/lza7fgP85ipAtNaEKZSL2++gDl3h yscdqDkMqOtR03S7z7DCl7K0dtb6rKliEwtxAG79F0usCoCdewFv2gy64iQEezfZfTXn leSrZhUBe6ukpAt3blnKkvqwK1UJwWMy97EQu6S/epckpz8D/KhSt1DWyAJa9UgLZm7C guRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736959629; x=1737564429; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GqaackKd3T3BDf0rqCuEaFapLeaySADmTZr7KUiCBW0=; b=J3jNOjmq4hCS+fHp8H313WA6Aggg7Vgj+FL5B6b9zW6Gvg7R0wkejBshU+1x5QlT7i UEJQRlJNJE3nOwEUZvFK/GBjtk3QskD33LS6t1qlm+DK9SsUsKChRYKBPPvFLwXDOc5W psJIoxb/TwWByzx4F3pvHC2Q2s9ai2qnIqzh0/HEZwQxISIGPHarpuHUu6YodPK4Wv6s ZO5Jodh0lU9XJLjDpHpvUTV0ADX1QTD05tWvKFISUgq0CATj0apyDpgV4L5YgDUmJVgi tw231v0dw0O9UY8IRafWo+buBNgSEnQuLTsKiVGVeU29DBsUQkfbSko3dw4/pAtT+0NB hgbQ== X-Forwarded-Encrypted: i=1; AJvYcCWfbJMCaQ99qFTYsgYvv2XOGONmNursCAKiafZ4v5vaCj6h28UNwM7sZzz1EFgEpSnsdByK3BKqMVCadJE=@vger.kernel.org X-Gm-Message-State: AOJu0Yxxu0udAp055ce92/fA0IWysLgZ1y9W7dWdjjxh/DWLxj8I0LXg elirxfxfkF507lzHHw1XKPdTR6WXbx99vuZgTmCNKp6VFnMDbpsiE3f2dWa9P7Q= X-Gm-Gg: ASbGncuePEisNNvHTkIj3oWcp90drOV94mfjSHd/pqxFxVDN2L03smxzwVqypkOCJtr Tjxy5kSO71mKrjHkvEHMxYGz3Ip5Mrs7Yd4gzXYKwATKOS0oy1/BsYQMN45F5qMZxCBVgUSW/7K 4BslQPvC0bsJsluWjz37CRD/7x95DNrlFSpFT5e6qc4RDkZE+CDUJn2t+wTqiu8HksI2I2Tp+nn jjiXegyF5fHvb7cw0xCWY7iipR36Frs4LT8SYBU2bJ26zy6wLKgrq2yDWf3TIzfcSEMXoN0+h+3 x+Esa0OvEaDch/kQ4kKL+g== X-Google-Smtp-Source: AGHT+IGzq+Se1a1P97rq+nHatNcsZtMIKoOoXMjgozGqU/HZGGKleug+9E9xSgNrD24RI7Ji9Uea5A== X-Received: by 2002:a17:902:f70f:b0:212:4c82:e3d4 with SMTP id d9443c01a7336-21a83fd90e6mr454163985ad.46.1736959629126; Wed, 15 Jan 2025 08:47:09 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f12ff3fsm84005725ad.79.2025.01.15.08.47.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 15 Jan 2025 08:47:08 -0800 (PST) From: "Doug Smythies" To: "'Len Brown'" Cc: "'Peter Zijlstra'" , , , "'Ingo Molnar'" , , "Doug Smythies" References: <20250107112606.GN20870@noisy.programming.kicks-ass.net> <20250107192340.GB36003@noisy.programming.kicks-ass.net> <001501db618c$67bd8170$37388450$@telus.net> <20250108131205.GO20870@noisy.programming.kicks-ass.net> <001b01db61e4$c3ce5a40$4b6b0ec0$@telus.net> <20250109105959.GA2981@noisy.programming.kicks-ass.net> <002f01db631d$d265a600$7730f200$@telus.net> <20250110115720.GA17405@noisy.programming.kicks-ass.net> <00c201db6547$b43b9a50$1cb2cef0$@telus.net> <20250113110312.GD5388@noisy.programming.kicks-ass.net> <20250114105847.GC8385@noisy.programming.kicks-ass.net> <001501db6697$32633960$9729ac20$@telus.net> In-Reply-To: Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Wed, 15 Jan 2025 08:47:09 -0800 Message-ID: <001701db676d$1ef85910$5ce90b30$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AQFlSYB1PWVBqj20pPXP7hZ0SBLYDgHtSwJ2AyWxwjwBnjptfQF6NJHoAFMZgMgCaxQKpgFP4l/5ASQn7WsCDo0DYQJ+LAk1AbobfE0C9E/kLLNPeB9w Hi Len, Thank you for chiming in on this thread. On 2025.01.14 18:09 Len Brown wrote: > Doug, > Your attention to detail and persistence has once again found a tricky > underlying bug -- kudos! > > Re: turbostat behaviour > > Yes, TSC_MHz -- "the measured rate of the TSC during an interval", is > printed as a sanity check. If there are any irregularities in it, as > you noticed, then something very strange in the hardware or software > is going wrong (and the actual turbostat results will likely not be > reliable). While I use turbostat almost every day, I am embarrassed to admit that until this investigation I did not know about the ability to add the "Time_Of_Day_Seconds" and "usec" columns. They have been incredibly useful. Early on, and until I discovered those two "show" options, I was using the sanity check calculation of TSC_MHz to reveal the anomaly. > Yes, the "usec" column measures how long it takes to migrate to a CPU > and collect stats there. So if you are hunting down a glitch in > migration all you need is this column to see it. "usec" on the > summary row is the difference between the 1st migration and after the > last -- excluding the sysfs/procfs time that is consumed on the last > CPU. So migration delays will also be reflected there. On a per CPU basis, it excludes the actual CPU migration step. Peter and I made a modification to turbostat to have the per CPU 'usec" column focus just on the CPU migration time. [1] > Note: we have a patch queued which changes the "usec" on the Summary > row to *include* the sysfs/procfs time on the last CPU. I did not realise that is was just for the last "sysfs/procfs" time. I'll take a closer look, and wonder if that can explain why I have been unable to catch the lingering >= 10 mSec stuff. > (The per-cpu > "usec" values are unchanged.) This is because we've noticed some > really weird delays in doing things like reading /proc/interrupts and > we want to be able to easily do A/B comparisons by simply including or > excluding counters. Yes, I saw the patch email on the linux-pm email list and have included it in my local turbostat for about a week now. > Also FYI, The scheme of migrating to each CPU so that collecting stats > there will be "local" isn't scaling so well on very large systems, and > I'm about to take a close look at it. In yogini we used a different > scheme, where a thread is bound to each CPU, so they can collect in > parallel; and we may be moving to something like that. > > cheers, > Len Brown, Intel Open Source Technology Center [1] https://lore.kernel.org/lkml/001b01db608a$56d3dc40$047b94c0$@telus.net/ ... Doug