From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B734156669 for ; Mon, 6 Jan 2025 15:01:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736175699; cv=none; b=WrkXaffjvni24ASlJ1s9yT17gEEnxapyq7jqx3CYCXct8Xd77cYR8Kie8mn9bVGOobGlbqQn7cr5tP8WZ2Dmw9FR3hYLeJgqutQ6+53TSHU45PXtIl+0fVObVd2OyVwTdEmgt7jA7DF3hlxpDfxGbkjip2wWal2cA7/zSGwo3CE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736175699; c=relaxed/simple; bh=p0XtuDV2Ei8AadlDnFYchC88sgQ/ABU3GRgGJonORmQ=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=W7+kG7qpq51246qYCDC9JtwYgY/RvhzK9iIsSiIrlmcw/YZ0EcRLwwx3DP3dc69UQ6T14DIbe419M/9I4q2+4TULsdwt83qILa7ACF9S5AlS7USV321RH/599N8cqG7hm2IyiE0N4fq/HEr/zmJagzdHdGqXT9OQmvAxxyVVjsk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=ECI4/cu4; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="ECI4/cu4" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-216728b1836so189334835ad.0 for ; Mon, 06 Jan 2025 07:01:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736175694; x=1736780494; darn=vger.kernel.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=/NIua4N6d/f7/bPFOF3VpBRBMHwi90fei1oq7u8gPJY=; b=ECI4/cu4CG+zHHO0HOAl65mpfrmGDrmDFVJtdY/MgPDxv4H6cQLxY/2SH2ZCC7Ih6e x6DgZZxueyiGjJ1TD8v5Dc7q3pTZQr+xc3D5lbmaCUFNZAxmSqC1W25SHfAAnPEvocQe lkDknt98ml67bstWeo3gZivoZ/3etssMhxq2AIxM7oRXfpRH9Mm/rSjP1IMbdC9NNe9/ ohSbeWAHyNQvrsKc56vfX0CWlIT0mOvpDyCBqsZV0/lMPAsEw5n8Tm97L17S7F450k7P j+nGKETDdnQ++t7n0wktvH/vCOa3zjEdMjSzR6up8qhosqoDqzrj3SxCSA2IrcnM/+cz XnrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736175694; x=1736780494; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/NIua4N6d/f7/bPFOF3VpBRBMHwi90fei1oq7u8gPJY=; b=fxUo6uFZGpZXK1mFk85RtPyHJwoPQxkqtt7jjK8svaaC9ObsFexFYW0hfVCZyBs04p x/RKZvOkqum9bLUPHzsYHrtvHePzvtcu4cmniKBR4ZW7HK7gHBgcXU8J8z89AMmVUFPb xTKkawLd5gQKG5SFmR9lgIZg7kcTwTFsO4pwBXmLiVTkPzOjnZAPziD3YtP58dj/evr0 1L960eqzo7na/TWGWwjuJ9qHmhY3kbO0M6SuZhDAeCIAnhyscBEVJuws0r9H8WZIgM7o R8yxyMBjntENDZPW5770b6++LoCDNMi20eBKdOHw8ECDdFmgJtM7400KppuTlApbvTDV WGVg== X-Gm-Message-State: AOJu0Ywgqn+xI6yf4q9LstQjuV554zfCEjgEKYK8VcesjKD/S1ujyJ2e Vl5g2qX7ERP7+dc+1ydHc6pigxIpuahNYzjyGMNSbhaF93R3ZzJNYCduxHYB02KQy2+KDC7phJU D X-Gm-Gg: ASbGncuE14/HQ3olbVmqnGtClH4s2wEq1zm8D6bkB9twL8kd1yUxnIKobzlJDAQ9JV7 NvgCn+gE2UB1am3RpTvTyi7teT9zD2QC7+pVCGDn0w7lb0EUcIa+It5V3gQP0QcL1qkR/9qXxi3 ur36npiQl7B1TcdgqxOZLDtLQWEnM20+VgH5lwoAraNA9Q7OWWerY9UPTBE7ougxlaoAoPV7FM5 HAzsMGl5CZ/TlT6nYugyxgke/OEcKzCZ8CCA235u4sBfCOqePb9K8O0ou45GN58x7ioBZj8Dje2 duhaj3g/76IzCsP3lkzqhw== X-Google-Smtp-Source: AGHT+IHP4Q4di1lI6lLGDdsDtJhQ6Wv1eee8Dws00TxGRyFN3yPKBuHkx+rUsg4AtG/qfwaI0g8u8g== X-Received: by 2002:a17:903:2281:b0:216:386e:dbf with SMTP id d9443c01a7336-219e6ea2660mr864299515ad.20.1736175693656; Mon, 06 Jan 2025 07:01:33 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc96e85csm293458715ad.61.2025.01.06.07.01.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Jan 2025 07:01:33 -0800 (PST) From: "Doug Smythies" To: "'Peter Zijlstra'" Cc: , , "Doug Smythies" References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> In-Reply-To: <20250106115732.GE20870@noisy.programming.kicks-ass.net> Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Mon, 6 Jan 2025 07:01:34 -0800 Message-ID: <000801db604b$e0f6b580$a2e42080$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AQJdvfVp9nqa9troPEySggPg+fzySgGTHC4vsfh/nSA= On 2025.01.06 03:58 Peter Zijlstra wrote: >On Sun, Dec 29, 2024 at 02:51:43PM -0800, Doug Smythies wrote: >> Hi Peter, >> >> I have been having trouble with turbostat reporting processor package power levels that can not possibly be true. >> After eliminating the turbostat program itself as the source of the issue I bisected the kernel. >> An edited summary (actual log attached): >> >> 82e9d0456e06 sched/fair: Avoid re-setting virtual deadline on 'migrations' >> b10 bad fc1892becd56 sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE >> b13 bad 54a58a787791 sched/fair: Implement DELAY_ZERO >> skip 152e11f6df29 sched/fair: Implement delayed dequeue >> skip e1459a50ba31 sched: Teach dequeue_task() about special task states >> skip a1c446611e31 sched,freezer: Mark TASK_FROZEN special >> skip 781773e3b680 sched/fair: Implement ENQUEUE_DELAYED >> skip f12e148892ed sched/fair: Prepare pick_next_task() for delayed dequeue >> skip 2e0199df252a sched/fair: Prepare exit/cleanup paths for delayed_dequeue >> b12 good e28b5f8bda01 sched/fair: Assert {set_next,put_prev}_entity() are properly balanced >> dfa0a574cbc4 sched/uclamg: Handle delayed dequeue >> b11 good abc158c82ae5 sched: Prepare generic code for delayed dequeue >> e8901061ca0c sched: Split DEQUEUE_SLEEP from deactivate_task() >> >> Where "bN" is just my assigned kernel name for each bisection step. >> >> In the linux-kernel email archives I found a thread that isolated these same commits. >> It was from late Novermebr / early December: >> >> https://lore.kernel.org/all/20240727105030.226163742@infradead.org/T/#m9aeb4d897e029cf7546513bb09499c320457c174 >> >> An example of the turbostat manifestation of the issue: >> >> doug@s19:~$ sudo ~/kernel/linux/tools/power/x86/turbostat/turbostat --quiet --Summary --show >> Busy%,Bzy_MHz,IRQ,PkgWatt,PkgTmp,TSC_MHz --interval 1 >> [sudo] password for doug: >> Busy% Bzy_MHz TSC_MHz IRQ PkgTmp PkgWatt >> 99.76 4800 4104 12304 73 80.08 >> 99.76 4800 4104 12047 73 80.23 >> 99.76 4800 879 12157 73 11.40 >> 99.76 4800 26667 84214 72 557.23 >> 99.76 4800 4104 12036 72 79.39 >> >> Where TSC_MHz was reported as 879, there was a big gap in time. >> Like 4.7 seconds instead of 1. >> Where TSC_MHz was reported as 26667, there was not a big gap in time. >> >> It happens for about 5% of the samples + or - a lot. >> It only happens when the workload is almost exactly 100%. >> More load, it doesn't occur. >> Less load, it doesn't occur. Although, I did get this once: >> >> Busy% Bzy_MHz TSC_MHz IRQ PkgTmp PkgWatt >> 91.46 4800 4104 11348 73 103.98 >> 91.46 4800 4104 11353 73 103.89 >> 91.50 4800 3903 11339 73 98.16 >> 91.43 4800 4271 12001 73 108.52 >> 91.45 4800 4148 11481 73 105.13 >> 91.46 4800 4104 11341 73 103.96 >> 91.46 4800 4104 11348 73 103.99 >> >> So, it might just be much less probable and less severe. >> >> It happens over many different types of workload that I have tried. > > In private email you've communicated it happens due to > sched_setaffinity() sometimes taking multiple seconds. > > I'm trying to reproduce by starting a bash 'while ;: do :; done' spinner > for each CPU, but so far am not able to reproduce. I have also been trying to reproduce the issue without using turbostat. No success. The other thing to note is that my test computer is otherwise very very idle with no GUI and few services. > > What is the easiest 100% load you're seeing this with? Lately, and specifically to be able to tell others, I have been using: yes > /dev/null & On my Intel i5-10600K, with 6 cores and 2 threads per core, 12 CPUs, I run 12 of those work loads. ... Doug