From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16EB21C3314 for ; Sun, 12 Jan 2025 19:59:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736711976; cv=none; b=mpswvzYumeM2+2RoYcvEhLKFpBMz+mZWkZxWRD6rB8X5Dqs1NOqBLhGAzCl6tFngw174BTnplIhqPnY0tpRQLaBX8LhCH2xa1McuR1/kfmVsPoUWZHPjv6MSmk9cksrG8Ps01bGgAlipKwZsWACYd31BvjNdGwnuX+zYs5SLlPk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736711976; c=relaxed/simple; bh=PMbYaTmi/eQZYCr0PQuLnTIc12FI1KBaavo3gpZzbfk=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=XErpGihXBVPMYTqOYeJj4M2d96Mjp59cbrk1pt2Idn1g+HtL3jLDJ2qNdLcnqjq90MQq2bg7+ZsOifnoGwXKpw5Eb8id1z1dxMRwPe335tRu06h2upv8pxf+qTnFHf4DgKvgDrwoNk2dI2tErgOmeCH4bk2WCHguaQZoEhGDTfs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=A2rAlp5/; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="A2rAlp5/" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2156e078563so52265125ad.2 for ; Sun, 12 Jan 2025 11:59:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1736711974; x=1737316774; darn=vger.kernel.org; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=dcz0VYF+h0rk2j7fdeRTiXxOeYpMgqZswNnLQH4Ybg0=; b=A2rAlp5/lG5R9KgFuyr89FQRnKV63lCm6YRad5FMf20DPTKvH/78mXPkpoipVDNqti eorYk5lj1jrj3pW5DhRO+WyJ3ZThuDZ6rnNsvyjLr5t/KQlZBkAF4bi2RoP/bspilMAy oTfJU9L0kCZ2YRgsDOcLAD8iyMWIbkC8gKUoyAVxjjRv2IuF+RzCtAkIBlOVlqFr49a+ eU/j9Fk12s70BZkf0/8u7rFaNEsDNq2LOMzn7udwK95BYaAdNChZEtSu1myo7Agkrm9P 7h1MTUrpW0avg4ygpwdDe0RhXe1wta2AmGVWSA/odG2IgZoesA5O7v6qkIok4oePXE10 DAaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736711974; x=1737316774; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dcz0VYF+h0rk2j7fdeRTiXxOeYpMgqZswNnLQH4Ybg0=; b=LWict14WgCqZsu4jgKC3K9YpyCRBJXYiTmczkLawA07z1vxzPdcxX95yuTG+QlfMP3 OI2/ocxIezFVgKbBw3cwawKv5ljZEPT8Av5JqYT6oNiptos/xFtFuMniSKChhLKA9tVV KZDn8N4IIPAnULlUjDAldT/+4Pazhl3WtBOSVs7OTnrUWIrrDtbsU1UYg6TfGRMJMisN g6Cu/s+bOV+Lk3gQ1JcCeGKtaU8tQcl46V1d+IZ8k+8bYjlstccOeltqQ4auHSc6NSwy gmNnBmlzcbGDw5UtSnAxzX3YUTG8YsRdY9DJPXnNe2ltIwkBRaMHQYZN3D/GBgGnpfns IAvw== X-Gm-Message-State: AOJu0Yxg+sc75D5fFd3XSvQr+LEUYbs9B2HuvcBahUVV8tgy3VLIpUgw 3nmk9moYdnWfl6CVORCNHqXmM/gL15Wdl1iSZJJP3NApboqLRRhbPRXH8FpJ+EM= X-Gm-Gg: ASbGncvjw+i0kIVhh/3b8mHGYZ/OT47CqquJ0wa+wkR7G7o6uZ2X/dqoBhNAGRS8Yks a7Uwi8jeaTLO8L9EBB5ZivoPSTphtQ2TZtXOIwFVFSnEVWYB0zr3mDS0KCkJpB2rFcuCkM7rsjY 95DDy3IshGnOPsDbHehli+Xw9KQcAGfMAiqugC5SmjU8U8opxw1AVojZEJ52qjdJWwzbs25xFOY 0FOxaJQqf8xDqHbSSq08DpP+lyrcf5SOOZcx+W7c0fDKrwqpnSWvyvQM7VdlkhipIU+79GLNVka AxnOafKaPr/H7LfdSIE8qQ== X-Google-Smtp-Source: AGHT+IHKfnvoLDdc53Ytfmbv5cI5u7gIf9aMi2fUOcXsrC6zWxMmY7isnnAX/nmLvMDnEDEx/gO5qA== X-Received: by 2002:a17:902:f693:b0:215:72a7:f39f with SMTP id d9443c01a7336-21a83fb156fmr232912365ad.36.1736711974326; Sun, 12 Jan 2025 11:59:34 -0800 (PST) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-a3184b68b48sm4896231a12.27.2025.01.12.11.59.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Jan 2025 11:59:33 -0800 (PST) From: "Doug Smythies" To: "'Peter Zijlstra'" Cc: , , "'Ingo Molnar'" , , "Doug Smythies" References: <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> <20250107112606.GN20870@noisy.programming.kicks-ass.net> <20250107192340.GB36003@noisy.programming.kicks-ass.net> <001501db618c$67bd8170$37388450$@telus.net> <20250108131205.GO20870@noisy.programming.kicks-ass.net> <001b01db61e4$c3ce5a40$4b6b0ec0$@telus.net> <20250109105959.GA2981@noisy.programming.kicks-ass.net> <002f01db631d$d265a600$7730f200$@telus.net> In-Reply-To: <002f01db631d$d265a600$7730f200$@telus.net> Subject: RE: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Date: Sun, 12 Jan 2025 11:59:35 -0800 Message-ID: <007d01db652c$819f78c0$84de6a40$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQGTHC4vE151303vdkbGCmbSm18RNwFSnUxsATwQWyAB9ViRCgICrP0fAWVJgHUB7UsCdgMlscI8AZ46bX0BejSR6ABTGYDIAmsUCqazDWxagA== Content-Language: en-ca Hi Peter, While we have moved on from this branch of this email thread, just for completeness, I'll give the additional data from the overnight test. There is also an observation that will be made and continued in the next email. On 2025.01.09 21:09 Doug Smythies wrote: > On 2025.01.09 03:00 Peter Zijlstra wrote: > > ... > >> This made me have a very hard look at reweight_entity(), and >> specifically the ->on_rq case, which is more prominent with >> DELAY_DEQUEUE. >> >> And indeed, it is all sorts of broken. While the computation of the new >> lag is correct, the computation for the new vruntime, using the new lag >> is broken for it does not consider the logic set out in place_entity(). >> >> With the below patch, I now see things like: >> >> migration/12-55 [012] d..3. 309.006650: reweight_entity: (ffff8881e0e6f600-ffff88885f235f40-12) >> { weight: 977582 avg_vruntime: 4860513347366 vruntime: 4860513347908 (-542) deadline: 4860516552475 > } -> >> { weight: 2 avg_vruntime: 4860528915984 vruntime: 4860793840706 (-264924722) deadline: 6427157349203 > } >> migration/14-62 [014] d..3. 309.006698: reweight_entity: (ffff8881e0e6cc00-ffff88885f3b5f40-15) >> { weight: 2 avg_vruntime: 4874472992283 vruntime: 4939833828823 (-65360836540) deadline: > 6316614641111 } -> >> { weight: 967149 avg_vruntime: 4874217684324 vruntime: 4874217688559 (-4235) deadline: 4874220535650 > } >> >> Which isn't perfect yet, but much closer. > > Agreed. > I tested the patch. Attached is a repeat of a graph I had sent before, with different y axis scale and old data deleted. > It still compares to the "b12" kernel (the last good one in the kernel bisection). > It was a 2 hour and 31 minute duration test, and the maximum CPU migration time was 24 milliseconds, > verses 6 seconds without the patch. > > I left things running for many hours and will let it continue overnight. > There seems to have been an issue at one spot in time: > > usec Time_Of_Day_Seconds CPU Busy% IRQ > 488994 1736476550.732222 - 99.76 12889 > 488520 1736476550.732222 11 99.76 1012 > 960999 1736476552.694222 - 99.76 17922 > 960587 1736476552.694222 11 99.76 1493 > 914999 1736476554.610222 - 99.76 23579 > 914597 1736476554.610222 11 99.76 1962 > 809999 1736476556.421222 - 99.76 23134 > 809598 1736476556.421222 11 99.76 1917 > 770998 1736476558.193221 - 99.76 21757 > 770603 1736476558.193221 11 99.76 1811 > 726999 1736476559.921222 - 99.76 21294 > 726600 1736476559.921222 11 99.76 1772 > 686998 1736476561.609221 - 99.76 20801 > 686600 1736476561.609221 11 99.76 1731 > 650998 1736476563.261221 - 99.76 20280 > 650601 1736476563.261221 11 99.76 1688 > 610998 1736476564.873221 - 99.76 19857 > 610606 1736476564.873221 11 99.76 1653 The test was continued overnight yielding this additional information: 868008 1736496040.956236 - 99.76 12668 867542 1736496040.956222 5 99.76 1046 5950010 1736496047.907233 - 99.76 22459 5949592 1736496047.907222 5 99.76 1871 5791008 1736496054.699232 - 99.76 83625 5790605 1736496054.699222 5 99.76 6957 1962999 1736502192.036227 - 99.76 12896 1962528 1736502192.036227 11 99.76 1030 434858 1736502193.472086 - 99.76 35824 434387 1736502193.472086 11 99.76 2965 Or 2 more continuous bursts, and a 5.9 second sample. Observation: There isn't any 10's of milliseconds data. Based on the graph, which is basically the same test done in ever so slightly a different way, there should be a lot of such data. Rather than re-attach the same graph, I'll present the Same data as a histogram: First the b12 kernel (the last good one in the kernel bisection): Time Occurrences 1.000000, 3282 1.001000, 1826 1.002000, 227 1.003000, 1852 1.004000, 1036 1.005000, 731 1.006000, 75 1.007000, 30 1.008000, 9 1.009000, 2 1.010000, 1 1.011000, 1 Total: 9072 : Total >= 10 mSec: 2 ( 0.02 percent) Second Kernel 6.13-rc6+this one patch Time Occurrences 1.000000, 1274 1.001000, 1474 1.002000, 512 1.003000, 3201 1.004000, 849 1.005000, 593 1.006000, 246 1.007000, 104 1.008000, 36 1.009000, 15 1.010000, 19 1.011000, 16 1.012000, 11 1.013000, 27 1.014000, 26 1.015000, 35 1.016000, 105 1.017000, 85 1.018000, 135 1.019000, 283 1.020000, 17 1.021000, 4 1.022000, 3 1.023000, 1 Total: 9071 : Total >= 10 mSec: 767 ( 8.46 percent) Where, and for example, this line: 1.005000, 593 Means that there were 593 occurrences of turbostat interval times between 1.005 and 1.005999 seconds. So, I would expect to see that reflected in the overnight test, but don't. It would appear that the slightly different way of doing the test effects the probability of occurrence significantly. I'll continue in a reply to your patch 2 email from Friday (Jan 10th). > > I had one of these the other day also, but they were all 6 seconds. > Its like a burst of problematic data. I have the data somewhere, > and can try to find it tomorrow. >> >> Fixes: eab03c23c2a1 ("sched/eevdf: Fix vruntime adjustment on reweight") >> Signed-off-by: Peter Zijlstra (Intel) > > ...