From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2F201E5207 for ; Tue, 24 Feb 2026 02:49:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771901373; cv=none; b=hTBvt2PquCPjQ2x2PSIOXyGJPoeZClIlT/TZqKnoOC7M7u3QSKV5dpUX/umUf6xYX5xQNKCNx7mkCpxzNYjy6dgdR37wRAnGJcSswFJMQPuCyxpwOvEpZgpOiRZZgWBmszNT3GRYncVsv1yUXcXcF8Zd2SOKvSl+6ECnzwgVM2A= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771901373; c=relaxed/simple; bh=R8BGa+OyizF3dAc3eCQRMJy3UsE/a/YClRE+Nd3efvk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=bPHkMlDmscNIcc568QXDZAbxrCKIv5Arq3Vx0muhGa7BONb2ykCw9qZj43GcCRpiX/ldeiXTR9AT4NEdg9gR73gjmVNMQMnQVSoBHePSOxS6RjYRd+A5QXo/D6PWEf1SYCQ42uSPuKDD6LNTv23xSULew+hkpBpqPt0pQRT8TZ8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io; spf=pass smtp.mailfrom=layalina.io; dkim=pass (2048-bit key) header.d=layalina-io.20230601.gappssmtp.com header.i=@layalina-io.20230601.gappssmtp.com header.b=lERew+0r; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=layalina.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=layalina-io.20230601.gappssmtp.com header.i=@layalina-io.20230601.gappssmtp.com header.b="lERew+0r" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-4358fb60802so3629966f8f.1 for ; Mon, 23 Feb 2026 18:49:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=layalina-io.20230601.gappssmtp.com; s=20230601; t=1771901370; x=1772506170; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MZO18IcEpuGcEqBcRFRTpNqZrB4w6yYu1NfbWR0y7DI=; b=lERew+0rSnFWKeZkBxXpQCTGG2pdAIHRCx5JVMIpEHimNPQSuIEG0miEVYTXblGT5i 3TkSaoTblAwf3Ox7Kp5D/m47Aamfu9dLnEN3zlfcwWPAFDjE29pFyynO2HpzBtly/4KG sIDgQwKedNJzCJeFyRG3oqI7PwO/4nw6dk+oyJK7Eby3qZjEE2PveEXsTQGbCPBgk6ba fGPclr4MXNqvWxNfmgSjX4zDDc0TKRgtUFIhRZlF3O3YZa2pFAP2+j6mGWo/Uy+QFP2M ZFiG1koiv8WSxuQ+5uQFvXyj9E3lmif9R0VdP6D3KLqN781J9x8z3i/wTI76h6qw+bYj b3Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771901370; x=1772506170; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MZO18IcEpuGcEqBcRFRTpNqZrB4w6yYu1NfbWR0y7DI=; b=mCpPY4ta0HcHdZ9t4GYIRIGyG499DrXcWXMklrlg4oCvZsFmQ5KME2OWVdLDAekvQj iMRZU9Tu68XKtZe+gMN6l7Xpm4Y1/cX6NEI3l57s999t2T3crdUG4MlrDYW/dzrHOy6k qbcrKbiaQhraGPPex8+X8ndar1CE0TY1pudB2Ybf3hrk3qAVvV1zE+VdtbNiEjy3kYzI Ol5Iu4ObXfl3+YoKzxx80Vs/0wwp+nJMQ7cAsCJkCjZ4DHAqSWgIE2MyMa/s+9V5rhps K0VIVE6Ts1iZ1jaOdfZOOphnGE3l2pqyu/7bDZss06JxzHVUXeFBlCckiVmOCOQ8KHum 7thQ== X-Forwarded-Encrypted: i=1; AJvYcCVGTWB1i/LbrwfMH5bdDNjJxovYYiphNtK3HhSYehAr2h2KoXKhNuBDmb7TPBD4aFr6NlQCDiFmVBG2gAM=@vger.kernel.org X-Gm-Message-State: AOJu0YwUva7TT/oHe3vybtQDEH2zv5oZU1dMywVpjBZwv4zR0BN3jUY6 gVZqa0Q6ql9iMIirZQt5mgyUQU773g+u5hnWj+rC8AO9P+Rgv+kugVga5y86WZpGwV0= X-Gm-Gg: ATEYQzxOdbswgXzZTlHl4AtIZ4gyTdWfcyhO0U8ccIycBwsY7JBk0mG3WhW6n2IJ56U GHakqqSgQXbI16hPWSqbDdmdmpTJZCFNtRZ/RIYXYSNz1T+Ny99fg+Hrb37tp7kl6MTql/fv7iS g2mE1EmrjuNWeKZ58YZxlLbgGe7E6r5lNg5zKxvxHCdapDEXz5KskF5prMnQD3bdAxIeNE5lwUo XWu1VnHYKpiyUKFmHz/W+6fRAj9sTTXJNAeZ7RkOVX0CDnCazA2dNGXfRKJnV2VS1Ewk+vze/+q fMzzWeOx9qZiElzBIfkpgV4aZihS2DRGE0rui/z0r44mAUS2TzMUU7sTJa/072d2o40flQPe5oo tb4rLtEeKHOU/EPOtPuFn0IP6b4U8VKR+pFtOJJVdajITe+uVK15iZ7VhH+IfbBZJgriDFcoTQl lgch+vctbOhLnZTR3741Gk5ihlfmMqinKA9s/a/P2uFaml9BZqvJ8bqBQvyG3DLoNr/aJd X-Received: by 2002:a05:6000:18a3:b0:437:71b2:6f23 with SMTP id ffacd0b85a97d-439625c9b7amr29945439f8f.5.1771901370046; Mon, 23 Feb 2026 18:49:30 -0800 (PST) Received: from airbuntu (host86-169-41-76.range86-169.btcentralplus.com. [86.169.41.76]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43970bfa001sm24837209f8f.2.2026.02.23.18.49.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Feb 2026 18:49:29 -0800 (PST) Date: Tue, 24 Feb 2026 02:49:27 +0000 From: Qais Yousef To: Peter Zijlstra Cc: Tim Chen , "Chen, Yu C" , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Libo Chen , linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 00/21] Cache Aware Scheduling Message-ID: <20260224024927.3nbywxfugfnt5eln@airbuntu> References: <20260219140828.a7pyzupun7lsdw34@airbuntu> <20260219144108.GI1282955@noisy.programming.kicks-ass.net> <66a23acb-973e-4987-a006-18c50e18d3aa@intel.com> <4514b6aef56d0ae144ebd56df9211c6599744633.camel@linux.intel.com> <20260220032941.cw7zdlcy7n6fxmsj@airbuntu> <20260220094303.GF2995752@noisy.programming.kicks-ass.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20260220094303.GF2995752@noisy.programming.kicks-ass.net> On 02/20/26 10:43, Peter Zijlstra wrote: > On Fri, Feb 20, 2026 at 03:29:41AM +0000, Qais Yousef wrote: > > > What's the reason wake up doesn't have the latest info? Is this a limitation of > > these large systems where stats updates are too expensive to do? Is it not > > fixable at all? > > Scalability is indeed the main problem. The periodic load-balancer, by > virtue of being 'slow' has two advantages: > > - the cost of aggregating the numbers is amortized by the relative low > frequency of aggregation > > - it can work with averages; it is less concerned with immediate > spikes. > > This obviously has the exact inverse set of problems in that it is not > able to deal with immediate/short term issues. Yes. And if we are to focus on providing better task placement based on QoS (which what I think this is essentially is), we have a constant problem of two paths producing results that are incompatible. Which is why I am trying to stress the importance of the wake up path. I understand for this initial drop we don't have a way to provide specific hints for tasks, but this is why we end up with this difficult choices always - which I think we don't have to. More on this at the bottom. > > > Anyway, we're already at the point where EAS wakeup path is getting far > too expensive for the current set of hardware. While we started with a > handful of asymmetric CPUs, we're now pushing 32 CPUs or so. Is this 32 perf domains? Expensive for what workloads? Folks can still use performance governor and plug it to a wall if they want ;-) > > (Look at Intel Nova Lake speculation online, that's supposedly going to > get us 2 dies of 8P+16E with another 4 bonus weaklings on the south > bridge or something, for a grand total of 52 asymmetric CPUs of 3 kinds) Not sure if my experience matters for whatever this is supposed to be used for, but the cost of wrong decision is really high on these topologies. It is bloody worthwhile spending more time to select a better CPU and worthwhile to have the push lb do frequent corrections. Not sure if you saw the other thread on one of Vincent's patches - but I am trying to completely disable overutilized (or regular LB) and rely on wakeup + push lb and seeing great success (and gains). But I am carrying a number of improvements that I discussed in various places on the list that makes this effective setup. Hopefully I'll share full findings properly at some point. > > > Then consider: > > - Intel Granite Rapids-SP at 8*86 cores for 688 cores / 1376 threads. > > - AMD Prometheus at 2*192 cores with 384 cores / 768 threads. These > are silly number of CPUs. > > - Power10, it is something like 16 sockets, 16 cores per socket, 8 > threads per core for a mere 2048 threads. > > Now, these are the extreme end of the spectrum systems, 'nobody' will > actually have them, but in a few generations they'll seem small again. > > > So whatever we build now, will have to deal with silly numbers of CPUs. True, but I think we ought to bite the bullet at some point. My line of thought is that we don't have to (and actually shouldn't) make the compromise at the kernel level. We can define the problem such that it is opt-in/opt-out where users who find the benefit can opt-in or find a disadvantage opt-out. Now the difficulty is that we don't have a way to describe such things, and this is what I am trying to solve with Sched QoS library. I am writing this now, but I think I should be able to help with this use case so that users can describe which workload wants to benefit from co-locating and these tasks will take the hit of harder task placement and frequent migration under loaded scenarios - the contract being that being co-located has significant performance impact they are happy to pay the price. Things that didn't subscribe will work as-is. Anyway, my major goal is to find how we can tie all these stories together as we need to add ability to do task placement based on special requirements and the conflict with LB is one major one that I think Vincent's proposal for push lb is quite neat and spot on. I am not sure if you saw our LPC talk about Sched QoS where we expanded on our overall thoughts. In my view, this problem belongs to the same class of problems of placement based on special requirements (latency, energy, cache etc) and hopefully we can address along the way. But if not, it would be good to know more so we can think how we can better incorporate it as part of the bigger story. So far I think if this can be made to go through the wake up path and rely on push lb; it is part of the same story. If not, then we need to think harder how to connect things together for a coherent approach. If I can successfully give you a way to describe the requirement of tasks needs to be co-located so that we don't have to make the assumption in the kernel that tasks belonging to the same process needs to stay in the same LLC, do you think wake up + push lb works? If not, how do you see it evolving? And more importantly, how do you view the role of regular LB in these cases? The way I see it is that it should trigger less for the reasons you mentioned at the top; and when it triggers it means heavy intervention is required and whatever special task placement requirements will need to be dropped at this stage since the push lb clearly failed to keep up and we are at a point where we need to do heavy handed balancing work. I think this activities are more relevant to multi-LLC systems - which has the added problem of defining when some imbalances are okay; which I believe the difficulty being hit here with wakeup path based approach. For single LLC systems I think this heavy handed approach can be made unnecessary if we do it correctly. Sorry a bit of divergent. But I am interested on how we can all move the ship in the same direction. I think this is all part of making the wake up path multi-modal and improve its co-ordination with LB.