From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9314324C676 for ; Fri, 20 Feb 2026 03:25:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771557933; cv=none; b=mFGaZ4P+l6aZ6ggfLFGGudRnxxBD+FtDzJCp/XxcOeoa+9IZ54+eEc5EBaPo7SvFpw2Xukg2gqeZeF5yX+/dEUfUwCLDda0iXs4HBhmCwEAybQhQcdyh8KAD/LEqdb5Vp1ZMNvh5T/zTggNVcC8vf21/L8yFsVr6wydtkBgiZtA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771557933; c=relaxed/simple; bh=A5GDCf59Bg5/P20O2+Y9ALNMa32yT+NkyCgWN/rbxxY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jwtYdZbjDg7h7fHjFxCcfM9Ban6N7edRFpx2DxioD/yKRwrReVKlTtVnOaBNgr4FmcyBZRhvM+CB0PlFWt1VHLYzM1Da4l004VOaoHe+TuRfGiOyM7BDu/unJoFnITqoYmn4ijxXJ67mvnK4UjfzQMJd6xqpv6IgJ3PNCGd0bMI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io; spf=pass smtp.mailfrom=layalina.io; dkim=pass (2048-bit key) header.d=layalina-io.20230601.gappssmtp.com header.i=@layalina-io.20230601.gappssmtp.com header.b=uXdIt837; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=layalina.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=layalina-io.20230601.gappssmtp.com header.i=@layalina-io.20230601.gappssmtp.com header.b="uXdIt837" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-4838c15e3cbso12879655e9.3 for ; Thu, 19 Feb 2026 19:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=layalina-io.20230601.gappssmtp.com; s=20230601; t=1771557930; x=1772162730; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Y3MHZYGYqoBmGcgXOq3kmrPteR6ZV9d4NZhyrNYeGjc=; b=uXdIt837uWuxYNsrQSlyN9wBTpa82SsUm29d7YOELhsnpk7w+zsRbbLgRDIosl2BGx cz+6iD2nWBEsoSB0Rxzv8B1vWXbRJNITLNjGE31qM57nEIqTeZjqFUKUUGQFzeAY3kYM os4SVUxq+MLoyn9EhHOtPZgdjPa7nS/y5GUiNYDt6r1cLy4r139MkTRKEd4C8xFSRioW BzOSFna5pqCABUbl6g3co6F6EzDol9sqvUOt5ze0jG1UNuuj/qGZ+Do+WzMtgYVHkM3P Q4qD/KK+O3ZjRsc7IJSTb6mI+sC7aIA5Bb3Kva4dILUl/w6X0lceAo2GS8QdAke4+JmP rajA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771557930; x=1772162730; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y3MHZYGYqoBmGcgXOq3kmrPteR6ZV9d4NZhyrNYeGjc=; b=iW+lT0JA0p5aLkh6Kq3HPqhA2C0yHH+qlYTbfT5o1hjyI9w7kHB/E0mNr8bJ+EovKA aI2+/YkMXMld8eJdNDFShXHV/srqkQU+00rI6CLTVpY35sh2BiNIhoNw18RqCzSI9VzA Oagbp+SYlEQ7uhDle4uvcvVPSmv6ssnOC7ZY+zHNFA7EEZncCPZgUUraTyJufD8lMW/D hAq+JEtpnCSw+de3T7Q782mtq0fy4CJtx8CH4sqdXWvRGlKAdcwJi115GV4oouOyTbVJ MychNkfA3POxCJ6rWf2R9u+Dp6QXSnaCeDBMd1dFfIdFq8d1Od0wrTxoIj8xIwyqy4YV 8fdQ== X-Forwarded-Encrypted: i=1; AJvYcCVHLstTuIUh0AccSKHAspqHNeoYOaXq7AMja2NCIjEggKjsxNSdIMnIfyyWenhxQA8NuA8wAUwoy4YJvAE=@vger.kernel.org X-Gm-Message-State: AOJu0YwkGa/BabQONw0j0taCPjEDQWm9ivdqonoUDAMO0z4GYw4xLRZ5 DFi/ctfEyJij4J70zXX0kicq7pWKYmFXbE0BdaVeRXTPn7HN3EFd8c7cD+m6Ol2cwiI= X-Gm-Gg: AZuq6aLkjGT3dtsVYwHlZ9L9YwA36dVaXbuNSruMB271q3ANSGyTK84Sduu+9h32K1+ q9FBUBnPGb42HNaBMInIAopaj3p8x/56LB36oNbj+y184GNVZIMAM23JqoxgvIfVTE4Xm7irEwk t2IjgH12do5S6hISmehBh+wSxeYdG034DKBz8hFI6avkqkZSB1psLDQ/04/g1dVA/noGtdP9rzO u6k2QukdS+5UbtafCpmSA4XApiJ5vb94h2CKcjzW0Vs4lxMQG+21fRUlLiEyD30xUV2xb/jNUf6 UtUzrThlCtWh5iXqY2Uu0N4LjKL+bJcdqkK3mZFFqPlxkKFboy3s51ypsAcphKn1GAgVYWvfsXY ti46MW9ArLu3aRGqDNkEYMyPikkHal8CaORji6rLLTVygMWfMdXZDeGDcUSgo83dpJhjwdgYAts lWVpg9qQv6tQenZPHe9Q2UMemPuwDsQ+s1wjgkxKhy3gYYWwghV/ZNsi31rfqxmabi/eY0 X-Received: by 2002:a05:600c:811b:b0:477:6d96:b3c8 with SMTP id 5b1f17b1804b1-483a5534ac0mr4042655e9.23.1771557929396; Thu, 19 Feb 2026 19:25:29 -0800 (PST) Received: from airbuntu (host86-169-41-76.range86-169.btcentralplus.com. [86.169.41.76]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-483a3d99c5asm13059015e9.0.2026.02.19.19.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Feb 2026 19:25:28 -0800 (PST) Date: Fri, 20 Feb 2026 03:25:27 +0000 From: Qais Yousef To: "Chen, Yu C" Cc: Peter Zijlstra , Tim Chen , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Libo Chen , linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 00/21] Cache Aware Scheduling Message-ID: <20260220032527.iing4vb2gylk5o5z@airbuntu> References: <20260219140828.a7pyzupun7lsdw34@airbuntu> <20260219144108.GI1282955@noisy.programming.kicks-ass.net> <66a23acb-973e-4987-a006-18c50e18d3aa@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <66a23acb-973e-4987-a006-18c50e18d3aa@intel.com> On 02/19/26 23:07, Chen, Yu C wrote: > Hi Peter, Qais, > > On 2/19/2026 10:41 PM, Peter Zijlstra wrote: > > On Thu, Feb 19, 2026 at 02:08:28PM +0000, Qais Yousef wrote: > > > On 02/10/26 14:18, Tim Chen wrote: > > [ ... ] > > > > > > > I admit yet to look fully at the series. But I must ask, why are you deferring > > > to load balance and not looking at wake up path? LB should be for corrections. > > > When wake up path is doing wrong decision all the time, LB (which is super slow > > > to react) is too late to start grouping tasks? What am I missing? > > > > There used to be wakeup steering, but I'm not sure that still exists in > > this version (still need to read beyond the first few patches). It isn't > > hard to add. > > > > Please let me explain a little more about why we did this in the > load balance path. Yes, the original version implemented cache-aware > scheduling only in the wakeup path. According to our testing, this appeared > to cause some task bouncing issues across LLCs. This was due to conflicts > with the legacy load balancer, which tries to spread tasks to different > LLCs. > So as Peter said, the load balancer should be taken care of anyway. Later, > we kept only the cache aware logic in the load balancer, and the test > results Yes, we need both. My concern is that the origin is for wake up path to keep tasks placed correctly as most task wake up and sleep often and this is the common case. If the decision tree is not unified, we will have problems. And this is not a specific problem to doing placement based on memory dependency. We need to extend the wake up path to do placement based on latency. Placement based on energy (EAS) has the same problem too. It disabled LB altogether, which is a problem we are trying to fix if you saw the other discussion about overutilized handling. Load balancer can destroy energy balance easily and it has no notion of how to distribute based on energy. This is a recurring theme for any new task placement decision that is not purely based on load. The LB will wreck havoc. > became much more stable, so we kept it as is. The wakeup path more or less > aggregates the wakees(threads within the same process) within the LLC in the > wakeup fast path, so we have not changed it for now. How expensive is it to use the new push lb, which unifies the decision with wake up path, to detect these bad task placement and steer them back to the right LLC? I think if we can construct the trigger right, we can simplify the load balance to keep tagged tasks within the same LLC much easier. In my view this bad task placement is just a new type of misfit where a task has strayed from its group for whatever reason at wake up and it is not sleeping and waking up again to be placed back with its clan - assuming the conditions has changed to warrant the move - which the wake up path should handle anyway. FWIW, I have been experimenting to use push lb to keep regular LB off and rely solely on it to manage the important corner cases (including overloaded one) - and seeing *very* promising results. But the systems I work with are small compared to yours. But essentially if we can construct the system to keep the wakeup path (via regular sleep/wakeup cycle and push lb) maintain the system relatively balanced and delay regular LB for when we need to do large intervention, we can simplify the problem space significantly IMHO. If the LB had to kick in, then the delays of not finding enough bandwidth to run are larger than the delays of not sharing the hottest LLC. IOW, keep the regular LB as-is for true load balance and handle the small exceptions via natural sleep/wakeup cycle or push lb. > > Let me copy the changelog from the previous patch version: > > " > In previous versions, aggregation of tasks were done in the > wake up path, without making load balancing paths aware of > LLC (Last-Level-Cache) preference. This led to the following > problems: > > 1) Aggregation of tasks during wake up led to load imbalance > between LLCs > 2) Load balancing tried to even out the load between LLCs > 3) Wake up tasks aggregation happened at a faster rate and > load balancing moved tasks in opposite directions, leading > to continuous and excessive task migrations and regressions > in benchmarks like schbench. Note this is an artefact of tagging all tasks belonging to the process as co-dependent. So somehow this is a case of shooting one self in the foot because processes with large number of tasks will create large imbalances and will start to require special handling. I guess the question, were they really that packed which means the steering logic needed to relax a little bit and say hey, this is an overcommit I must spill to the other LLCs, or was it really okay to pack them all in one LLC and LB was overzealous to kick in and needed to be aware the new case is not really a problem that requires its intervention? > > In this version, load balancing is made cache-aware. The main > idea of cache-aware load balancing consists of two parts: I think this might work under the conditions you care about. But will be hard to generalize. But I might need to go and read more. Note I am mainly concerned because the wake up path can't stay based purely on load forever and need to be able to do smarter decisions (latency being the most important one in the horizon). And they all will hit this problem. I think we need to find a good recipe for how to handle these problems in general. I don't think we can extend the LB to be energy aware, latency aware, cache aware etc without hitting a lot of hurdles. And it is too slow to react. > > 1) Identify tasks that prefer to run on their hottest LLC and > move them there. > 2) Prevent generic load balancing from moving a task out of > its hottest LLC. Isn't this 2nd part the fix to the wake up problem you faced? 1 should naturally be happening at wake up. And for random long running strayed tasks, I believe push lb is an easier way to manage them.