From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C028D35DA61 for ; Thu, 23 Apr 2026 15:06:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776956816; cv=none; b=sm7I824ZSbuPWsUdQYg+Cy1/NRP7QxEuelLbfB9hDh7YD2xyHlxEbf6f98bdarPpcKSW/fHF81qzFVZFPrMPSkgDFJm2y0KhSnrqgHXLeeJiYN4tQipgj4lwVcMtCvODoiB4qRqggVsAkrXYKualwJUyFBYWYcRqVvcr+eQkTH4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776956816; c=relaxed/simple; bh=7GX9/eaJcGm48SoTlPGC5tWvyFuqrYy3qe/AzP6/jOs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=stTKs5J2Ip0qggP25S6wq1nlJQ8O6KuUgkYTS4BfqRybFz5x3iVyyvW84aBYkDrIDlSTRtojKFmKADB4p1IZ15NrHu0O3IpxCnoXNpfMLxE9t82OFDwYKifc3h9GSoXoYnEMUtyQeq6ciMys/+vefUYaHY60mDu05N8fDGImtoU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io; spf=pass smtp.mailfrom=layalina.io; dkim=pass (2048-bit key) header.d=layalina-io.20251104.gappssmtp.com header.i=@layalina-io.20251104.gappssmtp.com header.b=kPQOl15F; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=layalina.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=layalina.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=layalina-io.20251104.gappssmtp.com header.i=@layalina-io.20251104.gappssmtp.com header.b="kPQOl15F" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-671c24f23b1so9815776a12.0 for ; Thu, 23 Apr 2026 08:06:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=layalina-io.20251104.gappssmtp.com; s=20251104; t=1776956813; x=1777561613; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wiXO18yywybr3Dg9jrvqYLuG/AjfJudbhHD0MMs0qL4=; b=kPQOl15FL5Ej4CsGkDiOlM691drvFMpfOPlcow/ixoKKBL9q3ZppvDp6vOtgVvWh1O XLEVFQKO1U9y+DwYvJvFcc6ENadgzAe+8TqcDQKdE/LMKE8uFiOYSXewojYSOxwaJhbj CMZ1gL6J2CCSOQbaWJVcKDUG0mpfDQMfoEIT/gntaCSNCOvAyiyivQiokw0Wp8M0PEZY yPQvC/PYPm+Ikgc9dSlKC3Fx4fdEyHvfJGmtdQrJJC/uIKkyArnm0SNC3g1OYl+HbfJS F/qJ5wTjVsvJoaslfkcSKuH1orj6Mtv2SHnLWiIQJ9lMHLaDicfmOIkBPk4oUM1TeSNC yeTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776956813; x=1777561613; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wiXO18yywybr3Dg9jrvqYLuG/AjfJudbhHD0MMs0qL4=; b=BpIURYXpNN2zfm41iqgrFWPp2VxUZkwhXo+1oVJf3/twa2BVnUZyE5N55zR53o7x0l G+C3ywnyLZaqW5s5kQYevavkOQBJRb9qEuqtsSqb+Ys75Qm57KE+ygtgcwbKdWZcxHSf +Gl4Z8PpCmeiFLNl6ZM7DNzPbh5UtWfoIIh+QSDqRqmTK7D8UyjsHIPer1thZZrnfSUV dt+5X/fsx4qWnQet7czdZjzSulfpiowQ/x9+uEIbtsjNsJQ5fn3cJ1HobFd6pt+PWkPB vIf4HtJCiD+2PsLS0fijvrdGIaWbBNfxj6l6B5NeqG9o013Dlpz7Ow/+6ARDb9u8YOFe xj3w== X-Forwarded-Encrypted: i=1; AFNElJ8ACB6wRn6gVWrPFPyYwoFMCmgwsTCF50DR1RDc7rNcbyo3EiZfL5/u3eaaJr6l9+VYJ7qpJeQAmc0VThg=@vger.kernel.org X-Gm-Message-State: AOJu0YzZPotAFTBG9MUL62Ji0PydGpb34VU29NRg/6qX9sXE0atY+7nQ 4Be4murCOQGlGzS3JTk+k7/EGTF/koz9Coc/4zem72jqTCi+hiwnDkonOLGFXe95WRM= X-Gm-Gg: AeBDiesRKdxl1+2P2B9R8QkWEqHl1WnEXS/xieQFhNIG1FTE4vB2WxWHLaL08+sjoYP 4j/awV84/52X6YRP6nw33IreloFXPSmH4ARYUF5Z8//lb589hTOpfAOJ8o9I+ODt1mjsVF4EwP4 Piu6xie4/jad6UcedQX+4pkMWmBDPgcmKRnWjc+5LWf9desySXxraEh7rT3J/o/5xO9IbtbNvG5 oJ6Kjd9S2lsVvyW66s8NwIIqju9OkUsRzgs+5M7VkrgSPK2q/vlv07kfHJbZ+76iV1M9c4lkBC8 +Pk5e2//FfJwfQniureGzj6ml/hAFUVSmGaIWCdGbQnnEjW9UWKQrs1d12OD2sevaetM28KZZli uUFtFLwPETyCKaPBT2ccRqcv/laO7nQh6KP0N3iOP7jyLHBM7Ng9MFmPkEu+5/1oCsnO+FO92s5 t57slNEWxmJpEEtTUO3KjVm8xKvO7u5qcY4SgYYtE= X-Received: by 2002:aa7:d9d3:0:b0:672:40fd:d7cc with SMTP id 4fb4d7f45d1cf-672bfd875f8mr9021191a12.6.1776956812937; Thu, 23 Apr 2026 08:06:52 -0700 (PDT) Received: from airbuntu ([146.70.179.104]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-672c4d4590asm4122990a12.19.2026.04.23.08.06.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2026 08:06:52 -0700 (PDT) Date: Thu, 23 Apr 2026 16:06:49 +0100 From: Qais Yousef To: Tim Chen Cc: "Chen, Yu C" , Peter Zijlstra , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Libo Chen , linux-kernel@vger.kernel.org Subject: Re: [Patch v4 00/22] Cache aware scheduling Message-ID: <20260423150649.y27ks53zldy2upwo@airbuntu> References: <20260416002749.muyrcycmtabksav4@airbuntu> <20260421003438.whnn2gvv4gkfcmx5@airbuntu> <83c14639f1d6baa1665ad367c308676e5f951ff7.camel@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <83c14639f1d6baa1665ad367c308676e5f951ff7.camel@linux.intel.com> On 04/21/26 13:57, Tim Chen wrote: > On Tue, 2026-04-21 at 01:34 +0100, Qais Yousef wrote: > > On 04/20/26 17:01, Chen, Yu C wrote: > > > On 4/16/2026 8:27 AM, Qais Yousef wrote: > > > > On 04/01/26 14:52, Tim Chen wrote: > > > > > > [ ... ] > > > > > > > > > > > I posted schedqos announcement yesterday, which I think (hope) would be the > > > > right way to address these concerns about tagging tasks. > > > > > > > > https://lore.kernel.org/lkml/20260415000910.2h5misvwc45bdumu@airbuntu/ > > > > > > > > > I think that's great. Will be a nice way to tag tasks that should be grouped > and aggregated together. To ensure it is clear, you can tag processes too, not just tasks. Though this is wasteful, but will allow easy tagging and quick results. > > > > Thanks, I'll take a look at this. > > > > > > > It would be trivial to add experimental branch to add new QoS flavour to say > > > > NUMA_SENSITIVE etc. I am still trying to think of a generic description to > > > > address a number of use cases (see Execution Profiles in README.md), not just > > > > this particular numa sensitive one, but the experimental branch should help > > > > iterate and drive the kernel development for wake up path + push lb instead of > > > > using load balance which I really doubt will work well in practice since this > > > > is slow to react, and you're relying on overcommitting the system by default by > > > > making every task of every process data dependent and require it to be > > > > co-located. > > > > > > I am not certain which strategy is preferable, as it largely depends > > > on the use case and workload. We intend to evaluate push-based load > > > balancing on top of the existing lb-based cache-aware placement logic. > > > > I'll defer to Vincent here, but I would have thought lb-based approach can go > > away. > > > > > > > > > I think in practice admins will care about specific applications to > > > > be kept within a single LLC, and if they are willing to spend the effort, they > > > > can tag specific tasks of a specific application. > > > > > > > > > > It seems to me that there are multiple use cases. In one scenario, > > > the administrator (including daemons) is responsible for tagging > > > workloads. In another, users prefer the OS to handle automatic > > > placement without any userspace involvement. > > > > How do you define this automatic placement? AFAICS you're just grouping all > > tasks of a specific process to stay within the same LLC and hitting overcommit > > issues which you're workingaround with this load balancer only based approach? > > The LLC chosen for aggregation (preferred LLC) is the one with most occupancy > by tasks in a process. > > However aggregation needs to be done with the load in target LLC and current > in mind. It is better to keep a task in its current LLC if plenty of idle CPUs are available > than move to an LLC where most of the threads are, but CPUs are frequently > busy. This is the main reason why we put the migration logic in the load > balancer where accurate load information is available and we could put > in load aware migration policy. > > It is fine to migrate tasks in the wake up path. But we need to resolve > the issue of over-aggregation, when multiple CPUs may push tasks > to a LLC independently of each other. It worsen things with > frequent tasks bouncing if we over-aggregate > and have to migrate tasks out of the LLC again. We encounter > such issues in our earlier implementations that have task migrations in the wake up path. Hmm I am still struggling to see how we can end up with these problem when we have few processes asking to be in the same LLC. With this series all processes are being grouped which I can see how this can lead to excessive ping ponging. But I acknowledge the task might not be straight forward due to lack of accurate load information. But this can't be fixable at all under more controlled and relaxed environment is where I struggle. We are simplifying the problem essentially to help the kernel do better and easier. Anyway. My goal is to help simplify the kernel details and defer the policy to the userspace completely. I am happy to work with you and Vincent to make sure we can address general QoS placement details on NUMA systems. FWIW, if we want to implement wake up based on latency, it's the exact same problems. That's why I am stressing this point. We need to be able to extend and scale. > > > > > > I think in practice there will be many corner cases where state is not optimal > > and we'd end up with heuristics to 'balance' things out and sensitivity to > > independent changes disturbing this fragile balance causing weird regressions > > and us slowly has less flexibility to move and shuffle code (okay, maybe too > > much doom and gloom, but we've been by this in the past :)). > > > > I am not sure how many of these tests stressed the system with multiple > > critical processes running concurrently? > > > > By making it a userspace problem they have to figure out the right balance and > > we can focus on providing the right mechanism. > > > > > > > > > Also QoS IMHO should be viewed as a scarce resource. For best effort delivery > > > > (which is the best we can do in reality, this is not hard real time system), it > > > > is easier to provide good best effort when the average noise level is low, ie: > > > > few tasks are required to be kept within the same LLC. If we overcommit often, > > > > we will crumble often. So IMHO the key is to delegate to userspace to tag, and > > > > > > I suppose there are two scenarios. The first is enabling/disabling > > > aggregation > > > for a group of tasks, and the second is task tagging. For the first > > > scenario, > > > this can be applied either process-wide or cgroup-wide by providing a flag, > > > > Cgroup-wide tagging doesn't make sense IMO. Process-wide yes. > > > > I think this depends on the usage scenario. In private discussion with > Vern from Tencent, he mentioned that such a cgroup based tagging is useful for them. We all want ponies :) I think this needs a why. It doesn't make sense to group procsses in general. It seems this requirement is tied to elaborate setup to force the kernel to deal with this elaborate setup in a generic manner. Anyway with the tagging approach we can easily allow process level LLC sharing via simple description like // shared cookie definition { "WEB_SERVICE_COOKIE": [ "nginx", "postgresql"], "TRANSCODING_COOKIE": [ "decoder", "encoder"] } Which simply tell the utility to reuse the cookie for these processes using the key as a unique identifier. By the way, cookie generation might need kernel help to create a unique id. Still, if someone wants such elaborate setup the first thing to suggest is static portioning via cpuset. Do you know why this is not sufficient?