From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7067828AB0B for ; Thu, 16 Jul 2026 15:30:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784215804; cv=none; b=rzZ6tIMsnf1/lylONf+btJvk+IEH3xG+bJEHhd2NI/FP0mHzO758jw0N8vtgKFDd9ukfNBruXK6AmvT2eu5poA7Il3C+9z6D/aW1yWkXjbfljco3ShSaInkwml3BsOEpGryBkaXq3AnJPmYUUfg9o+d5TsL10SIktVmcjSmF7zA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784215804; c=relaxed/simple; bh=8iV2D4LQwIFxirt282BnezgO4AAv0z0AJbV3lY+KhRE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WErKxNk3O8AGiR6gaAm/iw8qMZPW0Zx4Q4dmKQCH4Uc2wSemYPj7GhB77NMkEqTG16UfIMrBixmGjWIrmyuC2Lw7b5U6Lu8ObPavtrzHyM5VBjNsPGiraSPp+lJ61utpkBwUWQMh4O67pQfGQZLf2xTwHQhSwGL8irXbwa/eoqU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ahs+VdUf; arc=none smtp.client-ip=209.85.160.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ahs+VdUf" Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-4471b489240so2292891fac.2 for ; Thu, 16 Jul 2026 08:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784215802; x=1784820602; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HOgCZnnw3NvbVl3CM6pSt350xIpvqCivFKqtyhSvLeI=; b=ahs+VdUfBhiSwoi7GOTt8/5V59rKcUZaznsBXc35FJWqnBc+AnWexEXsxgBEMpKIri PkDjLEKCHObH/2CceRkgWrQFXNAeZ+dt4bfH0v58W6wRkFD1HA1liCWiSvpXecX/JDiq LweQ26C/0caRVG17NGQGiYSpc+wwmKCeYq6tuUpRPqC1TFqqa9saA2vMxEQfSnXi26Fn 6OsHrp+8/WF/cjdhbG3gYt5Jo28D8OQGuslsaZFWdtEbSz6GSuX4NE52ACS+o/mIJB7H MCUwbcvjbff9zNP67f2wcrVw58fXw8Do0K63xeX47j4eIxYyIdIAi+MS9Ksb8VMAdRNa JB4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784215802; x=1784820602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=HOgCZnnw3NvbVl3CM6pSt350xIpvqCivFKqtyhSvLeI=; b=eVin4Q/COb8R9GkWlTsfb8gFEp9jlOCb4sJD4xkdPgsgm9DWcXCRp32LPbFFwvc+/w HOtalKOjJWh8Agj0fDejmwb+qj5Hk3LHHvzqns7yGyvhbJmGP+pbjNRIhO4Fc70r/S6S GeIFGYaB+qZuO7WnoaEltbOc7XBoOAXwLTrKMpzwI7xRI5LIfdZXNkyJFP95m1Y7ZZD/ Z6WJYcmJeBcgeMZ1mFb4lRHu2mOYnAyxOROYL1dni/VghxcT+LZAPg/SomKbo7/iugMX URyMPE64V4i8a5cl23o3UtaloO5ua8sR342/jbCYHOFnTl/JQF9BJU6JYLGviLejWyF7 2Hyw== X-Forwarded-Encrypted: i=1; AHgh+Rrh7c2sopKo624AbTA7HLgB+AGCUpp3Nrb8hjt+dcbtqxXWRy+9vpGzv2GM6jjUSOnZe0F1DvCtHGV3R+o=@vger.kernel.org X-Gm-Message-State: AOJu0YwEeU/43snfLCYvZdULWLdyX/Na+SvYKT6QJPZMazTM7PCY/enQ Jx83DdGj6XPB6QtmByvuF0kz6dSUOMXMMfPQex6nrRPbvPlTbN6TnFgy X-Gm-Gg: AfdE7cnlZEDpaGXfk3j6s4JR2Mks0cSni6gs+Oacto8+MKJQOHfjKlbSuovNpN1Be8Z Bp3RhoZw9Zm9rFfc5CX2ITMyWg0NK2Hdy5fS1Dr2E4WIvEekI9XbQhclLwt71VeFwajOXxN+I12 iYLLZdHGQj9BfXRqStranerD38N2oo7t82OWm3mPhuaAsUEs6Ex5WFNaP2Ff5+FbAii3+rZHPuv eCJEZbrTx1DApvibdH2YHSekl7FIPxLyWIn3xM1IXi0qBOjgK+ro+OOW7wovSoECGzCnnDne9GL zYT7zcuo+8+5r3w+kLJT0FIW3mtbHUWa62ErtZQrjxK+F8G+UeFzJkdYVkAjF2lYkjIjl4Cz6Ls Sys8wcbe7DP4YT0TAUVikOhQ01kcjzIPCRLH8Dix50QlHG6vO72AxpPwlsYb7KyJs4Bgg2rdDUk yqoQAvgzKItlCB2dnbJt44Fx32N380E/U= X-Received: by 2002:a05:6808:15a5:b0:495:e0a9:8c0 with SMTP id 5614622812f47-4a495e79355mr5583029b6e.11.1784215801297; Thu, 16 Jul 2026 08:30:01 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:3::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4a1ad4ac92csm18484759b6e.5.2026.07.16.08.30.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jul 2026 08:30:00 -0700 (PDT) From: Joshua Hahn To: "David Hildenbrand (Arm)" Cc: Tejun Heo , Johannes Weiner , Michal Koutny , Michal Hocko , Roman Gushchin , Shakeel Butt , Andrew Morton , Chris Li , Kairui Song , Muchun Song , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , Youngjun Park , Qi Zheng , Axel Rasmussen , Yuanchu Xie , Wei Xu , Kaiyang Zhao , David Rientjes , Yiannis Nikolakopoulos , "Rao, Bharata Bhasker" , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: Re: [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware Date: Thu, 16 Jul 2026 08:29:58 -0700 Message-ID: <20260716152959.3524430-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit On Thu, 16 Jul 2026 15:11:30 +0200 "David Hildenbrand (Arm)" wrote: > Sorry for the late reply :) Yes, hopefully talk to you in person next time! Hello David, no worries! Thank you for taking the time to consider this series again. > >> Assume you have a bigger hierarchy (HBP, DRAM, CXL), or assume you have multiple > >> NUMA nodes with a hierarchy each. > >> > >> Your proposal doesn't really seem to be very versatile, or am I wrong? > > > > Let me address these comments separately! > > > > First, for the multi-numa-per-tier case, I think this is already pretty well > > handled by my series. Once we realize that a memcg is consuming too much memory > > from a tier, we trigger reclaim from that memcg via > > try_to_free_mem_cgroup_pages, > > which as far as I can tell already handles the multi-numa per memcg case. > > Other than restricting the scan_control's nodemask to target the nodes > > from that tier, I don't think there's anything else to be done. > > I am not really sure about the "toptier" notion, in particular if your memory > hierarchy is not static. > > Imagine that you have only DRAM in your system. Then you hotplug some slow CXL > memory. The you hotplug some HBM CXL memory. > > The meanting of "toptier" changes, which is rather confusing to me. While I do agree with the idea that tiers in general are not stable, I think that for the most part the "toptier" node is generally stable through hotplugging nodes. In your example above, the DRAM will always be the toptier node, right? In any case, I think if we are to support 3+ tiers I have to change the semantics to distinguish between higher and lower tiers instead to be generic anyways, so I will change the wording to reflect this! > > Next for the 3+ tier case, I think this is a lot more scalable than it seems > > at first. This series depends on another RFC that I sent out [1], > > which pushes the concept of "stock" from memcg to page_counter, which means > > that it is more scalable to just add more page counters to each memcg. > > This means that each tier would just need another page_counter to track its > > memory usage, and we trigger selective reclaim on the tier that is being > > targeted via the scan control nodemask we introduce in this series. > > > > At my talk in LSFMMBPF, Usama noted that the user-visible API should probably > > remain the same, no matter what. The way I have currently established the > > memcg files aren't really scalable, so Usama suggested turning the > > "memory.toptier_XXX" sysfs files to "memory.tiered_XXX", which would include > > a newline-separated / space-separated list of per-tier limits. Something like: > > > > $ cat memory.tiered_max > > tier_0 20971520 > > tier_1 31457280 > > ... > > memcgs are currently not NUMA aware, so this could be one possible direction. > > But is "tier" the right abstraction? I really don't know. >From my Pov after working on this for a while, I think tiering is the grouping that makes the most sense. It is my understanding that memory present in different nodes, but belonging to the same memcg are fungible; memory from one node isn't more or less scarce than the other node. However, if the system has multiple tiers then the memory is no longer fungible, and memcgs will be competing for the same scarce resource. So I think it makes sense to limit how much a memcg can make. > Are tier numbers really what we want? I don't know. If this doesn't make sense, I can change this to just tiers 0-3 (MAX_NR_TIERS is currently hardcoded to 4 across all architectures). > We'd really need some feedback from memcg folks. That would really be awesome : -) > > So we have a way to make both the user-facing side stable, and the internals > > also more scalable. > > > > With that said, I've opted to leave the internals to 2 tiers for now -- I think > > it is not too late to add the generalization series when we start seeing > > 3+ tier systems out there in the wild. My goal was to introduce tieredness, > > and we can work towards generalization in a future work. > > > > On that note, it seems like in general mm is aware of 3+ tiers, but most of the > > existing work revolves around distinguishing between toptier/everything else. > > I got this impression from reading mm/memory-tiers.c -- but please feel free > > to correct me if you feel like I have the wrong idea here : -) > > I assume you're right, but it's something different if we end up providing a > proper memcg API towards the users. > > > So perhaps the generalization work would benefit from first introducing more > > general tier awareness (not just toptier vs. rest) in memory-tiers.c. > > > > What do you think? Does this approach of introducing toptier restriction for > > now, and then generalizing in future work make sense to you? > > I'd suspect that we'd want to support multiple tiers from day one -- especially > get the interfaces right. So I think the interface that I have at the moment (in the v3) is stable across any number of memory tiers. My idea was to land something with a stable API and then as future work support 3+ tiers. But if it makes more sense to land the scalable tiered memcg work as the first iteration then I can work on that too. I've been working on v3 for a while, I will send it up sometime soon so that I can start the discussion on this again. I really want to discuss with memcg maintianers and migration maintainers to get some confirmation on whether the series's direction looks good and whether they are happy with my vision for the series. And then in v4, I will add the scalable tiers. How does that sound? : -) Again, thank you for coming back to this. It means a lot! Joshua