mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Joshua Hahn <joshua.hahnjy@gmail.com>
To: Gregory Price <gourry@gourry.net>
Cc: "David Hildenbrand (Arm)" <david@kernel.org>,
	linux-mm@kvack.org, Tejun Heo <tj@kernel.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Michal Koutny <mkoutny@suse.com>,
	Michal Hocko <mhocko@kernel.org>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Shakeel Butt <shakeel.butt@linux.dev>,
	Andrew Morton <akpm@linux-foundation.org>,
	Chris Li <chrisl@kernel.org>, Kairui Song <kasong@tencent.com>,
	Muchun Song <muchun.song@linux.dev>,
	Lorenzo Stoakes <ljs@kernel.org>,
	"Liam R. Howlett" <Liam.Howlett@oracle.com>,
	Vlastimil Babka <vbabka@kernel.org>,
	Mike Rapoport <rppt@kernel.org>,
	Suren Baghdasaryan <surenb@google.com>,
	Kemeng Shi <shikemeng@huaweicloud.com>,
	Nhat Pham <nphamcs@gmail.com>, Baoquan He <bhe@redhat.com>,
	Barry Song <baohua@kernel.org>,
	Youngjun Park <youngjun.park@lge.com>,
	Qi Zheng <qi.zheng@linux.dev>,
	Axel Rasmussen <axelrasmussen@google.com>,
	Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
	Kaiyang Zhao <kaiyang2@cs.cmu.edu>,
	David Rientjes <rientjes@google.com>,
	Yiannis Nikolakopoulos <yiannis@zptcorp.com>,
	"Rao, Bharata Bhasker" <bharata@amd.com>,
	cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
	kernel-team@meta.com
Subject: Re: [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware
Date: Thu, 23 Jul 2026 13:21:18 -0700	[thread overview]
Message-ID: <20260723202119.3688800-1-joshua.hahnjy@gmail.com> (raw)
In-Reply-To: <alkB4tnJxOLyVCi2@gourry-fedora-PF4VCD3F>

On Thu, 16 Jul 2026 12:55:55 -0400 Gregory Price <gourry@gourry.net> wrote:
> On Thu, Jul 16, 2026 at 03:11:30PM +0200, David Hildenbrand (Arm) wrote:
> > > 
> > > First, for the multi-numa-per-tier case, I think this is already pretty well
> > > handled by my series. Once we realize that a memcg is consuming too much memory
> > > from a tier, we trigger reclaim from that memcg via
> > > try_to_free_mem_cgroup_pages,
> > > which as far as I can tell already handles the multi-numa per memcg case.
> > > Other than restricting the scan_control's nodemask to target the nodes
> > > from that tier, I don't think there's anything else to be done.
> > 
> > I am not really sure about the "toptier" notion, in particular if your memory
> > hierarchy is not static.
> > 
> > Imagine that you have only DRAM in your system. Then you hotplug some slow CXL
> > memory. The you hotplug some HBM CXL memory.
> > 
> > The meanting of "toptier" changes, which is rather confusing to me.

Hello Gregory, hello David,

So circling back here a little bit (sorry that it took a while to get back)

> I think this is a naming problem.
> 
> Maybe we should start calling this "initiator tier" or something.
> Maybe tier is even the wrong notion here, but it's the best we have.

I think the confusion is that "toptier-ness" doesn't give the full picture
for whether a node is a promotion candidate, a promotion candidate node
needs to both be toptier (adistance < toptier_adistance) AND also have
CPUs. So a HBM node that is hotplugged can definitely be toptier
in that it has a smaller adistance than the toptier_adistance, and will
probably even have a smaller adistance than the DRAM node, but if it
doesn't have CPUs, it won't be a promotion candidate.

It gets a bit more confusing because this distinction is not a tier-wide
distinction; you can have two HBM nodes hotplugged where only one of them
is CPUless -- then we cannot make any statement on whether the tier is
a proper promotion candidate, only that one of the nodes within that
tier is a candidate.

> But I agree "toptier" naming doesn't generalize to complex systems.

The merits of calling a node a "toptier" node is therefore a little
dubious I think, it's just one part of whether a node can be promoted to.

Anyways, I think this is outside the scope of this series. I think that
for the scope of what I am working on I will try to move away from
referring to things as "toptier" nodes but rather call them 
"promotion candidate" nodes. I hope that distinction will make this
series easier to understand and read. 

> > > $ cat memory.tiered_max
> > > tier_0 20971520
> > > tier_1 31457280

One change here is that it is not really possible to have 0..N-1 indexed
tiers, since those are not stable. For instance, hotplugging a HBM
node (as David pointed out) would replace what tier_0 is. However,
one thing that remains stable is the abstract distance (adistance) the
tiers are associated with, so I'll preserve the ABI that I have right
now, so we can have something like

tier_42 20971520
tier_128 31457280

And if HBM does come up as the "highest tier" with the lowest abstract
distance we will just have

tier 16 1048576
tier_42 20971520
tier_128 31457280

And we will not have any problems with tier_0 defining DRAM and then
later switching to define HBM. It is certainly less interpretable but
I think this is the only way to have a stable set of tiers, since
adistance is pre-calculated for all possible nodes at lateinit time.

> > memcgs are currently not NUMA aware, so this could be one possible direction.
> > 
> > But is "tier" the right abstraction? I really don't know.

> Really I think we're trying to classify 3 general pools of memory
> 
> - Precious:      HBM
> - Preferred:     DRAM  (initiator preferred targets)
> - Non-Preferred: CXL   (fallback and demotion targets)
> 
> HBM nodes are already weird in terms of fallback ordering, but ignoring
> that for a moment.
> 
> Non-preferred "tiers" are slightly different than preferred "tiers".
> 
> Under normal operation (assume no mempolicy or cgroup weirdness):
> 
> 1) Direct reclaim NEVER targets non-preferred tiers (kswapd only)
> 2) Promotion NEVER targets non-preferred tiers (promotion -> toptier)
> 3) Under total system pressure (direct reclaim on top tier), we swap
>    directly from the preferred tier out to swap
> 
> So non-preferred tiers don't have the same reclaim behavior associated
> with them.  In fact i'm pretty sure we could summarize non-preferred
> tiers as being managed hyper-lazily in the current scheme.
> 
> (if there's space, have at it - otherwise you have to wait for kswapd).
> 
> In my head, I think what we mostly want is:
>   - Precious      - controlled entry,  Private node?
>   - Preferred     - semi-precious, fairness desired
>   - Non-preferred - not precious, lazy managed, no fairness
> 
> Just some food for thought.  I've been poking at making non-preferred
> tiers less lazily managed, and I'm just finding this results to massive
> regressions.

Thanks for your insights Gregory. All of what I am saying is just in
aggressive agreement with what you and David have pointed out.

I am working to make the next version generic across tiers.
Thank you both! I hope you all have a great weekend,
Joshua

  reply	other threads:[~2026-07-23 20:21 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-04-23 20:34 Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 1/9 v2] cgroup: Introduce memory_tiered_limits cgroup mount option Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 2/9 v2] mm/memory-tiers: Introduce toptier utility functions Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 3/9 v2] mm/memcontrol: Refactor page_counter charging in try_charge_memcg Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 4/9 v2] mm/memcontrol: charge/uncharge toptier memory to mem_cgroup Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 5/9 v2] mm/memcontrol: Set toptier limits proportional to memory limits Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 6/9 v2] mm/vmscan, memcontrol: Add nodemask to try_to_free_mem_cgroup_pages Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 7/9 v2] mm/memcontrol: Make memory.low and memory.min tier-aware Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 8/9 v2] mm/memcontrol: Make memory.high tier-aware Joshua Hahn
2026-04-23 20:34 ` [RFC PATCH 9/9 v2] mm/memcontrol: Make memory.max tier-aware Joshua Hahn
2026-05-11 15:56 ` [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware David Hildenbrand (Arm)
2026-05-11 20:03   ` Joshua Hahn
2026-07-16 13:11     ` David Hildenbrand (Arm)
2026-07-16 15:29       ` Joshua Hahn
2026-07-16 16:55       ` Gregory Price
2026-07-23 20:21         ` Joshua Hahn [this message]
2026-07-16 17:33       ` Joshua Hahn

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260723202119.3688800-1-joshua.hahnjy@gmail.com \
    --to=joshua.hahnjy@gmail.com \
    --cc=Liam.Howlett@oracle.com \
    --cc=akpm@linux-foundation.org \
    --cc=axelrasmussen@google.com \
    --cc=baohua@kernel.org \
    --cc=bharata@amd.com \
    --cc=bhe@redhat.com \
    --cc=cgroups@vger.kernel.org \
    --cc=chrisl@kernel.org \
    --cc=david@kernel.org \
    --cc=gourry@gourry.net \
    --cc=hannes@cmpxchg.org \
    --cc=kaiyang2@cs.cmu.edu \
    --cc=kasong@tencent.com \
    --cc=kernel-team@meta.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=ljs@kernel.org \
    --cc=mhocko@kernel.org \
    --cc=mkoutny@suse.com \
    --cc=muchun.song@linux.dev \
    --cc=nphamcs@gmail.com \
    --cc=qi.zheng@linux.dev \
    --cc=rientjes@google.com \
    --cc=roman.gushchin@linux.dev \
    --cc=rppt@kernel.org \
    --cc=shakeel.butt@linux.dev \
    --cc=shikemeng@huaweicloud.com \
    --cc=surenb@google.com \
    --cc=tj@kernel.org \
    --cc=vbabka@kernel.org \
    --cc=weixugc@google.com \
    --cc=yiannis@zptcorp.com \
    --cc=youngjun.park@lge.com \
    --cc=yuanchu@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®