From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8465C39658A for ; Tue, 24 Mar 2026 15:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.42 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774367059; cv=none; b=lx4yYNLGpcaORr8GNKNZSuAEC3J6cxOfKRL/gb+jP8HvW2iyYq5HsBeNNNT7XQ0vckZQAPX+9VgM5TYEaJjcT5l2W6YfVlbnQmHvI6dlZq14NUNqJ//X6J8c7o9oJnAZnKeh9N2vpLODAeqcvF0PfbF6rGXl4Q2GlIP6GJOBDHg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774367059; c=relaxed/simple; bh=Vehk0QM2fXERzlwYeIb5QCPQ2DmgSyzpx0RkxsHDeP0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fIW6WzdG9Z5E9Cg+DsEK3xX+MKzGOY60BGSUTkdmAcKt/isp4DhoggN3IbvxU5w/YPdfNfGLAFcYj+naRZy6yM7N9HqWMLNq+Lm5fTtRLZhD3fQ8fLRzozZ7Agy13W8d5Rs/5We+K+qTjIfisoUmbNTgbseulK+gZWSGp6lH2JI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=S0x6/rX/; arc=none smtp.client-ip=209.85.210.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S0x6/rX/" Received: by mail-ot1-f42.google.com with SMTP id 46e09a7af769-7d7ebe11bffso3730937a34.0 for ; Tue, 24 Mar 2026 08:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774367056; x=1774971856; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H/z+GYpxsMEozZLGaH6wDzqozrpwK3g4rVLvamGwvHc=; b=S0x6/rX/u04ODOeRtOqzK/5Cq2Ur+ZDEacYvhHkrZp44Bg2MX2XbC+2SnovCv6jpVf ojnVg92045PQiPRaetAVAbDxz55mzbk3Z75TmZgMIiO0ClRAFttR9ol8jpuKuXx+668l xTpgq3REg6NX3gKD2Kq6B2CZn6er4JIv8Z5MEb7dk+GEcQoQlGBSDKfE44o6sPkizCKo FIy2GQ4yGn/QneEB37gsQsU68SWLgQSr3rkEwaLSj/Me/7BA2+IIvHxNSJt4I28mJoXD X0+HC7Tog9VqFF0hm5T0omEZ31YNILcgSKoM70XxbeIJ8JPCEJtNoxVsNRQosq0kiTpZ r42A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774367056; x=1774971856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=H/z+GYpxsMEozZLGaH6wDzqozrpwK3g4rVLvamGwvHc=; b=mrEeU5o872idIrHYAIa7Y4DBwQrx1GGW+mUakQEeaKtt2NpwXDbpyhECG8AKFLONEw usuRwcwpKazuvM2xKBLqs7MNsR/APC7XpaQ0IJl5CINt2HrGNiBrrLiwcmlnHrqiN/pf G4MJJJlGRznVg+FNVc8lazdTwSR12cG5GupL16GcXXiruh57UyWRe9qYbuC3o0SuvB2e Hf0WEnh9K75xSOIhO7Z7R3gTSIEG93vzjeJXr2cGbpBmvOpZLkT2jBWzkpo9b9EoIUBy yKpnSLLQmuDsRNZzW/EABRT4WSX1tXvWxr863YjTFBnfKRrEuQNVAEynIg9bu/YkUIuC JOhg== X-Forwarded-Encrypted: i=1; AJvYcCX7owAnwe+wFPXskYSh/uv591TctsoViV9F6jS4iG2N6OqAT9X+9duS0jIKFMRbhMNrFbAUcLsxXgD/0M8=@vger.kernel.org X-Gm-Message-State: AOJu0Yx4yC+I0SKcvRR0wJvSeIrUgqq2nyFn4rdnEMnxc/YTCEsT6d1k VC+Y7SCv7bWmDkmL7hD6BwulNPLoQ61+tlX8AEeH/buINmqTztBJ2NVw X-Gm-Gg: ATEYQzxSEZzmR7qLnbRNtXuLdMlidRdoJIpqCjLKJ2kRdHNLM2aLM+QfgHJngKcdHEy jiQsPmZx8MXAB1BS8VeWp92w6v0ALbxFeIhM5r0kMN2yTeYdZ4Gn8fi9kTYw5ZVCTfLofHlkKJZ 8lc4pFdlgAtaFNwV3665c4bQ3EnYjF0IJug7YUXa5mW1dNUAgFcSNbaU6islf/lUNYomcqIarRr KFSqloIkMpTnG4Szx4xvFDLjykHYxkpJQJhy5sOa97IZ7LaXtdq7VfTdGiq2/hFPgtWqmgCuSDE ZSekk1YTHr3Pf38C3/ziQ/Y8UgXhSXRfrAxZqfqrqDvRMQyNmbQztpJKg7qkOaZK0xTLRhsQG68 xnoaxQdpvG/vsmqis3CUm4D/neC/fW7oRx5ZI5j0treGU1JtKEsy/sYa627H4LHxW2YKzlBtRuY hx4fUIw3vvdZW+XzkjlLRHdQ== X-Received: by 2002:a05:6830:43aa:b0:7d7:d673:c1c with SMTP id 46e09a7af769-7d7eb011282mr10512912a34.33.1774367056385; Tue, 24 Mar 2026 08:44:16 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:51::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d7eadd17d9sm13472356a34.14.2026.03.24.08.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2026 08:44:15 -0700 (PDT) From: Joshua Hahn To: Donet Tom Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Qi Zheng , Axel Rasmussen , Yuanchu Xie , Wei Xu , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: Re: [RFC PATCH 6/6] mm/memcontrol: Make memory.high tier-aware Date: Tue, 24 Mar 2026 08:44:12 -0700 Message-ID: <20260324154414.195150-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <90749965-ebc8-43b2-92e3-baec5f6e3de0@linux.ibm.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On Tue, 24 Mar 2026 16:21:06 +0530 Donet Tom wrote: > > On 2/24/26 4:08 AM, Joshua Hahn wrote: > > On machines serving multiple workloads whose memory is isolated via the > > memory cgroup controller, it is currently impossible to enforce a fair > > distribution of toptier memory among the workloads, as the only > > enforcable limits have to do with total memory footprint, but not where > > that memory resides. > > > > This makes ensuring a consistent and baseline performance difficult, as > > each workload's performance is heavily impacted by workload-external > > factors wuch as which other workloads are co-located in the same host, > > and the order at which different workloads are started. > > > > Extend the existing memory.high protection to be tier-aware in the > > charging and enforcement to limit toptier-hogging for workloads. > > > > Also, add a new nodemask parameter to try_to_free_mem_cgroup_pages, > > which can be used to selectively reclaim from memory at the > > memcg-tier interection of a cgroup. > > > > Signed-off-by: Joshua Hahn > > --- > > include/linux/swap.h | 3 +- > > mm/memcontrol-v1.c | 6 ++-- > > mm/memcontrol.c | 85 +++++++++++++++++++++++++++++++++++++------- > > mm/vmscan.c | 11 +++--- > > 4 files changed, 84 insertions(+), 21 deletions(-) > > > > diff --git a/include/linux/swap.h b/include/linux/swap.h > > index 0effe3cc50f5..c6037ac7bf6e 100644 > > --- a/include/linux/swap.h > > +++ b/include/linux/swap.h > > @@ -368,7 +368,8 @@ extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, > > unsigned long nr_pages, > > gfp_t gfp_mask, > > unsigned int reclaim_options, > > - int *swappiness); > > + int *swappiness, > > + nodemask_t *allowed); > > extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem, > > gfp_t gfp_mask, bool noswap, > > pg_data_t *pgdat, > > diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c > > index 0b39ba608109..29630c7f3567 100644 > > --- a/mm/memcontrol-v1.c > > +++ b/mm/memcontrol-v1.c > > @@ -1497,7 +1497,8 @@ static int mem_cgroup_resize_max(struct mem_cgroup *memcg, > > } > > > > if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, > > - memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, NULL)) { > > + memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, > > + NULL, NULL)) { > > ret = -EBUSY; > > break; > > } > > @@ -1529,7 +1530,8 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg) > > return -EINTR; > > > > if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, > > - MEMCG_RECLAIM_MAY_SWAP, NULL)) > > + MEMCG_RECLAIM_MAY_SWAP, > > + NULL, NULL)) > > nr_retries--; > > } > > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 8aa7ae361a73..ebd4a1b73c51 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -2184,18 +2184,30 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg, > > > > do { > > unsigned long pflags; > > - > > - if (page_counter_read(&memcg->memory) <= > > - READ_ONCE(memcg->memory.high)) > > + nodemask_t toptier_nodes, *reclaim_nodes; > > + bool mem_high_ok, toptier_high_ok; > > + > > + mt_get_toptier_nodemask(&toptier_nodes, NULL); > > + mem_high_ok = page_counter_read(&memcg->memory) <= > > + READ_ONCE(memcg->memory.high); > > + toptier_high_ok = !(tier_aware_memcg_limits && > > + mem_cgroup_toptier_usage(memcg) > > > + page_counter_toptier_high(&memcg->memory)); > > + if (mem_high_ok && toptier_high_ok) > > continue; > > > > + if (mem_high_ok && !toptier_high_ok) > > + reclaim_nodes = &toptier_nodes; > > + else > > + reclaim_nodes = NULL; > > > IIUC The intent of this patch is to partition cgroup memory such that > 0 → toptier_high is backed by higher-tier memory, and > toptier_high → max is backed by lower-tier memory. > > Based on this: > > 1.If top-tier usage exceeds toptier_high, pages should be >   demoted to the lower tier. > > 2. If lower-tier usage exceeds (max - toptier_high), pages >   should be swapped out. > > 3. If total memory usage exceeds max, demotion should be >   avoided and reclaim should directly swap out pages. > > I think we are only handling case (1) in this patch. When > mem_high_ok && !toptier_high_ok, we are reclaiming pages (demotion first) > > However, if !mem_high_ok, the memcg reclaim path works as if > there is no memory tiering  in cgroup. This can lead to more demotion > and may eventually result in OOM. > > Should we also handle cases (2) and (3) in this patch? Hello Donet! I hope you are doing well. For the second condition, should pages be swapped out? If a workload is using 0 toptier memory (extreme case, let's say they haven't set memory.low) then lower-tier should be able to use all the way up to max memory. Maybe you mean if lowtier_usage exceeds (max - toptier_usage) pages should be swapped out? But if we rearrange this lowtier_usage >= max - toptier_usage lowtier_usage + toptier_usage >= max total_usage >= max And this is just the memory.max check and is already handled by existing reclaim semantics : -) I think case 3 is a bit more nuanced. If we directly swap out from high tier and skip demotions, this is introducing a priority inversion since memory in toptier should be hotter than memory in lowtier, so we should prefer to swap out the colder memory in lowtier before swapping out memory in toptier. The idea was discussed at length at [1]. It also feels like an orthogonal discussion since the behavior isn't related to toptier high or low behaviors. Please let me know what you think. Thank you, I hope you have a great day! Joshua [1] https://lore.kernel.org/linux-mm/20260317230720.990329-3-bingjiao@google.com/