From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58BA33B4439 for ; Fri, 5 Jun 2026 15:36:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.42 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673776; cv=none; b=akA5NR2iKvLIS8xIo7O1l6Soy9MEcOKE4Rpfn/qGFv1BNZwteGNZkplPeK1yt5M6fOgv1wqgV0u/CdM1cnqTM5Vyv9X02K3dmKGTI/bCwmcpKVabR4eDNO0MIGm6lthauXJ9+G9oX6v9yhaJynAD/B4oFrZcALFryGDjPZZF/v0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673776; c=relaxed/simple; bh=bIEWn8F7ak6CcMSj5/RHqMkAoowSiMeeFwKRoqLy4oI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CqjJjoX04s1KMqPXkXDUJRyds2Chey029DUpGJaLvvfjoiFWOIGFk14KRtJY3jjMQS08/sowMSoXbNKjZdqGKmcZOsY4c6EMLGWxplIEaLpZJPRbGGTu7ZaFIMS+5MXjN4Dux4e/jxLGfzr7jU/O9VXyp6G0FCI57w4cfNGupm0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hGkRBjoo; arc=none smtp.client-ip=209.85.161.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hGkRBjoo" Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-69e59978deeso912130eaf.0 for ; Fri, 05 Jun 2026 08:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780673771; x=1781278571; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2POwf/K9X70lHx5WEcIKkTy5o0DDbStiw7GuBjFdd58=; b=hGkRBjoorE08vkSWw9XWZ9ybn3p19x/5BgvaFxUHZbac7XlPN7TU+dCnYRP2d5vlqE jSGjQ20foDou1QRiS9gvDh+8xX3BjVli9Q0EesOxw/TXdzDUeIjdTgDP2UssKhgawIJm CUMQ4T1u5BD1QmQJuQBSuvZCrVF+0KWstQflK4InW4/NxiH8ot41DWFfpBlKSjvRBKNd qkTiZASqL/Tsk9WPoHoF5kfU72hwLA7EKJQKDVVolysmE1EqgOteED938DwT3BZHG+ai vaQOGpOFhZ2ceVIQ4bMjLi795TCjfYR2ArXZchV4c//Zgf8wZOQ8URwIUJ0us6/RBkZp DxWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780673771; x=1781278571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=2POwf/K9X70lHx5WEcIKkTy5o0DDbStiw7GuBjFdd58=; b=izCBOWEfwgiHRIPrThIiVQD+1fvfkOPh4qfanWwWO0AaYodKeB9J2q7PfJEAj43Mja toqskTwRuenC+SXcDjyKd0F3RS0fw+dKNr4gqqO8trqpCAa2NMmCobw1D7zX2Sa1LwrO 9jAgeSBC1S37YBn1Pa9hAquEEbBvsFZQRTF9h/BBaJVxnO4pCBlcVcrpHJsCy97ICgXP GgWmbIi5vJLE442/E8ppQeb5S7/whlvABBCnj6CTfrSGm941R8txNvVQqNoj6BLG3Vj3 +/pzTK/eBVwYWdZB+so2+ZInsAb1ymnw5e6TTrGbh8OFd4DkBLpWwap7hQfJDuAFWhKq pnzQ== X-Forwarded-Encrypted: i=1; AFNElJ8zVLFVq/bPbIxA2Wxjw4/DtOwlihvINzhkjP1w+zJWEJI7jABLqTeZcCpTp7dFOFBr4XLxXmGG55SaKbM=@vger.kernel.org X-Gm-Message-State: AOJu0YxA8aROM5RVIXbNkLN3AaY2LAo+ySSy4c7AWe2zd9emmJsesj9T wVN8fSUYonpROsOwS1wBy7ZqL6EhHFQc34DfpFyy0nE6jLBJX2L7+ShD X-Gm-Gg: Acq92OHdjLCUsQoOJR9ktlEfSSjdxEqDE5wGJxA/7jcmz5o1XlVXVZ8loPXP4fOiz8P siUs/G/E8zmy//P3CaIOxNRR+DAXUpM0eAHVsgvVHeLrEKPmyUU8aJ21YWdFP4nT330oDsR0B5N eAEKnvVLozBXLNOaS4MDXiSSHH4bAuSyNjRG1GH0xcOuEUR+F6GYHgeq6D6T/nE1Dljc8a6SoCF c2jb0xsbjuRSxJR/2iYSKnXDWzHBvrrmLt8J8qd+M1IhYsaJlBlzxkOWDswdeUawkSbjJZi26t2 ffM6BmNdzn7p5yi9Rd9zuOhbumq+9CncaEv2bO/xaZDXD00WqrAvy8tXWb9Yv1vQIFpluf/j2Jq bPGikzNfG4gUaN3kahUZoTZUzqOP58O3f9aOqiyL03pKXP1GBNd45xS8k52kP8c5JJRH5B8p7hz sISM4tm9IGkYzgPOjQE6hoE3gJx4+lKt84ARyV/7dPS7H/WARlOCPBDz9FzqlYvu2tatWgZQ0aV w== X-Received: by 2002:a05:6820:4b03:b0:69e:59ec:f70b with SMTP id 006d021491bc7-69e68c19e22mr2043956eaf.37.1780673770989; Fri, 05 Jun 2026 08:36:10 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:3::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-69e4620b0casm5398168eaf.3.2026.06.05.08.36.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2026 08:36:10 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 5/6 v3] mm/memcontrol: optimize memsw stock for cgroup v1 Date: Fri, 5 Jun 2026 08:36:01 -0700 Message-ID: <20260605153603.234296-6-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com> References: <20260605153603.234296-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Previously, each memcg had its own stock, which was shared by all page counters within it. Specifically in try_charge_memcg, the stock limit check would occur before the memsw and memory page_counters were charged hierarchically. Now that the memcg stock was folded into the page_counter level, and we have replaced try_charge_memcg's stock check against the memory page_counter's stock, this leaves no fast path available for cgroup v1's memsw check. Introduce a new stock for the memsw page_counter, charged independently from the memory page_counter. This provides better caching on cgroup v1: The best case scenario is when both the memsw and memory page_counters can use their cached stock charge; this is the old behavior. The halfway scenario is when either the memsw or memory page_counter is within the stock size, but the other isn't. This requires one hierarchical charge. The worst case scenario is when both memsw and memory page_counters are over their limit, and must walk two page_counter hierarchies. This is the same as the old behavior. By introducing an independent stock for memsw, we can avoid the worst case scenario more often and can fail or succeed separately from the memory page counter. One user-visible change is that reported memsw usage may transiently be lower than memory usage. This happens because each counter independently batches the stock charges, so the visible values can differ by up to the stock batch size (MEMCG_CHARGE_BATCH) pages. Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 562ed9301f5a4..d0da2f842e2d4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2274,8 +2274,11 @@ static long drain_stock_on_cpu(void *arg) struct mem_cgroup *root_memcg = arg; struct mem_cgroup *memcg; - for_each_mem_cgroup_tree(memcg, root_memcg) + for_each_mem_cgroup_tree(memcg, root_memcg) { page_counter_drain_stock_local(&memcg->memory); + if (do_memsw_account()) + page_counter_drain_stock_local(&memcg->memsw); + } return 0; } @@ -2324,8 +2327,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) /* no need for the local lock */ drain_obj_stock(&per_cpu(obj_stock, cpu)); - for_each_mem_cgroup(memcg) + for_each_mem_cgroup(memcg) { page_counter_drain_stock_cpu(&memcg->memory, cpu); + if (do_memsw_account()) + page_counter_drain_stock_cpu(&memcg->memsw, cpu); + } return 0; } @@ -4237,6 +4243,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) /* failure is nonfatal, charges fall back to direct hierarchy */ page_counter_alloc_stock(&memcg->memory, MEMCG_CHARGE_BATCH); + if (do_memsw_account()) + page_counter_alloc_stock(&memcg->memsw, MEMCG_CHARGE_BATCH); /* * Ensure mem_cgroup_from_private_id() works once we're fully online. @@ -4299,6 +4307,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) lru_gen_offline_memcg(memcg); page_counter_disable_stock(&memcg->memory); + if (do_memsw_account()) + page_counter_disable_stock(&memcg->memsw); drain_all_stock(memcg); mem_cgroup_private_id_put(memcg, 1); -- 2.53.0-Meta