From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63C7D39902D for ; Fri, 22 May 2026 22:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487603; cv=none; b=EhX81sKAU8VJq/WVEV71Nelcx3eWopjy2zgtXYEfsABuHS4fAe0VV0NycPiNac9uJXKPD/wRO0IRN2cuBIMqZ/RIKz3Bc9aBjUAzq56Sk93rTMG51QQzHpSJXWQI14tFosRJk0LRiCPlwL8RQ3FNyazdFIIDYWgBy4xC5HwvGGc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487603; c=relaxed/simple; bh=4UYwrcyLg6NFLJwsNn/rvSTVnQ4sAFbwhsLF1ErPS6s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YFADcEwwcbgG4Ag4h8D1TnS17U2T+67iOks60I8yjLXGG+kK37PR0/kPj4+XvAU1OG6nyEYqDVjX+WMAlGULR8NpzXKqTSLqad/uHOi3EQavQSdLxi9KoHf4BktzmaEQFRk6PXyAm4WbiDxULPSgmed2NEq9sc9VcIbNk2JPHEo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M+UknO7K; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M+UknO7K" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-4855003fa6cso839136b6e.3 for ; Fri, 22 May 2026 15:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779487601; x=1780092401; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=31yASLdMSEM7nexMHwB6RiM0ndTUZyyCT5Ui47lS5/Q=; b=M+UknO7KedpAQJVmVfDNiBhxZosxKwfgE9FYCfCPY0sdnhwghGxjdkUGwL1aoF+8Lj 6qcW9xfW1aCvu+NFmNZFQ3sWKnWDD82fkGwA6/phTbJDJalVepRaFLUtrxmwVVEa23uq ZmpIInVVK7nARoBlw7T7O9ZTaYXsnzQpnFbpu0/Ada+Q3XPph8BhnhN8qG17xmGAIvYx OwCgYxC9hHw1l8vMsBFYci1dAI8rmu0PoxpcEeGSCKCkNvwnO/yeVf0VhIHgGZ94oMi1 tEicObjnLRRIMBN+QvXcIC24vXWill1chCoTL338KnoXL68ApNaO5UMl2vOVF0iM5YCy 6dow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779487601; x=1780092401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=31yASLdMSEM7nexMHwB6RiM0ndTUZyyCT5Ui47lS5/Q=; b=mAvW5OTVsKvq0TXwPyskLBv2HarVvFkxqOGwLx4eJJOqH39ji0Mp5o9O8zD+8VBRSY jTcthar4btGdbY6Y0mLfvKdd2TLmwlgI0sKxVUZEys8PifU7Uf+i9xTeIhPSN7WDxRe5 rrvM65OijoHtDvvJZInPlMnbGV923z0ig0mKjjIuVRuuV0TOPhxQtkzxGMDP10IJMTQR C0PjrOFIHWXGuhGjZZOOGQyMaVO0pBwQwSib4eIQFfMilS3MVA6URVEZJ0GzU/AAkpA5 32eahiYzBhwatnXkf2NywYDLP9b54fZKBDuTM6oe+8jpKQ1Yy2eSSiX+QPrmsJJJOOjU zXKQ== X-Forwarded-Encrypted: i=1; AFNElJ+wrnSAxLdJTmMvPDWOBATSIck4BCTIfpcAe/Kpy1buGeQXovOZAZ9Vi7ncuc1nMwBvexyXAO/27/IbYvE=@vger.kernel.org X-Gm-Message-State: AOJu0YyjDbAUJaGKz+GR0vENCaGHupL78YlfIAqqmv6uXd3rGJTPuNIU Bgk/aC3k/bF2wD4rfWRjV+ftCLFRk/LKofc+hTczKPOy5v3+DIZKzlwi X-Gm-Gg: Acq92OGSVhClDBE69e3Q9pcZ3hffe6oWfZTLknqE0zNny4qwq1j6EJgxHcdJKdOKDk1 qxMVjTTZhaEl6J27aZrJ74DCrGEDJ/oKyb4QM3GL+xauWTr/kvYFW1O6CmHyNbDw4oYuVzf7z7a 4bxnnz+XEeCVPmTBss99XcLHt/gsyBBDCH2QtRvVmJrIfX58asobTO4L2RoVGGhB78leZGIwMwc d1m3mrCsIEGw+7fRc+iaXMfsOTe6kH0x673qeGqnVQeWZPKnrO6jXnFQ2rk8sZyAsVo4QXDZFrQ yNJamvXyz1TaAMih+Ed8RR61zidKxICmcMfR4IQL3RdI538iI6QgGVrbsBeqYOp1Ih21ddnXU4m r6ULFufqBlaLOYVkSxHzSGVLe43eBUpC0UOeVW/JWMHU/qaUD1+MJnDKfvPhYVpwbJGGzYu4LyD 9NRQI+5U1jMSwzaxfxtvWZtUxSjUgFLn7OAFxzO8MimNr6DI5b9URWupINEvvb4uOUiflcA136H Qc= X-Received: by 2002:a05:6808:c2fa:b0:485:4441:721a with SMTP id 5614622812f47-4854a1d46f1mr3600895b6e.36.1779487601412; Fri, 22 May 2026 15:06:41 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:73::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-48554041ccasm1180385b6e.0.2026.05.22.15.06.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2026 15:06:40 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 5/7 v2] mm/memcontrol: optimize memsw stock for cgroup v1 Date: Fri, 22 May 2026 15:06:23 -0700 Message-ID: <20260522220627.1150804-6-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260522220627.1150804-1-joshua.hahnjy@gmail.com> References: <20260522220627.1150804-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Previously, each memcg had its own stock, which was shared by all page counters within it. Specifically in try_charge_memcg, the stock limit check would occur before the memsw and memory page_counters were charged hierarchically. Now that the memcg stock was folded into the page_counter level, and we have replaced try_charge_memcg's stock check against the memory page_counter's stock, this leaves no fast path available for cgroup v1's memsw check. Introduce a new stock for the memsw page_counter, charged independently from the memory page_counter. This provides better caching on cgroup v1: The best case scenario is when both the memsw and memory page_counters can use their cached stock charge; this is the old behavior. The halfway scenario is when either the memsw or memory page_counter is within the stock size, but the other isn't. This requires one hierarchical charge. The worst case scenario is when both memsw and memory page_counters are over their limit, and must walk two page_counter hierarchies. This is the same as the old behavior. By introducing an independent stock for memsw, we can avoid the worst case scenario more often and can fail or succeed separately from the memory page counter. One user-visible change is that reported memsw usage may transiently be lower than memory usage. This happens because each counter independently batches the stock charges, so the visible values can differ by up to the stock batch size (MEMCG_CHARGE_BATCH) pages. Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb1ea17e03730..06ec4d26cb519 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2058,8 +2058,11 @@ static long drain_stock_on_cpu(void *arg) struct mem_cgroup *root_memcg = arg; struct mem_cgroup *memcg; - for_each_mem_cgroup_tree(memcg, root_memcg) + for_each_mem_cgroup_tree(memcg, root_memcg) { page_counter_drain_stock_local(&memcg->memory); + if (do_memsw_account()) + page_counter_drain_stock_local(&memcg->memsw); + } return 0; } @@ -2106,8 +2109,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) /* no need for the local lock */ drain_obj_stock(&per_cpu(obj_stock, cpu)); - for_each_mem_cgroup(memcg) + for_each_mem_cgroup(memcg) { page_counter_drain_stock_cpu(&memcg->memory, cpu); + if (do_memsw_account()) + page_counter_drain_stock_cpu(&memcg->memsw, cpu); + } return 0; } @@ -3947,6 +3953,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) /* failure is nonfatal, charges fall back to direct hierarchy */ page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH); + if (do_memsw_account()) + page_counter_enable_stock(&memcg->memsw, MEMCG_CHARGE_BATCH); /* * Ensure mem_cgroup_from_private_id() works once we're fully online. @@ -3987,6 +3995,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); page_counter_disable_stock(&memcg->memory); + if (do_memsw_account()) + page_counter_disable_stock(&memcg->memsw); mem_cgroup_private_id_put(memcg, 1); } -- 2.53.0-Meta