From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D28CA2356D9 for ; Fri, 5 Jun 2026 15:36:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673770; cv=none; b=Un8byLd4AphWqvBYLOpCuroyfazixtEGVJqCYeTXtVBCb2OUp8MSTQQdBef8bbg7r9/S1rqiCgL/+Da/JWgMdB5wjBKNB0ehhnD5fSa1qCx5lift+VIYYlmGAhhy1Dg5JMw/Natj+GvMhuVsuEznQ3B3mUbgmG8mb+yFa4PkgUM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673770; c=relaxed/simple; bh=fhq8e8X9blJMJLSwX7ZK1MoKmj66J3cq3EZ6sKEAElA=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=KsePanwDI3jk0FaM4NphK+/6VUHuY8M+G/wFpcFFVSG5TuJxE95iYXyV40atRu19Nu1Ak1x5OnxG6P4MmXUCDU3w62qu3AoyDTnDzJNlDNJGRFceLu+FNKMHfh5bfmg7IBme4VIwZXCZ9iBrjLRXv8QzW4tYCXdhxG25y3wZ42k= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BR20jETF; arc=none smtp.client-ip=209.85.210.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BR20jETF" Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-7e6f27619e7so1528638a34.2 for ; Fri, 05 Jun 2026 08:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780673764; x=1781278564; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=OrISvpPCKWDj31WRr2OteniiAJNmHCrQHZNb3nzP7nc=; b=BR20jETFS+kbSwrFSmvfyKrzCl7Nl7Qk4PRD6B4nK6ixCpPCMESgibqrY4eGpD5Diz W631CApwethzktX8PzniFaEIBrknlDHqKg9zi3i6TmSySwXpL+MFHrYQSI32lPi2wsIq WDq7dfiaw/OLLLEm2zo9NLlEt47xCjFKvblmiYIhdMsxNj6CbNKmX4uYd/yFvgUOAwYv Qpj5RPPMqIurURSDd/yYhSquwgE9aAu7Czw/9ktKFJ2OpZ3AmvdO7V3a4GJo77D8aeNv prA48ZHyJcTNl/SFZsOpX4on+AaAJAWgBDAb4jJB3q+hMmtpfo+Xozmc45foUg4w/3tf Q0VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780673764; x=1781278564; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OrISvpPCKWDj31WRr2OteniiAJNmHCrQHZNb3nzP7nc=; b=qA+HTbr9B0+r1QLTKBGXKTFRzAaEFW2EjAs4nXWDjUfpZ5oEw70T3pv/pvsxSAK+KA Zq7hEbMqvt578tkrt7+h+c3Ss8dEwPqK9A0UePddww9+TsyyOQPwIbgaTMQmQCNBV5GC I1uoZe5bROCF4u4OBowE2ZKnsTZ5rJYo6NRpylXibV2MoI5e7w03K8W1raiotr0oovUI rCfpV96iupgRvbj49aPNTqERsqXfI1OKC8K2HASyX5rnUVi0EhD0DV8few+BP2kARS6u 99RuIlnqYM443oXXyn9ASKfJAUsUt+v7k/eC4uxN/TuVvdv6CoOxLKIJTVjfTLDgANZs xmog== X-Forwarded-Encrypted: i=1; AFNElJ818ZS7aKejft//+iHbmSvh4rwMxyN+T0sEsHtSuY5wTEdQqNj5V3C/v5nUhHeG+I3yl+08a4Ep4Htu2mE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5E1JZcobZPjTqGkl5mxbORXydBf8a+2EsfUf5aV/lMQ1bW3Vy iv94jmt77dpQmPiP7dPx3BE54RmmCVvZaOwh7en+mKmIHO1qPNFJDHyp X-Gm-Gg: Acq92OECByIT1/W8eZ6SGp06eOtiyQw6eXPdixkDtPlazSi2RV99QtlE9Gy8uU8JsQw E3MBbgIpqG3u1MbDiz+Siw0+n0jGD0PbHjAfZzxh/oCMirVb0JpwJ1YfAm0wx9irvSe9y+i9CId 1hIgSZ+hLFr6V3SV8WTwWsPahHpnmx7kHRFdzRayqKT4J55qgoyMm5G1BHdl65C20saLUVIJ3iO V0JP90TcuFwRv7by7E6Re0kg0dfkRMRn7WInIDdBD89VY2Sj3VylzwREEiCY1R+W4pDCkQl2f/6 70Ab6EBKuTGgv+48KcIbF8LojrDdYYfntnfUB5OkX6aPdpY3FG4kKf5T+8u8yT4E0dlSCItpL3A XS4wzL26bqtVKlysvyLtMIG479FYQFhh3ctWaOsjdvjPTLqq94eXMRVqmeJkUko30hbxCiWW7UG jPj1j4M+3TcdLGZm+AwNPNQKOcvwKJ+Obf1wuee7f3o/7YBw2xfL/2LaAlDA5KXqwx X-Received: by 2002:a05:6830:4425:b0:7e6:ff83:4b46 with SMTP id 46e09a7af769-7e70ca5c621mr2621260a34.23.1780673764515; Fri, 05 Jun 2026 08:36:04 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:54::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6e7468f8fsm6342595a34.6.2026.06.05.08.36.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2026 08:36:04 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 0/6 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter Date: Fri, 5 Jun 2026 08:35:56 -0700 Message-ID: <20260605153603.234296-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged allocations, allowing small allocations to avoid walking the expensive mem_cgroup hierarchy traversal and atomic operations on each charge. This design introduces a fastpath, but there is room for improvement: 1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When more than 7 mem_cgroups are actively charging on a single CPU, a random victim is evicted and its associated stock is drained. 2. Stock management is tightly coupled to struct mem_cgroup, which makes it difficult to add a new page_counter to mem_cgroup and have multiple sources of stock management, which is required when trying to introduce fastpaths to multiple hard limit checks. This series moves the per-cpu stock down into the page_counter which consolidates stock limit checking and page_counter limit checking into page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit, the random evictions (drain & refill), and slot traversal. In turn, we can add independent stock management for additional page_counters in each memcg, which is used in my tiered memory limits series to add a new page_counter to track toptier usage [1]. The resulting code in memcg is also easier to follow, as the caching becomes transparent from memcg's perspective and managed entirely within page_counter. There are, however, a few tradeoffs. First, the bound on how much memory can be overcharged (and remain stale as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages. Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines with many cgroups, this could be significant. There are four qualifying points: 1. Larger machines should be able to tolerate the additional overhead, 2. Stock should not remain stale as long as the cgroups are actively charging memory, 3. Getting close to this overhead is rare, as it would require a process to migrate across all CPUs and leave stock there. 4. These charges are not "real" allocated memory, but rather accounting done in memcg; they are easily returned on pressure. Secondly, we introduce some additional memory footprint. The new struct page_counter_stock adds 2 words of extra overhead per-(cpu x memcg). A small change is that for cgroupv1, reported memsw usage can be lower than reported memory usage, if the memsw page_counter overcharges to its stock whereas the memory page_counter does not. Finally, to keep the above memory footprint limited, I opted to not embed a work_struct into page_counter_stock, but rather decided to trigger synchronous stock draining, since the drain operation is rarer now, and only happens under memory pressure and on cgroup death. One side effect of doing synchronous work is that drain_all_stock holds the percpu_charge_mutex longer while it performs the work, which means chargers may be more likely to be unable to grab the mutex lock and exhaust MAX_RECLAIM_RETRIES and OOM, in theory. In practice, I have not been able to replicate this behavior in my experiments. Performance testing across single-cgroup, as well as 4-cgroup (under the 7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system shows moderate performance gains (~1%). In the tests, I repeatedly fault and release anonymous pages using madvise(MADV_DONTNEED) to stress the charge/uncharge path, across 30 trials of 50 iterations. Metric here is time it took across each iteration (ms). +----------+--------+-------+-----------+ | #cgroups | before | after | delta (%) | +----------+--------+-------+-----------+ | 1 | 357 | 350 | -1.960 | | 4 | 1221 | 1204 | -1.392 | | 32 | 9184 | 9032 | -1.682 | +----------+--------+-------+-----------+ Further testing on other stress-ng microbenchmarks also agreed with these results. v2 --> v3: - Dropped the cgroup v2 optimization, since it could indeed lead to too much time held with the cgroup_mutex. Instead we let the stock accumulate in the parent cgroups, which is not so bad; charges can still land on these cgroups, and if we ever reach the mem_cgroup limit, we can easily return those charges. - page_counter_disable_stock no longer drains, just prevents accumulating stock. The actual draining is done in the free_stock variant, where we know for sure there are no in-flight charges. - Reordering the page_counter_disable_stock path to disable before draining as to prevent accumulating stock first. - Skip isolated CPUs when draining synchronously - Rebase on newest mm-new - Wordsmithing v1 --> v2: - Dropped stock returning on uncharge to preserve same behavior as memcg stock. This resolves some race conditions present in v1. - Fixed many race conditions between disabling page_counter_stock and in-flight charges - Restructured drain_all_stock to iterate over all CPUs first before memcgs, to reduce the number of synchronous CPU work scheduling - Optimized cgroup v2 further to drain only on the first child and skip the root mem_cgroup - Dropped RFC - Wordsmithing cover letter [1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/ Joshua Hahn (6): mm/page_counter: introduce per-page_counter stock mm/page_counter: use page_counter_stock in page_counter_try_charge mm/page_counter: introduce stock drain APIs mm/memcontrol: convert memcg to use page_counter_stock mm/memcontrol: optimize memsw stock for cgroup v1 mm/memcontrol: remove unused memcg_stock code include/linux/page_counter.h | 16 ++ mm/memcontrol.c | 276 ++++++----------------------------- mm/page_counter.c | 129 +++++++++++++++- 3 files changed, 188 insertions(+), 233 deletions(-) -- 2.53.0-Meta