From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77D7D368277 for ; Fri, 22 May 2026 22:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487591; cv=none; b=YvUYiOzBnIy7ozQtRsNKZhqLkkTiOOA33edNJbUrxltCSWVluxV7Y4/86ubfuGaXvUcpF3mMeRzso9jVxAk23aJK7WA/FVOuJHiBQLsDWpE4mt7l6GtYvZLbjI2AwWit0JsIJZ+8R87AxCqby69iqFOPeOZJPcIVRcGugm/1w18= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487591; c=relaxed/simple; bh=962U/sFP2aV6aoQ/n5p22srRp1SFYr/o1PwxhK7nhK4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=jIX6nvdk6uOTSfd5vPzWHkcH+s8eRhF8JVSs4/wABL3DGM9clg8LEYohJwXoTNU5uUNbVbOPigwOIP2VUqiA2WoGeGUS49NDAV/TSJCU1TzKCmWf2GE40+1xvet2ihbF0NS09tFiAiEdcrGlr7toiITjZ9Hg4IGA1oI4cPc/buE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=qxP9KSXx; arc=none smtp.client-ip=209.85.161.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qxP9KSXx" Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-69d7e72b052so1003515eaf.2 for ; Fri, 22 May 2026 15:06:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779487589; x=1780092389; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=aitoRh8af8hU9OmkzJFXm6Hu0+SgnlMA/xdSbDz8UH4=; b=qxP9KSXxDRnPcNQSH/kFvPK/t82QLOLT3U84lEN2Qn8U8NFp0hsI7ZnNmGv/J6jxNy qqI9F36q6apPED1ErFvfLXSvT+6tP8oyvhXVmpwqs3+vS847Iqqz5qGT3gCbqgcxJyzR WMqdtoXRexBG6Zm9GRAyUAhXcXezOSk6tqWAgHpnnVtbWs0nvJ+UiYPbwSZwPl4s0B8s Kn5QQLxi64eAf5/N3+m7JUOAUAzmLFs7yG3+yoF6njYqnaD4e1XpexFSh1pUuy4su6rN Qhbc5XZja9fnsGIwWtMeF0BBDNl+5fuBVnmQEdGm11LS4xOQzAcq08enVOALDangNWn7 i8mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779487589; x=1780092389; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=aitoRh8af8hU9OmkzJFXm6Hu0+SgnlMA/xdSbDz8UH4=; b=tBtNjxF7+/VOj+2qDcC3pbvNnLYWiyTG20XLpLV3suFJshHy9PCKkPL/eeXSo+3UYe 0IoOgFOc+qAfFjZvZY6pDkg/cF6kuh/jbDq/jQjFBkxgXsQeDtKjUJ2kAx3m34Z8Swby 9Czd64+njVFmj11kkAtJpMyIAQSDqw2HG9GRfMehK/lX7DXyqs5l1g1M4GrT3KhE55zk qgvqcGdFNAt0szREI8h34n5Xrqt/aTNn+4MIDUvY0X5fkpPwuwu7Iuck7zJNmV1AdIaf KOzpPddwMCSYCmYoZYy1rUVNbTLj4pdvWhOmdoQ3QgBiScAP7wgP2i6Bcgy63O7cEB3O 5TCw== X-Forwarded-Encrypted: i=1; AFNElJ/+kwk94wwbLULFAdoWg4bh4116gtq+cbuIVUDoT18VwshOFX003IKZ1pQgE6IyTXb8wKnGq692DFm5xZw=@vger.kernel.org X-Gm-Message-State: AOJu0YzworxbwZO04YXuZgDsVo0nLNSSKDMLP904VRFnUFarif7NewHY 7DZiVqbtWE7NIRvlGGu/0DAm28mmVZui6iTz+K8eDcSpQA73anHELnZt X-Gm-Gg: Acq92OErJE7JxUIr4YCIWJAMM/Z+3GJFSEGtpTYzMAN2K9rXDR5GshPl01T+Dc4wJVX sgmDOJoG3np5xIPXPohGzk7rfk/mwDIrCo4RMWUIlmuRomqBc9lm11asZENhrMGN52thhb/bNRA vD9ME3TpRMKMl1oSwQ+xJ/FvlOL0pW+nkThvTw/+4/r6UKe6ra5hDLLGb86Ozcgy+4lf7T5WaUf nDYvJcl8vmhRLLCMQJGBSeq21e1Bs/RCP7Y2FsP3RXXDTMjTm2qePbxcwOjX9gH0i154D4EwTV8 hPr0HMEKS5YPkGsPqPFiBwmJCFCi4CrSmsbNhEHXxnYwKCtm67gAOT1hF4v1aZKAGCVmuzoBgFo aKC3gPSnldW1MjeQZW+lnV2Z88w8amRcDXrdTL6QCWGPt/bsAwy0EgIdEo9whE+akc9ka6EBu/H BPRIBwyIQW4e+3d2/l8Tyq4Nzeo89clGH0xOOwzQplgHiC+ary3/dlQQ== X-Received: by 2002:a4a:d813:0:b0:69d:7bdb:ade2 with SMTP id 006d021491bc7-69d7ec28651mr2091887eaf.32.1779487589309; Fri, 22 May 2026 15:06:29 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:56::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-43b635e373fsm2879308fac.8.2026.05.22.15.06.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2026 15:06:28 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 0/7 v2] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter Date: Fri, 22 May 2026 15:06:18 -0700 Message-ID: <20260522220627.1150804-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged allocations, allowing small allocations to avoid walking the expensive mem_cgroup hierarchy traversal and atomic operations on each charge. This design introduces a fastpath, but there is room for improvement: 1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When more than 7 mem_cgroups are actively charging on a single CPU, a random victim is evicted and its associated stock is drained. 2. Stock management is tightly coupled to struct mem_cgroup, which makes it difficult to add a new page_counter to mem_cgroup and have multiple sources of stock management, which is required when trying to introduce fastpaths to multiple hard limit checks. This series moves the per-cpu stock down into the page_counter which consolidates stock limit checking and page_counter limit checking into page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit, the random evictions (drain & refill), and slot traversal. In turn, we can add independent stock management for additional page_counters in each memcg, which is used in my tiered memory limits series to add a new page_counter to track toptier usage [1]. The resulting code in memcg is also easier to follow, as the caching becomes transparent from memcg's perspective and managed entirely within page_counter. There are, however, a few tradeoffs. First, the bound on how much memory can be overcharged (and remain stale as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages. Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines with many cgroups, this could be significant. There are three qualifying points: (1) larger machines should be able to tolerate the additional overhead, (2) the stock should not remain stale as long as the cgroups are actively charging memory, and (3) a process would have to migrate across all CPUs to incur this upper bound on overhead. Secondly, we introduce some additional memory footprint. The new struct page_counter_stock adds 2 words of extra overhead per-(cpu x memcg). A small change is that for cgroupv1, reported memsw usage can be lower than reported memory usage, if the memsw page_counter overcharges to its stock whereas the memory page_counter does not. Finally, to keep the above memory footprint limited, I opted to not embed a work_struct into page_counter_stock, but rather decided to trigger synchronous stock draining, since the drain operation is rarer now, and only happens under memory pressure and on cgroup death. Performance testing across single-cgroup, as well as 4-cgroup (under the 7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system shows negligible performance differences. In the tests, I repeatedly fault and release anonymous pages using madvise(MADV_DONTNEED) to stress the charge/uncharge path, across 30 trials of 50 iterations. Metric here is time it took across each iteration (ms). +----------+--------+-------+-----------+ | #cgroups | before | after | delta (%) | +----------+--------+-------+-----------+ | 1 | 352 | 353 | +0.283 | | 4 | 1198 | 1217 | +1.585 | | 32 | 8980 | 9027 | +0.526 | +----------+--------+-------+-----------+ Further testing on other stress-ng microbenchmarks also agreed with these results. v1 --> v2: - Dropped stock returning on uncharge to preserve same behavior as memcg stock. This resolves some race conditions present in v1. - Fixed many race conditions between disabling page_counter_stock and in-flight charges - Restructured drain_all_stock to iterate over all CPUs first before memcgs, to reduce the number of synchronous CPU work scheduling - Optimized cgroup v2 further to drain only on the first child and skip the root mem_cgroup - Dropped RFC - Wordsmithing cover letter [1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/ Joshua Hahn (7): mm/page_counter: introduce per-page_counter stock mm/page_counter: use page_counter_stock in page_counter_try_charge mm/page_counter: introduce stock drain APIs mm/memcontrol: convert memcg to use page_counter_stock mm/memcontrol: optimize memsw stock for cgroup v1 mm/memcontrol: optimize stock usage for cgroup v2 mm/memcontrol: remove unused memcg_stock code include/linux/page_counter.h | 16 ++ mm/memcontrol.c | 286 ++++++++--------------------------- mm/page_counter.c | 140 ++++++++++++++++- 3 files changed, 212 insertions(+), 230 deletions(-) -- 2.53.0-Meta