From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com [209.85.210.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E88252036E9 for ; Mon, 25 May 2026 19:04:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779735900; cv=none; b=HmohT9Ecz1RZ4oY3Iunak0EdsZQVZ8vLrGXLPCfdTO6oH50SCZh0upRfQbzyzAxYG+8S0aCuo5/R8etlwi6hDWeqYhPzr/8OoAWmp4odP6hcGDmfX3w5V5Q8AlQycZ2Yr1z3VYhEe5JWu8b2TkwhaBXuE8pS7fRILm0xNdMwoH4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779735900; c=relaxed/simple; bh=l5R1YGJHMK2ztHlqCD/O23AFreNTdT9N0zjySpshk+Q=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=KXK0RahfylidSrNrQ13G9Wx/k78ot4pKNZp/GJC7NgIftubOmNTHX/iJzExynpZF13eo4T+Zkjlrl+DTCmnr4g7o5VwchYFiqOmoT6EoK7crGKvNLkDioxpXoZy/fDuXLCVEPbJmxiXIXjFpZa10SwYApsoZ2/VzBJn0XgRb8N8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZjCYtrwY; arc=none smtp.client-ip=209.85.210.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZjCYtrwY" Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-7e568ab0bc5so10684487a34.0 for ; Mon, 25 May 2026 12:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779735897; x=1780340697; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ZsiSNZGlI52E7JWBNAgrZcZTSVxRjmA7Low1cc7dass=; b=ZjCYtrwYQxoxEJPMADr+CNEU/kpi9OAEhuQ2w5oVFQF4dzYeQgSRkNmkR6WzIIbd2+ Xng+OoOS1YjG995k6WGKem6/kg+QPC94+NsWqFv+YaRankgRYM6aYAU4kqtFwEZOEguP ylVLI5catZp2NWxdJjPs3lorZplfXKKQzczo4c7jt+hY4+YJ/N1QwSmYmeS3oyVso8di CWIWVa2ZzRy+l963eYGfPf+QPyFB0hCIN+cHkkYnP2ri5Pmlb/ZFDl5o2PW7ePJI2Hs1 acnZvszxvJPTrXqX04cmvxI7Ud5JFbEs1JARDKsqRW2dMrLoczpmKVyWDF0rc0vHyolu WY5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779735897; x=1780340697; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZsiSNZGlI52E7JWBNAgrZcZTSVxRjmA7Low1cc7dass=; b=LRewO30ErPEvrf9x2lo71l9BPaMWC049tlZAbEcJec/DyskIbSke7ZooF1pjk446tV AjHDwPIHzxgVsUj/qG0LmQT59ERrjgu1ux+//vq4KuQ6RpJUUwjerr5LuHRUbrv4cJSC tv6mxCbRjywhgZdTvaS7YXaM0oUQRXAQC3WWP0ynP10BCVJOZu0aDnZmegvfPVKhHEv/ 5cWVzUf18ZE5eQiE3Nb8PY9QED46mD6AdJG5foIFOxtsXUryCbE+H4qxqyYu3B4mNhiw LdpyGUZCqmRs81YAFyXKnV04QcEUt/rKln2LvNVhc7HuMNJjigxmi/lzo+JOGPXHJGh9 7v1w== X-Forwarded-Encrypted: i=1; AFNElJ811h7RfBXBDxp3qrlzo9on1O1O+rMT0L2Hl5Q0mNpJG6ZEt3WdqjdcItAzw3mlIzwpPg6YesJKyFp3Uwg=@vger.kernel.org X-Gm-Message-State: AOJu0YxnkFxjGB/xEk7TUbmVMDOlPgnZn0osVIHGHciSeb81MMaXafGj TtAZv7F2oIKeqZLTBhpHXrdPd7mg2ZB/IJLKAB+7lzI4JZLcnDhT0o+6 X-Gm-Gg: Acq92OG8jxA+SMdkekeNcItTsxCK/a1RSZQUF+mfspancfa9nk6opvBgohBdl7wRFo8 bLUCFcaPfGKermWZKpuVcT2osAK24bzVUR9rohFlBYEqTxPFAmz11Qe2aywsEeR6PUH2K9Xb3WT 9PDLxMnLlTw+2ssco+iEMxWMBMR25+vn8ZhW4dGGIEmyOx3XBBpPHH1mrHHcHA1N6ySAhzTBxtQ dIXVtnJGb84oHbv5C1hQ8wVo3VF/0DAMK6f2h73O0ATn1gX2cqVIZpmh+EGjgAaPF1dTx3dqqAt y+OWmPnviiikbFm/fZShBNXbU/+qaurK+PRf3PRkeLZVJ16aDEK3nlAq9zGoatT4UclhGunbH2O Ghi3f74/NDc2Fp+uUvsti9R0XcKPYU/QBI1Jh2RnVfxLfV52kuV4BznSGUM6VtqZ7etjQUlu9hP xyVdP+sATH130hNHdqbL5o8+7vmquxgTOoubM31Uf83hLVp6wTbqEu7w== X-Received: by 2002:a05:6830:41:b0:7e6:50c4:e954 with SMTP id 46e09a7af769-7e650c4ff36mr836626a34.11.1779735896872; Mon, 25 May 2026 12:04:56 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5d::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6060b2dffsm7751933a34.0.2026.05.25.12.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2026 12:04:56 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 0/7 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter Date: Mon, 25 May 2026 12:04:47 -0700 Message-ID: <20260525190455.2843786-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged allocations, allowing small allocations to avoid walking the expensive mem_cgroup hierarchy traversal and atomic operations on each charge. This design introduces a fastpath, but there is room for improvement: 1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When more than 7 mem_cgroups are actively charging on a single CPU, a random victim is evicted and its associated stock is drained. 2. Stock management is tightly coupled to struct mem_cgroup, which makes it difficult to add a new page_counter to mem_cgroup and have multiple sources of stock management, which is required when trying to introduce fastpaths to multiple hard limit checks. This series moves the per-cpu stock down into the page_counter which consolidates stock limit checking and page_counter limit checking into page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit, the random evictions (drain & refill), and slot traversal. In turn, we can add independent stock management for additional page_counters in each memcg, which is used in my tiered memory limits series to add a new page_counter to track toptier usage [1]. The resulting code in memcg is also easier to follow, as the caching becomes transparent from memcg's perspective and managed entirely within page_counter. There are, however, a few tradeoffs. First, the bound on how much memory can be overcharged (and remain stale as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages. Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines with many cgroups, this could be significant. There are three qualifying points: (1) larger machines should be able to tolerate the additional overhead, (2) the stock should not remain stale as long as the cgroups are actively charging memory, and (3) a process would have to migrate across all CPUs to incur this upper bound on overhead. Secondly, we introduce some additional memory footprint. The new struct page_counter_stock adds 2 words of extra overhead per-(cpu x memcg). A small change is that for cgroupv1, reported memsw usage can be lower than reported memory usage, if the memsw page_counter overcharges to its stock whereas the memory page_counter does not. Finally, to keep the above memory footprint limited, I opted to not embed a work_struct into page_counter_stock, but rather decided to trigger synchronous stock draining, since the drain operation is rarer now, and only happens under memory pressure and on cgroup death. Performance testing across single-cgroup, as well as 4-cgroup (under the 7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system shows negligible performance differences. In the tests, I repeatedly fault and release anonymous pages using madvise(MADV_DONTNEED) to stress the charge/uncharge path, across 40 trials of 50 iterations. Metric here is time it took across each iteration (ms). There are two testing versions below; the only difference is that v3 is based on top of mm-new, and v2 is based on top of mm-stable. The "after" on both sides are similar, but mm-new and mm-stable have different perforamnces. v3, tested against mm-new +----------+--------+-------+-----------+ | #cgroups | mm-new | after | delta (%) | +----------+--------+-------+-----------+ | 1 | 357 | 358 | +0.283 | | 4 | 1245 | 1214 | -2.430 | | 32 | 9281 | 8970 | -3.470 | +----------+--------+-------+-----------+ v2, tested against mm-stable +----------+-----------+-------+-----------+ | #cgroups | mm-stable | after | delta (%) | +----------+-----------+-------+-----------+ | 1 | 352 | 353 | +0.283 | | 4 | 1198 | 1217 | +1.585 | | 32 | 8980 | 9027 | +0.526 | +----------+-----------+-------+-----------+ Further testing on other stress-ng microbenchmarks also agreed with these results. v2 --> v3: - Rebased on top of latest mm-new, May 25, 2026, since the previous version could not be applied for Sashiko review. - Re-ran test numbers v1 --> v2: - Dropped stock returning on uncharge to preserve same behavior as memcg stock. This resolves some race conditions present in v1. - Fixed many race conditions between disabling page_counter_stock and in-flight charges - Restructured drain_all_stock to iterate over all CPUs first before memcgs, to reduce the number of synchronous CPU work scheduling - Optimized cgroup v2 further to drain only on the first child and skip the root mem_cgroup - Dropped RFC - Wordsmithing cover letter [1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/ Joshua Hahn (7): mm/page_counter: introduce per-page_counter stock mm/page_counter: use page_counter_stock in page_counter_try_charge mm/page_counter: introduce stock drain APIs mm/memcontrol: convert memcg to use page_counter_stock mm/memcontrol: optimize memsw stock for cgroup v1 mm/memcontrol: optimize stock usage for cgroup v2 mm/memcontrol: remove unused memcg_stock code include/linux/page_counter.h | 16 ++ mm/memcontrol.c | 289 +++++++---------------------------- mm/page_counter.c | 140 ++++++++++++++++- 3 files changed, 212 insertions(+), 233 deletions(-) -- 2.53.0-Meta