From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oa1-f51.google.com (mail-oa1-f51.google.com [209.85.160.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 385F838A722 for ; Tue, 23 Jun 2026 18:01:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782237688; cv=none; b=Jq1WcfkIZ8v0bsRwhhDgfyKrGYYJPYovWtBB9YANBv4+7d+KDNIkofCJahSN++KvqPoWhy5bIXvTn9WqHg9nGYB5xsANBh1ljBlnJyWOt/Y7dvRHXOaxZ778QEGFr72d/MQO9AQQIQNKnmhXuEcdzSWfxMGtSQb3396ciUPCQ0I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782237688; c=relaxed/simple; bh=JPyZk6FF6oQcAECXw3GAs0B362Mqgccf/vZbjhYIu40=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=WqdA5EdYUBWUeXYwNCE2PBFMBCGOLxPV4f8QMrqUPXTM5LhdHa8zxSFwy11NLFFP+HDVQA+NjyKQC7pUCxWfIL9Gql2j8IGoOdSZ0atHFTJVPj8QbtEgtpsxLjVISeOEz3fXLsdkzZ2n0I7THCUXiIGarIxH7HMujiW0Zl/ZTN8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=g9srL9+x; arc=none smtp.client-ip=209.85.160.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="g9srL9+x" Received: by mail-oa1-f51.google.com with SMTP id 586e51a60fabf-44755d837e8so155743fac.3 for ; Tue, 23 Jun 2026 11:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782237686; x=1782842486; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=MY1Ldz3B6HlKyVZ18RU6aMWYK7T1s/vC+KTUQ9kZi20=; b=g9srL9+x5XyBEMkEvwGTySIDxoEOYayoMXP8jirDRmmjz90HF8sCEnmbAbK/CDpWxj 3py6TXMVUgZBy3tiCnz6zULIiOfGf1ifU4zszdVS6LYVR2KuIppf07mbjAsx4okfh6gm cXxg2JzOmAnzLeQquGV7jsa/WjBs6nQTt8s2B9SsxQN4EeFV7ZWTykl1SYpO4oOJuKMo ezCRogdjHll184StOVzCyNhouO2j1EaAxZTvrX+PI2Lu7qAb83RRdl1R/eNBPQrdhngp lY0igPSyBFZqcijB+2qyC0tsYnvBW7vsuFydICBKWIBr5chMAngUB0XX1mKGZ4OOx5s/ 37Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782237686; x=1782842486; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MY1Ldz3B6HlKyVZ18RU6aMWYK7T1s/vC+KTUQ9kZi20=; b=hznZ3iAfE8wd4SrqU8mkb+MYXmADKvPwUdddEFRJwoDNpTh90Jf/VDCN6srERVpZxd 3NqMAhTnHjxXKpKeCRHm93WmZApU29GfQ1Y757IRaDiDJUiO8re7WEQtMMnJowzWYnyB EpPYfKipM2p4eV9SjAf4ySctiHxeIkyZ9Twow8kS4ZdXGwyafnaiiYXzzijoDUTsJ7dR ViZthq/Dv2kTJX8TxsqUDzvERlFsVfxcA9Zs7Sgw5lpY+PbEh49gpaoHaLA5BTmn/1L4 McseiziKjJTzZdG14NlEleM2GyzJD9Q2II9v72zjJC8JL5E85McI/79f4HWmfplQIKLL SCyQ== X-Forwarded-Encrypted: i=1; AHgh+RoX73QQZ66gdrh7Oa94LorWUVpwCth6RV+a1kdyoPRzm8IOc2M0Ahn7gh56EmG5W2PqxKJqfV9vhwyASIU=@vger.kernel.org X-Gm-Message-State: AOJu0YwJacVZFFHAvgk4+pvvyKIAnZuBEioBtH3ojFNQPoXHTPFzMBg6 XIvOEEfS6VLR7Ms14HOBtt3K1W88zOjmiao+ZB3n/to/r21fOssIFLxkcfI4qg== X-Gm-Gg: AfdE7clcvI/dDWeTUZWP8j7t8p5bQmfsb9YDZw8AeRpbv9jLQJQkePRPFTnvJCBlH/n uZYmIVwuHJsYL+Z39huECZIYiLWBO4DAsy7DpWbFYjwdBqUTggdR+bzWyeaiX50v83/12P795yM ltLZUeG5SepbB/hRvgaxg/VkgAqL9E1Eq/lN4j8QnKJnHLtmzuSNhiyvB3siWWDHqZBKBp8+JUp m013j0HJnbL/n8/P1dBcIr+nC3Ojt/dGrJXXzlQVezo7FvyxQOcK6NZ/bc4aBBTrvIV0puKRBLE Z+qPvdar7bF4dYPotne+Ze9KCTLlibXYtrAqRScandWWUvPxoasnxxAO7VKP6hehsEiWABIeLv8 m5em8fXuZne2q+ISTGg5BultU5+vfAvJ7qpWipJU80WFRIxxKPEidQzUYfnZRxuxh4LMVNq1o5E 9W1uzwvOSI/pFIUPTunhygz2kFntaN9E5OfsHsUCGcf9I= X-Received: by 2002:a05:6870:638c:b0:43d:3790:46c with SMTP id 586e51a60fabf-44707ef2b82mr16146481fac.29.1782237685727; Tue, 23 Jun 2026 11:01:25 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5a::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4472ecd3b51sm8900432fac.7.2026.06.23.11.01.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jun 2026 11:01:25 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 0/5] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter Date: Tue, 23 Jun 2026 11:01:18 -0700 Message-ID: <20260623180124.868655-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This series is intended for the next release cycle. v3 --> v4 ========= - Reduced memory footprint by 4x, from 16 bytes per-(cpu x memcg) to 4 bytes per-(cpu x memcg). Each page_counter_stock is a thin wrapper around an atomic_t. - Removed locking completely and uses atomic operations to use stock. - Removed synchronous work_on_cpu. All work is done via remote atomic_xchgs. - Added a patch to flatten page_counter charging in try_charge_memcg - Split page_counter_try_charge into stocked and non-stocked variants. INTRO ===== Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged allocations, allowing small and frequent allocations to avoid walking the expensive mem_cgroup hierarchy traversal each time. This fastpath offers real improvements, but there is room for improvement: 1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When more than 7 mem_cgroups have stock present on a single CPU, a random victim is evicted and its associated stock is drained. 2. When one cgroup runs out of memory and needs to drain stock across all CPUs it has stock cached in, those CPUs will drain all other memcgs' stock present in that CPU. This leads to inefficient stock caching and cross-memcg interference under memory pressure. 3. Stock management is tightly coupled to struct mem_cgroup, which makes it difficult to add a new page_counter to mem_cgroup and have multiple sources of stock management. This series moves the per-cpu stock down into page_counter which consolidates stock limit checking and page_counter limit checking into page_counter_try_charge_stock. This eliminates the 7 memcg-per-cpu slot limit, the random cross-memcg stock drains, and slot traversal. We also simplify memcontrol code, since we no longer need to maintain separate draining functions or manage the asynchronous workqueue. In turn, we can add independent stock management for additional page_counters in each memcg, which is used in my tiered memory limits series to add a new page_counter to track toptier usage [1]. There are a few tradeoffs, however. First, the bound on how much memory can be overcharged (and remain stale as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages. Now, it becomes nr_cpus x nr_cgroups x 64 pages. On large machines with many cgroups, this could be significant. There are 4 qualifying points: 1. Larger machines should be able to tolerate the additional overhead. 2. Stock should not remain stale as long as the cgroups are actively charging memory. 3. Getting anywhere close to this overhead is difficult and rare. It would require processes to bounce across CPUs and refill stock. 4. These charges are not "real" allocated memory, but rather accounting done in memcg; they are easily returned on pressure. Secondly, we introduce a small memory footprint. The new struct page_counter_stock is a wrapper around an atomic_t, which adds 4 bytes of overhead per-(cpu x memcg). On a 1024-CPU, 1024-memcg system, this adds 4MB of overhead. Smaller machines will see much smaller overhead. One small side effect for cgroupv1: this series decouples swap for the memory and memsw page_counters. Since stock charging can go out of sync, this means that users can transiently see memsw usage go below memory usage. Finally, by moving from asynchronous workqueue scheduling for draining to synchronous atomic_xchg, drain_all_stock holds the percpu_charge_mutex longer while it performs the work. This means that chargers may be more likely to be unable to grab the mutex lock and exhaust MAX_RECLAIM_RETRIES and OOM, in theory. In practice, I have not been able to replicate this behavior in my experiments. The series was built on top of latest akpm/mm-new as of Jun 23 2026, which is cdad4d4e4fc2e "mm/swap, PM: hibernate: atomically replace hibernation pin". TESTING ======= I tried to demonstrate the worst-case overhead this series introduces by writing a microbenchmark that pins multiple cgroup jobs to a single CPU and repeatedly faulting and releasing anon pages using madv(MADV_DONTNEED) in each cgroup. The data was collected over 30 trials of 15 iterations. Metric here is time each iteration took (ms) +----------+--------+-------+-----------+--------------+ | #cgroups | before | after | delta (%) | variance (%) | +----------+--------+-------+-----------+--------------+ | 1 | 112 | 112 | 0.000 | 1.1 | | 4 | 443 | 451 | +1.806 | 1.1 | | 32 | 3512 | 3584 | +2.051 | 2.0 | +----------+--------+-------+-----------+--------------+ It appears as though there is some small regression, although the magnitude is similar to the coefficient of variation (stddev / mean). CHANGELOG ========= v2 --> v3: - Dropped the cgroup v2 optimization, since it could indeed lead to too much time held with the cgroup_mutex. Instead we let the stock accumulate in the parent cgroups, which is not so bad; charges can still land on these cgroups, and if we ever reach the mem_cgroup limit, we can easily return those charges. - page_counter_disable_stock no longer drains, just prevents accumulating stock. The actual draining is done in the free_stock variant, where we know for sure there are no in-flight charges. - Reordering the page_counter_disable_stock path to disable before draining as to prevent accumulating stock first. - Skip isolated CPUs when draining synchronously - Rebase on newest mm-new - Wordsmithing v1 --> v2: - Dropped stock returning on uncharge to preserve same behavior as memcg stock. This resolves some race conditions present in v1. - Fixed many race conditions between disabling page_counter_stock and in-flight charges - Restructured drain_all_stock to iterate over all CPUs first before memcgs, to reduce the number of synchronous CPU work scheduling - Optimized cgroup v2 further to drain only on the first child and skip the root mem_cgroup - Dropped RFC - Wordsmithing cover letter [1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/ Joshua Hahn (5): mm/page_counter: introduce per-page_counter stock mm/memcontrol: flatten try_charge_memcg control flow mm/page_counter: introduce page_counter_try_charge_stock() mm/memcontrol: convert memcg to use page_counter_stock mm/memcontrol: remove unused memcg_stock code include/linux/page_counter.h | 19 +++ mm/memcontrol.c | 280 ++++++----------------------------- mm/page_counter.c | 90 +++++++++++ 3 files changed, 155 insertions(+), 234 deletions(-) -- 2.53.0-Meta