From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 309FF3446A7 for ; Fri, 10 Apr 2026 21:07:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775855267; cv=none; b=AdEWd0jHsE8YITctWC64kshwFmJG9uC+d89poAnZkiiKoSwMNVczxOOgGtm1MTI2VlveDdufspu0pH0jXARBVyDEPBPro26o26MNQgDb1C6BG545r+I75DcEqAIzIIByUpqaQmJHgxS1E1205wbbtmTn7gOmL+tSHCDjH/5A1hk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775855267; c=relaxed/simple; bh=1f/+Wg65Wzydtq/ARwlhRUIsAN+OxUJYfdFnUPasWW4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=X0dYJAp6QjiU3dTYif9LuPW7yyb7yi6CPT3iRLfYRW2ZR1uBaIXX1fWuW948yp97PmgzvmnUWMUt0sJkWp1HXASFOPz45ePv9Bk7Qwp2cw7TIC+8s8Lk4lX/Sm5MmgPkaNOPwjMSjiNmXnGsQZ7EpUrNvZO3Sn6GbYhXCFGwkk0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PSDCZNJ2; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PSDCZNJ2" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-7dbd8c6fc84so1539461a34.3 for ; Fri, 10 Apr 2026 14:07:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775855265; x=1776460065; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=OMjyo1BeE8Sjl3rJFGc/7waGNwkRpFV9y7vNe0a0k/I=; b=PSDCZNJ2OUPBOYMCX5rD2CXKwEHDZG9PuNlz32PV1lny6GGhrdvz0Y8pfPNI5Mio5q IvgLmWeM7zZX1bhlqej6ykZOFlAkowhkgPjXY5sVaG87ZMhNc69SpRlu669Ee8eaicLn vBlqxdQRseprprqPTVWOnLb23nCidsXIluNgtiuILpbCrh/FEvX04o/FVqIizLP5KXcX STI5TA3rHPYJYuv/4HU5vXgcK7WW0zBIjW7GzEdlNFUJtsiuF2sHjTsec/jucHQOlaZz iZkQqfM4NeawGFHSW7taJhHILTzTMHcTJzjIb9k5a4dO9crtUPFgyXvAmDAV3vX6N/Jk 1DsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775855265; x=1776460065; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OMjyo1BeE8Sjl3rJFGc/7waGNwkRpFV9y7vNe0a0k/I=; b=K2HnxfNWuxIvAPxREVdj87an48rHv61bBUyd+f4wDUI2/gs/Xub8J9qTECctqRS0tF 7+mv2nShrdzdD8oj3W4VOkpZB4rXl2l4Mx6oi/5jhDBfwCWCx/YVd7iMWDJdxjawoeLK GlN9oRW3ZsWN6kyaT/6NYOzE6WKsbbZQMyQ9Y4gZmiTU9lZlkbtG/op5yYWXn2Ji6Xsy bEfPo6E1AsGz5lbr5Qhi65P/GWISwib1DRYgfzCheDOnhqKLoyINEOfZGBT8GAWd0Isd qWfPgD/HuMgh/EeJPFtkV304IJdnxO6hJNTYfPrsWlOj1W8quIJ66yOE5enTT0I3iXKq XRDA== X-Forwarded-Encrypted: i=1; AJvYcCUsxvcOmQetITbHkBdA7Jsj4K5TrjzvzDoFQfQuYfG42RyhXO26nBOI0vcb5cJCrPDbxmxY7FYGaY5rFAY=@vger.kernel.org X-Gm-Message-State: AOJu0Yyt/xzuCZcGaFNHHvZrqfpQDlXkgR3DtY3NlJPo+ZX9JFZikBuV +jzsvbPH64/0cscS2inlIKZDtm3NnjtfkxPBx8iwGJkpncHWJjFnrRtS X-Gm-Gg: AeBDievx/KEM6m9oVno1J/1dv4hvT07zTjVfkOhN63t89QOWzaYC4Gef1ANcDJgg00P jP3RjsjjGe79LLJyRO6tzEPWV18v2vV72TD7lby/Li07/fKGVtb9yZ4hH6tuZguJlnalzlQ0MDS NS9bJQJETCGGEzYK24/kzwkfzgmSy2rzEUYDjlD/a10xrPvunRUG8QqSo6p8eycYMYhRJHA6YdI lDn8UnWy4hcBwEjl/aA6Yn9/efQ8LxPC5ojQc0rsfcmhfdxLRF2UsQ0HrlGpRHH4UOvMSO660e9 +eR1o1axJt5q/484+fGqyLAlSHBx8doNInS2taMiJWuuyxdxvHt8rMdz2PPhjEgKxaZGx24tWXo GQ6CEmUfRzTRBiO321bdkL10PVRTJ7oFZ4y1DTFSV0BD8Z6fuR3b9TTCDOo/m6EUvRF2AzFICLx FV+BihYjeNxe5sPPg13MJBKA== X-Received: by 2002:a05:6830:6b06:b0:7d7:4fc7:21a with SMTP id 46e09a7af769-7dc27ceee0cmr3297522a34.13.1775855265063; Fri, 10 Apr 2026 14:07:45 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:42::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7dc2660ef75sm2708244a34.10.2026.04.10.14.07.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:07:44 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner Cc: Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 0/8 RFC] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter Date: Fri, 10 Apr 2026 14:06:54 -0700 Message-ID: <20260410210742.550489-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged allocations, allowing small allocations and frees to avoid walking the expensive mem_cgroup hierarchy traversal on each charge. This design introduces a fastpath to charge/uncharge, but has several limitations: 1. Each CPU can track up to 7 (NR_MEMCG_STOCK) mem_cgroups. When more than 7 mem_cgroups are actively charging on a single CPU, a random victim is evicted, and its associated stock is drained, which triggers unnecessary hierarchy walks. Note that previously there used to be a 1-1 mapping between CPU and memcg stock; it was bumped up to 7 in f735eebe55f8f ("multi-memcg percpu charge cache") because it was observed that stock would frequently get flushed and refilled. 2. Stock management is tightly coupled to struct mem_cgroup, which makes it difficult to add a new page_counter to struct mem_cgroup and do its own stock management, since each operation has to be duplicated. 3. Each stock slot requires a css reference, as well as a traversal overhead on every stock operation to check which cpu-memcg we are trying to consume stock for. This series moves the per-cpu stock down into the page_counter, which consolidates stock limit checking and page_counter limit checking into page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit, the random evictions (drain & refill), slot traversal, and css refcounting. In addition, it makes independent stock management scalable for future users. As a demonstration, this series also introduces independent stock management for the cgroup v1 memsw page_counter, which curbs the likelihood of the worst-case scenario (traversing both the memsw and memory page_counter hierarchies). One change that should be noted is that draining is simplified to use work_on_cpu() for synchronous remote CPU drain. This eliminates the need for backpointers and embedded work_structs in the per-cpu stock struct, which minimizes memory overhead. This change over the existing async drain scheduling was done since the drain operation is much more rare now, only happening under memory pressure and on cgroup death (as opposed to the previous arbitrary scenario where more than 7 memcgs are charging to a CPU). Performance testing across single-cgroup, as well as 4-cgroup (under the 7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system shows negligible performance differences. In the tests, I repeatedly fault and release anonymous pages using madvise(MADV_DONTNEED) to stress the charge/uncharge path, across 30 trials of 50 iterations. Metric here is time it took across each iteration (ms). +----------+--------+-------+--------+-----------+ | #cgroups | before | after | stddev | delta (%) | +----------+--------+-------+--------+-----------+ | 1 | 446 | 441 | 5.097 | -1.195 | | 4 | 1832 | 1822 | 11.897 | -0.582 | | 32 | 14730 | 14739 | 54.089 | 0.061 | +----------+--------+-------+--------+-----------+ Signed-off-by: Joshua Hahn Joshua Hahn (8): mm/page_counter: introduce per-page_counter stock mm/page_counter: use page_counter_stock in page_counter_try_charge mm/page_counter: use page_counter_stock in page_counter_uncharge mm/page_counter: introduce stock drain APIs mm/memcontrol: convert memcg to use page_counter_stock mm/memcontrol: optimize memsw stock for cgroup v1 mm/memcontrol: optimize stock usage for cgroup v2 mm/memcontrol: remove unused memcg_stock code include/linux/page_counter.h | 15 ++ mm/memcontrol.c | 269 ++++++----------------------------- mm/page_counter.c | 173 +++++++++++++++++++++- 3 files changed, 224 insertions(+), 233 deletions(-) -- 2.52.0