From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1D9936C9F0 for ; Fri, 5 Jun 2026 15:36:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673776; cv=none; b=fVl0wgBXP6qW828y5oBEjNGdfz5UjOtFG1ReFhHg38dP0MmbqsydikdwhVHe1CiDvhRjN0pPUG+ir7j7bUA9wKN7XoFqqg306lWYYxrfzZG6/hak79aa2+HEJ6ib8aXf7KFshKotD0mqq9mkM7eySBR5s93nR/acj1SY6pk2SzA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673776; c=relaxed/simple; bh=YARM61p7OCvIlSHd3e+h/5q9hPtEiViLCkKf0hA0Dwg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tvSUTdaqaIufYe0u+lZA5aYXXeVeadunymM20rLQh25kN3iM4Rm0+O+o/BcYx5mDsMwE5eC3RswcU9nj+6ArTnFZHhAsL6F/57h9SSrzKoa0XVFWpAgXHOx7tGvkVolEVGHIeOmKsVyQL2/Sp2jMHrFmPJdhIvkgFa4Ky9Px7I4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BlIN9Kd8; arc=none smtp.client-ip=209.85.160.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BlIN9Kd8" Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-440df1c768bso1578424fac.1 for ; Fri, 05 Jun 2026 08:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780673770; x=1781278570; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QT7NBEIym2G0kzWniqTa/IOK3LykjdeMwTsMHCm4gYI=; b=BlIN9Kd8V0sZyE9w/u/W+jKR2KgCw1LX4b5olG3fsLhhu4oewo7H0geesRynJarwSW 1NM96ncEt1xtD0KMpOmsS3adN9DbpKSt9CDo9C09RAt6Rx9t3Aa0e/6hQFcWDGrJ7evL J1JAcrIkyk4ZdaclTbtujjljsYhpCotxaRnOvJ+l7aNjFd/PCazWhTWz01ftcQ5Eh9pd DtMXKFJuKC5wecIbG+WLndJC3XPF9b3FslZ5K9IDl6WtP4MngA941hsc69EGb2CYtCO/ vS3MhRwVMcA5Spfr1uzsQyYrfAtm0LKg8lECzUUELBZhdeh11KpP4jl7f7Y30m8v0m8C 6Ebw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780673770; x=1781278570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=QT7NBEIym2G0kzWniqTa/IOK3LykjdeMwTsMHCm4gYI=; b=LQ6Bg9mfm9rTZqYZ7w8KUdC9pHdKjBRzDFE7/LZrGyspr9m0hFR1UG5rZSlm5wqX8e tdP3nz+tTzUaPucS1YkXTq1gH/gigAJgiGyEE1WqtPURiZRaqMzPzF2tlvXfVZhczrzZ hS0rxg7/u+jbvBZM0KoStpfLuJj9nWKWKRWSELjM0o9RMcYS2cAZZMy71wbb6OFPbg9S Z1t3vTLuL2IlipE+4sjMuY/ZIZI2jnGIQlz/A4CGTSQhSgQr/GvGZ9kArJY0Lfu1C8y/ Mp4l2FZt/p6JQRjsZZoGRq29xub2i7qRIA/frzFP6nCeatrYjmSBOsuEN98DA1CqCV+X O5tg== X-Forwarded-Encrypted: i=1; AFNElJ8JJKzunjS86Bu9AGQk1PrP13lUyDKBf4x3vLgSK6fMrVRtv38avrzmO8tO5NLyotEeCyf25X3QNuNkkA0=@vger.kernel.org X-Gm-Message-State: AOJu0YwdD1CxLsI04hDPEtSnWD9NmrLm0vKqQ6vpn8D4WtceZqu9bh/s zcJn5f24MG/rGWLOfI7PImz5vXZ+cAnHeNa93heH5VfB3z6KC16inyp3 X-Gm-Gg: Acq92OFqZNRXj3dAXCsl+iRgpMH8Wi/i1PElVDBpQ6EiqoSVGHH6vAhZtsCrpwZYWce 79gkOg8dxFdr4TD4AKRJQjs8w6HTbR1rhsi1ACUik10R83aQydT9kFekGOrA6V0ZV9D28b1V8QK Xp2anT3XDlYLje0QVBwwFrKxaDQGd9lqKLSRYJJ03Ak4MIUcO4Qm1VKreYmUWWdvUo2DYywLB9i 1njSNDqJQ6f0MG1ja5xFhXJTG2HnqPSt+dNaNBbmFMwiGBC7/yxvzC9ieRry41KhZ17nPOHryhI 7n85AwCBKQ0Yt2WHRsLADTdZgvp+18GZ23uxcgEfcnWVfZzXuDJpUWxcRFA26avjC92bMWJoNcX NlTTZ6iXTsru4ZUJWb39wKtmrVuyLUpJ3SidGDBWohiP7wpfy+4Qfw0OPJi4k8mb6UIRTem/YIb 5tAViFMtxWCzg4xzbGHIULG8QR/D19CiXBwLKhMDJrHxqqijjl+3AMUsX2pEUxK6rB X-Received: by 2002:a05:6870:6488:b0:430:b01:1f79 with SMTP id 586e51a60fabf-4413d248508mr2239934fac.2.1780673769704; Fri, 05 Jun 2026 08:36:09 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:54::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-440d8295b38sm8359180fac.10.2026.06.05.08.36.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2026 08:36:09 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 4/6 v3] mm/memcontrol: convert memcg to use page_counter_stock Date: Fri, 5 Jun 2026 08:36:00 -0700 Message-ID: <20260605153603.234296-5-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com> References: <20260605153603.234296-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Now with all of the memcg_stock handling logic replicated in page_counter_stock, switch memcg to use the page_counter_stock. There are a few details that have changed: First, the old special-casing for the !allow_spinning check to avoid refilling and flushing of the old stock is removed. This special casing was important previously, because refilling the stock could do a lot of extra work by evicting one of 7 random victim memcgs in the percpu memcg_stock slots. In the new per-counter design, refilling stock just adds pages to the counter's own local cache without affecting other memcgs, so the original reason for the special case no longer applies. Also, we can now fail during page_counter_alloc_stock(), if there is not enough memory to allocate a percpu page_counter_stock. This failure is rare and nonfatal; the system can continue to operate, with the page counter working without stock and falling back to walking the hierarchy. Finally, drain_all_stock is restructured to iterate CPUs in the outer loop (rather than memcgs) to be able to schedule draining all memcgs via a single work_on_cpu call. It reduces the number of synchronous per-CPU work calls from O(memcgs * CPUs) to just O(CPUs). We also skip isolated CPUs, as schedule_drain_work() did before. We don't need its guard(rcu) here though; that rcu section existed to order async work scheduling against cpumask updates and workqueue flushes, which would lead to drain work pending. Since all work here is synchronous, we don't leave any work behind. Note that obj_stock remains untouched by these changes, and that memsw stock will be handled in the next patch. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 80 ++++++++++++++++++++++--------------------------- 1 file changed, 36 insertions(+), 44 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 56cd4af082326..562ed9301f5a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2269,6 +2269,17 @@ static void schedule_drain_work(int cpu, struct work_struct *work) queue_work_on(cpu, memcg_wq, work); } +static long drain_stock_on_cpu(void *arg) +{ + struct mem_cgroup *root_memcg = arg; + struct mem_cgroup *memcg; + + for_each_mem_cgroup_tree(memcg, root_memcg) + page_counter_drain_stock_local(&memcg->memory); + + return 0; +} + /* * Drains all per-CPU charge caches for given root_memcg resp. subtree * of the hierarchy under it. @@ -2280,28 +2291,18 @@ void drain_all_stock(struct mem_cgroup *root_memcg) /* If someone's already draining, avoid adding running more workers. */ if (!mutex_trylock(&percpu_charge_mutex)) return; - /* - * Notify other cpus that system-wide "drain" is running - * We do not care about races with the cpu hotplug because cpu down - * as well as workers from this path always operate on the local - * per-cpu data. CPU up doesn't touch memcg_stock at all. - */ + + for_each_online_cpu(cpu) { + if (!cpu_is_isolated(cpu)) + work_on_cpu(cpu, drain_stock_on_cpu, root_memcg); + } + + /* Drain obj_stock on all online CPUs */ migrate_disable(); curcpu = smp_processor_id(); for_each_online_cpu(cpu) { - struct memcg_stock_pcp *memcg_st = &per_cpu(memcg_stock, cpu); struct obj_stock_pcp *obj_st = &per_cpu(obj_stock, cpu); - if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) && - is_memcg_drain_needed(memcg_st, root_memcg) && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, - &memcg_st->flags)) { - if (cpu == curcpu) - drain_local_memcg_stock(&memcg_st->work); - else - schedule_drain_work(cpu, &memcg_st->work); - } - if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) && obj_stock_flush_required(obj_st, root_memcg) && !test_and_set_bit(FLUSHING_CACHED_CHARGE, @@ -2318,9 +2319,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { + struct mem_cgroup *memcg; + /* no need for the local lock */ drain_obj_stock(&per_cpu(obj_stock, cpu)); - drain_stock_fully(&per_cpu(memcg_stock, cpu)); + + for_each_mem_cgroup(memcg) + page_counter_drain_stock_cpu(&memcg->memory, cpu); return 0; } @@ -2595,7 +2600,6 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask) static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned int nr_pages) { - unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages); int nr_retries = MAX_RECLAIM_RETRIES; struct mem_cgroup *mem_over_limit; struct page_counter *counter; @@ -2608,31 +2612,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, bool allow_spinning = gfpflags_allow_spinning(gfp_mask); retry: - if (consume_stock(memcg, nr_pages)) - return 0; - - if (!allow_spinning) - /* Avoid the refill and flush of the older stock */ - batch = nr_pages; - reclaim_options = MEMCG_RECLAIM_MAY_SWAP; if (!do_memsw_account() || - page_counter_try_charge(&memcg->memsw, batch, &counter)) { - if (page_counter_try_charge(&memcg->memory, batch, &counter)) + page_counter_try_charge(&memcg->memsw, nr_pages, &counter)) { + if (page_counter_try_charge(&memcg->memory, nr_pages, &counter)) goto done_restock; if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, batch); + page_counter_uncharge(&memcg->memsw, nr_pages); mem_over_limit = mem_cgroup_from_counter(counter, memory); } else { mem_over_limit = mem_cgroup_from_counter(counter, memsw); reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP; } - if (batch > nr_pages) { - batch = nr_pages; - goto retry; - } - /* * Prevent unbounded recursion when reclaim operations need to * allocate memory. This might exceed the limits temporarily, @@ -2729,9 +2721,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, return 0; done_restock: - if (batch > nr_pages) - refill_stock(memcg, batch - nr_pages); - /* * If the hierarchy is above the normal consumption range, schedule * reclaim on returning to userland. We can perform reclaim here @@ -2768,7 +2757,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, * and distribute reclaim work and delay penalties * based on how much each task is actually allocating. */ - current->memcg_nr_pages_over_high += batch; + current->memcg_nr_pages_over_high += nr_pages; set_notify_resume(current); break; } @@ -3073,7 +3062,7 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, account_kmem_nmi_safe(memcg, -nr_pages); memcg1_account_kmem(memcg, -nr_pages); if (!mem_cgroup_is_root(memcg)) - refill_stock(memcg, nr_pages); + memcg_uncharge(memcg, nr_pages); css_put(&memcg->css); } @@ -4077,6 +4066,8 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) static void mem_cgroup_free(struct mem_cgroup *memcg) { + page_counter_free_stock(&memcg->memory); + page_counter_free_stock(&memcg->memsw); lru_gen_exit_memcg(memcg); memcg_wb_domain_exit(memcg); __mem_cgroup_free(memcg); @@ -4244,6 +4235,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) refcount_set(&memcg->id.ref, 1); css_get(css); + /* failure is nonfatal, charges fall back to direct hierarchy */ + page_counter_alloc_stock(&memcg->memory, MEMCG_CHARGE_BATCH); + /* * Ensure mem_cgroup_from_private_id() works once we're fully online. * @@ -4304,6 +4298,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) wb_memcg_offline(memcg); lru_gen_offline_memcg(memcg); + page_counter_disable_stock(&memcg->memory); drain_all_stock(memcg); mem_cgroup_private_id_put(memcg, 1); @@ -5499,7 +5494,7 @@ void mem_cgroup_sk_uncharge(const struct sock *sk, unsigned int nr_pages) mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages); - refill_stock(memcg, nr_pages); + page_counter_uncharge(&memcg->memory, nr_pages); } void mem_cgroup_flush_workqueue(void) @@ -5552,12 +5547,9 @@ int __init mem_cgroup_init(void) memcg_wq = alloc_workqueue("memcg", WQ_PERCPU, 0); WARN_ON(!memcg_wq); - for_each_possible_cpu(cpu) { - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_memcg_stock); + for_each_possible_cpu(cpu) INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work, drain_local_obj_stock); - } memcg_size = struct_size_t(struct mem_cgroup, nodeinfo, nr_node_ids); memcg_cachep = kmem_cache_create("mem_cgroup", memcg_size, 0, -- 2.53.0-Meta