From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22C8D3A9DAD for ; Fri, 10 Apr 2026 21:07:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775855276; cv=none; b=tL97Pmo8JE1IGQ6V0uUbzZ4vr6N2gi9k4SRGXnaPzyZ5Xg3jqi2VbYRw4nMCz7gDtC/rlVT2a0C7axqBzv0J89CHK8lOjx8ac/K/iTo0ew855Fl4W9IHp90xPJMAOXo3nvMLRllN1aOPJMcsN1vdMvRUwpMlaExf9jOy+EHg2QQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775855276; c=relaxed/simple; bh=fNBJAySEqaEJuZHafhBMXn/gZ8VxBkV7g0IuVKIaHHw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Kx5B2ThBBLPOPnRzNuuUDGkVB+M3X1fSscsmHKKbSzQaH6CPh/ovJ3RX1ci3QKqOP28lHKjSf/Y01CQD3yUj4jKm7Vo/qU5D6k5hRplwFYYSJ3XlEaKqg2GoSgSLfbxkdHGTIUN0NO7EL3IzyUaoaSbREQIhJnTAl7374dz0RuU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=akywdp/s; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="akywdp/s" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-7dbd8c6fc84so1539516a34.3 for ; Fri, 10 Apr 2026 14:07:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775855274; x=1776460074; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HPucxgysmex46OWvxebyn8gwYFbxkYWxk7oOg4s3POI=; b=akywdp/sY9WaRGsdI5CBnKquJwydApWBhDEI10AzpK95dTt+oBRmUOTenGqBOc1iAB 5XZOLawq+7z9F3JN9Cg8W6ackRo/bQZA9OeuNJT5u2Qb3q626PfoWNtn5NmPSaVL8e5Y yBhuUxgUk6G1IE5Jtkxiiq899UoGpKWAj4ALZBvkfuzrKo8axyqvu5JqgqJvpk1SAlZ+ G81SE0a7aoCOWs5lx/+apUVYrclKyJJkSJNOVHXoaEHr5LmB0YFYJm+UtRCLKdmbuBda oymgntVikPugPCRnKBGjGy34wsZGteDQ49uR6aQjgmx4ZOo3AbB108HUUQB18ywH7nHc LAaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775855274; x=1776460074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HPucxgysmex46OWvxebyn8gwYFbxkYWxk7oOg4s3POI=; b=X6tIFGNbMtoL5RQIQOsqcq/wD5QWm+3QwWWQVpA86vVtrEB1VotXWQtgQHxQ+DlrEF Ut0Zho+U4ml23hlrfJsfLFs6GP9YpZ+40+zssexQeMNDA5M0h1hWWkahL4Zb8vip7IkP jZV27/HGqChOH7VOytc07WQRO2ja4foVTgtF5P9QzMCgcyqIveLytSmJ4T3K8Kl//4qj whkNs35eNwgpnxpNIEl2sf4KIwvrh+KTK/qiHYwkUir962lYIry09yZkQ/mUpaUk4ibR V9YbidRM9HIO9nJtBGiijtaGkSJ6U5EAkw1TZK9P3onI70uht0uWqz96yF4cOKEoz95V /IPw== X-Forwarded-Encrypted: i=1; AJvYcCWLhvbWZdRxOwWSMmWTQi9DG02zOspjb13xPxHdzRgCfglsL/ZEvwnhbYj7Qfli2NhNumA3E1GcR1GHxKg=@vger.kernel.org X-Gm-Message-State: AOJu0YzyWPZs6v1dg+0Do+kbtn0ySaTBida90r11abDeZm11oYjUdmZ8 rDCKVLYUzOF4H00qlBcionpS/Gpza9NrMLSzqm4tdvjNfgU/p5M6tMKZ X-Gm-Gg: AeBDievmYpfpGZ8hYR1YJWf/Ii1P3VrtGQwb6GmfVA10gk8t92kdgqS0yTZ3Vk8VD9N ERjouXhndA7FfZCyFi2c4zgMnjTeaJ+t1WMfxGskZyrOeaU1SnPzplJ6V/IRrdQ8KqmiqmIAeyH l4m4JwyUrgll5CoUmb/WT0s/RmIeG3LlNJHYJiEHRJoH4H40wk1C1toSY4wUX/CdD+tks50fg1p ND+Qt6duc1opW0KAzmHS17u0AuM24DI1K05A/0w96nomR0Z2i9e7lKpnYFKRrOXFW3dmKqs5srG qOA4JsK9pQZfKhxxIqiaApK2TpYiUhyhKndTG8dooNLZPksQHOwJmw6HCxA1IY4pC2pj5Sc9Q/u hIMiWqwPUXHh1KYZQLIwJXzKsDMKxcJ7g3dYxRTQWj8VGHYx4E8x6tO2R+rbxDzHhPpG+fMp8F9 QH57DmSnryAqh/zucBd8UOTS7fVnSV9P4n X-Received: by 2002:a05:6830:828e:b0:7d7:5113:f83a with SMTP id 46e09a7af769-7dc27e4ed34mr2752617a34.25.1775855274137; Fri, 10 Apr 2026 14:07:54 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:56::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7dc26576d30sm3085434a34.4.2026.04.10.14.07.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:07:53 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner Cc: Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 6/8 RFC] mm/memcontrol: optimize memsw stock for cgroup v1 Date: Fri, 10 Apr 2026 14:07:00 -0700 Message-ID: <20260410210742.550489-7-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260410210742.550489-1-joshua.hahnjy@gmail.com> References: <20260410210742.550489-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Previously, each memcg had its own stock, which was shared by all page counters within it. Specifically in try_charge_memcg, the stock limit check would occur before the memsw and memory page_counters were charged hierarchically. Now that the memcg stock was folded into the page_counter level, and we have replaced try_charge_memcg's stock check against the memory page_counter's stock, this leaves no fast path available for cgroup v1's memsw check. Introduce a new stock for the memsw page_counter, charged and uncharged independently from the memory page_counter. This provides better caching on cgroup v1: The best case scenario is when both the memsw and memory page_counters can use their cached stock charge; this is the old behavior. The halfway scenario is when either the memsw or memory page_counter is within the stock size, but the other isn't. This requires one hierarchical charge. The worst case scenario is when both memsw and memory page_counters are over their limit, and must walk two page_counter hierarchies. This is the same as the old behavior. By introducing an indepednent stock for memsw, we can avoid the worst case scenario more often and can fail or succeed separately from the memory page counter. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 27d2edd5a7832..6d50f5d667434 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2245,8 +2245,10 @@ void drain_all_stock(struct mem_cgroup *root_memcg) if (!mutex_trylock(&percpu_charge_mutex)) return; - for_each_mem_cgroup_tree(memcg, root_memcg) + for_each_mem_cgroup_tree(memcg, root_memcg) { page_counter_drain_stock(&memcg->memory); + page_counter_drain_stock(&memcg->memsw); + } /* Drain obj_stock on all online CPUs */ migrate_disable(); @@ -2275,8 +2277,10 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) /* no need for the local lock */ drain_obj_stock(&per_cpu(obj_stock, cpu)); - for_each_mem_cgroup(memcg) + for_each_mem_cgroup(memcg) { page_counter_drain_cpu(&memcg->memory, cpu); + page_counter_drain_cpu(&memcg->memsw, cpu); + } return 0; } @@ -4111,6 +4115,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) /* failure is nonfatal, charges fall back to direct hierarchy */ page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH); + if (do_memsw_account()) + page_counter_enable_stock(&memcg->memsw, MEMCG_CHARGE_BATCH); /* * Ensure mem_cgroup_from_private_id() works once we're fully online. @@ -4175,6 +4181,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); page_counter_disable_stock(&memcg->memory); + page_counter_disable_stock(&memcg->memsw); mem_cgroup_private_id_put(memcg, 1); } -- 2.52.0