From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4A35392828 for ; Fri, 22 May 2026 22:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.42 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487596; cv=none; b=FpUyObuAnigYOzhQOWPKZt8g7BalX9E6QNLdKpfjDlyPLcX9wKkHWMgEEFY3n0S8CRxhKR5MD3SDlkx+ZvioaknxoEo0v0OsW57UzSxjgn2wbsU06BMdNuiCqqiCTiJl6IV3BJU7+V3QRyVMgQr3zQLhK07PAYMXmjgT7Avc71k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779487596; c=relaxed/simple; bh=ob+VYgFU6m5S1jM4nkjA7iYLIdVS3X8bkqvXmzMrSvg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rZSV3Ss5hZ9A31zgez0UfjQrmIWWCLzU1YIiQfnzbsX7gBrYhGhxP6DLF8q8HNEtnO+W+Xfpk9TRdD7zleWlRu2/cF+cCSNb0ebiPQbAykpDKPdOQLeO/ryr2GOhWDysKqHs62ubVe2wg88ozZ8gj54KfkB2A7ec65bTI8CZQLk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MXgjwohL; arc=none smtp.client-ip=209.85.210.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MXgjwohL" Received: by mail-ot1-f42.google.com with SMTP id 46e09a7af769-7dbd23bc684so4216418a34.2 for ; Fri, 22 May 2026 15:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779487594; x=1780092394; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ieNLELksxQ12VWfSTE2wcDumq9qPO5WfoZYKitlIUko=; b=MXgjwohLqY33VXXjW436tp0GNuIjXwIKMPptm+adFVNjY808wXPnrkGPKPdLRiQ+eS Eu8wEo7NidO/cI0vUN7nXFRiGMw2VHWzpf0WbRgd1YPXV0jwByeaouW55vUQF1GvTN+o EeMtk3R5zmzcqanCkPoRqdU3En9BTHCQCRysvJ/cQPv55XHHaeCgFIw1lBDq69TwvEkN yezWM1UQxG9+EtGDp22DwTWuwMJfu80fxuqV4l2SviOW5zKi/9YxDusZFspB4FrPu96F Cp+wZeU8rl0Kux0KuvtaDmSZ2+TxKIT5vSmhNJ1i25TkWLmdMjCIOyK4kLYaROfysPnd 7JbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779487594; x=1780092394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ieNLELksxQ12VWfSTE2wcDumq9qPO5WfoZYKitlIUko=; b=bzateObIi/BecB7LTM9P6PFLvebzP0ikd3ZrewuCFav4OLT51wOTOAF1yuA14XsCA5 RiqkAG2qlIbqTbHsa1En5BOMgZmcjNdo1yhy2ArtLzJUvTke05Egd7O3PTEsBQdimdTy HBg8Z3RnRNgFyDeVVfOPrx9o66VLcBuUvQuFMPXHJtmSDVSEZggymGDMT+90IMSC3O8/ xyP7QtxeL0UMXEFZSw2wVi2K2c+wRpDk6bTK3LSdIDL/oKGTxtdQ5bY4p7/4VPtxOsUV cYbjkYH/zHg6LTuh9HhD89DSnZnZ5FzvkEsYV8JxcuilUlrIFHrgEoKmVY5iR/MPPvGm tS7Q== X-Forwarded-Encrypted: i=1; AFNElJ8JyZuY5x46YNTSzwZONLsGAxyq47l66ZYnXdmRua1KMUUn5N8IuQlciJeIrH8lN/uMXBy9UDtorx73DeQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzyRHkwEF9RIDEw+ievFHWuiFnPVxgey1/0N8iG8qzmu/SUzib8 pglH1n6ZfNYsBIvEtql5LLmwcm099dTwjaD33gGYD6TwLQ8Bp3eE7q6x X-Gm-Gg: Acq92OFQY6LPREI5kgGvv7EH/qGsOu+1gTuZviVhuMXrB/nYPhaWisvVkYG1WzWJPea xdMGg1Ypc8A+bwY0UheYRfaNb4ZaqQFtsy9btAjWdj1nhtE36974zPan2w57xtdyBaS40hV8cK/ 7zIaDtORFUW9YEyH7OqK0vbA9OOrVMw1APDVZDHDYFcqGT5lwPGM0KMzLiUy/fJzr4dw/cjl9Wi Dk8eQodp2bjwdZNwX2DzBTyBBqgsGy2kYitphWm5STc2DYu61sabWQ5Ma6/oODnXiYuiDpSUNR/ Z7CbgNdTINtvaI6CZtxyjFdCmvFGDNVn3EAvmeEJfxP4RW/Xdv/atK0rwImckdOf2ECAqpcIzVM 74J65EEwDGotNCwku2CG1eeoeT5+v2tcNBW+nyc888mnm/Cgcpqrm9uD37D15/0LTz7HotoP5mc USdJchsgzWHaplflDbsUTpTBoXj7QdVMW7RXD5ErIC6YXGYi9It91e X-Received: by 2002:a05:6830:6d4d:b0:7e1:cbe3:bb1b with SMTP id 46e09a7af769-7e5fea05c4dmr3245063a34.0.1779487593743; Fri, 22 May 2026 15:06:33 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:1::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6064aa0bbsm1941729a34.12.2026.05.22.15.06.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 May 2026 15:06:33 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 2/7 v2] mm/page_counter: use page_counter_stock in page_counter_try_charge Date: Fri, 22 May 2026 15:06:20 -0700 Message-ID: <20260522220627.1150804-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260522220627.1150804-1-joshua.hahnjy@gmail.com> References: <20260522220627.1150804-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Make page_counter_try_charge() stock-aware. We preserve the same semantics as the existing stock handling logic in try_charge_memcg: 1. Limit-check against the stock. If there is enough, charge to the stock (non-hierarchical) and return immediately. 2. Greedily attempt to fulfill the charge request and fill the stock up at the same time via a hierarchical charge. 3. If we fail with this charge, retry again (once) with the exact number of pages requested. 4. If we succeed with the greedy attempt, then try to add those extra pages to the stock. If that fails (trylock), then uncharge those surplus pages hierarchically. As of this patch, the page_counter_stock is unused, as it has not been enabled on any memcg yet. No functional changes intended. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index a1a871a9d5c49..e002688bf7f1a 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail) { struct page_counter *c; + unsigned long charge = nr_pages; + unsigned long batch = READ_ONCE(counter->batch); bool protection = track_protection(counter); bool track_failcnt = counter->track_failcnt; + if (counter->stock && local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock = this_cpu_ptr(counter->stock); + + if (stock->nr_pages >= charge) { + stock->nr_pages -= charge; + local_unlock(&counter->stock->lock); + return true; + } + local_unlock(&counter->stock->lock); + } + + charge = max_t(unsigned long, batch, nr_pages); + +retry: for (c = counter; c; c = c->parent) { long new; /* @@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter, * we either see the new limit or the setter sees the * counter has changed and retries. */ - new = atomic_long_add_return(nr_pages, &c->usage); + new = atomic_long_add_return(charge, &c->usage); if (new > c->max) { - atomic_long_sub(nr_pages, &c->usage); + atomic_long_sub(charge, &c->usage); /* * This is racy, but we can live with some * inaccuracy in the failcnt which is only used @@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter, WRITE_ONCE(c->watermark, new); } } + + /* charge > nr_pages implies this page_counter has stock enabled */ + if (charge > nr_pages) { + if (local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock; + + stock = this_cpu_ptr(counter->stock); + stock->nr_pages += charge - nr_pages; + local_unlock(&counter->stock->lock); + } else { + page_counter_uncharge(counter, charge - nr_pages); + } + } + return true; failed: for (c = counter; c != *fail; c = c->parent) - page_counter_cancel(c, nr_pages); + page_counter_cancel(c, charge); + + if (charge > nr_pages) { + /* Retry without trying to grab extra pages to refill stock */ + charge = nr_pages; + goto retry; + } return false; } -- 2.53.0-Meta