From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812B2388E5A for ; Mon, 25 May 2026 19:05:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779735904; cv=none; b=Y6bEWJVqkSyk34xZIGSjvSGBkttB8EBpJOp739lnuOBNF4pXnZADqWxgqQZvGwnhEA+wFG8z8OZBAZIjwRTMRY2+cKv/ZF3esYAwYFGEpLXjzJr7RNfqKyjDy4/aD1+FBlrykB+S0vJdJttGqsFpkiliLzZfLDf3qg/67q834Kc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779735904; c=relaxed/simple; bh=ob+VYgFU6m5S1jM4nkjA7iYLIdVS3X8bkqvXmzMrSvg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pHfHByxTK8BeiAKEjEg2VS7zQ8srbepaVTEGoEF+EHbo7nv/6xR5aZNKNQ/elDkY/S0xCBwhKTe8dO4hAqFzTvattHnsasTj1pDazQsWyubQEVsoRQWhx5u3dwcq8wOJaw8Y82++FhGVUHieK5w6GElIMWglwuIW2RZiSXZ0Tkw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IkpwdsXA; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IkpwdsXA" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-7e6201aeb2aso2380212a34.3 for ; Mon, 25 May 2026 12:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779735901; x=1780340701; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ieNLELksxQ12VWfSTE2wcDumq9qPO5WfoZYKitlIUko=; b=IkpwdsXAVvNV1xx+KtG/7+0+bxXc+fUBGQXwLL3ScsBJXtFD7sv/fk1YEi9TGFtRRI bRffVHe0FDjXA4Rxi4m1GpPoUnPMd+dqgYawI8b66Sj0MJsOaPskdS+vu9uHEhKKJU6a 1QvTufAvtxbUxcQOZQg21UWlTA3AI0BWvp8ZhPO683zwkHYa1aQzrBOfjRL3gaNpeom9 mfwHUCxVe5PT19aF/UBP/GgwmDvYyT+czdJVAZJirIjlk9h8zaOzjj8gDBHWGBCOw+1E R6UC3y+60CYVO4lyG8PtOKL6qFVPuY5/GfcABjdqty2zhlAVT4Jsffa0AKAVHtN9Uiiq cl5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779735901; x=1780340701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ieNLELksxQ12VWfSTE2wcDumq9qPO5WfoZYKitlIUko=; b=Rm9v7ilrr/1aM3h0E2SIZeK2vBRPs/osID+vtqPi0QTgQs1izPxNLO6nzSEEdWDqwd hrgaZ+4+BrYVOMVKCVZIq8tLNRTT2W8Q1aQgztsSP4ITptYICryNxv4C5iLKcB8cUIyd +wPT29x0Q9h9hqqkUFsGPYRQqakQ1oOhWha1T8TF3lD62z0n0uqOPxpUbAcVEvVfcbsS PETfO/+BMGCOSSDolFIskHp8vbxkacYqyFpZY4g3bDtwktjlGjZx8wSo5fsJJ4/9X2KS FwewJlITyzopZ/CjIqmAnwOXJBXT1cj6CkvZ14VWAHh3ZRz1n8jUl5VJYgipHj8UJnAY 7NUQ== X-Forwarded-Encrypted: i=1; AFNElJ/QiXVOm28OKcQ9l8Bk5eX/WL/YvNXtUAZWY4iBt3TQobW2yvWkBHAN8LGRE8U1pJVXSu8JEaL+NxA+1wg=@vger.kernel.org X-Gm-Message-State: AOJu0YxIjhpA0vG+uEDfcETHSq7d8eSyiHJkjksc2vWMqiMJteYXG8wt bXQcbvdJq/Gw4Ih7YLxfL3a8KyFAXkfbn9xKKiEB183wcyDsKb2dLr0Y X-Gm-Gg: Acq92OHzbc9qDxOTh7izOIYg0XDkRjBlqWtbeQb769wW+IyRsrKW+vi6ukoL2r8kbLP 8BSfRzrhRwKKuRFmSXFhIqHtvXQ5Xo4Gk2Tm7a3lObGF5c9E96pIz7RN3x3GqiKHlUqCuAwW4EI HO+XPwv1CV7PBzs5rT4u7WVvGr9HLuZlXDSFMoPLPsEf3Ub7jTEHKhc6GqG9HrKB5A7kfZZ4eJN oS/P5u2sFEInINx7D6MB3Ar1WGwMOw0VPgg8S6MucGN0YoDihwbLuZoDBf08iO6wbFKCr9ZEMR9 mSLDEMmLXoLDsUOkcUevtgQolPAdHJLqYinu4cfkvI0dVyHA9r9d8ptY5hwMwa7j3PRfCenYR9W NHao4pXQGUrMYC5kGqMekPwawkntY1h0uH0PzDvMITM8g1YoRcubPjhrX5gR4fFcQY2RdPam421 EYtfH5e7QNDljmuoY/RLIskYw4sth4h7SwVLZlFK5CgGTbszrgYid3Kg== X-Received: by 2002:a05:6830:6a91:b0:7d7:d510:4bf9 with SMTP id 46e09a7af769-7e5fedd5083mr9959038a34.12.1779735901451; Mon, 25 May 2026 12:05:01 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:50::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6065e7cb6sm7850039a34.16.2026.05.25.12.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2026 12:05:01 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 2/7 v3] mm/page_counter: use page_counter_stock in page_counter_try_charge Date: Mon, 25 May 2026 12:04:49 -0700 Message-ID: <20260525190455.2843786-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com> References: <20260525190455.2843786-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Make page_counter_try_charge() stock-aware. We preserve the same semantics as the existing stock handling logic in try_charge_memcg: 1. Limit-check against the stock. If there is enough, charge to the stock (non-hierarchical) and return immediately. 2. Greedily attempt to fulfill the charge request and fill the stock up at the same time via a hierarchical charge. 3. If we fail with this charge, retry again (once) with the exact number of pages requested. 4. If we succeed with the greedy attempt, then try to add those extra pages to the stock. If that fails (trylock), then uncharge those surplus pages hierarchically. As of this patch, the page_counter_stock is unused, as it has not been enabled on any memcg yet. No functional changes intended. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index a1a871a9d5c49..e002688bf7f1a 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail) { struct page_counter *c; + unsigned long charge = nr_pages; + unsigned long batch = READ_ONCE(counter->batch); bool protection = track_protection(counter); bool track_failcnt = counter->track_failcnt; + if (counter->stock && local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock = this_cpu_ptr(counter->stock); + + if (stock->nr_pages >= charge) { + stock->nr_pages -= charge; + local_unlock(&counter->stock->lock); + return true; + } + local_unlock(&counter->stock->lock); + } + + charge = max_t(unsigned long, batch, nr_pages); + +retry: for (c = counter; c; c = c->parent) { long new; /* @@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter, * we either see the new limit or the setter sees the * counter has changed and retries. */ - new = atomic_long_add_return(nr_pages, &c->usage); + new = atomic_long_add_return(charge, &c->usage); if (new > c->max) { - atomic_long_sub(nr_pages, &c->usage); + atomic_long_sub(charge, &c->usage); /* * This is racy, but we can live with some * inaccuracy in the failcnt which is only used @@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter, WRITE_ONCE(c->watermark, new); } } + + /* charge > nr_pages implies this page_counter has stock enabled */ + if (charge > nr_pages) { + if (local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock; + + stock = this_cpu_ptr(counter->stock); + stock->nr_pages += charge - nr_pages; + local_unlock(&counter->stock->lock); + } else { + page_counter_uncharge(counter, charge - nr_pages); + } + } + return true; failed: for (c = counter; c != *fail; c = c->parent) - page_counter_cancel(c, nr_pages); + page_counter_cancel(c, charge); + + if (charge > nr_pages) { + /* Retry without trying to grab extra pages to refill stock */ + charge = nr_pages; + goto retry; + } return false; } -- 2.53.0-Meta