From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 636DA3E00B9 for ; Fri, 5 Jun 2026 15:36:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673773; cv=none; b=cdUztIu6B22LNJunm5Yv/aENBPmWPM2g9CSRjC8aVmO6gCjixwtkAi+MvqRdeBnbPN8UlgFx0LgEDm9mnsTVqvgP3oj0NsagRI3rRNtf8nmBTF/91oecrS0UnHhpfc616ZjVSsc3V+MGRnFl0q2+q/oeXWKVtsZoG4UFIeGqgMk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780673773; c=relaxed/simple; bh=gaqXcw7SaNGRC8ZlwI52O8ug9+jqeuYXrmBeNBSCLsc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JqzOS4gU8fQSZ7BE2xlPK9vo+AycPE1MKSi6xsvFlyGpjNmaKdtY5tOX6xccF0lABxScqIPlySppOa+87OnPcD/mQzXmy0FUTNl1pI4LSE/eL85j4kGKGiFeM+5TG24/OUMPM4n74AC6vlQCYN+j9U/CPVoE5IBodCi1nPaFte4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ox/7TD9x; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ox/7TD9x" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-7e615efd7d7so1828979a34.2 for ; Fri, 05 Jun 2026 08:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780673767; x=1781278567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J5dsckH9YwlwdD1mktr8FrA3biFKCktv90GPZj2FBiQ=; b=Ox/7TD9xcte/J7liCvcvq1VpwM2SIA9mJCuDaY/LxDBlVqx3ucTYFMib8Po7MGsmD4 igERZKncQP7PAoMnJt5ewbi/bz6n0MMrBxZyCRKpScfjpNCwrQfFeU2kqX2TAw8UU01+ iSRgDt/vwFXNHbQeGVBAfyznpXI8AuFJ2lNsPKKj8UJ5cTODdAzzkcZYyhdqAOi6hn6y NHVGtclcUs741ECZ1s2e7p1+NOzlnHFRGbcF+kz+G+s5vxfXFXww62en/e7rHQxqSzbZ mHM5RUl5dOa/Ik0S2m0+Gjmvz72kVLyZGJsHklj/gZW4EstEPoTLIHlDa/0mYIcrc3p1 EtLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780673767; x=1781278567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=J5dsckH9YwlwdD1mktr8FrA3biFKCktv90GPZj2FBiQ=; b=E5Yumc6THgWgH9CM6vzkYKC4mT/EMcmdGFhiuIe1UWo7GJ0Ei0QUq37GUVFAwJwtne Zad1OhXAjP0ZzPdjYD00eRieCDTBtn88a1eXW2i3r93B94qZ2AWvFR/NOy2U5d1j9fvd Rs6Jll+tnJywy3NXtLm2GYp4ymG0Z0w3iJdqqDdpvbaMUaGQnBw6P0A1CYOlqIm29ube +1YP189sFckQZqH2zla0oIW5Z8dY2+I/SuDSnKrqx2r1EMk0r7B8TUx2uNk4ykF4498i sg+aw3dtbIGfeQ8cx4XwXfxJwH0fX6TyrrqYWnaEMTszSdsIMfiDhbb67/caaUscqpza w83Q== X-Forwarded-Encrypted: i=1; AFNElJ9MZD3Zm+a83PMjYSIxGOSqZmEzWKLEoCo+MQTURVl7W2FICT7XWD+wCZZ/zrrKDiiwK3pJLaZ9OfInAMQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yzg7RJHFSMoDRH91wVGiX+shmOADlIGlM0Aicu7eJi9Z5dWNW1x yBgmn8ml767VBNTM1ITUAw3CB9SZ2StnG36kj4CwtWmHb3o0HtLLcDXu X-Gm-Gg: Acq92OGwWxmR8tU0uXnVWr0X1ycH3BdidQLraLX/mXTi/koYdv6lPUJt098nThuTlAd GWLWSqrjqXQ4RgItLkefAPr3tncjtizNWLrQnaKBi+D0GLhd4fmERxNnmoEVetoM4uZa4419a4H VQHfiEdsr2Hw4G8hNSSug/Rub+dczSuiPAOB7YjrHWpoKnOejcovXTv5t3q4gIw0b83UlRWJCQ9 SV3cO+aOYthWUyGl79i0Ly0jiVbjr2p2qw4LqIWUZeiD3TdzPgZCIpkEcMTBJykCXUOuCmfykIM yc956qpNZjEikS5tJuZ7oNnO/h/affouRJLNgvfg5pjYbuBnYMj/akwbSEImiQ8zlZu9mh/+PXD 6/+zTBAL66d5aFnUNlvUDhV184zEOBClRXeyImn3KQEh11/l21bslWmNW35ixRI9W2KyiDfaNpf It715dDH/P4QH2go66vRUfwxIrAGGMx9uKpKx2/xmgq/qxtNJLJCQTBbkyya6HJKw= X-Received: by 2002:a05:6830:8289:b0:7e3:d199:3164 with SMTP id 46e09a7af769-7e70c6a605cmr2472371a34.11.1780673767180; Fri, 05 Jun 2026 08:36:07 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:a::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6e7468f8fsm6342664a34.6.2026.06.05.08.36.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2026 08:36:06 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko Cc: Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 2/6 v3] mm/page_counter: use page_counter_stock in page_counter_try_charge Date: Fri, 5 Jun 2026 08:35:58 -0700 Message-ID: <20260605153603.234296-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com> References: <20260605153603.234296-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Make page_counter_try_charge() stock-aware. We preserve the same semantics as the existing stock handling logic in try_charge_memcg: 1. Limit-check against the stock. If there is enough, charge to the stock (non-hierarchical) and return immediately. 2. Greedily attempt to fulfill the charge request and fill the stock up at the same time via a hierarchical charge. 3. If we fail with this charge, retry again (once) with the exact number of pages requested. 4. If we succeed with the greedy attempt, then try to add those extra pages to the stock. If that fails (trylock), then uncharge those surplus pages hierarchically. As of this patch, the page_counter_stock is unused, as it has not been enabled on any memcg yet. No functional changes intended. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/mm/page_counter.c b/mm/page_counter.c index 9f3e3f8d896c4..1a71de4f43fd0 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail) { struct page_counter *c; + unsigned long charge = nr_pages; + unsigned long batch = READ_ONCE(counter->batch); bool protection = track_protection(counter); bool track_failcnt = counter->track_failcnt; + if (counter->stock && local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock = this_cpu_ptr(counter->stock); + + if (stock->nr_pages >= charge) { + stock->nr_pages -= charge; + local_unlock(&counter->stock->lock); + return true; + } + local_unlock(&counter->stock->lock); + } + + charge = max_t(unsigned long, batch, nr_pages); + +retry: for (c = counter; c; c = c->parent) { long new; /* @@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter, * we either see the new limit or the setter sees the * counter has changed and retries. */ - new = atomic_long_add_return(nr_pages, &c->usage); + new = atomic_long_add_return(charge, &c->usage); if (new > c->max) { - atomic_long_sub(nr_pages, &c->usage); + atomic_long_sub(charge, &c->usage); /* * This is racy, but we can live with some * inaccuracy in the failcnt which is only used @@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter, WRITE_ONCE(c->watermark, new); } } + + /* charge > nr_pages implies this page_counter has stock enabled */ + if (charge > nr_pages) { + if (local_trylock(&counter->stock->lock)) { + struct page_counter_stock *stock; + + stock = this_cpu_ptr(counter->stock); + stock->nr_pages += charge - nr_pages; + local_unlock(&counter->stock->lock); + } else { + page_counter_uncharge(counter, charge - nr_pages); + } + } + return true; failed: for (c = counter; c != *fail; c = c->parent) - page_counter_cancel(c, nr_pages); + page_counter_cancel(c, charge); + + if (charge > nr_pages) { + /* Retry without trying to grab extra pages to refill stock */ + charge = nr_pages; + goto retry; + } return false; } -- 2.53.0-Meta