From: Andrea Righi <arighi@nvidia.com>
To: Vlastimil Babka <vbabka@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Harry Yoo <harry@kernel.org>, Hao Li <hao.li@linux.dev>,
Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
David Hildenbrand <david@kernel.org>,
Gregory Price <gourry@gourry.net>, Zi Yan <ziy@nvidia.com>,
Joshua Hahn <joshua.hahnjy@gmail.com>,
Rakie Kim <rakie.kim@sk.com>,
Ying Huang <ying.huang@linux.alibaba.com>,
Jan Kara <jack@suse.cz>, Quan Liu <qualiu@nvidia.com>,
Lee Trager <ltrager@nvidia.com>,
Seth Forshee <sforshee@nvidia.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: [PATCH v2] mm/slub: Respect MPOL_BIND when taking remote partial slabs
Date: Thu, 19 Nov 2026 17:04:30 +0100 [thread overview]
Message-ID: <20261119160430.1113397-1-arighi@nvidia.com> (raw)
When the local node has no partial slabs, get_from_any_partial() and
__refill_objects_any() take partial slabs from other nodes. The scan
only checks cpuset_zone_allowed(), not the task's mempolicy, so a task
bound to a node with MPOL_BIND can get objects from nodes outside its
policy, even though new slabs for it are allocated on the allowed nodes.
slab_strict_numa doesn't help here: when the local node is in the bind
mask, apply_strict_numa_policy() leaves the node as NUMA_NO_NODE and the
allocation takes the same path.
Add mempolicy_slab_nodemask(), which returns the MPOL_BIND nodemask, and
use it to restrict the remote partial scan. The nodemask is used only
when the page allocator would also apply it to new slabs: the allocation
is not for an explicit node, not __GFP_THISNODE, not in interrupt
context or with BH disabled, spinning is allowed, the cache's zone is
subject to the policy and the nodemask is valid for the task's cpuset.
Sheaf refills allocate new slabs on the local node, so for them the
local node must also be in the nodemask. Otherwise new slabs can come
from any node anyway, and skipping remote partial slabs would only grow
the cache (e.g., a task bound to a node with only ZONE_MOVABLE memory,
or a GFP_DMA cache). Skipping it when spinning is not allowed also keeps
kmalloc_nolock() from reading current->mems_allowed_seq. Other policies
and tasks without a policy are not affected.
This was originally reported as a ~30% hackbench regression that turned
out to depend on the slab state of each boot. On a dual-socket NVIDIA
Vera system (node 0: CPUs 0-87, node 1: CPUs 88-175), running right
after boot without touching the slab caches:
$ numactl --physcpubind=0-87 --membind=0 \
hackbench -P -p -g 10 -f 20 -s 100 -l 100000
avg (5 runs) remote pipe buffer arrays
before 7.021s 160 / 1010
after 5.222s 5 / 1010
To make it deterministic, the slab state can be seeded before each run
by holding ~4K pipes created on a node 0 CPU (no partial slabs left on
node 0), then creating 1K pipes on a node 1 CPU and closing 7 out of 8
of them (sparse partial slabs on node 1). With seeding and the same
hackbench command:
avg (10 runs) remote pipe buffer arrays
before 9.885s 1682 / 2020
after 5.114s 0 / 2020
With --membind=0,1, --interleave=0,1, --preferred=0 or no policy,
nothing changes (within noise), as expected.
Reported-by: Quan Liu <qualiu@nvidia.com>
Link: https://lore.kernel.org/all/20261110065219.894219-1-arighi@nvidia.com/
Suggested-by: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
Changes in v2:
- Replace the pipe-specific kmalloc_node() approach with a generic fix
in SLUB: restrict the remote partial scan to the MPOL_BIND nodemask
(Vlastimil Babka)
- Link to v1:
https://lore.kernel.org/r/20261110065219.894219-1-arighi@nvidia.com
include/linux/mempolicy.h | 1 +
mm/mempolicy.c | 24 ++++++++++++++++++++++++
mm/slub.c | 25 +++++++++++++++++++++----
3 files changed, 46 insertions(+), 4 deletions(-)
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 65c732d440d2f..129ae718c75b0 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -147,6 +147,7 @@ extern bool init_nodemask_of_mempolicy(nodemask_t *mask);
extern bool mempolicy_in_oom_domain(struct task_struct *tsk,
const nodemask_t *mask);
extern unsigned int mempolicy_slab_node(void);
+extern nodemask_t *mempolicy_slab_nodemask(gfp_t gfp);
extern enum zone_type policy_zone;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index c7d4d149c9b88..8b29be7ee246c 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2198,6 +2198,30 @@ unsigned int mempolicy_slab_node(void)
}
}
+/*
+ * Return the nodes the task's MPOL_BIND policy restricts new slabs allocated
+ * with @gfp to, or NULL if they are not restricted. This follows the checks
+ * of alloc_frozen_pages_noprof() and policy_nodemask(), so that the nodes
+ * are returned only if the page allocator would also apply them.
+ */
+nodemask_t *mempolicy_slab_nodemask(gfp_t gfp)
+{
+ struct mempolicy *policy;
+
+ if (in_interrupt() || (gfp & __GFP_THISNODE))
+ return NULL;
+
+ policy = current->mempolicy;
+ if (!policy || policy->mode != MPOL_BIND)
+ return NULL;
+
+ if (!apply_policy_zone(policy, gfp_zone(current_gfp_context(gfp))) ||
+ !cpuset_nodemask_valid_mems_allowed(&policy->nodes))
+ return NULL;
+
+ return &policy->nodes;
+}
+
static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
{
struct weighted_interleave_state *state;
diff --git a/mm/slub.c b/mm/slub.c
index cd671b9347191..1177d76a1cb90 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3984,11 +3984,12 @@ static void *get_from_partial_node(struct kmem_cache *s,
/*
* Get an object from somewhere. Search in increasing NUMA distances.
*/
-static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
+static void *get_from_any_partial(struct kmem_cache *s, int node, gfp_t gfp_flags,
const struct slab_alloc_context *ac)
{
#ifdef CONFIG_NUMA
struct zonelist *zonelist;
+ nodemask_t *nodemask = NULL;
struct zoneref *z;
struct zone *zone;
enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
@@ -4027,8 +4028,19 @@ static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
if (allow_spin)
cpuset_mems_cookie = read_mems_allowed_begin();
+ /*
+ * Restrict the scan to the nodes of the task's MPOL_BIND
+ * policy, but only when new slabs are also allocated from
+ * them. With an explicit node or when spinning is not allowed,
+ * new slabs ignore the mempolicy. The latter case must also
+ * avoid mempolicy_slab_nodemask(), which reads
+ * current->mems_allowed_seq.
+ */
+ if (allow_spin && node == NUMA_NO_NODE)
+ nodemask = mempolicy_slab_nodemask(gfp_flags | s->allocflags);
+
zonelist = node_zonelist(mempolicy_slab_node(), gfp_flags);
- for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
+ for_each_zone_zonelist_nodemask(zone, z, zonelist, highest_zoneidx, nodemask) {
struct kmem_cache_node *n;
n = get_node(s, zone_to_nid(zone));
@@ -4072,7 +4084,7 @@ static void *get_from_partial(struct kmem_cache *s, int node, gfp_t flags,
if (object || (node != NUMA_NO_NODE && (flags & __GFP_THISNODE)))
return object;
- return get_from_any_partial(s, flags, ac);
+ return get_from_any_partial(s, node, flags, ac);
}
static bool has_pcs_used(int cpu, struct kmem_cache *s)
@@ -7461,6 +7473,7 @@ __refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min
unsigned int max)
{
struct zonelist *zonelist;
+ nodemask_t *nodemask;
struct zoneref *z;
struct zone *zone;
enum zone_type highest_zoneidx = gfp_zone(gfp);
@@ -7474,8 +7487,12 @@ __refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min
do {
cpuset_mems_cookie = read_mems_allowed_begin();
+ nodemask = mempolicy_slab_nodemask(gfp | s->allocflags);
+ /* new slabs for the refill are allocated on the local node */
+ if (nodemask && !node_isset(numa_mem_id(), *nodemask))
+ nodemask = NULL;
zonelist = node_zonelist(mempolicy_slab_node(), gfp);
- for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
+ for_each_zone_zonelist_nodemask(zone, z, zonelist, highest_zoneidx, nodemask) {
struct kmem_cache_node *n;
unsigned int r;
--
2.56.1
reply other threads:[~2026-11-19 16:04 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119160430.1113397-1-arighi@nvidia.com \
--to=arighi@nvidia.com \
--cc=akpm@linux-foundation.org \
--cc=cl@gentwo.org \
--cc=david@kernel.org \
--cc=gourry@gourry.net \
--cc=hao.li@linux.dev \
--cc=harry@kernel.org \
--cc=jack@suse.cz \
--cc=joshua.hahnjy@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ltrager@nvidia.com \
--cc=qualiu@nvidia.com \
--cc=rakie.kim@sk.com \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=sforshee@nvidia.com \
--cc=vbabka@kernel.org \
--cc=ying.huang@linux.alibaba.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®