From: "Vlastimil Babka (SUSE)" <vbabka@kernel.org>
To: Andrea Righi <arighi@nvidia.com>
Cc: Jan Kara <jack@suse.cz>, Alexander Viro <viro@zeniv.linux.org.uk>,
Christian Brauner <brauner@kernel.org>,
Quan Liu <qualiu@nvidia.com>, Lee Trager <ltrager@nvidia.com>,
Seth Forshee <sforshee@nvidia.com>,
Gregory Price <gourry@gourry.net>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, Harry Yoo <harry@kernel.org>,
Hao Li <hao.li@linux.dev>, Christoph Lameter <cl@gentwo.org>
Subject: Re: [PATCH] pipe: Prefer mempolicy node for metadata allocations
Date: Wed, 18 Nov 2026 15:24:03 +0100 [thread overview]
Message-ID: <eae20ada-01a0-4583-8c65-8f7b5c71cd0f@kernel.org> (raw)
In-Reply-To: <av2rqSzkvZYQRgAA@gpd4>
On 11/18/26 14:44, Andrea Righi wrote:
> Hi Vlastimil,
>
> thanks for taking a look and sorry for the late reply. Comments below.
>
> On Wed, Nov 11, 2026 at 09:12:44AM +0100, Vlastimil Babka (SUSE) wrote:
>> On 11/11/26 07:48, Andrea Righi wrote:
>> > Hi Jan,
>> >
>> > On Tue, Nov 10, 2026 at 01:27:53PM +0100, Jan Kara wrote:
>> >> On Tue 10-11-26 07:52:19, Andrea Righi wrote:
>> >> > Generic slab allocations can reuse partial slabs from another NUMA node,
>> >> > even when a pipe workload is bound to one node. Pipe metadata is shared
>> >> > between senders and receivers and accessed throughout the pipe lifetime.
>> >> > Reusing remote metadata can therefore make an IPC workload pay remote
>> >> > access and coherence costs on every exchange.
>> >> >
>> >> > The target use case is a workload that binds its workers and memory to
>> >> > one NUMA node with numactl. The policy node is then a useful placement
>> >> > hint for shared pipe metadata.
>> >> >
>> >> > Allocate pipe_inode_info and its descriptor array on the node selected
>> >> > by the caller's memory policy. Apply the same preference when replacing
>> >> > slab-sized descriptor arrays during ring resizing. An explicit node lets
>> >> > SLUB try local allocations before reusing remote partial slabs.
>> >> >
>> >> > For descriptor arrays larger than KMALLOC_MAX_CACHE_SIZE, pass
>> >> > NUMA_NO_NODE to keep the policy-aware page allocation path. An explicit
>> >> > node would otherwise lose the MPOL_BIND nodemask on fallback. Select the
>> >> > policy node only for slab-sized arrays, avoiding an extra advance of an
>> >> > interleave policy for large allocations.
>> >> >
>> >> > Moving a pipe's users to another node doesn't migrate its existing
>> >> > metadata, so migration can still result in remote accesses regardless of
>> >> > this change. Generic allocations may reuse remote partial slabs, making
>> >> > initial placement dependent on allocator state. In contrast, preferring
>> >> > the policy node leverages the workload's placement hint to improve
>> >> > locality when users remain on the intended node.
>> >> >
>> >> > Testing has been performed on a dual-socket NVIDIA Vera system, covering
>> >> > hackbench, pipe transfers, pipe creation and resizing, pipe_bench, and
>> >> > targeted stress tests.
>> >> >
>> >> > The primary hackbench reproducer is:
>> >> >
>> >> > $ numactl --physcpubind=0-87 --membind=0 \
>> >> > hackbench -P -p -g 10 -f 20 -s 100 -l 100000
>> >> >
>> >> > Results (over 10 measured runs per case):
>> >> >
>> >> > avg stdev
>> >> > before 9.811s 0.130s
>> >> > after 5.770s 0.058s
>> >> >
>> >> > Before this change, allocations could reuse remote pipe metadata. With
>> >> > this change, both pipe objects and descriptor arrays are allocated on
>> >> > the requested node, improving locality and reducing hackbench runtime by
>> >> > approximately 41%.
>> >> >
>> >> > Reported-by: Quan Liu <qualiu@nvidia.com>
>> >> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
>> >>
>> >> To me this makes sense so feel free to add:
>> >>
>> >> Acked-by: Jan Kara <jack@suse.cz>
>> >>
>> >> But IMHO it deserves a look from MM guys since this is more of an MM thing
>> >> than a filesystem one... Added to CC.
>> >
>> > Thanks for taking a look, and good point about getting MM feedback.
>> >
>> > Vlastimil, Gregory, do you think this SLUB locality and mempolicy
>> > aspects makes sense here? In particular, does preferring the policy node
>> > for slab-sized pipe metadata, while retaining the policy-aware path for
>> > larger arrays, look reasonable?
>>
>> It's not wrong, but feels a bit ad-hoc. If it's pipe metadata now, what's next?
>
> Agreed, after looking deeper this isn't really a pipe problem, any slab
> user can hit it, so it should probably be addressed in slab.
>
>>
>> Slab will generally return objects from the cpu's local node even if it's
>> not specified. It's not guaranteed, but it's not guaranteed also when it's
>> specified by kmalloc_node() (unless __GFP_THISNODE forces it).
>>
>> So the results above are surprising to me. Are they on recent kernels? Are
>> cpus 0-87 all local to node 0, or is membind 0 there to force a node that's
>> not local to some of them? If they are all local, I would expect the objects
>> to be from node 0 even without kmalloc_node(). So maybe we have some unknown
>> deficiency that should be investigated.
>
> The results were collected on 7.4-rc1 and I've reproduced them in
> 7.4-rc2. CPUs 0-87 are all local to node 0 (node 0 has CPUs 0-87 and
> their SMT siblings 176-263, node 1 has 88-175 and 264-351).
>
> I should have been clearer about the setup and the results I reported:
> numbers were collected after seeding the slab state to make the problem
> more reproducible. Before each run, a helper holds ~4K pipes created on
Yeah that important part of the setup should be always stated...
> a node 0 CPU, then it creates 1K pipes on a node 1 CPU and closes 7 out
> of 8 of them, leaving sparse partial slabs on node 1. Starting from a
> clean slab state, or with only the node 1 part, all objects are local
> and there's no difference, as you expected (100000 loops, 10 runs,
> placement traced with bpftrace):
>
> avg stdev remote objs/arrays
> not seeded 5.122s 0.027 0 / 0
> seeded 9.730s 0.125 1606 / 1780
> seeded, remote_node_defrag_ratio=0 5.101s 0.019 0 / 0
> seeded, slab_strict_numa 9.889s 0.188 1624 / 1482
> seeded, pipe patch 5.112s 0.016 0 / 0
> seeded, slab change below 5.106s 0.027 0 / 0
> not seeded, slab change below 5.103s 0.036 0 / 0
>
> When node 0 has no partial slabs, get_from_any_partial() and
> __refill_objects_any() take partial slabs from other nodes, and that
> scan only checks cpuset_zone_allowed(), not the task's MPOL_BIND
> nodemask.
>
> The seeding is only there to make the problem deterministic, but it also
> happens without any artificial setup: right after a boot, without
> touching the caches, ~24% of the buffer arrays came from node 1 (7.30s
> vs 5.92s with remote_node_defrag_ratio=0).
I see.
>
>>
>> Also note that it's possible to force this behavior of respecting mempolicy
>> for all slab objects, usigng the slab_strict_numa boot parameter. AFAIK this
>> was added for systems where the interconnect makes remote accesses
>> especially costly.
>
> I tried slab_strict_numa, but it doesn't seem to help in this case,
> because apply_strict_numa_policy() leaves the node as NUMA_NO_NODE when
> the policy is MPOL_BIND and the local node is in the mask, so the
> allocation still falls back to the remote partial scan.
>
> A more generic fix could be to make the remote partial scan respect the
> bind nodemask. Do you think something like the following could be
> reasonable? (only tested with this workload so far)
Might be acceptable even without slab_strict_numa as it's a slowpath.
Perhaps mempolicy_slab_node() itself could set the nodemask if a non-NULL
"nodemask_t **" is passed.
> Thanks,
> -Andrea
>
> mm/slub.c | 29 +++++++++++++++++++++++++++--
> 1 file changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index cd671b9347191..dd6f69cf2acdf 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3981,6 +3981,27 @@ static void *get_from_partial_node(struct kmem_cache *s,
> return object;
> }
>
> +#ifdef CONFIG_NUMA
> +/*
> + * Return the nodes allowed by the current task's bind policy, or NULL if
> + * the policy doesn't restrict them. Partial slabs on other nodes must not be
> + * used to satisfy an allocation that didn't request a specific node.
> + */
> +static nodemask_t *slab_bind_nodemask(void)
> +{
> + struct mempolicy *mpol;
> +
> + if (!in_task())
> + return NULL;
> +
> + mpol = current->mempolicy;
> + if (mpol && mpol->mode == MPOL_BIND)
> + return &mpol->nodes;
> +
> + return NULL;
> +}
> +#endif
> +
> /*
> * Get an object from somewhere. Search in increasing NUMA distances.
> */
> @@ -3994,6 +4015,7 @@ static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
> enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
> unsigned int cpuset_mems_cookie;
> bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
> + nodemask_t *nodemask = slab_bind_nodemask();
>
> /*
> * The defrag ratio allows a configuration of the tradeoffs between
> @@ -4028,7 +4050,8 @@ static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
> cpuset_mems_cookie = read_mems_allowed_begin();
>
> zonelist = node_zonelist(mempolicy_slab_node(), gfp_flags);
> - for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
> + for_each_zone_zonelist_nodemask(zone, z, zonelist,
> + highest_zoneidx, nodemask) {
> struct kmem_cache_node *n;
>
> n = get_node(s, zone_to_nid(zone));
> @@ -7466,6 +7489,7 @@ __refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min
> enum zone_type highest_zoneidx = gfp_zone(gfp);
> unsigned int cpuset_mems_cookie;
> unsigned int refilled = 0;
> + nodemask_t *nodemask = slab_bind_nodemask();
>
> /* see get_from_any_partial() for the defrag ratio description */
> if (!s->remote_node_defrag_ratio ||
> @@ -7475,7 +7499,8 @@ __refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min
> do {
> cpuset_mems_cookie = read_mems_allowed_begin();
> zonelist = node_zonelist(mempolicy_slab_node(), gfp);
> - for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
> + for_each_zone_zonelist_nodemask(zone, z, zonelist,
> + highest_zoneidx, nodemask) {
> struct kmem_cache_node *n;
> unsigned int r;
prev parent reply other threads:[~2026-11-18 14:24 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 6:52 Andrea Righi
2026-11-10 12:27 ` Jan Kara
2026-11-11 6:48 ` Andrea Righi
2026-11-11 8:12 ` Vlastimil Babka (SUSE)
2026-11-18 13:44 ` Andrea Righi
2026-11-18 14:24 ` Vlastimil Babka (SUSE) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=eae20ada-01a0-4583-8c65-8f7b5c71cd0f@kernel.org \
--to=vbabka@kernel.org \
--cc=arighi@nvidia.com \
--cc=brauner@kernel.org \
--cc=cl@gentwo.org \
--cc=gourry@gourry.net \
--cc=hao.li@linux.dev \
--cc=harry@kernel.org \
--cc=jack@suse.cz \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ltrager@nvidia.com \
--cc=qualiu@nvidia.com \
--cc=sforshee@nvidia.com \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®