From: Christoph Lameter <cl@linux-foundation.org>
To: David Rientjes <rientjes@google.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
Andrew Morton <akpm@linux-foundation.org>,
Pekka Enberg <penberg@cs.helsinki.fi>,
Matt Mackall <mpm@selenic.com>, Paul Menage <menage@google.com>,
Randy Dunlap <randy.dunlap@oracle.com>,
linux-kernel@vger.kernel.org
Subject: Re: [patch -mm] cpusets: add memory_slab_hardwall flag
Date: Mon, 9 Mar 2009 17:14:07 -0400 (EDT) [thread overview]
Message-ID: <alpine.DEB.1.10.0903091706540.1861@qirst.com> (raw)
In-Reply-To: <alpine.DEB.2.00.0903091313090.27253@chino.kir.corp.google.com>
On Mon, 9 Mar 2009, David Rientjes wrote:
> On Mon, 9 Mar 2009, KOSAKI Motohiro wrote:
>
> > My question mean, Why anyone need isolation?
> > your patch insert new branch into hotpath.
> > then, it makes slower hotpath a abit although a user don't use this feature.
> On large NUMA machines, it is currently possible for a very large
> percentage (if not all) of your slab allocations to come from memory that
> is distant from your application's set of allowable cpus. Such
> allocations that are long-lived would benefit from having affinity to
> those processors. Again, this is the typical use case for cpusets: to
> bind memory nodes to groups of cpus with affinity to it for the tasks
> attached to the cpuset.
Can you show us a real workload that suffers from this issue?
If you want to make sure that an allocation comes from a certain node then
specifying the node in kmalloc_node() will give you what you want.
> > typically, slab cache don't need strict node binding because
> > inode/dentry touched from multiple cpus.
> This change would obviously require inode and dentry objects to originate
> from a node on the cpuset's set of mems_allowed. That would incur a
> performance penalty if the cpu slab is not from such a node, but that is
> assumed by the user who has enabled the option.
The usage of kernel objects may not be cpuset specific. This is true for
other objects than inode and dentries well.
> > In addition, on large numa systems, slab cache is relatively small
> > than page cache. then this feature's improvement seems relatively small too.
> That's irrelevant, large NUMA machines may still require memory affinity
> to a specific group of cpus, the size of the global slab cache isn't
> important if that's the goal. When the option is enabled for cpusets
> that require that memory locality, we happily trade off partial list
> fragmentation and increased slab allocations for the long-lived local
> allocations.
Other memory may spill over too. F.e. two processes from disjunct cpu sets
cause faults in the same address range (its rather common for this to
happen to glibc code f.e.). Two processes may use another kernel feature
that buffers objects (are you going to want to search the LRU lists for objects
from the right node?)
NUMA affinity is there in the large picture. In detail the allocation
strategies over nodes etc etc may be disturbed by this and that in
particular if processes with disjoint cpusets run on the same processor.
Just dont do it. Dedicate a cpu to a cpuset. Overlapping cpusets can cause
other strange things as well.
next prev parent reply other threads:[~2009-03-09 21:17 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-08 16:27 David Rientjes
2009-03-08 16:53 ` Paul Menage
2009-03-08 21:38 ` David Rientjes
2009-03-09 7:08 ` Paul Menage
2009-03-09 9:18 ` David Rientjes
2009-03-08 17:01 ` Matt Mackall
2009-03-08 21:51 ` David Rientjes
2009-03-09 4:49 ` KOSAKI Motohiro
2009-03-09 9:12 ` David Rientjes
2009-03-09 10:19 ` KOSAKI Motohiro
2009-03-09 20:26 ` David Rientjes
2009-03-09 21:14 ` Christoph Lameter [this message]
2009-03-09 21:31 ` David Rientjes
2009-03-10 20:50 ` Christoph Lameter
2009-03-10 21:08 ` Matt Mackall
2009-03-12 16:08 ` Christoph Lameter
2009-03-12 17:23 ` Matt Mackall
2009-03-10 22:12 ` Paul Menage
2009-03-12 15:51 ` Christoph Lameter
2009-03-09 18:50 ` Christoph Lameter
2009-03-09 20:13 ` David Rientjes
2009-03-10 1:35 ` KOSAKI Motohiro
2009-03-10 2:01 ` David Rientjes
2009-03-10 4:05 ` Matt Mackall
2009-03-10 4:26 ` David Rientjes
2009-03-12 1:03 ` Paul E. McKenney
2009-03-12 5:51 ` David Rientjes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.1.10.0903091706540.1861@qirst.com \
--to=cl@linux-foundation.org \
--cc=akpm@linux-foundation.org \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=menage@google.com \
--cc=mpm@selenic.com \
--cc=penberg@cs.helsinki.fi \
--cc=randy.dunlap@oracle.com \
--cc=rientjes@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®