From: "Serge E. Hallyn" <serge@hallyn.com>
To: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Cc: Christoph Lameter <clameter@sgi.com>,
"Serge E. Hallyn" <serge@hallyn.com>,
Dhaval Giani <dhaval@linux.vnet.ibm.com>,
bob.picco@hp.com, nacc@us.ibm.com,
kamezawa.hiroyu@jp.fujitsu.com, mel@skynet.ie,
akpm@linux-foundation.org, Balbir Singh <balbir@in.ibm.com>,
Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>,
lkml <linux-kernel@vger.kernel.org>,
ckrm-tech <ckrm-tech@lists.sourceforge.net>
Subject: Re: Regression in 2.6.23-rc2-mm2, mounting cpusets causes a hang
Date: Wed, 15 Aug 2007 09:31:25 -0500 [thread overview]
Message-ID: <20070815143125.GA11582@vino.hallyn.com> (raw)
In-Reply-To: <1187185392.5422.13.camel@localhost>
Quoting Lee Schermerhorn (Lee.Schermerhorn@hp.com):
> On Tue, 2007-08-14 at 14:56 -0700, Christoph Lameter wrote:
> > On Tue, 14 Aug 2007, Lee Schermerhorn wrote:
> >
> > > > Ok then you did not have a NUMA system configured. So its okay for the
> > > > dummies to ignore the stuff. CONFIG_NODES_SHIFT is a constant and does not
> > > > change. The first bit is always set.
> > >
> > > The first bit [node 0] is only set for the N_ONLINE [and N_POSSIBLE]
> > > mask. We could add the static init for the other masks, but since
> > > non-numa platforms are going through the __build_all_zonelists, they
> > > might as well set the MEMORY bits explicitly. Or, maybe you'll
> > > disagree ;-).
> >
> > The bitmaps can be completely ignored if !NUMA.
> >
> > In the non NUMA case we define
> >
> > static inline int node_state(int node, enum node_states state)
> > {
> > return node == 0;
> > }
> >
> > So its always true for node 0. The "bit" is set.
>
> The issue is with the N_*_MEMORY masks. They don't get initialized
> properly because node_set_state() is a no-op if !NUMA. So, where we
> look for intersections with or where we AND with the N_*_MEMORY masks we
> get the empty set.
>
> >
> > We are trying to get cpusets to work with !NUMA?
>
>
> Well, yes. In Serge's case, he's trying to use cpusets with !NUMA.
> He'll have to comment on the reasons for that. Looking at all of the
So I can lock a container to a cpu on a non-numa machine.
> #ifdefs and init/Kconfig, CPUSET does not depend on NUMA--only SMP and
> CONTAINERS [altho' methinks CPUSET should select CONTAINERS rather than
> depend on it...]. So, you can use cpusets to partition of cpus in
> non-NUMA configs.
>
> In the more general case, tho', I'm looking at all uses of the
> node_online_map and for_each_online_node, for instances where they
> should be replaced with one of the *_MEMORY masks. IMO, generic code
> that is compiled independent of any CONFIG option, like NUMA, should
> just work, independent of the config. Currently, as Serge has shown,
> this is not the case. So, I think we should fix the *_MEMORY maps to be
> correctly populated in both the NUMA and !NUMA cases. A couple of
> options:
>
> 1) just use node_set() when populating the masks,
>
> 2) initialize all masks to include at least cpu/node 0 in the !NUMA
> case.
>
> Serge chose #1 to fix his problem. I followed his lead to fix the other
> 2 places where node_set_state() was being used to initialize the NORMAL
> memory node mask and the CPU node mask. This will add a few unnecessary
> instructions to !NUMA configs, so we could change to #2.
>
> Thoughts?
Paul, is the mems stuff in cpusets only really useful for NUMA cases?
(I think it is... but am not sure) If so I suppose one alternative
could be to just disable that when !NUMA. But disabling cpusets when
!NUMA is completely wrong.
I personally would think that 1) is still the best option. Otherwise
the action
echo $SOME_CPU > /cpusets/set1/cpu
echo $SOME_CPU > /cpusets/set1/mems
works on a numa machine, and is wrong on a non-numa machine. With
option 1, the second part doesn't actually restrict the memory, but
at least /cpusets/set1/mems exists and $SOME_CPU doesn't have to be 0 to
be valid.
-serge
next prev parent reply other threads:[~2007-08-15 14:31 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-12 15:21 Dhaval Giani
2007-08-13 20:12 ` Serge E. Hallyn
2007-08-14 15:03 ` Lee Schermerhorn
2007-08-14 18:03 ` Serge E. Hallyn
2007-08-14 18:13 ` Lee Schermerhorn
2007-08-14 19:23 ` Serge E. Hallyn
2007-08-14 20:49 ` Serge E. Hallyn
2007-08-14 21:07 ` Lee Schermerhorn
2007-08-14 21:28 ` Christoph Lameter
2007-08-14 21:41 ` Lee Schermerhorn
2007-08-14 21:56 ` Christoph Lameter
2007-08-15 13:43 ` Lee Schermerhorn
2007-08-15 14:31 ` Serge E. Hallyn [this message]
2007-08-15 16:23 ` Lee Schermerhorn
2007-08-15 16:31 ` Serge E. Hallyn
2007-08-15 16:52 ` Dhaval Giani
2007-08-15 17:08 ` Serge E. Hallyn
2007-08-15 18:07 ` Lee Schermerhorn
2007-08-15 20:39 ` Christoph Lameter
2007-08-16 13:26 ` Lee Schermerhorn
2007-08-16 19:14 ` Christoph Lameter
2007-08-15 20:38 ` Christoph Lameter
2007-08-15 16:31 ` Paul Jackson
2007-08-15 16:29 ` Paul Jackson
2007-08-15 17:12 ` Serge E. Hallyn
2007-08-15 18:00 ` Lee Schermerhorn
2007-08-15 18:33 ` Andrew Morton
2007-08-31 16:54 ` [ckrm-tech] " Paul Menage
2007-08-16 4:46 ` Paul Menage
2007-08-15 20:36 ` Christoph Lameter
2007-08-15 20:48 ` Lee Schermerhorn
2007-08-16 2:36 ` [ckrm-tech] " Paul Jackson
2007-08-14 21:37 ` Lee Schermerhorn
2007-08-14 21:39 ` Christoph Lameter
2007-08-14 21:01 ` Lee Schermerhorn
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070815143125.GA11582@vino.hallyn.com \
--to=serge@hallyn.com \
--cc=Lee.Schermerhorn@hp.com \
--cc=akpm@linux-foundation.org \
--cc=balbir@in.ibm.com \
--cc=bob.picco@hp.com \
--cc=ckrm-tech@lists.sourceforge.net \
--cc=clameter@sgi.com \
--cc=dhaval@linux.vnet.ibm.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mel@skynet.ie \
--cc=nacc@us.ibm.com \
--cc=vatsa@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®