mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Robin Holt <holt@sgi.com>
To: Samuel Thibault <samuel.thibault@ens-lyon.org>,
	linux-kernel@vger.kernel.org, lse-tech@lists.sourceforge.net
Cc: Jack Steiner <steiner@sgi.com>
Subject: Re: idle task's task_t allocation on NUMA machines
Date: Thu, 18 Aug 2005 13:27:19 -0500	[thread overview]
Message-ID: <20050818182718.GA21490@lnx-holt.americas.sgi.com> (raw)
In-Reply-To: <20050818140829.GB8123@implementation.labri.fr>

On Thu, Aug 18, 2005 at 04:08:29PM +0200, Samuel Thibault wrote:
> A solution would be to add to copy_process(), dup_task_struct(),
> alloc_task_struct() and kmem_cache_alloc() the node number on which
> allocation should be performed. This might also be useful if performing
> node load balancing at fork(): one could then allocate task_t directly
> on the new node. It might also be useful when allocating data for
> another node.

Can this be abstracted some?

Let me start with some background.  SGI has a kernel addition we made
on our previous kernel release something we called dplace.  It has a
userland piece and a library which gets configuration information passed
into a kernel driver.

Inside the kernel, we used Process Aggregates (pagg as found on
oss.sgi.com) to track children of a starting process and migrate them
to a desired cpu.

The problem we have with this method is the callout to pagg happens
far too late after the fork to help with some of the more important
user structures like page tables.  We find that most processes have
their pgd and many parts of the pmd allocated remotely.  Although it
is not a significant source of NUMA traffic, it does cause variability
in process run times which becomes exaggerated on larger MPI jobs which
rendezvous at a barrier.

It would be nice to be able to, early in fork, decide on a destination
numa node and cpu list for the task.  If this were done, then changing
allocation of structures like the task_t and page tables could be handled
on a case-by-case basis as we see benefit.  Additionally, it would be
nice if we could make the placement decision logic provide a callback
so we could add tailored placement.

I realize this is a very vague sketch of what I think needs to be done,
but I am sort of in a rush right now and wanted to at least start the
discussion.

Thanks,
Robin

      parent reply	other threads:[~2005-08-18 18:29 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2005-08-18 14:08 Samuel Thibault
2005-08-18 15:18 ` Eric Dumazet
2005-08-18 15:39   ` Samuel Thibault
2005-08-18 17:20     ` Christoph Lameter
2005-08-18 19:49   ` Samuel Thibault
2005-08-18 20:02     ` Samuel Thibault
2005-08-18 21:28       ` [Lse-tech] " Martin J. Bligh
2005-08-18 21:32       ` Andi Kleen
2005-08-18 18:27 ` Robin Holt [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20050818182718.GA21490@lnx-holt.americas.sgi.com \
    --to=holt@sgi.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lse-tech@lists.sourceforge.net \
    --cc=samuel.thibault@ens-lyon.org \
    --cc=steiner@sgi.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®