From: Andi Kleen <ak@suse.de>
To: Ravikiran G Thirumalai <kiran@scalex86.org>,
Andi Kleen <ak@suse.de>,
patches@x86-64.org, linux-kernel@vger.kernel.org
Subject: [PATCH] [7/48] x86_64: Set HASHDIST_DEFAULT to 1 for x86_64 NUMA
Date: Sun, 29 Apr 2007 12:52:55 +0200 (CEST) [thread overview]
Message-ID: <20070429105255.149C515108@wotan.suse.de> (raw)
In-Reply-To: <200704291252.514680000@suse.de>
From: Ravikiran G Thirumalai <kiran@scalex86.org>
Enable system hashtable memory to be distributed among nodes on x86_64 NUMA
Forcing the kernel to use node interleaved vmalloc instead of bootmem for
the system hashtable memory (alloc_large_system_hash) reduces the memory
imbalance on node 0 by around 40MB on a 8 node x86_64 NUMA box:
Before the following patch, on bootup of a 8 node box:
Node 0 MemTotal: 3407488 kB
Node 0 MemFree: 3206296 kB
Node 0 MemUsed: 201192 kB
Node 0 Active: 7012 kB
Node 0 Inactive: 512 kB
Node 0 Dirty: 0 kB
Node 0 Writeback: 0 kB
Node 0 FilePages: 1912 kB
Node 0 Mapped: 420 kB
Node 0 AnonPages: 5612 kB
Node 0 PageTables: 468 kB
Node 0 NFS_Unstable: 0 kB
Node 0 Bounce: 0 kB
Node 0 Slab: 5408 kB
Node 0 SReclaimable: 644 kB
Node 0 SUnreclaim: 4764 kB
After the patch (or using hashdist=1 on the kernel command line):
Node 0 MemTotal: 3407488 kB
Node 0 MemFree: 3247608 kB
Node 0 MemUsed: 159880 kB
Node 0 Active: 3012 kB
Node 0 Inactive: 616 kB
Node 0 Dirty: 0 kB
Node 0 Writeback: 0 kB
Node 0 FilePages: 2424 kB
Node 0 Mapped: 380 kB
Node 0 AnonPages: 1200 kB
Node 0 PageTables: 396 kB
Node 0 NFS_Unstable: 0 kB
Node 0 Bounce: 0 kB
Node 0 Slab: 6304 kB
Node 0 SReclaimable: 1596 kB
Node 0 SUnreclaim: 4708 kB
I guess it is a good idea to keep HASHDIST_DEFAULT "on" for x86_64 NUMA
since x86_64 has no dearth of vmalloc space? Or maybe enable hash
distribution for all 64bit NUMA arches? The following patch does it only
for x86_64.
I ran a HPC MPI benchmark -- 'Ansys wingsolid', which takes up quite a bit of
memory and uses up tlb entries. This was on a 4 way, 2 socket
Tyan AMD box (non vsmp), with 8G total memory (4G pernode).
The results with and without hash distribution are:
1. Vanilla - runtime of 1188.000s
2. With hashdist=1 runtime of 1154.000s
Oprofile output for the duration of run is:
1. Vanilla:
PU: AMD64 processors, speed 2411.16 MHz (estimated)
Counted L1_AND_L2_DTLB_MISSES events (L1 and L2 DTLB misses) with a unit
mask of 0x00 (No unit mask) count 500
samples % app name symbol name
163054 6.5513 libansys1.so MultiFront::decompose(int, int,
Elemset *, int *, int, int, int)
162061 6.5114 libansys3.so blockSaxpy6L_fd
162042 6.5107 libansys3.so blockInnerProduct6L_fd
156286 6.2794 libansys3.so maxb33_
87879 3.5309 libansys1.so elmatrixmultpcg_
84857 3.4095 libansys4.so saxpy_pcg
58637 2.3560 libansys4.so .st4560
46612 1.8728 libansys4.so .st4282
43043 1.7294 vmlinux-t copy_user_generic_string
41326 1.6604 libansys3.so blockSaxpyBackSolve6L_fd
41288 1.6589 libansys3.so blockInnerProductBackSolve6L_fd
2. With hashdist=1
CPU: AMD64 processors, speed 2411.13 MHz (estimated)
Counted L1_AND_L2_DTLB_MISSES events (L1 and L2 DTLB misses) with a unit
mask of 0x00 (No unit mask) count 500
samples % app name symbol name
162993 6.9814 libansys1.so MultiFront::decompose(int, int,
Elemset *, int *, int, int, int)
160799 6.8874 libansys3.so blockInnerProduct6L_fd
160459 6.8729 libansys3.so blockSaxpy6L_fd
156018 6.6826 libansys3.so maxb33_
84700 3.6279 libansys4.so saxpy_pcg
83434 3.5737 libansys1.so elmatrixmultpcg_
58074 2.4875 libansys4.so .st4560
46000 1.9703 libansys4.so .st4282
41166 1.7632 libansys3.so blockSaxpyBackSolve6L_fd
41033 1.7575 libansys3.so blockInnerProductBackSolve6L_fd
35762 1.5318 libansys1.so inner_product_sub
35591 1.5245 libansys1.so inner_product_sub2
28259 1.2104 libansys4.so addVectors
Signed-off-by: Pravin B. Shelar <pravin.shelar@calsoftinc.com>
Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>
Signed-off-by: Shai Fultheim <shai@scalex86.org>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: Christoph Lameter <clameter@engr.sgi.com>
Cc: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
include/linux/bootmem.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
Index: linux/include/linux/bootmem.h
===================================================================
--- linux.orig/include/linux/bootmem.h
+++ linux/include/linux/bootmem.h
@@ -122,9 +122,9 @@ extern void *alloc_large_system_hash(con
#define HASH_EARLY 0x00000001 /* Allocating during early boot? */
/* Only NUMA needs hash distribution.
- * IA64 is known to have sufficient vmalloc space.
+ * IA64 and x86_64 have sufficient vmalloc space.
*/
-#if defined(CONFIG_NUMA) && defined(CONFIG_IA64)
+#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
#define HASHDIST_DEFAULT 1
#else
#define HASHDIST_DEFAULT 0
next prev parent reply other threads:[~2007-04-29 11:05 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-04-29 10:52 [PATCH] [0/48] x86 candidate patches for review III: various stuff Andi Kleen
2007-04-29 10:52 ` [PATCH] [1/48] x86_64: fix x86_64-mm-sched-clock-share Andi Kleen
2007-04-29 10:52 ` [PATCH] [2/48] i386: Rewrite sched_clock Andi Kleen
2007-04-29 10:52 ` [PATCH] [3/48] x86_64: Use new shared sched_clock in x86-64 too Andi Kleen
2007-04-29 10:52 ` [PATCH] [4/48] x86_64: Don't disable basic block reordering Andi Kleen
2007-04-29 10:52 ` [PATCH] [5/48] x86_64: Allow sys_uselib unconditionally Andi Kleen
2007-04-29 10:52 ` [PATCH] [6/48] x86_64: Minor white space cleanup in traps.c Andi Kleen
2007-04-29 10:52 ` Andi Kleen [this message]
2007-04-29 10:52 ` [PATCH] [8/48] i386: modpost apic related warning fixes Andi Kleen
2007-04-29 10:52 ` [PATCH] [9/48] i386: make struct vmi_ops static Andi Kleen
2007-04-29 10:52 ` [PATCH] [10/48] i386: type cast clean up for find_next_zero_bit Andi Kleen
2007-04-29 10:52 ` [PATCH] [11/48] i386: workaround for a -Wmissing-prototypes warning Andi Kleen
2007-04-29 10:53 ` [PATCH] [12/48] x86: Log reason why TSC was marked unstable Andi Kleen
2007-04-29 10:53 ` [PATCH] [13/48] x86_64: fix ia32_binfmt.c build error Andi Kleen
2007-04-29 10:53 ` [PATCH] [14/48] x86_64: remove extra smp_processor_id calling Andi Kleen
2007-04-29 10:53 ` [PATCH] [15/48] x86_64: make simnow_init() static Andi Kleen
2007-04-29 10:53 ` [PATCH] [16/48] i386: vmi_pmd_clear() static Andi Kleen
2007-04-29 10:53 ` [PATCH] [18/48] x86_64: configurable fake numa node sizes Andi Kleen
2007-04-29 10:53 ` [PATCH] [19/48] x86_64: split remaining fake nodes equally Andi Kleen
2007-04-29 10:53 ` [PATCH] [20/48] x86_64: fixed size remaining fake nodes Andi Kleen
2007-04-29 10:53 ` [PATCH] [21/48] x86: remove constant_tsc reporting from /proc/cpuinfo' power flags Andi Kleen
2007-04-29 10:53 ` [PATCH] [22/48] x86_64: fake numa for cpusets document Andi Kleen
2007-04-29 10:53 ` [PATCH] [23/48] i386: VDSO_PRELINK warning fix Andi Kleen
2007-04-29 10:53 ` [PATCH] [24/48] i386: Initialize esp0 properly all the time Andi Kleen
2007-04-29 10:53 ` [PATCH] [25/48] x86_64: Introduce load_TLS to the "for" loop Andi Kleen
2007-04-29 10:53 ` [PATCH] [26/48] x86_64: Clarify CONFIG_REORDER explanation Andi Kleen
2007-04-29 10:53 ` [PATCH] [27/48] i386: Allow i386 crash kernels to handle x86_64 dumps Andi Kleen
2007-04-29 10:53 ` [PATCH] [28/48] i386: prevent ACPI quirk warning mass spamming in logs Andi Kleen
2007-04-29 10:53 ` [PATCH] [29/48] x86: add command line length to boot protocol Andi Kleen
2007-04-29 10:53 ` [PATCH] [30/48] i386: Use per-cpu variables for GDT, PDA Andi Kleen
2007-04-29 10:53 ` [PATCH] [31/48] i386: Use per-cpu GDT immediately upon boot Andi Kleen
2007-04-29 10:53 ` [PATCH] [32/48] i386: clean up cpu_init() Andi Kleen
2007-04-29 10:53 ` [PATCH] [33/48] i386: Rename boot_gdt_table to boot_gdt Andi Kleen
2007-04-29 10:53 ` [PATCH] [34/48] i386: rationalize paravirt wrappers Andi Kleen
2007-04-29 10:53 ` [PATCH] [35/48] x86: tighten kernel image page access rights Andi Kleen
2007-04-29 10:53 ` [PATCH] [36/48] i386: get rid of unused variables Andi Kleen
2007-04-29 10:53 ` [PATCH] [37/48] i386: ignore vgacon if hardware not present Andi Kleen
2007-04-29 12:17 ` Antonino A. Daplas
2007-04-29 13:24 ` Andi Kleen
2007-04-29 14:10 ` Antonino A. Daplas
2007-04-29 14:16 ` Andi Kleen
2007-04-29 17:16 ` Jeremy Fitzhardinge
2007-04-29 17:39 ` Andi Kleen
2007-04-29 10:53 ` [PATCH] [38/48] x86_64: Remove unused stext symbol Andi Kleen
2007-04-29 10:53 ` [PATCH] [39/48] i386: remove the APM_RTC_IS_GMT config option Andi Kleen
2007-04-29 10:53 ` [PATCH] [40/48] x86_64: use lru instead of page->index and page->private for pgd lists management Andi Kleen
2007-04-29 10:53 ` [PATCH] [41/48] x86: sys_ioperm() prototype cleanup Andi Kleen
2007-04-29 10:53 ` [PATCH] [42/48] x86: remove UNEXPECTED_IO_APIC() Andi Kleen
2007-04-29 10:53 ` [PATCH] [43/48] x86_64: fix vtime() vsyscall Andi Kleen
2007-04-29 10:53 ` [PATCH] [44/48] x86_64: vsyscall_gtod_data diet and vgettimeofday() fix Andi Kleen
2007-04-29 10:53 ` [PATCH] [45/48] x86_64: Inhibit machine from asserting an NMI when doing Alt-SysRq-M operation Andi Kleen
2007-04-29 10:53 ` [PATCH] [46/48] x86_64: adjust EDID retrieval Andi Kleen
2007-04-29 12:18 ` Antonino A. Daplas
2007-04-29 10:53 ` [PATCH] [47/48] x86_64: Fix "Section mismatch" compile warning Andi Kleen
2007-04-29 10:53 ` [PATCH] [48/48] i386: cleanup GDT Access Andi Kleen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070429105255.149C515108@wotan.suse.de \
--to=ak@suse.de \
--cc=kiran@scalex86.org \
--cc=linux-kernel@vger.kernel.org \
--cc=patches@x86-64.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®