From: Mike Galbraith <bitbucket@online.de>
To: Rik van Riel <riel@surriel.com>
Cc: torvalds@linux-foundation.org, davidlohr.bueso@hp.com,
linux-kernel@vger.kernel.org, akpm@linux-foundation.org,
hhuang@redhat.com, jason.low2@hp.com, walken@google.com,
lwoodman@redhat.com, chegu_vinod@hp.com
Subject: Re: ipc,sem: sysv semaphore scalability
Date: Fri, 22 Mar 2013 08:30:32 +0100 [thread overview]
Message-ID: <1363937432.4581.15.camel@marge.simpson.net> (raw)
In-Reply-To: <1363809337-29718-1-git-send-email-riel@surriel.com>
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote:
> Include lkml in the CC: this time... *sigh*
> ---8<---
>
> This series makes the sysv semaphore code more scalable,
> by reducing the time the semaphore lock is held, and making
> the locking more scalable for semaphore arrays with multiple
> semaphores.
>
> The first four patches were written by Davidlohr Buesso, and
> reduce the hold time of the semaphore lock.
>
> The last three patches change the sysv semaphore code locking
> to be more fine grained, providing a performance boost when
> multiple semaphores in a semaphore array are being manipulated
> simultaneously.
>
> On a 24 CPU system, performance numbers with the semop-multi
> test with N threads and N semaphores, look like this:
>
> vanilla Davidlohr's Davidlohr's + Davidlohr's +
> threads patches rwlock patches v3 patches
> 10 610652 726325 1783589 2142206
> 20 341570 365699 1520453 1977878
> 30 288102 307037 1498167 2037995
> 40 290714 305955 1612665 2256484
> 50 288620 312890 1733453 2650292
> 60 289987 306043 1649360 2388008
> 70 291298 306347 1723167 2717486
> 80 290948 305662 1729545 2763582
> 90 290996 306680 1736021 2757524
> 100 292243 306700 1773700 3059159
Hi Rik,
I plugged this set into enterprise -rt kernel, and beat on four boxen.
I ran into no trouble while giving boxen a generic drubbing fwtw.
Some semop-multi -rt numbers for an abby-normal 8 node box, and a
mundane 4 node box below.
32 cores+SMT, 3.0.68-rt92
numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 32733 MB
node 0 free: 29910 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 32768 MB
node 1 free: 30396 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 32768 MB
node 2 free: 30568 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 32767 MB
node 3 free: 28706 MB
node distances:
node 0 1 2 3
0: 10 21 21 21
1: 21 10 21 21
2: 21 21 10 21
3: 21 21 21 10
SCHED_OTHER
-v3 set +v3 set
threads
10 438485 1744599
20 376411 1580813
30 396639 1546841
40 449062 2152861
50 477594 2431344
60 446453 1874476
70 578705 2047884
80 607928 2144898
90 662136 2171074
100 622889 2295879
200 709668 2867273
300 661902 3008695
400 641758 3273250
500 614117 3403775
SCHED_FIFO
-v3 set +v3 set
threads
10 158656 914343
20 99784 1133775
30 84245 1099604
40 89725 1756577
50 85697 1607893
60 84033 1467272
70 86833 1979405
80 91451 1922250
90 92484 1990699
100 90691 2067705
200 103692 2308652
300 101161 2376921
400 103722 2417580
500 108838 2443349
64 core box (poor thing, smi free zone though), 3.0.68-rt92
numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 0 size: 8181 MB
node 0 free: 6309 MB
node distances:
node 0
0: 10
SCHED_OTHER
-v3 set +v3 set
threads
10 677534 2304417
20 451507 1873474
30 356876 1542688
40 329585 1500392
50 415761 1318676
60 403482 1380644
70 394089 1185165
80 407408 1191834
90 445764 1249156
100 430823 1245573
200 425470 1421686
300 427092 1480379
400 497900 1516599
500 421927 1551309
SCHED_FIFO
10 323560 1882543
20 226257 1806862
30 187851 1263163
40 205337 881331
50 196806 766666
60 193218 612709
70 209432 1241797
80 240445 1269146
90 219865 1482649
100 227970 1473038
200 201354 1719977
300 183728 1823074
400 175051 1808792
500 243708 1849803
next prev parent reply other threads:[~2013-03-22 7:30 UTC|newest]
Thread overview: 129+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-03-20 19:55 Rik van Riel
2013-03-20 19:55 ` [PATCH 1/7] ipc: remove bogus lock comment for ipc_checkid Rik van Riel
2013-03-20 19:55 ` [PATCH 2/7] ipc: introduce obtaining a lockless ipc object Rik van Riel
2013-03-20 19:55 ` [PATCH 3/7] ipc: introduce lockless pre_down ipcctl Rik van Riel
2013-03-20 19:55 ` [PATCH 4/7] ipc,sem: do not hold ipc lock more than necessary Rik van Riel
2013-03-20 19:55 ` [PATCH 5/7] ipc,sem: open code and rename sem_lock Rik van Riel
2013-03-22 1:14 ` Davidlohr Bueso
2013-03-20 19:55 ` [PATCH 6/7] ipc,sem: have only one list in struct sem_queue Rik van Riel
2013-03-22 1:14 ` Davidlohr Bueso
2013-03-20 19:55 ` [PATCH 7/7] ipc,sem: fine grained locking for semtimedop Rik van Riel
2013-03-22 1:14 ` Davidlohr Bueso
2013-03-22 23:01 ` Michel Lespinasse
2013-03-22 23:38 ` Rik van Riel
2013-03-22 23:42 ` [PATCH 7/7 part3] fix for sem_lock Rik van Riel
2013-03-20 20:49 ` ipc,sem: sysv semaphore scalability Linus Torvalds
2013-03-20 20:56 ` Linus Torvalds
2013-03-20 20:57 ` Davidlohr Bueso
2013-03-21 21:10 ` Andrew Morton
2013-03-21 21:47 ` Peter Hurley
2013-03-21 21:50 ` Peter Hurley
2013-03-21 22:01 ` Andrew Morton
2013-03-22 3:38 ` Rik van Riel
2013-03-26 19:28 ` Dave Jones
2013-03-26 19:43 ` Andrew Morton
2013-03-29 16:17 ` Dave Jones
2013-03-29 18:00 ` Linus Torvalds
2013-03-29 18:04 ` Dave Jones
2013-03-29 18:10 ` Linus Torvalds
2013-03-29 18:43 ` Linus Torvalds
2013-03-29 19:06 ` Dave Jones
2013-03-29 19:13 ` Linus Torvalds
2013-03-29 19:26 ` Linus Torvalds
2013-03-29 19:36 ` Peter Hurley
2013-04-02 16:08 ` Sasha Levin
2013-04-02 17:24 ` Linus Torvalds
2013-04-02 17:52 ` Linus Torvalds
2013-04-02 19:53 ` Sasha Levin
2013-04-02 20:00 ` Dave Jones
2013-03-29 19:33 ` Peter Hurley
2013-03-29 19:54 ` Linus Torvalds
2013-04-01 7:40 ` Stanislav Kinsbursky
2013-03-29 20:41 ` Linus Torvalds
2013-03-29 21:12 ` Linus Torvalds
2013-03-29 23:16 ` Linus Torvalds
2013-03-30 1:36 ` Emmanuel Benisty
2013-03-30 2:08 ` Davidlohr Bueso
2013-03-30 3:02 ` Emmanuel Benisty
2013-03-30 3:46 ` Linus Torvalds
2013-03-30 4:33 ` Emmanuel Benisty
2013-03-30 5:10 ` Linus Torvalds
2013-03-30 5:57 ` Emmanuel Benisty
2013-03-30 17:22 ` Linus Torvalds
2013-03-31 2:38 ` Emmanuel Benisty
2013-03-31 5:01 ` Davidlohr Bueso
2013-03-31 13:45 ` Rik van Riel
2013-03-31 17:10 ` Linus Torvalds
2013-03-31 17:02 ` Emmanuel Benisty
2013-03-30 2:09 ` Linus Torvalds
2013-03-30 2:55 ` Davidlohr Bueso
2013-03-29 19:01 ` Dave Jones
2013-05-03 15:03 ` Peter Hurley
2013-03-22 1:12 ` Davidlohr Bueso
2013-03-22 1:23 ` Linus Torvalds
2013-03-22 3:40 ` Rik van Riel
2013-03-22 7:30 ` Mike Galbraith [this message]
2013-03-22 11:04 ` Emmanuel Benisty
2013-03-22 15:37 ` Linus Torvalds
2013-03-23 3:19 ` Emmanuel Benisty
2013-03-23 19:45 ` Linus Torvalds
2013-03-24 13:46 ` Emmanuel Benisty
2013-03-24 17:10 ` Linus Torvalds
2013-03-25 13:47 ` Emmanuel Benisty
2013-03-25 14:00 ` Rik van Riel
2013-03-25 14:03 ` Rik van Riel
2013-03-25 15:20 ` Emmanuel Benisty
2013-03-25 15:53 ` Rik van Riel
2013-03-25 17:09 ` Emmanuel Benisty
2013-03-25 14:01 ` Rik van Riel
2013-03-25 14:21 ` Emmanuel Benisty
2013-03-26 17:59 ` Davidlohr Bueso
2013-03-26 18:14 ` Rik van Riel
2013-03-26 18:35 ` Andrew Morton
2013-04-16 23:30 ` Andrew Morton
2013-05-04 15:55 ` Jörn Engel
2013-05-04 18:12 ` Borislav Petkov
2013-05-06 14:47 ` Jörn Engel
2013-03-22 17:51 ` Davidlohr Bueso
2013-03-25 20:21 ` Sasha Levin
2013-03-25 20:38 ` [PATCH -mm -next] ipc,sem: fix lockdep false positive Rik van Riel
2013-03-25 21:42 ` Michel Lespinasse
2013-03-25 21:51 ` Michel Lespinasse
2013-03-25 21:56 ` Sasha Levin
2013-03-25 21:52 ` Sasha Levin
2013-03-26 13:19 ` Peter Zijlstra
2013-03-26 13:40 ` Michel Lespinasse
2013-03-26 14:27 ` Peter Zijlstra
2013-03-26 15:19 ` Rik van Riel
2013-03-27 8:40 ` Peter Zijlstra
2013-03-27 8:42 ` Peter Zijlstra
2013-03-27 11:22 ` Michel Lespinasse
2013-03-27 12:02 ` Peter Zijlstra
2013-03-27 20:00 ` Rik van Riel
2013-03-28 20:23 ` [PATCH v2 " Rik van Riel
2013-03-29 2:50 ` Michel Lespinasse
2013-03-29 9:57 ` Peter Zijlstra
2013-03-29 13:21 ` Michel Lespinasse
2013-03-29 12:07 ` Rik van Riel
2013-03-29 13:08 ` Michel Lespinasse
2013-03-29 13:24 ` Rik van Riel
2013-03-29 13:55 ` [PATCH v3 " Rik van Riel
2013-03-29 13:59 ` Michel Lespinasse
2013-03-26 14:25 ` [PATCH " Rik van Riel
2013-03-26 17:33 ` ipc,sem: sysv semaphore scalability Sasha Levin
2013-03-26 17:51 ` Davidlohr Bueso
2013-03-26 18:07 ` Sasha Levin
2013-03-26 18:17 ` Rik van Riel
2013-03-26 20:00 ` [PATCH -mm -next] ipc,sem: untangle RCU locking with find_alloc_undo Rik van Riel
2013-04-05 4:38 ` Mike Galbraith
2013-04-05 13:21 ` Rik van Riel
2013-04-05 16:26 ` Mike Galbraith
2013-04-16 12:37 ` Mike Galbraith
2013-03-26 17:55 ` ipc,sem: sysv semaphore scalability Paul E. McKenney
2013-03-28 15:32 ` [PATCH -mm -next] ipc,sem: untangle RCU locking with find_alloc_undo Rik van Riel
2013-03-28 21:05 ` Davidlohr Bueso
2013-03-29 1:00 ` Michel Lespinasse
2013-03-29 1:14 ` Sasha Levin
2013-03-30 13:35 ` Sasha Levin
2013-03-31 1:30 ` Rik van Riel
2013-03-31 4:09 ` Davidlohr Bueso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1363937432.4581.15.camel@marge.simpson.net \
--to=bitbucket@online.de \
--cc=akpm@linux-foundation.org \
--cc=chegu_vinod@hp.com \
--cc=davidlohr.bueso@hp.com \
--cc=hhuang@redhat.com \
--cc=jason.low2@hp.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lwoodman@redhat.com \
--cc=riel@surriel.com \
--cc=torvalds@linux-foundation.org \
--cc=walken@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome