* [question] IPC queue filling-up problem?
@ 2007-09-05 13:37 Fortier,Vincent [Montreal]
2007-09-05 14:16 ` Eric Dumazet
0 siblings, 1 reply; 3+ messages in thread
From: Fortier,Vincent [Montreal] @ 2007-09-05 13:37 UTC (permalink / raw)
To: linux-kernel
Hi all,
We are testing new hardware and planning a switch from our old redhat
7.3 to a Debian Etch 4.0 for our radar forecast analysis systems. We
found out that our main IPC dispatcher software module would use 100% of
a CPU all the time and that the IPC queues would fill up quickly on a
2.6 kernel. We first tought that it would be a problem of compatibility
between a 2.4 vs 2.6 IPC calls vs our radar analysis software but after
a lot of work we have been able to test a 2.4 kernel on that same
hardware and got the exact same problem.
So curiously, on our actual systems (see SYSTEM 1 below) our IPC
dispatcher module works like a charm and queues gets near 0. On our
test system which are way more powerfull systems (see SYSTEM 2) our IPC
dispatcher module queue fills up rapidly (depending of the msgmnb queue
size it will wimply take a bit longer to fill).
We have tested both our already compiled binaries from rh73 using gcc
2.9 and a recompiled version of the modules on a debian sarge system and
got the exact same problem on either a Debian Sarge 3.1 (running a 2.4
or 2.6 kernel) and on a Etch 64bit system (using 32bit compat layer)
with a 2.6 kernel. In all cases the queues would simply fill-up.
After strac'ing the module I noticed that the time needed to handle the
signal & ipc calls are way lower on the new system hence I don't see why
the dispatcher queue does fill-up like that?!?!?!
Does anyone experienced something similar? Could this be a kernel issue
vs material, kernel option? Might this be related to libc?
Help / Clues very much appreciated.
Thnx
- vin
Debian Sarge 3.1 kernel 2.4.35 notes:
-------------------------------------
Pre-built 2.4.35.1 kernel + backported mpt fusion & megasas drivers for
Debian Sarge 3.1 available at:
http://linux-dev.qc.ec.gc.ca/kernel/debian/sarge/i386/2.4.35/
Megaraid SAS backport patch for a 2.4.35.1 kernel available at:
http://linux-dev.qc.ec.gc.ca/kernel/patches/megaraid_sas-linux_2.4.35.1-
v00.00.03.09.patch
2.4.35.1 config file:
http://linux-dev.qc.ec.gc.ca/kernel/debian/CONFIG-i686-2.4.35.1-008
RedHat 7.3 kernel 2.4.33 notes:
-------------------------------
Pre-built 2.4.33 kernel for RH73:
http://linux-dev.qc.ec.gc.ca/kernel/redhat/rh73/
2.4.33 config file:
http://linux-dev.qc.ec.gc.ca/kernel/redhat/rh73/config-2.4.33-01.rh73.en
vcanbigmem
SYSTEM INFORMATION:
SYSTEM 1:
---------
HPDL580 G2
Quad Intel Xeon 1.90GHz
4G ram
DRBD disks on dual-gigabit adapter
OS: RedHat 7.3 / kernel: 2.4.33 / libc: 2.2.5 / gcc 2.96
SYSTEM 2:
---------
Dell PE2950
Dual Intel Quad-Core 2.66GHz
16G ram
local 300G 15000 RPM SCSI.
OS1: Debian Etch 4.0 / kernels 2.6.18 -> 2.6.22 / libc 2.3.6 / gcc 4.1.2
OS2: Debian Sarge 3.1 / kernels 2.4.35, 2.6.18 -> 2.6.22 / libc 2.3.2 /
gcc 3.3.5
=============================
SYSTEM 1
=============================
top
---
12:29pm up 147 days, 15:47, 4 users, load average: 3.00, 1.65, 1.56
229 processes: 227 sleeping, 2 running, 0 zombie, 0 stopped
CPU0 states: 48.0% user, 28.0% system, 0.0% nice, 23.0% idle
CPU1 states: 7.0% user, 3.0% system, 0.0% nice, 88.0% idle
CPU2 states: 71.0% user, 13.0% system, 0.0% nice, 15.0% idle
CPU3 states: 6.0% user, 7.0% system, 0.0% nice, 85.0% idle
CPU4 states: 35.0% user, 3.0% system, 0.0% nice, 60.0% idle
CPU5 states: 18.0% user, 3.0% system, 15.0% nice, 77.0% idle
CPU6 states: 11.0% user, 19.0% system, 0.0% nice, 68.0% idle
CPU7 states: 15.0% user, 9.0% system, 0.0% nice, 75.0% idle
Mem: 5952612K av, 5772232K used, 180380K free, 0K shrd, 145816K
buff
Swap: 2097112K av, 7560K used, 2089552K free 4820588K
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
29308 urp 14 5 10920 10M 496 S N 15.0 0.1 0:46
URPDispatcher
ipcs -q
-------
[root@localhost]$ ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
0x41008050 1130594304 urp 660 0 0
0x370041d6 1132494849 urp 660 0 0
0x140041d8 1132396546 urp 660 0 0
0xfa0041d6 1133084675 urp 660 0 0
...
strace -ss -tt -T -v -e trace=ipc
---------------------------------
11:51:45.238031 msgsnd(1104543768, {1, ""...}, 91, IPC_NOWAIT) = 0
<0.000029>
11:51:45.260221 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 82
<0.000092>
11:51:45.280938 msgsnd(1103790090, {1, ""...}, 82, IPC_NOWAIT) = 0
<0.000034>
11:51:45.368730 msgsnd(1104543768, {1, ""...}, 82, IPC_NOWAIT) = 0
<0.023403>
11:51:45.414185 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 84
<0.000020>
11:51:45.483523 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 84
<0.000042>
11:51:45.543332 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 89
<0.000027>
11:51:45.578211 msgsnd(1104543768, {1, ""...}, 89, IPC_NOWAIT) = 0
<0.000057>
11:51:45.592104 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 91
<0.000032>
11:51:45.596233 msgsnd(1103790090, {1, ""...}, 91, IPC_NOWAIT) = 0
<0.000037>
11:51:45.660060 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 101
<0.000053>
11:51:45.717638 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 89
<0.000045>
strace -ss -tt -T -v -e trace=signal
------------------------------------
07:11:27.150753 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000006>
07:11:27.434360 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000016>
07:11:27.717707 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000016>
07:11:28.001054 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000004>
07:11:28.286200 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
07:11:28.569836 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
07:11:28.853549 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000015>
07:11:29.143072 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000011>
07:11:29.436344 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
=============================
SYSTEM 2
=============================
top
---
top - 07:26:58 up 18:18, 4 users, load average: 7.06, 7.20, 7.48
Tasks: 134 total, 6 running, 128 sleeping, 0 stopped, 0 zombie
Cpu0 : 16.6% user, 0.7% system, 0.0% nice, 82.7% idle
Cpu1 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Cpu2 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Cpu3 : 61.8% user, 1.3% system, 0.0% nice, 36.9% idle
Cpu4 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Cpu5 : 54.8% user, 0.3% system, 0.0% nice, 44.9% idle
Cpu6 : 13.3% user, 1.0% system, 0.0% nice, 85.7% idle
Cpu7 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Mem: 16567608k total, 16530352k used, 37256k free, 35068k buffers
Swap: 2048276k total, 4k used, 2048272k free, 15368132k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1963 urp 17 0 11308 11m 548 R 99.9 0.1 18:23.56
URPDispatcher
ipcs -q
-------
[root@localhost /root]# ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
0x41018018 950272 urp 660 23732 276
0x41018024 983041 urp 660 0 0
0x4101800b 1015810 urp 660 0 0
0x41018011 1048579 urp 660 0 0
...
strace -ss -tt -T -v -e trace=ipc
---------------------------------
06:51:20.450471 msgsnd(557073, {1, ""...}, 78, IPC_NOWAIT) = 0
<0.000008>
06:51:20.538398 msgrcv(0, {1, ""...}, 2565, 0, 0) = 82 <0.000007>
06:51:20.569654 msgsnd(32769, {1, ""...}, 81, IPC_NOWAIT) = 0 <0.000007>
06:51:20.706519 msgsnd(360459, {1, ""...}, 81, IPC_NOWAIT) = 0
<0.000014>
06:51:20.832973 msgrcv(0, {1, ""...}, 2565, 0, 0) = 78 <0.000006>
06:51:21.032484 msgsnd(557073, {1, ""...}, 77, IPC_NOWAIT) = 0
<0.000009>
06:51:21.156817 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000007>
06:51:21.447914 msgrcv(0, {1, ""...}, 2565, 0, 0) = 78 <0.000009>
06:51:21.645212 msgsnd(557073, {1, ""...}, 77, IPC_NOWAIT) = 0
<0.000021>
06:51:21.730485 msgrcv(0, {1, ""...}, 2565, 0, 0) = 109 <0.000009>
06:51:22.028139 msgrcv(0, {1, ""...}, 2565, 0, 0) = 76 <0.000009>
06:51:22.229706 msgsnd(557073, {1, ""...}, 75, IPC_NOWAIT) = 0
<0.000013>
06:51:22.316948 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000012>
06:51:22.608471 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000009>
strace -ss -tt -T -v -e trace=signal
------------------------------------
06:59:01.979280 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.036669>
06:59:02.357971 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000012>
06:59:02.648930 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
06:59:02.939051 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000011>
06:59:03.229976 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
06:59:03.521447 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000012>
06:59:03.813863 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
06:59:04.107771 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000010>
06:59:04.404351 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000005>
06:59:04.748842 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
06:59:05.043842 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000006>
06:59:05.353598 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000013>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [question] IPC queue filling-up problem?
2007-09-05 13:37 [question] IPC queue filling-up problem? Fortier,Vincent [Montreal]
@ 2007-09-05 14:16 ` Eric Dumazet
2007-09-05 15:48 ` Fortier,Vincent [Montreal]
0 siblings, 1 reply; 3+ messages in thread
From: Eric Dumazet @ 2007-09-05 14:16 UTC (permalink / raw)
To: Fortier,Vincent [Montreal]; +Cc: linux-kernel
On Wed, 5 Sep 2007 09:37:50 -0400
"Fortier,Vincent [Montreal]" <Vincent.Fortier1@EC.GC.CA> wrote:
>
> Hi all,
>
> We are testing new hardware and planning a switch from our old redhat
> 7.3 to a Debian Etch 4.0 for our radar forecast analysis systems. We
> found out that our main IPC dispatcher software module would use 100% of
> a CPU all the time and that the IPC queues would fill up quickly on a
> 2.6 kernel. We first tought that it would be a problem of compatibility
> between a 2.4 vs 2.6 IPC calls vs our radar analysis software but after
> a lot of work we have been able to test a 2.4 kernel on that same
> hardware and got the exact same problem.
>
> So curiously, on our actual systems (see SYSTEM 1 below) our IPC
> dispatcher module works like a charm and queues gets near 0. On our
> test system which are way more powerfull systems (see SYSTEM 2) our IPC
> dispatcher module queue fills up rapidly (depending of the msgmnb queue
> size it will wimply take a bit longer to fill).
>
> We have tested both our already compiled binaries from rh73 using gcc
> 2.9 and a recompiled version of the modules on a debian sarge system and
> got the exact same problem on either a Debian Sarge 3.1 (running a 2.4
> or 2.6 kernel) and on a Etch 64bit system (using 32bit compat layer)
> with a 2.6 kernel. In all cases the queues would simply fill-up.
>
> After strac'ing the module I noticed that the time needed to handle the
> signal & ipc calls are way lower on the new system hence I don't see why
> the dispatcher queue does fill-up like that?!?!?!
>
> Does anyone experienced something similar? Could this be a kernel issue
> vs material, kernel option? Might this be related to libc?
>
> Help / Clues very much appreciated.
Hi Vincent
top shows that something is eating cpu cycles in User mode on your new platform,
while old platform consumes cycles both in User and System land.
This might be related to some programing error, maybe some spinlock in user mode
or bad multi-threading synchronization, or scheduling assumptions, that break because of
the quad core cpus of your new machine.
So the thread that is supposed to consume IPC messages is not scheduled in time,
because CPU starves. (beware the four cores of each CPU compete for ressources)
You could issue "ps auxm" to check which threads are spining in User mode and try to trace them ?
Eric
>
> Thnx
>
> - vin
>
>
> Debian Sarge 3.1 kernel 2.4.35 notes:
> -------------------------------------
> Pre-built 2.4.35.1 kernel + backported mpt fusion & megasas drivers for
> Debian Sarge 3.1 available at:
> http://linux-dev.qc.ec.gc.ca/kernel/debian/sarge/i386/2.4.35/
> Megaraid SAS backport patch for a 2.4.35.1 kernel available at:
> http://linux-dev.qc.ec.gc.ca/kernel/patches/megaraid_sas-linux_2.4.35.1-
> v00.00.03.09.patch
> 2.4.35.1 config file:
> http://linux-dev.qc.ec.gc.ca/kernel/debian/CONFIG-i686-2.4.35.1-008
>
>
> RedHat 7.3 kernel 2.4.33 notes:
> -------------------------------
> Pre-built 2.4.33 kernel for RH73:
> http://linux-dev.qc.ec.gc.ca/kernel/redhat/rh73/
> 2.4.33 config file:
> http://linux-dev.qc.ec.gc.ca/kernel/redhat/rh73/config-2.4.33-01.rh73.en
> vcanbigmem
>
>
> SYSTEM INFORMATION:
>
> SYSTEM 1:
> ---------
> HPDL580 G2
> Quad Intel Xeon 1.90GHz
> 4G ram
> DRBD disks on dual-gigabit adapter
> OS: RedHat 7.3 / kernel: 2.4.33 / libc: 2.2.5 / gcc 2.96
>
> SYSTEM 2:
> ---------
> Dell PE2950
> Dual Intel Quad-Core 2.66GHz
> 16G ram
> local 300G 15000 RPM SCSI.
> OS1: Debian Etch 4.0 / kernels 2.6.18 -> 2.6.22 / libc 2.3.6 / gcc 4.1.2
> OS2: Debian Sarge 3.1 / kernels 2.4.35, 2.6.18 -> 2.6.22 / libc 2.3.2 /
> gcc 3.3.5
>
>
>
> =============================
> SYSTEM 1
> =============================
>
> top
> ---
> 12:29pm up 147 days, 15:47, 4 users, load average: 3.00, 1.65, 1.56
> 229 processes: 227 sleeping, 2 running, 0 zombie, 0 stopped
> CPU0 states: 48.0% user, 28.0% system, 0.0% nice, 23.0% idle
> CPU1 states: 7.0% user, 3.0% system, 0.0% nice, 88.0% idle
> CPU2 states: 71.0% user, 13.0% system, 0.0% nice, 15.0% idle
> CPU3 states: 6.0% user, 7.0% system, 0.0% nice, 85.0% idle
> CPU4 states: 35.0% user, 3.0% system, 0.0% nice, 60.0% idle
> CPU5 states: 18.0% user, 3.0% system, 15.0% nice, 77.0% idle
> CPU6 states: 11.0% user, 19.0% system, 0.0% nice, 68.0% idle
> CPU7 states: 15.0% user, 9.0% system, 0.0% nice, 75.0% idle
> Mem: 5952612K av, 5772232K used, 180380K free, 0K shrd, 145816K
> buff
> Swap: 2097112K av, 7560K used, 2089552K free 4820588K
> cached
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 29308 urp 14 5 10920 10M 496 S N 15.0 0.1 0:46
> URPDispatcher
>
>
> ipcs -q
> -------
> [root@localhost]$ ipcs -q
>
> ------ Message Queues --------
> key msqid owner perms used-bytes messages
> 0x41008050 1130594304 urp 660 0 0
> 0x370041d6 1132494849 urp 660 0 0
> 0x140041d8 1132396546 urp 660 0 0
> 0xfa0041d6 1133084675 urp 660 0 0
> ...
>
>
> strace -ss -tt -T -v -e trace=ipc
> ---------------------------------
> 11:51:45.238031 msgsnd(1104543768, {1, ""...}, 91, IPC_NOWAIT) = 0
> <0.000029>
> 11:51:45.260221 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 82
> <0.000092>
> 11:51:45.280938 msgsnd(1103790090, {1, ""...}, 82, IPC_NOWAIT) = 0
> <0.000034>
> 11:51:45.368730 msgsnd(1104543768, {1, ""...}, 82, IPC_NOWAIT) = 0
> <0.023403>
> 11:51:45.414185 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 84
> <0.000020>
> 11:51:45.483523 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 84
> <0.000042>
> 11:51:45.543332 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 89
> <0.000027>
> 11:51:45.578211 msgsnd(1104543768, {1, ""...}, 89, IPC_NOWAIT) = 0
> <0.000057>
> 11:51:45.592104 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 91
> <0.000032>
> 11:51:45.596233 msgsnd(1103790090, {1, ""...}, 91, IPC_NOWAIT) = 0
> <0.000037>
> 11:51:45.660060 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 101
> <0.000053>
> 11:51:45.717638 msgrcv(1100054529, {1, ""...}, 2565, 0, 0) = 89
> <0.000045>
>
>
> strace -ss -tt -T -v -e trace=signal
> ------------------------------------
> 07:11:27.150753 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000006>
> 07:11:27.434360 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000016>
> 07:11:27.717707 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000016>
> 07:11:28.001054 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000004>
> 07:11:28.286200 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
> 07:11:28.569836 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
> 07:11:28.853549 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000015>
> 07:11:29.143072 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000011>
> 07:11:29.436344 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
>
>
>
> =============================
> SYSTEM 2
> =============================
>
> top
> ---
>
> top - 07:26:58 up 18:18, 4 users, load average: 7.06, 7.20, 7.48
> Tasks: 134 total, 6 running, 128 sleeping, 0 stopped, 0 zombie
> Cpu0 : 16.6% user, 0.7% system, 0.0% nice, 82.7% idle
> Cpu1 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
> Cpu2 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
> Cpu3 : 61.8% user, 1.3% system, 0.0% nice, 36.9% idle
> Cpu4 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
> Cpu5 : 54.8% user, 0.3% system, 0.0% nice, 44.9% idle
> Cpu6 : 13.3% user, 1.0% system, 0.0% nice, 85.7% idle
> Cpu7 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
> Mem: 16567608k total, 16530352k used, 37256k free, 35068k buffers
> Swap: 2048276k total, 4k used, 2048272k free, 15368132k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 1963 urp 17 0 11308 11m 548 R 99.9 0.1 18:23.56
> URPDispatcher
>
>
> ipcs -q
> -------
> [root@localhost /root]# ipcs -q
>
> ------ Message Queues --------
> key msqid owner perms used-bytes messages
> 0x41018018 950272 urp 660 23732 276
> 0x41018024 983041 urp 660 0 0
> 0x4101800b 1015810 urp 660 0 0
> 0x41018011 1048579 urp 660 0 0
> ...
>
>
> strace -ss -tt -T -v -e trace=ipc
> ---------------------------------
> 06:51:20.450471 msgsnd(557073, {1, ""...}, 78, IPC_NOWAIT) = 0
> <0.000008>
> 06:51:20.538398 msgrcv(0, {1, ""...}, 2565, 0, 0) = 82 <0.000007>
> 06:51:20.569654 msgsnd(32769, {1, ""...}, 81, IPC_NOWAIT) = 0 <0.000007>
> 06:51:20.706519 msgsnd(360459, {1, ""...}, 81, IPC_NOWAIT) = 0
> <0.000014>
> 06:51:20.832973 msgrcv(0, {1, ""...}, 2565, 0, 0) = 78 <0.000006>
> 06:51:21.032484 msgsnd(557073, {1, ""...}, 77, IPC_NOWAIT) = 0
> <0.000009>
> 06:51:21.156817 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000007>
> 06:51:21.447914 msgrcv(0, {1, ""...}, 2565, 0, 0) = 78 <0.000009>
> 06:51:21.645212 msgsnd(557073, {1, ""...}, 77, IPC_NOWAIT) = 0
> <0.000021>
> 06:51:21.730485 msgrcv(0, {1, ""...}, 2565, 0, 0) = 109 <0.000009>
> 06:51:22.028139 msgrcv(0, {1, ""...}, 2565, 0, 0) = 76 <0.000009>
> 06:51:22.229706 msgsnd(557073, {1, ""...}, 75, IPC_NOWAIT) = 0
> <0.000013>
> 06:51:22.316948 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000012>
> 06:51:22.608471 msgrcv(0, {1, ""...}, 2565, 0, 0) = 79 <0.000009>
>
>
> strace -ss -tt -T -v -e trace=signal
> ------------------------------------
> 06:59:01.979280 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.036669>
> 06:59:02.357971 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000012>
> 06:59:02.648930 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
> 06:59:02.939051 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000011>
> 06:59:03.229976 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000009>
> 06:59:03.521447 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000012>
> 06:59:03.813863 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
> 06:59:04.107771 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000010>
> 06:59:04.404351 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000005>
> 06:59:04.748842 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000014>
> 06:59:05.043842 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000006>
> 06:59:05.353598 rt_sigaction(SIGALRM, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, {0x804a888, [ALRM],
> SA_RESTORER|SA_RESTART, 0x4006f678}, 8) = 0 <0.000013>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* RE: [question] IPC queue filling-up problem?
2007-09-05 14:16 ` Eric Dumazet
@ 2007-09-05 15:48 ` Fortier,Vincent [Montreal]
0 siblings, 0 replies; 3+ messages in thread
From: Fortier,Vincent [Montreal] @ 2007-09-05 15:48 UTC (permalink / raw)
To: Eric Dumazet; +Cc: linux-kernel, Ingo Molnar
[-- Attachment #1: Type: text/plain, Size: 4097 bytes --]
> -----Message d'origine-----
> De : linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org] De la part de Eric Dumazet
>
> On Wed, 5 Sep 2007 09:37:50 -0400
> "Fortier,Vincent [Montreal]" <Vincent.Fortier1@EC.GC.CA> wrote:
>
> >
> > Hi all,
> >
> > We are testing new hardware and planning a switch from our old
redhat
> > 7.3 to a Debian Etch 4.0 for our radar forecast analysis systems.
We
> > found out that our main IPC dispatcher software module would use
100%
> > of a CPU all the time and that the IPC queues would fill up quickly
on a
> > 2.6 kernel. We first tought that it would be a problem of
> > compatibility between a 2.4 vs 2.6 IPC calls vs our radar analysis
> > software but after a lot of work we have been able to test a 2.4
> > kernel on that same hardware and got the exact same problem.
> >
> > So curiously, on our actual systems (see SYSTEM 1 below) our IPC
> > dispatcher module works like a charm and queues gets near 0. On our
> > test system which are way more powerfull systems (see SYSTEM 2) our
> > IPC dispatcher module queue fills up rapidly (depending of the
msgmnb
> > queue size it will wimply take a bit longer to fill).
> >
> > We have tested both our already compiled binaries from rh73 using
gcc
> > 2.9 and a recompiled version of the modules on a debian sarge system
> > and got the exact same problem on either a Debian Sarge 3.1 (running
a
> > 2.4 or 2.6 kernel) and on a Etch 64bit system (using 32bit compat
> > layer) with a 2.6 kernel. In all cases the queues would simply
fill-up.
> >
> > After strac'ing the module I noticed that the time needed to handle
> > the signal & ipc calls are way lower on the new system hence I don't
> > see why the dispatcher queue does fill-up like that?!?!?!
> >
> > Does anyone experienced something similar? Could this be a kernel
> > issue vs material, kernel option? Might this be related to libc?
> >
> > Help / Clues very much appreciated.
>
> Hi Vincent
>
> top shows that something is eating cpu cycles in User mode on
> your new platform, while old platform consumes cycles both in
> User and System land.
>
> This might be related to some programing error, maybe some
> spinlock in user mode or bad multi-threading synchronization,
> or scheduling assumptions, that break because of the quad
> core cpus of your new machine.
Actually, could this be worth trying (adding Ingo in CC):
http://lkml.org/lkml/2007/9/5/75
> So the thread that is supposed to consume IPC messages is not
> scheduled in time, because CPU starves. (beware the four
> cores of each CPU compete for ressources)
>
> You could issue "ps auxm" to check which threads are spining
> in User mode and try to trace them ?
Effectively in this specific test case I had 5 stuck process using each
100% of a CPU... although 3 other cores where still available so there
is (I believe) no reason why it should had starved that much.
Anyhow, that did not happend on all the other testing I made during the
past 3 weeks (except this one).
I restarted the this test (again using a 2.4.35.1 kernel), mde sure no
process where stuck :), and grabbed the ps aux + ipcs -q info (attached)
Again, the ipcs -q shows that the queue is getting full comparing to
SYSTEM 1 which always has a queue of 0.
Note: Also attached a top.txt file showing that the dispatcher uses 100%
of a CPU on SYSTEM 2. This never occurs on SYSTEM 1.
> Eric
PS, thnx for replying.
- vin
> > SYSTEM INFORMATION:
> >
> > SYSTEM 1:
> > ---------
> > HPDL580 G2
> > Quad Intel Xeon 1.90GHz
> > 4G ram
> > DRBD disks on dual-gigabit adapter
> > OS: RedHat 7.3 / kernel: 2.4.33 / libc: 2.2.5 / gcc 2.96
> >
> > SYSTEM 2:
> > ---------
> > Dell PE2950
> > Dual Intel Quad-Core 2.66GHz
> > 16G ram
> > local 300G 15000 RPM SCSI.
> > OS1: Debian Etch 4.0 / kernels 2.6.18 -> 2.6.22 / libc 2.3.6 / gcc
4.1.2
> > OS2: Debian Sarge 3.1 / kernels 2.4.35, 2.6.18 -> 2.6.22 / libc
2.3.2 / gcc 3.3.5
[-- Attachment #2: ipcs-q.txt --]
[-- Type: text/plain, Size: 2133 bytes --]
------ Message Queues --------
key msqid owner perms used-bytes messages
0x41018018 0 urp 660 60784 748
0x41018024 32769 urp 660 0 0
0x4101800b 65538 urp 660 0 0
0x41018011 98307 urp 660 0 0
0x4101800e 131076 urp 660 0 0
0x4101803f 163845 urp 660 0 0
0x41018019 196614 urp 660 0 0
0x41018026 229383 urp 660 0 0
0x4101802f 262152 urp 660 0 0
0x4101803a 294921 urp 660 0 0
0x41018036 327690 urp 660 0 0
0x4101802b 360459 urp 660 0 0
0x41018040 393228 urp 660 0 0
0x41018027 425997 urp 660 0 0
0x41018028 458766 urp 660 0 0
0x41018020 491535 urp 660 0 0
0x4101800c 524304 urp 660 0 0
0x4101800d 557073 urp 660 0 0
0x4101803c 589842 urp 660 0 0
0x4101802e 622611 urp 660 0 0
0x41018021 655380 urp 660 0 0
0x41018006 688149 urp 660 0 0
0x4101802a 720918 urp 660 0 0
0x4101800a 753687 urp 660 0 0
0x4101801e 786456 urp 660 0 0
0x41018029 819225 urp 660 0 0
0x41018037 851994 urp 660 0 0
0x41018038 884763 urp 660 0 0
0x41018022 917532 urp 660 0 0
[-- Attachment #3: ps.auxm.txt --]
[-- Type: text/plain, Size: 16811 bytes --]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 0.0 1492 500 ? - 10:06 0:04 init [2]
root - 0.4 - - - - S 10:06 0:04 -
root 2 0.0 0.0 0 0 ? - 10:06 0:00 [keventd]
root - 0.0 - - - - S 10:06 0:00 -
root 3 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU0]
root - 0.0 - - - - SN 10:06 0:00 -
root 4 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU1]
root - 0.0 - - - - SN 10:06 0:00 -
root 5 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU2]
root - 0.0 - - - - SN 10:06 0:00 -
root 6 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU3]
root - 0.0 - - - - SN 10:06 0:00 -
root 7 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU4]
root - 0.0 - - - - SN 10:06 0:00 -
root 8 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU5]
root - 0.0 - - - - SN 10:06 0:00 -
root 9 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU6]
root - 0.0 - - - - SN 10:06 0:00 -
root 10 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU7]
root - 0.0 - - - - SN 10:06 0:00 -
root 11 0.0 0.0 0 0 ? - 10:06 0:00 [kswapd]
root - 0.0 - - - - S 10:06 0:00 -
root 12 0.0 0.0 0 0 ? - 10:06 0:00 [bdflush]
root - 0.0 - - - - S 10:06 0:00 -
root 13 0.0 0.0 0 0 ? - 10:06 0:00 [kupdated]
root - 0.0 - - - - S 10:06 0:00 -
root 26 0.0 0.0 0 0 ? - 10:06 0:00 [scsi_eh_0]
root - 0.0 - - - - S 10:06 0:00 -
root 66 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 85 0.0 0.0 2920 2036 ? - 10:06 0:00 /sbin/devfsd /dev
root - 0.0 - - - - Ss 10:06 0:00 -
root 190 0.0 0.0 0 0 ? - 10:06 0:00 [khubd]
root - 0.0 - - - - S 10:06 0:00 -
root 231 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 232 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 233 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 234 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 236 0.0 0.0 0 0 ? - 10:06 0:00 [xfsbufd]
root - 0.0 - - - - S 10:06 0:00 -
root 237 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/0]
root - 0.0 - - - - S 10:06 0:00 -
root 238 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/1]
root - 0.0 - - - - S 10:06 0:00 -
root 239 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/2]
root - 0.0 - - - - S 10:06 0:00 -
root 240 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/3]
root - 0.0 - - - - S 10:06 0:00 -
root 241 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/4]
root - 0.0 - - - - S 10:06 0:00 -
root 242 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/5]
root - 0.0 - - - - S 10:06 0:00 -
root 243 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/6]
root - 0.0 - - - - S 10:06 0:00 -
root 244 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/7]
root - 0.0 - - - - S 10:06 0:00 -
root 245 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/0]
root - 0.0 - - - - S 10:06 0:00 -
root 246 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/1]
root - 0.0 - - - - S 10:06 0:00 -
root 247 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/2]
root - 0.0 - - - - S 10:06 0:00 -
root 248 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/3]
root - 0.0 - - - - S 10:06 0:00 -
root 249 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/4]
root - 0.0 - - - - S 10:06 0:00 -
root 250 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/5]
root - 0.0 - - - - S 10:06 0:00 -
root 251 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/6]
root - 0.0 - - - - S 10:06 0:00 -
root 252 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/7]
root - 0.0 - - - - S 10:06 0:00 -
root 253 0.0 0.0 0 0 ? - 10:06 0:00 [xfssyncd]
root - 0.0 - - - - S 10:06 0:00 -
root 254 0.0 0.0 0 0 ? - 10:06 0:01 [kjournald]
root - 0.0 - - - - S 10:06 0:01 -
daemon 436 0.0 0.0 1608 452 ? - 10:06 0:00 /sbin/portmap
daemon - 0.0 - - - - Ss 10:06 0:00 -
root 520 0.1 0.0 1544 624 ? - 10:07 0:01 /sbin/syslogd
root - 0.1 - - - - Ss 10:07 0:01 -
root 523 0.0 0.0 2176 1236 ? - 10:07 0:00 /sbin/klogd
root - 0.0 - - - - Ss 10:07 0:00 -
clamav 609 0.0 0.0 3032 1228 ? - 10:07 0:00 /usr/bin/freshclam -d --quiet
clamav - 0.0 - - - - Ss 10:07 0:00 -
root 619 0.0 0.0 5784 2664 ? - 10:07 0:00 /usr/sbin/cupsd -F
root - 0.0 - - - - Ss 10:07 0:00 -
message 622 0.0 0.0 2476 976 ? - 10:07 0:00 /usr/bin/dbus-daemon-1 --system
message - 0.0 - - - - Ss 10:07 0:00 -
root 634 0.0 0.0 2504 1072 ? - 10:07 0:00 /usr/sbin/lisa
root - 0.0 - - - - Ss 10:07 0:00 -
root 666 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - Ss 10:07 0:00 -
root 667 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 668 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 669 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 670 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 672 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 673 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 793 0.0 0.0 2888 1084 ? - 10:07 0:00 /usr/lib/postfix/master
root - 0.0 - - - - Ss 10:07 0:00 -
rwhod 799 0.0 0.0 1524 576 ? - 10:07 0:00 /usr/sbin/rwhod -b
rwhod - 0.0 - - - - Ss 10:07 0:00 -
rwhod 801 0.0 0.0 1524 636 ? - 10:07 0:00 /usr/sbin/rwhod -b
rwhod - 0.0 - - - - S 10:07 0:00 -
postfix 805 0.0 0.0 2964 1072 ? - 10:07 0:00 pickup -l -t fifo -u -c
postfix - 0.0 - - - - S 10:07 0:00 -
postfix 806 0.0 0.0 2996 1100 ? - 10:07 0:00 qmgr -l -t fifo -u -c
postfix - 0.0 - - - - S 10:07 0:00 -
root 808 0.0 0.0 4048 1568 ? - 10:07 0:00 /usr/sbin/sshd
root - 0.0 - - - - Ss 10:07 0:00 -
root 814 0.0 0.0 3048 820 ? - 10:07 0:00 /usr/sbin/vsftpd
root - 0.0 - - - - Ss 10:07 0:00 -
root 823 0.0 0.0 2084 840 ? - 10:07 0:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.pid -stayalive
root - 0.0 - - - - Ss 10:07 0:00 -
root 824 0.0 0.0 1668 728 ? - 10:07 0:00 /sbin/rpc.statd
root - 0.0 - - - - Ss 10:07 0:00 -
daemon 829 0.0 0.0 1480 480 ? - 10:07 0:00 /usr/sbin/atd
daemon - 0.0 - - - - Ss 10:07 0:00 -
root 833 0.0 0.0 1568 692 ? - 10:07 0:00 /usr/sbin/cron
root - 0.0 - - - - Ss 10:07 0:00 -
root 843 0.0 0.0 4572 2108 ? - 10:07 0:00 /usr/sbin/apache
root - 0.0 - - - - S 10:07 0:00 -
www-data 845 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 846 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 847 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 848 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 849 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
root 860 0.0 0.0 1484 472 vc/1 - 10:07 0:00 /sbin/getty 38400 tty1
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 861 0.0 0.0 1484 472 vc/2 - 10:07 0:00 /sbin/getty 38400 tty2
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 862 0.0 0.0 1484 472 vc/3 - 10:07 0:00 /sbin/getty 38400 tty3
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 863 0.0 0.0 1484 472 vc/4 - 10:07 0:00 /sbin/getty 38400 tty4
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 864 0.0 0.0 1484 472 vc/5 - 10:07 0:00 /sbin/getty 38400 tty5
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 865 0.0 0.0 1484 472 vc/6 - 10:07 0:00 /sbin/getty 38400 tty6
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 872 0.0 0.0 6892 2084 ? - 10:07 0:00 sshd: root@pts/0
root - 0.0 - - - - Ss 10:07 0:00 -
root 874 0.0 0.0 3884 1844 pts/0 - 10:07 0:00 -zsh
root - 0.0 - - - - Ss 10:07 0:00 -
root 885 0.0 0.0 7212 2236 ? - 10:07 0:00 sshd: root@pts/1
root - 0.0 - - - - Ss 10:07 0:00 -
root 890 0.0 0.0 3884 1844 pts/1 - 10:08 0:00 -zsh
root - 0.0 - - - - Ss 10:08 0:00 -
urp 895 0.0 0.0 4352 2384 pts/1 - 10:08 0:00 -su
urp - 0.0 - - - - S 10:08 0:00 -
urp 944 0.0 0.0 3384 1916 pts/0 - 10:09 0:00 -su
urp - 0.0 - - - - S 10:09 0:00 -
urp 22762 90.9 0.0 12292 11304 pts/1 - 10:14 9:24 URPDispatcher
urp - 90.9 - - - - R 10:14 9:24 -
urp 22805 18.6 0.1 57636 31292 pts/1 - 10:14 1:55 bin/URPJobMaker
urp - 18.6 - - - - R 10:14 1:55 -
urp 22822 2.4 0.1 34996 23228 pts/1 - 10:14 0:14 bin/URPCappi
urp - 2.4 - - - - S 10:14 0:14 -
urp 22836 0.3 0.0 15320 12780 pts/1 - 10:14 0:02 bin/URPConvPPI
urp - 0.3 - - - - S 10:14 0:02 -
urp 22850 0.4 0.0 19584 12224 pts/1 - 10:14 0:02 bin/URPClogzPPI
urp - 0.4 - - - - S 10:14 0:02 -
urp 22865 0.6 0.0 20160 13168 pts/1 - 10:14 0:04 bin/URPVrPPI
urp - 0.6 - - - - S 10:14 0:04 -
urp 22880 2.4 0.1 26552 20420 pts/1 - 10:14 0:14 bin/URPEchoTop
urp - 2.4 - - - - S 10:14 0:14 -
urp 22897 0.7 0.0 15920 13140 pts/1 - 10:14 0:04 bin/URPMaxR
urp - 0.7 - - - - S 10:14 0:04 -
urp 22911 0.3 0.0 16512 13456 pts/1 - 10:14 0:02 bin/URPPrecip
urp - 0.3 - - - - S 10:14 0:02 -
urp 22925 1.0 0.1 21932 18908 pts/1 - 10:14 0:06 bin/URPSvrWx
urp - 1.0 - - - - S 10:14 0:06 -
urp 22939 0.0 0.0 10504 9032 pts/1 - 10:14 0:00 bin/URPRFA
urp - 0.0 - - - - S 10:14 0:00 -
urp 22953 2.3 0.1 42032 19324 pts/1 - 10:14 0:13 bin/URPPA
urp - 2.3 - - - - S 10:14 0:13 -
urp 22970 5.0 0.1 28532 21524 pts/1 - 10:14 0:29 bin/URPVVW
urp - 5.0 - - - - S 10:14 0:29 -
urp 23004 0.3 0.0 16704 13020 pts/1 - 10:14 0:01 bin/URPMesoCyclone
urp - 0.3 - - - - S 10:14 0:01 -
urp 23018 0.1 0.0 14668 11208 pts/1 - 10:14 0:00 bin/URPMicroBurst
urp - 0.1 - - - - S 10:14 0:00 -
urp 23032 0.0 0.0 12556 10424 pts/1 - 10:14 0:00 bin/URPGust
urp - 0.0 - - - - S 10:14 0:00 -
urp 23056 1.1 0.0 26996 14180 pts/1 - 10:14 0:06 bin/URPCellID
urp - 1.1 - - - - S 10:14 0:06 -
urp 23073 3.8 0.0 22948 13000 pts/1 - 10:14 0:22 bin/URPCellProps
urp - 3.8 - - - - S 10:14 0:22 -
urp 23089 0.0 0.0 10632 9068 pts/1 - 10:15 0:00 bin/URPTracker
urp - 0.0 - - - - S 10:15 0:00 -
urp 23103 0.3 0.0 17476 11940 pts/1 - 10:15 0:01 bin/URPPPI
urp - 0.3 - - - - S 10:15 0:01 -
urp 23147 1.9 0.1 23568 19508 pts/1 - 10:15 0:11 bin/URPHail
urp - 1.9 - - - - S 10:15 0:11 -
urp 23161 0.0 0.0 10828 9248 pts/1 - 10:15 0:00 bin/URPAssessmwin
urp - 0.0 - - - - S 10:15 0:00 -
urp 23178 1.3 0.0 25124 16312 pts/1 - 10:15 0:08 bin/URPNids
urp - 1.3 - - - - S 10:15 0:08 -
urp 23192 0.0 0.0 12520 9848 pts/1 - 10:15 0:00 bin/URPBWER
urp - 0.0 - - - - S 10:15 0:00 -
urp 23206 0.3 0.0 16660 11600 pts/1 - 10:15 0:02 bin/URPGradient
urp - 0.3 - - - - S 10:15 0:02 -
urp 23220 0.0 0.0 10724 9372 pts/1 - 10:15 0:00 bin/URPMultiRadarMerge
urp - 0.0 - - - - S 10:15 0:00 -
urp 23234 0.0 0.0 10696 9072 pts/1 - 10:15 0:00 bin/URPSAC
urp - 0.0 - - - - S 10:15 0:00 -
urp 23251 0.0 0.0 10500 9004 pts/1 - 10:15 0:00 bin/URPSCIT
urp - 0.0 - - - - S 10:15 0:00 -
urp 23265 0.4 0.0 10360 9188 pts/1 - 10:15 0:02 bin/URPIngester
urp - 0.4 - - - - S 10:15 0:02 -
root 24580 0.0 0.0 7048 2100 ? - 10:17 0:00 sshd: root@pts/2
root - 0.0 - - - - Ss 10:17 0:00 -
root 24601 0.0 0.0 4712 2132 pts/2 - 10:17 0:00 -zsh
root - 0.0 - - - - Ss 10:17 0:00 -
root 24623 0.0 0.0 2572 588 pts/2 - 10:17 0:00 tail -f urplog.log
root - 0.0 - - - - S+ 10:17 0:00 -
urp 26020 0.0 0.0 2024 500 pts/0 - 10:24 0:00 sleep 2
urp - 0.0 - - - - S+ 10:24 0:00 -
urp 26021 0.0 0.0 2292 728 pts/1 - 10:24 0:00 ps auxm
urp - 0.0 - - - - R+ 10:24 0:00 -
[-- Attachment #4: top.txt --]
[-- Type: text/plain, Size: 3551 bytes --]
top - 10:45:13 up 38 min, 3 users, load average: 1.31, 1.40, 1.30
Tasks: 119 total, 5 running, 114 sleeping, 0 stopped, 0 zombie
Cpu0 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Cpu1 : 1.3% user, 0.0% system, 0.0% nice, 98.7% idle
Cpu2 : 0.7% user, 0.0% system, 0.0% nice, 99.3% idle
Cpu3 : 32.3% user, 0.0% system, 0.0% nice, 67.7% idle
Cpu4 : 0.7% user, 0.3% system, 0.0% nice, 99.0% idle
Cpu5 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Cpu6 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Cpu7 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Mem: 16567608k total, 3797076k used, 12770532k free, 442836k buffers
Swap: 2048276k total, 0k used, 2048276k free, 2690988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31337 urp 18 0 11308 11m 548 R 99.9 0.1 7:53.39 URPDispatcher
31373 urp 9 0 32296 31m 612 S 31.0 0.2 2:05.00 URPJobMaker
31667 urp 9 0 14132 13m 716 S 1.0 0.1 0:03.81 URPCellID
31838 urp 9 0 11632 11m 704 S 1.0 0.1 0:01.47 URPGradient
885 root 9 0 2236 2236 1732 S 0.3 0.0 0:00.22 sshd
31424 urp 18 0 12228 11m 11m S 0.3 0.1 0:01.32 URPClogzPPI
31444 urp 10 0 12696 12m 700 R 0.3 0.1 0:02.46 URPVrPPI
580 urp 10 0 992 992 760 R 0.3 0.0 0:00.55 top
1205 urp 16 0 13420 13m 11m R 0.3 0.1 0:00.01 URPClogzPPI
1206 urp 18 0 13420 13m 11m R 0.3 0.1 0:00.01 URPClogzPPI
1 root 8 0 500 500 448 S 0.0 0.0 0:04.38 init
2 root 9 0 0 0 0 S 0.0 0.0 0:00.00 keventd
3 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU0
4 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU1
5 root 19 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU2
6 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU3
7 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU4
8 root 19 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU5
9 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU6
10 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU7
11 root 9 0 0 0 0 S 0.0 0.0 0:00.03 kswapd
12 root 9 0 0 0 0 S 0.0 0.0 0:00.00 bdflush
13 root 9 0 0 0 0 S 0.0 0.0 0:00.11 kupdated
26 root 9 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
66 root 9 0 0 0 0 S 0.0 0.0 0:00.04 kjournald
85 root 9 0 2036 2036 648 S 0.0 0.0 0:00.02 devfsd
190 root 9 0 0 0 0 S 0.0 0.0 0:00.00 khubd
231 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
232 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
233 root 9 0 0 0 0 S 0.0 0.0 0:00.01 kjournald
234 root 9 0 0 0 0 S 0.0 0.0 0:00.21 kjournald
236 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfsbufd
237 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/0
238 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/1
239 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/2
240 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/3
241 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/4
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2007-09-05 15:49 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-09-05 13:37 [question] IPC queue filling-up problem? Fortier,Vincent [Montreal]
2007-09-05 14:16 ` Eric Dumazet
2007-09-05 15:48 ` Fortier,Vincent [Montreal]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®