Network throughput (localhost)

Network throughput (localhost)

[Aki Niimura]

Hello Linux experts,

I developed a Python program, which tries to take advantage of
available Multi CPU resources.
Two programs (Server and Client) are running in separate processes
communicating each other using Socket IPC.
The performance I'm getting is somewhat in line with my expectation
under Solaris/x86 but significantly slower
under Linux (CentOS 5) even both use the identical hardware.

My investigation revealed that the performance is bottlenecked by the
communication (Socket to localhost).

So, I wrote much simpler program (socket server and socket client) to
measure the network throughput.
What I found was that the network throughput on both platforms
(Solaris/x86 and CentOS Linux) are almost
identical if packets are going out but the network throughput of
Solaris/x86 is almost twice (up to three times)
the throughput Linux can deliver if the packets are destined to localhost.

I know that Solaris treats packets destined to localhost differently.

Because of this, my program is slower (than one process) under Linux
(and is practically useless).
(I see meaningful performance gain under Solaris/x86, though)
I haven't tried any other Linux distributions.
Is there any Linux distribution known for better localhost network throughput?
Is there any way to improve the localhost throughput under Linux (CentOS)?

Of course, some people would point out that other types of IPC (such
as shared memory) are better choices
for the communication between closely coupled multi-processing
program. Unfortunately, there are
reasons that my program needs to use Socket.

Any suggestions will be greatly appreciated.

Best regards,
Aki Niimura

Re: Network throughput (localhost)

[Bernd Petrovitsch]

On Fri, 2009-03-13 at 00:25 -0700, Aki Niimura wrote:
[...]
> I developed a Python program, which tries to take advantage of
> available Multi CPU resources.
> Two programs (Server and Client) are running in separate processes
> communicating each other using Socket IPC.
> The performance I'm getting is somewhat in line with my expectation
> under Solaris/x86 but significantly slower
> under Linux (CentOS 5) even both use the identical hardware.
> 
> My investigation revealed that the performance is bottlenecked by the
> communication (Socket to localhost).
> 
> So, I wrote much simpler program (socket server and socket client) to
> measure the network throughput.
> What I found was that the network throughput on both platforms
> (Solaris/x86 and CentOS Linux) are almost
> identical if packets are going out but the network throughput of
> Solaris/x86 is almost twice (up to three times)
> the throughput Linux can deliver if the packets are destined to localhost.
> 
> I know that Solaris treats packets destined to localhost differently.
> 
> Because of this, my program is slower (than one process) under Linux
> (and is practically useless).
> (I see meaningful performance gain under Solaris/x86, though)
> I haven't tried any other Linux distributions.
> Is there any Linux distribution known for better localhost network throughput?
> Is there any way to improve the localhost throughput under Linux (CentOS)?
> 
> Of course, some people would point out that other types of IPC (such
> as shared memory) are better choices
> for the communication between closely coupled multi-processing
> program. Unfortunately, there are
> reasons that my program needs to use Socket.
> 
> Any suggestions will be greatly appreciated.

Have you tried AF_UNIX sockets?

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: Network throughput (localhost)

[David Miller]

From: Aki Niimura <akineko@gmail.com>
Date: Fri, 13 Mar 2009 00:25:48 -0700

> Any suggestions will be greatly appreciated.

Use AF_UNIX sockets if you want optimal performance.

Solaris treats TCP connections over localhost specially,
and we have no plans to add that optimization any time
soon.

Re: Network throughput (localhost)

[Aki Niimura]

Hello David,

Thank you for your prompt response to my posting.
I have changed my program temporarily to use AF_UNIX and tested the performance.
I got approx 20% throughput improvement in all testing conditions.
Thank you!

However, even with the use of AF_UNIX, the localhost throughput of
Solaris/x86 is still around 30% faster than Linux's AF_UNIX throughput
(using the same h/w).
I like the way Solaris did because I don't need to change my program
as my program may need to send packets to another machine rather than
another CPU next to.
Hope Linux will incorporate such scheme in the future.

Best regards,
Aki Niimura

On Fri, Mar 13, 2009 at 10:08 AM, David Miller <davem@davemloft.net> wrote:
> From: Aki Niimura <akineko@gmail.com>
> Date: Fri, 13 Mar 2009 00:25:48 -0700
>
>> Any suggestions will be greatly appreciated.
>
> Use AF_UNIX sockets if you want optimal performance.
>
> Solaris treats TCP connections over localhost specially,
> and we have no plans to add that optimization any time
> soon.
>