Poor performance with pcnet32 on SMP

Poor performance with pcnet32 on SMP

[Mel Gorman]

I have being noticing some seriously poor network performance with SMP
enabled. The machine is a xSeries 350 with quad xeon procesors. Under UP,
I get decent transfer rates but with SMP enabled, it won't get over 2kB/s.
I tried binding interrupts to one processor with

echo 1 > /proc/irq/16/smp_affinity

which, according to mail archives fixed the problem for other people, but
the result is the same. A CVS checkout from a server on the local lan of a
source tree less than 1MB took over an hour to complete but under UP
completes in about 2 seconds. This occurs with both 2.4.20 and 2.5.64.

In case it is relevant, the dmesg output related to the card on 2.5.64 is

pcnet32.c:v1.27b 01.10.2002 tsbogend@alpha.franken.de
pcnet32: PCnet/FAST III 79C975 at 0x2200, 00 02 55 fc 6d 21 assigned IRQ 16.
eth0: registered as PCnet/FAST III 79C975
pcnet32: 1 cards_found.
eth0: link up, 10Mbps, half-duplex, lpa 0x0021
Module pcnet32 cannot be unloaded due to unsafe usage in include/linux/module.h:457

Any suggestions on how to fix this are appreciated

-- 
Mel Gorman

Re: Poor performance with pcnet32 on SMP

[Dave Hansen]

Mel Gorman wrote:
> I have being noticing some seriously poor network performance with SMP
> enabled. The machine is a xSeries 350 with quad xeon procesors. Under UP,
> I get decent transfer rates but with SMP enabled, it won't get over 2kB/s.

I have the same kind of machine.  I've seen the same problems, but
they're intermittent; I haven't been able to really discern a pattern.
I have the feeling that part of the problem may be with the machine on
the other side of the connection.

I have no problems copying between two 2.5.65 machines.  In fact, my
speeds are ~10 MBytes/sec.  (yes, megabytes)

> I tried binding interrupts to one processor with
> 
> echo 1 > /proc/irq/16/smp_affinity
> 
> which, according to mail archives fixed the problem for other people, but
> the result is the same. A CVS checkout from a server on the local lan of a
> source tree less than 1MB took over an hour to complete but under UP
> completes in about 2 seconds. This occurs with both 2.4.20 and 2.5.64.

Yeah, kirq was causing that and it _was_ broken for a bit.  It should
have been fixed by 2.5.64, though.

> In case it is relevant, the dmesg output related to the card on 2.5.64 is
> 
> pcnet32.c:v1.27b 01.10.2002 tsbogend@alpha.franken.de
> pcnet32: PCnet/FAST III 79C975 at 0x2200, 00 02 55 fc 6d 21 assigned IRQ 16.
> eth0: registered as PCnet/FAST III 79C975
> pcnet32: 1 cards_found.
> eth0: link up, 10Mbps, half-duplex, lpa 0x0021
> Module pcnet32 cannot be unloaded due to unsafe usage in include/linux/module.h:457

I'm running on a full-duplex 100Mbps network, so maybe these things have
some kind of issue with 10Mbps.  I'll try plugging an old hub in and see
what happens.

Here's mine:

pcnet32.c:v1.27b 01.10.2002 tsbogend@alpha.franken.de
pcnet32: PCnet/FAST III 79C975 at 0x2240, 00 02 55 fc 43 20 assigned IRQ 16.
eth2: registered as PCnet/FAST III 79C975
pcnet32: 1 cards_found.
eth2: link up, 100Mbps, full-duplex, lpa 0x45E1

-- 
Dave Hansen
haveblue@us.ibm.com

Re: Poor performance with pcnet32 on SMP

[Mel Gorman]

On Fri, 21 Mar 2003, Dave Hansen wrote:

> I have the same kind of machine.  I've seen the same problems, but
> they're intermittent; I haven't been able to really discern a pattern.

Neither can I. I did not that sometimes between subsequent boots, I could
get either 300B/s or 7kB/s for no apparent reason.

> I have the feeling that part of the problem may be with the machine on
> the other side of the connection.
>

I tried a few different machines but they are all using a 2.4.x kernel of
some description. It still is a lot more reliable if SMP is disabled.

> Yeah, kirq was causing that and it _was_ broken for a bit.  It should
> have been fixed by 2.5.64, though.
>

I'm not sure and I don't have the expertise to track down the problem.
After a session of Devils Guide To Debugging[1], I could not find any
pattern that determined performance. I tried loading the module full
duplex at 100mb (it still resolves to 10mb at half-duplex) and various
combinations of smp_affinity but it still is very poor. Curiously enough,
trying to load it at 100mb and full_duplex makes it perform more
predictably. It's still bad (3 to 7 kB/s), but bad at a consistant level.
I suspect this observation exists purely inside my own head though. I
tried turning off ACPI but it made no difference.

I did note that I get decent performance to a 2.4 UP box that is on the
same subnet as me. I am not sure if it is relevant but the computer I am
getting really bad performance with is on another subnet behind a
firewall even though it is on the same LAN.

I am going to leave it for the moment anyway as I am getting nowhere with
the problem. Thanks for your time and if there is any other suggestions,
I'd be happy to try them out.

Mel

[1] Change things at random until something works

Re: Poor performance with pcnet32 on SMP

[Mel Gorman]

On Fri, 21 Mar 2003, Dave Hansen wrote:

> I have no problems copying between two 2.5.65 machines.  In fact, my
> speeds are ~10 MBytes/sec.  (yes, megabytes)
>

I lied, I didn't give up. This struck a chord with me for some reason. I
have the following

A - 2.5 SMP machine
B - 2.4 UP machine on *same* subnet as A
C - Web server on different subnet, behind firewall but same LAN

I have a 10MB file that I dd'd from /dev/zero on the webserver

wget http://C/~mel/10mb_file

Speed starts at 7kB/s and drops slowly to a crawl. Down with that sort of
thing. I set up a port forwarder (called aproxy) on machine B that
forwards port 80 on B to port 80 on C and try the wget again.

wget http://B/~mel/10mb_file (obviously this forwarding to machine C)

Download starts at 30kB/s and maintains it. A wget from B to C can
download at about 50kB/s but I am assuming that it is related to port
forwarding overhead as much as anything else. I am not sure what this
result means but I thought it was significant. It would seem that as well
as being SMP related, connecting to a different subnet or connecting
through a firewall is also significant.

-- 
Mel Gorman
MSc Student, University of Limerick
http://www.csn.ul.ie/~mel

Re: Poor performance with pcnet32 on SMP

[Willy Tarreau]

Hello,
 
> I have being noticing some seriously poor network performance with SMP
> enabled. The machine is a xSeries 350 with quad xeon procesors. Under UP,
> I get decent transfer rates but with SMP enabled, it won't get over 2kB/s.
> I tried binding interrupts to one processor with

I've already had problems with pcnet32 (but they were 79c971). Basically,
connected to a 100 full-duplex, they would suddenly switch back to half
duplex under heavy load, and then I got very very low speeds of course.

Since yours works in UP, this may not be related to the same problem, but who
knows ?  I see in your dmesg that the card is connected to a 10 half link. Have
you tried a 100 Mbps just in case ?

Regards,
Willy