Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[Thomas Pollinger]

Hi Stephen

>Let me see if I understand this correct:
>
>         Client  Router  Server
>         ------------------------------------
>         Linux   Linux   HPUX            Bad
>         WinNT   Linux   HPUX            Good
>         HPUX    Linux   HPUX            Good
>
>With all Linux boxes version 2.2.something
>
>Is this correct?

Yes, this is right.

>This is slightly different from my situation; I had a Linux client and
>Linux server directly connected to the same switch (3Com Superstack
>II, FWIW)

This is probably a configuration I'm going to try as well: put the cvs 
server on a linux box and have it communicate to different clients. 
However, the configuration I have now simply suggests that there cannot be 
a problem on the IP level as the Linux box as router is working as expected.


>Perhaps if I get time I'll repeat the experiment with (a) the most
>recent Alan pre-2.2 kernel and (b) the most recent 2.4-test kernel...
>
>I'll re-iterate my original request, which was not "it's broke - can
>you fix it" but was "okay, how do I go about tracking this one down?"

At this stage, I am uncertain of the cause of the problem. I am pretty much 
sure that the card, drivers and the IP level should work as expected. It is 
lilkely that there could be a problem with CVS which gets caught in a kind 
of deadlock - however, observing the TCP-packet communication and the 
system calls done by both CVS client and server shows always the same 
result (I put a printf before and after the read/recv call in the client):
         - The clilent is stuck in the read system call on the socket for 
reads and waits forever
         - The server gets EWOULDBLOCK notifications when writing to the 
socket and eventually goes
           to sleep
         - The last packages sent by the server will be resent roughly 
every 30-60 seconds.
           The client either responds no packages accepted or the last 
packages are accepted by the
           client but the system call still blocks.

Regards,
-Thomas

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[kuznet]

Hello!

172.30.0.3 is HPUX?

Upgrade it, you were so unlucky to purchase some invalid OS release.
Each TCP connection to it from any sane OS will freeze miximum after
656 seconds.

I think this bug has been fixed.

[ Dave: one puzzle less, this bogon wraps his own timestamps at 16 bits. 8) ]

Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[Thomas Pollinger]

>
>Hello!
>
>172.30.0.3 is HPUX?

Hello Alexey,
yes, this is the HPUX.

>Upgrade it, you were so unlucky to purchase some invalid OS release.
>Each TCP connection to it from any sane OS will freeze miximum after
>656 seconds.

Thanks for the hint. This stroke me as well as I left it once the whole 
night and the server did not terminate the connection.


>I think this bug has been fixed.

Just out of curiosity, was that bug related to a special configuration with 
the HPUX or could it have happened between two Linux boxes as well?

-Thomas

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[kuznet]

Hello!

> Just out of curiosity, was that bug related to a special configuration with 
> the HPUX 

This is bug of HPUX, to be more exact.


>	or could it have happened between two Linux boxes as well?

No.

Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[Stephen Landamore]

Hi Thomas,

On Fri, 3 Nov 2000, Thomas Pollinger wrote:
> Running a 'cvs get' on the Linux clients of a larger source tree
> eventually hangs the client in the middle of the get process. The
> hang is *always* reproduceable (however it does not always hang at
> the same place, sometimes after 1', sometimes after 5' to
> 10'). Several runs on Win NT did not show this problem.
[...]
> At last, I tried to do the same on another HP-UX box and there was
> no blocking at all.
>
> What is interesting to know is that between the HP-UX server and the
> HP client is a Linux router with a 3c905 card, 2.2.14 kernel.

Let me see if I understand this correct:

	Client	Router	Server
	------------------------------------
	Linux	Linux	HPUX		Bad
	WinNT	Linux	HPUX		Good
	HPUX	Linux	HPUX		Good

With all Linux boxes version 2.2.something

Is this correct?

This is slightly different from my situation; I had a Linux client and
Linux server directly connected to the same switch (3Com Superstack
II, FWIW)

I found the client always hung in my test; this was only visible at
the end of a SPEC run. Analysing tcpdump etc showed that in fact it
died somewhere early in the test, almost always before 10 minutes
worth of run.

In the end I 'fixed' the problem by using a HPUX server :-/ ... I
observe that there _are_ some SPECweb99 submissions with Linux (the
Tux result), I'm curious to know if the people who actually did the
test run experienced any problems...

Perhaps if I get time I'll repeat the experiment with (a) the most
recent Alan pre-2.2 kernel and (b) the most recent 2.4-test kernel...

I'll re-iterate my original request, which was not "it's broke - can
you fix it" but was "okay, how do I go about tracking this one down?"

cheers,
stephen

--
Stephen Landamore, <slandamore@zeus.com>              Zeus Technology
Tel: +44 1223 525000                      Universally Serving the Net
Fax: +44 1223 525100                              http://www.zeus.com
Zeus Technology, Zeus House, Cowley Road, Cambridge, CB4 0ZT, ENGLAND

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [BUG REPORT] TCP/IP weirdness in 2.2.15

[Thomas Pollinger]

Hi Stephen

I've been experiencing similar problems with what would seem at first a 
completely unrelated problem. To make things short:

I have the following configuration:
- CVS server (vers. 1.10.4) running on HPUX B.11.00
- different CVS clients (running different cvs client versions: 1.10.4/7/8):
  - Win NT (PIII, 512MB, 3c905 10/100)
  - Debian 2.2.17 (PII, 256MB, 3c905, main client tester)
  - Mandrake 2.2.x (3c905)
  - RedHat 6.2 (Laptop, different Ethernet card)
  - SuSE 2.2.10 (PII, 256MB, 3c905)

Running a 'cvs get' on the Linux clients of a larger source tree eventually 
hangs the client in the middle of the get process. The hang is *always* 
reproduceable (however it does not always hang at the same place, sometimes 
after 1', sometimes after 5' to 10'). Several runs on Win NT did not show 
this problem.

Running a strace on the client and server reveals that the client gets 
stuck in a read(3, resp. recv(3, call (a recompiled the cvs client to not 
use file descriptors for an additional test) and blocks forever (I left it 
once during one night). The server tries first to send to the client, but 
gets an EWOULDBLOCK error back. The server tries to send to the client for 
about 5' during which no packets are accepted by the client. tcpdump 
revealed that after the hang, sometimes the client did not accept any 
packets (ack 0), sometimes it replied . ack 7678261 back (this will go 
forever if the client won't be killed). The server replies: P 
7678261:76784009 (1448) ack 612 win 32768 ... indefinitely.

To keep the mail short, I can run a test and send all related logs if needed.

I first thought that I got not a recent network driver and compiled a more 
recent one (v0.99H 12Jun00 was old, new: 16Aug00 <2.2.17-pre17>) - but the 
behaviour was the same.

At last, I tried to do the same on another HP-UX box and there was no 
blocking at all.
What is interesting to know is that between the HP-UX server and the HP 
client is a Linux router with a 3c905 card, 2.2.14 kernel.

Regards,
Pollinger Thomas


--------------------------------------------------------------------------------------------------------------
Below the original mail from Stephen (as it has been a while ago this 
message was posted):

Hi Folks,
I'm seeing some very strange behaviour here with TCP/IP connections
(or at least, it looks strange to me) - under heavy load, it appears
connections are spuriously dropped.
Bear with me while I give a little background...
...I'm actually trying to perform some SPECweb99 benchmarking; I have
a test bed of 5 clients and 1 server. For those of you who are not
familiar with SPECweb99, let's just say that it tries to simulate
'real world' traffic - a mixture of HTTP GET, POST, static and dynamic
GET, keepalive connections, non-keepalive connections... basically
hammer the server as hard as you can until it grinds to a halt. Each
client has one controlling (parent) process which forks off N children
to generate the HTTP load. One client is a designated 'prime client'
which synchronises the rest of the clients. If you're interested in
more SPEC info, there's a very informative whitepaper on their
website, have a look at http://www.spec.org/osg/web99
The "dropped connection" manifests itself by the SPEC run 'hanging' -
at the end of the test there is one or more child processes on one or
more clients stuck in a read(2), trying to read from a
socket. (usually, one child process on one client).
If I do a netstat on the client I see an ESTABLISHED connection:
[stephenl@spec1 stephenl]$ netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
[...]
tcp 0 0 spec1.specweb.zeus:6426 spec6.specweb.zeus.:www ESTABLISHED
[...]
[stephenl@spec1 stephenl]$
(spec1 is the client, spec6 is the server. As it happens, spec1 is
also my prime client - this is just a coincidence in this particular
lockup)
If I hop over to the server and do a netstat there, I
see... nothing. No ESTABLISHED connection, no TIME_WAIT, no
CLOSE_WAIT. Nothing.
Question: is this valid? (I think it may be, if the server
crashed. Which it didn't - all machines have enjoyed a nice long
uptime :-)
I've done more digging. I captured tcpdumps of the entire SPEC run, on
all machines. I added lots of extra debugging logs to the client code,
I ran 'netstat -tn' every 10 seconds on all machines for the duration
of the test. Oh yes, and the webserver logs.
Here is what I've found...
Debugging logfile
-----------------
The client creates a socket, binds a port to it and connects to the
webserver. It write()s a HTTP GET request, and read()s a file back. So
far so good. Without closing the socket, another request is sent and
another file is received (on keepalives, there are a random number of
HTTP requests performed - on average, 10). Now here is where it gets
interesting... still without closing the socket, write() another HTTP
request and go to read() the file... the read() sits there and this
child is effectively dead. I've observed it to hang on anywhere from
the 2nd request to the 7th or 8th request.
tcpdump logs
------------
This matches perfectly what I've just detailed above... I see all the
previous requests on this port, I see the server send the previous
file, I see the client ACK this, I see the client send the final,
fatal GET... I see the server ACK this. I see this on both the client
AND server tcpdump logfiles - they both perfectly match. After the
final, fatal ACK I see... nothing. The client never sends any more
packets, the server never sends any more packets. Specifically I see
no FIN from either end.
webserver logs
--------------
I see all requests from the offending client up to and including the
previous request. The offending GET never makes it to the server logs.
netstat logs
------------
Running a 'netstat -tn' every 10 seconds on the client, I see the
connection spring into existence somewhere through the test (sometimes
near the beginning of the test, sometimes near the end. On average
about halfway through, on a 10-minute run). Once the connection has
been made, it is reported as 'ESTABLISHED' until I terminate the stuck
process... Running a 'netstat -tn' every 10 seconds on the server, I
see the connection spring into existence at the same time; I see the
same connection 10 seconds later; 20 seconds later it has vanished. No
CLOSE, no CLOSE_WAIT. Not, even, a TIME_WAIT. According to netstat,
the connection on the server simply vanished. I'm *sure* this is just
plain wrong - but it's what I see.
misc logs
---------
No error messages in the webserver log; nothing in 'dmesg', nothing in
/var/spool/*
---

I've searched the archives best as I can but couldn't find anything
relevant (actually I've been loosely following linux-kernel for the
past 2 years, and I would expect to remember if I'd seen this before,
but I'm sure better minds can advise!)

Please help me folks, I've spent hours tracking it down this far, and
it's starting to wear me down :-/

I'd really appreciate info on how to go about debugging from
here... or a pointer as to what I'm doing wrong (am I doing something
wrong?)

---

I'm deliberately not going to include .config files etc here - this
mail is getting long enough as it is. Of course all information is
available - just ask. But I will give a concise system description:

All clients are identical, Celeron 500's with IDE subsystem and twin
EEPRO10/100 NICs (actually I'm only using 1 NIC for my tests at
present).

The server is basically the same as the client, except it has a PIII
700.

All systems were originally installed with RedHat 6.0, the only thing
that has changed is now ALL machines have linux kernel 2.2.15 (and I
hoped that would fix my problems :-)

Webserver software is Zeus 3.3.6.

The server has had /proc/sys/fs/{file,inode}-max raised to cope with
the big fileset (and the PAM limits raised)

That's all I can think of for now, so I'll stop typing here. Any
feedback would be deeply appreciated,

Best regards,
Stephen

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to "majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/