TCP stack behaviour question

TCP stack behaviour question

[Stuart MacDonald]

I'm having some trouble with a network application I've written. I've
done a lot of research the last few days; man 7 ip, man 7 tcp, kernel
2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some
reason we don't have Vol 2), Usenet, websites. I'm hoping someone here
can help me out, or point me in the correct direction.

Distro: Debian 3.0 r2
Kernel: Stock 2.4.24

tcp_retries1 == 3
tcp_retries2 == 15

I have an application that setups up a TCP connection to a server. If
the server has a power failure, TCP starts retransmitting the packet
that wasn't ACKed. I see the exponential backoff.

Question 1: There's the original packet, plus 7 retransmitted packets
for a total of 8, then TCP gives up. How is 7 (or 8) derived from the
tcp_retries[12] settings?

Question 1a: The time between last and second-last retransmit packets
is only about 27 seconds. I've read there's a maximum time, but also
that it's usually 100 or 120 seconds. Where can I find that setting in
/proc?

Question 1b: If RTO is that high, why is retransmit stopping?

Question 2: After the retransmit has given up, the app is still
making an occasional write(), which succeeds! However, tearing down
and attemting a new connection results in an immediate EHOSTUNREACH
error. Why is the write() succeeding?

Question 2a: How can my app find out the EHOSTUNREACH error
immediately? IP_RECVERR is not implemented on TCP, and SO_ERROR always
reports no error (0).

..Stu

Re: TCP stack behaviour question

[Andi Kleen]

"Stuart MacDonald" <stuartm@connecttech.com> writes:

> I'm having some trouble with a network application I've written. I've
> done a lot of research the last few days; man 7 ip, man 7 tcp, kernel
> 2.4.31 source code, Stevens' Illustrated TCP/IP Vol 1 & 3 (for some
> reason we don't have Vol 2), Usenet, websites. I'm hoping someone here
> can help me out, or point me in the correct direction.

Stevens is a good reference to BSD networking, but not necessarily to Linux.

> Question 1: There's the original packet, plus 7 retransmitted packets
> for a total of 8, then TCP gives up. How is 7 (or 8) derived from the
> tcp_retries[12] settings?

Documented in tcp(7)
 
> Question 1a: The time between last and second-last retransmit packets
> is only about 27 seconds. I've read there's a maximum time, but also
> that it's usually 100 or 120 seconds. Where can I find that setting in
> /proc?

It's fixed by the TCP specification.

> Question 2: After the retransmit has given up, the app is still
> making an occasional write(), which succeeds! However, tearing down
> and attemting a new connection results in an immediate EHOSTUNREACH
> error. Why is the write() succeeding?

It goes all in the local buffer, until it starts blocking (or EAGAIN
for non blocking sockets)

> 
> Question 2a: How can my app find out the EHOSTUNREACH error
> immediately? IP_RECVERR is not implemented on TCP, and SO_ERROR always
> reports no error (0).

Did you really read the manpages?  It is implemented and it's documented.

-Andi (your new temporary manpage proxy)  

RE: TCP stack behaviour question

[Stuart MacDonald]

From: On Behalf Of Andi Kleen

Thanks for replying, I appreciate it.

> Stevens is a good reference to BSD networking, but not 
> necessarily to Linux.

Which is why I posted here. I was wondering about the specific
implementation.

What happened was this: I had a run where I captured output with
tcpdump. My original post was based on that, and the results of the
debug output from my app. For whatever reason, it appears the stack
didn't generate all of the packets it should have. When the log showed
a second-last to last retransmit time of about 27 seconds, and then a
gap of about 400 to the very next packet of any kind, I assumed that
meant the stack had given up on the retransmits when it appears
something else was going on.

I did some digging into the kernel and on the next run found that all
the expected retransmit packets were being generated, and that once
the stack gives up on the retransmits then system calls return errors.

> > Question 2a: How can my app find out the EHOSTUNREACH error
> > immediately? IP_RECVERR is not implemented on TCP, and 
> SO_ERROR always
> > reports no error (0).
> 
> Did you really read the manpages?  It is implemented and it's 
> documented.

Yes I did and no it's not, according to the man page. I quoth:
# man 7 ip
..
               Note that TCP has no error queue; MSG_ERRQUEUE is
illegal on SOCK_STREAM sockets.  Thus all errors are returned by
socket function return or SO_ERROR only.

Maybe the man page is wrong? That's from my FC 3 install.

..Stu

Re: TCP stack behaviour question

[Andi Kleen]

> # man 7 ip
> ..
>                Note that TCP has no error queue; MSG_ERRQUEUE is
> illegal on SOCK_STREAM sockets.  Thus all errors are returned by
> socket function return or SO_ERROR only.
> 
> Maybe the man page is wrong? That's from my FC 3 install.

The sentence is correct, but TCP has a IP_RECVERR that works
differently without a queue. Basically it doesn't delay the error 
reporting for incoming ICMPs to the last retransmit, but reports
them immediately. This is documented in tcp(7)

-Andi

RE: TCP stack behaviour question

[Stuart MacDonald]

From: Andi Kleen [mailto:ak@suse.de] 
> > # man 7 ip
> > ..
> >                Note that TCP has no error queue; MSG_ERRQUEUE is
> > illegal on SOCK_STREAM sockets.  Thus all errors are returned by
> > socket function return or SO_ERROR only.
> > 
> > Maybe the man page is wrong? That's from my FC 3 install.
> 
> The sentence is correct, but TCP has a IP_RECVERR that works
> differently without a queue. Basically it doesn't delay the error 
> reporting for incoming ICMPs to the last retransmit, but reports
> them immediately. This is documented in tcp(7)

I read that too, but didn't know which one was correct, so I erred on
the side of caution and believed ip(7).

Good to know, I'll look into that. Thanks.

..Stu

Re: TCP stack behaviour question

[Andi Kleen]

On Monday 18 September 2006 16:19, Stuart MacDonald wrote:
> From: Andi Kleen [mailto:ak@suse.de] 
> > > # man 7 ip
> > > ..
> > >                Note that TCP has no error queue; MSG_ERRQUEUE is
> > > illegal on SOCK_STREAM sockets.  Thus all errors are returned by
> > > socket function return or SO_ERROR only.
> > > 
> > > Maybe the man page is wrong? That's from my FC 3 install.
> > 
> > The sentence is correct, but TCP has a IP_RECVERR that works
> > differently without a queue. Basically it doesn't delay the error 
> > reporting for incoming ICMPs to the last retransmit, but reports
> > them immediately. This is documented in tcp(7)
> 
> I read that too, but didn't know which one was correct, so I erred on
> the side of caution and believed ip(7).

Ok maybe it's a bit misleading. Michael, you might want to clarify.

-Andi

Re: TCP stack behaviour question

[Michael Kerrisk]

Von: Andi Kleen <ak@suse.de>
> On Monday 18 September 2006 16:19, Stuart MacDonald wrote:
> > From: Andi Kleen [mailto:ak@suse.de] 
> > > > # man 7 ip
> > > > ..
> > > >                Note that TCP has no error queue; MSG_ERRQUEUE is
> > > > illegal on SOCK_STREAM sockets.  Thus all errors are returned by
> > > > socket function return or SO_ERROR only.
> > > > 
> > > > Maybe the man page is wrong? That's from my FC 3 install.
> > > 
> > > The sentence is correct, but TCP has a IP_RECVERR that works
> > > differently without a queue. Basically it doesn't delay the error 
> > > reporting for incoming ICMPs to the last retransmit, but reports
> > > them immediately. This is documented in tcp(7)
> > 
> > I read that too, but didn't know which one was correct, so I erred on
> > the side of caution and believed ip(7).
> 
> Ok maybe it's a bit misleading. Michael, you might want to clarify.

Can some one of you propose a better text please?

Cheers,

Michael
-- 
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.

RE: TCP stack behaviour question

[Stuart MacDonald]

From: Michael Kerrisk [mailto:mtk-manpages@gmx.net] 
> Von: Andi Kleen <ak@suse.de>
> > On Monday 18 September 2006 16:19, Stuart MacDonald wrote:
> > > From: Andi Kleen [mailto:ak@suse.de] 
> > > > > # man 7 ip
> > > > > ..
> > > > >                Note that TCP has no error queue; 
> MSG_ERRQUEUE is
> > > > > illegal on SOCK_STREAM sockets.  Thus all errors are 
> returned by
> > > > > socket function return or SO_ERROR only.
> > > > > 
> > > > > Maybe the man page is wrong? That's from my FC 3 install.
> > > > 
> > > > The sentence is correct, but TCP has a IP_RECVERR that works
> > > > differently without a queue. Basically it doesn't delay 
> the error 
> > > > reporting for incoming ICMPs to the last retransmit, but reports
> > > > them immediately. This is documented in tcp(7)
> > > 
> > > I read that too, but didn't know which one was correct, 
> so I erred on
> > > the side of caution and believed ip(7).
> > 
> > Ok maybe it's a bit misleading. Michael, you might want to clarify.
> 
> Can some one of you propose a better text please?

Perhaps

Note that TCP has no error queue; MSG_ERRQUEUE is illegal on
SOCK_STREAM sockets.  IP_RECVERR is valid for TCP, but all errors are
returned by socket function return or SO_ERROR only.

?

..Stu

Re: RE: TCP stack behaviour question

[Michael Kerrisk]

Von: "Stuart MacDonald" <stuartm@connecttech.com>

> > > Ok maybe it's a bit misleading. Michael, you might want to clarify.
> > 
> > Can some one of you propose a better text please?
> 
> Perhaps
> 
> Note that TCP has no error queue; MSG_ERRQUEUE is illegal on
> SOCK_STREAM sockets.  IP_RECVERR is valid for TCP, but all errors are
> returned by socket function return or SO_ERROR only.
> 
> ?

Sound okay to you Andi?

Cheers,

Michael
-- 
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.

Re: TCP stack behaviour question

[Andi Kleen]

On Tuesday 19 September 2006 08:13, Michael Kerrisk wrote:
> Von: "Stuart MacDonald" <stuartm@connecttech.com>
> 
> > > > Ok maybe it's a bit misleading. Michael, you might want to clarify.
> > > 
> > > Can some one of you propose a better text please?
> > 
> > Perhaps
> > 
> > Note that TCP has no error queue; MSG_ERRQUEUE is illegal on
> > SOCK_STREAM sockets.  IP_RECVERR is valid for TCP, but all errors are
> > returned by socket function return or SO_ERROR only.
> > 
> > ?
> 
> Sound okay to you Andi?

Yes.
-Andi

Re: TCP stack behaviour question

[Michael Kerrisk]

> Perhaps
> 
> Note that TCP has no error queue; MSG_ERRQUEUE is illegal on
> SOCK_STREAM sockets.  IP_RECVERR is valid for TCP, but all errors are
> returned by socket function return or SO_ERROR only.

Stuart -- thanks, added for man-pages-2.41.

Interestingly, at this point in the man pages source there
is the following commented out text:

.\" FIXME . Is it a good idea to document that? It is a dubious feature.
.\" On
.\" .B SOCK_STREAM
.\" sockets,
.\" .I IP_RECVERR
.\" has slightly different semantics. Instead of
.\" saving the errors for the next timeout, it passes all incoming
.\" errors immediately to the user.
.\" This might be useful for very short-lived TCP connections which
.\" need fast error handling. Use this option with care:
.\" it makes TCP unreliable
.\" by not allowing it to recover properly from routing
.\" shifts and other normal
.\" conditions and breaks the protocol specification.

Cheers,

Michael
-- 
Michael Kerrisk
maintainer of Linux man pages Sections 2, 3, 4, 5, and 7 

Want to help with man page maintenance?  
Grab the latest tarball at
ftp://ftp.win.tue.nl/pub/linux-local/manpages/, 
read the HOWTOHELP file and grep the source 
files for 'FIXME'.

Re: TCP stack behaviour question

[Andi Kleen]

> Interestingly, at this point in the man pages source there
> is the following commented out text:

Yes that was me. On second thought I suppose I was right back then that this 
feature is too dubious to be documented. So better keep it
undocumented and drop the change.

-Andi

> 
> .\" FIXME . Is it a good idea to document that? It is a dubious feature.
> .\" On
> .\" .B SOCK_STREAM
> .\" sockets,
> .\" .I IP_RECVERR
> .\" has slightly different semantics. Instead of
> .\" saving the errors for the next timeout, it passes all incoming
> .\" errors immediately to the user.
> .\" This might be useful for very short-lived TCP connections which
> .\" need fast error handling. Use this option with care:
> .\" it makes TCP unreliable
> .\" by not allowing it to recover properly from routing
> .\" shifts and other normal
> .\" conditions and breaks the protocol specification.
> 
> Cheers,
> 
> Michael