mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "James Nichols" <jamesnichols3@gmail.com>
To: "Glen Turner" <gdt@gdt.id.au>
Cc: "Jan Engelhardt" <jengelh@computergmbh.de>,
	"Eric Dumazet" <dada1@cosmosbay.com>,
	linux-kernel@vger.kernel.org,
	"Linux Netdev List" <netdev@vger.kernel.org>
Subject: Re: After many hours all outbound connections get stuck in SYN_SENT
Date: Thu, 20 Dec 2007 11:37:39 -0500	[thread overview]
Message-ID: <83a51e120712200837p9e3d1a4g15b5f4763597073e@mail.gmail.com> (raw)
In-Reply-To: <1198161695.6154.47.camel@andromache>

> But I'd be very surprised if the router is acting as anything more
> that a network-layer device. It might perhaps have some soft connection
> state being used for generating accounting records.  Being Cisco
> it's probably a switch-router, so it might carry some per-port hard
> state for validating source IP addresses and ARPs on each port.
>
> The firewall is much more likely to be carrying per-flow Sack
> state. The Cisco PIX had a bug with SACK handling (CSCse14419,
> fixed in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141) but perhaps it
> has regressed). A simple trace either side of the firewall will
> show the inconsistency between the TCP sequence number (which
> gets randomised) and the Sack sequence number (which didn't).
> You could disable the TCP Sequence Number Randomisation feature
> and see if the fault reoccurs.

I do have TCP Sequence # Randomization enabled on my router.  However,
if this was causing an issue, wouldn't it always occur and cause
connection issues, not just after 38 hours of correct operation?  I
can look into turning this off, but I'll likely have to jump through
several hoops which will be challenging if I don't have a very clear
definitive reason why this is causing this issue.  Plus, I've had this
problem with at least 2 other sets of network switches over the past 4
years.  I'm actually running 7.0(6), which doesn't have the fix you
mentioned.  If it really is possible that this issue wouldn't always
cause problems, but only after hours of succesful operation, then I
could probably motivate the upgrade.  I can try to setup a trace, but
this is a lot of work for other people in my organization, so it will
take quite some time.


> You'd probably should also investigate the Linux kernel,
> especially the size and locks of the components of the Sack data
> structures and what happens to those data structures after Sack is
> disabled (presumably the Sack data structure is in some unhappy
> circumstance, and disabling Sack allows the data to be discarded,
> magically unclaging the box).
>
> In the absence of the reporter wanting to dump the kernel's
> core, how about a patch to print the Sack datastructure when
> the command to disable Sack is received by the kernel?
> Maybe just print the last 16b of the IP address?

Given the fact that I've had this problem for so long, over a variety
of networking hardware vendors and colo-facilities, this really sounds
good to me.  It will be challenging for me to justify a kernel core
dump, but a simple patch to dump the Sack data would be do-able.

  reply	other threads:[~2007-12-20 16:37 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-12-14 20:39 James Nichols
2007-12-17 23:14 ` Jan Engelhardt
2007-12-18 15:34   ` James Nichols
2007-12-18 16:05     ` Jan Engelhardt
2007-12-18 16:45       ` James Nichols
2007-12-18 17:19         ` Jan Engelhardt
2007-12-18 18:09           ` James Nichols
2007-12-18 18:14             ` Jan Engelhardt
2007-12-18 18:21               ` James Nichols
2007-12-18 18:30                 ` Jan Engelhardt
2007-12-18 18:32                 ` Eric Dumazet
2007-12-18 19:44                   ` James Nichols
2007-12-18 20:37                     ` Eric Dumazet
2007-12-18 21:20                       ` Jan Engelhardt
2007-12-19 16:53                       ` James Nichols
2007-12-19 17:07                         ` Eric Dumazet
2007-12-19 17:43                           ` James Nichols
2007-12-19 17:58                             ` Jan Engelhardt
2007-12-19 18:12                               ` James Nichols
2007-12-20 14:41                               ` Glen Turner
2007-12-20 16:37                                 ` James Nichols [this message]
2007-12-20 21:05                                   ` Ilpo Järvinen
2007-12-21  6:06                                     ` Jan Engelhardt
2007-12-21  4:51                                   ` Glen Turner
2007-12-21 13:57                                     ` James Nichols
2007-12-19 18:03                             ` Eric Dumazet
2007-12-19 21:27                               ` Ilpo Järvinen
2007-12-20 16:08                               ` James Nichols
2007-12-20 20:44                                 ` Ilpo Järvinen
2007-12-20 20:49                                 ` Justin Banks
2007-12-18 19:45                   ` James Nichols
2007-12-18 20:28                     ` Chuck Ebbert

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=83a51e120712200837p9e3d1a4g15b5f4763597073e@mail.gmail.com \
    --to=jamesnichols3@gmail.com \
    --cc=dada1@cosmosbay.com \
    --cc=gdt@gdt.id.au \
    --cc=jengelh@computergmbh.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®