Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+

[Ingo Molnar <mingo@elte.hu>]

* Eric Dumazet <dada1@cosmosbay.com> wrote:

>>>  22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
>>>  22003 listen(4, 10)                     = 0
>>>  22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>>>
>>> i'll queue up your reverts for testing in -tip.

I turned off localhost distcc two days ago and there has not been a 
single hung socket since then, so we now know it for sure that without 
localhost distcc connections, -tip's QA will not produce any hung 
sockets in about 1000 random-kernel-build+boot iterations.

i've added those reverts this morning and added back the localhost 
distcc rules - we'll see whether the hung sockets are back.

> I believe Ingo problems come on long lived sockets (were many bytes 
> were exchanged between the peers), so I dont think DEFER_ACCEPT is the 
> cullprit.
>
> I suggest to enable CONFIG_TIMER_STATS and to check timers, because 
> /proc/net/tcp can display apparently large timer values when the timer 
> is elapsed (jiffies > icsk->icsk_timeout) and 
> jiffies_to_clock_t(timer_expires - jiffies) is then overflowing doing 
> a multiply and a divide.

i'm wondering whether your suspicion on broken TCP timers is consistent 
with the symptoms i've seen: the hung sockets clearly produced periodic 
packet activity every 180 seconds, up to 8 hours, without ever changing 
their receive of send queue. So at least a part of the TCP timer 
mechanism for that specific stuck socket was working fine.

is there no sysctl or other debug mechanism to somehow get its full TCP 
state and the reasons for why it is stuck? I'm wondering how you debug 
broken TCP state machines without enabling testers to be able to dump 
all state and passing it to developers.

I have a clearly reproducable testcase and i'd like to help out, but the 
whole effort is stalled on 'not enough information' it appears. Doing 
random reverts might help in truly helpless situations where a bug has 
no debuggable state - but this situation seems really routine to me: 
it's very difficult to trigger the bug but once it triggers the bug 
scenario is stable and analyzable. I'd be glad to test any 
instrumentation patch that makes similar scenarios more analyzable.

	Ingo