Re: 2.6.17: networking bug??

[David Miller <davem@davemloft.net>]

From: Mark Lord <lkml@rtr.ca>
Date: Tue, 13 Jun 2006 17:49:21 -0400

> I suppose the most important objection to our current behaviour
> is that this behaviour *changes* when something totally unrelated
> (to Joe User) happens:  adding or removing a stick of RAM.

We are pretty much required to choose the TCP memory parameters
based upon how much physical memory is in the machine, and these
parameters in-turn are inextricably linked to what kind of window
scale we try to use for connections.

The behavior is unfortunate, but more unfortunate are the boxes that
create these problems in the first place.  I believe their lifespan is
quite limited.

> We should perhaps just have a fixed upper memory setting, as we
> currently do in 2.6.16, so that the behaviour is predictable.

The change in 2.6.17 was exactly that we needed to increase this
upper limit to ~4MB.

> On a related note.. I wonder if we can choose better values for
> the window size, so that if the scale factor is ignored, we still
> end up with reasonably sized packets?  So that the other box
> will not think our window is a mere "94" when the scale factor
> is lost?

We have an algorithm that tries to pick something based upon the
set of the values we might need to represent in the window field.

If the scale is too high, you lose accuracy, since the lower bits
get chopped off when the TCP header is being built and the computed
window size is shifted down.

So we try to pick the smallest scale necessary to represent the
largest window size we might end up needing to advertise.

A complication here is that we dynamically size both receive and send
buffers in response to our growing knowledge of the connection's
characteristics over time.  So at the beginning we'll use a small
buffer size, and as the congestion window grows we'll increase our
buffer sizes to fill the pipe.

This adds even more considerations for window scale selection, as you
can imagine.

One final word about window sizes.  If you have a connection whose
bandwidth-delay-product needs an N byte buffer to fill, you actually
have to have an "N * 2" sized buffer available in order for fast
retransmit to work.