slow tcp acks on loopback device

[Steve French <smfltc@us.ibm.com>]

I am seeing odd tcp characteristics on the loopback device.

In analyzing cifs read performance to samba, I see a fairly consistent
pattern.

TCP frame containing SMBRead request (asking for size 16K)
1st 16K of SMB Read Response  (1 ms later or less) sent from samba
	Samba's response is just over 16K (due to protocol header)
wait 40ms doing nothing AFAIK
tcp ack from client to server
TCP frame with last few bytes of the SMB ReadResponse
TCP frame containing next SMB Read Request (asking for size 16K) etc.

I added a setsockopt call to cifs's ipv4_connect to set TCP_NODELAY to 1
(done once just after the connect) which did not seem to help.

Noticing that the loopback device (at least on RHEL4) has an unfortunate
mtu size 16384 (which is about 50 bytes too small for SMB read
responses), I did try increasing the MTU slightly.  Changing that to
18000 did avoid the fragmentation and the 40ms delay - but what puzzled
me was why setting TCP_NODELAY after the socket was created did not
eliminate the delay on the ack and if there is a way to avoid the huge
tcp ack delay by either doing something else to force client acking
immediately or to do something on the client side of the stack to get
the server to send the whole 16K+ frame - it looks like the tcp windows
is 32K if the value in the tcp acks in the network trace is to be
trusted.