mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* Re: Kernel 2.4.6 and 2.4.7 networking performance: Seeing serious delays
@ 2001-08-15 17:30 Nivedita Singhvi
  2001-08-16 23:06 ` Alexey Kuznetsov
  0 siblings, 1 reply; 3+ messages in thread
From: Nivedita Singhvi @ 2001-08-15 17:30 UTC (permalink / raw)
  To: ron.flory; +Cc: linux-kernel


> [1.] One line summary of the problem:
>
>  Depending upon length of TCP socket write requests, I'm seeing
> extremely long TCP delays, limiting throughput to only 25 datagrams
> per second. Data blocks above a 'magic' length have a much higher
> throughput than shorter blocks, which is counter-intuitive.

This is correct (but unfortunate) operation of TCP. Youre just
hitting a well known phenomenon: the worst case interaction of
delayed acks and Nagle at a boundary condition.

Note that in the first case, your  write is 1447 bytes, which is
1 byte less than an MSS, i.e. subject to Nagle if there is an
outstanding packet not acked.

Whats happening, we suspect, is that the ack for the 8 byte request
(Frame 10) is delayed once delayed acks kick in (no longer in quick
ack mode). Delayed ack timeout is 40ms, and youre seeing around 36ms.

Since the other side has bytes outstanding and has to send a small
packet (1447 bytes, 1 less than MSS), it will delay the send (Nagle)
until it gets the ack.

> # Time Src Dst Port Info
>  8 0.000111 .100 .101 41359 > 4180 [PSH, ACK] Seq=4186558043
>       Ack=124141127 Win=5840 Len=8
>  9 0.000062 .101 .100 4180 > 41359 [PSH, ACK] Seq=124141127
>       Ack=4186558051 Win=5792 Len=8
> 10 0.036399 .100 .101 41359 > 4180 [ACK] Seq=4186558051
>       Ack=124141135 Win=5840 Len=0 <- DELAY
> 11 0.000074 .101 .100 4180 > 41359 [PSH, ACK] Seq=124141135
>       Ack=4186558051 Win=5792 Len=1447
> 12 0.000362 .100 .101 41359 > 4180 [ACK] Seq=4186558051
>       Ack=124142582 Win=8682 Len=0
> 13 0.000279 .100 .101 41359 > 4180 [FIN, ACK] Seq=4186558051
>       Ack=124142582 Win=8682 Len=0
> 14 0.000073 .101 .100 4180 > 41359 [FIN, ACK] Seq=124142582
>       Ack=4186558052 Win=5792 Len=0

In the next session, when youre sending the 1448 bytes, (exactly MSS),
the ack is delayed, but the write isnt delayed since we have a big enough
packet to send. The other end sends the ack as soon as it receives the
1447 bytes. (Acks are delayed unless we have more than one MSS
outstanding),
so it doesnt need to wait the full delayed ack timeout.

>  # Time Src Dst Port Info
>  8 0.000122 .100 .101 41363 > 4180 [PSH, ACK] Seq=4268751138
>       Ack=189815010 Win=5840 Len=8
>  9 0.000061 .101 .100 4180 > 41363 [PSH, ACK] Seq=189815010
>       Ack=4268751146 Win=5792 Len=8
> 10 0.000034 .101 .100 4180 > 41363 [ACK] Seq=189815018
>       Ack=4268751146 Win=5792 Len=1448
> 11 0.000351 .100 .101 41363 > 4180 [ACK] Seq=4268751146
>       Ack=189816466 Win=8688 Len=0
> 12 0.000520 .100 .101 41363 > 4180 [FIN, ACK] Seq=4268751146
>       Ack=189816466 Win=8688 Len=0

> Of particular interest is line #10 of each above.
>
> For some reason, in the case of the 'short' packet, the server machine
> waits for a TCP ACK from the client machine (delaying 36 millisec),
> which does not happen if the packet is just 1-byte longer, in which
> case the server immediately follows the 8-byte block with the
> 1448-byte payload.
[ snip ]

Hope that helps. Its a consequence of sending repeated small packets.
You can turn off Nagle (delayed sends) by setting the TCP socket option
TCP_NODELAY, or alter your application sequence of writes, if possible.

I'm still wondering why the next ack isnt delayed as well, though.
I dont believe we should still be in quickack mode, and
so I would have expected subsequent acks (there's just one, the FIN comes
down as a result of the close, which might be a factor, masked by ethereal
somehow) to be delayed as well.

Thanks to Sridhar Samudrala, Dave Stevens, Shirley Ma who got interested
in the question and contributed in shedding light on whats happening..

thanks,
Nivedita
----
Nivedita Singhvi                            (503) 578-4580
Linux Technology Center           nivedita@us.ibm.com
IBM Beaverton, OR                      nivedita@sequent.com


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Kernel 2.4.6 and 2.4.7 networking performance: Seeing serious delays
  2001-08-15 17:30 Kernel 2.4.6 and 2.4.7 networking performance: Seeing serious delays Nivedita Singhvi
@ 2001-08-16 23:06 ` Alexey Kuznetsov
  0 siblings, 0 replies; 3+ messages in thread
From: Alexey Kuznetsov @ 2001-08-16 23:06 UTC (permalink / raw)
  To: Nivedita Singhvi; +Cc: linux-kernel

Hello!

> I'm still wondering why the next ack isnt delayed as well, though.

Because out tcp is enough clever to understand that it made
a shit delaying ACK for previous packet. But not enough psychic
to foresee future and to avoid delay. :-)


> I dont believe we should still be in quickack mode, 

However, we are there. :-)


 4 0.000061 .100 .101 41359 > 4180 [PSH, ACK] Seq=4186558035 Ack=124141119 Win=5
840 Len=8
 5 0.000061 .101 .100 4180 > 41359 [ACK] Seq=124141119 Ack=4186558043 Win=5792 L
en=0

.101 quickacks

 6 0.002295 .101 .100 4180 > 41359 [PSH, ACK] Seq=124141119 Ack=4186558043 Win=5
792 Len=8

.101 leaves quickacks setting pingpong


 7 0.000098 .100 .101 41359 > 4180 [ACK] Seq=4186558043 Ack=124141127 Win=5840 L
en=0

.100 quickacks

 8 0.000111 .100 .101 41359 > 4180 [PSH, ACK] Seq=4186558043 Ack=124141127 Win=5
840 Len=8

.100 leaves quickacks setting pingpong


 9 0.000062 .101 .100 4180 > 41359 [PSH, ACK] Seq=124141127 Ack=4186558051 Win=5
792 Len=8
10 0.036399 .100 .101 41359 > 4180 [ACK] Seq=4186558051 Ack=124141135 Win=5840 L
en=0  <- DELAY

.100 experiences delack timeout, which is considered as an error.
It is really error. So, tcp clears pingpong and returns to quickack.

11 0.000074 .101 .100 4180 > 41359 [PSH, ACK] Seq=124141135 Ack=4186558051 Win=5
792 Len=1447
12 0.000362 .100 .101 41359 > 4180 [ACK] Seq=4186558051 Ack=124142582 Win=8682 L
en=0  

.100 quickacks

This loop would repeat infinitely, if connection were not closed.
Kernel sees that user replies fastly, switches to canonical BSD mode
to allow ack piggibacking and tries to delay ack, but silly user does
not want to reply more. So, we have reduced amount of timeouts twice,
but this is not a big win in this case.

I am writing this mostly because it is the best demonstration
why TCP_QUICKACK:=0 is invalidated. :-)


What's about advice to use TCP_NODELAY in this case, it is _absolutely_
wrong and in fact presents violation of protocol, this app sends silly
spagetti of pakets which must be coalesced. It is exactly the situation
when TCP_NODELAY is straight bug. This app should use TCP_CORK or,
even better from the viewpoint of performance, to coalesce writes at user user
level not abusing tcp.

Alexey

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Kernel 2.4.6 and 2.4.7 networking performance: Seeing serious delays
@ 2001-08-21  4:00 Nivedita Singhvi
  0 siblings, 0 replies; 3+ messages in thread
From: Nivedita Singhvi @ 2001-08-21  4:00 UTC (permalink / raw)
  To: kuznet; +Cc: linux-kernel


Alexey,

Thanks for the clarification on the pingpong behaviour!
Some more questions..

>> I'm still wondering why the next ack isnt delayed as well, though.
>
>Because out tcp is enough clever to understand that it made
>a shit delaying ACK for previous packet. But not enough psychic
>to foresee future and to avoid delay. :-)


>.100 experiences delack timeout, which is considered as an error.
>It is really error. So, tcp clears pingpong and returns to quickack.

So I understand that we're allowing pingpong to retain state of whether
it makes sense to delay right now or not, based on whats been happening,
and trying to avoid a delayed ack timer expiry..

But...

>This loop would repeat infinitely, if connection were not closed.
>Kernel sees that user replies fastly, switches to canonical BSD mode
>to allow ack piggibacking and tries to delay ack, but silly user does
>not want to reply more. So, we have reduced amount of timeouts twice,
>but this is not a big win in this case.
>
>I am writing this mostly because it is the best demonstration
>why TCP_QUICKACK:=0 is invalidated. :-)

Then whats the point of TCP_QUICKACK socket option at all? If its only
a user hint that gets erased on the first go through, that is one full system call just
 to avoid a delay on one packet. The user isnt going to write an application full of
setsockopt() calls based on his data sends, right?


>What's about advice to use TCP_NODELAY in this case, it is _absolutely_
>wrong and in fact presents violation of protocol, this app sends silly
>spagetti of pakets which must be coalesced. It is exactly the situation
>when TCP_NODELAY is straight bug. This app should use TCP_CORK or,
>even better from the viewpoint of performance, to coalesce writes at user
>user  level not abusing tcp.

True, this is an abuse of TCP :). But we should have the freedom to do so, right? :)
i.e. User may need to send small amounts of data at a time.  The user might even
want to have that data sent out immediately, and not want  the kernel to delay the send.
After all, if the application is sending down large full frames, the kernel is by default
not going to be halted by Nagle. Its really  only when you are sending small packets
at a time that youre affected by Nagle and might want to set NODELAY, especially
if that application is highly interactive?.

thanks,
Nivedita


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2001-08-21  4:39 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2001-08-15 17:30 Kernel 2.4.6 and 2.4.7 networking performance: Seeing serious delays Nivedita Singhvi
2001-08-16 23:06 ` Alexey Kuznetsov
2001-08-21  4:00 Nivedita Singhvi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®