[BUG] TCP connection deadlock under simultaneous bidirectional ICSK_ACK_NOMEM (OOM)

[xietangxin <xietangxin@yeah.net>]

Hi all,

We have observed a TCP connection deadlock on stable 6.6 under heavy stress testing.

1.Both Peer A and Peer B enter the ICSK_ACK_NOMEM branch in tcp_select_window().
After commit 8c670bdfa58e ("tcp: correct handling of extreme memory squeeze"),
Both peers freeze their rcv_nxt and set rcv_wnd = 0.

2.Prior to freezing, both sides had already sent out flight data.
Since both sides are dropping incoming data packets due to OOM, rcv_nxt stops advancing,
but the peer's seq of subsequent packets continues to grow.

3.When Peer A receives Peer B's Zero Window ACK,
the packet's seq is far ahead of Peer A's frozen rcv_nxt.
Both peers drop each other's packet, also no Zero Window Probes are triggered
because snd_wnd is never updated to 0.


Simplified Packet Trace:

Assume Peer A's rcv_nxt = 1000, and Peer B's rcv_nxt = 5000 initially.

Time  Dir      Type        Seq   Ack   Win  Len  Status
------------------------------------------------------------------------
T1:   B -> A   [PSH, ACK]  1000  5000  3000 100  (A hits OOM, rcv_nxt=1000)
T2:   B -> A   [ACK]       1100  5000  3000 200  (Dropped due to A's OOM)
T3:   B -> A   [PSH, ACK]  1300  5000  3000 200  (Dropped due to A's OOM)

T4:   A -> B   [PSH, ACK]  5000  1000  3000 100  (B hits OOM, rcv_nxt=5000)
T5:   A -> B   [ACK]       5100  1000  3000 200  (Dropped due to B's OOM)
T6:   A -> B   [PSH, ACK]  5300  1000  3000 200  (Dropped due to B's OOM)

-- Both sides are now in OOM. B's Seq is 1500; A's Seq is 5500 --

T7:   B -> A   [ZeroWin]   1500  5000  0    0    (Dropped: Seq 1500 != 1000)
T8:   A -> B   [ZeroWin]   5500  1000  0    0    (Dropped: Seq 5500 != 5000)
T9:   A -> B   [WinUpdate] 5500  1000  20   0    (Dropped: Seq 5500 != 5000)

Should we relax the sequence check in tcp_sequence() for zero window ACK?

Any feedback or guidance would be greatly appreciated.

-- 
Best regards,
Tangxin Xie