Re: ZONE_NORMAL memory exhausted by 4000 TCP sockets

["Zhao Xiaoming" <xiaoming.nj@gmail.com>]

2006/11/6, Eric Dumazet <dada1@cosmosbay.com>:
> Zhao Xiaoming a écrit :
> > Dears,
> >    I'm running a linux box with kernel version 2.6.16. The hardware
> > has 2 Woodcrest Xeon CPUs (2 cores each) and 4G RAM. The NIC cards is
> > Intel 82571 on PCI-e bus.
> >    The box is acting as ethernet bridge between 2 Gigabit Ethernets.
> > By configuring ebtables and iptables, an application is running as TCP
> > proxy which will intercept all TCP connections requests from the
> > network and setup another TCP connection to the acture server.  The
> > TCP proxy then relays all traffics in both directions.
> >    The problem is the memory. Since the box must support thousands of
> > concurrent connections, I know the memory size of ZONE_NORMAL would be
> > a bottleneck as TCP packets would need many buffers. After setting
> > upper limit of net.ipv4.tcp_rmem and net.ipv4.tcp_wmem to 32K bytes,
> > our test began.
> >    My test scenario employs 2000 concurrent downloading connections
> > to a IIS server's port 80. The throughput is about 500~600 Mbps which
> > is limited by the capability of the client application. Because all
> > traffics are from server to client and the capability of client
> > machine is bottleneck, I believe the receiver side of the sockets
> > connected with server and the sender side of the sockets connected
> > with client should be filled with packets in correspondent windows.
> > Thus, roughly there should be about 32K * 2000+ 32K*2000 = 128M bytes
> > memory occupied by TCP/IP stack for packet buffering. Data from
> > slabtop confermed it. it's about 140M bytes memory cost after I start
> > the traffic. That reasonablly matched with my estimation. However,
> > /proc/meminfo had a different story. The 'LowFree' dropped from about
> > 710M to 80M. In other words, there's addtional 500M memory in
> > ZONE_NORMAL allocated by someone other than the slab. Why?
>
> We dont know. You might post some data so that we can have some ideas.
>
> Also, these kind of question is better handled by linux netdev mailing list,
> so I added a CC to this list.
>
> cat /proc/slabinfo
> cat /proc/meminfo
> cat /proc/net/sockstat
> cat /proc/buddyinfo
>
> >   I also made another test that the upper limit of tcp_rmem and
> > tcp_wmem being set to 64K. After 2000 connections transfering a lot of
> > data for several seconds, the linux box showed some error messages
> > such as error allocating memory pages, etc. and became unstable.
> >   My questions are:
> >
> > 1. To calculate memory request of TCP sockets, is there any other
> > large amount of memory requested besides send and receive buffer?
> > 2. Is there any logics that emploied by TCP/IP stack that will
> > dynamically allocating memory pages directly instead of from slab?
>
> TCP stack is one thing, but other things may consume ram on your kernel.
>
> Also, kernel memory allocation might use twice the ram you intend to use
> because of power of two alignments.
>
> Are you using iptables connection tracking ?
>
> If you plan to use a lot of RAM in kernel, why dont you use a 64 bits kernel,
> so that all ram is available for kernel, not only 900 MB ?
>
> Eric
>
>
Thanks for the answer. I know it's more likely relats to netdev.
However, it's always a strange thing to have 400~500M bytes LOWMEM
'gone' while it's not reported to be occupied by slab. Both meminfo
and buddyinfo tell the same.

with traffics of 2000 concurrent sessions:

cat /proc/meminfo
MemTotal:      4136580 kB
MemFree:       3298460 kB
Buffers:          4096 kB
Cached:          21124 kB
SwapCached:          0 kB
Active:          47416 kB
Inactive:        12532 kB
HighTotal:     3276160 kB
HighFree:      3214592 kB
LowTotal:       860420 kB
LowFree:         83868 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:              12 kB
Writeback:           0 kB
Mapped:          42104 kB
Slab:           293952 kB
CommitLimit:   2068288 kB
Committed_AS:    58892 kB
PageTables:       1112 kB
VmallocTotal:   116728 kB
VmallocUsed:      2940 kB
VmallocChunk:   110548 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB


without traffics:

cat /proc/meminfo
MemTotal:      4136580 kB
MemFree:       3924276 kB
Buffers:          4460 kB
Cached:          21020 kB
SwapCached:          0 kB
Active:          47592 kB
Inactive:        12848 kB
HighTotal:     3276160 kB
HighFree:      3214716 kB
LowTotal:       860420 kB
LowFree:        709560 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:              40 kB
Writeback:           0 kB
Mapped:          42368 kB
Slab:           132172 kB
CommitLimit:   2068288 kB
Committed_AS:    59220 kB
PageTables:       1140 kB
VmallocTotal:   116728 kB
VmallocUsed:      2940 kB
VmallocChunk:   110548 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB


Xiaoming.