To try to figure out some buffering problems my network application is having, 
I wrote 2 tiny utilities
  - pwrite connects to a socket and floods it with data until it sleeps
  - pread accepts an incoming connection but never actually reads data.
    It just lets it pile up until the read buffer is full.

Therefore, pwrite will go to sleep once both its write buffer and pread's read 
buffer have filled up.

echo 4096 4096 4096 > /proc/sys/net/ipv4/tcp_rmem
echo 4096 4096 4096 > /proc/sys/net/ipv4/tcp_wmem

Now that I've established a  4K read buffer and a 4K write buffer, pwrite 
should go to go to sleep after writing 8K of data, right?  Except that it 
only manages to write 5K of data before sleeping!


Here is the output from running pread:

bronson@lea:~/linkanywhere/sockbuf/kernel$ ./pread 
waiting on port 1111...
Read buffer is 4096 bytes, with 0 bytes used.        (printed at startup)
Read buffer is 4096 bytes, with 3072 bytes used.   (printed on exit)


And simultaneously running pwrite with 1024-byte writes:

bronson@lea:~/kernel$ ./pwrite 1024
Allocating 1024 bytes for write buffer.
 0: SO_SNDBUF=  4096, SIOCOUTQ=     0:   4096 avail.  Wrote 1024 bytes, 1024 
total.
 1: SO_SNDBUF=  4096, SIOCOUTQ=     0:   4096 avail.  Wrote 1024 bytes, 2048 
total.
 2: SO_SNDBUF=  4096, SIOCOUTQ=  1024:   3072 avail.  Wrote 1024 bytes, 3072 
total.
 3: SO_SNDBUF=  4096, SIOCOUTQ=  2048:   2048 avail.  Wrote 1024 bytes, 4096 
total.
 4: SO_SNDBUF=  4096, SIOCOUTQ=  2048:   2048 avail.  Wrote 1024 bytes, 5120 
total.


Why on earth is it sleeping after only 5K?  According to the SIOC calls, the 
write buffer still had 2048 bytes available, and the read buffer 1024 bytes.  
The kernel claims that there's still 3K of space that can be filled.  So why 
isn't it?

Thanks,

    - Scott