Denys Vlasenko wrote: > On Sunday 30 March 2008 07:43, Deomid Ryabkov wrote: > >> This has started recently and i'm at a loss as to why. >> Send-Q on a moderately active UDP socket keeps growing steadily until it >> reaches ~128K (wmem_max?) at which point socket writes start failing. >> The application in question is standard ntpd from Fedora 7, kernel is >> the latest available for the distro, that is >> 2.6.23.15-80.fc7 #1 SMP Sun Feb 10 16:52:18 EST 2008 x86_64 >> >> BIND, running on the same machine, does not exhibit this problem, but >> that may be because it does not get nearly as much load as ntpd, >> which is part of the pool.ntp.org. That said, load is really not very >> high, on the order of 10 QPS, and machine is 99+% idle. >> ntpd seems to be doing its usual select-recvmsg-sendto routine, nothing >> out of the ordinary. >> > > Wher does it (tries to) send these packets? > all over the world :) > I managed to reproduced something like this if I try to send > UDPs to nonexistent host on local subnet. Kernel tries to find it, > it emits ARP probes but no reply is coming. As long as kernel > doesn't know how to send queued UDP packet, I see nonempty > queue. > > However, in my simple case kernel decides that it is a lost case > in a few seconds, and drops packets (queue len 0). > ok, it happened again. no, it's not arp - there are no entries in the arp table. > I imagine whit routing table tricks and/or iptables/arptables > you may end up with situation where kernel is stuck in > "I don't know how to send these packets" mode forever. > nothing fancy on this box - there are no firewall rules, except for on nat rule that does not apply to these packets. > You can strace ntpd, get a list of IPs it is trying to send packets > to, and then do "echo TEST | nc -u 123" for each of these. > will nc's queue become nonempty (at least for some IP)? > as far as i can tell, apart from this one socket networking on the box works normally. this is what i see in netstat: udp 0 125280 89.111.168.177:123 0.0.0.0:* this is how strace looks like (nothing suspicious): select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 382485}) = 1 (in [22], left {0, 125000}) select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 0}) = 1 (in [22], left {0, 0}) recvmsg(22, {msg_name(16)={sa_family=AF_INET, sin_port=htons(101), sin_addr=inet_addr("80.250.211.2")}, msg_iov(1)=[{"#\3\n\356\0\0\17v\0\0\25 \302!\277E\313\324`\257\317\3K\16\0\0\0\0\0\0\0\0"..., 1092}], msg_controllen=32, {cmsg_len=32, cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 48 recvmsg(22, 0x7fffb14354a0, 0) = -1 EAGAIN (Resource temporarily unavailable) sendto(22, "$\4\n\354\0\0\r\345\0\0!\31\n\0\0\2\313\324aR!\334\3242\313\324a\250\t\32\206\224"..., 48, 0, {sa_family=AF_INET, sin_port=htons(101), sin_addr=inet_addr("80.250.211.2")}, 16) = -1 EAGAIN (Resource temporarily unavailable) select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 123523}) = 1 (in [22], left {0, 73000}) select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 0}) = 1 (in [22], left {0, 0}) recvmsg(22, {msg_name(16)={sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("217.77.53.12")}, msg_iov(1)=[{"\31\3\4\372\0\0\16\200\0\7\334F\301}\217\214\313\324R\4+g\\/\0\0\0\0\0\0\0\0"..., 1092}], msg_controllen=32, {cmsg_len=32, cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 48 recvmsg(22, 0x7fffb14354a0, 0) = -1 EAGAIN (Resource temporarily unavailable) sendto(22, "\31\4\4\354\0\0\r\345\0\0!\31\n\0\0\2\313\324aR!\334\3242\313\324a\236$\366\274\212"..., 48, 0, {sa_family=AF_INET, sin_port=htons(123), sin_addr=inet_addr("217.77.53.12")}, 16) = -1 EAGAIN (Resource temporarily unavailable) select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 71771}) = 1 (in [22], left {0, 39000}) select(26, [16 17 18 19 20 21 22 23 24 25], NULL, NULL, {0, 0}) = 1 (in [22], left {0, 0}) recvmsg(22, {msg_name(16)={sa_family=AF_INET, sin_port=htons(29080), sin_addr=inet_addr("213.33.220.118")}, msg_iov(1)=[{"\331\3\4\372\0\0\7\v\0\2\6\262Yl|\4\313\324S\272\322\235e\326\313\324#\260\247\367\352\r"..., 1092}], msg_controllen=32, {cmsg_len=32, cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 48 recvmsg(22, 0x7fffb14354a0, 0) = -1 EAGAIN (Resource temporarily unavailable) sendto(22, "\31\4\4\354\0\0\r\345\0\0!\31\n\0\0\2\313\324aR!\334\3242\313\324a\247;\244\206\\"..., 48, 0, {sa_family=AF_INET, sin_port=htons(29080), sin_addr=inet_addr("213.33.220.118")}, 16) = -1 EAGAIN (Resource temporarily unavailable) etc, etc, etc > > -- > vda > -- Deomid Ryabkov aka Rojer myself@rojer.pp.ru rojer@sysadmins.ru ICQ: 8025844