Hello All, [1.]--------------------Short summary: GRE tunnel uses wrong SRC in IP packet. [2.]--------------------Summary: After GRE tunnel disconnection, despite SRC in tunnel settings (from "ip tunnel") outcoming packets use interface settings (randomly? from "ip addr"). When interface has more then one IP address, packets could use wrong SRC and so ignored on destination host. [3.]------------------- Keywords gre, tcp/ip, tunnel --------------------System: HostB connects to HostA (which has multiply IPs on interface eth0) via gre. HostA: Red Hat Linux version 2.6.12-05 (gcc version 3.4.3) (and kernel 2.6.10 too), iproute2-ss040831 HostB: Not tested, kernel 2.4.30 [6.]--------------------When: Not always, but usually happens in that sequence: - Reboot of HostA, - Establishing GRE tunnel, works fine - Disconnect HostB for long period of time, 10-30 minutes - Delete existing tunnels - Establish GRE tunnel. - Problem is exist, packets are not delivered any more. - I do have multiply tunnels (6 at the some time), all of them seems to be independent and each of them could be affected or non-affected independently. --------------------Work around: "ip route add *.*.*.* via 64.26.181.225 src 64.26.161.230 table T1" command has no effect on SRC of gre traffic. The only one way to connect gre (except reboot HostA) is to delete tunnel on hostB and re-establish it to other IP address which currently HostA puts in SRC field. Seems as compiling gre as module or as non-module does not affect the bug. --------------------Bug Reproducing Commands to follow: HostA range=192.168.2.0/24 remote=64.200.201.202 local=64.26.161.230 ip=192.168.1.1 ip tunnel del tnl0 ip tunnel add tnl0 mode gre local $local remote $remote ttl 255 ip addr add $ip dev tnl0 ip link set tnl0 up ip route add $range dev tnl0 HostB range=192.168.1.0/24 remote=64.26.161.230 local=64.100.101.102 ip=192.168.2.1 ip tunnel del tnl0 ip tunnel add tnl0 mode gre local $local remote $remote ttl 255 ip addr add $ip dev tnl0 ip link set $tnl up ip route add $range dev tnl0 Check: [HostA]# ping 192.168.2.1 >> /dev/null & Expected (without bug): [HostA]# tcpdump -i eth0 host 64.100.101.102 -n 17:53:09.919555 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 21 17:53:09.951859 IP 64.200.201.202 > 64.26.161.230: IP 192.168.2.1 > 192.168.1.1: icmp 64: echo reply seq 21 17:53:10.920393 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 22 17:53:10.943320 IP 64.200.201.202 > 64.26.161.230: IP 192.168.2.1 > 192.168.1.1: icmp 64: echo reply seq 22 17:53:11.921231 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 23 17:53:11.943070 IP 64.200.201.202 > 64.26.161.230: IP 192.168.2.1 > 192.168.1.1: icmp 64: echo reply seq 23 17:53:12.922074 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 24 17:53:12.942243 IP 64.200.201.202 > 64.26.161.230: IP 192.168.2.1 > 192.168.1.1: icmp 64: echo reply seq 24 17:53:13.922922 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 25 17:53:13.943235 IP 64.200.201.202 > 64.26.161.230: IP 192.168.2.1 > 192.168.1.1: icmp 64: echo reply seq 25 17:53:14.923754 IP 64.26.161.230 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 26 Received (with bug): [HostA]# tcpdump -i eth0 host 64.100.101.102 -n 17:59:11.242867 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 34 17:59:11.276509 IP 64.200.201.202 > 64.26.181.226: icmp 116: 64.200.201.202 protocol 47 unreachable 17:59:12.242719 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 35 17:59:12.270244 IP 64.200.201.202 > 64.26.181.226: icmp 116: 64.200.201.202 protocol 47 unreachable 17:59:13.242554 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 36 17:59:13.267199 IP 64.200.201.202 > 64.26.181.226: icmp 116: 64.200.201.202 protocol 47 unreachable 17:59:14.242398 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 37 17:59:14.267350 IP 64.200.201.202 > 64.26.181.226: icmp 116: 64.200.201.202 protocol 47 unreachable 17:59:15.242247 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 38 17:59:15.268835 IP 64.200.201.202 > 64.26.181.226: icmp 116: 64.200.201.202 protocol 47 unreachable 17:59:16.242075 IP 64.26.181.226 > 64.200.201.202: IP 192.168.1.1 > 192.168.2.1: icmp 64: echo request seq 39 As You see, outcoming packet has wrong SRC and ignored by destination HostB. ---------------------------- Here some configuration output (due I have 6 tunnels I deleted lines which belongs to other five): [HostA]# ip addr 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0: mtu 1480 qdisc noop link/ipip 0.0.0.0 brd 0.0.0.0 3: gre0: mtu 1476 qdisc noop link/gre 0.0.0.0 brd 0.0.0.0 4: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0e:2e:21:0d:8b brd ff:ff:ff:ff:ff:ff inet 64.26.161.230/28 brd 64.26.161.239 scope global eth0 inet 64.26.181.226/28 scope global secondary eth0 inet 64.26.181.231/28 scope global secondary eth0 inet6 fe80::20e:2eff:fe21:d8b/64 scope link valid_lft forever preferred_lft forever 5: eth1: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0e:2e:21:27:10 brd ff:ff:ff:ff:ff:ff inet 192.168.1.1/24 brd 192.168.1.255 scope global eth1 inet6 fe80::20e:2eff:fe21:2710/64 scope link valid_lft forever preferred_lft forever [HostA]# ip tunnel tunl0: ip/ip remote any local any ttl inherit nopmtudisc gre0: gre/ip remote any local any ttl inherit nopmtudisc sit0: ipv6/ip remote any local any ttl 64 nopmtudisc tnl0: gre/ip remote 64.200.201.202 local 64.26.161.230 ttl 255 [HostA]# ip link 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: tunl0: mtu 1480 qdisc noop link/ipip 0.0.0.0 brd 0.0.0.0 3: gre0: mtu 1476 qdisc noop link/gre 0.0.0.0 brd 0.0.0.0 4: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0e:2e:21:0d:8b brd ff:ff:ff:ff:ff:ff 5: eth1: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0e:2e:21:27:10 brd ff:ff:ff:ff:ff:ff 8: sit0: mtu 1480 qdisc noop link/sit 0.0.0.0 brd 0.0.0.0 33: ppp1: mtu 1492 qdisc pfifo_fast qlen 3 link/ppp 45: ppp0: mtu 1492 qdisc pfifo_fast qlen 3 link/ppp 71: tnl0@NONE: mtu 1476 qdisc noqueue link/gre 64.26.161.230 peer 64.200.201.202 [HostA]# ip route 64.26.181.224/28 dev eth0 proto kernel scope link src 64.26.161.230 192.168.2.0/24 dev tnl0 scope link 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.1 default via 64.26.181.225 dev eth0 [HostA] lsmod [root@helix usr]# lsmod Module Size Used by ppp_synctty 14976 0 ppp_async 16512 2 crc_ccitt 6400 1 ppp_async ip_nat_ftp 7808 0 ip_conntrack_ftp 77072 1 ip_nat_ftp ipt_REJECT 9728 2 ipt_limit 6656 1 ipt_LOG 11392 14 iptable_mangle 7040 1 iptable_filter 7168 1 iptable_nat 27068 2 ip_nat_ftp ip_conntrack 47464 3 ip_nat_ftp,ip_conntrack_ftp,iptable_nat ip_tables 25856 6 ipt_REJECT,ipt_limit,ipt_LOG,iptable_mangle,iptable_filter,iptable_nat md5 8320 1 ipv6 267840 26 autofs4 23684 0 sunrpc 143172 1 video 19844 0 button 10896 0 battery 13572 0 ac 9092 0 uhci_hcd 35856 0 ehci_hcd 38664 0 i2c_i801 12812 0 i2c_core 25856 1 i2c_i801 b44 27012 0 8139too 30336 0 mii 9856 2 b44,8139too floppy 63252 0 [HostA] cat /proc/modules ppp_synctty 14976 0 - Live 0xe0b05000 ppp_async 16512 2 - Live 0xe0ac0000 crc_ccitt 6400 1 ppp_async, Live 0xe0abd000 ip_nat_ftp 7808 0 - Live 0xe0ab6000 ip_conntrack_ftp 77072 1 ip_nat_ftp, Live 0xe0af1000 ipt_REJECT 9728 2 - Live 0xe0ab9000 ipt_limit 6656 1 - Live 0xe0ab3000 ipt_LOG 11392 14 - Live 0xe0aab000 iptable_mangle 7040 1 - Live 0xe0ac8000 iptable_filter 7168 1 - Live 0xe0a8d000 iptable_nat 27068 2 ip_nat_ftp, Live 0xe0a85000 ip_conntrack 47464 3 ip_nat_ftp,ip_conntrack_ftp,iptable_nat, Live 0xe0b54000 ip_tables 25856 6 ipt_REJECT,ipt_limit,ipt_LOG,iptable_mangle,iptable_filter,iptable_nat, Live 0xe0a7d000 md5 8320 1 - Live 0xe0a91000 ipv6 267840 24 - Live 0xe0a3a000 autofs4 23684 0 - Live 0xe0a9e000 sunrpc 143172 1 - Live 0xe0bb5000 video 19844 0 - Live 0xe087a000 button 10896 0 - Live 0xe0876000 battery 13572 0 - Live 0xe0871000 ac 9092 0 - Live 0xe0851000 uhci_hcd 35856 0 - Live 0xe0867000 ehci_hcd 38664 0 - Live 0xe0846000 i2c_i801 12812 0 - Live 0xe0833000 i2c_core 25856 1 i2c_i801, Live 0xe083e000 b44 27012 0 - Live 0xe0824000 8139too 30336 0 - Live 0xe081b000 mii 9856 2 b44,8139too, Live 0xe083a000 floppy 63252 0 - Live 0xe0856000 [HostA]/usr/src/linux-2.6.12/scripts/ver_linux If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. Linux helix 2.6.12-05 #2 SMP Mon Jul 4 11:39:22 EDT 2005 i686 i686 i386 GNU/Linux Gnu C 3.4.2 Gnu make 3.80 binutils 2.15.92.0.2 util-linux 2.12a mount 2.12a module-init-tools 3.1-pre5 e2fsprogs 1.35 reiserfsprogs line reiser4progs line pcmcia-cs 3.2.7 quota-tools 3.12. PPP 2.4.2 isdn4k-utils 3.3 nfs-utils 1.0.6 Linux C Library 2.3.3 Dynamic linker (ldd) 2.3.3 Procps 3.2.3 Net-tools 1.60 Kbd 1.12 Sh-utils 5.2.1 udev 039 Modules Loaded ppp_synctty ppp_async crc_ccitt ip_nat_ftp ip_conntrack_ftp ipt_REJECT ipt_limit ipt_LOG iptable_mangle iptable_filter iptable_nat ip_conntrack ip_tables md5 ipv6 autofs4 sunrpc video button battery ac uhci_hcd ehci_hcd i2c_i801 i2c_core b44 8139too mii floppy ----------------- [HostB]# ip addr 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo 2: teql0: mtu 1500 qdisc noop qlen 100 link/void 3: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:fc:ec:74:39 brd ff:ff:ff:ff:ff:ff inet 64.200.201.202/18 brd ******* scope global eth0 4: eth1: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:fc:ec:74:4e brd ff:ff:ff:ff:ff:ff inet 192.168.2.1/24 brd 127.255.255.255 scope global eth1 5: tunl0@NONE: mtu 1480 qdisc noop link/ipip 0.0.0.0 brd 0.0.0.0 6: gre0@NONE: mtu 1476 qdisc noop link/gre 0.0.0.0 brd 0.0.0.0 30: tnl0@NONE: mtu 1476 qdisc noqueue link/gre 64.200.201.202 peer 64.26.161.230 inet 192.168.2.1/32 scope global tnl0 [HostB]# ip link 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: teql0: mtu 1500 qdisc noop qlen 100 link/void 3: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:fc:ec:74:39 brd ff:ff:ff:ff:ff:ff 4: eth1: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:fc:ec:74:4e brd ff:ff:ff:ff:ff:ff 5: tunl0@NONE: mtu 1480 qdisc noop link/ipip 0.0.0.0 brd 0.0.0.0 6: gre0@NONE: mtu 1476 qdisc noop link/gre 0.0.0.0 brd 0.0.0.0 30: tnl0@NONE: mtu 1476 qdisc noqueue link/gre 64.200.201.202 peer 64.26.161.230 [HostB]# ip tunnel tunl0: ip/ip remote any local any ttl inherit nopmtudisc gre0: gre/ip remote any local any ttl inherit nopmtudisc tnl0: gre/ip remote 64.26.161.230 local 64.200.201.202 ttl 255 ------------------------------------------------------- Kernel config attached. Thank You for Your time!