IPv4 saddr do not match with selected output device in double default gateways scene

IPv4 saddr do not match with selected output device in double default gateways scene

[Ziyang Xuan (William)]

Create VLAN devices and add default gateways with following commands:

# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.41/24 dev eth2.71
# ip addr add 192.168.72.41/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up
# route add -net default gw 192.168.71.1 dev eth2.71
# route add -net default gw 192.168.72.1 dev eth2.72

Add a nameserver configuration in the following file:
# cat /etc/resolv.conf
nameserver 8.8.8.8

Use the following command trigger DNS packet:
# ping www.baidu.com

Assume the above test machine is client.

Of course, we should also create VLAN devices in peer server as following:

# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.1/24 dev eth2.71
# ip addr add 192.168.72.1/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up

We capture packets with tcpdump in client machine when ping:
# tcpdump -i eth2 -vne
...
20:30:22.996044 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25407, offset 0, flags [DF], proto UDP (17), length 59)
    192.168.72.41.42666 > 8.8.8.8.domain: 58562+ A? www.baidu.com. (31)
20:30:22.996125 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25408, offset 0, flags [DF], proto UDP (17), length 59)
    192.168.72.41.42666 > 8.8.8.8.domain: 25803+ AAAA? www.baidu.com. (31)
...

We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".

I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.

The problem happens in the connect() process. Analysis as following with codes:

static inline struct rtable *ip_route_connect(struct flowi4 *fl4,
					      __be32 dst, __be32 src, u32 tos,
					      int oif, u8 protocol,
					      __be16 sport, __be16 dport,
					      struct sock *sk)
{
	struct net *net = sock_net(sk);
	struct rtable *rt;

	ip_route_connect_init(fl4, dst, src, tos, oif, protocol,
			      sport, dport, sk);

	if (!dst || !src) {

		/* rtable and fl4 are matched after the first __ip_route_output_key().
		 * rtable->dst.dev->name == "eth2.72" && rtable->rt_gw4 == 0x148a8c0
		 * fl4->saddr == 0x2948a8c0
		 */
		rt = __ip_route_output_key(net, fl4);
		if (IS_ERR(rt))
			return rt;
		ip_rt_put(rt);
		flowi4_update_output(fl4, oif, tos, fl4->daddr, fl4->saddr);
	}
	security_sk_classify_flow(sk, flowi4_to_flowi_common(fl4));

	/* rtable and fl4 do not match after the second __ip_route_output_key().
	 * rtable->dst.dev->name == "eth2.71" && rtable->rt_gw4 == 0x147a8c0
	 * fl4->saddr == 0x2948a8c0
	 */
	return ip_route_output_flow(net, fl4, sk);
}

Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
So the phenomenon described at the beginning happens.

Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?

Re: IPv4 saddr do not match with selected output device in double default gateways scene

[Ziyang Xuan (William)]

> Create VLAN devices and add default gateways with following commands:
> 
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.41/24 dev eth2.71
> # ip addr add 192.168.72.41/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
> # route add -net default gw 192.168.71.1 dev eth2.71
> # route add -net default gw 192.168.72.1 dev eth2.72
> 
> Add a nameserver configuration in the following file:
> # cat /etc/resolv.conf
> nameserver 8.8.8.8
> 
> Use the following command trigger DNS packet:
> # ping www.baidu.com
> 
> Assume the above test machine is client.
> 
> Of course, we should also create VLAN devices in peer server as following:
> 
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.1/24 dev eth2.71
> # ip addr add 192.168.72.1/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
> 
> We capture packets with tcpdump in client machine when ping:
> # tcpdump -i eth2 -vne
> ...
> 20:30:22.996044 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25407, offset 0, flags [DF], proto UDP (17), length 59)
>     192.168.72.41.42666 > 8.8.8.8.domain: 58562+ A? www.baidu.com. (31)
> 20:30:22.996125 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25408, offset 0, flags [DF], proto UDP (17), length 59)
>     192.168.72.41.42666 > 8.8.8.8.domain: 25803+ AAAA? www.baidu.com. (31)
> ...
> 
> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".

Is there anyone familiar with route/fib realization? And thank you for your warm-hearted help!

> 
> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
> 
> The problem happens in the connect() process. Analysis as following with codes:
> 
> static inline struct rtable *ip_route_connect(struct flowi4 *fl4,
> 					      __be32 dst, __be32 src, u32 tos,
> 					      int oif, u8 protocol,
> 					      __be16 sport, __be16 dport,
> 					      struct sock *sk)
> {
> 	struct net *net = sock_net(sk);
> 	struct rtable *rt;
> 
> 	ip_route_connect_init(fl4, dst, src, tos, oif, protocol,
> 			      sport, dport, sk);
> 
> 	if (!dst || !src) {
> 
> 		/* rtable and fl4 are matched after the first __ip_route_output_key().
> 		 * rtable->dst.dev->name == "eth2.72" && rtable->rt_gw4 == 0x148a8c0
> 		 * fl4->saddr == 0x2948a8c0
> 		 */
> 		rt = __ip_route_output_key(net, fl4);
> 		if (IS_ERR(rt))
> 			return rt;
> 		ip_rt_put(rt);
> 		flowi4_update_output(fl4, oif, tos, fl4->daddr, fl4->saddr);
> 	}
> 	security_sk_classify_flow(sk, flowi4_to_flowi_common(fl4));
> 
> 	/* rtable and fl4 do not match after the second __ip_route_output_key().
> 	 * rtable->dst.dev->name == "eth2.71" && rtable->rt_gw4 == 0x147a8c0
> 	 * fl4->saddr == 0x2948a8c0
> 	 */
> 	return ip_route_output_flow(net, fl4, sk);
> }
> 
> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
> So the phenomenon described at the beginning happens.
> 
> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
> 

Re: IPv4 saddr do not match with selected output device in double default gateways scene

[David Ahern]

On 3/7/22 11:41 PM, Ziyang Xuan (William) wrote:
> Create VLAN devices and add default gateways with following commands:
> 
> # ip link add link eth2 dev eth2.71 type vlan id 71
> # ip link add link eth2 dev eth2.72 type vlan id 72
> # ip addr add 192.168.71.41/24 dev eth2.71
> # ip addr add 192.168.72.41/24 dev eth2.72
> # ip link set eth2.71 up
> # ip link set eth2.72 up
> # route add -net default gw 192.168.71.1 dev eth2.71
> # route add -net default gw 192.168.72.1 dev eth2.72
> 

...

> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
> 
> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
> 

...

> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
> So the phenomenon described at the beginning happens.
> 
> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?

That is a known problem with multipath routes.

Re: IPv4 saddr do not match with selected output device in double default gateways scene

[Ziyang Xuan (William)]

> On 3/7/22 11:41 PM, Ziyang Xuan (William) wrote:
>> Create VLAN devices and add default gateways with following commands:
>>
>> # ip link add link eth2 dev eth2.71 type vlan id 71
>> # ip link add link eth2 dev eth2.72 type vlan id 72
>> # ip addr add 192.168.71.41/24 dev eth2.71
>> # ip addr add 192.168.72.41/24 dev eth2.72
>> # ip link set eth2.71 up
>> # ip link set eth2.72 up
>> # route add -net default gw 192.168.71.1 dev eth2.71
>> # route add -net default gw 192.168.72.1 dev eth2.72
>>
> 
> ...
> 
>> We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".
>>
>> I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.
>>
> 
> ...
> 
>> Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
>> and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
>> So the phenomenon described at the beginning happens.
>>
>> Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?
> 
> That is a known problem with multipath routes.
> .
> 

Does the community have a plan to address it?

Re: IPv4 saddr do not match with selected output device in double default gateways scene

[David Ahern]

On 3/9/22 8:20 PM, Ziyang Xuan (William) wrote:
> Does the community have a plan to address it?

i do not. someone with the knowledge and time is welcome to create a patch.