Re: Kernel 5.4 regression - memory leak in network layer

[Byron Stanoszek <gandalf@winds.org>]

On Wed, 20 Nov 2019, Florian Westphal wrote:

> Florian Westphal <fw@strlen.de> wrote:
>> Byron Stanoszek <gandalf@winds.org> wrote:
>>> unreferenced object 0xffff88821a48a180 (size 64):
>>>   comm "softirq", pid 0, jiffies 4294709480 (age 192.558s)
>>>   hex dump (first 32 bytes):
>>>     01 00 00 00 01 06 ff ff 00 00 00 00 00 00 00 00  ................
>>>     00 20 72 3d 82 88 ff ff 00 00 00 00 00 00 00 00  . r=............
>>>   backtrace:
>>>     [<00000000edf73c5e>] skb_ext_add+0xc0/0xf0
>>>     [<00000000ca960770>] br_nf_pre_routing+0x171/0x489
>>>     [<0000000063a55d83>] br_handle_frame+0x171/0x300
>>
>> Brnf related, I will have a look.
>
> Not reproducible.
>
> I'm on
>
> c74386d50fbaf4a54fd3fe560f1abc709c0cff4b ("afs: Fix missing timeout reset").

I confirm I still see the issue on that commit.

> Does your setup use any other settings (ethtool, sysctl, qdiscs, tunnels
> and the like)?

Yeah, I'm using macvlan. Here are my settings:

$ ethtool -i eth0
driver: e1000e
version: 3.2.6-k
firmware-version: 0.13-4
expansion-rom-version:
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

$ ethtool -i eth1
driver: igb
version: 5.6.0-k
firmware-version: 3.25, 0x800005cf
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Commands to set up network:

ethtool -K eth0 tx off rx off
ethtool -K eth1 tx off rx off
ifconfig lo 127.0.0.1
ifconfig eth0 up
brctl addbr br0
brctl addif br0 eth0
brctl setfd br0 0
ifconfig eth1 up
brctl addbr br1
brctl addif br1 eth1
brctl setfd br1 0
ifconfig br0 172.17.2.10 netmask 255.255.0.0
ifconfig br1 192.168.0.1 netmask 255.255.255.0
ip l add link br1 mac1 address BE:77:00:00:00:70 type macvlan mode bridge
ip l set mac1 up
ip a add 192.168.0.70/24 broadcast + dev mac1

$ iptables-save -c
# Generated by iptables-save v1.8.3 on Wed Nov 20 17:26:29 2019
*raw
:PREROUTING ACCEPT [3701999:2657924997]
:OUTPUT ACCEPT [1122825:291796686]
COMMIT
# Completed on Wed Nov 20 17:26:29 2019
# Generated by iptables-save v1.8.3 on Wed Nov 20 17:26:29 2019
*nat
:PREROUTING ACCEPT [612068:41087443]
:INPUT ACCEPT [17:2254]
:OUTPUT ACCEPT [55:3780]
:POSTROUTING ACCEPT [36:2340]
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 102 -j DNAT --to-destination 192.168.0.2
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 192.168.0.2
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 5900 -j DNAT --to-destination 192.168.0.4
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 44818 -j DNAT --to-destination 192.168.0.2
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 51234 -j DNAT --to-destination 192.168.0.9
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 51235 -j DNAT --to-destination 192.168.0.9
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 51236 -j DNAT --to-destination 192.168.0.9
[0:0] -A PREROUTING -d 172.17.2.10/32 -i br0 -p tcp -m tcp --dport 44444 -j DNAT --to-destination 192.168.0.9
[2:120] -A POSTROUTING -o br0 -j MASQUERADE
[17:1320] -A POSTROUTING -o br1 -j MASQUERADE
[0:0] -A POSTROUTING -o eth2 -j MASQUERADE
COMMIT
# Completed on Wed Nov 20 17:26:29 2019
# Generated by iptables-save v1.8.3 on Wed Nov 20 17:26:29 2019
*filter
:INPUT ACCEPT [3093143:2617432037]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1122803:291795238]
COMMIT
# Completed on Wed Nov 20 17:26:29 2019

Setting up another box as IP 172.17.2.11 and 192.168.0.99 and running this
command from the original box reliably adds about 2MB of memory marked used
according to "free":

netperf -H 172.17.2.11 -t UDP_RR

or

netperf -H 192.168.0.99 -t UDP_RR

Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

212992 212992 1        1       10.00    4000.98
212992 212992

Nothing else at the moment is attached to the bridges:

$ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.2046a101b1fb       no              eth0
br1             8000.2046a101b1fc       no              eth1

As for network-related sysctls, I've got:

# Enable IP Forwarding
net.ipv4.ip_forward = 1

# Increase the number of in-flight AF_UNIX datagrams per socket
net.unix.max_dgram_qlen = 1000

Regards,
  -Byron