From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86C6CC83F2D for ; Wed, 30 Aug 2023 19:03:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239642AbjH3TAp (ORCPT ); Wed, 30 Aug 2023 15:00:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245279AbjH3PCp (ORCPT ); Wed, 30 Aug 2023 11:02:45 -0400 Received: from px.funlab.cc (px.funlab.cc [IPv6:2a01:4f8:c010:6bd5::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 998DF1A2; Wed, 30 Aug 2023 08:02:41 -0700 (PDT) Received: from [192.168.1.40] (unknown [83.27.115.100]) (Authenticated sender: doka@funlab.cc) by px.funlab.cc (Postfix) with ESMTPSA id 116E6602BA; Wed, 30 Aug 2023 18:02:39 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=funlab.cc; s=dkim; t=1693407759; bh=wLWCndZBolb584JQz+iw/o1bq04QE7Zr3wQ9bkJ9P3o=; h=Date:Cc:Subject:To:References:From:In-Reply-To:From; b=KUXFBHTAZ+YMZmxMa0nunbW7rosTHmwDlcNs0M1AtpkTiAtY0gAVqPCLHiclbLckW +zMF2uGKz5NUGHseT47Pz2xUfS3B2ppn9BAMUU4CHEQpKKrxilzp//JlkyWXbhYcNQ ZFqk0vwCjCyrHP2oQurScFy5okjtMS6IBeFXWSIchIK97azV8MFdtvfmvKsO9KXjhq Kgpet42bfs2PY9uX5UWJ9OI2ah/FB18fnvmMAO7h42ljlRxb5i4qpKnYBMAhD+IY2p aphfJUiy6R/oa4qYuQXsqxV2A34E3w3FmUWk1oEEHJQgfZqBwHb76xpoiZLlVI93nC suEilXdrEbsEQ== Message-ID: <49f87e48-7898-97d5-0140-1dab840ce0f2@funlab.cc> Date: Wed, 30 Aug 2023 17:02:28 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Cc: doka@funlab.cc, "David S. Miller" , David Ahern , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Pablo Neira Ayuso , Jozsef Kadlecsik , Florian Westphal , Linux Networking , Linux Netfilter Subject: Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets Content-Language: en-US To: Bagas Sanjaya , linux-kernel@vger.kernel.org References: <4b5b5ce0-e7a0-db7d-f23f-dde4b041f2fe@funlab.cc> From: Volodymyr Litovka In-Reply-To: <4b5b5ce0-e7a0-db7d-f23f-dde4b041f2fe@funlab.cc> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi colleagues, sorry bothering you, but can anyone shed light on this issue? This stops me and I will be glad to hear, where I'm wrong and/or where try to look into the problem. Thank you very much. On 8/27/23 10:34, Volodymyr Litovka wrote: > Hi Bagas, > > this tested on: > > - 5.19.0-42 on Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection >   -- this is host hardware > - 6.2.0-32 on Virtio network device (under KVM 6.2 on host hardware > above) > - 6.5.0-060500rc7 on Virtio network device (under KVM 6.2 on host > hardware above) > > Result is the same for all cases. > > Thank you. > > On 8/27/23 04:07, Bagas Sanjaya wrote: >> On Sat, Aug 26, 2023 at 09:55:30PM +0200, Volodymyr Litovka wrote: >>> Hi colleagues, >>> >>> I'm trying to catch and process (in 3rd party analytics app) DHCP >>> packets >>> from ERSPAN session, but cannot do this due to absence of DHCP unicast >>> packets after decapsulation. >>> >>> The model is pretty simple: there is PHY interface (enp2s0) which >>> receive >>> ERSPAN traffic and erspan-type interface to get decapsulated packets >>> (inspan, created using command "ip link add inspan type erspan seq >>> key 10 >>> local 10.171.165.65 erspan_ver 1", where 10.171.165.65 is ERSPAN >>> target). >>> Then I'm going to rewrite headers in the proper ways (nftable's netdev >>> family) and forward packets to the pool of workers. >>> >>> Having this, I'm expecting everything, which is encapsulated inside >>> ERSPAN, >>> on 'inspan' interface. And there is _almost_ everything except DHCP >>> unicast >>> packets - tcpdump shows about 1kps on this interface of decapsulated >>> packets, but no DHCP unicast (see below traces). >>> >>> To avoid any interactions, I removed and disabled everything that >>> can catch >>> DHCP in userspace - systemd-networkd, netplan, dhcp-client. There is >>> no DHCP >>> server and ifupdown - for test purposes, I'm bringing networking >>> manually. >>> Apparmor disabled as well. Kernel (Linux 5.19.0-42-generic >>> #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC) compiled without CONFIG_IP_PNP >>> (according to /boot/config-5.19.0-42-generic). Nothing in userspace >>> listens >>> on UDP/68 and UDP/67: >> Can you reproduce this on latest mainline? >> >>> # netstat -tunlpa >>> Active Internet connections (servers and established) >>> Proto Recv-Q Send-Q Local Address           Foreign Address >>> State       PID/Program name >>> tcp        0      0 0.0.0.0:22 0.0.0.0:* LISTEN      544/sshd: >>> /usr/sbin >>> tcp6       0      0 :::22 :::*                    LISTEN 544/sshd: >>> /usr/sbin >>> >>> I have no ideas, why this is happening. Decapsulation itself works, but >>> particular kind of packets get lost. >>> >>> I will appreciate if anyone can help me understand where is the bug >>> - in my >>> configuration or somewhere inside the kernel? >>> >>> Evidence of traffic presence/absence is below. >>> >>> Thank you. >>> >>> Encapsulated ERSPAN session (udp and port 67/68) contains lot of >>> different >>> kinds of DHCP packets: >>> >>> # tcpdump -s0 -w- -i enp2s0 'proto gre and ether[73:1]=17 and >>> (ether[84:2]=67 or ether[84:2]=68)' | tshark -r- -l >>>   [ ... ] >>>      7   0.001942  0.0.0.0 → 255.255.255.255 DHCP 392 DHCP Discover - >>> Transaction ID 0x25c096fc >>>      8   0.003432  z.z.z.z → a.a.a.a         DHCP 418 DHCP ACK      - >>> Transaction ID 0x5515126a >>>      9   0.005170  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover - >>> Transaction ID 0xa7b7 >>>     10   0.005171  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover - >>> Transaction ID 0xa7b7 >>>     11   0.015399  n.n.n.n → z.z.z.z         DHCP 690 DHCP Request  - >>> Transaction ID 0x54955233 >>>     12   0.025537  z.z.z.z → n.n.n.n         DHCP 420 DHCP ACK      - >>> Transaction ID 0x54955233 >>>     13   0.030313  z.z.z.z → m.m.m.m         DHCP 413 DHCP Offer    - >>> Transaction ID 0xa7b7 >>> >>> but decapsulated traffic (which I'm seeing on inspan interface) >>> contains >>> just the following: >>> >>> # tcpdump -i inspan 'port 67 or port 68' >>> listening on inspan, link-type EN10MB (Ethernet), snapshot length >>> 262144 >>> bytes >>> 17:23:36.540721 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, >>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300 >>> 17:23:39.760036 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, >>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300 >>> 17:23:44.135711 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, >>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300 >>> 17:23:52.008504 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, >>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300 >>> >> What hardware? >> -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison