eBPF verifier does not load libxdp dispatcher eBPF program

eBPF verifier does not load libxdp dispatcher eBPF program

[Chris Ward]

I am trying to use the 'bleeding edge' kernel to determine whether a
problem I see has already been fixed, but with this kernel the eBPF
verifier will not load the dispatcher program that is contained within
libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
running the test case from
https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
build it go to the AF_XDP-filter directory and type 'make', and to run
it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
The lines from the run output indicating the failure are
libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Func#11 is safe for any args that match its prototype
btf_vmlinux is malformed
reg type unsupported for arg#0 function xdp_dispatcher#29
0: R1=ctx(off=0,imm=0) R10=fp0
; int xdp_dispatcher(struct xdp_md *ctx)
0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
R6_w=ctx(off=0,imm=0)
1: (b7) r0 = 2                        ; R0_w=2
; __u8 num_progs_enabled = conf.num_progs_enabled;
2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
R8=map_value(off=0,ks=4,vs=84,imm=0)
; if (num_progs_enabled < 1)
5: (15) if r7 == 0x0 goto pc+141      ; R7=1
; ret = prog0(ctx);
6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
R6=ctx(off=0,imm=0)
7: (85) call pc+140
btf_vmlinux is malformed
R1 type=ctx expected=fp
Caller passes invalid args into func#1
processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
9 peak_states 9 mark_read 1
-- END PROG LOAD LOG --
libbpf: prog 'xdp_dispatcher': failed to load: -22
libbpf: failed to load object 'xdp-dispatcher.o'
libxdp: Failed to load dispatcher: Invalid argument
libxdp: Falling back to loading single prog without dispatcher

Can this regression be fixed before kernel 6.2 ships ?

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Chris Ward]

The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
So it seems that something Canonical did for the Ubuntu 20.04 kernel
causes eBPF to work correctly.

On Mon, 23 Jan 2023 at 11:06, Chris Ward <tjcw01@gmail.com> wrote:
>
> I am trying to use the 'bleeding edge' kernel to determine whether a
> problem I see has already been fixed, but with this kernel the eBPF
> verifier will not load the dispatcher program that is contained within
> libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
> kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
> running the test case from
> https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
> build it go to the AF_XDP-filter directory and type 'make', and to run
> it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
> type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
> The lines from the run output indicating the failure are
> libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> Func#11 is safe for any args that match its prototype
> btf_vmlinux is malformed
> reg type unsupported for arg#0 function xdp_dispatcher#29
> 0: R1=ctx(off=0,imm=0) R10=fp0
> ; int xdp_dispatcher(struct xdp_md *ctx)
> 0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
> R6_w=ctx(off=0,imm=0)
> 1: (b7) r0 = 2                        ; R0_w=2
> ; __u8 num_progs_enabled = conf.num_progs_enabled;
> 2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
> 4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
> R8=map_value(off=0,ks=4,vs=84,imm=0)
> ; if (num_progs_enabled < 1)
> 5: (15) if r7 == 0x0 goto pc+141      ; R7=1
> ; ret = prog0(ctx);
> 6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
> R6=ctx(off=0,imm=0)
> 7: (85) call pc+140
> btf_vmlinux is malformed
> R1 type=ctx expected=fp
> Caller passes invalid args into func#1
> processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
> 9 peak_states 9 mark_read 1
> -- END PROG LOAD LOG --
> libbpf: prog 'xdp_dispatcher': failed to load: -22
> libbpf: failed to load object 'xdp-dispatcher.o'
> libxdp: Failed to load dispatcher: Invalid argument
> libxdp: Falling back to loading single prog without dispatcher
>
> Can this regression be fixed before kernel 6.2 ships ?

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 3002 bytes --]

On 23.01.2023 12:35:41, Chris Ward wrote:
> The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
> torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
> So it seems that something Canonical did for the Ubuntu 20.04 kernel
> causes eBPF to work correctly.
> 
> On Mon, 23 Jan 2023 at 11:06, Chris Ward <tjcw01@gmail.com> wrote:
> >
> > I am trying to use the 'bleeding edge' kernel to determine whether a
> > problem I see has already been fixed, but with this kernel the eBPF
> > verifier will not load the dispatcher program that is contained within
> > libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
> > kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
> > running the test case from
> > https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
> > build it go to the AF_XDP-filter directory and type 'make', and to run
> > it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
> > type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
> > The lines from the run output indicating the failure are
> > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > Func#11 is safe for any args that match its prototype
> > btf_vmlinux is malformed
> > reg type unsupported for arg#0 function xdp_dispatcher#29
> > 0: R1=ctx(off=0,imm=0) R10=fp0
> > ; int xdp_dispatcher(struct xdp_md *ctx)
> > 0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
> > R6_w=ctx(off=0,imm=0)
> > 1: (b7) r0 = 2                        ; R0_w=2
> > ; __u8 num_progs_enabled = conf.num_progs_enabled;
> > 2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
> > 4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
> > R8=map_value(off=0,ks=4,vs=84,imm=0)
> > ; if (num_progs_enabled < 1)
> > 5: (15) if r7 == 0x0 goto pc+141      ; R7=1
> > ; ret = prog0(ctx);
> > 6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
> > R6=ctx(off=0,imm=0)
> > 7: (85) call pc+140
> > btf_vmlinux is malformed
> > R1 type=ctx expected=fp
> > Caller passes invalid args into func#1
> > processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
> > 9 peak_states 9 mark_read 1
> > -- END PROG LOAD LOG --
> > libbpf: prog 'xdp_dispatcher': failed to load: -22
> > libbpf: failed to load object 'xdp-dispatcher.o'
> > libxdp: Failed to load dispatcher: Invalid argument
> > libxdp: Falling back to loading single prog without dispatcher
> >
> > Can this regression be fixed before kernel 6.2 ships ?

I'm seeing the same failure on 32 bit ARM on v6.13.

Have you found a solution?

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Eduard Zingerman]

 Skip to content [79] 
Navigation Menu Toggle navigation 




[80] 
Sign in
[81] 
* 

Product













GitHub Copilot
Write better code with AI



[37]





Security
Find and fix vulnerabilities



[71]





Actions
Automate any workflow



[72]





Codespaces
Instant dev environments



[73]





Issues
Plan and track work



[74]





Code Review
Manage code changes



[75]





Discussions
Collaborate outside of code



[76]





Code Search
Find more, search less



[77]




Explore


All features


[35]

Documentation




[22]

GitHub Skills




[26]

Blog




[15]



* 

Solutions







By company size


Enterprises


[36]

Small and medium teams


[40]

Startups


[69]

Nonprofits


[70]


By use case


DevSecOps


[65]

DevOps


[66]

CI/CD


[67]

View all use cases


[68]




By industry


Healthcare


[60]

Financial services


[61]

Manufacturing


[62]

Government


[63]

View all industries


[64]




View all solutions



[78] 

* 

Resources







Topics


AI


[55]

DevOps


[56]

Security


[57]

Software Development


[58]

View all


[59]




Explore


Learning Pathways




[51]

Events & Webinars




[41]

Ebooks & Whitepapers


[52]

Customer Stories


[53]

Partners




[30]

Executive Insights


[54]



* 

Open Source











GitHub Sponsors
Fund open source developers



[50]






The ReadME Project
GitHub community articles



[16]


Repositories


Topics


[47]

Trending


[48]

Collections


[49]



* 

Enterprise













Enterprise platform
AI-powered developer platform



[36]


Available add-ons






Advanced Security
Enterprise-grade security features



[44]





GitHub Copilot
Enterprise-grade AI features



[45]





Premium Support
Enterprise-grade 24/7 support



[46]



* 
Pricing [39]
undefined
Search or jump to... undefinedundefined 
Search code, repositories, users, issues, pull requests... 

Provide feedback 
undefined


We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel Submit feedback
Saved searches Use saved searches to filter your results more quickly 
undefined




Cancel Create saved search

Sign in
[81] Sign in to GitHub Username or email address Password Forgot password? [82] 










Sign up
[83] Reseting focus 




404 “This is not the web page you are looking for” 

Find code, projects, and people on GitHub: Search 
Contact Support [84] — GitHub Status [85] — @githubstatus [86] 
Site-wide Links 


[80] Subscribe to our developer newsletter Get tips, technical guides, and best practices. Twice a month. Right in your inbox.

Subscribe


[87] Product * 
Features [35]
* 
Enterprise [36]
* 
Copilot [37]
* 
Security [38]
* 
Pricing [39]
* 
Team [40]
* 
Resources [41]
* 
Roadmap [42]
* 
Compare GitHub [43]
Platform * 
Developer API [29]
* 
Partners [30]
* 
Education [31]
* 
GitHub CLI [32]
* 
GitHub Desktop [33]
* 
GitHub Mobile [34]
Support * 
Docs [22]
* 
Community Forum [23]
* 
Professional Services [24]

* 
Premium Support [25]
* 
Skills [26]
* 
Status [27]
* 
Contact GitHub [28]
Company * 
About [13]
* 
Customer stories [14]
* 
Blog [15]
* 
The ReadME Project [16]
* 
Careers [17]
* 
Newsroom [18]
* 
Inclusion [19]
* 
Social Impact [20]
* 
Shop [21]

* 
© 2025 GitHub, Inc.
* 
Terms [8]
* 
Privacy [10]
(Updated 02/2024 [9])
* 
Sitemap [11]
* 
What is Git? [12]
* 


Manage cookies


* 


Do not share my personal information


* 



GitHub on LinkedIn
[1]
* 

Instagram

GitHub on Instagram
[2]
* 



GitHub on YouTube
[3]
* 



GitHub on X
[4]
* 

TikTok

GitHub on TikTok
[5]
* 

Twitch

GitHub on Twitch
[6]
* 



GitHub’s organization on GitHub
[7]



   

[1]                    GitHub on LinkedIn        https://www.linkedin.com/company/github
[2]          Instagram          GitHub on Instagram        https://www.instagram.com/github
[3]                    GitHub on YouTube        https://www.youtube.com/github
[4]                    GitHub on X        https://x.com/github
[5]          TikTok          GitHub on TikTok        https://www.tiktok.com/@github
[6]          Twitch          GitHub on Twitch        https://www.twitch.tv/github
[7]                    GitHub’s organization on GitHub        https://github.com/github
[8] Terms https://docs.github.com/site-policy/github-terms/github-terms-of-service
[9] Updated 02/2024 https://github.com/github/site-policy/pull/582
[10] Privacy https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
[11] Sitemap https://github.com/sitemap
[12] What is Git? https://github.com/git-guides
[13] About https://github.com/about
[14] Customer stories https://github.com/customer-stories?type=enterprise
[15] Blog https://github.blog/
[16] The ReadME Project https://github.com/readme
[17] Careers https://github.careers/
[18] Newsroom https://github.com/newsroom
[19] Inclusion https://github.com/about/diversity
[20] Social Impact https://socialimpact.github.com/
[21] Shop https://shop.github.com/
[22] Docs https://docs.github.com/
[23] Community Forum https://github.community/
[24] Professional Services https://services.github.com/
[25] Premium Support https://github.com/enterprise/premium-support
[26] Skills https://skills.github.com/
[27] Status https://www.githubstatus.com/
[28] Contact GitHub https://support.github.com/?tags=dotcom-footer
[29] Developer API https://docs.github.com/get-started/exploring-integrations/about-building-integrations
[30] Partners https://partner.github.com/
[31] Education https://github.com/edu
[32] GitHub CLI https://cli.github.com/
[33] GitHub Desktop https://desktop.github.com/
[34] GitHub Mobile https://github.com/mobile
[35] Features https://github.com/features
[36] Enterprise https://github.com/enterprise
[37] Copilot https://github.com/features/copilot
[38] Security https://github.com/security
[39] Pricing https://github.com/pricing
[40] Team https://github.com/team
[41] Resources https://resources.github.com/
[42] Roadmap https://github.com/github/roadmap
[43] Compare GitHub https://resources.github.com/devops/tools/compare
[44]                              Advanced Security         Enterprise-grade security features               https://github.com/enterprise/advanced-security
[45]                              GitHub Copilot         Enterprise-grade AI features               https://github.com/features/copilot#enterprise
[46]                              Premium Support         Enterprise-grade 24/7 support               https://github.com/premium-support
[47]        Topics        https://github.com/topics
[48]        Trending        https://github.com/trending
[49]        Collections        https://github.com/collections
[50]                        GitHub Sponsors         Fund open source developers               https://github.com/sponsors
[51]        Learning Pathways              https://resources.github.com/learn/pathways
[52]        Ebooks & Whitepapers        https://github.com/resources/whitepapers
[53]        Customer Stories        https://github.com/customer-stories
[54]        Executive Insights        https://github.com/solutions/executive-insights
[55]        AI        https://github.com/resources/articles/ai
[56]        DevOps        https://github.com/resources/articles/devops
[57]        Security        https://github.com/resources/articles/security
[58]        Software Development        https://github.com/resources/articles/software-development
[59]        View all        https://github.com/resources/articles
[60]        Healthcare        https://github.com/solutions/industry/healthcare
[61]        Financial services        https://github.com/solutions/industry/financial-services
[62]        Manufacturing        https://github.com/solutions/industry/manufacturing
[63]        Government        https://github.com/solutions/industry/government
[64]        View all industries        https://github.com/solutions/industry
[65]        DevSecOps        https://github.com/solutions/use-case/devsecops
[66]        DevOps        https://github.com/solutions/use-case/devops
[67]        CI/CD        https://github.com/solutions/use-case/ci-cd
[68]        View all use cases        https://github.com/solutions/use-case
[69]        Startups        https://github.com/enterprise/startups
[70]        Nonprofits        https://github.com/solutions/industry/nonprofits
[71]                              Security         Find and fix vulnerabilities               https://github.com/features/security
[72]                              Actions         Automate any workflow               https://github.com/features/actions
[73]                              Codespaces         Instant dev environments               https://github.com/features/codespaces
[74]                              Issues         Plan and track work               https://github.com/features/issues
[75]                              Code Review         Manage code changes               https://github.com/features/code-review
[76]                              Discussions         Collaborate outside of code               https://github.com/features/discussions
[77]                              Code Search         Find more, search less               https://github.com/features/code-search
[78]                View all solutions                       https://github.com/solutions
[79] Skip to content https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth#start-of-content
[80]                        https://github.com/
[81]              Sign in            https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftjcw%2Fbpf-examples%2Ftree%2Ftjcw-explore-sameeth
[82] Forgot password? https://github.com/password_reset
[83]                  Sign up                https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2Ftjcw%2Fbpf-examples%2Ftree%2Ftjcw-explore-sameeth&source=header
[84] Contact Support https://support.github.com/?tags=dotcom-404
[85] GitHub Status https://githubstatus.com/
[86] @githubstatus https://twitter.com/githubstatus
[87]    Subscribe        https://resources.github.com/newsletter/

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Eduard Zingerman]

On Tue, 2025-02-25 at 16:55 +0100, Marc Kleine-Budde wrote:
> On 23.01.2023 12:35:41, Chris Ward wrote:
> > The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
> > torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
> > So it seems that something Canonical did for the Ubuntu 20.04 kernel
> > causes eBPF to work correctly.
> > 
> > On Mon, 23 Jan 2023 at 11:06, Chris Ward <tjcw01@gmail.com> wrote:
> > > 
> > > I am trying to use the 'bleeding edge' kernel to determine whether a
> > > problem I see has already been fixed, but with this kernel the eBPF
> > > verifier will not load the dispatcher program that is contained within
> > > libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
> > > kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
> > > running the test case from
> > > https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
> > > build it go to the AF_XDP-filter directory and type 'make', and to run
> > > it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
> > > type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
> > > The lines from the run output indicating the failure are
> > > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > > Func#11 is safe for any args that match its prototype
> > > btf_vmlinux is malformed
> > > reg type unsupported for arg#0 function xdp_dispatcher#29
> > > 0: R1=ctx(off=0,imm=0) R10=fp0
> > > ; int xdp_dispatcher(struct xdp_md *ctx)
> > > 0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
> > > R6_w=ctx(off=0,imm=0)
> > > 1: (b7) r0 = 2                        ; R0_w=2
> > > ; __u8 num_progs_enabled = conf.num_progs_enabled;
> > > 2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
> > > 4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
> > > R8=map_value(off=0,ks=4,vs=84,imm=0)
> > > ; if (num_progs_enabled < 1)
> > > 5: (15) if r7 == 0x0 goto pc+141      ; R7=1
> > > ; ret = prog0(ctx);
> > > 6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
> > > R6=ctx(off=0,imm=0)
> > > 7: (85) call pc+140
> > > btf_vmlinux is malformed
> > > R1 type=ctx expected=fp
> > > Caller passes invalid args into func#1
> > > processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
> > > 9 peak_states 9 mark_read 1
> > > -- END PROG LOAD LOG --
> > > libbpf: prog 'xdp_dispatcher': failed to load: -22
> > > libbpf: failed to load object 'xdp-dispatcher.o'
> > > libxdp: Failed to load dispatcher: Invalid argument
> > > libxdp: Falling back to loading single prog without dispatcher
> > > 
> > > Can this regression be fixed before kernel 6.2 ships ?
> 
> I'm seeing the same failure on 32 bit ARM on v6.13.
> 
> Have you found a solution?
> 
> regards,
> Marc
> 

Hello,

When I try the link from the discussion:
https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth
I get a 404 error from github.

(Sorry for previous email, it was sent unintentionally).

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 8080 bytes --]

On 25.02.2025 10:21:11, Eduard Zingerman wrote:
> On Tue, 2025-02-25 at 16:55 +0100, Marc Kleine-Budde wrote:
> > On 23.01.2023 12:35:41, Chris Ward wrote:
> > > The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
> > > torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
> > > So it seems that something Canonical did for the Ubuntu 20.04 kernel
> > > causes eBPF to work correctly.
> > >
> > > On Mon, 23 Jan 2023 at 11:06, Chris Ward <tjcw01@gmail.com> wrote:
> > > >
> > > > I am trying to use the 'bleeding edge' kernel to determine whether a
> > > > problem I see has already been fixed, but with this kernel the eBPF
> > > > verifier will not load the dispatcher program that is contained within
> > > > libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
> > > > kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
> > > > running the test case from
> > > > https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
> > > > build it go to the AF_XDP-filter directory and type 'make', and to run
> > > > it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
> > > > type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
> > > > The lines from the run output indicating the failure are
> > > > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > > > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > > > Func#11 is safe for any args that match its prototype
> > > > btf_vmlinux is malformed
> > > > reg type unsupported for arg#0 function xdp_dispatcher#29
> > > > 0: R1=ctx(off=0,imm=0) R10=fp0
> > > > ; int xdp_dispatcher(struct xdp_md *ctx)
> > > > 0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
> > > > R6_w=ctx(off=0,imm=0)
> > > > 1: (b7) r0 = 2                        ; R0_w=2
> > > > ; __u8 num_progs_enabled = conf.num_progs_enabled;
> > > > 2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
> > > > 4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
> > > > R8=map_value(off=0,ks=4,vs=84,imm=0)
> > > > ; if (num_progs_enabled < 1)
> > > > 5: (15) if r7 == 0x0 goto pc+141      ; R7=1
> > > > ; ret = prog0(ctx);
> > > > 6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
> > > > R6=ctx(off=0,imm=0)
> > > > 7: (85) call pc+140
> > > > btf_vmlinux is malformed
> > > > R1 type=ctx expected=fp
> > > > Caller passes invalid args into func#1
> > > > processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
> > > > 9 peak_states 9 mark_read 1
> > > > -- END PROG LOAD LOG --
> > > > libbpf: prog 'xdp_dispatcher': failed to load: -22
> > > > libbpf: failed to load object 'xdp-dispatcher.o'
> > > > libxdp: Failed to load dispatcher: Invalid argument
> > > > libxdp: Falling back to loading single prog without dispatcher
> > > >
> > > > Can this regression be fixed before kernel 6.2 ships ?
> >
> > I'm seeing the same failure on 32 bit ARM on v6.13.
> >
> > Have you found a solution?

> When I try the link from the discussion:
> https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth
> I get a 404 error from github.

I'm have the same error as Chris Ward wrote in their original mail. But
I'm using the xdp-tutorial's [1] basic01-xdp-pass/xdp_pass_user example.

[1] https://github.com/xdp-project/xdp-tutorial.git

This is my error message.

| sudo ./xdp_pass_user -d lan0
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
| libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
| btf_vmlinux is malformed
  ^^^^^^^^^^^^^^^^^^^^^^^^

Now I understand, what this error message wants to tell me. I should
recompile my kernel with CONFIG_DEBUG_INFO_BTF=y.

| 0: R1=ctx() R10=fp0
| ; int xdp_dispatcher(struct xdp_md *ctx) @ xdp-dispatcher.c:118
| 0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
| ; __u8 num_progs_enabled = conf.num_progs_enabled; @ xdp-dispatcher.c:120
| 1: (18) r8 = 0xc3b45cc8               ; R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
| 3: (71) r7 = *(u8 *)(r8 +2)           ; R7_w=1 R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
| 4: (b7) r0 = 2                        ; R0_w=2
| ; if (num_progs_enabled < 1) @ xdp-dispatcher.c:123
| 5: (15) if r7 == 0x0 goto pc+136      ; R7_w=1
| ; ret = prog0(ctx); @ xdp-dispatcher.c:125
| 6: (bf) r1 = r6                       ; R1_w=ctx() R6_w=ctx()
| 7: (85) call pc+135
| btf_vmlinux is malformed
| R1 type=ctx expected=fp
| Caller passes invalid args into func#1 ('prog0')
| processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
| -- END PROG LOAD LOG --
| libbpf: prog 'xdp_dispatcher': failed to load: -22
| libbpf: failed to load object 'xdp-dispatcher.o'
| libxdp: Failed to load dispatcher: Invalid argument
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:118) on device:lan0(ifindex:4)


With the CONFIG_DEBUG_INFO_BTF=y kernel the verifier seems to be more
happy. Now it fails with "-22":

| sudo ./xdp_pass_user -d lan0
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
| libbpf: prog 'xdp_pass': -- BEGIN PROG LOAD LOG --
| Extension programs should be JITed
| processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
| -- END PROG LOAD LOG --
| libbpf: prog 'xdp_pass': failed to load: -22
| libbpf: failed to load object 'xdp-dispatcher.o'
| libxdp: Compatibility check for dispatcher program failed: Invalid argument
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:20) on device:lan0(ifindex:4)


After unloading and enabling the JIT...

| ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) echo 1 |sudo tee /proc/sys/net/core/bpf_jit_enable                                   

... the dispatcher fails to load with "524". Yes, the number is
positive.

| ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0 --unload-all
| Success: Unloading XDP prog name: xdp_prog_simple
| ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0             
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libxdp: Compatibility check for dispatcher program failed: Unknown error 524
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:48) on device:lan0(ifindex:4)

strace indicates this syscalls fails:

| bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=17}}, 16) = -1 ENOTSUPP (Unknown error 524)

I'm on a armv7l, i.e. a 32 bit ARM system. Maybe I'm missing some kernel
option or BPF_RAW_TRACEPOINT_OPEN is not supported on armv7l. Will look
deeper into the kernel config options tomorrow.

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

On 25.02.2025 23:57:47, Marc Kleine-Budde wrote:

[...]

> With the CONFIG_DEBUG_INFO_BTF=y kernel the verifier seems to be more
> happy. Now it fails with "-22":
> 
> | sudo ./xdp_pass_user -d lan0
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
> | libbpf: prog 'xdp_pass': -- BEGIN PROG LOAD LOG --
> | Extension programs should be JITed
> | processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
> | -- END PROG LOAD LOG --
> | libbpf: prog 'xdp_pass': failed to load: -22
> | libbpf: failed to load object 'xdp-dispatcher.o'
> | libxdp: Compatibility check for dispatcher program failed: Invalid argument
> | libxdp: Falling back to loading single prog without dispatcher
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | Success: Loading XDP prog name:xdp_prog_simple(id:20) on device:lan0(ifindex:4)
> 
> 
> After unloading and enabling the JIT...
> 
> | ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) echo 1 |sudo tee /proc/sys/net/core/bpf_jit_enable                                   
> 
> ... the dispatcher fails to load with "524". Yes, the number is
> positive.
> 
> | ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0 --unload-all
> | Success: Unloading XDP prog name: xdp_prog_simple
> | ➜ (pts/0) frogger@riot:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0             
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | libxdp: Compatibility check for dispatcher program failed: Unknown error 524
> | libxdp: Falling back to loading single prog without dispatcher
> | libbpf: elf: skipping unrecognized data section(7) xdp_metadata
> | Success: Loading XDP prog name:xdp_prog_simple(id:48) on device:lan0(ifindex:4)
> 
> strace indicates this syscalls fails:
> 
> | bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=17}}, 16) = -1 ENOTSUPP (Unknown error 524)
> 
> I'm on a armv7l, i.e. a 32 bit ARM system. Maybe I'm missing some kernel
> option or BPF_RAW_TRACEPOINT_OPEN is not supported on armv7l. Will look
> deeper into the kernel config options tomorrow.

FTR: I figured out that the dispatcher needs eBPF trampoline support,
which doesn't exist on ARM.

| https://docs.ebpf.io/linux/concepts/trampolines/

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: eBPF verifier does not load libxdp dispatcher eBPF program

[Chris Ward]

The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
So it seems that something Canonical did for the Ubuntu 20.04 kernel
causes eBPF to work correctly.

Chris Ward