[PATCH bpf-next 00/16] Add percpu variable support for bpf arena

[PATCH bpf-next 00/16] Add percpu variable support for bpf arena

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

To use percpu data in a bpf arena today, we need to define an array
sized to the maximum possible cpu number, like the cpu_ctxs array
defined in tools/sched_ext/scx_qmap.h. This is not a convenient
pattern. It lacks performance and scalability, and wastes memory.

Inspired by [1], this series introduces percpu variables for the
arena. The basic idea is to use a separate ELF section to collect the
percpu variables declared in the program. At load time, a separate
chunk is created for each cpu, and the initial data from the ELF
section is copied to each cpu's chunk. Every chunk is page aligned,
and is populated on the target cpu's NUMA node at creation time.

To declare a percpu variable in the arena, a bpf prog declares the
variable with the __arena_percpu macro:

__u64 __arena_percpu hits;

The macro tells the compiler to collect the variable into the
dedicated ELF section. The variable itself is read or written by the
bpf prog as a normal variable. At run time, its address is mapped by
the JIT to the address of the replica on the running cpu.

User space sees the same data through its arena mapping and can
address any cpu's copy directly:

total += *bpf_map__arena_percpu_ptr(arena, &skel->arena_percpu->hits, cpu);

Cross cpu access is only allowed for user space. A bpf prog can only
read or write the replica of the running cpu. The other replicas are
not visible to the bpf prog.

# arena layout

When the global arena data sits at the end of the arena, the percpu chunks
are placed right before it:

+------------+---------+---------+----+---------+--------+
|            | percpu  | percpu  |... | percpu  | global |
|            | chunk 0 | chunk 1 |    | chunk N | data   |
+------------+---------+---------+----+---------+--------+

Otherwise, the global data stays at the start of the arena, and the
percpu chunks are placed right after it:

+---------+---------+---------+----+---------+------------+
| global  | percpu  | percpu  |... | percpu  |            |
| data    | chunk 0 | chunk 1 |    | chunk N |            |
+---------+---------+---------+----+---------+------------+

Any kernel that supports arena percpu variables is new enough to
accept these offsets, so in practice the first layout is the one in
use.

A chunk is created for every possible cpu, so N + 1 is the number of
possible cpus on the system.

# pseudo instruction

A pseudo instruction is introduced for arena percpu variables. The
normal ld_imm64 instruction that references an arena percpu variable
is rewritten to the pseudo instruction by libbpf at relocation time.
The pseudo instruction carries the symbol offset in the arena, and
the verifier checks the offset to make sure it is inside cpu 0's
chunk. The pseudo instruction is JITed to compute the actual address
in the running cpu's replica, using:

dst_reg = address in chunk 0 + chunk_size * smp_processor_id()

[1] https://lore.kernel.org/bpf/20260813152324.97937-1-leon.hwang@linux.dev/

Xu Kuohai (16):
  libbpf: Collect arena percpu data section
  libbpf: Create arena percpu chunk from the arena percpu ELF section
  libbpf: Place the percpu chunks in the bpf arena
  libbpf: Copy the percpu chunks into the bpf arena
  libbpf: Add the arena percpu chunk attribute to map creation options
  libbpf: Relocate arena percpu access instructions
  libbpf: Add arena percpu feature probe
  bpf: Extend syscall to accept arena percpu information
  bpf: Teach verifier about arena percpu instructions
  bpf, x86: Add support for arena percpu
  bpf, arm64: Add support for arena percpu
  libbpf: Introduce arena percpu helpers for user space
  bpftool: Emit the arena percpu datasec into skeleton
  selftests/bpf: Add test for arena percpu-only object
  selftests/bpf: Add test for mixed arena percpu object
  selftests/bpf: Add test for arena percpu bounds

 arch/arm64/net/bpf_jit_comp.c                 |  51 ++-
 arch/x86/net/bpf_jit_comp.c                   |  33 ++
 include/linux/bpf.h                           |   5 +-
 include/linux/filter.h                        |   1 +
 include/uapi/linux/bpf.h                      |  17 +
 kernel/bpf/arena.c                            |  43 ++-
 kernel/bpf/core.c                             |  22 +-
 kernel/bpf/syscall.c                          |   2 +-
 kernel/bpf/verifier.c                         |  69 +++-
 tools/bpf/bpftool/gen.c                       |  69 +++-
 tools/include/uapi/linux/bpf.h                |  17 +
 tools/lib/bpf/bpf.c                           |   4 +-
 tools/lib/bpf/bpf.h                           |   4 +
 tools/lib/bpf/features.c                      |  50 +++
 tools/lib/bpf/libbpf.c                        | 248 +++++++++++-
 tools/lib/bpf/libbpf.h                        |  33 ++
 tools/lib/bpf/libbpf.map                      |   2 +
 tools/lib/bpf/libbpf_internal.h               |   2 +
 .../bpf/libarena/include/bpf_arena_common.h   |   2 +
 .../selftests/bpf/prog_tests/arena_percpu.c   | 361 ++++++++++++++++++
 .../selftests/bpf/progs/arena_percpu_mixed.c  |  29 ++
 .../selftests/bpf/progs/arena_percpu_only.c   |  21 +
 22 files changed, 1054 insertions(+), 31 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_percpu.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_percpu_mixed.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_percpu_only.c

-- 
2.47.3

[PATCH bpf-next 01/16] libbpf: Collect arena percpu data section

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Introduce a new section ".arena.percpu" for the bpf arena. A variable
in this section lives in the bpf arena and has a separate value for
each possible CPU. The initial value is replicated for each CPU at
load time.

Collect the section's data and index so the later steps can size,
place and relocate it.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/libbpf.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 8fcb8cf39cc0..d79fc382319e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -573,6 +573,13 @@ struct bpf_struct_ops {
  */
 #define ARENA_DATA_SEC ".arena.data"
 
+/*
+ * A variable in this section is arena percpu data. It lives in the bpf
+ * arena and has a separate value for each possible CPU. The initial value
+ * is replicated for each CPU at load time.
+ */
+#define ARENA_PERCPU_SEC ".arena.percpu"
+
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
 	LIBBPF_MAP_DATA,
@@ -718,6 +725,7 @@ struct elf_state {
 	Elf64_Ehdr *ehdr;
 	Elf_Data *symbols;
 	Elf_Data *arena_data;
+	Elf_Data *arena_percpu_data;
 	size_t shstrndx; /* section index for section name strings */
 	size_t strtabidx;
 	struct elf_sec_desc *secs;
@@ -728,6 +736,7 @@ struct elf_state {
 	int symbols_shndx;
 	bool has_st_ops;
 	int arena_data_shndx;
+	int arena_percpu_shndx;
 	int jumptables_data_shndx;
 };
 
@@ -1645,6 +1654,7 @@ static struct bpf_object *bpf_object__new(const char *path,
 	obj->efile.obj_buf_sz = obj_buf_sz;
 	obj->efile.btf_maps_shndx = -1;
 	obj->efile.arena_data_shndx = -1;
+	obj->efile.arena_percpu_shndx = -1;
 	obj->kconfig_map_idx = -1;
 	obj->arena_map_idx = -1;
 
@@ -1664,6 +1674,7 @@ static void bpf_object__elf_finish(struct bpf_object *obj)
 	obj->efile.ehdr = NULL;
 	obj->efile.symbols = NULL;
 	obj->efile.arena_data = NULL;
+	obj->efile.arena_percpu_data = NULL;
 
 	zfree(&obj->efile.secs);
 	obj->efile.sec_cnt = 0;
@@ -3201,7 +3212,7 @@ static int bpf_object__init_arena_data(struct bpf_object *obj)
 
 	if (obj->arena_map_idx < 0) {
 		/* bpf_object__init_user_btf_maps() didn't look: there is no .maps section */
-		if (obj->efile.arena_data) {
+		if (obj->efile.arena_data || obj->efile.arena_percpu_data) {
 			pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
 				ARENA_SEC);
 			return -ENOENT;
@@ -3331,6 +3342,12 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
 		return -ENOENT;
 	}
 
+	if (obj->efile.arena_percpu_data && obj->arena_map_idx < 0) {
+		pr_warn("elf: sec '%s': to use arena percpu variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
+			ARENA_PERCPU_SEC);
+		return -ENOENT;
+	}
+
 	return 0;
 }
 
@@ -4286,6 +4303,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				obj->efile.arena_data_shndx = idx;
 			} else if (strcmp(name, ARENA_DATA_SEC) == 0) {
 				/* the marker, see the first pass */
+			} else if (strcmp(name, ARENA_PERCPU_SEC) == 0) {
+				obj->efile.arena_percpu_data = data;
+				obj->efile.arena_percpu_shndx = idx;
 			} else if (strcmp(name, JUMPTABLES_SEC) == 0) {
 				obj->jumptables_data = malloc(data->d_size);
 				if (!obj->jumptables_data)
-- 
2.47.3

Re: [PATCH bpf-next 01/16] libbpf: Collect arena percpu data section

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]

> libbpf: Collect arena percpu data section

TL;DR: Objects with .arena.percpu but no .maps section may get a misleading
libbpf error naming .addr_space.1 and "global __arena variables"; it's a
message-wording question, not a functional problem.

> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 8fcb8cf39cc0..d79fc382319e 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c

[ ... ]

> @@ -3201,7 +3212,7 @@ static int bpf_object__init_arena_data(struct bpf_object *obj)
>
>  	if (obj->arena_map_idx < 0) {
>  		/* bpf_object__init_user_btf_maps() didn't look: there is no .maps section */
> -		if (obj->efile.arena_data) {
> +		if (obj->efile.arena_data || obj->efile.arena_percpu_data) {
>  			pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
>  				ARENA_SEC);
>  			return -ENOENT;

The condition now also fires on obj->efile.arena_percpu_data, but the warning
still names ARENA_SEC and talks about "global __arena variables".  Is that the
intended message for the percpu case?

Consider an object with no SEC(".maps"), the ".arena.data" marker (the
data_in_arena case), some .data/.bss, a ".arena.percpu" section and no
".addr_space.1" section.  bpf_object__init_user_btf_maps() returns 0 early
because btf_maps_shndx < 0, so its new ARENA_PERCPU_SEC message never runs.

bpf_object__init_arena_data() then reaches this block with arena_data == NULL
and arena_percpu_data != NULL, and prints:

  elf: sec '.addr_space.1': to use global __arena variables the ARENA map
  should be explicitly declared in SEC(".maps")

That points at a section the object does not have and at a kind of variable
the user did not declare.  The real problem is the .arena.percpu variables.

The same patch adds a separate message for this case in
bpf_object__init_user_btf_maps():

  pr_warn("elf: sec '%s': to use arena percpu variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
          ARENA_PERCPU_SEC);

So the two paths report the same condition differently.  Should the no-.maps
path in bpf_object__init_arena_data() choose its section name and wording
based on which of arena_data / arena_percpu_data is set?

The message is unchanged at the end of the series (08224ba2de19
"selftests/bpf: Add test for arena percpu bounds"), so no later commit
addresses this.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 02/16] libbpf: Create arena percpu chunk from the arena percpu ELF section

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Before replicating the initial values for each possible cpu, create
one chunk holding a single cpu's variables. The chunk size is rounded
up to a page multiple, so each cpu's copy gets its own pages and can
be placed on the cpu's NUMA node. The data comes from the ELF section,
zero-padded to the chunk size.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/libbpf.c | 75 ++++++++++++++++++++++++++++++++++++++++--
 tools/lib/bpf/libbpf.h |  7 ++++
 2 files changed, 79 insertions(+), 3 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index d79fc382319e..02ee57e3f6c7 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -829,6 +829,13 @@ struct bpf_object {
 	} *data_ptrs;
 	size_t data_ptr_cnt;
 
+	struct {
+		void *chunk; /* copy from section .arena.percpu, zero-padded to chunk size */
+		size_t chunk_sz;	/* chunk size, page-aligned */
+		size_t nr_chunks;	/* how many chunks, one chunk per possible cpu */
+		bool present;
+	} arena_percpu;
+
 	void *jumptables_data;
 	size_t jumptables_data_sz;
 
@@ -3267,6 +3274,41 @@ static int bpf_object__init_arena_data(struct bpf_object *obj)
 	return 0;
 }
 
+static int init_arena_percpu_data(struct bpf_object *obj, void *percpu_data, size_t percpu_sz)
+{
+	static const char *fcpu = "/sys/devices/system/cpu/possible";
+	bool *mask = NULL;
+	int possible_cpus;
+	int err;
+
+	if (percpu_sz == 0) {
+		pr_warn("arena percpu: empty %s section\n", ARENA_PERCPU_SEC);
+		return -EINVAL;
+	}
+
+	err = parse_cpu_mask_file(fcpu, &mask, &possible_cpus);
+	if (err) {
+		pr_warn("arena percpu: failed to get possible cpus: %s\n",
+			errstr(err));
+		return err;
+	}
+
+	obj->arena_percpu.chunk_sz = roundup(percpu_sz, sysconf(_SC_PAGE_SIZE));
+	obj->arena_percpu.chunk = calloc(1, obj->arena_percpu.chunk_sz); /* zero padding */
+	if (!obj->arena_percpu.chunk) {
+		free(mask);
+		return -ENOMEM;
+	}
+
+	memcpy(obj->arena_percpu.chunk, percpu_data, percpu_sz);
+
+	obj->arena_percpu.nr_chunks = possible_cpus;
+	obj->arena_percpu.present = true;
+	free(mask);
+
+	return 0;
+}
+
 static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
 					  const char *pin_root_path)
 {
@@ -3335,6 +3377,13 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
 			if (err)
 				return err;
 		}
+
+		if (obj->efile.arena_percpu_data) {
+			err = init_arena_percpu_data(obj, obj->efile.arena_percpu_data->d_buf,
+						     obj->efile.arena_percpu_data->d_size);
+			if (err)
+				return err;
+		}
 	}
 	if (obj->efile.arena_data && obj->arena_map_idx < 0) {
 		pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
@@ -8141,12 +8190,24 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 		struct bpf_map *arena_map = &obj->maps[obj->arena_map_idx];
 		size_t data_sz = roundup(obj->arena_data_sz, sysconf(_SC_PAGE_SIZE));
 		size_t mmap_sz = bpf_map_mmap_sz(arena_map);
+		size_t percpu_sz = obj->arena_percpu.nr_chunks * obj->arena_percpu.chunk_sz;
+
+		/*
+		 * the light skeleton loader doesn't mmap the arena in user space, so the percpu
+		 * chunks could not be populated
+		 */
+		if (obj->gen_loader && obj->arena_percpu.present) {
+			pr_warn("map '%s': light skeletons don't support arena percpu variables\n",
+				arena_map->name);
+			return -EOPNOTSUPP;
+		}
 
-		if (data_sz > mmap_sz) {
-			pr_warn("map '%s': declared ARENA map size (%zu) is too small to hold global %s of size %zu\n",
+		if (data_sz + percpu_sz > mmap_sz) {
+			pr_warn("map '%s': declared ARENA map size (%zu) is too small to hold "
+				"global %s of size %zu plus the percpu data size %zu\n",
 				arena_map->name, mmap_sz,
 				obj->data_in_arena ? "data" : "__arena variables",
-				obj->arena_data_sz);
+				obj->arena_data_sz, percpu_sz);
 			return -E2BIG;
 		}
 
@@ -10649,6 +10710,7 @@ void bpf_object__close(struct bpf_object *obj)
 		close(obj->token_fd);
 
 	zfree(&obj->arena_data);
+	zfree(&obj->arena_percpu.chunk);
 
 	zfree(&obj->jumptables_data);
 	obj->jumptables_data_sz = 0;
@@ -12103,6 +12165,13 @@ int bpf_map__set_initial_value(struct bpf_map *map,
 	return 0;
 }
 
+/*
+ * NOTE: For an ARENA map, the initial value returned covers the global
+ * data only. Percpu variables live in a different region from the
+ * global variables. We cannot return a flat initial value for both
+ * global and percpu arena data. Read and set the arena percpu data
+ * through the bpf skeleton.
+ */
 void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
 {
 	if (bpf_map__is_struct_ops(map)) {
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index 03f794f4adb0..3e0b7bc91bef 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -1221,6 +1221,13 @@ LIBBPF_API int bpf_map__set_map_extra(struct bpf_map *map, __u64 map_extra);
 
 LIBBPF_API int bpf_map__set_initial_value(struct bpf_map *map,
 					  const void *data, size_t size);
+/*
+ * NOTE: For an ARENA map, the initial value returned covers the global
+ * data only. Percpu variables live in a different region from the
+ * global variables. We cannot return a flat initial value for both
+ * global and percpu arena data. Read and set the arena percpu data
+ * through the bpf skeleton.
+ */
 LIBBPF_API void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize);
 
 /**
-- 
2.47.3

Re: [PATCH bpf-next 02/16] libbpf: Create arena percpu chunk from the arena percpu ELF section

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3080 bytes --]

> libbpf: Create arena percpu chunk from the arena percpu ELF section

TL;DR: Reading the possible-CPUs sysfs file at open time may make
bpf_object_open() fail, breaking 'bpftool gen skeleton' in sandboxes without
/sys; suggests computing nr_chunks lazily in relocate.

> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c

[ ... ]

> @@ -3267,6 +3274,41 @@ static int bpf_object__init_arena_data(struct bpf_object *obj)
>  	return 0;
>  }
>
> +static int init_arena_percpu_data(struct bpf_object *obj, void *percpu_data, size_t percpu_sz)
> +{
> +	static const char *fcpu = "/sys/devices/system/cpu/possible";
> +	bool *mask = NULL;
> +	int possible_cpus;
> +	int err;
> +
> +	if (percpu_sz == 0) {
> +		pr_warn("arena percpu: empty %s section\n", ARENA_PERCPU_SEC);
> +		return -EINVAL;
> +	}
> +
> +	err = parse_cpu_mask_file(fcpu, &mask, &possible_cpus);
> +	if (err) {
> +		pr_warn("arena percpu: failed to get possible cpus: %s\n",
> +			errstr(err));
> +		return err;
> +	}

Can this make bpf_object_open() fail on hosts where
/sys/devices/system/cpu/possible cannot be read?

init_arena_percpu_data() is reached from bpf_object__init_user_btf_maps()
during bpf_object_open(), so opening any object with a .arena.percpu section
now reads this sysfs file.  Until now libbpf only needed the possible CPU
count at load or runtime, through map_set_def_max_entries(), the per-cpu
value size checks and perf_buffer setup.

The value computed here, obj->arena_percpu.nr_chunks, is first used in
bpf_object__relocate(), which runs from bpf_object_prepare():

> +		size_t percpu_sz = obj->arena_percpu.nr_chunks * obj->arena_percpu.chunk_sz;

Open-only consumers such as 'bpftool gen skeleton' and 'bpftool gen
subskeleton' call bpf_object__open_mem() and never load, so they now depend
on the build host's sysfs.  In a build chroot or sandbox without /sys, open
fails with "arena percpu: failed to get possible cpus" and skeleton
generation breaks, even though the CPU count is never used there.

With 'gen skeleton -L', open fails with this error before the intended
-EOPNOTSUPP check in bpf_object__relocate() is reached:

> +		if (obj->gen_loader && obj->arena_percpu.present) {
> +			pr_warn("map '%s': light skeletons don't support arena percpu variables\n",
> +				arena_map->name);
> +			return -EOPNOTSUPP;
> +		}

Would it work to allocate and copy the chunk at open time, and compute
nr_chunks lazily in bpf_object__relocate(), where it is first needed?  No
later commit in the series changes this.

> +
> +	obj->arena_percpu.chunk_sz = roundup(percpu_sz, sysconf(_SC_PAGE_SIZE));
> +	obj->arena_percpu.chunk = calloc(1, obj->arena_percpu.chunk_sz); /* zero padding */
> +	if (!obj->arena_percpu.chunk) {
> +		free(mask);
> +		return -ENOMEM;
> +	}

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 03/16] libbpf: Place the percpu chunks in the bpf arena

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Place the percpu chunks in the bpf arena, alongside the global arena
data.

When full-range ldimm64 offset is supported, the global data sits at
the end of the arena, place the percpu chunks right before it:

+------------+---------+---------+----+---------+--------+
|            | percpu  | percpu  |... | percpu  | global |
|            | chunk 0 | chunk 1 |    | chunk N | data   |
+------------+---------+---------+----+---------+--------+

Otherwise, the global data region sits at the beginning, place the
percpu chunks right after it:

+---------+---------+---------+----+---------+------------+
| global  | percpu  | percpu  |... | percpu  |            |
| data    | chunk 0 | chunk 1 |    | chunk N |            |
+---------+---------+---------+----+---------+------------+

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/libbpf.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 02ee57e3f6c7..6fc8c7f549d9 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -833,6 +833,7 @@ struct bpf_object {
 		void *chunk; /* copy from section .arena.percpu, zero-padded to chunk size */
 		size_t chunk_sz;	/* chunk size, page-aligned */
 		size_t nr_chunks;	/* how many chunks, one chunk per possible cpu */
+		size_t offset;		/* offset within the arena */
 		bool present;
 	} arena_percpu;
 
@@ -8212,8 +8213,12 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 		}
 
 		/* place globals at the end of the arena (if supported) */
-		if (kernel_supports(obj, FEAT_LDIMM64_FULL_RANGE_OFF))
-			obj->arena_data_off = mmap_sz - data_sz;
+		if (kernel_supports(obj, FEAT_LDIMM64_FULL_RANGE_OFF)) {
+			obj->arena_data_off = data_sz ? mmap_sz - data_sz : 0;
+			obj->arena_percpu.offset = mmap_sz - data_sz - percpu_sz;
+		} else {
+			obj->arena_percpu.offset = data_sz;
+		}
 	}
 
 	err = bpf_object__relocate_data_ptrs(obj);
-- 
2.47.3

[PATCH bpf-next 04/16] libbpf: Copy the percpu chunks into the bpf arena

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

After the arena is mmapped, copy the chunk into every cpu's replica
in the arena. Before each copy, pin the copying thread to the target
cpu, so each chunk's pages fault in from that cpu's NUMA node.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/libbpf.c | 46 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 6fc8c7f549d9..3d18ba483f6e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -25,6 +25,7 @@
 #include <fcntl.h>
 #include <errno.h>
 #include <ctype.h>
+#include <sched.h>
 #include <asm/unistd.h>
 #include <linux/err.h>
 #include <linux/kernel.h>
@@ -5649,6 +5650,49 @@ static struct bpf_map *bpf_object__sec_map(struct bpf_object *obj, int sec_idx)
 	return NULL;
 }
 
+/*
+ * Copy the percpu chunks into the mmapped bpf arena, one chunk at a
+ * time. Before copying each chunk, pin the thread to the chunk's cpu,
+ * so the copy faults the chunk's pages in from that cpu's NUMA node.
+ * If pinning fails, fall back to copying on the current cpu.
+ */
+static void arena_percpu_populate(struct bpf_object *obj, void *arena_mmaped)
+{
+	size_t chunk_sz = obj->arena_percpu.chunk_sz;
+	cpu_set_t *saved_set = NULL, *one = NULL;
+	size_t set_sz;
+	void *base;
+	int cpu;
+
+	base = arena_mmaped + obj->arena_percpu.offset;
+
+	set_sz = CPU_ALLOC_SIZE(obj->arena_percpu.nr_chunks);
+	saved_set = CPU_ALLOC(obj->arena_percpu.nr_chunks);
+	one = CPU_ALLOC(obj->arena_percpu.nr_chunks);
+	if (!saved_set || !one || sched_getaffinity(0, set_sz, saved_set) != 0) {
+		/* no affinity info, fall back to copying on the current cpu */
+		for (cpu = 0; cpu < obj->arena_percpu.nr_chunks; cpu++)
+			memcpy(base + cpu * chunk_sz, obj->arena_percpu.chunk, chunk_sz);
+		goto out;
+	}
+
+	for (cpu = 0; cpu < obj->arena_percpu.nr_chunks; cpu++) {
+		CPU_ZERO_S(set_sz, one);
+		CPU_SET_S(cpu, set_sz, one);
+		sched_setaffinity(0, set_sz, one);
+
+		/* faults the chunk pages in from the target cpu's NUMA node */
+		memcpy(base + cpu * chunk_sz, obj->arena_percpu.chunk, chunk_sz);
+	}
+
+	sched_setaffinity(0, set_sz, saved_set);
+
+out:
+	CPU_FREE(saved_set);
+	CPU_FREE(one);
+	zfree(&obj->arena_percpu.chunk);
+}
+
 static int bpf_object__mmap_arena(struct bpf_object *obj, struct bpf_map *map)
 {
 	int i, err;
@@ -5669,6 +5713,8 @@ static int bpf_object__mmap_arena(struct bpf_object *obj, struct bpf_map *map)
 			obj->arena_data_sz);
 		zfree(&obj->arena_data);
 	}
+	if (obj->arena_percpu.present)
+		arena_percpu_populate(obj, map->mmaped);
 	for (i = 0; i < obj->nr_maps; i++) {
 		struct bpf_map *m = &obj->maps[i];
 
-- 
2.47.3

Re: [PATCH bpf-next 04/16] libbpf: Copy the percpu chunks into the bpf arena

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4971 bytes --]

> libbpf: Copy the percpu chunks into the bpf arena

TL;DR:
- arena_percpu_populate() may pin the loader thread to cpus the app excluded
  (isolcpus)
- the final affinity restore may permanently shrink the thread's cpu mask, and
  its failure is unchecked

> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c

[ ... ]

> @@ -5649,6 +5650,49 @@ static struct bpf_map *bpf_object__sec_map(struct bpf_object *obj, int sec_idx)
> +/*
> + * Copy the percpu chunks into the mmapped bpf arena, one chunk at a
> + * time. Before copying each chunk, pin the thread to the chunk's cpu,
> + * so the copy faults the chunk's pages in from that cpu's NUMA node.
> + * If pinning fails, fall back to copying on the current cpu.
> + */
> +static void arena_percpu_populate(struct bpf_object *obj, void *arena_mmaped)
> +{
> +	size_t chunk_sz = obj->arena_percpu.chunk_sz;
> +	cpu_set_t *saved_set = NULL, *one = NULL;
> +	size_t set_sz;
> +	void *base;
> +	int cpu;
> +
> +	base = arena_mmaped + obj->arena_percpu.offset;
> +
> +	set_sz = CPU_ALLOC_SIZE(obj->arena_percpu.nr_chunks);
> +	saved_set = CPU_ALLOC(obj->arena_percpu.nr_chunks);
> +	one = CPU_ALLOC(obj->arena_percpu.nr_chunks);
> +	if (!saved_set || !one || sched_getaffinity(0, set_sz, saved_set) != 0) {
> +		/* no affinity info, fall back to copying on the current cpu */
> +		for (cpu = 0; cpu < obj->arena_percpu.nr_chunks; cpu++)
> +			memcpy(base + cpu * chunk_sz, obj->arena_percpu.chunk, chunk_sz);
> +		goto out;
> +	}
> +
> +	for (cpu = 0; cpu < obj->arena_percpu.nr_chunks; cpu++) {
> +		CPU_ZERO_S(set_sz, one);
> +		CPU_SET_S(cpu, set_sz, one);
> +		sched_setaffinity(0, set_sz, one);

Can this loop move the calling thread onto cpus the application deliberately
excluded?  saved_set is fetched but never consulted, so the loop pins to
every cpu in [0, nr_chunks) regardless of the caller's mask.

The kernel's __sched_setaffinity() only intersects the new mask with
cpuset_cpus_allowed(), not with the thread's earlier mask.  An application
that limited its threads with taskset or sched_setaffinity() (for example to
housekeeping cpus 0-3 on a box booted with isolcpus= or nohz_full=4-63) would
have libbpf run on every isolated cpu during bpf_object__load().  Each pin
also forces a migration through stop_one_cpu() that preempts whatever is
running on that cpu.

Would it work to pin only to cpus that are set in saved_set, and copy the
remaining chunks without pinning?

> +
> +		/* faults the chunk pages in from the target cpu's NUMA node */
> +		memcpy(base + cpu * chunk_sz, obj->arena_percpu.chunk, chunk_sz);
> +	}
> +
> +	sched_setaffinity(0, set_sz, saved_set);

Does this final sched_setaffinity() actually give back the original state?
The kernel's sched_getaffinity() returns:

    cpumask_and(mask, &p->cpus_mask, cpu_active_mask);

so saved_set holds only the cpus that are active at the time of the call, not
the thread's full cpus_mask.  A thread with default affinity has a cpus_mask
covering all possible cpus, so after this call its cpus_mask shrinks to the
cpus that were online when the object was loaded.

Because the syscall sets SCA_USER, that snapshot is also stored as
p->user_cpus_ptr, and __set_cpus_allowed_ptr() clamps later cpuset updates
to it:

    if (p->user_cpus_ptr && !(ctx->flags & (SCA_USER | ...)) &&
        cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
            ctx->new_mask = rq->scratch_mask;

For example, take a vm with possible=0-63 and online=0-15, loading an object
that has .arena.percpu variables.  From then on the loader thread, and every
thread or process it creates afterwards (cpus_mask and user_cpus_ptr are
inherited), could never run on cpus 16-63, even after they are hot-added or
the cpuset is widened.  Before this patch, bpf_object__load() never touched
scheduling state.

The return value of this sched_setaffinity() is also ignored.  If it fails
(-ENOMEM from alloc_user_cpus_ptr(), or -EINVAL when a concurrent cpuset or
hotplug change leaves saved_set with no allowed or active cpus), the thread
stays pinned to the last cpu it was moved to, and nothing reports it.

The commit message and the function comment only cover the case where pinning
fails.  They do not mention changing or restoring the caller's affinity, and
no later patch in this series changes arena_percpu_populate().

Would doing the per-cpu copies in a short-lived helper thread avoid touching
the caller's thread altogether?  Otherwise, could the return value of the
restore at least be checked and reported with pr_warn()?

> +
> +out:
> +	CPU_FREE(saved_set);
> +	CPU_FREE(one);
> +	zfree(&obj->arena_percpu.chunk);
> +}

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 05/16] libbpf: Add the arena percpu chunk attribute to map creation options

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add the percpu chunk offset and size information to the map creation
attribute, and set the values when creating an arena map.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/include/uapi/linux/bpf.h | 4 ++++
 tools/lib/bpf/bpf.c            | 4 +++-
 tools/lib/bpf/bpf.h            | 4 ++++
 tools/lib/bpf/libbpf.c         | 6 ++++++
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6f70db7515dd..246f52361372 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1615,6 +1615,10 @@ union bpf_attr {
 		__aligned_u64 excl_prog_hash;
 		/* Size of the passed excl_prog_hash. */
 		__u32 excl_prog_hash_size;
+
+		/* offset and size of an arena percpu chunk */
+		__u32	arena_percpu_chunk_off;
+		__u32	arena_percpu_chunk_sz;
 	};
 
 	struct { /* anonymous struct used by BPF_MAP_*_ELEM and BPF_MAP_FREEZE commands */
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index b49822d212ae..3d5fd7e23898 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -210,7 +210,7 @@ int bpf_map_create(enum bpf_map_type map_type,
 		   __u32 max_entries,
 		   const struct bpf_map_create_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size);
+	const size_t attr_sz = offsetofend(union bpf_attr, arena_percpu_chunk_sz);
 	const size_t attr_common_sz = sizeof(struct bpf_common_attr);
 	struct bpf_common_attr attr_common;
 	struct bpf_log_opts *log_opts;
@@ -246,6 +246,8 @@ int bpf_map_create(enum bpf_map_type map_type,
 	attr.map_token_fd = OPTS_GET(opts, token_fd, 0);
 	attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL));
 	attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0);
+	attr.arena_percpu_chunk_off = OPTS_GET(opts, arena_percpu_chunk_off, 0);
+	attr.arena_percpu_chunk_sz = OPTS_GET(opts, arena_percpu_chunk_sz, 0);
 
 	log_opts = OPTS_GET(opts, log_opts, NULL);
 	if (!OPTS_VALID(log_opts, bpf_log_opts))
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 826d9cc9ab65..15a544488eb5 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -70,6 +70,10 @@ struct bpf_map_create_opts {
 	const void *excl_prog_hash;
 	__u32 excl_prog_hash_size;
 
+	/* offset and size of an arena percpu chunk */
+	__u32 arena_percpu_chunk_off;
+	__u32 arena_percpu_chunk_sz;
+
 	struct bpf_log_opts *log_opts;
 
 	size_t :0;
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 3d18ba483f6e..0b0c9d72b353 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -5816,6 +5816,12 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b
 	create_attr.map_flags = def->map_flags;
 	create_attr.numa_node = map->numa_node;
 	create_attr.map_extra = map->map_extra;
+
+	if (obj->arena_percpu.present && map == &obj->maps[obj->arena_map_idx]) {
+		create_attr.arena_percpu_chunk_off = obj->arena_percpu.offset;
+		create_attr.arena_percpu_chunk_sz = obj->arena_percpu.chunk_sz;
+	}
+
 	create_attr.token_fd = obj->token_fd;
 	if (obj->token_fd)
 		create_attr.map_flags |= BPF_F_TOKEN_FD;
-- 
2.47.3

Re: [PATCH bpf-next 05/16] libbpf: Add the arena percpu chunk attribute to map creation options

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3701 bytes --]

> libbpf: Add the arena percpu chunk attribute to map creation options

TL;DR: New fields inserted before log_opts in struct bpf_map_create_opts
appear to break libbpf ABI: binaries built against 1.8 may have log_opts
ignored or map creation fail with -EINVAL. They should likely be appended
after log_opts.

> diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
> --- a/tools/lib/bpf/bpf.h
> +++ b/tools/lib/bpf/bpf.h
> @@ -70,6 +70,10 @@ struct bpf_map_create_opts {
>  	const void *excl_prog_hash;
>  	__u32 excl_prog_hash_size;
>
> +	/* offset and size of an arena percpu chunk */
> +	__u32 arena_percpu_chunk_off;
> +	__u32 arena_percpu_chunk_sz;
> +
>  	struct bpf_log_opts *log_opts;
                              ^^^^

Does inserting these two fields ahead of log_opts break the layout of
struct bpf_map_create_opts for existing users?

log_opts was added by 702259006f93 ("libbpf: Add syscall common attributes
support for map_create") and is already in a released libbpf (3ead4ead078f
("libbpf: Start v1.9 development cycle") says libbpf 1.8.0 has been
released, and 702259006f93 is an ancestor of it).

The OPTS_HAS(opts, field) scheme relies on the caller-supplied sz and
offsetofend(type, field), which only works if new fields are appended at the
end.  On x86_64 the layout changes like this:

  old (1.8): excl_prog_hash_size @64, log_opts @72, sizeof == 80
  new:       excl_prog_hash_size @64, arena_percpu_chunk_off @68,
             arena_percpu_chunk_sz @72, log_opts @80, sizeof == 88

Consider an application built against the 1.8 headers that sets
opts.log_opts (opts.sz == 80), running against a shared libbpf that includes
this patch.  In bpf_map_create():

  attr.arena_percpu_chunk_off = OPTS_GET(opts, arena_percpu_chunk_off, 0);
      /* 80 >= 72: reads old padding */
  attr.arena_percpu_chunk_sz = OPTS_GET(opts, arena_percpu_chunk_sz, 0);
      /* 80 >= 76: reads low 32 bits of the old log_opts pointer */
  log_opts = OPTS_GET(opts, log_opts, NULL);
      /* 80 < 88: NULL */

So the caller's log_opts is silently ignored, no log buffer is filled, and
true_size is never written back.  The kernel also receives a non-zero
arena_percpu_chunk_sz.

On kernels without the arena percpu attr, BPF_MAP_CREATE_LAST_FIELD is still
excl_prog_hash_size in kernel/bpf/syscall.c at this commit, so
CHECK_ATTR(BPF_MAP_CREATE) sees non-zero bytes past that field and returns
-EINVAL.  Does that mean every map creation that passes log_opts, which used
to succeed, now fails?

On kernels with the later d7935093885b, an ARENA map would get a bogus chunk
size (rejected as unaligned, or accepted with a wrong layout), and other map
types would silently lose their log.

The reverse case also looks problematic.  An application built with the new
header sets arena_percpu_chunk_sz but not log_opts, then runs against libbpf
1.8.  The 1.8 library reads offset 72 (arena_percpu_chunk_sz plus padding) as
a struct bpf_log_opts pointer, and OPTS_VALID(log_opts, bpf_log_opts)
dereferences it.

Nothing later in this series (a6e6ab7a4b2f..08224ba2de19) touches
tools/lib/bpf/bpf.h, so this is not fixed by a following patch.

Should the two new fields be appended after log_opts instead, with the
last-field marker updated?

	struct bpf_log_opts *log_opts;

	/* offset and size of an arena percpu chunk */
	__u32 arena_percpu_chunk_off;
	__u32 arena_percpu_chunk_sz;

	size_t :0;
};
#define bpf_map_create_opts__last_field arena_percpu_chunk_sz


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 06/16] libbpf: Relocate arena percpu access instructions

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Introduce a pseudo instruction for arena percpu, and relocate the normal
arena percpu access instruction to it.

The pseudo instruction is encoded as a BPF_PSEUDO_ARENA_PERCPU ldimm64,
and carries the offset of the value in cpu 0's chunk. At run time, the JIT
translates the offset to the value's address in the running cpu's replica.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 include/uapi/linux/bpf.h       | 13 +++++++++++++
 tools/include/uapi/linux/bpf.h | 13 +++++++++++++
 tools/lib/bpf/libbpf.c         | 35 +++++++++++++++++++++++++++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6f70db7515dd..9e5fa52c9bc2 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1418,6 +1418,19 @@ enum {
  */
 #define BPF_PSEUDO_FUNC		4
 
+/* insn[0].src_reg:  BPF_PSEUDO_ARENA_PERCPU
+ * insn[0].imm:      map fd
+ * insn[1].imm:      offset of the value in cpu 0's chunk, including the chunk's
+ *                   offset in the arena and the value's offset in the chunk
+ * insn[0].off:      0
+ * insn[1].off:      0
+ * ldimm64 rewrite:  address of the value in cpu 0's chunk. The JIT adds
+ *                   smp_processor_id() * chunk_sz to step it into the
+ *                   running cpu's replica
+ * verifier type:    PTR_TO_ARENA
+ */
+#define BPF_PSEUDO_ARENA_PERCPU	7
+
 /* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative
  * offset to another bpf function
  */
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 246f52361372..6b17dcfc36b4 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1418,6 +1418,19 @@ enum {
  */
 #define BPF_PSEUDO_FUNC		4
 
+/* insn[0].src_reg:  BPF_PSEUDO_ARENA_PERCPU
+ * insn[0].imm:      map fd
+ * insn[1].imm:      offset of the value in cpu 0's chunk, including the chunk's
+ *                   offset in the arena and the value's offset in the chunk
+ * insn[0].off:      0
+ * insn[1].off:      0
+ * ldimm64 rewrite:  address of the value in cpu 0's chunk. The JIT adds
+ *                   smp_processor_id() * chunk_sz to step it into the
+ *                   running cpu's replica
+ * verifier type:    PTR_TO_ARENA
+ */
+#define BPF_PSEUDO_ARENA_PERCPU	7
+
 /* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative
  * offset to another bpf function
  */
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 0b0c9d72b353..8634778e8ba5 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -378,6 +378,7 @@ enum reloc_type {
 	RELO_SUBPROG_ADDR,
 	RELO_CORE,
 	RELO_INSN_ARRAY,
+	RELO_ARENA_PERCPU,
 };
 
 struct reloc_desc {
@@ -4934,7 +4935,8 @@ static int bpf_prog_compute_hash(struct bpf_program *prog)
 		purged[i] = prog->insns[i];
 		if (purged[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
 		    (purged[i].src_reg == BPF_PSEUDO_MAP_FD ||
-		     purged[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
+		     purged[i].src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     purged[i].src_reg == BPF_PSEUDO_ARENA_PERCPU)) {
 			purged[i].imm = 0;
 			i++;
 			if (i >= prog->insns_cnt ||
@@ -5070,6 +5072,20 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
 		return 0;
 	}
 
+	/* arena percpu data relocation */
+	if (shdr_idx == obj->efile.arena_percpu_shndx) {
+		if (obj->arena_map_idx < 0) {
+			pr_warn("prog '%s': bad arena percpu data relocation at insn %u, no arena maps defined\n",
+				prog->name, insn_idx);
+			return -LIBBPF_ERRNO__RELOC;
+		}
+		reloc_desc->type = RELO_ARENA_PERCPU;
+		reloc_desc->insn_idx = insn_idx;
+		reloc_desc->map_idx = obj->arena_map_idx;
+		reloc_desc->sym_off = sym->st_value;
+		return 0;
+	}
+
 	/* jump table data relocation */
 	if (shdr_idx == obj->efile.jumptables_data_shndx) {
 		reloc_desc->type = RELO_INSN_ARRAY;
@@ -7105,6 +7121,23 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
 						   relo->map_idx, map);
 			}
 			break;
+		case RELO_ARENA_PERCPU:
+			/*
+			 * resolve the symbol to its arena-relative offset in cpu 0's chunk. The JIT
+			 * will add smp_processor_id() * chunk_sz to it at run time, so the address
+			 * ends up in the running cpu's replica.
+			 */
+			insn[1].imm = insn[0].imm + relo->sym_off + obj->arena_percpu.offset;
+
+			map = &obj->maps[relo->map_idx];
+			if (map->autocreate) {
+				insn[0].src_reg = BPF_PSEUDO_ARENA_PERCPU;
+				insn[0].imm = map->fd;
+			} else {
+				poison_map_ldimm64(prog, i, relo->insn_idx, insn,
+						   relo->map_idx, map);
+			}
+			break;
 		case RELO_EXTERN_LD64:
 			ext = &obj->externs[relo->ext_idx];
 			if (ext->type == EXT_KCFG) {
-- 
2.47.3

Re: [PATCH bpf-next 06/16] libbpf: Relocate arena percpu access instructions

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]

> libbpf: Relocate arena percpu access instructions

TL;DR: Nit only: the new BPF_PSEUDO_ARENA_PERCPU comment in bpf.h (and its
tools/ copy) starts text on the "/*" line instead of BPF comment style, though
it may match neighbouring comments.

> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 6f70db7515dd..9e5fa52c9bc2 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -1418,6 +1418,19 @@ enum {
>   */
>  #define BPF_PSEUDO_FUNC		4
>  
> +/* insn[0].src_reg:  BPF_PSEUDO_ARENA_PERCPU
                ^^^^

This isn't a bug, but the text of this new multi-line comment starts on the
same line as the opening "/*".  Should it use the BPF comment style, with
"/*" on a line of its own?

/*
 * insn[0].src_reg:  BPF_PSEUDO_ARENA_PERCPU
 * insn[0].imm:      map fd
 * ...
 */

The same comment is duplicated in tools/include/uapi/linux/bpf.h.  The
neighbouring BPF_PSEUDO_* comments in these headers use the old format, so
this may have been done to keep that table consistent.

> + * insn[0].imm:      map fd
> + * insn[1].imm:      offset of the value in cpu 0's chunk, including the chunk's
> + *                   offset in the arena and the value's offset in the chunk
> + * insn[0].off:      0
> + * insn[1].off:      0
> + * ldimm64 rewrite:  address of the value in cpu 0's chunk. The JIT adds
> + *                   smp_processor_id() * chunk_sz to step it into the
> + *                   running cpu's replica
> + * verifier type:    PTR_TO_ARENA
> + */
> +#define BPF_PSEUDO_ARENA_PERCPU	7

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 07/16] libbpf: Add arena percpu feature probe

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Arena percpu requires kernel and JIT support for the percpu pseudo
ldimm64. Probe it at load time and fail objects carrying arena percpu
data with -EOPNOTSUPP when it is missing.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/features.c        | 50 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.c          |  6 ++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 3 files changed, 58 insertions(+)

diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c
index adb5d2e7fdfa..f9c7d1545c1e 100644
--- a/tools/lib/bpf/features.c
+++ b/tools/lib/bpf/features.c
@@ -2,6 +2,7 @@
 /* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
 #include <linux/kernel.h>
 #include <linux/filter.h>
+#include <sys/mman.h>
 #include "bpf.h"
 #include "libbpf.h"
 #include "libbpf_common.h"
@@ -728,6 +729,52 @@ static int probe_kern_percpu_data(int token_fd)
 	return probe_fd(ret);
 }
 
+static int probe_kern_arena_percpu_data(int token_fd)
+{
+	int possible_cpus;
+	size_t page_sz = sysconf(_SC_PAGE_SIZE);
+	bool *mask = NULL;
+	struct bpf_insn insns[] = {
+		BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_ARENA_PERCPU, 0, 0, 0, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = BPF_F_MMAPABLE | (token_fd ? BPF_F_TOKEN_FD : 0),
+		.arena_percpu_chunk_sz = page_sz,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	void *mmaped;
+	int ret, map_fd, insn_cnt = ARRAY_SIZE(insns);
+
+	ret = parse_cpu_mask_file("/sys/devices/system/cpu/possible", &mask, &possible_cpus);
+	if (ret < 0 || possible_cpus <= 0) {
+		free(mask);
+		return -EINVAL;
+	}
+	free(mask);
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_ARENA, "arena_percpu", 0, 0, possible_cpus, &map_opts);
+	if (map_fd < 0)
+		return 0;
+
+	mmaped = mmap(NULL, possible_cpus * page_sz, PROT_READ | PROT_WRITE, MAP_SHARED, map_fd, 0);
+	if (mmaped == MAP_FAILED) {
+		close(map_fd);
+		return 0;
+	}
+
+	insns[0].imm = map_fd;
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	munmap(mmaped, possible_cpus * page_sz);
+	close(map_fd);
+
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -824,6 +871,9 @@ static struct kern_feature_desc {
 	[FEAT_EXCL_MAP_FD_ARRAY_CNT] = {
 		"exclusive map bound through fd_array_cnt", probe_kern_excl_map_fd_array_cnt,
 	},
+	[FEAT_ARENA_PERCPU_DATA] = {
+		"kernel supports arena percpu data", probe_kern_arena_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 8634778e8ba5..be2ddaf96631 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -8288,6 +8288,12 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 			return -EOPNOTSUPP;
 		}
 
+		if (obj->arena_percpu.present && !kernel_supports(obj, FEAT_ARENA_PERCPU_DATA)) {
+			pr_warn("map '%s': kernel doesn't support %s variables\n",
+				arena_map->name, ARENA_PERCPU_SEC);
+			return -EOPNOTSUPP;
+		}
+
 		if (data_sz + percpu_sz > mmap_sz) {
 			pr_warn("map '%s': declared ARENA map size (%zu) is too small to hold "
 				"global %s of size %zu plus the percpu data size %zu\n",
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index e90b5336bac0..2aaf681f8cb8 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -408,6 +408,8 @@ enum kern_feature_id {
 	FEAT_FD_ARRAY_CNT,
 	/* Kernel checks an exclusive map bound through fd_array_cnt against the final digest */
 	FEAT_EXCL_MAP_FD_ARRAY_CNT,
+	/* Kernel supports arena percpu data */
+	FEAT_ARENA_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
-- 
2.47.3

Re: [PATCH bpf-next 07/16] libbpf: Add arena percpu feature probe

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3570 bytes --]

> libbpf: Add arena percpu feature probe

TL;DR: The new arena percpu probe check in bpf_object__relocate() lacks an
arena_map->autocreate condition, so on kernels without the feature, objects
with .arena.percpu data may fail to load with -EOPNOTSUPP even when the arena
is disabled.

> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 8634778e8ba5..be2ddaf96631 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c

[ ... ]

> @@ -8288,6 +8288,12 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
>  			return -EOPNOTSUPP;
>  		}
>
> +		if (obj->arena_percpu.present && !kernel_supports(obj, FEAT_ARENA_PERCPU_DATA)) {
> +			pr_warn("map '%s': kernel doesn't support %s variables\n",
> +				arena_map->name, ARENA_PERCPU_SEC);
> +			return -EOPNOTSUPP;
> +		}
> +

Should this check also require arena_map->autocreate?

As written, any object with a .arena.percpu section fails to load on a
kernel without this feature, even when the arena map is not going to be
created.

The rest of the series supports an arena map with autocreate == false:

  bpf_object__relocate_data(), RELO_ARENA_PERCPU:
      if (map->autocreate) {
          insn[0].src_reg = BPF_PSEUDO_ARENA_PERCPU; ...
      } else {
          poison_map_ldimm64(...);
      }

  bpf_object__relocate_data_ptrs():
      if (!arena->autocreate)
          return 0;

  bpf_object__bind_arena():
      if (!map->autocreate || map->fd < 0 || ...)
          return 0;

The later commit 6714f66de847 ("libbpf: Introduce arena percpu helpers for
user space") also documents bpf_map__arena_percpu_base() as returning NULL
for a map with autocreate == false.

The related FEAT_PERCPU_DATA and FEAT_GLOBAL_DATA handling in
bpf_object__create_maps() skips creating the map and only fails the
programs that are loaded and reference it:

  If those global variable-using programs are not loaded at runtime due to
  bpf_program__set_autoload(prog, false), bpf_object loading will succeed
  just fine even on old kernels. Same skipping applies to percpu data.

Consider a kernel that supports arenas but not arena percpu, or an arch
whose JIT lacks bpf_jit_supports_arena_percpu_insn().  The application
calls bpf_map__set_autocreate(arena, false) and
bpf_program__set_autoload(prog_using_percpu, false), then
bpf_object__load():

bpf_object__load()
  bpf_object_load()
    bpf_object_prepare()
      bpf_object__relocate()
        return -EOPNOTSUPP;

At the parent commit 422f64147751 this same object loaded: the arena was
skipped in bpf_object__create_maps(), percpu ldimm64 instructions in
programs that are not autoloaded were never relocated, and poisoned ones
are only rejected if the program is loaded.

libbpf has no public probe for this feature, and retrying with the arena
disabled hits the same check.  Is there any configuration on such kernels
that can load an object carrying arena percpu data, including its programs
that do not use the arena?

Adding arena_map->autocreate && to the condition would keep the early, clear
error for the normal case while keeping the autocreate == false path
working.

The commit message says "fail objects carrying arena percpu data with
-EOPNOTSUPP when it is missing", but it does not say why this should apply
when the arena map is not going to be created.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 08/16] bpf: Extend syscall to accept arena percpu information

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Extend the bpf syscall to accept the arena percpu chunk offset and
size. Both are checked at arena creation time to make sure they are
page-aligned and the percpu chunks live inside the arena vm_range. If
the checks pass, the values are stored in struct bpf_arena for the
verifier and the JIT.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 include/uapi/linux/bpf.h |  4 ++++
 kernel/bpf/arena.c       | 18 ++++++++++++++++++
 kernel/bpf/syscall.c     |  2 +-
 3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 9e5fa52c9bc2..6b17dcfc36b4 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1628,6 +1628,10 @@ union bpf_attr {
 		__aligned_u64 excl_prog_hash;
 		/* Size of the passed excl_prog_hash. */
 		__u32 excl_prog_hash_size;
+
+		/* offset and size of an arena percpu chunk */
+		__u32	arena_percpu_chunk_off;
+		__u32	arena_percpu_chunk_sz;
 	};
 
 	struct { /* anonymous struct used by BPF_MAP_*_ELEM and BPF_MAP_FREEZE commands */
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 868589f07330..05cd6397adae 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -67,6 +67,9 @@ struct bpf_arena {
 	struct irq_work     free_irq;
 	struct work_struct  free_work;
 	struct llist_head   free_spans;
+	/* percpu chunk offset and size */
+	u32 percpu_chunk_off;
+	u32 percpu_chunk_sz;
 };
 
 static void arena_free_worker(struct work_struct *work);
@@ -298,6 +301,17 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 		/* user vma must not cross 32-bit boundary */
 		return ERR_PTR(-ERANGE);
 
+	if (attr->arena_percpu_chunk_sz) {
+		u64 offset = attr->arena_percpu_chunk_off;
+		u64 size = attr->arena_percpu_chunk_sz;
+
+		if (!IS_ALIGNED(offset, PAGE_SIZE) || !IS_ALIGNED(size, PAGE_SIZE))
+			return ERR_PTR(-EINVAL);
+
+		if (offset > vm_range || size > (vm_range - offset) / nr_cpu_ids)
+			return ERR_PTR(-E2BIG);
+	}
+
 	kern_vm = get_vm_area(KERN_VM_SZ, VM_SPARSE | VM_USERMAP);
 	if (!kern_vm)
 		return ERR_PTR(-ENOMEM);
@@ -311,6 +325,10 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 	if (arena->user_vm_start)
 		arena->user_vm_end = arena->user_vm_start + vm_range;
 
+	arena->percpu_chunk_sz = attr->arena_percpu_chunk_sz;
+	if (attr->arena_percpu_chunk_sz)
+		arena->percpu_chunk_off = attr->arena_percpu_chunk_off;
+
 	INIT_LIST_HEAD(&arena->vma_list);
 	init_llist_head(&arena->free_spans);
 	init_irq_work(&arena->free_irq, arena_free_irq);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 849bc9136299..240c7349284d 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1396,7 +1396,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
 	return ret;
 }
 
-#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size
+#define BPF_MAP_CREATE_LAST_FIELD arena_percpu_chunk_sz
 /* called via syscall */
 static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,
 			    struct bpf_map **mapp, struct bpf_token **tokenp)
-- 
2.47.3

Re: [PATCH bpf-next 08/16] bpf: Extend syscall to accept arena percpu information

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]

> bpf: Extend syscall to accept arena percpu information

TL;DR:
- BPF_MAP_CREATE may silently accept arena percpu fields for non-arena map
  types, blocking future uapi use
- A nonzero percpu chunk offset with zero size is not validated

> diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
> index 868589f07330..05cd6397adae 100644
> --- a/kernel/bpf/arena.c
> +++ b/kernel/bpf/arena.c

[ ... ]

> @@ -298,6 +301,17 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
>  		/* user vma must not cross 32-bit boundary */
>  		return ERR_PTR(-ERANGE);
>
> +	if (attr->arena_percpu_chunk_sz) {
> +		u64 offset = attr->arena_percpu_chunk_off;
> +		u64 size = attr->arena_percpu_chunk_sz;
> +
> +		if (!IS_ALIGNED(offset, PAGE_SIZE) || !IS_ALIGNED(size, PAGE_SIZE))
> +			return ERR_PTR(-EINVAL);
> +
> +		if (offset > vm_range || size > (vm_range - offset) / nr_cpu_ids)
> +			return ERR_PTR(-E2BIG);
> +	}
> +
>  	kern_vm = get_vm_area(KERN_VM_SZ, VM_SPARSE | VM_USERMAP);
>  	if (!kern_vm)
>  		return ERR_PTR(-ENOMEM);

[ ... ]

> @@ -311,6 +325,10 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
>  	if (arena->user_vm_start)
>  		arena->user_vm_end = arena->user_vm_start + vm_range;
>
> +	arena->percpu_chunk_sz = attr->arena_percpu_chunk_sz;
> +	if (attr->arena_percpu_chunk_sz)
> +		arena->percpu_chunk_off = attr->arena_percpu_chunk_off;
> +

If userspace passes arena_percpu_chunk_off != 0 with arena_percpu_chunk_sz
== 0, is the offset ever validated?

In that case the alignment and range checks above are skipped, the offset is
ignored, and arena->percpu_chunk_off stays 0.

The commit message says "Both are checked at arena creation time", which
only holds when arena_percpu_chunk_sz is nonzero.  The excl_prog_hash
handling in map_create_alloc() already rejects the matching case of a size
without a hash:

	} else if (attr->excl_prog_hash_size) {
		...
		err = -EINVAL;

Should arena_map_alloc() return -EINVAL when arena_percpu_chunk_off is
nonzero but arena_percpu_chunk_sz is 0?

[ ... ]

> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index 849bc9136299..240c7349284d 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -1396,7 +1396,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
>  	return ret;
>  }
>
> -#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size
> +#define BPF_MAP_CREATE_LAST_FIELD arena_percpu_chunk_sz
>  /* called via syscall */
>  static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,
>  			    struct bpf_map **mapp, struct bpf_token **tokenp)

Does raising BPF_MAP_CREATE_LAST_FIELD here make CHECK_ATTR accept
arena_percpu_chunk_off and arena_percpu_chunk_sz for every map type?

Only arena_map_alloc() reads them, and map_create_alloc() has no check for
these fields.  For example, BPF_MAP_CREATE with map_type =
BPF_MAP_TYPE_HASH and arena_percpu_chunk_sz = 4096 succeeds and the values
are silently dropped.

map_create_alloc() already rejects the comparable per-type field:

	if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER &&
	    attr->map_type != BPF_MAP_TYPE_ARENA &&
	    attr->map_type != BPF_MAP_TYPE_RHASH &&
	    attr->map_extra != 0) {
		bpf_log(log, "Invalid map_extra.\n");
		return -EINVAL;
	}

Since current kernels will accept any value in these fields, they can never
later be given a meaning for other map types, or for the offset-only case
above.  Bad loader input, such as percpu info attached to the wrong map, is
also never reported.  No later commit in the series (through 08224ba2de19)
adds this check.

Should map_create_alloc() return -EINVAL when map_type is not
BPF_MAP_TYPE_ARENA and either arena_percpu_chunk_off or
arena_percpu_chunk_sz is nonzero?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 09/16] bpf: Teach verifier about arena percpu instructions

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Teach the verifier about the arena percpu pseudo instruction. The
percpu chunks of all cpus have the same size and layout, so the
verifier only checks the address in the pseudo instruction against
cpu 0's chunk. If the offset or the access size goes past cpu 0's
chunk, the prog is rejected.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 include/linux/bpf.h    |  3 ++
 include/linux/filter.h |  1 +
 kernel/bpf/arena.c     | 23 ++++++++++++++
 kernel/bpf/core.c      | 22 +++++++++++++-
 kernel/bpf/verifier.c  | 69 +++++++++++++++++++++++++++++++++++++++---
 5 files changed, 113 insertions(+), 5 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7750df973494..5ffefcd5f15e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -674,6 +674,9 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
 u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
 u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
 u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
+u32 bpf_arena_percpu_chunk_off(const struct bpf_map *map);
+u32 bpf_arena_percpu_chunk_sz(const struct bpf_map *map);
+bool bpf_arena_percpu_off_in_range(const struct bpf_map *map, u32 off);
 struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
 int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
 
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 6f4c9bd024a0..d4d80e14d594 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1254,6 +1254,7 @@ bool bpf_jit_needs_zext(void);
 bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
+bool bpf_jit_supports_arena_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
 bool bpf_jit_supports_callx(void);
 bool bpf_jit_supports_kfunc_ret_reg_pair(void);
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 05cd6397adae..310573a2661b 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -102,6 +102,29 @@ u64 bpf_arena_map_kern_vm_start(struct bpf_map *map)
 	return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map));
 }
 
+u32 bpf_arena_percpu_chunk_off(const struct bpf_map *map)
+{
+	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
+
+	return arena->percpu_chunk_off;
+}
+
+u32 bpf_arena_percpu_chunk_sz(const struct bpf_map *map)
+{
+	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
+
+	return arena->percpu_chunk_sz;
+}
+
+bool bpf_arena_percpu_off_in_range(const struct bpf_map *map, u32 off)
+{
+	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
+	u32 chunk_off = arena->percpu_chunk_off;
+	u32 chunk_sz = arena->percpu_chunk_sz;
+
+	return off >= chunk_off && off - chunk_off < chunk_sz;
+}
+
 /**
  * bpf_prog_arena - return the bpf_map of the arena referenced by @prog
  * @prog: a loaded BPF program
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index fc22d66d194a..29291101416a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -319,7 +319,8 @@ int bpf_prog_calc_tag(struct bpf_prog *fp)
 		if (!was_ld_map &&
 		    dst[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
 		    (dst[i].src_reg == BPF_PSEUDO_MAP_FD ||
-		     dst[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
+		     dst[i].src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     dst[i].src_reg == BPF_PSEUDO_ARENA_PERCPU)) {
 			was_ld_map = true;
 			dst[i].imm = 0;
 		} else if (was_ld_map &&
@@ -1600,6 +1601,20 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bp
 			continue;
 		}
 
+		if (insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
+			/*
+			 * The imm of an arena percpu pseudo insn is the
+			 * loader resolved address of the value in cpu 0's
+			 * chunk, and the JIT translates the pseudo insn to
+			 * step it into the running cpu's chunk. No need to
+			 * blind it.
+			 */
+			insn++;
+			i++;
+			continue;
+		}
+
 		/* We temporarily need to hold the original ld64 insn
 		 * so that we can still access the first part in the
 		 * second blinding run.
@@ -3383,6 +3398,11 @@ bool __weak bpf_jit_supports_percpu_insn(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_arena_percpu_insn(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_kfunc_call(void)
 {
 	return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 01434e1d23b5..0c0d3662ec76 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4724,6 +4724,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state
 		verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n",
 			mem_size, off, size);
 		break;
+	case PTR_TO_ARENA:
+		verbose(env, "per-cpu arena access at offset %d size %d outside cpu 0's chunk\n",
+			off, size);
+		break;
 	case PTR_TO_MEM:
 	default:
 		verbose(env, "invalid access to memory, mem_size=%u off=%d size=%d\n",
@@ -7415,6 +7419,17 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 		if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ))
 			mark_reg_unknown(env, regs, value_regno);
 	} else if (reg->type == PTR_TO_ARENA) {
+		/*
+		 * A per-cpu reference is bounded to cpu 0's chunk through
+		 * mem_size. A plain arena pointer has no bound and only
+		 * faults at runtime.
+		 */
+		if (reg->mem_size) {
+			err = check_mem_region_access(env, reg, argno, off, size,
+						      reg->mem_size, false);
+			if (err)
+				return err;
+		}
 		if (t == BPF_READ && value_regno >= 0)
 			mark_reg_unknown(env, regs, value_regno);
 	} else {
@@ -17456,7 +17471,12 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 			aux->zext_dst = true;
 		}
 
-		/* Any arithmetic operations are allowed on arena pointers */
+		/*
+		 * Any arithmetic operations are allowed on arena pointers.
+		 * Arithmetic on a per-cpu reference drops its chunk bound.
+		 * The register rolls back to a plain arena pointer.
+		 */
+		dst_reg->mem_size = 0;
 		return 0;
 	}
 
@@ -19080,10 +19100,27 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	map = env->used_maps[aux->map_index];
 
 	if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
-	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
+	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE ||
+	    insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
 		if (map->map_type == BPF_MAP_TYPE_ARENA) {
-			__mark_reg_unknown(env, dst_reg);
-			dst_reg->map_ptr = map;
+			if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
+				u64 pos = aux->map_off - bpf_arena_percpu_chunk_off(map);
+
+				/*
+				 * The register value tracks the position of
+				 * the reference inside cpu 0's chunk and
+				 * mem_size bounds accesses to the chunk.
+				 * map_ptr and mem_size share a union and
+				 * map_ptr is never read for PTR_TO_ARENA,
+				 * so only mem_size is set.
+				 */
+				__mark_reg_known(dst_reg, pos);
+				dst_reg->type = PTR_TO_ARENA;
+				dst_reg->mem_size = bpf_arena_percpu_chunk_sz(map);
+			} else {
+				__mark_reg_unknown(env, dst_reg);
+				dst_reg->map_ptr = map;
+			}
 			bpf_diag_mod_end(env);
 			return 0;
 		}
@@ -21372,6 +21409,7 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 			switch (insn[0].src_reg) {
 			case BPF_PSEUDO_MAP_VALUE:
 			case BPF_PSEUDO_MAP_IDX_VALUE:
+			case BPF_PSEUDO_ARENA_PERCPU:
 				break;
 			case BPF_PSEUDO_MAP_FD:
 			case BPF_PSEUDO_MAP_IDX:
@@ -21410,6 +21448,26 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 			} else {
 				u32 off = insn[1].imm;
 
+				if (insn[0].src_reg == BPF_PSEUDO_ARENA_PERCPU) {
+					if (map->map_type != BPF_MAP_TYPE_ARENA) {
+						verbose(env, "per-cpu value pseudo insn only allowed on arena maps\n");
+						return -EINVAL;
+					}
+					if (!bpf_arena_percpu_chunk_sz(map)) {
+						verbose(env, "arena map has no per-cpu chunks\n");
+						return -EINVAL;
+					}
+					if (!bpf_arena_percpu_off_in_range(map, off)) {
+						verbose(env, "per-cpu arena reference at offset %u outside cpu 0's chunk\n",
+							off);
+						return -ERANGE;
+					}
+					if (!bpf_jit_supports_arena_percpu_insn()) {
+						verbose(env, "JIT doesn't support arena percpu insn\n");
+						return -EOPNOTSUPP;
+					}
+				}
+
 				if (!map->ops->map_direct_value_addr) {
 					verbose(env, "no direct value access support for this map type\n");
 					return -EINVAL;
@@ -21579,6 +21637,9 @@ static void convert_pseudo_ld_imm64(struct bpf_verifier_env *env)
 			continue;
 		if (insn->src_reg == BPF_PSEUDO_FUNC)
 			continue;
+		/* The JIT reads this pseudo to emit the per-cpu step */
+		if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU)
+			continue;
 		insn->src_reg = 0;
 	}
 }
-- 
2.47.3

Re: [PATCH bpf-next 09/16] bpf: Teach verifier about arena percpu instructions

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 8557 bytes --]

> bpf: Teach verifier about arena percpu instructions

TL;DR:
- Verifier state pruning may skip arena percpu bounds checks, so out-of-range
  accesses could load
- Possible vmlinux link failure on 32-bit/!MMU: no stubs for new arena helpers
- JIT constant blinding skipped for loader-controlled immediates

> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index 7750df973494..5ffefcd5f15e 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -674,6 +674,9 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
>  u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
>  u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
>  u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
> +u32 bpf_arena_percpu_chunk_off(const struct bpf_map *map);
> +u32 bpf_arena_percpu_chunk_sz(const struct bpf_map *map);
> +bool bpf_arena_percpu_off_in_range(const struct bpf_map *map, u32 off);
>  struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
>  int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
>  

Is this declaration reachable from a build that doesn't compile arena.c?

kernel/bpf/Makefile only builds arena.c when both options are set:

    ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)
    obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o
    endif

verifier.o is built for every CONFIG_BPF_SYSCALL config, and this patch
calls the new helpers from it behind runtime map_type checks only, which the
compiler cannot remove:

    check_ld_imm():
        u64 pos = aux->map_off - bpf_arena_percpu_chunk_off(map);
        dst_reg->mem_size = bpf_arena_percpu_chunk_sz(map);

    check_and_resolve_insns():
        if (!bpf_arena_percpu_chunk_sz(map)) {
        if (!bpf_arena_percpu_off_in_range(map, off)) {

Can this fail to link vmlinux with undefined references to
bpf_arena_percpu_chunk_off, bpf_arena_percpu_chunk_sz and
bpf_arena_percpu_off_in_range on 32-bit or !MMU kernels with
CONFIG_BPF_SYSCALL=y?  In-tree examples are
arch/arm/configs/imx_v6_v7_defconfig, aspeed_g5_defconfig and
imxrt_defconfig (!MMU).

The other arena.c symbols used by always-built code all have a fallback.
kernel/bpf/core.c has __weak bpf_arena_get_user_vm_start() and
bpf_arena_get_kern_vm_start() stubs "for configs without MMU or 32-bit", and
include/linux/bpf.h wraps the arena kfuncs in
"#if defined(CONFIG_MMU) && defined(CONFIG_64BIT)" with static inline stubs.

Nothing later in the series adds stubs for these three helpers.  The JIT
callers added later are 64-bit only, so stubs (or a CONFIG_MMU &&
CONFIG_64BIT guard) would be needed only for the verifier call sites.

> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index fc22d66d194a..29291101416a 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c

[ ... ]

> @@ -1600,6 +1601,20 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bp
>  			continue;
>  		}
>  
> +		if (insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
> +		    insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
> +			/*
> +			 * The imm of an arena percpu pseudo insn is the
> +			 * loader resolved address of the value in cpu 0's
> +			 * chunk, and the JIT translates the pseudo insn to
> +			 * step it into the running cpu's chunk. No need to
> +			 * blind it.
> +			 */
> +			insn++;
> +			i++;
> +			continue;
> +		}
> +
>  		/* We temporarily need to hold the original ld64 insn
>  		 * so that we can still access the first part in the
>  		 * second blinding run.

Is it safe to skip constant blinding for this instruction?

The new comment describes the imm as the "loader resolved address".
check_and_resolve_insns() builds it from loader chosen values:

    arena_map_direct_value_addr() + off
      == arena->user_vm_start + insn[1].imm

user_vm_start comes from attr->map_extra (any page aligned value that does
not cross a 32-bit boundary), and off can be any value inside the chunk.  So
the loader controls essentially all 64 bits that the JIT emits as an
immediate (movabs on x86, a mov sequence on arm64).

The bpf_pseudo_func() skip just above is justified because that address "is
not a user controlled constant".  This one is.  By comparison, a regular
arena global (BPF_PSEUDO_MAP_VALUE) has its src_reg cleared by
convert_pseudo_ld_imm64() and is blinded.

With net.core.bpf_jit_harden=2, can a CAP_BPF + CAP_PERFMON loader now place
chosen 8-byte immediates in the JIT image through this instruction?

The comment's reason ("the JIT translates the pseudo insn") explains why the
generic rewrite cannot be applied, since that would drop the src_reg marker.
It doesn't show that blinding is unnecessary, and the commit message does
not mention the exemption.

Would blinding the value while keeping the per-cpu step work, for example by
emitting the per-cpu add as a separate step after a blinded load?

[ ... ]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 01434e1d23b5..0c0d3662ec76 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -19080,10 +19100,27 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
>  	map = env->used_maps[aux->map_index];
>  
>  	if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
> -	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
> +	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE ||
> +	    insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
>  		if (map->map_type == BPF_MAP_TYPE_ARENA) {
> -			__mark_reg_unknown(env, dst_reg);
> -			dst_reg->map_ptr = map;
> +			if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
> +				u64 pos = aux->map_off - bpf_arena_percpu_chunk_off(map);
> +
> +				/*
> +				 * The register value tracks the position of
> +				 * the reference inside cpu 0's chunk and
> +				 * mem_size bounds accesses to the chunk.
> +				 * map_ptr and mem_size share a union and
> +				 * map_ptr is never read for PTR_TO_ARENA,
> +				 * so only mem_size is set.
> +				 */
> +				__mark_reg_known(dst_reg, pos);
> +				dst_reg->type = PTR_TO_ARENA;
> +				dst_reg->mem_size = bpf_arena_percpu_chunk_sz(map);
> +			} else {
> +				__mark_reg_unknown(env, dst_reg);
> +				dst_reg->map_ptr = map;
> +			}
>  			bpf_diag_mod_end(env);
>  			return 0;
>  		}

Does state pruning still distinguish these registers?

This patch gives PTR_TO_ARENA registers safety state.  var_off holds the
position inside cpu 0's chunk and mem_size holds the bound that
check_mem_access() enforces.  regsafe() in kernel/bpf/states.c was not
updated, and it still treats any two PTR_TO_ARENA registers as equivalent:

    case PTR_TO_ARENA:
        return true;

A path whose per-cpu reference is out of range can then be pruned against an
explored path whose reference is in range (or is a plain unbounded arena
pointer), and the bound check never runs for it.  For example:

    r0 = bpf_get_prandom_u32()
    r1 = ld_imm64 ARENA_PERCPU(off = chunk_off + chunk_sz - 4)
                                       ; pos = chunk_sz - 4
    if r0 == 0 goto L
    r1 = ld_imm64 ARENA_PERCPU(off = chunk_off)
                                       ; pos = 0
  L:
    r2 = *(u64 *)(r1 + 0)
    r0 = 0
    exit

The fall-through path (pos 0) is explored first, passes the 8 byte load at
L, and leaves a checkpoint there.  That needs enough preceding insns/jumps,
or BPF_F_TEST_STATE_FREQ.

When the pushed branch (pos chunk_sz - 4) reaches L, states_equal() ->
regsafe() returns true for r1.  r0 is not precise, so it matches too.  The
path is pruned and the program loads, even though that load reads 4 bytes
past cpu 0's chunk.

A plain arena pointer (e.g. after addr_space_cast) on the first path hides
an out-of-range per-cpu reference on the second path in the same way, so
whether the program is accepted depends on exploration order.

That contradicts the commit message ("If the offset or the access size goes
past cpu 0's chunk, the prog is rejected") and what the percpu_bounds
selftest added in 08224ba2de19 expects.  At runtime the access lands in the
next CPU's chunk, or past the last chunk.  Arena accesses are PROBE_MEM32,
so kernel memory is not exposed, but per-cpu data is silently shared across
CPUs.

Should regsafe() compare mem_size, and check range_within()/tnum_in() on
the position, for PTR_TO_ARENA registers with mem_size != 0?

This is not fixed later in the series.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 10/16] bpf, x86: Add support for arena percpu

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add x86 JIT support for the arena percpu pseudo instruction.
Translate the address for each cpu by adding the offset
smp_processor_id() * chunk_sz to cpu 0's address.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/x86/net/bpf_jit_comp.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 72a2a0a89e03..7c09a4d90cd1 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2772,6 +2772,34 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_LD | BPF_IMM | BPF_DW:
 			emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm);
+
+#ifdef CONFIG_SMP
+			if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
+				u32 chunk_sz = bpf_arena_percpu_chunk_sz(bpf_prog_arena(bpf_prog));
+
+				/* aux = smp_processor_id(); mov aux, gs:[cpu_number] */
+				EMIT2(0x65, add_2mod(0x40, 0, AUX_REG));
+				EMIT3(0x8B, add_2reg(0x04, 0, AUX_REG), 0x25);
+				EMIT((u32)(unsigned long)&cpu_number, 4);
+
+				/*
+				 * aux *= chunk_sz
+				 * arena_map_alloc() guarantees
+				 * chunk_sz * nr_cpu_ids <= vm_range <= 4 GiB and
+				 * cpu_number < nr_cpu_ids, so a 32-bit imul
+				 * cannot overflow here.
+				 */
+				maybe_emit_mod(&prog, AUX_REG, AUX_REG, false);
+				/* imul aux, aux, imm32 */
+				EMIT2_off32(0x69, add_2reg(0xC0, AUX_REG, AUX_REG),
+					    chunk_sz);
+
+				/* dst += aux */
+				maybe_emit_mod(&prog, dst_reg, AUX_REG, true);
+				EMIT2(0x01, add_2reg(0xC0, dst_reg, AUX_REG));
+			}
+#endif
+
 			insn++;
 			i++;
 			break;
@@ -5072,6 +5100,11 @@ bool bpf_jit_supports_percpu_insn(void)
 	return true;
 }
 
+bool bpf_jit_supports_arena_percpu_insn(void)
+{
+	return true;
+}
+
 void bpf_jit_free(struct bpf_prog *prog)
 {
 	if (prog->jited) {
-- 
2.47.3

[PATCH bpf-next 11/16] bpf, arm64: Add support for arena percpu

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add arm64 JIT support for the arena percpu pseudo instruction.
Translate the address for each cpu by adding the offset
smp_processor_id() * chunk_sz to cpu 0's address.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/arm64/net/bpf_jit_comp.c | 51 ++++++++++++++++++++++++++++-------
 include/linux/bpf.h           |  2 +-
 kernel/bpf/arena.c            |  2 +-
 3 files changed, 43 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 1d5f4cf88d33..998806cc700e 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -500,6 +500,21 @@ static void emit_percpu_ptr(const u8 dst_reg, void __percpu *ptr,
 	emit(A64_ADD(1, dst_reg, dst_reg, tmp), ctx);
 }
 
+static void emit_current_cpu_id(u8 dst, struct jit_ctx *ctx)
+{
+	u8 tmp = bpf2a64[TMP_REG_1];
+	u8 tmp2 = bpf2a64[TMP_REG_2];
+	u32 cpu_offset = offsetof(struct thread_info, cpu);
+
+	emit(A64_MRS_SP_EL0(tmp), ctx);
+	if (is_lsi_offset(cpu_offset, 2)) {
+		emit(A64_LDR32I(dst, tmp, cpu_offset), ctx);
+	} else {
+		emit_a64_mov_i(1, tmp2, cpu_offset, ctx);
+		emit(A64_LDR32(dst, tmp, tmp2), ctx);
+	}
+}
+
 #define BTI_INSNS (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) ? 1 : 0)
 #define PAC_INSNS (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) ? 1 : 0)
 
@@ -1742,19 +1757,10 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		const u8 r0 = bpf2a64[BPF_REG_0];
 		bool func_addr_fixed;
 		u64 func_addr;
-		u32 cpu_offset;
 
 		/* Implement helper call to bpf_get_smp_processor_id() inline */
 		if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
-			cpu_offset = offsetof(struct thread_info, cpu);
-
-			emit(A64_MRS_SP_EL0(tmp), ctx);
-			if (is_lsi_offset(cpu_offset, 2)) {
-				emit(A64_LDR32I(r0, tmp, cpu_offset), ctx);
-			} else {
-				emit_a64_mov_i(1, tmp2, cpu_offset, ctx);
-				emit(A64_LDR32(r0, tmp, tmp2), ctx);
-			}
+			emit_current_cpu_id(r0, ctx);
 			break;
 		}
 
@@ -1829,6 +1835,26 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		else
 			emit_a64_mov_i64(dst, imm64, ctx);
 
+#ifdef CONFIG_SMP
+		if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
+			u32 chunk_sz = bpf_arena_percpu_chunk_sz(bpf_prog_arena(ctx->prog));
+
+			/* tmp = smp_processor_id() */
+			emit_current_cpu_id(tmp, ctx);
+			/*
+			 * tmp *= chunk_sz
+			 * arena_map_alloc() guarantees
+			 * chunk_sz * nr_cpu_ids <= vm_range <= 4 GiB and
+			 * cpu_number < nr_cpu_ids, so a 32-bit mul
+			 * cannot overflow here.
+			 */
+			emit_a64_mov_i(0, tmp2, chunk_sz, ctx);
+			emit(A64_MUL(0, tmp, tmp, tmp2), ctx);
+			/* dst += tmp */
+			emit(A64_ADD(1, dst, dst, tmp), ctx);
+		}
+#endif
+
 		return 1;
 	}
 
@@ -3446,6 +3472,11 @@ bool bpf_jit_supports_percpu_insn(void)
 	return true;
 }
 
+bool bpf_jit_supports_arena_percpu_insn(void)
+{
+	return true;
+}
+
 bool bpf_jit_bypass_spec_v4(void)
 {
 	/* In case of arm64, we rely on the firmware mitigation of Speculative
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5ffefcd5f15e..4a7af8433c19 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -677,7 +677,7 @@ u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
 u32 bpf_arena_percpu_chunk_off(const struct bpf_map *map);
 u32 bpf_arena_percpu_chunk_sz(const struct bpf_map *map);
 bool bpf_arena_percpu_off_in_range(const struct bpf_map *map, u32 off);
-struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
+struct bpf_map *bpf_prog_arena(const struct bpf_prog *prog);
 int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
 
 struct bpf_offload_dev;
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 310573a2661b..4d2df31d1811 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -133,7 +133,7 @@ bool bpf_arena_percpu_off_in_range(const struct bpf_map *map, u32 off)
  * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if
  * @prog does not reference an arena.
  */
-struct bpf_map *bpf_prog_arena(struct bpf_prog *prog)
+struct bpf_map *bpf_prog_arena(const struct bpf_prog *prog)
 {
 	struct bpf_arena *arena = prog->aux->arena;
 
-- 
2.47.3

Re: [PATCH bpf-next 11/16] bpf, arm64: Add support for arena percpu

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4713 bytes --]

> bpf, arm64: Add support for arena percpu

TL;DR: On arm64 (and likely x86), the arena percpu ld_imm64 yields a full
64-bit user address, so a direct load/store without addr_space_cast may hit
the wrong kernel address instead of the arena.

> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 1d5f4cf88d33..998806cc700e 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c

[ ... ]

> @@ -1829,6 +1835,26 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
>  		else
>  			emit_a64_mov_i64(dst, imm64, ctx);
>  
> +#ifdef CONFIG_SMP
> +		if (insn->src_reg == BPF_PSEUDO_ARENA_PERCPU) {
> +			u32 chunk_sz = bpf_arena_percpu_chunk_sz(bpf_prog_arena(ctx->prog));
> +
> +			/* tmp = smp_processor_id() */
> +			emit_current_cpu_id(tmp, ctx);
> +			/*
> +			 * tmp *= chunk_sz
> +			 * arena_map_alloc() guarantees
> +			 * chunk_sz * nr_cpu_ids <= vm_range <= 4 GiB and
> +			 * cpu_number < nr_cpu_ids, so a 32-bit mul
> +			 * cannot overflow here.
> +			 */
> +			emit_a64_mov_i(0, tmp2, chunk_sz, ctx);
> +			emit(A64_MUL(0, tmp, tmp, tmp2), ctx);
> +			/* dst += tmp */
> +			emit(A64_ADD(1, dst, dst, tmp), ctx);

Can a direct dereference of this register end up outside the arena window?

For BPF_PSEUDO_ARENA_PERCPU, imm64 is user_vm_start + off (set up by
check_and_resolve_insns() from arena_map_direct_value_addr()), so after this
sequence dst holds the full 64-bit user virtual address plus cpu * chunk_sz.

The verifier types this register as PTR_TO_ARENA in check_ld_imm(), and
convert_ctx_accesses() rewrites loads and stores through it to
BPF_PROBE_MEM32 without any truncation.

Elsewhere a PTR_TO_ARENA register only holds a 32-bit arena offset.  The
cast_kern conversion is a mov32, and adjust_reg_min_max_vals() turns ALU64 on
PTR_TO_ARENA into ALU32.  Arena pages are mapped at kern_vm_start +
(u32)uaddr (see arena_vm_fault()).

The arm64 JIT then forms the address with a 64-bit add.  For LDX in
build_insn():

    if (BPF_MODE(insn->code) == BPF_PROBE_MEM32 ||
        BPF_MODE(insn->code) == BPF_PROBE_MEM32SX) {
            emit(A64_ADD(1, tmp2, src, arena_vm_base), ctx);
            src = tmp2;
    }

The ST and STX cases in build_insn() use the same pattern with dst, and so do
the PROBE_ATOMIC paths in emit_atomic_ld_st() and emit_lse_atomic():

    emit(A64_ADD(1, tmp, reg, arena_vm_base), ctx);

So a direct access through the pseudo-load result would touch
kern_vm_start + user_vm_start + off + cpu * chunk_sz, instead of
kern_vm_start + (u32)(user_vm_start + off) + cpu * chunk_sz.

These only agree when upper_32_bits(user_vm_start) == 0.  The verifier
requires user_vm_start to be set, and it is normally a high mmap address or
whatever map_extra the loader picked.

If the resulting address is unmapped, the exception handler zeroes the load
or skips the store, so the access silently fails.  If something else in
vmalloc space is mapped there, the program reads or writes unrelated kernel
memory, and map_extra lets the loader steer the offset.

There appear to be two ways to reach this.  One is the raw instruction
pattern used by the selftest at the end of this series (08224ba2de19):

    BPF_LD_IMM64_RAW(BPF_REG_1, BPF_PSEUDO_ARENA_PERCPU, 0)
    BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, 0)

which loads successfully and only needs to be run.  The other is an object
built with the bpf_arena_common.h fallback (#define __arena_percpu
SEC(".arena.percpu") without __BPF_FEATURE_ADDR_SPACE_CAST, or with
BPF_ARENA_FORCE_ASM), where the compiler emits no addr_space_cast between the
ld_imm64 and the load or store.  LLVM output that does insert cast_kern is
not affected, since the mov32 truncates the value.

Before this patch arm64 rejected the pseudo instruction with -EOPNOTSUPP, so
returning true from bpf_jit_supports_arena_percpu_insn() is what makes this
reachable on arm64.  The x86 lowering in a27c971d206b looks to have the same
problem, since its PROBE_MEM32 addressing is [src + r12 + off] with a 64-bit
src.  No later commit in the series touches the kernel code.

Would keeping only the low 32 bits for the PTR_TO_ARENA view work, for
example a 32-bit move of (u32)imm64 followed by a 32-bit ADD of the per-cpu
offset?  Alternatively, should the verifier type the result like a regular
arena ld_imm64 (scalar / user address), so that the kernel-side access goes
through the existing 32-bit conversion?

> +		}
> +#endif


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 12/16] libbpf: Introduce arena percpu helpers for user space

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Introduce two helpers for user space.
- bpf_map__arena_percpu_base(): return the base address of the arena
  percpu. The generated skeleton will call it to set up its arena
  percpu data.
- bpf_map__arena_percpu_ptr(): translate a variable address in cpu 0's
  chunk to the address in the target cpu's chunk.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/lib/bpf/libbpf.c   | 49 ++++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h   | 26 +++++++++++++++++++++
 tools/lib/bpf/libbpf.map |  2 ++
 3 files changed, 77 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index be2ddaf96631..0edce476451c 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -12287,6 +12287,55 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize)
 	return map->mmaped;
 }
 
+void *bpf_map__arena_percpu_ptr(const struct bpf_map *map, void *ptr, unsigned int cpu)
+{
+	struct bpf_object *obj;
+	uint64_t base;
+
+	if (!map || !ptr)
+		return libbpf_err_ptr(-EINVAL);
+
+	obj = map->obj;
+	if (map->def.type != BPF_MAP_TYPE_ARENA || !obj->arena_percpu.present)
+		return libbpf_err_ptr(-EINVAL);
+
+	if (cpu >= obj->arena_percpu.nr_chunks)
+		return libbpf_err_ptr(-ERANGE);
+
+	/*
+	 * obj->arena_percpu.chunk is freed by arena_percpu_populate(), and only
+	 * then the map is mmaped
+	 */
+	if (obj->arena_percpu.chunk)
+		return libbpf_err_ptr(-EINVAL);
+
+	base = (uint64_t)map->mmaped + obj->arena_percpu.offset;
+	if ((uint64_t)ptr < base || (uint64_t)ptr >= base + obj->arena_percpu.chunk_sz)
+		return libbpf_err_ptr(-EINVAL);
+
+	return (void *)((uint64_t)ptr + (uint64_t)cpu * obj->arena_percpu.chunk_sz);
+}
+
+void *bpf_map__arena_percpu_base(const struct bpf_map *map)
+{
+	struct bpf_object *obj;
+
+	if (!map)
+		return NULL;
+
+	obj = map->obj;
+	/*
+	 * The per-cpu chunks exist once the map is created and populated;
+	 * obj->arena_percpu.chunk is freed by arena_percpu_populate(), and
+	 * only then the map is mmaped
+	 */
+	if (map->def.type != BPF_MAP_TYPE_ARENA || !obj->arena_percpu.present ||
+	    obj->arena_percpu.chunk)
+		return NULL;
+
+	return map->mmaped + obj->arena_percpu.offset;
+}
+
 bool bpf_map__is_internal(const struct bpf_map *map)
 {
 	return map->libbpf_type != LIBBPF_MAP_UNSPEC;
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index 3e0b7bc91bef..31d3982226cc 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -1230,6 +1230,32 @@ LIBBPF_API int bpf_map__set_initial_value(struct bpf_map *map,
  */
 LIBBPF_API void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize);
 
+/**
+ * @brief **bpf_map__arena_percpu_ptr()** translates an address within
+ * cpu 0's chunk of the arena's per-cpu area (e.g. the address of an
+ * `__arena_percpu` variable as seen in the skeleton's arena_percpu
+ * section) to the per-cpu chunk of the requested @cpu.
+ * @param map the ARENA map
+ * @param ptr address of an `__arena_percpu` variable (cpu 0's chunk)
+ * @param cpu cpu id to obtain the chunk for
+ * @return pointer to the variable in the chunk of @cpu; error pointer, if the
+ * map is not a loaded ARENA map with a per-cpu area, @cpu is out of range,
+ * or @ptr is not within cpu 0's chunk
+ */
+LIBBPF_API void *bpf_map__arena_percpu_ptr(const struct bpf_map *map, void *ptr,
+					   unsigned int cpu);
+
+/**
+ * @brief **bpf_map__arena_percpu_base()** returns the base of cpu 0's
+ * chunk of the arena's per-cpu area in the user-space-visible mmap of
+ * the arena, i.e. where the skeleton's arena_percpu section points.
+ * @param map the ARENA map
+ * @return pointer to the start of cpu 0's chunk; NULL if @map is not a
+ * loaded ARENA map whose per-cpu chunks have been populated (e.g.
+ * before load, or for a map with `autocreate == false`)
+ */
+LIBBPF_API void *bpf_map__arena_percpu_base(const struct bpf_map *map);
+
 /**
  * @brief **bpf_map__is_internal()** tells the caller whether or not the
  * passed map is a special map created by libbpf automatically for things like
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index eeba4072a5da..eda89ab60543 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -458,6 +458,8 @@ LIBBPF_1.7.0 {
 
 LIBBPF_1.8.0 {
 	global:
+		bpf_map__arena_percpu_base;
+		bpf_map__arena_percpu_ptr;
 		bpf_map__attach_cgroup_opts;
 		bpf_prog_stream_open;
 		bpf_program__add_flags;
-- 
2.47.3

Re: [PATCH bpf-next 12/16] libbpf: Introduce arena percpu helpers for user space

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]

> libbpf: Introduce arena percpu helpers for user space

TL;DR: The new libbpf arena percpu symbols are exported under the
already-released LIBBPF_1.8.0 node instead of LIBBPF_1.9.0, which changes that
version's ABI and can cause undefined symbol errors at runtime.

> diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> index eeba4072a5da..eda89ab60543 100644
> --- a/tools/lib/bpf/libbpf.map
> +++ b/tools/lib/bpf/libbpf.map
> @@ -458,6 +458,8 @@ LIBBPF_1.7.0 {
>
>  LIBBPF_1.8.0 {
>  	global:
> +		bpf_map__arena_percpu_base;
> +		bpf_map__arena_percpu_ptr;
>  		bpf_map__attach_cgroup_opts;
>  		bpf_prog_stream_open;
>  		bpf_program__add_flags;

Should these two symbols be added to the LIBBPF_1.9.0 node instead of
LIBBPF_1.8.0?

Commit 3ead4ead078f ("libbpf: Start v1.9 development cycle") states that
libbpf 1.8.0 has been released, bumped LIBBPF_MINOR_VERSION to 9, and added
an empty node at the end of libbpf.map:

    LIBBPF_1.9.0 {
    } LIBBPF_1.8.0;

That node is still empty here, so exporting the new functions under
LIBBPF_1.8.0 changes the ABI of an already released version.

Binaries that use these symbols will record them as @LIBBPF_1.8.0, so
symbol-version based dependency generators (for example rpm's
libbpf.so.1(LIBBPF_1.8.0)) would accept the real libbpf 1.8.0 release, which
does not have bpf_map__arena_percpu_base or bpf_map__arena_percpu_ptr.  At
runtime the dynamic linker then fails with an undefined symbol error instead
of a clear "version LIBBPF_1.9.0 not found" message.

No later commit in this series changes libbpf.map.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 13/16] bpftool: Emit the arena percpu datasec into skeleton

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Emit the arena percpu data into skeleton.

For an object with

__u64 arena_global __arena;
__u64 __arena_percpu data1;
__u64 __arena_percpu data2;

the skeleton gets two data sections for the arena map

struct obj__arena {
	__u64 arena_global;
} *arena;

struct obj__arena_percpu {
	__u64 data1;
	__u64 data2;
} *arena_percpu;

and the load function points arena_percpu at cpu 0's chunk

static inline int
obj__load(struct obj *obj)
{
	int err;

	err = bpf_object__load_skeleton(obj->skeleton);
	if (err)
		return err;

	obj->arena_percpu = bpf_map__arena_percpu_base(obj->maps.arena);

	return 0;
}

Other cpus can be accessed through bpf_map__arena_percpu_ptr().

For an object with only

__u64 __arena_percpu counter;

the skeleton carries a single arena_percpu section

struct obj__arena_percpu {
	__u64 counter;
} *arena_percpu;

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 tools/bpf/bpftool/gen.c | 69 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 64 insertions(+), 5 deletions(-)

diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c
index ee680f7c2116..8a321ec1155a 100644
--- a/tools/bpf/bpftool/gen.c
+++ b/tools/bpf/bpftool/gen.c
@@ -132,6 +132,10 @@ static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 		snprintf(buf, buf_sz, "arena");
 		return true;
 	}
+	if (strcmp(sec_name, ".arena.percpu") == 0) {
+		snprintf(buf, buf_sz, "arena_percpu");
+		return true;
+	}
 	for  (i = 0, n = ARRAY_SIZE(pfxs); i < n; i++) {
 		const char *pfx = pfxs[i];
 
@@ -326,6 +330,14 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 		}
 	}
 
+	/* expose arena percpu  as a separate data section */
+	sec = find_type_for_map(btf, "arena_percpu");
+	if (sec) {
+		err = codegen_datasec_def(obj, btf, d, sec, obj_name);
+		if (err)
+			goto out;
+	}
+
 
 out:
 	btf_dump__free(d);
@@ -1511,11 +1523,58 @@ static int do_skeleton(int argc, char **argv)
 			return %1$s__open_opts(NULL);			    \n\
 		}							    \n\
 									    \n\
-		static inline int					    \n\
-		%1$s__load(struct %1$s *obj)				    \n\
-		{							    \n\
-			return bpf_object__load_skeleton(obj->skeleton);    \n\
-		}							    \n\
+		",
+		obj_name
+	);
+
+	/* The per-cpu arena data has no mmaped slot in the map skeleton.
+	 * Point obj->arena_percpu at cpu 0's chunk once the object is
+	 * loaded.
+	 */
+	{
+		char arena_map_ident[256] = "";
+		bool have_percpu_arena = false;
+
+		bpf_object__for_each_map(map, obj) {
+			if (bpf_map__type(map) != BPF_MAP_TYPE_ARENA)
+				continue;
+			if (find_type_for_map(bpf_object__btf(obj), "arena_percpu") &&
+			    get_map_ident(map, arena_map_ident, sizeof(arena_map_ident))) {
+				have_percpu_arena = true;
+				break;
+			}
+		}
+
+		if (have_percpu_arena) {
+			codegen("\
+				\n\
+				static inline int \n\
+				%1$s__load(struct %1$s *obj) \n\
+				{ \n\
+					int err; \n\
+					 \n\
+					err = bpf_object__load_skeleton(obj->skeleton); \n\
+					if (err) \n\
+						return err; \n\
+					obj->arena_percpu = \n\
+						bpf_map__arena_percpu_base(obj->maps.%2$s); \n\
+					return 0; \n\
+				} \n\
+				", obj_name, arena_map_ident);
+		} else {
+			codegen("\
+				\n\
+				static inline int \n\
+				%1$s__load(struct %1$s *obj) \n\
+				{ \n\
+					return bpf_object__load_skeleton(obj->skeleton); \n\
+				} \n\
+				", obj_name);
+		}
+	}
+
+	codegen("\
+		\n\
 									    \n\
 		static inline struct %1$s *				    \n\
 		%1$s__open_and_load(void)				    \n\
-- 
2.47.3

[PATCH bpf-next 14/16] selftests/bpf: Add test for arena percpu-only object

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add the __arena_percpu macro to bpf_arena_common.h and a test object
with only a arena percpu variable.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 .../bpf/libarena/include/bpf_arena_common.h   |   2 +
 .../selftests/bpf/prog_tests/arena_percpu.c   | 148 ++++++++++++++++++
 .../selftests/bpf/progs/arena_percpu_only.c   |  21 +++
 3 files changed, 171 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_percpu.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_percpu_only.c

diff --git a/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h b/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
index 82aafe879fae..7cdee141764e 100644
--- a/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
+++ b/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
@@ -35,11 +35,13 @@
 #if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && !defined(BPF_ARENA_FORCE_ASM)
 #define __arena __attribute__((address_space(1))) __attribute__((btf_type_tag("arena")))
 #define __arena_global __attribute__((address_space(1)))
+#define __arena_percpu __attribute__((address_space(1))) __attribute__((section(".arena.percpu")))
 #define cast_kern(ptr) /* nop for bpf prog. emitted by LLVM */
 #define cast_user(ptr) /* nop for bpf prog. emitted by LLVM */
 #else
 #define __arena __attribute__((btf_type_tag("arena")))
 #define __arena_global SEC(".addr_space.1")
+#define __arena_percpu SEC(".arena.percpu")
 #define cast_kern(ptr) bpf_addr_space_cast(ptr, 0, 1)
 #define cast_user(ptr) bpf_addr_space_cast(ptr, 1, 0)
 #endif
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
new file mode 100644
index 000000000000..27d4721c7ae9
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
@@ -0,0 +1,148 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Huawei Technologies Inc. */
+#include <test_progs.h>
+#include <unistd.h>
+#include <sched.h>
+#include <sys/mman.h>
+#include "bpf/libbpf_internal.h"
+#include "arena_percpu_only.skel.h"
+
+static cpu_set_t affinity_save;
+
+static int pin_to(int cpu)
+{
+	cpu_set_t cpuset;
+
+	CPU_ZERO(&cpuset);
+	CPU_SET(cpu, &cpuset);
+	return sched_setaffinity(0, sizeof(cpuset), &cpuset);
+}
+
+static int run_on_cpu(int cpu, struct bpf_program *prog, const char *name)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	int err;
+
+	if (pin_to(cpu))
+		return -1;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &opts);
+
+	if (!ASSERT_OK(err, name))
+		return -1;
+	if (!ASSERT_OK(opts.retval, name))
+		return -1;
+	return 0;
+}
+
+static int save_affinity(void)
+{
+	if (!ASSERT_OK(sched_getaffinity(0, sizeof(affinity_save), &affinity_save),
+		       "save_affinity"))
+		return -1;
+	return 0;
+}
+
+static void restore_affinity(void)
+{
+	sched_setaffinity(0, sizeof(affinity_save), &affinity_save);
+}
+
+static int parse_possible_cpus(bool **possible, int *nr_possible)
+{
+	if (!ASSERT_OK(parse_cpu_mask_file("/sys/devices/system/cpu/possible",
+					   possible, nr_possible),
+		       "parse_possible"))
+		return -1;
+	return 0;
+}
+
+/* check one cpu's copy of a per-cpu global */
+static void check_percpu_value(struct bpf_map *arena, void *var, int cpu,
+			       __u64 expected, const char *name)
+{
+	__u64 *p = bpf_map__arena_percpu_ptr(arena, var, cpu);
+
+	if (!ASSERT_OK_PTR(p, name))
+		return;
+	ASSERT_EQ(*p, expected, name);
+}
+
+static void test_arena_percpu_only(void)
+{
+	struct arena_percpu_only *skel;
+	struct bpf_map *arena;
+	bool *possible = NULL;
+	int possible_cpus, cpu, picked_cpu = -1, i, err;
+
+	if (save_affinity())
+		return;
+
+	skel = arena_percpu_only__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (parse_possible_cpus(&possible, &possible_cpus))
+		goto out;
+
+	/* size the arena for the per-cpu chunks and load the object */
+	if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.arena, possible_cpus),
+		       "set_max_entries"))
+		goto out;
+	if (!ASSERT_OK(arena_percpu_only__load(skel), "load"))
+		goto out;
+	arena = skel->maps.arena;
+
+	if (!ASSERT_OK_PTR(skel->arena_percpu, "arena_percpu"))
+		goto out;
+
+	/* every chunk starts as a copy of the initialized section */
+	for (cpu = 0; cpu < possible_cpus; cpu++) {
+		if (!possible[cpu])
+			continue;
+		check_percpu_value(arena, &skel->arena_percpu->counter, cpu,
+				   0x123456, "counter_init");
+	}
+
+	/* pick one cpu to pin */
+	for (i = 0; i < CPU_SETSIZE; i++) {
+		if (CPU_ISSET(i, &affinity_save)) {
+			picked_cpu = i;
+			break;
+		}
+	}
+	if (!ASSERT_GE(picked_cpu, 0, "pick_cpu"))
+		goto out;
+
+	err = run_on_cpu(picked_cpu, skel->progs.arena_percpu_only_write, "run on cpu");
+	if (err < 0)
+		goto out;
+
+	/* only the cpu the program ran on got written */
+	for (cpu = 0; cpu < possible_cpus; cpu++) {
+		if (!possible[cpu])
+			continue;
+
+		if (cpu == picked_cpu)
+			check_percpu_value(arena, &skel->arena_percpu->counter,
+					   cpu, picked_cpu, "counter_written");
+		else
+			check_percpu_value(arena, &skel->arena_percpu->counter,
+					   cpu, 0x123456, "counter_untouched");
+	}
+
+out:
+	free(possible);
+	arena_percpu_only__destroy(skel);
+	restore_affinity();
+}
+
+void test_arena_percpu(void)
+{
+	if (!feat_supported(NULL, FEAT_ARENA_PERCPU_DATA)) {
+		test__skip();
+		return;
+	}
+	if (test__start_subtest("percpu_only"))
+		test_arena_percpu_only();
+}
diff --git a/tools/testing/selftests/bpf/progs/arena_percpu_only.c b/tools/testing/selftests/bpf/progs/arena_percpu_only.c
new file mode 100644
index 000000000000..35e4ff983e65
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_percpu_only.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Huawei Technologies Inc. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_arena_common.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+} arena SEC(".maps");
+
+__u64 __arena_percpu counter = 0x123456;
+
+SEC("syscall")
+int arena_percpu_only_write(void *ctx)
+{
+	counter = bpf_get_smp_processor_id();
+	return 0;
+}
+
+char LICENSE[] SEC("license") = "GPL";
-- 
2.47.3

Re: [PATCH bpf-next 14/16] selftests/bpf: Add test for arena percpu-only object

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5248 bytes --]

> selftests/bpf: Add test for arena percpu-only object

TL;DR:
- arena_percpu_only.c lacks an addr_space_cast gate, so in the fallback config
  the store may hit a user address and oops the kernel
- test skips silently without a reason (minor)

> diff --git a/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h b/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
> index 82aafe879fae..7cdee141764e 100644
> --- a/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
> +++ b/tools/testing/selftests/bpf/libarena/include/bpf_arena_common.h
> @@ -35,11 +35,13 @@
>  #if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && !defined(BPF_ARENA_FORCE_ASM)

[ ... ]

>  #else
>  #define __arena __attribute__((btf_type_tag("arena")))
>  #define __arena_global SEC(".addr_space.1")
> +#define __arena_percpu SEC(".arena.percpu")
>  #define cast_kern(ptr) bpf_addr_space_cast(ptr, 0, 1)
>  #define cast_user(ptr) bpf_addr_space_cast(ptr, 1, 0)
>  #endif

In this fallback branch (clang without __BPF_FEATURE_ADDR_SPACE_CAST, or
with BPF_ARENA_FORCE_ASM), __arena_percpu only selects the section, so no
addr_space_cast is emitted for accesses to the variable.  Can the access in
arena_percpu_only.c work in this configuration?

    __u64 __arena_percpu counter = 0x123456;
    ...
    counter = bpf_get_smp_processor_id();

This compiles to roughly:

    r1 = counter ll
    *(u64 *)(r1 + 0) = r0

with no cast_kern() between the two instructions.  libbpf turns the ld_imm64
into BPF_PSEUDO_ARENA_PERCPU, and resolve_pseudo_ldimm64() fills in the full
64-bit value from arena_map_direct_value_addr(), i.e. user_vm_start + off.
check_ld_imm() still types the register as PTR_TO_ARENA:

    __mark_reg_known(dst_reg, pos);
    dst_reg->type = PTR_TO_ARENA;
    dst_reg->mem_size = bpf_arena_percpu_chunk_sz(map);

so the verifier accepts the store, and the fixups turn it into
BPF_PROBE_MEM32.  The x86 jit emits PROBE_MEM32 as [reg + r12 + off] using
the whole 64-bit register, while the arena layout in kernel/bpf/arena.c
assumes that register holds only the lower 32 bits of the user pointer.

Using the example values from the comment in arena.c, user_vm_start
0x7f7d26200000 and kern_vm_start 0xffffc90001e69000, the store targets
0x0000487d28069000.  That is a user-space address and not in the arena.

On x86 with supervisor-mode access prevention enabled, a supervisor access
to a user address goes through do_user_addr_fault() -> page_fault_oops()
without consulting the bpf extable, so running this test would oops the
kernel.  Without it, the store lands on whatever the current process has
mapped at that address, and other user_vm_start values can make the sum land
in kernel space.

Every other arena selftest that touches arena globals directly
(arena_atomics.c, linked_arena1.c, data_in_arena_decl.c, arena_htab.c, ...)
gates that code on __BPF_FEATURE_ADDR_SPACE_CAST and skips otherwise.  In
the same fallback branch, __arena_global variables need an explicit
cast_kern().  arena_percpu_only.c has no such gate.  Should it have one?

The underlying problem appears to be on the kernel side of this series.
Commit 2e94f202e80e ("bpf: Teach verifier about arena percpu instructions")
types an uncast 64-bit user address as PTR_TO_ARENA, and the jit commits
a27c971d206b ("bpf, x86: Add support for arena percpu") and 0c3a9efd447d
("bpf, arm64: Add support for arena percpu") do not truncate it.

Commit 08224ba2de19 ("selftests/bpf: Add test for arena percpu bounds")
loads exactly this uncast ld_imm64 + ldx pattern but never runs it.  No fix
for this was found later in the series (searched e06485c3c3fb..08224ba2de19).

> diff --git a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
> new file mode 100644
> index 000000000000..27d4721c7ae9
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
> @@ -0,0 +1,148 @@

[ ... ]

> +void test_arena_percpu(void)
> +{
> +	if (!feat_supported(NULL, FEAT_ARENA_PERCPU_DATA)) {
> +		test__skip();
> +		return;
> +	}
> +	if (test__start_subtest("percpu_only"))
> +		test_arena_percpu_only();
> +}

A subsystem pattern flags this as potentially concerning: a test that skips
silently, with no message.  When the kernel lacks arena percpu support,
test__skip() is called without printing a reason, and feat_supported() is
also silent when the probe returns 0.  The output only shows the test as
skipped and never says that FEAT_ARENA_PERCPU_DATA (kernel, jit, or arena
percpu chunk support) was missing.

test__skip() does mark the test as skipped in the test_progs output, and
many bpf selftests call it without a message, so this may be acceptable as
is.  The skip condition itself looks correct, since the probe loads a
BPF_PSEUDO_ARENA_PERCPU program and therefore also covers jits without
bpf_jit_supports_arena_percpu_insn().

Would it be worth printing a short reason before test__skip(), for example
that arena percpu data is not supported?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

[PATCH bpf-next 15/16] selftests/bpf: Add test for mixed arena percpu object

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add a test object that mixes percpu variables and a regular global in
one arena.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 .../selftests/bpf/prog_tests/arena_percpu.c   | 96 +++++++++++++++++++
 .../selftests/bpf/progs/arena_percpu_mixed.c  | 29 ++++++
 2 files changed, 125 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/arena_percpu_mixed.c

diff --git a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
index 27d4721c7ae9..b450be5f2b36 100644
--- a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
+++ b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
@@ -6,6 +6,7 @@
 #include <sys/mman.h>
 #include "bpf/libbpf_internal.h"
 #include "arena_percpu_only.skel.h"
+#include "arena_percpu_mixed.skel.h"
 
 static cpu_set_t affinity_save;
 
@@ -137,6 +138,99 @@ static void test_arena_percpu_only(void)
 	restore_affinity();
 }
 
+static void test_arena_percpu_mixed(void)
+{
+	struct arena_percpu_mixed *skel;
+	struct bpf_map *arena, *bss_map;
+	bool *possible = NULL, *online = NULL, *written = NULL;
+	int possible_cpus, online_cpus, nr_written = 0, cpu, err;
+
+	if (save_affinity())
+		return;
+
+	skel = arena_percpu_mixed__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (parse_possible_cpus(&possible, &possible_cpus))
+		goto out;
+
+	/* size the arena for the per-cpu chunks and load the object */
+	if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.arena, possible_cpus + 16),
+		       "set_max_entries"))
+		goto out;
+	if (!ASSERT_OK(arena_percpu_mixed__load(skel), "load"))
+		goto out;
+	arena = skel->maps.arena;
+
+	for (cpu = 0; cpu < possible_cpus; cpu++) {
+		if (!possible[cpu])
+			continue;
+
+		check_percpu_value(arena, &skel->arena_percpu->data1, cpu,
+				   0x123456789, "data1_init");
+		check_percpu_value(arena, &skel->arena_percpu->data2, cpu,
+				   0x987654321, "data2_init");
+	}
+
+	if (!ASSERT_OK(parse_cpu_mask_file("/sys/devices/system/cpu/online",
+					   &online, &online_cpus),
+		       "parse_online"))
+		goto out;
+
+	written = calloc(online_cpus, sizeof(*written));
+	if (!ASSERT_OK_PTR(written, "written_alloc"))
+		goto out;
+
+	for (cpu = 0; cpu < online_cpus; cpu++) {
+		if (!online[cpu] || cpu >= CPU_SETSIZE || !CPU_ISSET(cpu, &affinity_save))
+			continue;
+
+		err = run_on_cpu(cpu, skel->progs.arena_percpu_mixed_write, "write");
+		if (err < 0)
+			goto out;
+		if (err > 0)
+			continue;
+		written[cpu] = true;
+		nr_written++;
+	}
+	if (!ASSERT_GE(nr_written, 1, "cpus_exercised"))
+		goto out;
+
+	/* check every written chunk from user space */
+	for (cpu = 0; cpu < online_cpus; cpu++) {
+		if (!written[cpu])
+			continue;
+
+		check_percpu_value(arena, &skel->arena_percpu->data1, cpu,
+				   cpu, "data1_written");
+		check_percpu_value(arena, &skel->arena_percpu->data2, cpu,
+				   cpu << 2, "data2_written");
+	}
+
+	/* the __arena global should not be touched */
+	ASSERT_EQ(skel->arena->arena_global, 0x5a5ba5a5c0dec0de, "arena_global_intact");
+
+	/* api error paths: cpu out of range, pointer outside cpu 0's chunk */
+	ASSERT_ERR_PTR(bpf_map__arena_percpu_ptr(arena, &skel->arena_percpu->data1,
+						 possible_cpus), "api_cpu_range");
+	ASSERT_ERR_PTR(bpf_map__arena_percpu_ptr(arena, possible, 0),
+		       "api_not_in_chunk");
+
+	/* a non-ARENA map has no per-cpu chunks to index */
+	bss_map = bpf_object__find_map_by_name(skel->obj, ".bss");
+	if (ASSERT_OK_PTR(bss_map, "bss_map"))
+		ASSERT_EQ(libbpf_get_error(bpf_map__arena_percpu_ptr(bss_map, &cpu, 0)),
+			  -EINVAL, "api_not_arena");
+
+out:
+	free(written);
+	free(online);
+	free(possible);
+	arena_percpu_mixed__destroy(skel);
+	restore_affinity();
+}
+
 void test_arena_percpu(void)
 {
 	if (!feat_supported(NULL, FEAT_ARENA_PERCPU_DATA)) {
@@ -145,4 +239,6 @@ void test_arena_percpu(void)
 	}
 	if (test__start_subtest("percpu_only"))
 		test_arena_percpu_only();
+	if (test__start_subtest("percpu_mixed"))
+		test_arena_percpu_mixed();
 }
diff --git a/tools/testing/selftests/bpf/progs/arena_percpu_mixed.c b/tools/testing/selftests/bpf/progs/arena_percpu_mixed.c
new file mode 100644
index 000000000000..9bc199e86031
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_percpu_mixed.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Huawei Technologies Inc. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_arena_common.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+} arena SEC(".maps");
+
+__u64 __arena_percpu data1 = 0x123456789;
+__u64 __arena_percpu data2 = 0x987654321;
+
+__u64 __arena_global arena_global = 0x5a5ba5a5c0dec0de;
+
+int bss_var;
+
+SEC("syscall")
+int arena_percpu_mixed_write(void *ctx)
+{
+	u32 cpu = bpf_get_smp_processor_id();
+
+	data1 = cpu;
+	data2 = cpu << 2;
+	return 0;
+}
+
+char LICENSE[] SEC("license") = "GPL";
-- 
2.47.3

[PATCH bpf-next 16/16] selftests/bpf: Add test for arena percpu bounds

[Xu Kuohai]

From: Xu Kuohai <xukuohai@huawei.com>

Add a raw insn test. It loads a hand built arena percpu pseudo
instruction against an arena map with a one page chunk to test in
range accesses and out of range failures.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 .../selftests/bpf/prog_tests/arena_percpu.c   | 117 ++++++++++++++++++
 1 file changed, 117 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
index b450be5f2b36..1f9a22cc9730 100644
--- a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
+++ b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
@@ -8,6 +8,7 @@
 #include "arena_percpu_only.skel.h"
 #include "arena_percpu_mixed.skel.h"
 
+static char log_buf[16384];
 static cpu_set_t affinity_save;
 
 static int pin_to(int cpu)
@@ -231,6 +232,120 @@ static void test_arena_percpu_mixed(void)
 	restore_affinity();
 }
 
+/* check out of range pseudo instructions and accesses */
+static void test_arena_percpu_percpu_bounds(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_IMM64_RAW(BPF_REG_1, BPF_PSEUDO_ARENA_PERCPU, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, 0),
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts);
+	bool *possible = NULL;
+	__u32 page_sz = getpagesize();
+	__u32 bad_offs[2];
+	char expected[128];
+	void *arena = MAP_FAILED;
+	int possible_cpus, nr_pages, map_fd, prog_fd, i;
+
+	if (parse_possible_cpus(&possible, &possible_cpus))
+		return;
+
+	/* a one-page per-cpu chunk at page 1 of the arena */
+	map_opts.map_flags = BPF_F_MMAPABLE;
+	map_opts.arena_percpu_chunk_off = page_sz;
+	map_opts.arena_percpu_chunk_sz = page_sz;
+
+	prog_opts.log_buf = log_buf;
+	prog_opts.log_size = sizeof(log_buf);
+	prog_opts.log_level = 1;
+
+	/* one page per possible cpu plus one for the chunk offset */
+	nr_pages = possible_cpus + 1;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_ARENA, "arena_percpu_ref",
+				0, 0, nr_pages, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		goto out;
+
+	arena = mmap(NULL, nr_pages * page_sz, PROT_READ | PROT_WRITE,
+		     MAP_SHARED, map_fd, 0);
+	if (!ASSERT_NEQ(arena, MAP_FAILED, "arena_mmap"))
+		goto out;
+
+	insns[0].imm = map_fd;
+
+	/* a reference at the start of the chunk loads */
+	insns[1].imm = page_sz;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
+				"arena_percpu_ref", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_GE(prog_fd, 0, "in_range_load"))
+		goto out;
+	close(prog_fd);
+
+	/* the first u64 below the chunk and one past its end are rejected */
+	bad_offs[0] = page_sz - 8;
+	bad_offs[1] = page_sz + page_sz;
+
+	for (i = 0; i < ARRAY_SIZE(bad_offs); i++) {
+		insns[1].imm = bad_offs[i];
+		prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
+					"arena_percpu_ref", "GPL", insns,
+					ARRAY_SIZE(insns), &prog_opts);
+		if (!ASSERT_LT(prog_fd, 0, "out_of_range_load"))
+			close(prog_fd);
+		snprintf(expected, sizeof(expected),
+			 "per-cpu arena reference at offset %u "
+			 "outside cpu 0's chunk", bad_offs[i]);
+		ASSERT_HAS_SUBSTR(log_buf, expected, "out_of_range_log");
+	}
+
+	/* the last u64 of the chunk is in range and loads */
+	insns[1].imm = page_sz + page_sz - 8;
+	insns[2].off = 0;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
+				"arena_percpu_ref", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_GE(prog_fd, 0, "last_u64_load"))
+		goto out;
+	close(prog_fd);
+
+	/* one u64 past the chunk end is rejected */
+	insns[2].off = 8;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
+				"arena_percpu_ref", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "out_of_range_access"))
+		close(prog_fd);
+	snprintf(expected, sizeof(expected),
+		 "per-cpu arena access at offset %d size 8 "
+		 "outside cpu 0's chunk", (int)page_sz);
+	ASSERT_HAS_SUBSTR(log_buf, expected, "out_of_range_access_log");
+
+	/* an access below the chunk start is rejected */
+	insns[1].imm = page_sz;
+	insns[2].off = -8;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
+				"arena_percpu_ref", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "below_chunk_access"))
+		close(prog_fd);
+	snprintf(expected, sizeof(expected),
+		 "per-cpu arena access at offset -8 size 8 "
+		 "outside cpu 0's chunk");
+	ASSERT_HAS_SUBSTR(log_buf, expected, "below_chunk_access_log");
+
+out:
+	if (arena != MAP_FAILED)
+		munmap(arena, nr_pages * page_sz);
+	if (map_fd >= 0)
+		close(map_fd);
+	free(possible);
+}
+
 void test_arena_percpu(void)
 {
 	if (!feat_supported(NULL, FEAT_ARENA_PERCPU_DATA)) {
@@ -241,4 +356,6 @@ void test_arena_percpu(void)
 		test_arena_percpu_only();
 	if (test__start_subtest("percpu_mixed"))
 		test_arena_percpu_mixed();
+	if (test__start_subtest("percpu_bounds"))
+		test_arena_percpu_percpu_bounds();
 }
-- 
2.47.3

Re: [PATCH bpf-next 16/16] selftests/bpf: Add test for arena percpu bounds

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3811 bytes --]

> selftests/bpf: Add test for arena percpu bounds

TL;DR: Nits only in the arena percpu bounds selftest: a redundant insns[2].off
= 0 assignment, and the repeated bpf_prog_load and log-check code could share
a helper.

> diff --git a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
> --- a/tools/testing/selftests/bpf/prog_tests/arena_percpu.c
> +++ b/tools/testing/selftests/bpf/prog_tests/arena_percpu.c

[ ... ]

> +/* check out of range pseudo instructions and accesses */
> +static void test_arena_percpu_percpu_bounds(void)
> +{

[ ... ]

> +	/* a reference at the start of the chunk loads */
> +	insns[1].imm = page_sz;
> +	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
> +				"arena_percpu_ref", "GPL", insns,
> +				ARRAY_SIZE(insns), &prog_opts);
> +	if (!ASSERT_GE(prog_fd, 0, "in_range_load"))
> +		goto out;
> +	close(prog_fd);
> +
> +	/* the first u64 below the chunk and one past its end are rejected */
> +	bad_offs[0] = page_sz - 8;
> +	bad_offs[1] = page_sz + page_sz;
> +
> +	for (i = 0; i < ARRAY_SIZE(bad_offs); i++) {
> +		insns[1].imm = bad_offs[i];
> +		prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
> +					"arena_percpu_ref", "GPL", insns,
> +					ARRAY_SIZE(insns), &prog_opts);
> +		if (!ASSERT_LT(prog_fd, 0, "out_of_range_load"))
> +			close(prog_fd);
> +		snprintf(expected, sizeof(expected),
> +			 "per-cpu arena reference at offset %u "
> +			 "outside cpu 0's chunk", bad_offs[i]);
> +		ASSERT_HAS_SUBSTR(log_buf, expected, "out_of_range_log");
> +	}
> +
> +	/* the last u64 of the chunk is in range and loads */
> +	insns[1].imm = page_sz + page_sz - 8;
> +	insns[2].off = 0;

This isn't a bug, but is this insns[2].off = 0 assignment needed?  The field
is still 0 from the initializer at this point.

> +	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
> +				"arena_percpu_ref", "GPL", insns,
> +				ARRAY_SIZE(insns), &prog_opts);
> +	if (!ASSERT_GE(prog_fd, 0, "last_u64_load"))
> +		goto out;
> +	close(prog_fd);
> +
> +	/* one u64 past the chunk end is rejected */
> +	insns[2].off = 8;
> +	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
> +				"arena_percpu_ref", "GPL", insns,
> +				ARRAY_SIZE(insns), &prog_opts);
> +	if (!ASSERT_LT(prog_fd, 0, "out_of_range_access"))
> +		close(prog_fd);
> +	snprintf(expected, sizeof(expected),
> +		 "per-cpu arena access at offset %d size 8 "
> +		 "outside cpu 0's chunk", (int)page_sz);
> +	ASSERT_HAS_SUBSTR(log_buf, expected, "out_of_range_access_log");
> +
> +	/* an access below the chunk start is rejected */
> +	insns[1].imm = page_sz;
> +	insns[2].off = -8;
> +	prog_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT,
> +				"arena_percpu_ref", "GPL", insns,
> +				ARRAY_SIZE(insns), &prog_opts);
> +	if (!ASSERT_LT(prog_fd, 0, "below_chunk_access"))
> +		close(prog_fd);
> +	snprintf(expected, sizeof(expected),
> +		 "per-cpu arena access at offset -8 size 8 "
> +		 "outside cpu 0's chunk");
> +	ASSERT_HAS_SUBSTR(log_buf, expected, "below_chunk_access_log");

This isn't a bug, but could the five identical bpf_prog_load() calls and the
repeated reject/log checks share a small helper?

Only insns[1].imm and insns[2].off differ between the cases, and the
ASSERT_LT + close, snprintf into expected, ASSERT_HAS_SUBSTR sequence is
repeated three times.

A helper taking the reference offset and the access offset, and returning
the prog fd or checking the expected log, would make each case a single
line, like run_on_cpu() and check_percpu_value() do elsewhere in this file.

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39434884362

Re: [PATCH bpf-next 00/16] Add percpu variable support for bpf arena

[Alexei Starovoitov]

On Tue, Oct 20, 2026 at 06:22 AM Xu Kuohai <xukuohai@huaweicloud.com> wrote:
> To use percpu data in a bpf arena today, we need to define an array
> sized to the maximum possible cpu number, like the cpu_ctxs array
> defined in tools/sched_ext/scx_qmap.h. This is not a convenient
> pattern. It lacks performance and scalability, and wastes memory.

It depends on how you use it.
Patches 10 and 11 emit the same thing as
&arr[bpf_get_smp_processor_id()]: load the cpu id, multiply, add.
The x86 verifier and the arm64 JIT inline that helper already.

__u64 hits takes a page per possible cpu with this series and 8 bytes
per cpu in an array.

cpu_ctxs[] is indexed by cid and qmap_dump_cid() reads other cpus'
slots from bpf, so scx_qmap cannot use this.

Keep it in the bpf prog.

Any arena offset is safe to access, so the kernel doesn't need to know
about percpu chunks. bpf_arena_alloc.h does it already with
page_frag_cur_page[NR_CPUS]: a table of pointers and a page that the
cpu allocates on the first use. NUMA_NO_NODE gives the local node and
user space reads the table through its mmap.
Then no need for BPF_PSEUDO_ARENA_PERCPU, the map_create attrs, verifier
and JIT changes, new libbpf APIs and libbpf changing thread affinity
during load. What is lost is plain 'hits++', but kernel C doesn't have
that either. Initial values are left to the prog below.

Something like this:

+/*
+ * One copy of 'type' per cpu. A cpu allocates its copy on the first use,
+ * so it comes from the local NUMA node and cpus that never run the prog
+ * cost nothing. User space reads any cpu's copy through name[cpu].
+ */
+#define DEFINE_ARENA_PER_CPU(type, name)				\
+type __arena * __arena name[NR_CPUS];					\
+									\
+static inline type __arena *this_cpu_##name(void)			\
+{									\
+	__u32 cpu = bpf_get_smp_processor_id();				\
+	type __arena *p = name[cpu];					\
+									\
+	if (!p) {							\
+		p = bpf_arena_alloc_pages(&arena, NULL,			\
+					  round_up(sizeof(type), PAGE_SIZE) / PAGE_SIZE, \
+					  NUMA_NO_NODE, 0);		\
+		name[cpu] = p;						\
+	}								\
+	cast_kern(p);							\
+	return p;							\
+}

Orthogonal to that people are working on slab-quality allocator
for arena. It's also going to be done completely on bpf side.

pw-bot: cr