mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Samuel Wu <wusamuel@google.com>
To: Andrii Nakryiko <andrii@kernel.org>,
	Eduard Zingerman <eddyz87@gmail.com>,
	 Ihor Solodrai <ihor.solodrai@linux.dev>,
	Alexei Starovoitov <ast@kernel.org>,
	 Daniel Borkmann <daniel@iogearbox.net>,
	Kumar Kartikeya Dwivedi <memxor@gmail.com>,
	 Martin KaFai Lau <martin.lau@linux.dev>,
	Song Liu <song@kernel.org>,
	 Yonghong Song <yonghong.song@linux.dev>,
	Jiri Olsa <jolsa@kernel.org>,
	 Emil Tsalapatis <emil@etsalapatis.com>,
	David Vernet <void@manifault.com>
Cc: Samuel Wu <wusamuel@google.com>,
	kernel-team@android.com,  Andrii Nakryiko <andriin@fb.com>,
	bpf@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH bpf-next] libbpf: Fix ring buffers for 32-bit userspace on 64-bit kernels
Date: Thu, 19 Nov 2026 17:36:42 -0800	[thread overview]
Message-ID: <20261120013642.1093852-2-wusamuel@google.com> (raw)

The kernel keeps ring buffer positions in unsigned long, so a 32-bit
little-endian process on a 64-bit kernel accesses only the low 32 bits.
Reading them is fine, as positions are never more than one ring size
apart, but the upper 32 bits are always zero when writing. After 4GB:
- RINGBUF: the kernel sees consumer_pos 4GB behind, so
  bpf_ringbuf_reserve() always fails and epoll consumers spin.
- USER_RINGBUF: the kernel sees producer_pos behind consumer_pos, so
  bpf_user_ringbuf_drain() stops returning samples.

In a 32-bit little-endian process, write the position as a 64-bit value
that carries into the upper half. On a 32-bit kernel, the upper 4 bytes
land in user-writable memory that the kernel does not use for that map
type: consumer-page padding (RINGBUF), or padding/pending_pos
(USER_RINGBUF). Big-endian 32-bit compat is not fixed because offset 0
maps the upper 32 bits, so even reads do not work without knowing the
kernel word size.

Tested transferring >4GB over BPF_MAP_TYPE_RINGBUF with 32-bit
userspace (armeabi-v7a) on a 64-bit kernel (aarch64), and verified no
regression on 32-bit i386.

This needs commit fdd4fad0bbbd ("libbpf: Fix ring buffer consumer loop
on 32-bit position wrap"): without it, the consumer stops when the low
32 bits wrap.

Fixes: bf99c936f947 ("libbpf: Add BPF ring buffer support")
Fixes: b66ccae01f1d ("bpf: Add libbpf logic for user-space ring buffer")
Signed-off-by: Samuel Wu <wusamuel@google.com>
---
 tools/lib/bpf/ringbuf.c | 38 +++++++++++++++++++++++++++++++++-----
 1 file changed, 33 insertions(+), 5 deletions(-)

diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c
index 78fcb3df243f..254e19ec51f2 100644
--- a/tools/lib/bpf/ringbuf.c
+++ b/tools/lib/bpf/ringbuf.c
@@ -58,6 +58,35 @@ struct ringbuf_hdr {
 	__u32 pad;
 };
 
+/* 8-byte alignment lets compiler inline 64-bit __atomic_store_n() on i386. */
+typedef __u64 __attribute__((aligned(8), may_alias)) ringbuf_pos64_t;
+
+/*
+ * Carry 32-bit little-endian position updates into the upper 32 bits for
+ * 64-bit kernels. On 32-bit kernels the upper half lands in unused padding
+ * (RINGBUF) or unused pending_pos (USER_RINGBUF).
+ */
+static void ringbuf_store_pos(unsigned long *pos, unsigned long val)
+{
+	ringbuf_pos64_t *p = (void *)pos;
+	__u64 pos64;
+
+	if (sizeof(long) == 8 || __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) {
+		/* Pairs with smp_load_acquire() in the kernel. */
+		smp_store_release(pos, val);
+		return;
+	}
+
+	pos64 = READ_ONCE(*p);
+	pos64 += (__u32)(val - pos64);
+#if __GCC_ATOMIC_LLONG_LOCK_FREE == 2
+	__atomic_store_n(p, pos64, __ATOMIC_RELEASE);
+#else
+	/* Non-atomic store, but it only transiently fails reserve/peek. */
+	smp_store_release(p, pos64);
+#endif
+}
+
 static void ringbuf_free_ring(struct ring_buffer *rb, struct ring *r)
 {
 	if (r->consumer_pos) {
@@ -290,14 +319,13 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
 				err = r->sample_cb(r->ctx, sample, len);
 				if (err < 0) {
 					/* update consumer pos and bail out */
-					smp_store_release(r->consumer_pos,
-							  cons_pos);
+					ringbuf_store_pos(r->consumer_pos, cons_pos);
 					return err;
 				}
 				cnt++;
 			}
 
-			smp_store_release(r->consumer_pos, cons_pos);
+			ringbuf_store_pos(r->consumer_pos, cons_pos);
 
 			if (cnt >= n)
 				goto done;
@@ -412,7 +440,7 @@ struct ring *ring_buffer__ring(struct ring_buffer *rb, unsigned int idx)
 
 unsigned long ring__consumer_pos(const struct ring *r)
 {
-	/* Synchronizes with smp_store_release() in ringbuf_process_ring(). */
+	/* Synchronizes with ringbuf_store_pos() in ringbuf_process_ring(). */
 	return smp_load_acquire(r->consumer_pos);
 }
 
@@ -652,7 +680,7 @@ void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size)
 	/* Synchronizes with smp_load_acquire() in __bpf_user_ringbuf_peek() in
 	 * the kernel.
 	 */
-	smp_store_release(rb->producer_pos, prod_pos + total_size);
+	ringbuf_store_pos(rb->producer_pos, prod_pos + total_size);
 
 	return (void *)rb->data + ((prod_pos + BPF_RINGBUF_HDR_SZ) & rb->mask);
 }
-- 
2.56.1.809.ga0b61035b5-goog


                 reply	other threads:[~2026-11-20  1:36 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120013642.1093852-2-wusamuel@google.com \
    --to=wusamuel@google.com \
    --cc=andrii@kernel.org \
    --cc=andriin@fb.com \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=eddyz87@gmail.com \
    --cc=emil@etsalapatis.com \
    --cc=ihor.solodrai@linux.dev \
    --cc=jolsa@kernel.org \
    --cc=kernel-team@android.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=martin.lau@linux.dev \
    --cc=memxor@gmail.com \
    --cc=song@kernel.org \
    --cc=void@manifault.com \
    --cc=yonghong.song@linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®