[PATCH] ring-buffer: Do not allow reader to see unfinished moved head

[Steven Rostedt <rostedt@goodmis.org>]

From: Steven Rostedt <rostedt@goodmis.org>

The reader of the ring buffer swaps out a blank sub-buffer with the "head"
of the writable ring buffer so that it has sole access to the sub-buffer
and doesn't race with new writes. To do this, when the writer fills the
ring buffer and needs to overwrite the oldest sub-buffer, it has to mvoe
the HEAD of the buffer that the reader will take next.

While it is moving the head, the reader is prevented from doing the swap.
This is accomplished by the writer changing the HEAD flag of the head page
with the UPDATE flag. It then sets the next sub-buffer to have HEAD flag
set as well.

The cpu_buffer->head_page points to the last place the reader saw the head
page. When it needs to acquire a new page from the ring buffer, it will
start at the cpu_buffer->head_page and search for the sub-buffer with
either the HEAD or UPDATE flag set. It will only do a swap if the
sub-buffer has the HEAD flag set, otherwise it waits for the UPDATE flag
to clear to find the new head page. The problem is that this
cpu_buffer->head_page can be stale, and if an interrupt happens while an
event is being written, and the interrupt writes a lot of events, it can
move the head page several sub-buffers from the original head page. If the
cpu_buffer->head_page ended up between the original head page that has the
UPDATE flag set and the new head page with the HEAD flag set, it can allow
the reader to see the new head page and swap that one out while a commit
is still in progress. This can corrupt the ring buffer.

Have the writer update the cpu_buffer->head_page to keep it pointing to
the old head page before it does the update to make sure that the reader
will always start its search outside the UPDATE and new HEAD sub-buffers.

Cc: stable@vger.kernel.org
Fixes: 77ae365eca89 ("ring-buffer: make lockless")
Closes: https://lore.kernel.org/all/20261105-ring-buffer-head-moved-v2-1-05da40cebccc@debian.org/
Reported-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/ring_buffer.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index c64e001c3fc0..0d37f8b529d5 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -1494,7 +1494,13 @@ rb_set_head_page(struct ring_buffer_per_cpu *cpu_buffer)
 	if (RB_WARN_ON(cpu_buffer, rb_list_head(list->prev->next) != list))
 		return NULL;
 
-	page = head = cpu_buffer->head_page;
+	/*
+	 * Make sure that the start of the search of the head_page does not
+	 * start between the writer adding the UPDATE and the new HEAD that it
+	 * moves forward.
+	 */
+	/* Sync with the writer moving the HEAD in rb_handle_head_page() */
+	page = head = smp_load_acquire(&cpu_buffer->head_page);
 	/*
 	 * It is possible that the writer moves the header behind
 	 * where we started, and we miss in one loop.
@@ -3721,6 +3727,15 @@ rb_handle_head_page(struct ring_buffer_per_cpu *cpu_buffer,
 		local_sub(rb_page_commit(next_page), &cpu_buffer->entries_bytes);
 		local_inc(&cpu_buffer->pages_lost);
 
+		/*
+		 * To prevent a stale head_page from being between
+		 * the UPDATE and new HEAD pages, set it here to have
+		 * the reader check always find the update before
+		 * finding the HEAD page.
+		 */
+		/* Sync with the reader setting the head page */
+		smp_store_release(&cpu_buffer->head_page, next_page);
+
 		if (cpu_buffer->ring_meta)
 			rb_update_meta_head(cpu_buffer, next_page);
 		/*
-- 
2.53.0