[BUG] tracing: Too many tries to read user space

[BUG] tracing: Too many tries to read user space

[Jeongho Choi]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hello,

We are seeing a reproducible kernel panic related to the tracing code
when it fails to read user-space memory.

The issue was originally reported through the Android/Google Issue
Tracker, and we were advised to report it to the upstream trace mailing
list because the affected code is upstream.

Environment:

Architecture: arm64
Kernel: Linux 6.18.21
Base: Android Common Kernel (android17-6.18)
Affected area: kernel/trace/

The relevant error/panic log is:
[48916.569148] [9:           lmkd:  536] Error: Too many tries to read
user space
[48916.569156] [9:           lmkd:  536] WARNING: CPU: 9 PID: 536 at
kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
[48916.569443] [9:           lmkd:  536] CPU: 9 UID: 1069 PID: 536 Comm:
lmkd Tainted: G           OE       6.18.21-android17-5-ga1a8e8cab9ec-4k
#1 PREEMPT  25372cd4750dcac3c0fe86b57d47c665f97a6046

[48916.569450] [9:           lmkd:  536] pstate: 63402005 (nZCv daif
+PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[48916.569452] [9:           lmkd:  536] pc :
trace_user_fault_read+0x334/0x360
[48916.569454] [9:           lmkd:  536] lr :
trace_user_fault_read+0x330/0x360
[48916.569456] [9:           lmkd:  536] sp : ffffffc096993cc0
[48916.569457] [9:           lmkd:  536] x29: ffffffc096993cd0 x28:
0000000000000065 x27: ffffff8817e45200
[48916.569461] [9:           lmkd:  536] x26: 0000000000000000 x25:
000000011616c082 x24: 0000000000000000
[48916.569463] [9:           lmkd:  536] x23: 0000000000000009 x22:
000000000000002b x21: ffffff880597f740
[48916.569466] [9:           lmkd:  536] x20: 0000007fc42dd940 x19:
ffffff8817e45770 x18: ffffffd5947ce240
[48916.569468] [9:           lmkd:  536] x17: 2073656972742079 x16:
6e616d206f6f5420 x15: 3a726f727245205d
[48916.569471] [9:           lmkd:  536] x14: 36333520203a646b x13:
6563617073207265 x12: 0000000000000001
[48916.569473] [9:           lmkd:  536] x11: 6f74207365697274 x10:
0000000000000001 x9 : 4bd9ad4516d75100
[48916.569476] [9:           lmkd:  536] x8 : 4bd9ad4516d75100 x7 :
205d383431393635 x6 : 2e36313938345b0a
[48916.569479] [9:           lmkd:  536] x5 : ffffffc080fa5998 x4 :
ffffffd591707202 x3 : 0001360a00000000
[48916.569481] [9:           lmkd:  536] x2 : ffffffc096993af4 x1 :
00000000000000c0 x0 : 0000000000000028
[48916.569484] [9:           lmkd:  536] Call trace:
[48916.569486] [9:           lmkd:  536]
trace_user_fault_read+0x334/0x360 (P)
[48916.569488] [9:           lmkd:  536]  tracing_mark_write+0x84/0x174
[48916.569491] [9:           lmkd:  536]  __arm64_sys_write+0x2a0/0x5c0
[48916.569494] [9:           lmkd:  536]  invoke_syscall+0x58/0xe4
[48916.569498] [9:           lmkd:  536]  do_el0_svc+0x48/0xdc
[48916.569500] [9:           lmkd:  536]  el0_svc+0x3c/0x98
[48916.569503] [9:           lmkd:  536]
el0t_64_sync_handler+0x20/0x130
[48916.569505] [9:           lmkd:  536]  el0t_64_sync+0x1c4/0x1c8
[48916.569508] [9:           lmkd:  536] Kernel panic - not syncing:
kernel: panic_on_warn set ...


The code at the WARN location mentioned in the log above is as follows.

7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
  tries to read user space"))
7375                         return NULL;


Our current analysis is as follows:

In the Gmail process, during a low memory situation, LMKD writes strings
to /sys/kernel/tracing/trace_marker for systrace recording. At the same
time, it broadcasts a sigkill due to low memory, which is causing the
LMKD trace marker operation to stall.


Please review this issue.
Thanks,
Jeongho Choi

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [BUG] tracing: Too many tries to read user space

[Steven Rostedt]

On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:

> The code at the WARN location mentioned in the log above is as follows.
> 
> 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
>   tries to read user space"))
> 7375                         return NULL;
> 

This happens when something forces a schedule.

> 
> Our current analysis is as follows:
> 
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.
> 

Can you see what is being scheduled in? Perhaps use the persistent ring
buffer (if you can) and enable sched_switch tracepoint in it.

The loop is this:

	do {
		/*
		 * It is possible that something is trying to migrate this
		 * task. What happens then, is when preemption is enabled,
		 * the migration thread will preempt this task, try to
		 * migrate it, fail, then let it run again. That will
		 * cause this to loop again and never succeed.
		 * On failures, enabled and disable preemption with
		 * migration enabled, to allow the migration thread to
		 * migrate this task.
		 */
		if (trys) {
			preempt_enable_notrace();
			preempt_disable_notrace();
			cpu = smp_processor_id();
			buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
		}

		/*
		 * If for some reason, copy_from_user() always causes a context
		 * switch, this would then cause an infinite loop.
		 * If this task is preempted by another user space task, it
		 * will cause this task to try again. But just in case something
		 * changes where the copying from user space causes another task
		 * to run, prevent this from going into an infinite loop.
		 * 100 tries should be plenty.
		 */
		if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
			return NULL;

		/* Read the current CPU context switch counter */
		cnt = nr_context_switches_cpu(cpu);

		/*
		 * Preemption is going to be enabled, but this task must
		 * remain on this CPU.
		 */
		migrate_disable();

		/*
		 * Now preemption is being enabled and another task can come in
		 * and use the same buffer and corrupt our data.
		 */
		preempt_enable_notrace();

		/* Make sure preemption is enabled here */
		lockdep_assert_preemption_enabled();

		if (copy_func) {
			ret = copy_func(buffer, ptr, size, data);
		} else {
			ret = __copy_from_user(buffer, ptr, size);
		}

		preempt_disable_notrace();
		migrate_enable();

		/* if it faulted, no need to test if the buffer was corrupted */
		if (ret)
			return NULL;

		/*
		 * Preemption is disabled again, now check the per CPU context
		 * switch counter. If it doesn't match, then another user space
		 * process may have schedule in and corrupted our buffer. In that
		 * case the copying must be retried.
		 */
	} while (nr_context_switches_cpu(cpu) != cnt);

What it does is to write into a per CPU buffer from user space. If it
schedules out while trying this, it will try again as another task could
have come in and corrupted the buffer.

For some reason, when preemption is enabled to copy from user, something
is forcing a schedule of the process. If we can figure out what is doing
that, perhaps we can fix this. A similar thing happened with the migration
thread that commit edca33a56297d ("tracing: Fix failure to read user space
from system call trace events") fixed.

-- Steve

Re: [BUG] tracing: Too many tries to read user space

[Masami Hiramatsu]

On Wed, 8 Jul 2026 09:18:04 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 8 Jul 2026 21:37:53 +0900
> Jeongho Choi <jh1012.choi@samsung.com> wrote:
> 
> > The code at the WARN location mentioned in the log above is as follows.
> > 
> > 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
> >   tries to read user space"))
> > 7375                         return NULL;
> > 
> 
> This happens when something forces a schedule.
> 
> > 
> > Our current analysis is as follows:
> > 
> > In the Gmail process, during a low memory situation, LMKD writes strings
> > to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> > time, it broadcasts a sigkill due to low memory, which is causing the
> > LMKD trace marker operation to stall.
> > 
> 
> Can you see what is being scheduled in? Perhaps use the persistent ring
> buffer (if you can) and enable sched_switch tracepoint in it.

Nit: If the device just caused oom and not crash/reboot the kernel, you can
just use normal ring buffer (or perfetto).

Thanks, 

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [BUG] tracing: Too many tries to read user space

[Masami Hiramatsu]

On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:

> Hello,
> 
> We are seeing a reproducible kernel panic related to the tracing code
> when it fails to read user-space memory.
> 
> The issue was originally reported through the Android/Google Issue
> Tracker, and we were advised to report it to the upstream trace mailing
> list because the affected code is upstream.
> 
> Environment:
> 
> Architecture: arm64
> Kernel: Linux 6.18.21
> Base: Android Common Kernel (android17-6.18)
> Affected area: kernel/trace/
> 
> The relevant error/panic log is:
> [48916.569148] [9:           lmkd:  536] Error: Too many tries to read
> user space
> [48916.569156] [9:           lmkd:  536] WARNING: CPU: 9 PID: 536 at
> kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
> [48916.569443] [9:           lmkd:  536] CPU: 9 UID: 1069 PID: 536 Comm:
> lmkd Tainted: G           OE       6.18.21-android17-5-ga1a8e8cab9ec-4k
> #1 PREEMPT  25372cd4750dcac3c0fe86b57d47c665f97a6046
> 
> [48916.569450] [9:           lmkd:  536] pstate: 63402005 (nZCv daif
> +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [48916.569452] [9:           lmkd:  536] pc :
> trace_user_fault_read+0x334/0x360
> [48916.569454] [9:           lmkd:  536] lr :
> trace_user_fault_read+0x330/0x360
> [48916.569456] [9:           lmkd:  536] sp : ffffffc096993cc0
> [48916.569457] [9:           lmkd:  536] x29: ffffffc096993cd0 x28:
> 0000000000000065 x27: ffffff8817e45200
> [48916.569461] [9:           lmkd:  536] x26: 0000000000000000 x25:
> 000000011616c082 x24: 0000000000000000
> [48916.569463] [9:           lmkd:  536] x23: 0000000000000009 x22:
> 000000000000002b x21: ffffff880597f740
> [48916.569466] [9:           lmkd:  536] x20: 0000007fc42dd940 x19:
> ffffff8817e45770 x18: ffffffd5947ce240
> [48916.569468] [9:           lmkd:  536] x17: 2073656972742079 x16:
> 6e616d206f6f5420 x15: 3a726f727245205d
> [48916.569471] [9:           lmkd:  536] x14: 36333520203a646b x13:
> 6563617073207265 x12: 0000000000000001
> [48916.569473] [9:           lmkd:  536] x11: 6f74207365697274 x10:
> 0000000000000001 x9 : 4bd9ad4516d75100
> [48916.569476] [9:           lmkd:  536] x8 : 4bd9ad4516d75100 x7 :
> 205d383431393635 x6 : 2e36313938345b0a
> [48916.569479] [9:           lmkd:  536] x5 : ffffffc080fa5998 x4 :
> ffffffd591707202 x3 : 0001360a00000000
> [48916.569481] [9:           lmkd:  536] x2 : ffffffc096993af4 x1 :
> 00000000000000c0 x0 : 0000000000000028
> [48916.569484] [9:           lmkd:  536] Call trace:
> [48916.569486] [9:           lmkd:  536]
> trace_user_fault_read+0x334/0x360 (P)
> [48916.569488] [9:           lmkd:  536]  tracing_mark_write+0x84/0x174
> [48916.569491] [9:           lmkd:  536]  __arm64_sys_write+0x2a0/0x5c0
> [48916.569494] [9:           lmkd:  536]  invoke_syscall+0x58/0xe4
> [48916.569498] [9:           lmkd:  536]  do_el0_svc+0x48/0xdc
> [48916.569500] [9:           lmkd:  536]  el0_svc+0x3c/0x98
> [48916.569503] [9:           lmkd:  536]
> el0t_64_sync_handler+0x20/0x130
> [48916.569505] [9:           lmkd:  536]  el0t_64_sync+0x1c4/0x1c8
> [48916.569508] [9:           lmkd:  536] Kernel panic - not syncing:
> kernel: panic_on_warn set ...
> 
> 
> The code at the WARN location mentioned in the log above is as follows.
> 
> 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
>   tries to read user space"))
> 7375                         return NULL;
> 
> 
> Our current analysis is as follows:
> 
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.

Hm, in my view, this warning indicates that the circuit breaker has
triggered correctly, so that is not a bug. Under the heavy memory
pressure and low-memory situation, the page can be reclaimed soon
after it is copied.

However, this seems a bit strange that we only checks the CPU-wide context
switching in the loop. Instead, can we introduce a per-cpu sequence counter
to per-cpu buffer, and check it? 

Could you try this ?

From f76d8e4400a5961725d17899f4290c9334987e2b Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <mhiramat@kernel.org>
Date: Fri, 10 Jul 2026 12:11:00 +0900
Subject: [PATCH] tracing: Use per-CPU sequence counter in
 trace_user_fault_read

trace_user_fault_read() copies trace data from user space to the
per-CPU trace buffer. When preemption is enabled during the copy, it
checks if any context switches occurred on the current CPU via
nr_context_switches_cpu() to detect whether the buffer may have been
corrupted by another trace writer.

However, under heavy memory pressure, copying from user space can trigger
page faults (e.g., for swapped-out BSS or anonymous pages) that block and
cause a context switch. Because nr_context_switches_cpu() detects any
context switch (even unrelated ones), it mistakenly assumes the buffer was
corrupted. This leads to repeated retries (up to the 100-try limit), which
causes a WARN_ONCE backtrace and returns -EFAULT to user space, even if
no other task ever accessed the trace buffer.

To mitigate this issue, replace the CPU-wide context switch check with
a dedicated per-CPU sequence counter in struct trace_user_buf.

Since only other tasks invoking trace_user_fault_read() on the same CPU
will increment this counter, unrelated context switches (including those
from page fault sleep) will no longer trigger retries.

Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
 kernel/trace/trace.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 1bc27c0ad029..46cec63f5798 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5989,6 +5989,7 @@ static ssize_t write_marker_to_buffer(struct trace_array *tr, const char *buf,
 
 struct trace_user_buf {
 	char		*buf;
+	unsigned int	sequence;
 };
 
 static DEFINE_MUTEX(trace_user_buffer_mutex);
@@ -6031,7 +6032,10 @@ static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t si
 
 	/* Clear each buffer in case of error */
 	for_each_possible_cpu(cpu) {
-		per_cpu_ptr(tinfo->tbuf, cpu)->buf = NULL;
+		struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+
+		tbuf->buf = NULL;
+		tbuf->sequence = 0;
 	}
 
 	for_each_possible_cpu(cpu) {
@@ -6196,8 +6200,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			     trace_user_buf_copy copy_func, void *data)
 {
 	int cpu = smp_processor_id();
-	char *buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
-	unsigned int cnt;
+	struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+	char *buffer = tbuf->buf;
+	unsigned int seq;
 	int trys = 0;
 	int ret;
 
@@ -6211,10 +6216,10 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		return NULL;
 
 	/*
-	 * This acts similar to a seqcount. The per CPU context switches are
+	 * This acts similar to a seqcount. The per CPU sequence counters are
 	 * recorded, migration is disabled and preemption is enabled. The
 	 * read of the user space memory is copied into the per CPU buffer.
-	 * Preemption is disabled again, and if the per CPU context switches count
+	 * Preemption is disabled again, and if the per CPU sequence count
 	 * is still the same, it means the buffer has not been corrupted.
 	 * If the count is different, it is assumed the buffer is corrupted
 	 * and reading must be tried again.
@@ -6235,7 +6240,8 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			preempt_enable_notrace();
 			preempt_disable_notrace();
 			cpu = smp_processor_id();
-			buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
+			tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+			buffer = tbuf->buf;
 		}
 
 		/*
@@ -6250,8 +6256,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
 			return NULL;
 
-		/* Read the current CPU context switch counter */
-		cnt = nr_context_switches_cpu(cpu);
+		/* Increment the per-CPU buffer sequence counter */
+		tbuf->sequence++;
+		seq = tbuf->sequence;
 
 		/*
 		 * Preemption is going to be enabled, but this task must
@@ -6282,12 +6289,12 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/*
-		 * Preemption is disabled again, now check the per CPU context
-		 * switch counter. If it doesn't match, then another user space
+		 * Preemption is disabled again, now check the per CPU sequence
+		 * counter. If it doesn't match, then another user space
 		 * process may have schedule in and corrupted our buffer. In that
 		 * case the copying must be retried.
 		 */
-	} while (nr_context_switches_cpu(cpu) != cnt);
+	} while (tbuf->sequence != seq);
 
 	return buffer;
 }
-- 
2.43.0



-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [BUG] tracing: Too many tries to read user space

[Steven Rostedt]

On Fri, 10 Jul 2026 12:22:31 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> However, this seems a bit strange that we only checks the CPU-wide context
> switching in the loop. Instead, can we introduce a per-cpu sequence counter
> to per-cpu buffer, and check it? 

I originally tried this but found a situation that it fails:

  tbuf->sequence = 0;

	Task 1				Task 2
	------				------

   tbuf->sequence++;
   seq = tbuf->sequence; (seq = 1)

   preempt_enable();

   [schedule] ---------------------->

				  tbuf->sequence++;
				  seq = tbuf->sequence; (seq = 2);

				  preempt_enable();

				  copy_from_user(buffer);

	     <--------------------[schedule]

   copy_from_user(buffer);

   *** BUFFER NOW CORRUPTED ***

   [schedule] ---------------------->

				  preempt_disable();

				} while (tubf->sequence != seq); // tbuf->sequence == seq !!!!


This is why we use a CPU wide counter.

-- Steve
  

Re: [BUG] tracing: Too many tries to read user space

[Masami Hiramatsu]

On Fri, 10 Jul 2026 07:46:05 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 10 Jul 2026 12:22:31 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> 
> > However, this seems a bit strange that we only checks the CPU-wide context
> > switching in the loop. Instead, can we introduce a per-cpu sequence counter
> > to per-cpu buffer, and check it? 
> 
> I originally tried this but found a situation that it fails:
> 
>   tbuf->sequence = 0;
> 
> 	Task 1				Task 2
> 	------				------
> 
>    tbuf->sequence++;
>    seq = tbuf->sequence; (seq = 1)
> 
>    preempt_enable();
> 
>    [schedule] ---------------------->
> 
> 				  tbuf->sequence++;
> 				  seq = tbuf->sequence; (seq = 2);
> 
> 				  preempt_enable();
> 
> 				  copy_from_user(buffer);
> 
> 	     <--------------------[schedule]
> 
>    copy_from_user(buffer);
> 
>    *** BUFFER NOW CORRUPTED ***
> 
>    [schedule] ---------------------->
> 
> 				  preempt_disable();
> 
> 				} while (tubf->sequence != seq); // tbuf->sequence == seq !!!!
> 
> 
> This is why we use a CPU wide counter.

Ah, indeed. As similar to seqlock, maybe we need to increment sequence
number after the process so that we can find the buffer is used some
other process. Or, just use a mutex to wait. But anyway, that will be
make it slower?

BTW, it is just an idea, but can we split fast path (per-cpu buffer)
and slow path (allocate an intermadiate buffer) for the page which
can cause page fault ?

Thanks,

> 
> -- Steve
>   
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [BUG] tracing: Too many tries to read user space

[Steven Rostedt]

On Fri, 10 Jul 2026 12:22:31 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> Hm, in my view, this warning indicates that the circuit breaker has
> triggered correctly, so that is not a bug. Under the heavy memory
> pressure and low-memory situation, the page can be reclaimed soon
> after it is copied.

So you are saying that every time the copy_from_user() is executed, the
page is reclaimed? And this causes a schedule?

Now, I did have a version that used sched_switch and only incremented the
counter when a non-kernel thread was scheduled in. Then the test would
check if the counter increased by 2 or more. As an increase by 1 meant that
only kernel threads scheduled in which would not corrupt the buffer. The 1
increment was the current task scheduling back.

This is based on that work (I'm glad I save old versions in my git tree :-)

Funny, the comments were from the original change I did back in August of
2025, which mentions kernel threads scheduling in to handle the fault.

I also kept this around in case it was needed. Looks like it may be needed.

-- Steve

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 18710c190c92..19354fe2fca1 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -53,6 +53,8 @@
 #include <linux/io.h> /* vmap_page_range() */
 #include <linux/fs_context.h>
 
+#include <trace/events/sched.h>
+
 #include <asm/setup.h> /* COMMAND_LINE_SIZE */
 
 #include "trace.h"
@@ -5984,6 +5986,32 @@ struct trace_user_buf {
 static DEFINE_MUTEX(trace_user_buffer_mutex);
 static struct trace_user_buf_info *trace_user_buffer;
 
+static DEFINE_PER_CPU(unsigned long, sched_switch_cnt);
+
+/*
+ * The per CPU buffer trace_user_buffer is written to optimstically.
+ * The counter sched_switch_cnt is taken, preemption is enabled,
+ * the copying of the user space memory is placed into the trace_user_buffer,
+ * Preeption is re-enabled and the count is read again. If the count is greater
+ * than one from its previous reading, it means that another user space
+ * task scheduled in and the buffer is unreliable for use.
+ */
+static void
+probe_sched_switch(void *ignore, bool preempt,
+		   struct task_struct *prev, struct task_struct *next,
+		   unsigned int prev_state)
+{
+	/*
+	 * The buffer can only be corrupted by another user space task.
+	 * Ignore kernel tasks that may be scheduled in order to process
+	 * the faulting memory.
+	 */
+	if (!is_user_task(next))
+		return;
+
+	this_cpu_inc(sched_switch_cnt);
+}
+
 /**
  * trace_user_fault_destroy - free up allocated memory of a trace user buffer
  * @tinfo: The descriptor to free up
@@ -6003,6 +6031,8 @@ void trace_user_fault_destroy(struct trace_user_buf_info *tinfo)
 		kfree(buf);
 	}
 	free_percpu(tinfo->tbuf);
+
+	unregister_trace_sched_switch(probe_sched_switch, NULL);
 }
 
 static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t size)
@@ -6053,11 +6083,17 @@ static int user_buffer_init(struct trace_user_buf_info **tinfo, size_t size)
 
 	lockdep_assert_held(&trace_user_buffer_mutex);
 
+	ret = register_trace_sched_switch(probe_sched_switch, NULL);
+	if (ret < 0)
+		return ret;
+
 	if (!*tinfo) {
 		alloc = true;
 		*tinfo = kzalloc_obj(**tinfo);
-		if (!*tinfo)
+		if (!*tinfo) {
+			unregister_trace_sched_switch(probe_sched_switch, NULL);
 			return -ENOMEM;
+		}
 	}
 
 	ret = user_fault_buffer_enable(*tinfo, size);
@@ -6241,7 +6277,7 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/* Read the current CPU context switch counter */
-		cnt = nr_context_switches_cpu(cpu);
+		cnt = this_cpu_read(sched_switch_cnt);
 
 		/*
 		 * Preemption is going to be enabled, but this task must
@@ -6272,12 +6308,19 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/*
-		 * Preemption is disabled again, now check the per CPU context
-		 * switch counter. If it doesn't match, then another user space
-		 * process may have schedule in and corrupted our buffer. In that
-		 * case the copying must be retried.
+		 * Preemption is disabled again, now check the sched_switch_cnt.
+		 * If it increased by two or more, then another user space process
+		 * may have schedule in and corrupted our buffer. In that case
+		 * the copying must be retried.
+		 *
+		 * Note, if this task was scheduled out and only kernel threads
+		 * were scheduled in (maybe to process the fault), then the
+		 * counter would increment again when this task scheduled in.
+		 * If this task scheduled out and another user task scheduled
+		 * in, this task would still need to be scheduled back in and
+		 * the counter would increment by at least two.
 		 */
-	} while (nr_context_switches_cpu(cpu) != cnt);
+	} while (this_cpu_read(sched_switch_cnt) > cnt + 1);
 
 	return buffer;
 }

Re: [BUG] tracing: Too many tries to read user space

[Masami Hiramatsu]

On Fri, 10 Jul 2026 08:33:57 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 10 Jul 2026 12:22:31 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> 
> > Hm, in my view, this warning indicates that the circuit breaker has
> > triggered correctly, so that is not a bug. Under the heavy memory
> > pressure and low-memory situation, the page can be reclaimed soon
> > after it is copied.
> 
> So you are saying that every time the copy_from_user() is executed, the
> page is reclaimed? And this causes a schedule?
> 
> Now, I did have a version that used sched_switch and only incremented the
> counter when a non-kernel thread was scheduled in. Then the test would
> check if the counter increased by 2 or more. As an increase by 1 meant that
> only kernel threads scheduled in which would not corrupt the buffer. The 1
> increment was the current task scheduling back.
> 
> This is based on that work (I'm glad I save old versions in my git tree :-)
> 
> Funny, the comments were from the original change I did back in August of
> 2025, which mentions kernel threads scheduling in to handle the fault.
> 
> I also kept this around in case it was needed. Looks like it may be needed.

Hmm, it's not clear whether this is the result of a scheduling storm
caused by kernel threads. But maybe worth to try.

Jeongho, can you try this patch? Or/and you maybe also try using
mlock the LMKD process memory which is passed to trace_marker.

Thank you, 

> 
> -- Steve
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 18710c190c92..19354fe2fca1 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -53,6 +53,8 @@
>  #include <linux/io.h> /* vmap_page_range() */
>  #include <linux/fs_context.h>
>  
> +#include <trace/events/sched.h>
> +
>  #include <asm/setup.h> /* COMMAND_LINE_SIZE */
>  
>  #include "trace.h"
> @@ -5984,6 +5986,32 @@ struct trace_user_buf {
>  static DEFINE_MUTEX(trace_user_buffer_mutex);
>  static struct trace_user_buf_info *trace_user_buffer;
>  
> +static DEFINE_PER_CPU(unsigned long, sched_switch_cnt);
> +
> +/*
> + * The per CPU buffer trace_user_buffer is written to optimstically.
> + * The counter sched_switch_cnt is taken, preemption is enabled,
> + * the copying of the user space memory is placed into the trace_user_buffer,
> + * Preeption is re-enabled and the count is read again. If the count is greater
> + * than one from its previous reading, it means that another user space
> + * task scheduled in and the buffer is unreliable for use.
> + */
> +static void
> +probe_sched_switch(void *ignore, bool preempt,
> +		   struct task_struct *prev, struct task_struct *next,
> +		   unsigned int prev_state)
> +{
> +	/*
> +	 * The buffer can only be corrupted by another user space task.
> +	 * Ignore kernel tasks that may be scheduled in order to process
> +	 * the faulting memory.
> +	 */
> +	if (!is_user_task(next))
> +		return;
> +
> +	this_cpu_inc(sched_switch_cnt);
> +}
> +
>  /**
>   * trace_user_fault_destroy - free up allocated memory of a trace user buffer
>   * @tinfo: The descriptor to free up
> @@ -6003,6 +6031,8 @@ void trace_user_fault_destroy(struct trace_user_buf_info *tinfo)
>  		kfree(buf);
>  	}
>  	free_percpu(tinfo->tbuf);
> +
> +	unregister_trace_sched_switch(probe_sched_switch, NULL);
>  }
>  
>  static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t size)
> @@ -6053,11 +6083,17 @@ static int user_buffer_init(struct trace_user_buf_info **tinfo, size_t size)
>  
>  	lockdep_assert_held(&trace_user_buffer_mutex);
>  
> +	ret = register_trace_sched_switch(probe_sched_switch, NULL);
> +	if (ret < 0)
> +		return ret;
> +
>  	if (!*tinfo) {
>  		alloc = true;
>  		*tinfo = kzalloc_obj(**tinfo);
> -		if (!*tinfo)
> +		if (!*tinfo) {
> +			unregister_trace_sched_switch(probe_sched_switch, NULL);
>  			return -ENOMEM;
> +		}
>  	}
>  
>  	ret = user_fault_buffer_enable(*tinfo, size);
> @@ -6241,7 +6277,7 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
>  			return NULL;
>  
>  		/* Read the current CPU context switch counter */
> -		cnt = nr_context_switches_cpu(cpu);
> +		cnt = this_cpu_read(sched_switch_cnt);
>  
>  		/*
>  		 * Preemption is going to be enabled, but this task must
> @@ -6272,12 +6308,19 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
>  			return NULL;
>  
>  		/*
> -		 * Preemption is disabled again, now check the per CPU context
> -		 * switch counter. If it doesn't match, then another user space
> -		 * process may have schedule in and corrupted our buffer. In that
> -		 * case the copying must be retried.
> +		 * Preemption is disabled again, now check the sched_switch_cnt.
> +		 * If it increased by two or more, then another user space process
> +		 * may have schedule in and corrupted our buffer. In that case
> +		 * the copying must be retried.
> +		 *
> +		 * Note, if this task was scheduled out and only kernel threads
> +		 * were scheduled in (maybe to process the fault), then the
> +		 * counter would increment again when this task scheduled in.
> +		 * If this task scheduled out and another user task scheduled
> +		 * in, this task would still need to be scheduled back in and
> +		 * the counter would increment by at least two.
>  		 */
> -	} while (nr_context_switches_cpu(cpu) != cnt);
> +	} while (this_cpu_read(sched_switch_cnt) > cnt + 1);
>  
>  	return buffer;
>  }


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [BUG] tracing: Too many tries to read user space

[Steven Rostedt]

On Mon, 13 Jul 2026 12:16:59 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> > I also kept this around in case it was needed. Looks like it may be needed.  
> 
> Hmm, it's not clear whether this is the result of a scheduling storm
> caused by kernel threads. But maybe worth to try.

I don't see what else it could be.

> 
> Jeongho, can you try this patch? Or/and you maybe also try using
> mlock the LMKD process memory which is passed to trace_marker.

Yes, Jeongho, can you please test this patch and also see if mlocking the
memory used by trace_marker would help too?

-- Steve

Re: [BUG] tracing: Too many tries to read user space

[Jeongho Choi]

[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]

On Mon, Jul 13, 2026 at 10:23:43AM -0400, Steven Rostedt wrote:
> On Mon, 13 Jul 2026 12:16:59 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> 
> > > I also kept this around in case it was needed. Looks like it may be needed.  
> > 
> > Hmm, it's not clear whether this is the result of a scheduling storm
> > caused by kernel threads. But maybe worth to try.
> 
> I don't see what else it could be.
> 
> > 
> > Jeongho, can you try this patch? Or/and you maybe also try using
> > mlock the LMKD process memory which is passed to trace_marker.
> 
> Yes, Jeongho, can you please test this patch and also see if mlocking the
> memory used by trace_marker would help too?
> 
> -- Steve
>
Hi Steven adn Masami,

Sorry for the delayed response.
I would like to share the current situation.

This issue occurred during the verification process of our latest Exynos
chip development.
Unfortunately, this problem is not being further reproduced.
We have continued testing the unmodified kernel on about 40 devices for
roughly 10 hours per day for approximately one week.

Because the baseline kernel no longer reproduces the problem at a
measurable rate, I do not think that a test with the proposed patch, or
with the LMKD buffer mlocked, would currently allow us to determine
whether either change improves the issue. A non-reproduction with the
patch would be indistinguishable from the current baseline result.

Versions that can be deployed for verification with a large
number of devices are limited to official release-ready versions.
I can only test the test patches you shared by building them
individually and testing them on the few devices I have.

Therefore, at this point I cannot provide a conclusive test result for
the proposed patch. I can still perform limited build and functional
testing on a small number of devices, but I would not be able to verify
whether the patch changes the reproduction rate.

Thanks,
Jeongho


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]