perf: fuzzer leads to trace_kprobe: Could not insert message flood

perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Vince Weaver]

Author: Song Liu <songliubraving@fb.com>
Date:   Wed Dec 6 14:45:15 2017 -0800

When running the perf_fuzzer on a current git checkout my logs are flooded 
with messages such as this:
[71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
[71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22

Presumably this is due to the introduction of the perf_kprobe PMU in
	commit e12f03d7031a977356e3d7b75a68c2185ff8d155
	Author: Song Liu <songliubraving@fb.com>
	Date:   Wed Dec 6 14:45:15 2017 -0800

Is there a way to get this error disabled, or else rate-limited?

Vince

Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Song Liu]

> On Apr 10, 2018, at 7:48 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> 
> Author: Song Liu <songliubraving@fb.com>
> Date:   Wed Dec 6 14:45:15 2017 -0800
> 
> When running the perf_fuzzer on a current git checkout my logs are flooded 
> with messages such as this:
> [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
> [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
> 
> Presumably this is due to the introduction of the perf_kprobe PMU in
> 	commit e12f03d7031a977356e3d7b75a68c2185ff8d155
> 	Author: Song Liu <songliubraving@fb.com>
> 	Date:   Wed Dec 6 14:45:15 2017 -0800
> 
> Is there a way to get this error disabled, or else rate-limited?
> 
> Vince

Hi Vince, 

Thanks for the report. 

This is a new API that creates probe together with perf_event_open(). Based on 
my limited understanding of perf_fuzzer, it doesn't understand this API, and uses 
it in an abnormal way. I would recommend perf_fuzzer to understand this new
API and test it. For more information about using this API, please refer to the 
man-page diff available at:

      https://patchwork.kernel.org/patch/10097283/

Thanks again for the test and report. Please let me know if you have further 
questions. 

Best, 
Song

Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Ingo Molnar]

* Song Liu <songliubraving@fb.com> wrote:

> 
> 
> > On Apr 10, 2018, at 7:48 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> > 
> > Author: Song Liu <songliubraving@fb.com>
> > Date:   Wed Dec 6 14:45:15 2017 -0800
> > 
> > When running the perf_fuzzer on a current git checkout my logs are flooded 
> > with messages such as this:
> > [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
> > [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
> > 
> > Presumably this is due to the introduction of the perf_kprobe PMU in
> > 	commit e12f03d7031a977356e3d7b75a68c2185ff8d155
> > 	Author: Song Liu <songliubraving@fb.com>
> > 	Date:   Wed Dec 6 14:45:15 2017 -0800
> > 
> > Is there a way to get this error disabled, or else rate-limited?
> > 
> > Vince
> 
> Hi Vince, 
> 
> Thanks for the report. 
> 
> This is a new API that creates probe together with perf_event_open(). Based on 
> my limited understanding of perf_fuzzer, it doesn't understand this API, and uses 
> it in an abnormal way. [...]

Vince's point is valid: the kernel log should not be flooded with pointless 
messages as a response to user-space ABI uses ...

Why is there a kernel log message at all, isn't an error returned?

> [...] I would recommend perf_fuzzer to understand this new API and test it. 
> [...]

This bug needs to be fixed: a new API must not effectively DoS fuzzing efforts by 
spamming the kernel log ...

Thanks,

	Ingo

Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Song Liu]

> On Apr 11, 2018, at 5:04 AM, Ingo Molnar <mingo@kernel.org> wrote:
> 
> 
> * Song Liu <songliubraving@fb.com> wrote:
> 
>> 
>> 
>>> On Apr 10, 2018, at 7:48 AM, Vince Weaver <vincent.weaver@maine.edu> wrote:
>>> 
>>> Author: Song Liu <songliubraving@fb.com>
>>> Date:   Wed Dec 6 14:45:15 2017 -0800
>>> 
>>> When running the perf_fuzzer on a current git checkout my logs are flooded 
>>> with messages such as this:
>>> [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
>>> [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
>>> 
>>> Presumably this is due to the introduction of the perf_kprobe PMU in
>>> 	commit e12f03d7031a977356e3d7b75a68c2185ff8d155
>>> 	Author: Song Liu <songliubraving@fb.com>
>>> 	Date:   Wed Dec 6 14:45:15 2017 -0800
>>> 
>>> Is there a way to get this error disabled, or else rate-limited?
>>> 
>>> Vince
>> 
>> Hi Vince, 
>> 
>> Thanks for the report. 
>> 
>> This is a new API that creates probe together with perf_event_open(). Based on 
>> my limited understanding of perf_fuzzer, it doesn't understand this API, and uses 
>> it in an abnormal way. [...]
> 
> Vince's point is valid: the kernel log should not be flooded with pointless 
> messages as a response to user-space ABI uses ...
> 
> Why is there a kernel log message at all, isn't an error returned?
> 
>> [...] I would recommend perf_fuzzer to understand this new API and test it. 
>> [...]
> 
> This bug needs to be fixed: a new API must not effectively DoS fuzzing efforts by 
> spamming the kernel log ...

Yeah, the new API allows non-root user to trigger this message. We should only 
allow root to create kprobe with perf_event_open(). 

On the other hand, do we need to fix this for root? In fact, a simple bash loop 
can create something similar through the text interface (with root):

root@virt-test:~# for x in {0..5} ; do echo p:xx xx+$x >> /sys/kernel/debug/tracing/kprobe_events ; done
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
root@virt-test:~# dmesg | tail -n 5
[  664.208374] trace_kprobe: Could not insert probe at xx+1: -2
[  664.237882] trace_kprobe: Could not insert probe at xx+2: -2
[  664.268067] trace_kprobe: Could not insert probe at xx+3: -2
[  664.297395] trace_kprobe: Could not insert probe at xx+4: -2
[  664.327614] trace_kprobe: Could not insert probe at xx+5: -2

This happens before the new API is introduced. 

The following patch does capable(CAP_SYS_ADMIN) for perf_kprobe and 
perf_uprobe at an earlier stage, so non-root user cannot trigger 
this error message. Please let me know whether we need to fix this 
for root. 

Thanks,
Song



>From c6708e9e3cd5ba7afb5a7f693b04abf64fec031e Mon Sep 17 00:00:00 2001
From: Song Liu <songliubraving@fb.com>
Date: Wed, 11 Apr 2018 10:37:00 -0700
Subject: [PATCH] perf: need CAP_SYS_ADMIN to create k/uprobe with
 perf_event_open()

Non-root user cannot create kprobe or uprobe through the text-based
interface (kprobe_events, uprobe_events). So they cannot create the
probes with perf_event_open(). To ensure this, we check
capable(CAP_SYS_ADMIN) at perf_[k,u]probe_event_init().

Fixes: e12f03d7031a ("perf/core: Implement the 'perf_kprobe' PMU")
Fixes: 33ea4b24277b ("perf/core: Implement the 'perf_uprobe' PMU")

Signed-off-by: Song Liu <songliubraving@fb.com>
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Cc: Ingo Molnar <mingo@kernel.org>
---
 kernel/events/core.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index d7af828..2d5fe26 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)

        if (event->attr.type != perf_kprobe.type)
                return -ENOENT;
+
+       if (!capable(CAP_SYS_ADMIN))
+               return -EACCES;
+
        /*
         * no branch sampling for probe events
         */
@@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)

        if (event->attr.type != perf_uprobe.type)
                return -ENOENT;
+
+       if (!capable(CAP_SYS_ADMIN))
+               return -EACCES;
+
        /*
         * no branch sampling for probe events
         */
--
2.9.5

Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Ingo Molnar]

* Song Liu <songliubraving@fb.com> wrote:

> > spamming the kernel log ...
> 
> Yeah, the new API allows non-root user to trigger this message. We should only 
> allow root to create kprobe with perf_event_open(). 
> 
> On the other hand, do we need to fix this for root? In fact, a simple bash loop 
> can create something similar through the text interface (with root):
> 
> root@virt-test:~# for x in {0..5} ; do echo p:xx xx+$x >> /sys/kernel/debug/tracing/kprobe_events ; done
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> root@virt-test:~# dmesg | tail -n 5
> [  664.208374] trace_kprobe: Could not insert probe at xx+1: -2
> [  664.237882] trace_kprobe: Could not insert probe at xx+2: -2
> [  664.268067] trace_kprobe: Could not insert probe at xx+3: -2
> [  664.297395] trace_kprobe: Could not insert probe at xx+4: -2
> [  664.327614] trace_kprobe: Could not insert probe at xx+5: -2
> 
> This happens before the new API is introduced. 
> 
> The following patch does capable(CAP_SYS_ADMIN) for perf_kprobe and 
> perf_uprobe at an earlier stage, so non-root user cannot trigger 
> this error message. Please let me know whether we need to fix this 
> for root. 

That's two bugs then, and yes, I think we should fix the log spamming: what's the 
point? We already get an error code from the write.

I'll apply your CAP_SYS_ADMIN fix.

Thanks,

	Ingo

Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood

[Ingo Molnar]

* Song Liu <songliubraving@fb.com> wrote:

> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index d7af828..2d5fe26 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)
> 
>         if (event->attr.type != perf_kprobe.type)
>                 return -ENOENT;
> +
> +       if (!capable(CAP_SYS_ADMIN))
> +               return -EACCES;
> +
>         /*
>          * no branch sampling for probe events
>          */
> @@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)
> 
>         if (event->attr.type != perf_uprobe.type)
>                 return -ENOENT;
> +
> +       if (!capable(CAP_SYS_ADMIN))
> +               return -EACCES;

This is seriously whitespace damaged: all tabs are spaces ...

Thanks,

	Ingo

[tip:perf/urgent] perf/core: Need CAP_SYS_ADMIN to create k/uprobe with perf_event_open()

[tip-bot for Song Liu]

Commit-ID:  32e6e967fb36bf77ed99221ae3ce1909f045d8f9
Gitweb:     https://git.kernel.org/tip/32e6e967fb36bf77ed99221ae3ce1909f045d8f9
Author:     Song Liu <songliubraving@fb.com>
AuthorDate: Wed, 11 Apr 2018 18:02:37 +0000
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 12 Apr 2018 09:55:50 +0200

perf/core: Need CAP_SYS_ADMIN to create k/uprobe with perf_event_open()

Non-root user cannot create kprobe or uprobe through the text-based
interface (kprobe_events, uprobe_events),so they should not be able
to create probes via perf_event_open() either.

Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Signed-off-by: Song Liu <songliubraving@fb.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Fixes: 33ea4b24277b ("perf/core: Implement the 'perf_uprobe' PMU")
Fixes: e12f03d7031a ("perf/core: Implement the 'perf_kprobe' PMU")
Link: http://lkml.kernel.org/r/C0B2EFB5-C403-4BDB-9046-C14B3EE66999@fb.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/events/core.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index d7af82827373..2d5fe26551f8 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)
 
 	if (event->attr.type != perf_kprobe.type)
 		return -ENOENT;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
 	/*
 	 * no branch sampling for probe events
 	 */
@@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)
 
 	if (event->attr.type != perf_uprobe.type)
 		return -ENOENT;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
 	/*
 	 * no branch sampling for probe events
 	 */