Linux 4.15.0-rc3 perf probe/uprobe issue with address randomization

Linux 4.15.0-rc3 perf probe/uprobe issue with address randomization

[Thomas-Mich Richter]

During debugging of perf probe tool I discovered an issue with
uprobes and address randomization.

To set a uprobe on a function named inet_pton in libc library, you
obtain the address of the symbol inet_pton using command nm and
then use the following command to set the uprobe:

# echo "p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x142060"
	> /sys/kernel/debug/tracing/uprobe_events

0x142060 is the address of inet_pton on my system.
This works nicely and the uprobe is usable.

The issue is with the output:
# cat /sys/kernel/debug/tracing/uprobe_events
p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x000000002d0f8952
#

The displayed address 0x000000002d0f8952 is wrong, probably
randomized and post processing of this output with the perf
probe tool fails due to this random address:

# linux/tools/perf/perf probe -l
Failed to find debug information for address 2d0f8952
  probe_libc:inet_pton (on 0x2d0f8952 in /usr/lib64/libc-2.26.so)
# 

So how to fix this (if at all)?
Is replacing %p by %llx in line 612 of file kernel/trace/trace_uprobe.c
   seq_printf(m, "0x%p", (void *)tu->offset)
an option?
Or is this broken by design and intention?

Thanks for hints and suggestions.
-- 
Thomas Richter, Dept 3303, IBM LTC Boeblingen Germany
--
Vorsitzende des Aufsichtsrats: Martina Koederitz 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: Linux 4.15.0-rc3 perf probe/uprobe issue with address randomization

[Peter Zijlstra]

On Fri, Dec 15, 2017 at 03:14:37PM +0100, Thomas-Mich Richter wrote:
> During debugging of perf probe tool I discovered an issue with
> uprobes and address randomization.
> 
> To set a uprobe on a function named inet_pton in libc library, you
> obtain the address of the symbol inet_pton using command nm and
> then use the following command to set the uprobe:
> 
> # echo "p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x142060"
> 	> /sys/kernel/debug/tracing/uprobe_events
> 
> 0x142060 is the address of inet_pton on my system.
> This works nicely and the uprobe is usable.
> 
> The issue is with the output:
> # cat /sys/kernel/debug/tracing/uprobe_events
> p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x000000002d0f8952
> #
> 
> The displayed address 0x000000002d0f8952 is wrong, probably
> randomized and post processing of this output with the perf
> probe tool fails due to this random address:
> 
> # linux/tools/perf/perf probe -l
> Failed to find debug information for address 2d0f8952
>   probe_libc:inet_pton (on 0x2d0f8952 in /usr/lib64/libc-2.26.so)
> # 
> 
> So how to fix this (if at all)?
> Is replacing %p by %llx in line 612 of file kernel/trace/trace_uprobe.c
>    seq_printf(m, "0x%p", (void *)tu->offset)
> an option?
> Or is this broken by design and intention?

So recently %p got changed to hash pointers in order to avoid leaking
kernel addresses.

  ad67b74d2469 ("printk: hash addresses printed with %p")

I'm not sure what privilidges are required for reading that kprobe
state, but I suspect its root only, so changing this to %px might be
what is needed.

Re: Linux 4.15.0-rc3 perf probe/uprobe issue with address randomization

[Thomas-Mich Richter]

On 12/15/2017 03:21 PM, Peter Zijlstra wrote:
> On Fri, Dec 15, 2017 at 03:14:37PM +0100, Thomas-Mich Richter wrote:
>> During debugging of perf probe tool I discovered an issue with
>> uprobes and address randomization.
>>
>> To set a uprobe on a function named inet_pton in libc library, you
>> obtain the address of the symbol inet_pton using command nm and
>> then use the following command to set the uprobe:
>>
>> # echo "p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x142060"
>> 	> /sys/kernel/debug/tracing/uprobe_events
>>
>> 0x142060 is the address of inet_pton on my system.
>> This works nicely and the uprobe is usable.
>>
>> The issue is with the output:
>> # cat /sys/kernel/debug/tracing/uprobe_events
>> p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x000000002d0f8952
>> #
>>
>> The displayed address 0x000000002d0f8952 is wrong, probably
>> randomized and post processing of this output with the perf
>> probe tool fails due to this random address:
>>
>> # linux/tools/perf/perf probe -l
>> Failed to find debug information for address 2d0f8952
>>   probe_libc:inet_pton (on 0x2d0f8952 in /usr/lib64/libc-2.26.so)
>> # 
>>
>> So how to fix this (if at all)?
>> Is replacing %p by %llx in line 612 of file kernel/trace/trace_uprobe.c
>>    seq_printf(m, "0x%p", (void *)tu->offset)
>> an option?
>> Or is this broken by design and intention?
> 
> So recently %p got changed to hash pointers in order to avoid leaking
> kernel addresses.
> 
>   ad67b74d2469 ("printk: hash addresses printed with %p")
> 
> I'm not sure what privilidges are required for reading that kprobe
> state, but I suspect its root only, so changing this to %px might be
> what is needed.
> 

Thanks for the hint. The %px printk format string fixes the issue:


# echo "p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x142060" 
            > /sys/kernel/debug/tracing/uprobe_events
# cat /sys/kernel/debug/tracing/uprobe_events
p:probe_libc/inet_pton /usr/lib64/libc-2.26.so:0x0000000000142060
# ./tools/perf/perf probe -l
  probe_libc:inet_pton (on __inet_pton@resolv/inet_pton.c in /usr/lib64/libc-2.26.so)
# 

I will send a patch to fix this.
-- 
Thomas Richter, Dept 3303, IBM LTC Boeblingen Germany
--
Vorsitzende des Aufsichtsrats: Martina Koederitz 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294