[PATCH] trace : use kvmalloc instead of kmalloc

[PATCH] trace : use kvmalloc instead of kmalloc

[Zhaoyang Huang]

High order memory stuff within trace could introduce OOM, use kvmalloc instead.

Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low. 

traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),  order=2, oom_score_adj=-1

traced_probes cpuset=system-background mems_allowed=0
CPU: 3 PID: 588 Comm: traced_probes Tainted: G        W  O    4.14.181 #1
Hardware name: Generic DT based system
(unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
(show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
(dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
(dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
(oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
(out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
(__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
(kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
(kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
(tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
(do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
(vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
(path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
(do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
(do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
(SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)

Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
---
changes since v1: change kfree to kvfree
---
 kernel/trace/trace.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ca1ee65..8d70c79 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
 	if (trace_array_get(tr) < 0)
 		return -ENODEV;
 
-	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	info = kvmalloc(sizeof(*info), GFP_KERNEL);
 	if (!info) {
 		trace_array_put(tr);
 		return -ENOMEM;
@@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
 	if (info->spare)
 		ring_buffer_free_read_page(iter->trace_buffer->buffer,
 					   info->spare_cpu, info->spare);
-	kfree(info);
+	kvfree(info);
 
 	mutex_unlock(&trace_types_lock);
 
-- 
1.9.1

Re: [PATCH] trace : use kvmalloc instead of kmalloc

[Steven Rostedt]

On Thu, 30 Jul 2020 19:04:12 +0800
Zhaoyang Huang <huangzhaoyang@gmail.com> wrote:

> High order memory stuff within trace could introduce OOM, use kvmalloc instead.
> 
> Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low. 

Please limit your column width in the description of patches to 76
characters.

> 
> traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),  order=2, oom_score_adj=-1
> 

What does this traced_probes thing do?

> traced_probes cpuset=system-background mems_allowed=0
> CPU: 3 PID: 588 Comm: traced_probes Tainted: G        W  O    4.14.181 #1
> Hardware name: Generic DT based system
> (unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
> (show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
> (dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
> (dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
> (oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
> (out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
> (__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
> (kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
> (kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
> (tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
> (do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
> (vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
> (path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
> (do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
> (do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
> (SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)
> 
> Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
> ---
> changes since v1: change kfree to kvfree
> ---
>  kernel/trace/trace.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index ca1ee65..8d70c79 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
>  	if (trace_array_get(tr) < 0)
>  		return -ENODEV;
>  
> -	info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	info = kvmalloc(sizeof(*info), GFP_KERNEL);

The above is a bug. It converts kzalloc() to kvmalloc() instead of
kvzalloc().

-- Steve



>  	if (!info) {
>  		trace_array_put(tr);
>  		return -ENOMEM;
> @@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
>  	if (info->spare)
>  		ring_buffer_free_read_page(iter->trace_buffer->buffer,
>  					   info->spare_cpu, info->spare);
> -	kfree(info);
> +	kvfree(info);
>  
>  	mutex_unlock(&trace_types_lock);
>  

Re: [PATCH] trace : use kvmalloc instead of kmalloc

[Zhaoyang Huang]

On Thu, Jul 30, 2020 at 9:58 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 30 Jul 2020 19:04:12 +0800
> Zhaoyang Huang <huangzhaoyang@gmail.com> wrote:
>
> > High order memory stuff within trace could introduce OOM, use kvmalloc instead.
> >
> > Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low.
>
> Please limit your column width in the description of patches to 76
> characters.
>
> >
> > traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),  order=2, oom_score_adj=-1
> >
>
> What does this traced_probes thing do?
traced_probes is an android exe which reads ftrace and writes them to
files. I think kzalloc works fine for most of context but will cause
OOM in high ratio ftrace access in memory constraint system, like
mobilephone etc.
>
> > traced_probes cpuset=system-background mems_allowed=0
> > CPU: 3 PID: 588 Comm: traced_probes Tainted: G        W  O    4.14.181 #1
> > Hardware name: Generic DT based system
> > (unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
> > (show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
> > (dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
> > (dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
> > (oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
> > (out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
> > (__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
> > (kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
> > (kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
> > (tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
> > (do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
> > (vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
> > (path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
> > (do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
> > (do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
> > (SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)
> >
> > Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
> > ---
> > changes since v1: change kfree to kvfree
> > ---
> >  kernel/trace/trace.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> > index ca1ee65..8d70c79 100644
> > --- a/kernel/trace/trace.c
> > +++ b/kernel/trace/trace.c
> > @@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
> >       if (trace_array_get(tr) < 0)
> >               return -ENODEV;
> >
> > -     info = kzalloc(sizeof(*info), GFP_KERNEL);
> > +     info = kvmalloc(sizeof(*info), GFP_KERNEL);
>
> The above is a bug. It converts kzalloc() to kvmalloc() instead of
> kvzalloc().
fixed and resend with patch v2
>
> -- Steve
>
>
>
> >       if (!info) {
> >               trace_array_put(tr);
> >               return -ENOMEM;
> > @@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
> >       if (info->spare)
> >               ring_buffer_free_read_page(iter->trace_buffer->buffer,
> >                                          info->spare_cpu, info->spare);
> > -     kfree(info);
> > +     kvfree(info);
> >
> >       mutex_unlock(&trace_types_lock);
> >
>

Re: [PATCH] trace : use kvmalloc instead of kmalloc

[Steven Rostedt]

This is also the second version of the patch. Please add a v2 to the
subject, otherwise it gets confusing to know which one to pull in.

 "[PATCH v2] ..."

-- Steve


On Thu, 30 Jul 2020 19:04:12 +0800
Zhaoyang Huang <huangzhaoyang@gmail.com> wrote:

> High order memory stuff within trace could introduce OOM, use kvmalloc instead.
> 
> Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low. 
> 

[PATCH] trace : use kvmalloc instead of kmalloc

[Zhaoyang Huang]

High order memory stuff within trace could introduce OOM, use kvmalloc instead.

Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low. 

traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),  order=2, oom_score_adj=-1

traced_probes cpuset=system-background mems_allowed=0
CPU: 3 PID: 588 Comm: traced_probes Tainted: G        W  O    4.14.181 #1
Hardware name: Generic DT based system
(unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
(show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
(dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
(dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
(oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
(out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
(__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
(kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
(kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
(tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
(do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
(vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
(path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
(do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
(do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
(SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)

Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
---
 kernel/trace/trace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ca1ee65..d4eb7ea 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
 	if (trace_array_get(tr) < 0)
 		return -ENODEV;
 
-	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	info = kvmalloc(sizeof(*info), GFP_KERNEL);
 	if (!info) {
 		trace_array_put(tr);
 		return -ENOMEM;
-- 
1.9.1

Re: [PATCH] trace : use kvmalloc instead of kmalloc

[Michal Hocko]

On Wed 29-07-20 19:48:29, Zhaoyang Huang wrote:
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index ca1ee65..d4eb7ea 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
>  	if (trace_array_get(tr) < 0)
>  		return -ENODEV;
>  
> -	info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	info = kvmalloc(sizeof(*info), GFP_KERNEL);

I am not familiar with the code but it is quite clear that the patch is
incomplete without matching kvfree call in the release path. You cannot
simply change k.malloc to kvmalloc without the pairing kvfree.

>  	if (!info) {
>  		trace_array_put(tr);
>  		return -ENOMEM;
> -- 
> 1.9.1

-- 
Michal Hocko
SUSE Labs

[PATCH] trace : use kvmalloc instead of kmalloc

[Zhaoyang Huang]

High order memory stuff within trace could introduce OOM, use kvmalloc instead.

traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),  order=2, oom_score_adj=-1

traced_probes cpuset=system-background mems_allowed=0
CPU: 3 PID: 588 Comm: traced_probes Tainted: G        W  O    4.14.181 #1
Hardware name: Generic DT based system
(unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
(show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
(dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
(dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
(oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
(out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
(__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
(kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
(kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
(tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
(do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
(vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
(path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
(do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
(do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
(SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)

Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
---
 kernel/trace/trace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ca1ee65..d4eb7ea 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
 	if (trace_array_get(tr) < 0)
 		return -ENODEV;
 
-	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	info = kvmalloc(sizeof(*info), GFP_KERNEL);
 	if (!info) {
 		trace_array_put(tr);
 		return -ENOMEM;
-- 
1.9.1