[patch] oom: dump stack and VM state when oom killer panics

[patch] oom: dump stack and VM state when oom killer panics

[David Rientjes]

The oom killer header, including information such as the allocation order
and gfp mask, current's cpuset and memory controller, call trace, and VM
state information is currently only shown when the oom killer has
selected a task to kill.

This information is omitted, however, when the oom killer panics either
because of panic_on_oom sysctl settings or when no killable task was
found.  It is still relevant to know crucial pieces of information such
as the allocation order and VM state when diagnosing such issues,
especially at boot.

This patch displays the oom killer header whenever it panics so that bug
reports can include pertinent information to debug the issue, if
possible.

Signed-off-by: David Rientjes <rientjes@google.com>
---
 mm/oom_kill.c |   45 +++++++++++++++++++++++++++------------------
 1 files changed, 27 insertions(+), 18 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -310,6 +310,23 @@ static void dump_tasks(const struct mem_cgroup *mem)
 	} while_each_thread(g, p);
 }
 
+static void dump_header(gfp_t gfp_mask, int order, struct mem_cgroup *mem)
+{
+	if (printk_ratelimit()) {
+		printk(KERN_WARNING "%s invoked oom-killer: "
+			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",
+			current->comm, gfp_mask, order, current->oomkilladj);
+		task_lock(current);
+		cpuset_print_task_mems_allowed(current);
+		task_unlock(current);
+		dump_stack();
+		mem_cgroup_print_oom_info(mem, current);
+		show_mem();
+		if (sysctl_oom_dump_tasks)
+			dump_tasks(mem);
+	}
+}
+
 /*
  * Send SIGKILL to the selected  process irrespective of  CAP_SYS_RAW_IO
  * flag though it's unlikely that  we select a process with CAP_SYS_RAW_IO
@@ -392,20 +409,7 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
 {
 	struct task_struct *c;
 
-	if (printk_ratelimit()) {
-		printk(KERN_WARNING "%s invoked oom-killer: "
-			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",
-			current->comm, gfp_mask, order, current->oomkilladj);
-		task_lock(current);
-		cpuset_print_task_mems_allowed(current);
-		task_unlock(current);
-		dump_stack();
-		mem_cgroup_print_oom_info(mem, current);
-		show_mem();
-		if (sysctl_oom_dump_tasks)
-			dump_tasks(mem);
-	}
-
+	dump_header(gfp_mask, order, mem);
 	/*
 	 * If the task is already exiting, don't alarm the sysadmin or kill
 	 * its children or threads, just set TIF_MEMDIE so it can die quickly
@@ -540,6 +544,7 @@ retry:
 	/* Found nothing?!?! Either we hang forever, or we panic. */
 	if (!p) {
 		read_unlock(&tasklist_lock);
+		dump_header(gfp_mask, order, NULL);
 		panic("Out of memory and no killable processes...\n");
 	}
 
@@ -605,8 +610,10 @@ void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
 		/* Got some memory back in the last second. */
 		return;
 
-	if (sysctl_panic_on_oom == 2)
-		panic("out of memory. Compulsory panic_on_oom is selected.\n");
+	if (sysctl_panic_on_oom == 2) {
+		dump_header(gfp_mask, order, NULL);
+		panic("Out of memory. Compulsory panic_on_oom is selected.\n");
+	}
 
 	/*
 	 * Check if there were limitations on the allocation (only relevant for
@@ -622,8 +629,10 @@ void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
 		break;
 
 	case CONSTRAINT_NONE:
-		if (sysctl_panic_on_oom)
-			panic("out of memory. panic_on_oom is selected\n");
+		if (sysctl_panic_on_oom) {
+			dump_header(gfp_mask, order, NULL);
+			panic("Out of memory. panic_on_oom is selected.\n");
+		}
 		/* Fall-through */
 	case CONSTRAINT_CPUSET:
 		__out_of_memory(gfp_mask, order);

Re: [patch] oom: dump stack and VM state when oom killer panics

[KAMEZAWA Hiroyuki]

On Thu, 17 Sep 2009 13:55:22 -0700 (PDT)
David Rientjes <rientjes@google.com> wrote:

> The oom killer header, including information such as the allocation order
> and gfp mask, current's cpuset and memory controller, call trace, and VM
> state information is currently only shown when the oom killer has
> selected a task to kill.
> 
> This information is omitted, however, when the oom killer panics either
> because of panic_on_oom sysctl settings or when no killable task was
> found.  It is still relevant to know crucial pieces of information such
> as the allocation order and VM state when diagnosing such issues,
> especially at boot.
> 
> This patch displays the oom killer header whenever it panics so that bug
> reports can include pertinent information to debug the issue, if
> possible.
> 
> Signed-off-by: David Rientjes <rientjes@google.com>

Seems nice.
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>


> ---
>  mm/oom_kill.c |   45 +++++++++++++++++++++++++++------------------
>  1 files changed, 27 insertions(+), 18 deletions(-)
> 
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -310,6 +310,23 @@ static void dump_tasks(const struct mem_cgroup *mem)
>  	} while_each_thread(g, p);
>  }
>  
> +static void dump_header(gfp_t gfp_mask, int order, struct mem_cgroup *mem)
> +{
> +	if (printk_ratelimit()) {
> +		printk(KERN_WARNING "%s invoked oom-killer: "
> +			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",
> +			current->comm, gfp_mask, order, current->oomkilladj);
> +		task_lock(current);
> +		cpuset_print_task_mems_allowed(current);
> +		task_unlock(current);
> +		dump_stack();
> +		mem_cgroup_print_oom_info(mem, current);
> +		show_mem();
> +		if (sysctl_oom_dump_tasks)
> +			dump_tasks(mem);
> +	}
> +}
> +
>  /*
>   * Send SIGKILL to the selected  process irrespective of  CAP_SYS_RAW_IO
>   * flag though it's unlikely that  we select a process with CAP_SYS_RAW_IO
> @@ -392,20 +409,7 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
>  {
>  	struct task_struct *c;
>  
> -	if (printk_ratelimit()) {
> -		printk(KERN_WARNING "%s invoked oom-killer: "
> -			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",
> -			current->comm, gfp_mask, order, current->oomkilladj);
> -		task_lock(current);
> -		cpuset_print_task_mems_allowed(current);
> -		task_unlock(current);
> -		dump_stack();
> -		mem_cgroup_print_oom_info(mem, current);
> -		show_mem();
> -		if (sysctl_oom_dump_tasks)
> -			dump_tasks(mem);
> -	}
> -
> +	dump_header(gfp_mask, order, mem);
>  	/*
>  	 * If the task is already exiting, don't alarm the sysadmin or kill
>  	 * its children or threads, just set TIF_MEMDIE so it can die quickly
> @@ -540,6 +544,7 @@ retry:
>  	/* Found nothing?!?! Either we hang forever, or we panic. */
>  	if (!p) {
>  		read_unlock(&tasklist_lock);
> +		dump_header(gfp_mask, order, NULL);
>  		panic("Out of memory and no killable processes...\n");
>  	}
>  
> @@ -605,8 +610,10 @@ void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
>  		/* Got some memory back in the last second. */
>  		return;
>  
> -	if (sysctl_panic_on_oom == 2)
> -		panic("out of memory. Compulsory panic_on_oom is selected.\n");
> +	if (sysctl_panic_on_oom == 2) {
> +		dump_header(gfp_mask, order, NULL);
> +		panic("Out of memory. Compulsory panic_on_oom is selected.\n");
> +	}
>  
>  	/*
>  	 * Check if there were limitations on the allocation (only relevant for
> @@ -622,8 +629,10 @@ void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
>  		break;
>  
>  	case CONSTRAINT_NONE:
> -		if (sysctl_panic_on_oom)
> -			panic("out of memory. panic_on_oom is selected\n");
> +		if (sysctl_panic_on_oom) {
> +			dump_header(gfp_mask, order, NULL);
> +			panic("Out of memory. panic_on_oom is selected.\n");
> +		}
>  		/* Fall-through */
>  	case CONSTRAINT_CPUSET:
>  		__out_of_memory(gfp_mask, order);
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [patch] oom: dump stack and VM state when oom killer panics

[KOSAKI Motohiro]

Hello

> The oom killer header, including information such as the allocation order
> and gfp mask, current's cpuset and memory controller, call trace, and VM
> state information is currently only shown when the oom killer has
> selected a task to kill.
> 
> This information is omitted, however, when the oom killer panics either
> because of panic_on_oom sysctl settings or when no killable task was
> found.  It is still relevant to know crucial pieces of information such
> as the allocation order and VM state when diagnosing such issues,
> especially at boot.
> 
> This patch displays the oom killer header whenever it panics so that bug
> reports can include pertinent information to debug the issue, if
> possible.
> 
> Signed-off-by: David Rientjes <rientjes@google.com>

Good patch :)

I have one request. In panic case, I don't hope to call printk_ratelimit().
the infomation should be displayed always.

Thanks.

Re: [patch] oom: dump stack and VM state when oom killer panics

[Andrew Morton]

On Fri, 18 Sep 2009 09:26:37 +0900 (JST)
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> wrote:

> Hello
> 
> > The oom killer header, including information such as the allocation order
> > and gfp mask, current's cpuset and memory controller, call trace, and VM
> > state information is currently only shown when the oom killer has
> > selected a task to kill.
> > 
> > This information is omitted, however, when the oom killer panics either
> > because of panic_on_oom sysctl settings or when no killable task was
> > found.  It is still relevant to know crucial pieces of information such
> > as the allocation order and VM state when diagnosing such issues,
> > especially at boot.
> > 
> > This patch displays the oom killer header whenever it panics so that bug
> > reports can include pertinent information to debug the issue, if
> > possible.
> > 
> > Signed-off-by: David Rientjes <rientjes@google.com>
> 
> Good patch :)
> 
> I have one request. In panic case, I don't hope to call printk_ratelimit().
> the infomation should be displayed always.
> 

The code shouldn't be using printk_ratelimit() - in fact nothing should
be using it and we should remove printk_ratelimit() from the kernel, I
suspect.

Because it uses global state, so a printk_ratelimit() in some random
net driver could cause printks from the oom-killer to be suppressed.

Switching to __ratelimit() would fix such interactions.

Re: [patch] oom: dump stack and VM state when oom killer panics

[Joe Perches]

On Mon, 2009-09-21 at 13:41 -0700, Andrew Morton wrote:
> The code shouldn't be using printk_ratelimit() - in fact nothing should
> be using it and we should remove printk_ratelimit() from the kernel, I
> suspect.

Perhaps a per instance use instead.  Something like:

#define printk_ratelimited(fmt, arg...)			\
({	static struct ratelimit_state _rs = {		\
		.interval = DEFAULT_RATELIMIT_INTERVAL,	\
		.burst = DEFAULT_RATELIMIT_BURST,	\
	};						\
	int rtn;					\
							\
	if (!__ratelimit(&_rs))				\
		rtn = printk(fmt, ##arg);		\
	else						\
		rtn = 0;				\
	rtn;						\
})

#define pr_info_rl(fmt, arg) \
	printk_ratelimited(KERN_INFO pr_fmt(fmt), ##arg)
etc...