Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

[KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>]

[-- Attachment #1: Type: text/plain, Size: 3387 bytes --]

> On 30/08/10 05:28 AM, KOSAKI Motohiro wrote:
> >> [snip]
> >>
> >> The interesting thing is that even though there is a lot of free memory at the
> >> end, it still hangs. 
> > 
> > Grr. I'm surprised this result ;-)
> > shrink_all_memory() finish to shrink memory successfully. but your
> > system still hang immediately after. I have no idea why this mysterious
> > occur. 
> > 
> > I prepared next debugging patch. It added prenty debug printk. I hope
> > it enlighten up which path makes system hang-up.
> > 
> > 1. apply my new patch
> > 
> > 2. Enable following PM debug option in Kconfig
> > 
> >   [*] Power Management support
> >   [*]   Power Management Debug Support
> >   [*]     Extra PM attributes in sysfs for low-level debugging/testing
> >   [*]     Verbose Power Management debugging
> > 
> > 3. append following kernel boot option into grub configration file
> > 
> > 	no_console_suspend=1
> > 
> > 3. kernel build and reboot
> > 4. some prepare
> >    # echo 8 > /proc/sysrq-trigger
> >    # cd /sys/power
> >    # echo 1 > pm_trace
> >    # echo 0 > pm_async
> > 
> > 5. run your test program
> > 
> > 
> >> I also included the timestamps; note the one and two second
> >> delays between the passes.
> > 
> > This is expected result because tmpfs shrink need swap-out. then
> > we need i/o time.
> > 
> >>
> >> Please let me know if there is anything I can do.
> > 
> > Please send me your .config and full dmesg.
> > 
> > 
> > Thanks many and many help us!
> 
> Hello,
> 
> I have followed your instructions, with one exception: I have also
> enabled CONFIG_PM_TRACE so that I would have /sys/power/pm_trace.
> 
> This time I had some more output, as expected. I double checked what
> I typed while looking at the screen-shot I took with my camera. Here's
> the output:
> 
> === 8< ===
> PM: Marking nosave pages: ...0009f000 - ...000100000
> PM: basic memory bitmaps created
> PM: Syncing filesystems ... done
> Freezing user space processes ... (elapsed 0.01 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
> PM: Preallocating image memory...
> shrink_all_memory start
> PM: shrink memory: pass=1, req:310171 reclaimed:15492 free:360936
> PM: shrink memory: pass=2, req:294679 reclaimed:28864 free:373981
> PM: shrink memory: pass=3, req:265815 reclaimed:60311 free:405374
> PM: shrink memory: pass=4, req:205504 reclaimed:97870 free:443024
> PM: shrink memory: pass=5, req:107634 reclaimed:146948 free:492141
> shrink_all_memory: req:107634 reclaimed:146948 free:492141
> PM: preallocate_image_highmem 556658 278329
> PM: preallocate_image_memory 103139 103139
> PM: preallocate_highmem_fraction 183908 556658 760831 -> 183908
> === >8 ===
> 
> According to your patch, the next output should have been
> "preallocate_image_memory ...", but it never gets printed, so the
> hang point should be that function.

Great!
I've attached more verbose debug message patch and trial bug fixing patch.
Could you please to try this?



> I am attaching my dmesg output which I got after the failed hibernation
> attempt and my .config file. Please note that the attached .config file
> is a trimmed version of the .config I usually use on my computer. I trimmed
> it so that it compiles faster, but (mostly) has support for devices I might
> use.
> 
> Thanks a lot for your help, and please let me know if I can do anything else.
> 


[-- Attachment #2: 0001-debug2.patch --]
[-- Type: application/octet-stream, Size: 5261 bytes --]

From c3a93b865e76afd02bf3856c63c50dd9493afafd Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Date: Tue, 31 Aug 2010 13:10:15 +0900
Subject: [PATCH 1/2] debug2

---
 kernel/power/snapshot.c |   66 ++++++++++++++++++++++++++++------------------
 1 files changed, 40 insertions(+), 26 deletions(-)

diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c
index c864806..3144ab4 100644
--- a/kernel/power/snapshot.c
+++ b/kernel/power/snapshot.c
@@ -1258,10 +1258,13 @@ static unsigned long minimum_image_size(unsigned long saveable)
 int hibernate_preallocate_memory(void)
 {
 	struct zone *zone;
-	unsigned long saveable, size, max_size, count, highmem, pages = 0;
+	unsigned long saveable, max_size, count, highmem, pages = 0;
 	unsigned long alloc, save_highmem, pages_highmem;
 	struct timeval start, stop;
 	int error;
+	unsigned long additional_size, img_size;
+	unsigned long pages_fraction;
+	unsigned long img_pages;
 
 	printk(KERN_INFO "PM: Preallocating image memory... \n");
 	do_gettimeofday(&start);
@@ -1288,9 +1291,9 @@ int hibernate_preallocate_memory(void)
 	count = saveable;
 	saveable += save_highmem;
 	highmem = save_highmem;
-	size = 0;
+	additional_size = 0;
 	for_each_populated_zone(zone) {
-		size += snapshot_additional_pages(zone);
+		additional_size += snapshot_additional_pages(zone);
 		if (is_highmem(zone))
 			highmem += zone_page_state(zone, NR_FREE_PAGES);
 		else
@@ -1299,16 +1302,23 @@ int hibernate_preallocate_memory(void)
 	count += highmem;
 	count -= totalreserve_pages;
 
+	printk(KERN_INFO "PM: save_highmem(%lu), saveable(%lu) count=(%lu)\n",
+	       save_highmem, saveable, count);
+
 	/* Compute the maximum number of saveable pages to leave in memory. */
-	max_size = (count - (size + PAGES_FOR_IO)) / 2 - 2 * SPARE_PAGES;
-	size = DIV_ROUND_UP(image_size, PAGE_SIZE);
-	if (size > max_size)
-		size = max_size;
+	max_size = (count - (additional_size + PAGES_FOR_IO)) / 2 - 2 * SPARE_PAGES;
+	img_size = DIV_ROUND_UP(image_size, PAGE_SIZE);
+	if (img_size > max_size)
+		img_size = max_size;
+
+	printk(KERN_INFO "PM: max_size(%lu), image_size(%lu), img_size(%lu)\n",
+	       max_size, image_size, img_size);
+
 	/*
 	 * If the maximum is not less than the current number of saveable pages
 	 * in memory, allocate page frames for the image and we're done.
 	 */
-	if (size >= saveable) {
+	if (img_size >= saveable) {
 		printk(KERN_INFO "PM: size >= saveable. skip \n");
 		pages = preallocate_image_highmem(save_highmem);
 		pages += preallocate_image_memory(saveable - pages);
@@ -1316,9 +1326,12 @@ int hibernate_preallocate_memory(void)
 	}
 
 	/* Estimate the minimum size of the image. */
-	pages = minimum_image_size(saveable);
-	if (size < pages)
-		size = min_t(unsigned long, pages, max_size);
+	img_pages = minimum_image_size(saveable);
+	if (img_size < img_pages)
+		img_size = min_t(unsigned long, img_pages, max_size);
+
+	printk(KERN_INFO "PM: img_pages(%lu), img_size(%lu)\n",
+	       img_pages, img_size);
 
 	/*
 	 * Let the memory management subsystem know that we're going to need a
@@ -1326,7 +1339,7 @@ int hibernate_preallocate_memory(void)
 	 * NOTE: If this is not done, performance will be hurt badly in some
 	 * test cases.
 	 */
-	shrink_all_memory(saveable - size);
+	shrink_all_memory(saveable - img_size);
 
 	/*
 	 * The number of saveable pages in memory was too high, so apply some
@@ -1336,30 +1349,31 @@ int hibernate_preallocate_memory(void)
 	 * highmem and non-highmem zones separately.
 	 */
 	pages_highmem = preallocate_image_highmem(highmem / 2);
-	printk(KERN_INFO "PM: preallocate_image_highmem %lu %lu\n",
-	       highmem, pages_highmem);
+	printk(KERN_INFO "PM: preallocate_image_highmem %lu %lu free(%u/%lu)\n",
+	       highmem, pages_highmem, nr_free_highpages(), global_page_state(NR_FREE_PAGES));
 
 	alloc = (count - max_size) - pages_highmem;
 	pages = preallocate_image_memory(alloc);
-	printk(KERN_INFO "PM: preallocate_image_memory %lu %lu\n",
-	       alloc, pages);
+	printk(KERN_INFO "PM: preallocate_image_memory %lu %lu free(%u/%lu)\n",
+	       alloc, pages, nr_free_highpages(), global_page_state(NR_FREE_PAGES));
 
 	if (pages < alloc) {
 		printk(KERN_INFO "PM: pages(%lu) < alloc(%lu). error. \n",
 		       pages, alloc);
 		goto err_out;
 	}
-	size = max_size - size;
-	alloc = size;
-	size = preallocate_highmem_fraction(size, highmem, count);
-	printk(KERN_INFO "PM: preallocate_highmem_fraction %lu %lu %lu -> %lu\n",
-	       size, highmem, count, size);
-
-	pages_highmem += size;
-	alloc -= size;
+
+	alloc = max_size - img_size;
+
+	pages_fraction = preallocate_highmem_fraction(alloc, highmem, count);
+	printk(KERN_INFO "PM: preallocate_highmem_fraction %lu %lu %lu -> %lu free(%u/%lu)\n",
+	       alloc, highmem, count, pages_fraction, nr_free_highpages(), global_page_state(NR_FREE_PAGES));
+
+	pages_highmem += pages_fraction;
+	alloc -= pages_fraction;
 	pages += preallocate_image_memory(alloc);
-	printk(KERN_INFO "PM: preallocate_image_memory %lu %lu\n",
-	       alloc, pages);
+	printk(KERN_INFO "PM: preallocate_image_memory %lu %lu free(%u/%lu)\n",
+	       alloc, pages, nr_free_highpages(), global_page_state(NR_FREE_PAGES));
 	pages += pages_highmem;
 
 	/*
-- 
1.6.5.2


[-- Attachment #3: 0002-add-gfp_noretry.patch --]
[-- Type: application/octet-stream, Size: 825 bytes --]

From 561bf4eed48ce99bd8d8b933390603f84ac4cacb Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Date: Tue, 31 Aug 2010 14:58:04 +0900
Subject: [PATCH 2/2] add gfp_noretry

Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
---
 kernel/power/snapshot.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c
index 3144ab4..c488704 100644
--- a/kernel/power/snapshot.c
+++ b/kernel/power/snapshot.c
@@ -1091,7 +1091,7 @@ void swsusp_free(void)
 
 /* Helper functions used for the shrinking of memory. */
 
-#define GFP_IMAGE	(GFP_KERNEL | __GFP_NOWARN)
+#define GFP_IMAGE	(GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY)
 
 /**
  * preallocate_image_pages - Allocate a number of pages for hibernation image
-- 
1.6.5.2