RFC for 2.6: avoid OOM at bounce buffer storm

RFC for 2.6: avoid OOM at bounce buffer storm

[Martin Wilck]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]

Hello,

I have recently seen massive problems with the bounce buffer allocation 
implementation in 2.6. My system got OOM with an ata_piix controller 
(4GB bounce limit), 2 P4 CPUs and 8GB of RAM with a very simple 
copy-compare test: About 800MB worth of data are copied with cp(1) in 7 
synchronous jobs. Each job does a sync(1) after the copy.

All available ZONE_NORMAL memory is filled with bounce buffers in 
fractions of a second after the sync(1) calls are issued.
The system goes OOM (becomes unusable, OOM killer strikes several times 
against innocent processes).

The attached quick-and-dirty patch gives you an idea of what I figured 
could be a useful workaround for that problem. It follows 2 main ideas:

1 When bounce buffer allocations fail, it is wrong to trigger the page
   reclaim mechanism which is likely to genetate even more bounce buffer
   requests. Thus there should be no wakeup_bdflush() call for the
   page_pool.

2 bounce buffer allocations should not simply use alloc_page() because
   there is no limit for allocations that way. Rather, bounce buffer
   allocations should stop if ZONE_NORMAL is full to a certain degree
   (my patch stops at 51% - chosen arbitrarily).

For the case I described, the system behavior is clearly improved with 
the patch. The system remains usable, and the OOM killer isn't triggered.

(The patch is against an RH enterprise kernel, but it applies against 
almost all recent 2.6.kernels except 2.6.12-rcX).

Idea 1) is already implemented in a recent patch by Nick Piggin which 
went into 2.6.12-rc5. I haven't tested yet if that patch alone fixes my 
problem because I couldn't get 2.6.12-rc5 to run on my system. I doubt 
so, though, because I started with idea 1) alone and it didn't help much.

Here is another idea how the bounce buffer behavior could be enhanced:
Introduce a new memory zone ZONE_4G with memory from 896M-4096M
and allocate bounce buffers preferably in that zone. Advantage: Bounce 
buffers would't interfere with valuable ZONE_NORMAL memory. But that may 
be too much effort just for systems which have too much memory for their 
IO controllers...

Regards
Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy


[-- Attachment #2: bounce.diff --]
[-- Type: text/x-patch, Size: 4078 bytes --]

diff -rupX /root/diff-exludes linux-2.6.9-6.37.EL-orig/arch/i386/mm/pgtable.c linux-2.6.9-6.37.EL/arch/i386/mm/pgtable.c
--- linux-2.6.9-6.37.EL-orig/arch/i386/mm/pgtable.c	2005-04-11 14:19:03.000000000 +0200
+++ linux-2.6.9-6.37.EL/arch/i386/mm/pgtable.c	2005-05-31 10:27:57.000000000 +0200
@@ -24,6 +24,7 @@
 #include <asm/tlbflush.h>
 #include <asm/atomic_kmap.h>
 
+extern atomic_t n_bounce_pages;
 void show_mem(void)
 {
 	int total = 0, reserved = 0;
@@ -55,6 +56,7 @@ void show_mem(void)
 	printk("%d reserved pages\n",reserved);
 	printk("%d pages shared\n",shared);
 	printk("%d pages swap cached\n",cached);
+	printk("%d bounce buffers used\n",atomic_read(&n_bounce_pages));
 }
 
 EXPORT_SYMBOL_GPL(show_mem);
diff -rupX /root/diff-exludes linux-2.6.9-6.37.EL-orig/Makefile linux-2.6.9-6.37.EL/Makefile
--- linux-2.6.9-6.37.EL-orig/mm/highmem.c	2004-10-18 23:54:37.000000000 +0200
+++ linux-2.6.9-6.37.EL/mm/highmem.c	2005-06-03 15:24:03.000000000 +0200
@@ -28,18 +28,45 @@
 #include <linux/highmem.h>
 #include <asm/tlbflush.h>
 
-static mempool_t *page_pool, *isa_page_pool;
+mempool_t *page_pool, *isa_page_pool;
+atomic_t n_bounce_pages = ATOMIC_INIT(0);
+static unsigned int min_free_pages[MAX_NR_ZONES];
+
+static unsigned int zone_free_pages(int type)
+{
+	pg_data_t *pgdat;
+	unsigned int pages = 0;
+	for_each_pgdat(pgdat)
+		pages += pgdat->node_zones[type].free_pages;
+	return pages;
+}
+
+static int sufficient_lowpages(int gfp)
+{
+	int type = (gfp & __GFP_DMA ? ZONE_DMA : ZONE_NORMAL);
+
+	return (zone_free_pages(type) >= min_free_pages[type]);
+}
 
 static void *page_pool_alloc(int gfp_mask, void *data)
 {
 	int gfp = gfp_mask | (int) (long) data;
+	void *page;
+
+	if (!sufficient_lowpages(gfp))
+		return NULL;
 
-	return alloc_page(gfp);
+	page = alloc_page(gfp);
+
+	if (page)
+		atomic_inc(&n_bounce_pages);
+	return page;
 }
 
 static void page_pool_free(void *page, void *data)
 {
 	__free_page(page);
+	atomic_dec(&n_bounce_pages);
 }
 
 /*
@@ -212,20 +239,25 @@ void fastcall kunmap_high(struct page *p
 EXPORT_SYMBOL(kunmap_high);
 
 #define POOL_SIZE	64
+#define POOL_FRAC(x) ((x)/2)
+#define SMALL_POOL_FRAC(x) ((x)/100)
 
 static __init int init_emergency_pool(void)
 {
+	unsigned int n = zone_free_pages(ZONE_NORMAL);
 	struct sysinfo i;
 	si_meminfo(&i);
 	si_swapinfo(&i);
         
+	min_free_pages[ZONE_NORMAL] = POOL_FRAC(n);
 	if (!i.totalhigh)
 		return 0;
-
-	page_pool = mempool_create(POOL_SIZE, page_pool_alloc, page_pool_free, NULL);
+	n = SMALL_POOL_FRAC(n);
+	n = (n > POOL_SIZE ? n : POOL_SIZE);
+	page_pool = mempool_create(n, page_pool_alloc, page_pool_free, NULL);
 	if (!page_pool)
 		BUG();
-	printk("highmem bounce pool size: %d pages\n", POOL_SIZE);
+	printk("highmem bounce pool size: %d pages, min: %u\n", n, min_free_pages[ZONE_NORMAL]);
 
 	return 0;
 }
@@ -265,11 +297,12 @@ int init_emergency_isa_pool(void)
 	if (isa_page_pool)
 		return 0;
 
+	min_free_pages[ZONE_DMA] = POOL_FRAC(zone_free_pages(ZONE_DMA));
 	isa_page_pool = mempool_create(ISA_POOL_SIZE, page_pool_alloc, page_pool_free, (void *) __GFP_DMA);
 	if (!isa_page_pool)
 		BUG();
 
-	printk("isa bounce pool size: %d pages\n", ISA_POOL_SIZE);
+	printk("isa bounce pool size: %d pages, min: %u\n", ISA_POOL_SIZE, min_free_pages[ZONE_DMA]);
 	return 0;
 }
 
diff -rupX /root/diff-exludes linux-2.6.9-6.37.EL-orig/mm/mempool.c linux-2.6.9-6.37.EL/mm/mempool.c
--- linux-2.6.9-6.37.EL-orig/mm/mempool.c	2004-10-18 23:54:37.000000000 +0200
+++ linux-2.6.9-6.37.EL/mm/mempool.c	2005-06-03 15:35:01.000000000 +0200
@@ -15,6 +15,8 @@
 #include <linux/blkdev.h>
 #include <linux/writeback.h>
 
+extern mempool_t *page_pool, *isa_page_pool;
+
 static void add_element(mempool_t *pool, void *element)
 {
 	BUG_ON(pool->curr_nr >= pool->min_nr);
@@ -211,11 +213,12 @@ repeat_alloc:
 		if (likely(element != NULL))
 			return element;
 	}
-
+	
 	/*
 	 * Kick the VM at this point.
 	 */
-	wakeup_bdflush(0);
+	if (pool != page_pool && pool != isa_page_pool)
+		wakeup_bdflush(0);
 
 	spin_lock_irqsave(&pool->lock, flags);
 	if (likely(pool->curr_nr)) {

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Andrew Morton]

Martin Wilck <martin.wilck@fujitsu-siemens.com> wrote:
>
> Hello,
> 
> I have recently seen massive problems with the bounce buffer allocation 
> implementation in 2.6. My system got OOM with an ata_piix controller 
> (4GB bounce limit), 2 P4 CPUs and 8GB of RAM with a very simple 
> copy-compare test: About 800MB worth of data are copied with cp(1) in 7 
> synchronous jobs. Each job does a sync(1) after the copy.

Fair enough.

> ...
> 
> The attached quick-and-dirty patch gives you an idea of what I figured 
> could be a useful workaround for that problem. It follows 2 main ideas:
> 
> 1 When bounce buffer allocations fail, it is wrong to trigger the page
>    reclaim mechanism which is likely to genetate even more bounce buffer
>    requests. Thus there should be no wakeup_bdflush() call for the
>    page_pool.
> 
> 2 bounce buffer allocations should not simply use alloc_page() because
>    there is no limit for allocations that way. Rather, bounce buffer
>    allocations should stop if ZONE_NORMAL is full to a certain degree
>    (my patch stops at 51% - chosen arbitrarily).

It might be neater to do this at the mempool level: that way we're adding
general-purpose infrastructure and then just using it, rather than
special-casing the bounce code.

See below a (n untested) patch against the latest devel tree.  It won't be
stunningly scalable on big SMP, but the overhead of bouncing will probably
hide that.


> I couldn't get 2.6.12-rc5 to run on my system.

Ow.  Could you please investigate further?  Any boot messages for us to
see?  it's quite possibly some missing config option..



 include/linux/mempool.h |   17 +++++++++++++++--
 mm/highmem.c            |    4 +++-
 mm/mempool.c            |   17 +++++++++++++----
 3 files changed, 31 insertions(+), 7 deletions(-)

diff -puN include/linux/mempool.h~mempool-bounce-buffer-restriction include/linux/mempool.h
--- 25/include/linux/mempool.h~mempool-bounce-buffer-restriction	Fri Jun  3 16:01:58 2005
+++ 25-akpm/include/linux/mempool.h	Fri Jun  3 16:01:58 2005
@@ -5,6 +5,7 @@
 #define _LINUX_MEMPOOL_H
 
 #include <linux/wait.h>
+#include <asm/semaphore.h>
 
 typedef void * (mempool_alloc_t)(unsigned int __nocast gfp_mask, void *pool_data);
 typedef void (mempool_free_t)(void *element, void *pool_data);
@@ -14,14 +15,26 @@ typedef struct mempool_s {
 	int min_nr;		/* nr of elements at *elements */
 	int curr_nr;		/* Current nr of elements at *elements */
 	void **elements;
+	int limit;
 
 	void *pool_data;
 	mempool_alloc_t *alloc;
 	mempool_free_t *free;
 	wait_queue_head_t wait;
+	struct semaphore limit_sem;
 } mempool_t;
-extern mempool_t * mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
-				 mempool_free_t *free_fn, void *pool_data);
+
+extern mempool_t *mempool_create_restricted(int min_nr,
+	mempool_alloc_t *alloc_fn, mempool_free_t *free_fn,
+	void *pool_data, int limit);
+
+static inline mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
+			mempool_free_t *free_fn, void *pool_data)
+{
+	return mempool_create_restricted(min_nr, alloc_fn, free_fn,
+					pool_data, 0);
+}
+
 extern int mempool_resize(mempool_t *pool, int new_min_nr, unsigned int __nocast gfp_mask);
 extern void mempool_destroy(mempool_t *pool);
 extern void * mempool_alloc(mempool_t *pool, unsigned int __nocast gfp_mask);
diff -puN mm/highmem.c~mempool-bounce-buffer-restriction mm/highmem.c
--- 25/mm/highmem.c~mempool-bounce-buffer-restriction	Fri Jun  3 16:01:58 2005
+++ 25-akpm/mm/highmem.c	Fri Jun  3 16:01:58 2005
@@ -224,7 +224,9 @@ static __init int init_emergency_pool(vo
 	if (!i.totalhigh)
 		return 0;
 
-	page_pool = mempool_create(POOL_SIZE, page_pool_alloc, page_pool_free, NULL);
+	page_pool = mempool_create_restricted(POOL_SIZE, page_pool_alloc,
+				page_pool_free, NULL,
+				200 * 1024 * 1024 / PAGE_SIZE);
 	if (!page_pool)
 		BUG();
 	printk("highmem bounce pool size: %d pages\n", POOL_SIZE);
diff -puN mm/mempool.c~mempool-bounce-buffer-restriction mm/mempool.c
--- 25/mm/mempool.c~mempool-bounce-buffer-restriction	Fri Jun  3 16:01:58 2005
+++ 25-akpm/mm/mempool.c	Fri Jun  3 16:01:58 2005
@@ -38,12 +38,13 @@ static void free_pool(mempool_t *pool)
 }
 
 /**
- * mempool_create - create a memory pool
+ * mempool_create_restricted - create a memory pool
  * @min_nr:    the minimum number of elements guaranteed to be
  *             allocated for this pool.
  * @alloc_fn:  user-defined element-allocation function.
  * @free_fn:   user-defined element-freeing function.
  * @pool_data: optional private data available to the user-defined functions.
+ * @limit:     maximum number of in-flight objects
  *
  * this function creates and allocates a guaranteed size, preallocated
  * memory pool. The pool can be used from the mempool_alloc and mempool_free
@@ -51,8 +52,8 @@ static void free_pool(mempool_t *pool)
  * functions might sleep - as long as the mempool_alloc function is not called
  * from IRQ contexts.
  */
-mempool_t * mempool_create(int min_nr, mempool_alloc_t *alloc_fn,
-				mempool_free_t *free_fn, void *pool_data)
+mempool_t *mempool_create_restricted(int min_nr, mempool_alloc_t *alloc_fn,
+			mempool_free_t *free_fn, void *pool_data, int limit)
 {
 	mempool_t *pool;
 
@@ -71,6 +72,8 @@ mempool_t * mempool_create(int min_nr, m
 	init_waitqueue_head(&pool->wait);
 	pool->alloc = alloc_fn;
 	pool->free = free_fn;
+	pool->limit = limit;
+	sema_init(&pool->limit_sem, limit);
 
 	/*
 	 * First pre-allocate the guaranteed number of buffers.
@@ -87,7 +90,7 @@ mempool_t * mempool_create(int min_nr, m
 	}
 	return pool;
 }
-EXPORT_SYMBOL(mempool_create);
+EXPORT_SYMBOL(mempool_create_restricted);
 
 /**
  * mempool_resize - resize an existing memory pool
@@ -208,6 +211,9 @@ void * mempool_alloc(mempool_t *pool, un
 
 	gfp_temp = gfp_mask & ~(__GFP_WAIT|__GFP_IO);
 
+	if (unlikely(pool->limit))
+		down(&pool->limit_sem);
+
 repeat_alloc:
 
 	element = pool->alloc(gfp_temp, pool->pool_data);
@@ -250,6 +256,9 @@ void mempool_free(void *element, mempool
 {
 	unsigned long flags;
 
+	if (unlikely(pool->limit))
+		up(&pool->limit_sem);
+
 	smp_mb();
 	if (pool->curr_nr < pool->min_nr) {
 		spin_lock_irqsave(&pool->lock, flags);
_

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Martin Wilck]

Andrew Morton wrote:

> It might be neater to do this at the mempool level: that way we're adding
> general-purpose infrastructure and then just using it, rather than
> special-casing the bounce code.
> 
> See below a (n untested) patch against the latest devel tree.  It won't be
> stunningly scalable on big SMP, but the overhead of bouncing will probably
> hide that.

I don't quite understand your patch. You introduce a "limit" field but 
you never actually use it. You also don't count the allocated pages.
Are you using the semaphore for slowing things down on purpose?

(Note that the problem is not in the mempool allocation itself but in 
the "normal" allocation path (page_pool_alloc() -> alloc_page()))

Anyway, I think could figure out your patch but with 2.6.12-rc5-mm2 I 
couldn't reproduce the problem any more. It appears to run much more 
smoothly now, perhaps because wakeup_bdflush() isn't called any more. 
Are you still interested in more data?

>>I couldn't get 2.6.12-rc5 to run on my system.
> 
> Ow.  Could you please investigate further?  Any boot messages for us to
> see?  it's quite possibly some missing config option..

It turned out to be a problem with Red Hat's nash that didn't check the 
returned pid in it's wait4() call and thus ended up insmod'ing mutliple 
modules simultaneously, leading to "Unkown symbol" errors. Yuck, it took 
me a day figure that out.

That bug is fixed in redhat's "mkinitrd" package 4.2.0.3-1 and later, 
but that package is currently only in Fedora's "Development" tree.

Thanks,
Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Andrew Morton]

Martin Wilck <martin.wilck@fujitsu-siemens.com> wrote:
>
> > It might be neater to do this at the mempool level: that way we're adding
>  > general-purpose infrastructure and then just using it, rather than
>  > special-casing the bounce code.
>  > 
>  > See below a (n untested) patch against the latest devel tree.  It won't be
>  > stunningly scalable on big SMP, but the overhead of bouncing will probably
>  > hide that.
> 
>  I don't quite understand your patch. You introduce a "limit" field but 
>  you never actually use it. You also don't count the allocated pages.
>  Are you using the semaphore for slowing things down on purpose?

The semaphore is initialised with the limit level, so once it has been
down()ed more than `limit' times, processes will block until someone does
up().

>  (Note that the problem is not in the mempool allocation itself but in 
>  the "normal" allocation path (page_pool_alloc() -> alloc_page()))

yup.  The semaphore will prevent more than `limit' pages being allocated at
any point in time.

>  Anyway, I think could figure out your patch but with 2.6.12-rc5-mm2 I 
>  couldn't reproduce the problem any more.

Oh bugger.

> It appears to run much more 
>  smoothly now, perhaps because wakeup_bdflush() isn't called any more. 
>  Are you still interested in more data?

Perhaps the newer kernel has writeback thresholding fixes so it's not
possible to dirty as much memory with write().

You can probably trigger the same problem if the memory is instead dirtied
with mmap(MAP_SHARED).

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Martin Wilck]

Hello Andrew,

> The semaphore is initialised with the limit level, so once it has been
> down()ed more than `limit' times, processes will block until someone does
> up().

Oh - of course. Neat.

>>It appears to run much more 
>> smoothly now, perhaps because wakeup_bdflush() isn't called any more. 
>> Are you still interested in more data?
> 
> Perhaps the newer kernel has writeback thresholding fixes so it's not
> possible to dirty as much memory with write().

I have collected more data and the behavior with 2.6.12-rc5-mm2 is 
flawless, there is a continuous writeback flow close to the maximum rate 
possible, and the bounce buffer usage never gets anywhere near the limit 
where it'd become dangerous. At least not in my test setup. The latest 
fedora kernel 2.6.11-1.27 also behaves ok, although it doesn't adapt to 
changing io load as smoothly as 2.6.12-rc5-mm2 does, and the writeback 
rate is oscillating more strongly.

The kernels where I observe the problem are 2.6.9 kernels from RedHat 
EL4. I have posted this here because I saw that the highmem bounce 
buffer/memory pool implementation was identical between the 2.6.9 kernel 
and all but the very latest development kernels, and I concluded 
prematurely that the behavior under my scenario must also be the same -- 
which it wasn't. I apologize for not having looked more closely.

Many thanks for looking into this anyway. From a theoretical point of 
view, I still think I had a valid point :-/.

Your patch sure looks good to me.

Regards
Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Andrew Morton]

Martin Wilck <martin.wilck@fujitsu-siemens.com> wrote:
>
> Hello Andrew,
> 
> > The semaphore is initialised with the limit level, so once it has been
> > down()ed more than `limit' times, processes will block until someone does
> > up().
> 
> Oh - of course. Neat.
> 
> >>It appears to run much more 
> >> smoothly now, perhaps because wakeup_bdflush() isn't called any more. 
> >> Are you still interested in more data?
> > 
> > Perhaps the newer kernel has writeback thresholding fixes so it's not
> > possible to dirty as much memory with write().
> 
> I have collected more data and the behavior with 2.6.12-rc5-mm2 is 
> flawless, there is a continuous writeback flow close to the maximum rate 
> possible, and the bounce buffer usage never gets anywhere near the limit 
> where it'd become dangerous. At least not in my test setup. The latest 
> fedora kernel 2.6.11-1.27 also behaves ok, although it doesn't adapt to 
> changing io load as smoothly as 2.6.12-rc5-mm2 does, and the writeback 
> rate is oscillating more strongly.
> 
> The kernels where I observe the problem are 2.6.9 kernels from RedHat 
> EL4. I have posted this here because I saw that the highmem bounce 
> buffer/memory pool implementation was identical between the 2.6.9 kernel 
> and all but the very latest development kernels, and I concluded 
> prematurely that the behavior under my scenario must also be the same -- 
> which it wasn't. I apologize for not having looked more closely.
> 
> Many thanks for looking into this anyway. From a theoretical point of 
> view, I still think I had a valid point :-/.
> 
> Your patch sure looks good to me.

Well.  As I said, I think what you're seeing here is recent changes to
mm/page-writeback.c which reduce the amount of memory which we'll permit to
be dirtied due to write() calls.  You'll probably find that the bounce
buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in
2.6.9, for the same reasons.

What concerns me is that there are other ways of dirtying lots of memory
apart from write(): namely mmap(MAP_SHARED).  If someone dirties 90% of all
memory via mmap() then we might again get into bounce buffer starvation.

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Martin Wilck]

Hi Andrew,

> Well.  As I said, I think what you're seeing here is recent changes to
> mm/page-writeback.c which reduce the amount of memory which we'll permit to
> be dirtied due to write() calls.  You'll probably find that the bounce
> buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in
> 2.6.9, for the same reasons.
> 
> What concerns me is that there are other ways of dirtying lots of memory
> apart from write(): namely mmap(MAP_SHARED).  If someone dirties 90% of all
> memory via mmap() then we might again get into bounce buffer starvation.

I have tried the mmap(MAP_SHARED) method now extensively. I haven't been 
able to come anywhere near the catastrophic situations I saw with the 
2.6.9 kernel, even by dirtying the full 8GB in fractions of a second.

There was another strangeness there though: Even with the high memory 
pressure applied, The ZONE_NORMAL free memory would never go below 
~300MB. When the mem pressure got too high, the kernel would rather free 
almost slabs and start swapping than use those remaining 300M. It seems 
to me that the new logic is a bit too conservative with ZONE_NORMAL 
allocations.

Regards
Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy

Re: RFC for 2.6: avoid OOM at bounce buffer storm

[Andrew Morton]

Martin Wilck <martin.wilck@fujitsu-siemens.com> wrote:
>
> Hi Andrew,
> 
> > Well.  As I said, I think what you're seeing here is recent changes to
> > mm/page-writeback.c which reduce the amount of memory which we'll permit to
> > be dirtied due to write() calls.  You'll probably find that the bounce
> > buffer problem is also fixable by reducing /proc/sys/vm/dirty_ratio in
> > 2.6.9, for the same reasons.
> > 
> > What concerns me is that there are other ways of dirtying lots of memory
> > apart from write(): namely mmap(MAP_SHARED).  If someone dirties 90% of all
> > memory via mmap() then we might again get into bounce buffer starvation.
> 
> I have tried the mmap(MAP_SHARED) method now extensively. I haven't been 
> able to come anywhere near the catastrophic situations I saw with the 
> 2.6.9 kernel, even by dirtying the full 8GB in fractions of a second.

OK, thanks.

> There was another strangeness there though: Even with the high memory 
> pressure applied, The ZONE_NORMAL free memory would never go below 
> ~300MB. When the mem pressure got too high, the kernel would rather free 
> almost slabs and start swapping than use those remaining 300M. It seems 
> to me that the new logic is a bit too conservative with ZONE_NORMAL 
> allocations.

Yes, we reserve lots of ZONE_NORMAL memory when performing GFP_HIGHMEM
allocations to avoid a weird corner case in which all of the lowmem memory
is pinned down because some application mlocked a lot of memory.

You can do

	echo 1000 > /proc/sys/vm/lowmem_reserve_ratio 

to get the old behaviour back (I do this all the time).