re: mmap over nfs leads to excessive system load

re: mmap over nfs leads to excessive system load

[Kenny Simpson]

Just another data point....
If I open the file with O_DIRECT.. not much changes:

samples  %        image name               app name                 symbol name
12585321 18.9373  vmlinux-2.6.14           vmlinux-2.6.14           find_get_pages_tag
8608887  12.9539  vmlinux-2.6.14           vmlinux-2.6.14           mpage_writepages
6870600  10.3383  vmlinux-2.6.14           vmlinux-2.6.14           unlock_page
6605417   9.9393  vmlinux-2.6.14           vmlinux-2.6.14           clear_page_dirty_for_io
6259207   9.4183  vmlinux-2.6.14           vmlinux-2.6.14           release_pages
3249493   4.8896  vmlinux-2.6.14           vmlinux-2.6.14           __lookup_tag
3248871   4.8886  vmlinux-2.6.14           vmlinux-2.6.14           pci_conf1_write
2677914   4.0295  vmlinux-2.6.14           vmlinux-2.6.14           page_waitqueue
982811    1.4789  vmlinux-2.6.14           vmlinux-2.6.14           _read_lock_irqsave
917165    1.3801  vmlinux-2.6.14           vmlinux-2.6.14           _read_unlock_irq
758960    1.1420  vmlinux-2.6.14           vmlinux-2.6.14           __wake_up_bit
706607    1.0632  vmlinux-2.6.14           vmlinux-2.6.14           _spin_lock_irqsave


-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Tue, 2005-11-08 at 11:25 -0800, Kenny Simpson wrote:
> Just another data point....
> If I open the file with O_DIRECT.. not much changes:

Hmm... Are you mounting using the -osync or -onoac options? Doing
synchronous writes will tend to slow down flushing considerably, and the
VM appears to be very fragile w.r.t. slow filesystems.

Cheers,
  Trond


> samples  %        image name               app name                 symbol name
> 12585321 18.9373  vmlinux-2.6.14           vmlinux-2.6.14           find_get_pages_tag
> 8608887  12.9539  vmlinux-2.6.14           vmlinux-2.6.14           mpage_writepages
> 6870600  10.3383  vmlinux-2.6.14           vmlinux-2.6.14           unlock_page
> 6605417   9.9393  vmlinux-2.6.14           vmlinux-2.6.14           clear_page_dirty_for_io
> 6259207   9.4183  vmlinux-2.6.14           vmlinux-2.6.14           release_pages
> 3249493   4.8896  vmlinux-2.6.14           vmlinux-2.6.14           __lookup_tag
> 3248871   4.8886  vmlinux-2.6.14           vmlinux-2.6.14           pci_conf1_write
> 2677914   4.0295  vmlinux-2.6.14           vmlinux-2.6.14           page_waitqueue
> 982811    1.4789  vmlinux-2.6.14           vmlinux-2.6.14           _read_lock_irqsave
> 917165    1.3801  vmlinux-2.6.14           vmlinux-2.6.14           _read_unlock_irq
> 758960    1.1420  vmlinux-2.6.14           vmlinux-2.6.14           __wake_up_bit
> 706607    1.0632  vmlinux-2.6.14           vmlinux-2.6.14           _spin_lock_irqsave
> 
> 
> -Kenny
> 
> 
> 
> 	
> 		
> __________________________________ 
> Yahoo! Mail - PC Magazine Editors' Choice 2005 
> http://mail.yahoo.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

re: mmap over nfs leads to excessive system load

[Kenny Simpson]

I knew I forgoet something important in the info:
  type nfs (rw,tcp,rsize=32768,wsize=32768,hard,intr,vers=3,tcp,rsize=32768,wsize=32768,hard,intr

-Kenny


--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> On Tue, 2005-11-08 at 11:25 -0800, Kenny Simpson wrote:
> > Just another data point....
> > If I open the file with O_DIRECT.. not much changes:
> 
> Hmm... Are you mounting using the -osync or -onoac options? Doing
> synchronous writes will tend to slow down flushing considerably, and the
> VM appears to be very fragile w.r.t. slow filesystems.
> 
> Cheers,
>   Trond



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

I am attaching a sample piece of code to show the behavior.
This simply tries to grow a file as fast as it can using different methods.
Use with caution as it does not stop, and will fill the disk if you let it run.

Run as "nfstest -m <filename>" where <filename> is on an nfs mount.

-Kenny


		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 652126289-nfstest.c --]
[-- Type: text/x-csrc; name="nfstest.c", Size: 3153 bytes --]

/* test the write throughput of a sliding mmap window vs simple FILE*  */
#define _GNU_SOURCE

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>

int total_bytes = 0;
int last_bytes = 0;

struct timeval last_time;

void timeout(int i)
{
  struct timeval now;
  gettimeofday(&now, 0);

  (void) i;

  {
    double time_diff = (now.tv_sec - last_time.tv_sec) + ((now.tv_usec - last_time.tv_usec) / 1000000.);

    int new_bytes = total_bytes;
    int diff = new_bytes - last_bytes;
    printf("wrote %dk %dM bytes in %f seconds -> %fM/sec\n", diff / 1024, diff / (1024 * 1024), time_diff, (diff / (time_diff * 1000000.)));
    last_bytes = new_bytes;
    last_time = now;
  }
}



void do_mapwrite(int fd)
{
  /* lets write as fast as we can.... */
  int const window_size = 2 * 1024 * 1024; /* 16k */
  int const window_pages = window_size / 4096;

  int file_page_offset = 0;
  long long file_size = window_size;

  ftruncate64(fd, file_size);

  {
    char* mapping_start = (char*)mmap64(0, window_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);


    /* scribble into buffer and walk the window */
    for (;;) {
      memset(mapping_start, 0, window_size);

      /* grow file */
      file_size += window_size;
      ftruncate64(fd, file_size);
      file_page_offset += window_pages;
      /*msync(mapping_start, window_size, MS_SYNC);*/
      /*munmap(mapping_start, window_size);*/
      /*mapping_start = (char*)mmap64(mapping_start, window_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, file_size - window_size);*/
      remap_file_pages(mapping_start, window_size, 0, file_page_offset, MAP_SHARED);

      total_bytes += window_size;
    }
  }
}

void do_filewrite(int fd)
{
  FILE* ptr = fdopen(fd, "w+");

  int line_len = 80;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    fwrite(buf, line_len, 1, ptr);
    total_bytes += line_len;
  }
}

void do_syswrite(int fd)
{
  int line_len = 64 * 1024;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    write(fd, buf, line_len);
    total_bytes += line_len;
  }
}


int main(int argc, char* argv[])
{
  if (argc != 3) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  if ((argv[1][0] != '-') || 
      ((argv[1][1] != 'm') &&
       (argv[1][1] != 'w') &&
       (argv[1][1] != 'f'))) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  int fd = open(argv[2], O_RDWR | O_CREAT | O_LARGEFILE | O_DIRECT, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

  /* start the clock...  */
  signal(SIGALRM, timeout);
  {
    struct itimerval itv;
    itv.it_interval.tv_sec = 1;
    itv.it_interval.tv_usec = 0;
    itv.it_value = itv.it_interval;

    setitimer(ITIMER_REAL, &itv, 0);
  }

  switch (argv[1][1]) {
  case 'm': do_mapwrite(fd); break;
  case 'f': do_filewrite(fd); break;
  case 'w': do_syswrite(fd); break;
  }

  return 0;
}

re: mmap over nfs leads to excessive system load

[Kenny Simpson]

I ran the same test again against 2.6.15-rc, and got pretty much the same thing.  It starts nice
and fast (30+MB/s, but drops down to under 10MB/s with the system time pegging one CPU).

Here is the oprofile result:

CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
of 0x01 (mandatory) count 100000
samples  %        symbol name
412585   14.6687  find_get_pages_tag
343898   12.2267  mpage_writepages
290144   10.3155  release_pages
288631   10.2617  unlock_page
286181   10.1746  pci_conf1_write
267619    9.5147  clear_page_dirty_for_io
128128    4.5554  __lookup_tag
120895    4.2982  page_waitqueue
52739     1.8750  _spin_lock_irqsave
43623     1.5509  skb_copy_bits
30157     1.0722  __wake_up_bit
29973     1.0656  _read_lock_irqsave


-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[William Lee Irwin III]

On Tue, Nov 15, 2005 at 03:47:30PM -0800, Kenny Simpson wrote:
> CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
> of 0x01 (mandatory) count 100000
> samples  %        symbol name
> 412585   14.6687  find_get_pages_tag
> 343898   12.2267  mpage_writepages
> 290144   10.3155  release_pages
> 288631   10.2617  unlock_page
> 286181   10.1746  pci_conf1_write
> 267619    9.5147  clear_page_dirty_for_io
> 128128    4.5554  __lookup_tag
> 120895    4.2982  page_waitqueue
> 52739     1.8750  _spin_lock_irqsave
> 43623     1.5509  skb_copy_bits
> 30157     1.0722  __wake_up_bit
> 29973     1.0656  _read_lock_irqsave

67%, or 2/3 of the samples, are in the top 6 functions. Have you tried
instruction-level profiling? It would be interesting to see what
codepaths within the functions are the largest offenders.


-- wli

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- William Lee Irwin III <wli@holomorphy.com> wrote:
> 67%, or 2/3 of the samples, are in the top 6 functions. Have you tried
> instruction-level profiling? It would be interesting to see what
> codepaths within the functions are the largest offenders.
> 
> 
> -- wli
> 
I'm a little new to oprofile, but I'm willing to try any configuration or set of flags that could
be useful.
Are you referring to the -d option in opreport?
 --details / -d
              Show per-instruction details for all selected symbols.

I'll give it a go when I get back to work.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Kenny Simpson <theonetruekenny@yahoo.com> wrote:
>
> I ran the same test again against 2.6.15-rc, and got pretty much the same thing.  It starts nice
> and fast (30+MB/s, but drops down to under 10MB/s with the system time pegging one CPU).
> 
> Here is the oprofile result:
> 
> CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
> of 0x01 (mandatory) count 100000
> samples  %        symbol name
> 412585   14.6687  find_get_pages_tag
> 343898   12.2267  mpage_writepages
> 290144   10.3155  release_pages
> 288631   10.2617  unlock_page
> 286181   10.1746  pci_conf1_write
> 267619    9.5147  clear_page_dirty_for_io
> 128128    4.5554  __lookup_tag
> 120895    4.2982  page_waitqueue
> 52739     1.8750  _spin_lock_irqsave
> 43623     1.5509  skb_copy_bits
> 30157     1.0722  __wake_up_bit
> 29973     1.0656  _read_lock_irqsave
> 

Your application walks the file in 2MB hunks, doing ftruncate() each time
to expand the file by another 2MB.

nfs_setattr() implements the truncate.  It syncs the whole file, using
filemap_write_and_wait() (that seems a bit suboptimal.  All we're doing is
increasing i_size??)

So filemap_write_and_wait() has to write 2MB's worth of pages.  Problem is,
_all_ the pages, even the 99% which are clean are tagged as dirty in the
pagecache radix tree.  So find_get_pages_tag() ends up visiting each page
in the file, and blows much CPU doing so.

The writeout happens in mpage_writepages(), which uses
clear_page_dirty_for_io() to clear PG_dirty.  But it doesn't clear the
dirty tag in the radix tree.  It relies upon the filesystem to do the right
thing later on.  Which is all very unpleasant, sorry.  See the explanatory
comment over clear_page_dirty_for_io().

nfs_writepage() doesn't do any of the things which that comment says it
should, hence the radix tree tags are getting out of sync, hence this
problem.

NFS does strange, incomprehensible-to-little-akpms things in its writeout
path.  Ideally, it should run set_page_writeback() prior to unlocking the
page and end_page_writeback() when I/O completes.  That'll keep the VM
happier while fixing this performance glitch.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Tue, 2005-11-15 at 23:45 -0800, Andrew Morton wrote:

> So filemap_write_and_wait() has to write 2MB's worth of pages.  Problem is,
> _all_ the pages, even the 99% which are clean are tagged as dirty in the
> pagecache radix tree.  So find_get_pages_tag() ends up visiting each page
> in the file, and blows much CPU doing so.
> 
> The writeout happens in mpage_writepages(), which uses
> clear_page_dirty_for_io() to clear PG_dirty.  But it doesn't clear the
> dirty tag in the radix tree.  It relies upon the filesystem to do the right
> thing later on.  Which is all very unpleasant, sorry.  See the explanatory
> comment over clear_page_dirty_for_io().

> nfs_writepage() doesn't do any of the things which that comment says it
> should, hence the radix tree tags are getting out of sync, hence this
> problem.
> 
> NFS does strange, incomprehensible-to-little-akpms things in its writeout
> path.  Ideally, it should run set_page_writeback() prior to unlocking the
> page and end_page_writeback() when I/O completes.  That'll keep the VM
> happier while fixing this performance glitch.

Actually that will screw over performance even further by forcing us to
send out loads of little RPC requests to write 4k pages instead of
allowing us to gather those writes into 32k (or larger) chunks.

Anyhow, does the following patch help?

Cheers,
  Trond
------
NFS: resync to yet more writepage() changes...

 Ensure that we call clear_page_dirty() for pages that have been written
 via writepage().

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..ea77da5 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
 	} while (count);
 	/* Update file length */
 	nfs_grow_file(page, offset, written);
+	clear_page_dirty(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, written);
 
@@ -238,6 +239,7 @@ static int nfs_writepage_async(struct nf
 		goto out;
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
+	clear_page_dirty(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> Anyhow, does the following patch help?

Unfortunately, not:

samples  %        symbol name
545009   15.2546  find_get_pages_tag
450595   12.6120  mpage_writepages
383196   10.7255  release_pages
381479   10.6775  unlock_page
351513    9.8387  clear_page_dirty_for_io
317784    8.8947  pci_conf1_write
167918    4.7000  __lookup_tag
160701    4.4980  page_waitqueue
59142     1.6554  _spin_lock_irqsave
47655     1.3338  skb_copy_bits
39136     1.0954  __wake_up_bit
38143     1.0676  _read_lock_irqsave


With reducing the window size to 32k, things aren't much different:
samples  %        symbol name
474589   21.2001  find_get_pages_tag
370512   16.5509  mpage_writepages
310556   13.8727  release_pages
302571   13.5160  unlock_page
286541   12.7999  clear_page_dirty_for_io
119717    5.3478  page_waitqueue
109920    4.9102  __lookup_tag
33313     1.4881  pci_conf1_write
29198     1.3043  __wake_up_bit
27075     1.2095  _read_lock_irqsave
25009     1.1172  _read_unlock_irq

... except the performance is much worse than with the 2M buffer (hence the 2M choice).  With the
smaller buffer, the throughput starts at 8M/sec and quickly drops to 1M/sec.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 07:01 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > Anyhow, does the following patch help?
> 
> Unfortunately, not:
> 
> samples  %        symbol name
> 545009   15.2546  find_get_pages_tag

Argh... I totally missed the point there with the last patch. We should
be resyncing the page tag with the value of the PG_dirty flag...

OK, please back out the patch that I sent you, and try this one instead.

Cheers,
 Trond

------
NFS: resync to yet more writepage() changes...

 Ensure that we call clear_page_dirty() for pages that have been written
 via writepage().

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c      |    2 ++
 include/linux/mm.h  |    1 +
 mm/page-writeback.c |   20 ++++++++++++++++++++
 3 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..61ec355 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
 	} while (count);
 	/* Update file length */
 	nfs_grow_file(page, offset, written);
+	clear_page_dirty_tag(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, written);
 
@@ -238,6 +239,7 @@ static int nfs_writepage_async(struct nf
 		goto out;
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
+	clear_page_dirty_tag(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 1013a42..cb1cfe1 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -730,6 +730,7 @@ int redirty_page_for_writepage(struct wr
 int FASTCALL(set_page_dirty(struct page *page));
 int set_page_dirty_lock(struct page *page);
 int clear_page_dirty_for_io(struct page *page);
+int clear_page_dirty_tag(struct page *page);
 
 extern unsigned long do_mremap(unsigned long addr,
 			       unsigned long old_len, unsigned long new_len,
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 74138c9..65c58fa 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -751,6 +751,26 @@ int clear_page_dirty_for_io(struct page 
 	return TestClearPageDirty(page);
 }
 
+/*
+ * Clears the page dirty tag. See comment in clear_page_dirty_for_io()
+ */
+int clear_page_dirty_tag(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (mapping) {
+		unsigned long flags;
+
+		write_lock_irqsave(&mapping->tree_lock, flags);
+		if (!PageDirty(page))
+			radix_tree_tag_clear(&mapping->page_tree,
+						page_index(page),
+						PAGECACHE_TAG_DIRTY);
+		write_unlock_irqrestore(&mapping->tree_lock, flags);
+	}
+}
+EXPORT_SYMBOL(clear_page_dirty_tag);
+
 int test_clear_page_writeback(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 07:01 -0800, Kenny Simpson wrote:
> > --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > > Anyhow, does the following patch help?
> > 
> > Unfortunately, not:
> > 
> > samples  %        symbol name
> > 545009   15.2546  find_get_pages_tag
> 
> Argh... I totally missed the point there with the last patch. We should
> be resyncing the page tag with the value of the PG_dirty flag...
> 
> OK, please back out the patch that I sent you, and try this one instead.
> 
> ...
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index 8f71e76..61ec355 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
>  	} while (count);
>  	/* Update file length */
>  	nfs_grow_file(page, offset, written);
> +	clear_page_dirty_tag(page);
>  	/* Set the PG_uptodate flag? */
>  	nfs_mark_uptodate(page, offset, written);
>
> ....  
> +int clear_page_dirty_tag(struct page *page)
> +{
> +	struct address_space *mapping = page_mapping(page);
> +
> +	if (mapping) {
> +		unsigned long flags;
> +
> +		write_lock_irqsave(&mapping->tree_lock, flags);
> +		if (!PageDirty(page))
> +			radix_tree_tag_clear(&mapping->page_tree,
> +						page_index(page),
> +						PAGECACHE_TAG_DIRTY);
> +		write_unlock_irqrestore(&mapping->tree_lock, flags);
> +	}
> +}

That will fix it, but the PageWriteback accounting is still wrong.

Is it not possible to use set_page_writeback()/end_page_writeback()?

Are these pages marked "unstable" at this time?

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:

> That will fix it, but the PageWriteback accounting is still wrong.
> 
> Is it not possible to use set_page_writeback()/end_page_writeback()?

Not really. The pages aren't flushed at this time. We the point is to
gather several pages and coalesce them into one over-the-wire RPC call.
That means we cannot really do it from inside ->writepage().

We do start the actual RPC calls in ->writepages(), though.

> Are these pages marked "unstable" at this time?

No. "unstable" means that the RPC call to send the pages to the server
has completed, but the pages have not been flushed to disk by the
server. In this case we haven't even sent the pages to the server.

Instead the pages are accounted for in nr_dirty, and are tracked by the
internal NFS 'dirty request' lists. We also mark the inode as being
dirty in order to ensure that pdflush will kick off the actual RPC calls
if nobody else does so first.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Christoph Hellwig]

On Wed, Nov 16, 2005 at 01:34:22PM -0500, Trond Myklebust wrote:
> Not really. The pages aren't flushed at this time. We the point is to
> gather several pages and coalesce them into one over-the-wire RPC call.
> That means we cannot really do it from inside ->writepage().
> 
> We do start the actual RPC calls in ->writepages(), though.

This is a problem we have in various filesystems.  Except for really
bad OOM situations the filesystem should never get a writeout request
for a single file.  We should really stop having ->writepage called by
the VM and move this kind of batching code into the VM.  I'm runnin into
similar issues for XFS and unwritten/delayed extent conversion once again.

Re: mmap over nfs leads to excessive system load

[Nikita Danilov]

Christoph Hellwig writes:
 > On Wed, Nov 16, 2005 at 01:34:22PM -0500, Trond Myklebust wrote:
 > > Not really. The pages aren't flushed at this time. We the point is to
 > > gather several pages and coalesce them into one over-the-wire RPC call.
 > > That means we cannot really do it from inside ->writepage().
 > > 
 > > We do start the actual RPC calls in ->writepages(), though.
 > 
 > This is a problem we have in various filesystems.  Except for really
 > bad OOM situations the filesystem should never get a writeout request
 > for a single file.  We should really stop having ->writepage called by
 > the VM and move this kind of batching code into the VM.  I'm runnin into
 > similar issues for XFS and unwritten/delayed extent conversion once again.

Simplistic version if such batching is implemented in the patch below
(also available at

http://linuxhacker.ru/~nikita/patches/2.6.15-rc1/05-cluster-pageout.patch

it depends on page_referenced-move-dirty patch from the same place)

This version pokes into address_space radix tree to find a cluster of
pages suitable for page-out and then calls ->writepage() on pages in
that cluster in the proper order. This relies on the underlying layer
(e.g., block device) to perform request coalescing.

My earlier attempts to do this through ->writepages() were all racy,
because at some point ->writepages() has to release a lock at the
original page around which the cluster is built, and that lock is the
only thing that protects inode/address_space from the destruction. As
was already noted by Andrew, one cannot use igrab/iput in the VM scanner
to deal with that.

I still think it's possible to do higher layer batching, but that would
require more extensive changes to both VM scanner and ->writepages().

 > 

Nikita.
--
Implement pageout clustering at the VM level.

With this patch VM scanner calls pageout_cluster() instead of
->writepage(). pageout_cluster() tries to find a group of dirty pages around
target page, called "pivot" page of the cluster. If group of suitable size is
found, ->writepages() is called for it, otherwise, page_cluster() falls back
to ->writepage().

This is supposed to help in work-loads with significant page-out of
file-system pages from tail of the inactive list (for example, heavy dirtying
through mmap), because file system usually writes multiple pages more
efficiently. Should also be advantageous for file-systems doing delayed
allocation, as in this case they will allocate whole extents at once.

Few points:

 - swap-cache pages are not clustered (although they can be, but by
   page->private rather than page->index)

 - only kswapd does clustering, because direct reclaim path should be low
   latency.

 - Original version of this patch added new fields to struct writeback_control
   and expected ->writepages() to interpret them. This led to hard-to-fix races
   against inode reclamation. Current version simply calls ->writepage() in the
   "correct" order, i.e., in the order of increasing page indices.

Signed-off-by: Nikita Danilov <nikita@clusterfs.com>


 mm/shmem.c  |   14 ++++++-
 mm/vmscan.c |  112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 124 insertions(+), 2 deletions(-)

diff -puN mm/vmscan.c~cluster-pageout mm/vmscan.c
--- git-linux/mm/vmscan.c~cluster-pageout	2005-11-13 21:25:15.000000000 +0300
+++ git-linux-nikita/mm/vmscan.c	2005-11-13 21:25:15.000000000 +0300
@@ -360,6 +360,116 @@ static void send_page_to_kpgout(struct p
 	spin_unlock(&kpgout_queue_lock);
 }
 
+enum {
+	PAGE_CLUSTER_WING = 16,
+	PAGE_CLUSTER_SIZE = 2 * PAGE_CLUSTER_WING,
+};
+
+static int page_fits_cluster(struct address_space *mapping, struct page *page)
+{
+	int result;
+
+	if (page != NULL && !PageActive(page) && !TestSetPageLocked(page)) {
+		/*
+		 * unlock ->tree_lock to avoid lock inversion with
+		 * ->i_mmap_lock in page_referenced().
+		 */
+		read_unlock_irq(&mapping->tree_lock);
+		result =
+			/* try_to_unmap(page) == SWAP_SUCCESS && */
+			PageDirty(page) && !PageWriteback(page) &&
+			!page_referenced(page, 1,
+					 page_zone(page)->temp_priority <= 0,
+					 1);
+		if (result == 0)
+			unlock_page(page);
+		read_lock_irq(&mapping->tree_lock);
+	} else
+		result = 0;
+	return result;
+}
+
+static void call_writepage(struct page *page, struct address_space *mapping,
+			   struct writeback_control *wbc)
+{
+	if (clear_page_dirty_for_io(page)) {
+		int result;
+
+		BUG_ON(!PageLocked(page));
+		BUG_ON(PageWriteback(page));
+
+		result = mapping->a_ops->writepage(page, wbc);
+		if (result == WRITEPAGE_ACTIVATE)
+			unlock_page(page);
+	}
+}
+
+int __pageout_cluster(struct page *page, struct address_space *mapping,
+		      struct writeback_control *wbc)
+{
+	int result;
+	int used;
+
+	pgoff_t punct;
+	pgoff_t start;
+	pgoff_t end;
+
+	struct page *pages_out[PAGE_CLUSTER_WING];
+	struct page *scan;
+
+	BUG_ON(PageAnon(page));
+
+	punct = page->index;
+	read_lock_irq(&mapping->tree_lock);
+	for (start = punct - 1, used = 0;
+	     start < punct && punct - start <= PAGE_CLUSTER_WING; start --) {
+		scan = radix_tree_lookup(&mapping->page_tree, start);
+		if (!page_fits_cluster(mapping, scan))
+			/*
+			 * no suitable page, stop cluster at this point
+			 */
+			break;
+		pages_out[used ++] = scan;
+		if ((start % PAGE_CLUSTER_SIZE) == 0)
+			/*
+			 * we reached aligned page.
+			 */
+			break;
+	}
+	read_unlock_irq(&mapping->tree_lock);
+
+	while (used > 0)
+		call_writepage(pages_out[--used], mapping, wbc);
+
+	result = mapping->a_ops->writepage(page, wbc);
+
+	for (end = punct + 1;
+	     end > punct && end - start < PAGE_CLUSTER_SIZE; ++ end) {
+		int enough;
+
+		/*
+		 * XXX nikita: consider find_get_pages_tag()
+		 */
+		read_lock_irq(&mapping->tree_lock);
+		scan = radix_tree_lookup(&mapping->page_tree, end);
+		enough = !page_fits_cluster(mapping, scan);
+		read_unlock_irq(&mapping->tree_lock);
+		if (enough)
+			break;
+		call_writepage(scan, mapping, wbc);
+	}
+	return result;
+}
+
+static int pageout_cluster(struct page *page, struct address_space *mapping,
+			   struct writeback_control *wbc)
+{
+	if (PageSwapCache(page)	|| !current_is_kswapd())
+		return mapping->a_ops->writepage(page, wbc);
+	else
+		return __pageout_cluster(page, mapping, wbc);
+}
+
 /*
  * Called by shrink_list() for each dirty page. Calls ->writepage().
  */
@@ -445,7 +555,7 @@ static pageout_t pageout(struct page *pa
 
 		ClearPageSkipped(page);
 		SetPageReclaim(page);
-		res = mapping->a_ops->writepage(page, &wbc);
+		res = pageout_cluster(page, mapping, &wbc);
 
 		if (res < 0)
 			handle_write_error(mapping, page, res);
diff -puN include/linux/writeback.h~cluster-pageout include/linux/writeback.h
diff -puN fs/mpage.c~cluster-pageout fs/mpage.c
diff -puN mm/shmem.c~cluster-pageout mm/shmem.c
--- git-linux/mm/shmem.c~cluster-pageout	2005-11-13 21:25:15.000000000 +0300
+++ git-linux-nikita/mm/shmem.c	2005-11-13 21:25:15.000000000 +0300
@@ -45,6 +45,7 @@
 #include <linux/swapops.h>
 #include <linux/mempolicy.h>
 #include <linux/namei.h>
+#include <linux/rmap.h>
 #include <asm/uaccess.h>
 #include <asm/div64.h>
 #include <asm/pgtable.h>
@@ -813,7 +814,18 @@ static int shmem_writepage(struct page *
 	struct inode *inode;
 
 	BUG_ON(!PageLocked(page));
-	BUG_ON(page_mapped(page));
+
+	/*
+	 * If shmem_writepage() is called on mapped page, a problem arises for
+	 * a tmpfs file mapped shared into different mms. Viz. shmem_writepage
+	 * changes the tmpfs-file identity of the page to swap identity: so if
+	 * it's unmapped later, the instances would then become private (to be
+	 * COWed) instead of shared.
+	 *
+	 * Just unmap page.
+	 */
+	if (page_mapped(page) && try_to_unmap(page) != SWAP_SUCCESS)
+		goto redirty;
 
 	mapping = page->mapping;
 	index = page->index;

_

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> 
> > That will fix it, but the PageWriteback accounting is still wrong.
> > 
> > Is it not possible to use set_page_writeback()/end_page_writeback()?
> 
> Not really. The pages aren't flushed at this time. We the point is to
> gather several pages and coalesce them into one over-the-wire RPC call.
> That means we cannot really do it from inside ->writepage().
> 

I still don't get it.

Once nfs_writepage() has been called, the page is conceptually "under
writeback", yes?  In that, at some point in the future, it will be written
to backing store.

Hence it's perfectly appropriate to run set_page_writepage() within
nfs_writepage().  It's a matter of finding the right place for the
end_page_writeback().

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 11:09 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> > 
> > > That will fix it, but the PageWriteback accounting is still wrong.
> > > 
> > > Is it not possible to use set_page_writeback()/end_page_writeback()?
> > 
> > Not really. The pages aren't flushed at this time. We the point is to
> > gather several pages and coalesce them into one over-the-wire RPC call.
> > That means we cannot really do it from inside ->writepage().
> > 
> 
> I still don't get it.
> 
> Once nfs_writepage() has been called, the page is conceptually "under
> writeback", yes?  In that, at some point in the future, it will be written
> to backing store.
> 
> Hence it's perfectly appropriate to run set_page_writepage() within
> nfs_writepage().  It's a matter of finding the right place for the
> end_page_writeback().

The point is that the process of flushing has not been started at that
time, so anybody that calls wait_on_page_writeback() immediately after
calling writepage() may end up waiting for a very long time indeed
(probably until the next pdflush).

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

I tried the same test, but instead of ftruncate64, I simply did a pwrite64 to get the file
extended... and got 40M+ with mostly outbound traffic, and much less CPU usage..... 

Unfortunately, once my test file hit 4295065601, Bad Things (TM) started to happen.  The system
time went to 100% of a CPU, and the nfs traffic on that mount stopped.

I got an oprofile of the spinning system:
samples  %        symbol name
301039   27.9748  zap_pte_range
156234   14.5184  unmap_vmas
111760   10.3856  __bitmap_weight
103624    9.6295  _spin_lock
97063     9.0198  unmap_page_range
67011     6.2272  unmap_mapping_range
59382     5.5182  sub_preempt_count
51258     4.7633  zap_page_range
25235     2.3450  page_address
16768     1.5582  unmap_mapping_range_vma
13257     1.2319  debug_smp_processor_id
11594     1.0774  add_preempt_count

I also seem unable to kill the test process.

Any ideas?  (2**32 file size issue somewhere?)

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Kenny Simpson <theonetruekenny@yahoo.com> wrote:
> I also seem unable to kill the test process.

Root is also unable to kill the process (even -9).
top shows its status a R.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 12:56 -0800, Kenny Simpson wrote:
> I tried the same test, but instead of ftruncate64, I simply did a pwrite64 to get the file
> extended... and got 40M+ with mostly outbound traffic, and much less CPU usage..... 
> 
> Unfortunately, once my test file hit 4295065601, Bad Things (TM) started to happen.  The system
> time went to 100% of a CPU, and the nfs traffic on that mount stopped.
> 
> I got an oprofile of the spinning system:
> samples  %        symbol name
> 301039   27.9748  zap_pte_range
> 156234   14.5184  unmap_vmas
> 111760   10.3856  __bitmap_weight
> 103624    9.6295  _spin_lock
> 97063     9.0198  unmap_page_range
> 67011     6.2272  unmap_mapping_range
> 59382     5.5182  sub_preempt_count
> 51258     4.7633  zap_page_range
> 25235     2.3450  page_address
> 16768     1.5582  unmap_mapping_range_vma
> 13257     1.2319  debug_smp_processor_id
> 11594     1.0774  add_preempt_count
> 
> I also seem unable to kill the test process.
> 
> Any ideas?  (2**32 file size issue somewhere?)

Is this NFSv2?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> Is this NFSv2?
> 
> Cheers,
>   Trond
> 

Not according to mount(1): 
 (rw,vers=3,tcp,rsize=32768,wsize=32768,hard,intr,addr=x.x.x.x)

-Kenny





		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> Is this NFSv2?
> 
> Cheers,
>   Trond
> 
This is reproducible with O_DIRECT, but not without.

The profile looks the same:
samples  %        symbol name
647042   28.4114  zap_pte_range
572195   25.1249  unmap_mapping_range
324291   14.2395  _spin_lock
139259    6.1148  __bitmap_weight
137048    6.0177  zap_page_range
104614    4.5936  unmap_mapping_range_vma
63406     2.7841  debug_smp_processor_id
48906     2.1474  sub_preempt_count
46090     2.0238  unmap_vmas
27966     1.2280  add_preempt_count
23224     1.0198  invalidate_inode_pages2_range
21676     0.9518  unmap_page_range
17825     0.7827  _spin_unlock

I've had mixed results with a local ext3 file with the same test.  One run had a 37 second delay
while crossing 4GB, another happily went by without incident.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 13:41 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > 
> > Is this NFSv2?
> > 
> > Cheers,
> >   Trond
> > 
> This is reproducible with O_DIRECT, but not without.

I'm getting lost here. Please could you spell out the testcases that are
not working.

Are you saying that the combination mmap() + pwrite64() fails on
O_DIRECT, but works on ordinary open, and that mmap() + ftruncate64()
always works?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> I'm getting lost here. Please could you spell out the testcases that are
> not working.
> 
> Are you saying that the combination mmap() + pwrite64() fails on
> O_DIRECT, but works on ordinary open, and that mmap() + ftruncate64()
> always works?
> 
> Cheers,
>   Trond
> 
ftruncate64 works with O_DIRECT
ftruncate64 works w/o O_DIRECT

pwrite64 FAILS with O_DIRECT at ~4GB
pwrite64 works w/o O_DIRECT.

I am re-running these tests to confirm (could take a minute).

All opens are with O_RDWR | O_CREAT | O_LARGEFILE.
All test over GbE w/ jumbo frames (8160 mtu) to a netapp filer (via x-over cable).

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> I'm getting lost here. Please could you spell out the testcases that are
> not working.

I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
condition.

The cases that are fine are:
  pwrite64
  ftruncate with O_DIRECT
  ftruncate

Also, when the system is in this state, if I try to 'ls' the file,
the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
shows it is in a call to stat64.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:39 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > I'm getting lost here. Please could you spell out the testcases that are
> > not working.
> 
> I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
> condition.
> 
> The cases that are fine are:
>   pwrite64
>   ftruncate with O_DIRECT
>   ftruncate
> 
> Also, when the system is in this state, if I try to 'ls' the file,
> the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
> shows it is in a call to stat64.
> 
> -Kenny

Chuck, can you take a look at this?

Kenny is seeing what a hang when using pwrite64() on an O_DIRECT file
and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.

I had a quick look at nfs_file_direct_write(), and among other things,
it would appear that it is not doing any of the usual overflow checks on
*pos and the count size (see generic_write_checks()). In particular,
checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
not set (and also against overflow vs. s_maxbytes, but that is less
relevant here).

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

Trond Myklebust wrote:
> On Wed, 2005-11-16 at 14:39 -0800, Kenny Simpson wrote:
> 
>>--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>>>I'm getting lost here. Please could you spell out the testcases that are
>>>not working.
>>
>>I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
>>condition.
>>
>>The cases that are fine are:
>>  pwrite64
>>  ftruncate with O_DIRECT
>>  ftruncate
>>
>>Also, when the system is in this state, if I try to 'ls' the file,
>>the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
>>shows it is in a call to stat64.
>>
>>-Kenny
> 
> 
> Chuck, can you take a look at this?
> 
> Kenny is seeing a hang when using pwrite64() on an O_DIRECT file
> and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.
> 
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).

'uname -a' on the client?


[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Chuck Lever <cel@citi.umich.edu> wrote:
> 
> 'uname -a' on the client?

Linux tux6127 2.6.15-rc1 #6 SMP PREEMPT Wed Nov 16 14:47:14 EST 2005 i686 GNU/Linux

I also sent the .config on a previous posting.  I can send it again if you'd like.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> Chuck, can you take a look at this?
> 
> Kenny is seeing what a hang when using pwrite64() on an O_DIRECT file
> and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.
> 
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).
> 
> Cheers,
>   Trond

I tried the same test, but starting closer to 4GB... here is the final lines from strace:
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1047544, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8564768768)          = 1
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1048056, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8566865920)          = 1
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1048568, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8568963072

The pwrite never returns.
So it seems to be a problem NOT with an absolute 4GB, but with a total of 4GB having been written.

Here are the first few lines from the strace to show all the options being used:
open("/mnt/bar", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0644) = 3
pwrite(3, "\0", 1, 4280287232)          = 1
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xff000) = 0xb7b8e000
pwrite(3, "\0", 1, 4282384384)          = 1
remap_file_pages(0xb7b8e000, 2097152, PROT_NONE, 2552, MAP_SHARED) = 0
pwrite(3, "\0", 1, 4284481536)          = 1
remap_file_pages(0xb7b8e000, 2097152, PROT_NONE, 3064, MAP_SHARED) = 0

/mnt is an nfs mount over GbE w/ jumbo frames (8160 mtu) cross-over directly to a netapp filer.

The mount options are: (from /proc/mounts)
 /mnt nfs rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=tcp,addr=x.x.x.x 0 0

The card is an Intel e1000 - default module options (NAPI-enabled)
on a 64-bit PCIX 100MHz.
Kernel is 2.6.15-rc w/ Trond's nfs patch.
Machine is a 2x Pentium 4 Xeon 2.66GHz (HT enabled), w/ 2GB ram and 4GB swap.

vmstat shows:
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0      0 1336864 123212 203936    0    0     0    20 1111  1129  1 26 73  0
 1  0      0 1336608 123212 203936    0    0     0     0 1078  1076  1 25 74  0
 1  0      0 1336864 123212 203936    0    0     0     0 1077  1087  1 26 73  0

the sy of 25 is one virtual CPU with 100% system.

Oprofile shows time being spent:

samples  %        symbol name
303102   42.4732  zap_pte_range
133702   18.7355  _spin_lock
61145     8.5682  __bitmap_weight
43169     6.0492  page_address
42196     5.9129  unmap_vmas
30132     4.2224  unmap_page_range

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Kenny Simpson <theonetruekenny@yahoo.com> wrote:
>
> The pwrite never returns.
> So it seems to be a problem NOT with an absolute 4GB, but with a total of 4GB having been written.

Could you send the test app please?  (Apologies if you've already done so
and I missed it).

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

--- Andrew Morton <akpm@osdl.org> wrote:
> Could you send the test app please?  (Apologies if you've already done so
> and I missed it).
> 

Here it is again... this one skips to just under 4GB before starting.
run with "writetest -m <filename>" for the mmap test.

-Kenny



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3360621278-writetest.cpp --]
[-- Type: text/x-c++src; name="writetest.cpp", Size: 3615 bytes --]

// test the write throughput of a sliding mmap window vs simple FILE*

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>

int total_bytes = 0;
int last_bytes = 0;

struct timeval last_time;

void timeout(int)
{
  struct timeval now;
  gettimeofday(&now, 0);

  double time_diff = (now.tv_sec - last_time.tv_sec) + ((now.tv_usec - last_time.tv_usec) / 1000000.);
  last_time = now;

  int new_bytes = total_bytes;
  int diff = new_bytes - last_bytes;
  printf("wrote %dk %dM bytes in %f seconds -> %fM/sec\n", diff / 1024, diff / (1024 * 1024), time_diff, (diff / (time_diff * 1000000.)));
  last_bytes = new_bytes;
}


// tests O_DIRECT + pwrite = fail
//       O_DIRECT + truncate = happy
//                  truncate = happy

//                  pwrite = happy
//       O_DIRECT + truncate = happy
//                  truncate = happy
//       O_DIRECT + pwrite = fail

void do_mapwrite(int fd)
{
  // lets write as fast as we can....
  int const window_size = 2 * 1024 * 1024; // 2 * 1024 * 1024; // 16k
  int const window_pages = window_size / 4096;

  int file_page_offset = 0;
  long long file_size = window_size;


  // fast-forward... by 2046 windows
  file_size += 2046u * (2 * 1024 * 1024);
  file_page_offset += 2046;


  //ftruncate64(fd, file_size);
  printf("pwrite to %llx  %llu\n", file_size, file_size);
  pwrite64(fd, "", 1, file_size);

  char* mapping_start = static_cast<char*>(mmap64(0, window_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, file_size - window_size));


  // scribble into buffer and walk the window
  for (;;) {
    memset(mapping_start, 0, window_size);

    // grow file
    file_size += window_size;

    //ftruncate64(fd, file_size);
    pwrite64(fd, "", 1, file_size);

    file_page_offset += window_pages;
    //munmap(mapping_start, window_size);
    //mapping_start = static_cast<char*>(mmap64(mapping_start, window_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, file_size - window_size));
    while (remap_file_pages(mapping_start, window_size, 0, file_page_offset, MAP_SHARED /*| MAP_NONBLOCK*/) <0)
      perror("remap_file_pages");

    total_bytes += window_size;
  }
}



void do_filewrite(int fd)
{
  FILE* ptr = fdopen(fd, "w+");

  int line_len = 80;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    fwrite(buf, line_len, 1, ptr);
    total_bytes += line_len;
  }
}

void do_syswrite(int fd)
{
  int line_len = 64 * 1024;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    write(fd, buf, line_len);
    total_bytes += line_len;
  }
}


int main(int argc, char* argv[])
{
  if (argc != 3) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  if ((argv[1][0] != '-') || 
      ((argv[1][1] != 'm') &&
       (argv[1][1] != 'w') &&
       (argv[1][1] != 'f'))) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  int fd = open(argv[2], O_RDWR | O_CREAT | O_LARGEFILE | O_DIRECT, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

  // start the clock...
  signal(SIGALRM, timeout);
  {
    struct itimerval itv;
    itv.it_interval.tv_sec = 1;
    itv.it_interval.tv_usec = 0;
    itv.it_value = itv.it_interval;

    setitimer(ITIMER_REAL, &itv, 0);
  }

  switch (argv[1][1]) {
  case 'm': do_mapwrite(fd); break;
  case 'f': do_filewrite(fd); break;
  case 'w': do_syswrite(fd); break;
  }

  return 0;
}

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

Same test now done w/ rc1-mm2...
similar result.

However, instead of pegging 1 virtual cpu in system time, two virtual CPUs on the same core now
share the 100% load (so each shows 50% load and 50% idle).

Oprofile shows:

5610635  66.9136  zap_pte_range
711147    8.4813  _raw_spin_trylock
498282    5.9426  unmap_mapping_range
196369    2.3419  add_preempt_count
126243    1.5056  unmap_page_range
111791    1.3332  __bitmap_weight
96284     1.1483  page_address
88998     1.0614  _raw_spin_unlock

And I'm attaching the sysrq dump

-Kenny


		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Attachment #2: 3794922530-sysrq --]
[-- Type: application/octet-stream, Size: 54031 bytes --]

Nov 18 11:53:23 tux6127 kernel: [ 1309.383101] SysRq : Show State
Nov 18 11:53:23 tux6127 kernel: [ 1309.383112] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383113]                                                sibling
Nov 18 11:53:23 tux6127 kernel: [ 1309.383116]   task             PC      pid father child younger older
Nov 18 11:53:23 tux6127 kernel: [ 1309.383121] init          S 00000001   716     1      0     2               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383153] c2985ea4 c2985e94 00000004 00000001 c0143c42 c04bf900 c04bf000 c2985e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383187]        c02a6df8 c01275c3 c281ffe0 c281ffe0 00000286 c2985e6c c042b140 c281ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383218]        c281ffe0 c2985eb8 00000286 c2985eb8 c2985e80 c281f460 00000001 00003f1f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383245] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383249]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.383268]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.383283]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.383298]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.383314] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383320] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383327] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383333] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383339] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383349] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383361] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383373] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383383] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383387] migration/0   S 00000002  3364     2      1             3       (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383415] c299cf94 c299cf84 00000004 00000002 ffffe658 ffffffff c28174b8 0000654e 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383440]        00000000 c281f460 00000001 00000001 0000007d c281f4b4 c281fe00 00000001 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383465]        c299cf5c c042b48a c281f460 c299cf94 c0118006 c2817460 00000000 00000c53 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383493] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383499]  [<c0119a1b>] migration_thread+0x81/0x10d
Nov 18 11:53:23 tux6127 kernel: [ 1309.383517]  [<c0132844>] kthread+0xb7/0xbc
Nov 18 11:53:23 tux6127 kernel: [ 1309.383531]  [<c0101181>] kernel_thread_helper+0x5/0xb
Nov 18 11:53:23 tux6127 kernel: [ 1309.383543] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383548] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383553] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383558] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383563] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383574] .....[<c0119a1b>] ..   ( <= migration_thread+0x81/0x10d)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383584] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383595] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383606] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383611] ksoftirqd/0   S 00000002  3580     3      1             4     2 (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383642] c29a0f98 c29a0f88 00000004 00000002 c05b3000 c29a0f68 c29a0f38 c012381f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383668]        c29a0f40 c29a0f60 c2a759d0 c0118a46 ffffffff c29a0000 00000000 000000b1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383697]        c05a6380 c29a0000 c05a6380 c29a0fb4 c010390e c2817460 00000000 0000012d 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383730] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383735]  [<c0123d02>] ksoftirqd+0xde/0x106
Nov 18 11:53:23 tux6127 kernel: [ 1309.383746]  [<c0132844>] kthread+0xb7/0xbc
Nov 18 11:53:23 tux6127 kernel: [ 1309.383760]  [<c0101181>] kernel_thread_helper+0x5/0xb
Nov 18 11:53:23 tux6127 kernel: [ 1309.383774] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383779] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383784] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383789] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383795] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383806] .....[<c0123d02>] ..   ( <= ksoftirqd+0xde/0x106)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383816] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383830] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383838] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383842] watchdog/0    S 00000002  3304     4      1             5     3 (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383872] c29a4f44 c29a4f30 00000004 00000002 00000286 c29a4eec c02a6df8 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383897]        c29899d0 c01275c3 c2817fe0 c2817fe0 00000286 c29a4f0c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383929]        0000002d c29a4f58 f1e7f9d0 00000000 c2817e00 c2817460 00000000 00000ff3 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383959] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383964]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.383977]  [<c042a5c0>] schedule_timeout_interruptible+0x1a/0x1c
Nov 18 11:53:23 tux6127 kernel: [ 1309.383990]  [<c0128356>] msleep_interruptible+0x3a/0x48
Nov 18 11:53:23 tux6127 kernel: [ 1309.384007]  [<c013d982>] watchdo900 000000d0 dc505ea4 00000130 0001a90e 00000000 f24909d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396571]        c0132be1 ef7ec1e8 f24909d0 00000000 c282fe00 c282f460 00000003 00008795 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396581] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396583]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396589]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396594]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396599]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396603] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396605] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396607] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396609] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396612] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396616] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396621] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396625] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396629] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396631] kwin          S 00000003   784  4084   4059          4092  4075 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396641] f7f71ea4 f7f71e94 00000004 00000003 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396650]        c18ff820 c04bf900 000000d0 f2e659d0 f7f71e8c c0143c97 f7f71e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396661]        c0132be1 effd600c f2e60e04 00000286 00000286 c282f460 00000003 00004001 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396671] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396673]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396679]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396684]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396689]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396693] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396695] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396697] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396700] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396702] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396706] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396711] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396715] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396719] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396721] khotkeys      S 00000003   784  4087      1          4089  4083 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396731] f5ac2ea4 f5ac2e94 00000004 00000003 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396740]        c19300c0 c04bf900 000000d0 f24909d0 f5ac2e8c c0143c97 f5ac2e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396750]        c0132be1 f5ac2e7c c02a6df8 c0132be1 f100407c c282f460 00000003 0000b903 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396761] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396763]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396768]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396773]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396778]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396782] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396785] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396787] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396789] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396791] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396795] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396800] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396804] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396808] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396810] kdesktop      S 00000004   700  4089      1  4122    4091  4087 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396821] f2b36ea4 f2b36e90 00000004 00000004 00000286 f2402dec f2b36ed4 f2b05e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396830]        f2e659d0 c01275c3 c2827fe0 c2827fe0 00000286 f2b36e6c c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396841]        c2827fe0 f2b36eb8 f7b319d0 00000000 c2827e00 c2827460 00000002 000090b5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396851] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396853]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396858]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396863]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396868]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396873] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396875] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396877] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396879] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396882] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396886] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396890] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396895] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396899] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396900] kicker        S 00000004   784  4091      1          4094  4089 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396911] f6ba9ea4 f6ba9e90 00000004 00000004 000200d0 00000002 00000001 f4d72e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396920]        f24909d0 c01275c3 c2827fe0 c2827fe0 00000286 f6ba9e6c c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396930]        c2827fe0 f6ba9eb8 f7b319d0 00000000 c2827e00 c2827460 00000002 00007bc5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396940] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396943]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396948]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396953]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396957]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396962] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396964] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396966] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396968] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396971] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396975] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396979] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396983] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396987] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396989] kio_file      S 00000001   784  4092   4059          4096  4084 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397000] f5fecea4 f5fece94 00000004 00000001 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397009]        c1972ae0 c04bf900 c29899d0 f736a9d0 f5fece8c c0143c97 f5fece70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397019]        c0132be1 f263a00c f25e2e04 00000286 00000286 c281f460 00000001 000001f5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397030] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397032]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397037]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397043]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397047]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397052] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397054] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397056] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397058] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397061] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397065] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397069] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397073] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397077] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397079] klipper       S 00000002   764  4094      1          4099  4091 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397090] f2b54ea4 f2b54e94 00000004 00000002 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397099]        c18d3600 c04bf900 000000d0 f26c09d0 f2b54e8c c0143c97 f2b54e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397109]        c0132be1 ef12000c f26d4e04 00000286 00000286 c2817460 00000000 0000fcc7 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397119] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397122]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397127]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397132]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397137]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397141] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397143] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397145] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397147] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397150] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397154] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397158] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397163] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397167] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397169] evolution-ala S 00000003   772  4096   4059          4097  4092 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397179] f266df00 f266def0 00000004 00000003 c04bf900 000000d0 f266deac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397188]        c0132be1 ef69f07c f26f6e04 00000286 00000286 f26f6e04 f266dec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397199]        f26f6e04 ef69f07c f266ded8 c0132bff ef69f000 c282f460 00000003 000181d5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397209] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397211]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397217]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397221]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397226]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397231] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397233] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397235] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397237] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397240] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397244] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397248] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397253] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397256] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397259] evolution-ala S 00000003  3140  4106   4059                4097 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397270] f2648f00 f2648ef0 00000004 00000003 c04bf900 000000d0 f2648eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397279]        c0132be1 f1d060ec f21fae04 00000286 00000286 f21fae04 f2648ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397290]        f21fae04 f1d060ec f2648ed8 c0132bff f1d06000 c282f460 00000003 000040c6 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397301] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397303]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397308]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397313]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397318]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397323] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397325] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397327] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397329] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397331] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397336] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397340] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397344] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397348] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397350] gkrellm       S 00000003   772  4097   4059          4106  4096 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397360] f6bb9f00 f6bb9eec 00000004 00000003 c04bf900 000000d0 f6bb9eac 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397369]        c29909d0 c01275c3 c282ffe0 c282ffe0 00000286 f6bb9ec8 c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397379]        00000045 f6bb9f14 f7b319d0 00000000 c282fe00 c282f460 00000003 00014414 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397390] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397392]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397397]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397402]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397407]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397411] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397414] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397416] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397418] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397420] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397424] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397429] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397435] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397439] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397441] gconfd-2      S 00000002   752  4099      1          4102  4094 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397452] f6a83f00 f6a83ef0 00000004 00000002 c04bf900 000000d0 f6a83eac f6a83eb0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397461]        c02a6df8 c01275c3 c2817fe0 c2817fe0 00000286 f6a83ec8 c042b140 c2817fe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397471]        c2817fe0 f6a83f14 00000286 f6a83f14 f6a83edc c2817460 00000000 00001d41 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397482] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397484]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397489]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397494]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397499]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397504] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397506] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397508] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397510] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397513] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397517] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397522] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397526] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397530] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397532] bonobo-activa S 00000002  1768  4102      1          4110  4099 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397543] f6a57f00 f6a57ef0 00000004 00000002 c04bf900 000000d0 f6a57eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397553]        c0132be1 f1c270b4 f1cfee04 00000286 00000286 f1cfee04 f6a57ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397563]        f1cfee04 f1c270b4 f6a57ed8 c0132bff f1c27000 c2817460 00000000 00017814 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397573] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397576]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397581]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397585]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397590]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397595] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397597] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397599] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397601] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397604] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397608] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397612] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397617] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397621] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397623] evolution-dat S 00000004   812  4110      1          4111  4102 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397633] f2aaff00 f2aafef0 00000004 00000004 c04bf900 000000d0 f2b039d0 f2aafed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397642]        c0143c97 f2aafeb8 c02a6df8 c0132be1 f1d34044 f2e37df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397652]        f2e37df8 f2aafed0 c042b524 f2e37df8 f1d34044 c2827460 00000002 00000e63 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397663] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397665]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397670]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397675]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397680]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397684] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397686] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397689] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397691] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397693] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397697] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397702] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397706] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397710] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397712] evolution-dat S 00000004  3004  4111      1          4118  4110 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397724] f6508f00 f6508ef0 00000004 00000004 c04bf900 000000d0 f6508eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397733]        c0132be1 f1d490ec c29909d0 00000286 00000286 f2046e04 f6508ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397743]        f2046e04 f1d490ec f6508ed8 c0132bff f1d49000 c2827460 00000002 000004a6 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397754] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397756]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397762]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397766]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397771]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397776] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397779] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397782] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397785] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397788] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397792] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397796] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397800] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397804] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397806] evolution-dat S 00000003  3048  4118      1          4701  4111 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397818] f256bf00 f256bef0 00000004 00000003 c04bf900 000000d0 f20749d0 f256bed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397827]        c0143c97 f256beb8 c02a6df8 c0132be1 f221d00c f2048df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397837]        f2048df8 f256bed0 c042b524 f2048df8 f221d00c c282f460 00000003 00004383 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397848] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397850]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397856]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397860]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397865]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397870] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397872] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397874] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397876] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397879] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397883] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397887] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397892] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397895] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397897] xterm         S 00000003   724  4122   4089  4123    4135       (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397908] f2a86ea4 f2a86e94 00000004 00000003 c02e0004 f46c87f8 c04cf964 f2a86e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397917]        c02a6df8 c01275c3 c282ffe0 c282ffe0 00000286 f2a86e6c c042b140 c282ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397928]        c282ffe0 f2a86eb8 00000286 f2a86eb8 f2a86e80 c282f460 00000003 000013c2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397938] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397940]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397945]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397951]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397955]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397960] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397962] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397964] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397966] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397969] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397973] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397978] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397982] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397986] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397988] bash          S 00000002   688  4123   4122  4238               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397998] f26e5f08 f26e5ef8 00000004 00000002 00000003 c25a6230 c04bf928 000200d2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398007]        f58739d0 f26e5ee4 f26e5ec4 f26e5ec8 c04cb6dc c04cb6c0 f26e5ed4 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398018]        f26e5edc c02a6df8 c0132be1 f26e5f54 f5e4fe7c c2817460 00000000 000005d8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398028] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398030]  [<c0121f28>] do_wait+0x193/0x37f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398035]  [<c01221d4>] sys_wait4+0x3e/0x42
Nov 18 11:53:23 tux6127 kernel: [ 1309.398040]  [<c01221ff>] sys_waitpid+0x27/0x2b
Nov 18 11:53:23 tux6127 kernel: [ 1309.398045]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398049] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398051] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398053] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398056] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398058] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398062] .....[<c0121f28>] ..   ( <= do_wait+0x193/0x37f)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398066] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398070] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398074] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398076] xterm         S 00000001   724  4135   4089  4136    4241  4122 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398087] f2227ea4 f2227e90 00000004 00000001 c02e0004 f1dc77f8 c04cf964 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398096]        c04b9ae0 c01275c3 c281ffe0 c281ffe0 00000286 f2227e6c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398107]        00000013 f2227eb8 f1e7f9d0 00000000 c281fe00 c281f460 00000001 0001075f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398117] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398119]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398124]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398129]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398134]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398138] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398140] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398142] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398144] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398147] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398151] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398155] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398160] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398164] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398166] bash          S 00000003   724  4136   4135                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398176] f646ce78 f646ce68 00000004 00000003 f1dc77f8 f646ce20 f646ce24 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398185]        c02e0004 f646ce30 c02a6df8 c02e0120 f2bde804 c04cf964 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398196]        00000286 f646ce48 c042b524 c04cf964 f2bde804 c282f460 00000003 0000c649 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398206] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398208]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398214]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398220]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398225]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398230]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398235]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398239] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398241] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398243] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398245] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398248] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398252] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398256] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398261] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398265] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398266] writetest     R running   776  4238   4123                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398276] xterm         S 00000003   784  4241   4089  4242    4798  4135 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398287] f45f4ea4 f45f4e94 00000004 00000003 c02e0004 f1d2b7f8 c04cf964 f45f4e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398296]        c02a6df8 c01275c3 c282ffe0 c282ffe0 00000286 f45f4e6c c042b140 c282ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398307]        c282ffe0 f45f4eb8 00000286 f45f4eb8 f45f4e80 c282f460 00000003 00000de7 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398317] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398319]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398324]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398330]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398334]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398339] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398341] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398343] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398345] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398348] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398352] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398356] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398361] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398364] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398366] bash          S 00000001   784  4242   4241                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398376] f1a06e78 f1a06e68 00000004 00000001 f1d2b7f8 f1a06e20 f1a06e24 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398386]        c02e0004 f1a06e30 c02a6df8 c02e0120 f1703804 c04cf964 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398396]        00000286 f1a06e48 c042b524 c04cf964 f1703804 c281f460 00000001 000101a8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398407] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398409]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398414]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398420]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398424]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398431]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398437]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398442] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398444] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398446] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398448] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398450] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398455] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398459] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398465] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398469] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398472] oprofiled     S 00000001  1396  4701      1                4118 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398482] f0a08f04 f0a08ef4 00000004 00000001 00030002 f0a08f80 c012293c f09a5ec8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398492]        f0a08f5c 00030002 c04c0b84 00000000 c01187e8 c04c0b84 00000003 f0a08ed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398502]        c02a6df8 c0132ce2 f0a08f38 f8b99b04 00000286 c281f460 00000001 000736eb 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398512] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398515]  [<f8b935b1>] event_buffer_read+0x157/0x15c [oprofile]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398526]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398531]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398536]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398541] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398543] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398545] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398547] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398549] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398554] .....[<f8b935b1>] ..   ( <= event_buffer_read+0x157/0x15c [oprofile])
Nov 18 11:53:23 tux6127 kernel: [ 1309.398562] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398566] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398570] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398572] xterm         S 00000002   724  4798   4089  4799    4809  4241 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398583] ef65aea4 ef65ae94 00000004 00000002 c02e0004 f1dc67f8 c04cf964 ef65ae54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398592]        c02a6df8 c01275c3 c2817fe0 c2817fe0 00000286 ef65ae6c c042b140 c2817fe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398602]        c2817fe0 ef65aeb8 00000286 ef65aeb8 ef65ae80 c2817460 00000000 000013fd 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398613] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398615]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398620]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398625]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398630]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398634] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398636] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398638] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398641] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398643] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398648] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398652] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398656] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398660] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398662] bash          S 000000DF   724  4799   4798                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398673] efbf3e78 f272a9d0 c281f4b4 000000df 00000001 00000001 00000082 f1e28e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398682]        f1e7f9d0 00000001 efbf3e30 4d1775c4 000000df 00001add 00000000 f272a9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398692]        0000003f ef6bf044 f272a9d0 00000000 c281fe00 c281f460 00000001 00000c2a 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398702] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398704]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398709]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398715]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398719]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398724]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398729]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398734] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398736] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398738] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398740] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398742] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398746] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398751] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398755] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398759] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398761] xterm         S 00000004   724  4809   4089  4810          4798 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398771] ef70cea4 ef70ce90 00000004 00000004 c02e0004 f17847f8 c04cf964 f2b05e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398780]        f2e659d0 c01275c3 c2827fe0 c2827fe0 00000286 ef70ce6c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398791]        c2827fe0 ef70ceb8 f1e7f9d0 00000000 c2827e00 c2827460 00000002 00003c7b 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398801] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398803]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398808]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398813]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398818]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398823] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398825] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398827] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398829] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398831] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398835] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398840] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398844] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398848] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398850] bash          S 00000002   724  4810   4809  4820               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398860] ef683f08 ef683ef8 00000004 00000002 00000003 c25fae60 c04bf928 000200d2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398869]        ef7699d0 ef683ee4 ef683ec4 c02a6df8 c014cab3 c25b1930 ef683ed4 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398880]        ef683edc c02a6df8 c0132be1 ef683f54 ef7efe7c c2817460 00000000 000059b4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398890] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398893]  [<c0121f28>] do_wait+0x193/0x37f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398898]  [<c01221d4>] sys_wait4+0x3e/0x42
Nov 18 11:53:23 tux6127 kernel: [ 1309.398902]  [<c01221ff>] sys_waitpid+0x27/0x2b
Nov 18 11:53:23 tux6127 kernel: [ 1309.398907]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398911] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398913] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398916] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398918] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398920] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398924] .....[<c0121f28>] ..   ( <= do_wait+0x193/0x37f)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398928] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398933] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398937] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398939] firefox-bin   S 00000002   512  4820   4810          4851       (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398949] ef682f00 ef682ef0 00000004 00000002 ef682ec8 c0143c97 ef682eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398958]        c0132be1 ef682eb8 c04b9ae0 c0132be1 efb930d0 ef7b0df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398969]        ef7b0df8 ef682ed0 c042b524 ef7b0df8 efb930d0 c2817460 00000000 000002b1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398979] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398981]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398986]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.398991]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398996]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399001] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399003] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399005] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399007] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399009] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399014] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399018] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399022] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399026] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399028] firefox-bin   S 00000004   960  4851   4810          4853  4820 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399039] ef79df00 ef79def0 00000004 00000004 ef79dea4 c02a6df8 c0132be1 ef06f028 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399048]        eed24e04 00000286 00000286 eed24e04 ef79debc c042b524 eed24e04 ef06f028 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399058]        ef79ded0 c0132bff ef06f000 ef6f21e4 ef79df98 c2827460 00000002 00001391 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399069] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399071]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.399076]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.399081]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.399086]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399090] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399093] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399095] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399097] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399099] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399103] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399108] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399112] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399116] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399118] firefox-bin   S 00000004  3256  4853   4810                4851 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399130] ef70be6c ef70be58 00000004 00000004 ef70be10 ef70be14 c02a6df8 f4ed7e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399140]        f7b319d0 c01275c3 c2827fe0 c2827fe0 00000286 ef70be34 c042b140 f2e659d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399150]        00000022 ef70be80 f2e659d0 00000000 c2827e00 c2827460 00000002 000016a1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399160] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399163]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.399168]  [<c013707f>] futex_wait+0x217/0x2a4
Nov 18 11:53:23 tux6127 kernel: [ 1309.399174]  [<c0137394>] do_futex+0x42/0xaf
Nov 18 11:53:23 tux6127 kernel: [ 1309.399178]  [<c01374ec>] sys_futex+0xeb/0xf7
Nov 18 11:53:23 tux6127 kernel: [ 1309.399183]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399188] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399190] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399192] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399194] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399196] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399201] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399205] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399209] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399213] 

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

Instead of sysrq with 't', here is the sysrq with 'p'... it agrees with oprofile

Has anyone else been able to repoduce this?

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

[-- Attachment #2: 3794922530-sysrq --]
[-- Type: application/octet-stream, Size: 11561 bytes --]

Nov 18 12:17:10 tux6127 kernel: [ 2735.531683] SysRq : Show Regs
Nov 18 12:17:10 tux6127 kernel: [ 2735.531777] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.531825] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.531876] EIP: 0060:[<c02d6299>] CPU: 0
Nov 18 12:17:10 tux6127 kernel: [ 2735.531934] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.531986]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532040] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c056b000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532094] ESI: f7ab9c6c EDI: f7ab9bf8 EBP: c056bf88 DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532185] CR0: 8005003b CR2: b7f33e64 CR3: 364a5000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532239] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532289] | preempt count: 00010001 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.532340] | 1 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.532391] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532443] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532533] .....[<c01002dd>] ..   ( <= _stext+0x45/0x49)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532623] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.532670]  ----------- IPI show regs -----------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532766] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.532772] EIP: 0060:[<c02d6299>] CPU: 1
Nov 18 12:17:10 tux6127 kernel: [ 2735.532777] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.532780]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532784] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c298a000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532787] ESI: f7abec6c EDI: f7abebf8 EBP: c298af5c DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532798] CR0: 8005003b CR2: 0804d054 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532801]  [<c0101155>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532806]  [<c0112afd>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.532814]  [<c0110a58>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532821]  [<c0104cae>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532827]  [<c0103b0e>] nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.532832]  [<c02d63c9>] acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532839]  [<c0100e10>] cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532848]  [<c0111032>] start_secondary+0x17f/0x2ef
Nov 18 12:17:10 tux6127 kernel: [ 2735.532854]  [<00000000>] 0x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.532865]  [<c298afb4>] 0xc298afb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532871] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532874] | preempt count: 00020002 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.532876] | 2 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.532878] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532881] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532885] .....[<c0111032>] ..   ( <= start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532892] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.532904] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532909] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.532911]  ----------- IPI show regs -----------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532918] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.532923] EIP: 0060:[<c02d6299>] CPU: 3
Nov 18 12:17:10 tux6127 kernel: [ 2735.532931] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.532935]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532939] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c2997000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532943] ESI: f7abfc6c EDI: f7abfbf8 EBP: c2997f5c DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532948] CR0: 8005003b CR2: b7ef9310 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532952]  [<c0101155>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532959]  [<c0112afd>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.532967]  [<c0110a58>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532972]  [<c0104cae>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532978]  [<c0103b0e>] nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.532983]  [<c02d63c9>] acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532989]  [<c0100e10>] cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532994]  [<c0111032>] start_secondary+0x17f/0x2ef
Nov 18 12:17:10 tux6127 kernel: [ 2735.532999]  [<00000000>] 0x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533005]  [<c2997fb4>] 0xc2997fb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533009] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533012] | preempt count: 00020002 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533015] | 2 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533017] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533020] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533025] .....[<c0111032>] ..   ( <= start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533031] .. [<c042b0ef>] .... _spin_lock+0x6c/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533037] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533042] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533045] ----------- IPI show regs -----------<3>Debug: sleeping function called from invalid context at drivers/char/vt.c:2843
Nov 18 12:17:10 tux6127 kernel: [ 2735.533054] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533056] in_atomic():1, irqs_disabled():1
Nov 18 12:17:10 tux6127 kernel: [ 2735.533060] Pid: 4238, comm:            writetest
Nov 18 12:17:10 tux6127 kernel: [ 2735.533063]  [<c0103df0>] EIP: 0060:[<c02a6841>] CPU: 2
Nov 18 12:17:10 tux6127 kernel: [ 2735.533068] dump_stack+0x1e/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.533072] EIP is at _raw_spin_trylock+0xb/0x2d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533076]  [<c011b38e>]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533083] EAX: 00000001 EBX: c19589ec ECX: f73ad000 EDX: f1e7f9d0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533087] __might_sleep+0xa7/0xafESI: f73ad000 EDI: b7c00000 EBP: f73adbf4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533095]  DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533098]  [<c02f179d>] CR0: 8005003b CR2: b7fb8000 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.533104] do_unblank_screen+0x150/0x155 [<c0101155>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533109]  [<c02f17b4>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533114] unblank_screen+0x12/0x16 [<c0112afd>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533119]  [<c0114cb0>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.533123] bust_spinlocks+0x2c/0x54 [<c0110a58>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533129]  [<c0112b13>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533134] smp_show_regs+0x50/0x57 [<c0104cae>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533139]  [<c0110a58>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533144] smp_nmi_callback+0x69/0x7d [<c0103b0e>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533149]  [<c0104cae>] nmi_stack_correct+0x1d/0x22do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533154] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533156]  [<c042b0aa>]  [<c0103b0e>] _spin_lock+0x27/0x90nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.533163] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533166]  [<c014bb29>]  [<c02d63c9>] zap_pte_range+0x5b/0x2f7acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533175] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533178]  [<c014be76>]  [<c0100e10>] unmap_page_range+0xb1/0x117cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533186] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533188]  [<c014bfb8>]  [<c0111032>] unmap_vmas+0xdc/0x221
Nov 18 12:17:10 tux6127 kernel: [ 2735.533193] start_secondary+0x17f/0x2ef [<c014c1a3>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533199]  [<00000000>] zap_page_range+0xa6/0x1100x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533204] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533207]  [<c014cd62>]  [<c2997fb4>] unmap_mapping_range_vma+0x46/0xbb0xc2997fb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533215] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533218]  [<c014cf0a>] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533222] | preempt count: 00020001 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533226] unmap_mapping_range+0x133/0x1ed| 1 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533232] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533234] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533237]  [<c0147dc2>] .. [<c0100dc6>] .... invalidate_inode_pages2_range+0x1f4/0x22acpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533245] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533247] .....[<c0111032>] ..   ( <=  [<c0147e19>] start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533254] invalidate_inode_pages2+0x21/0x28
Nov 18 12:17:10 tux6127 kernel: [ 2735.533258] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533260]   [<c01eba54>] nfs_file_direct_write+0x1c3/0x1ff
Nov 18 12:17:10 tux6127 kernel: [ 2735.533271]  [<c01ca23e>] nfs_file_write+0x113/0x115
Nov 18 12:17:10 tux6127 kernel: [ 2735.533276]  [<c015f99d>] do_sync_write+0xa7/0xe7
Nov 18 12:17:10 tux6127 kernel: [ 2735.533282]  [<c015fa83>] vfs_write+0xa6/0x17d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533287]  [<c015fd42>] sys_pwrite64+0x7d/0x81
Nov 18 12:17:10 tux6127 kernel: [ 2735.533291]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 12:17:10 tux6127 kernel: [ 2735.533295] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533297] | preempt count: 00020004 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533299] | 4 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533300] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533303] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533307] .....[<c014ce6e>] ..   ( <= unmap_mapping_range+0x97/0x1ed)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533311] .. [<c014c127>] .... zap_page_range+0x2a/0x110
Nov 18 12:17:10 tux6127 kernel: [ 2735.533315] .....[<c014cd62>] ..   ( <= unmap_mapping_range_vma+0x46/0xbb)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533319] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533322] .....[<c014bb29>] ..   ( <= zap_pte_range+0x5b/0x2f7)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533326] .. [<c042b0ef>] .... _spin_lock+0x6c/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533330] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533334] 

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Andrew Morton <akpm@osdl.org> wrote:
> Could you send the test app please?  (Apologies if you've already done so
> and I missed it).

Here is an strace from a reduced test:
open("/mnt/bar", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0644) = 3
pwrite(3, "\0", 1, 4292870144)          = 1
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xffc00) = 0xb7ace000
pwrite(3, "\0", 1, 4294967296)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0xffe00) = 0xb7ace000
pwrite(3, "\0", 1, 4297064448)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0x100000) = 0xb7ace000
pwrite(3, "\0", 1, 4299161600)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0x100200) = 0xb7ace000
pwrite(3, "\0", 1, 4301258752


The final pwrite never returns.
This is with 2.6.15-rc1 + Trond's NFS patch (-mm2 would die on strace).

The only change from the previous test is that this one uses munmap/mmap64 instead of
remap_file_pages.

showPc shows:

SysRq : Show Regs

Pid: 4271, comm:            writetest
EIP: 0060:[<c029cd2b>] CPU: 0
EIP is at prio_tree_first+0x26/0xb8
 EFLAGS: 00000292    Tainted: P       (2.6.15-rc1)
EAX: 00000000 EBX: f6765d94 ECX: f6765d94 EDX: 00000200
ESI: 00000000 EDI: f5da475c EBP: f6765db4 DS: 007b ES: 007b
CR0: 8005003b CR2: b7590000 CR3: 36b5d000 CR4: 000006d0
 [<c029ce5e>] prio_tree_next+0xa1/0xa3
 [<c014822a>] vma_prio_tree_next+0x27/0x51
 [<c014b1dc>] unmap_mapping_range+0x18b/0x210
 [<c0120d06>] __do_softirq+0x6a/0xd1
 [<c0103a24>] apic_timer_interrupt+0x1c/0x24
 [<c0146577>] invalidate_inode_pages2_range+0x215/0x24c
 [<c01465cd>] invalidate_inode_pages2+0x1f/0x26
 [<c01e4031>] nfs_file_direct_write+0x1e1/0x21a
 [<c0159ea4>] do_sync_write+0xc7/0x10d
 [<c012ff32>] autoremove_wake_function+0x0/0x57
 [<c0159f92>] vfs_write+0xa8/0x177
 [<c015a275>] sys_pwrite64+0x88/0x8c
 [<c0102f51>] syscall_call+0x7/0xb


-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

Trond Myklebust wrote:
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).

the architecture is to allow the NFS protocol and server to do these checks.

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Thu, 2005-11-17 at 12:02 -0500, Chuck Lever wrote:
> Trond Myklebust wrote:
> > I had a quick look at nfs_file_direct_write(), and among other things,
> > it would appear that it is not doing any of the usual overflow checks on
> > *pos and the count size (see generic_write_checks()). In particular,
> > checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> > not set (and also against overflow vs. s_maxbytes, but that is less
> > relevant here).
> 
> the architecture is to allow the NFS protocol and server to do these checks.

No it isn't.

The NFS protocol has no clue as to whether or not you opened the file
using O_LARGEFILE. For NFSv2, we do _not_ want file pointers to wrap
once they hit the 32-bit boundary.

The protocol and server cannot be involved in any of those checks. They
must be done on the client.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

Yet another data point:

Under 2.6.8-2 (debain sarge kernel), the test does not cause a spin.
Instead, the file extension via pwrite does not allow the new pages to be usable by
remap_file_pages.
However, munmap/mmap are happy to use pages intoduces by the pwrite...
and happily writes more than 4GB.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 11:09 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> > > 
> > > > That will fix it, but the PageWriteback accounting is still wrong.
> > > > 
> > > > Is it not possible to use set_page_writeback()/end_page_writeback()?
> > > 
> > > Not really. The pages aren't flushed at this time. We the point is to
> > > gather several pages and coalesce them into one over-the-wire RPC call.
> > > That means we cannot really do it from inside ->writepage().
> > > 
> > 
> > I still don't get it.
> > 
> > Once nfs_writepage() has been called, the page is conceptually "under
> > writeback", yes?  In that, at some point in the future, it will be written
> > to backing store.
> > 
> > Hence it's perfectly appropriate to run set_page_writepage() within
> > nfs_writepage().  It's a matter of finding the right place for the
> > end_page_writeback().
> 
> The point is that the process of flushing has not been started at that
> time, so anybody that calls wait_on_page_writeback() immediately after
> calling writepage() may end up waiting for a very long time indeed
> (probably until the next pdflush).

But block-backed filesytems have the same concern: we don't want to do a
whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
appropriate place to be building up these large I/Os.

NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
large I/Os it leaves the I/O pending on return from nfs_writepage().  It
appears to flush any pending pages on the exit path from nfs_writepages().

If that's a correct reading then there doesn't appear to be any way in
which there's dangling I/O left to do after nfs_writepages() completes.

If there _is_ dandling I/O left over then that's problematic, and probably
doesn't buy us much in the way of performance benefit.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 13:31 -0800, Andrew Morton wrote:
> But block-backed filesytems have the same concern: we don't want to do a
> whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
> appropriate place to be building up these large I/Os.
> 
> NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
> large I/Os it leaves the I/O pending on return from nfs_writepage().  It
> appears to flush any pending pages on the exit path from nfs_writepages().
> 
> If that's a correct reading then there doesn't appear to be any way in
> which there's dangling I/O left to do after nfs_writepages() completes.

Agreed. AFAICS, nfs_writepages should be quite OK, however writepage()
on its own _is_ problematic.

Look at the usage in write_one_page(), which calls directly down to
->writepage(), and then immediately does a wait_on_page_writeback().

How is the filesystem supposed to distinguish between the cases
"VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 13:31 -0800, Andrew Morton wrote:
> > But block-backed filesytems have the same concern: we don't want to do a
> > whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
> > appropriate place to be building up these large I/Os.
> > 
> > NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
> > large I/Os it leaves the I/O pending on return from nfs_writepage().  It
> > appears to flush any pending pages on the exit path from nfs_writepages().
> > 
> > If that's a correct reading then there doesn't appear to be any way in
> > which there's dangling I/O left to do after nfs_writepages() completes.
> 
> Agreed. AFAICS, nfs_writepages should be quite OK, however writepage()
> on its own _is_ problematic.
> 
> Look at the usage in write_one_page(), which calls directly down to
> ->writepage(), and then immediately does a wait_on_page_writeback().
> 
> How is the filesystem supposed to distinguish between the cases
> "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> 

Via the writeback_control, hopefully.

For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
immediately (it appears to not do so).

For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
should be pushed immediately.  nfs_writepage() seems to dtrt here.

With the proposed changes, we don't need that iput() in nfs_writepage(). 
That worries me because I recall from a couple of years back that there are
really subtle races with doing iput() on the vmscan->writepage() path. 
Cannot remember what they were though...

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:

> > How is the filesystem supposed to distinguish between the cases
> > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > 
> 
> Via the writeback_control, hopefully.
> 
> For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> immediately (it appears to not do so).

Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
discriminate between a writepages() and a single writepage() situation,
whatever the original intention was.

> For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> should be pushed immediately.  nfs_writepage() seems to dtrt here.
> 
> With the proposed changes, we don't need that iput() in nfs_writepage(). 
> That worries me because I recall from a couple of years back that there are
> really subtle races with doing iput() on the vmscan->writepage() path. 
> Cannot remember what they were though...

Possibly to do with block filesystems that may trigger ->writepage()
while inside iput_final()? NFS can't do that.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 17:23 -0500, Trond Myklebust wrote:
> On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> 
> > > How is the filesystem supposed to distinguish between the cases
> > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > 
> > 
> > Via the writeback_control, hopefully.
> > 
> > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > immediately (it appears to not do so).
> 
> Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> discriminate between a writepages() and a single writepage() situation,
> whatever the original intention was.

IMHO, the correct way to distinguish between the two would be to use the
wbc->nr_to_write field. If all the instances of writepage() were to set
that field to '1', then the filesystems could do the right thing.

As it is, you have shrink_list() that sets it to the value
"SWAP_CLUSTER_MAX" for no apparent reason...

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 17:23 -0500, Trond Myklebust wrote:
> > On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> > 
> > > > How is the filesystem supposed to distinguish between the cases
> > > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > > 
> > > 
> > > Via the writeback_control, hopefully.
> > > 
> > > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > > immediately (it appears to not do so).
> > 
> > Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> > discriminate between a writepages() and a single writepage() situation,
> > whatever the original intention was.
> 
> IMHO, the correct way to distinguish between the two would be to use the
> wbc->nr_to_write field. If all the instances of writepage() were to set
> that field to '1', then the filesystems could do the right thing.

yes, except ->writepages is supposed to decrement nr_to_write as it proceeds,
so it'll end up at `1' by accident on the last go around the loop.

I think a separate boolean is better - it's just a single bit.

> As it is, you have shrink_list() that sets it to the value
> "SWAP_CLUSTER_MAX" for no apparent reason...

How weird.  That's presumably wrong, but I'd need to check the changelogs
to doublecheck.  ugh, 264 of them.

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> 
> > > How is the filesystem supposed to distinguish between the cases
> > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > 
> > 
> > Via the writeback_control, hopefully.
> > 
> > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > immediately (it appears to not do so).
> 
> Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> discriminate between a writepages() and a single writepage() situation,
> whatever the original intention was.

Could peek at wbc->nr_pages, or add another boolean to writeback_control
for this.

diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
--- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
+++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
@@ -53,10 +53,11 @@ struct writeback_control {
 	loff_t start;
 	loff_t end;
 
-	unsigned nonblocking:1;			/* Don't get stuck on request queues */
-	unsigned encountered_congestion:1;	/* An output: a queue is full */
-	unsigned for_kupdate:1;			/* A kupdate writeback */
-	unsigned for_reclaim:1;			/* Invoked from the page allocator */
+	unsigned nonblocking:1;		/* Don't get stuck on request queues */
+	unsigned encountered_congestion:1; /* An output: a queue is full */
+	unsigned for_kupdate:1;		/* A kupdate writeback */
+	unsigned for_reclaim:1;		/* Invoked from the page allocator */
+	unsigned for_writepages:1;	/* This is a writepages() call */
 };
 
 /*
diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
--- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
+++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
@@ -550,11 +550,17 @@ void __init page_writeback_init(void)
 
 int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
+	int ret;
+
 	if (wbc->nr_to_write <= 0)
 		return 0;
+	wbc->for_writepages = 1;
 	if (mapping->a_ops->writepages)
-		return mapping->a_ops->writepages(mapping, wbc);
-	return generic_writepages(mapping, wbc);
+		ret =  mapping->a_ops->writepages(mapping, wbc);
+	else
+		ret = generic_writepages(mapping, wbc);
+	wbc->for_writepages = 0;
+	return ret;
 }
 
 /**
_

> > For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> > should be pushed immediately.  nfs_writepage() seems to dtrt here.
> > 
> > With the proposed changes, we don't need that iput() in nfs_writepage(). 
> > That worries me because I recall from a couple of years back that there are
> > really subtle races with doing iput() on the vmscan->writepage() path. 
> > Cannot remember what they were though...
> 
> Possibly to do with block filesystems that may trigger ->writepage()
> while inside iput_final()? NFS can't do that.

iput_final() can call truncate_inode_pages - maybe it was a deadlock, but
I'm fairly sure it was a race.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:44 -0800, Andrew Morton wrote:

> Could peek at wbc->nr_pages, or add another boolean to writeback_control
> for this.
> 
> diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
> --- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
> @@ -53,10 +53,11 @@ struct writeback_control {
>  	loff_t start;
>  	loff_t end;
>  
> -	unsigned nonblocking:1;			/* Don't get stuck on request queues */
> -	unsigned encountered_congestion:1;	/* An output: a queue is full */
> -	unsigned for_kupdate:1;			/* A kupdate writeback */
> -	unsigned for_reclaim:1;			/* Invoked from the page allocator */
> +	unsigned nonblocking:1;		/* Don't get stuck on request queues */
> +	unsigned encountered_congestion:1; /* An output: a queue is full */
> +	unsigned for_kupdate:1;		/* A kupdate writeback */
> +	unsigned for_reclaim:1;		/* Invoked from the page allocator */
> +	unsigned for_writepages:1;	/* This is a writepages() call */
>  };
>  
>  /*
> diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
> --- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
> @@ -550,11 +550,17 @@ void __init page_writeback_init(void)
>  
>  int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
>  {
> +	int ret;
> +
>  	if (wbc->nr_to_write <= 0)
>  		return 0;
> +	wbc->for_writepages = 1;
>  	if (mapping->a_ops->writepages)
> -		return mapping->a_ops->writepages(mapping, wbc);
> -	return generic_writepages(mapping, wbc);
> +		ret =  mapping->a_ops->writepages(mapping, wbc);
> +	else
> +		ret = generic_writepages(mapping, wbc);
> +	wbc->for_writepages = 0;
> +	return ret;
>  }

That would work...

> > > For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> > > should be pushed immediately.  nfs_writepage() seems to dtrt here.
> > > 
> > > With the proposed changes, we don't need that iput() in nfs_writepage(). 
> > > That worries me because I recall from a couple of years back that there are
> > > really subtle races with doing iput() on the vmscan->writepage() path. 
> > > Cannot remember what they were though...
> > 
> > Possibly to do with block filesystems that may trigger ->writepage()
> > while inside iput_final()? NFS can't do that.
> 
> iput_final() can call truncate_inode_pages - maybe it was a deadlock, but
> I'm fairly sure it was a race.

Doesn't matter. There can be no dirty pages when NFS hits iput_final().
We make sure that we flush them into the filesystem accounting before we
release the file descriptor, then we make sure that we don't release the
dentry before the inode has been synced up.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:44 -0800, Andrew Morton wrote:
> diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
> --- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
> @@ -53,10 +53,11 @@ struct writeback_control {
>  	loff_t start;
>  	loff_t end;
>  
> -	unsigned nonblocking:1;			/* Don't get stuck on request queues */
> -	unsigned encountered_congestion:1;	/* An output: a queue is full */
> -	unsigned for_kupdate:1;			/* A kupdate writeback */
> -	unsigned for_reclaim:1;			/* Invoked from the page allocator */
> +	unsigned nonblocking:1;		/* Don't get stuck on request queues */
> +	unsigned encountered_congestion:1; /* An output: a queue is full */
> +	unsigned for_kupdate:1;		/* A kupdate writeback */
> +	unsigned for_reclaim:1;		/* Invoked from the page allocator */
> +	unsigned for_writepages:1;	/* This is a writepages() call */
>  };
>  
>  /*
> diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
> --- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
> @@ -550,11 +550,17 @@ void __init page_writeback_init(void)
>  
>  int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
>  {
> +	int ret;
> +
>  	if (wbc->nr_to_write <= 0)
>  		return 0;
> +	wbc->for_writepages = 1;
>  	if (mapping->a_ops->writepages)
> -		return mapping->a_ops->writepages(mapping, wbc);
> -	return generic_writepages(mapping, wbc);
> +		ret =  mapping->a_ops->writepages(mapping, wbc);
> +	else
> +		ret = generic_writepages(mapping, wbc);
> +	wbc->for_writepages = 0;
> +	return ret;
>  }

The accompanying NFS patch makes use of this in order to figure out when
to flush the data correctly.
-------------
NFS: Work correctly with single-page ->writepage() calls

 Ensure that we use set_page_writeback() in the appropriate places
 to help the VM in keeping its page radix_tree in sync.
 Ensure that we always initiate flushing of data before we exit
 a single-page ->writepage() call.

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c |   22 +++++++++-------------
 1 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..95d00f9 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -189,6 +189,7 @@ static int nfs_writepage_sync(struct nfs
 		(long long)NFS_FILEID(inode),
 		count, (long long)(page_offset(page) + offset));
 
+	set_page_writeback(page);
 	nfs_begin_data_update(inode);
 	do {
 		if (count < wsize)
@@ -221,6 +222,7 @@ static int nfs_writepage_sync(struct nfs
 
 io_error:
 	nfs_end_data_update(inode);
+	end_page_writeback(page);
 	nfs_writedata_free(wdata);
 	return written ? written : result;
 }
@@ -230,19 +232,16 @@ static int nfs_writepage_async(struct nf
 		unsigned int offset, unsigned int count)
 {
 	struct nfs_page	*req;
-	int		status;
 
 	req = nfs_update_request(ctx, inode, page, offset, count);
-	status = (IS_ERR(req)) ? PTR_ERR(req) : 0;
-	if (status < 0)
-		goto out;
+	if (IS_ERR(req))
+		return PTR_ERR(req);
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);
- out:
-	return status;
+	return 0;
 }
 
 static int wb_priority(struct writeback_control *wbc)
@@ -302,11 +301,8 @@ do_it:
 	lock_kernel();
 	if (!IS_SYNC(inode) && inode_referenced) {
 		err = nfs_writepage_async(ctx, inode, page, 0, offset);
-		if (err >= 0) {
-			err = 0;
-			if (wbc->for_reclaim)
-				nfs_flush_inode(inode, 0, 0, FLUSH_STABLE);
-		}
+		if (!wbc->for_writepages)
+			nfs_flush_inode(inode, 0, 0, wb_priority(wbc));
 	} else {
 		err = nfs_writepage_sync(ctx, inode, page, 0,
 						offset, priority);
@@ -929,7 +925,7 @@ static int nfs_flush_multi(struct list_h
 	atomic_set(&req->wb_complete, requests);
 
 	ClearPageError(page);
-	SetPageWriteback(page);
+	set_page_writeback(page);
 	offset = 0;
 	nbytes = req->wb_bytes;
 	do {
@@ -992,7 +988,7 @@ static int nfs_flush_one(struct list_hea
 		nfs_list_remove_request(req);
 		nfs_list_add_request(req, &data->pages);
 		ClearPageError(req->wb_page);
-		SetPageWriteback(req->wb_page);
+		set_page_writeback(req->wb_page);
 		*pages++ = req->wb_page;
 		count += req->wb_bytes;
 	}

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> The accompanying NFS patch makes use of this in order to figure out when
>  to flush the data correctly.

OK.  So with that patch, nfs_writepages() may still leave I/O pending,
uninitiated, yes?

I don't understand why NFS hasn't been BUGging as it stands at present.  It
has several end_page_writeback() calls but no set_page_writeback()s. 
end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
wasn't PageWriteback().

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 16:25 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > The accompanying NFS patch makes use of this in order to figure out when
> >  to flush the data correctly.
> 
> OK.  So with that patch, nfs_writepages() may still leave I/O pending,
> uninitiated, yes?
> 
> I don't understand why NFS hasn't been BUGging as it stands at present.  It
> has several end_page_writeback() calls but no set_page_writeback()s. 
> end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
> wasn't PageWriteback().

It does have SetPageWriteback() calls in the asynchronous writeback
path. As you can see from the patch I just sent, I only needed to
replace them with set_page_writebacks().

Cheers,
 Trond

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 16:25 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > The accompanying NFS patch makes use of this in order to figure out when
> > >  to flush the data correctly.
> > 
> > OK.  So with that patch, nfs_writepages() may still leave I/O pending,
> > uninitiated, yes?

This?

I don't know if it'll be a problem.  One factor is that when the VFS is
doing an fsync() or whatever, it will fail to notice these left-over pages
are "dirty", so it won't launch writepage() against them.

But if they are marked PageWriteback(), sync will notice them on the second
pass and will wait upon them, which apparently could mean a stall until
pdflush kicks off the I/O?

If they're not marked PageDirty() or PageWriteback(), the VFS will miss
them altogether during the sync.  But perhaps NFS's own page tracking will
flush them and wait upon the result?

> > I don't understand why NFS hasn't been BUGging as it stands at present.  It
> > has several end_page_writeback() calls but no set_page_writeback()s. 
> > end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
> > wasn't PageWriteback().
> 
> It does have SetPageWriteback() calls in the asynchronous writeback
> path. As you can see from the patch I just sent, I only needed to
> replace them with set_page_writebacks().

Ah, OK.   Things are improved.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 16:38 -0800, Andrew Morton wrote:

> I don't know if it'll be a problem.  One factor is that when the VFS is
> doing an fsync() or whatever, it will fail to notice these left-over pages
> are "dirty", so it won't launch writepage() against them.

That doesn't matter. They are being tracked by the NFS client. We don't
want anyone to call writepage() against them again because that will
cause them to be written out twice.

> But if they are marked PageWriteback(), sync will notice them on the second
> pass and will wait upon them, which apparently could mean a stall until
> pdflush kicks off the I/O?
> 
> If they're not marked PageDirty() or PageWriteback(), the VFS will miss
> them altogether during the sync.  But perhaps NFS's own page tracking will
> flush them and wait upon the result?

Yes. There is no chance of data loss (unless someone physically pulls
the plug on the client - there's no protecting against that).

Note that writepages() will normally end up calling nfs_flush_inode().

It will only fail to do so if

  - generic_writepages() returns an error
or
  - there is write congestion, and wbc->nonblocking is set.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> OK, please back out the patch that I sent you, and try this one instead.

THAT'S IT!

Very nice..  30MB+/sec sustained for several minutes..
only 25% system CPU and the new profile is:

samples  %        symbol name
1047754  32.9054  pci_conf1_write
193876    6.0888  _spin_lock_irqsave
152897    4.8018  skb_copy_bits
74745     2.3474  _spin_lock
73273     2.3012  __copy_from_user_ll
69273     2.1756  __lookup_tag
65084     2.0440  _spin_unlock_irqrestore
43803     1.3757  sub_preempt_count
32047     1.0065  tcp_v4_rcv
30895     0.9703  schedule
26161     0.8216  kfree

Thank you!

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> OK, please back out the patch that I sent you, and try this one instead.

With jumbo frames, the profile is even happier:
(throughput is a little higher and CPU usage is a little lower too)

samples  %        symbol name
74463    12.1129  skb_copy_bits
30351     4.9372  __lookup_tag
24520     3.9887  _spin_lock
20353     3.3108  _spin_lock_irqsave
19306     3.1405  __copy_from_user_ll
15393     2.5040  __copy_user_zeroing_intel
10014     1.6290  isolate_lru_pages
9002      1.4644  sub_preempt_count
7997      1.3009  debug_smp_processor_id
7691      1.2511  schedule
6999      1.1385  shrink_list
6699      1.0897  tcp_sendmsg
6669      1.0848  radix_tree_delete
6532      1.0626  _write_lock_irqsave
6413      1.0432  __mod_page_state
6170      1.0037  acpi_safe_halt

Again... this is excellent.
So will this make 2.6.16? or can this be called a bug fix for 2.6.15?

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

I seem to have hit the same issue as Fabio Brugnara
http://www.ussg.iu.edu/hypermail/linux/kernel/0505.0/1061.html
but although he said the problem went away for him in 2.6.11, I saw the problem
in 2.6.8, and still see it in 2.6.14.

When using a sliding mmap window over a file on nfs, the system time increases
with time, and the throughput drops.
Using the same program on a local file I get about 30MB/s with very little
system time... just a bunch of iowait.

I am using a gigabit connection to a netapp, and initially get 30+MB/s, but
after about 30 seconds, this decays to only about 8MB/s with the system time
close to 80%.

I did a quick Oprofile run, and here is the top of the list:

1739810  16.8926  vmlinux-2.6.14           vmlinux-2.6.14          
find_get_pages_tag
1305104  12.6718  vmlinux-2.6.14           vmlinux-2.6.14          
mpage_writepages
1041147  10.1089  vmlinux-2.6.14           vmlinux-2.6.14           unlock_page
998692    9.6967  vmlinux-2.6.14           vmlinux-2.6.14          
clear_page_dirty_for_io
966458    9.3838  vmlinux-2.6.14           vmlinux-2.6.14          
release_pages
674434    6.5484  vmlinux-2.6.14           vmlinux-2.6.14          
pci_conf1_write
486345    4.7221  vmlinux-2.6.14           vmlinux-2.6.14          
__lookup_tag
399627    3.8801  vmlinux-2.6.14           vmlinux-2.6.14          
page_waitqueue
134250    1.3035  vmlinux-2.6.14           vmlinux-2.6.14          
_spin_lock_irqsave
130594    1.2680  vmlinux-2.6.14           vmlinux-2.6.14          
_read_lock_irqsave


I'm attaching the config.
Other details:
  Dual Xeon 2.66 w/ 2GB RAM
  Debian sarge + 2.6.14 kernel + glibc 2.3.5
  nvidia drivers
  broadcom gigabit driver (8.2.18)

thanks,
-Kenny


		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

[-- Attachment #2: 1636389412-config-2.6.14 --]
[-- Type: application/octet-stream, Size: 43684 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.14
# Mon Nov  7 14:00:32 2005
#
CONFIG_X86=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
# CONFIG_CLEAN_COMPILE is not set
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
# CONFIG_CPUSETS is not set
CONFIG_INITRAMFS_SOURCE=""
CONFIG_EMBEDDED=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_SMP=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=m
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_TOSHIBA=m
CONFIG_I8K=m
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_DELL_RBU is not set
CONFIG_DCDBAS=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
# CONFIG_IRQBALANCE is not set
# CONFIG_REGPARM is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_PHYSICAL_START=0x100000
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set
CONFIG_ACPI_VIDEO=y
# CONFIG_ACPI_HOTKEY is not set
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
# CONFIG_ACPI_THERMAL is not set
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
# CONFIG_CPU_FREQ_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_POWERNOW_K8_ACPI=y
# CONFIG_X86_GX_SUSPMOD is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_CENTRINO_ACPI=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
# CONFIG_X86_SPEEDSTEP_ICH is not set
# CONFIG_X86_SPEEDSTEP_SMI is not set
CONFIG_X86_P4_CLOCKMOD=m
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set

#
# shared options
#
# CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
CONFIG_X86_SPEEDSTEP_LIB=m

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCIEPORTBUS=y
# CONFIG_HOTPLUG_PCI_PCIE is not set
CONFIG_PCI_MSI=y
# CONFIG_PCI_LEGACY_PROC is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_HOTPLUG_CPU is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_ACPI=m
# CONFIG_HOTPLUG_PCI_ACPI_IBM is not set
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
# CONFIG_HOTPLUG_PCI_SHPC_POLL_EVENT_MODE is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=m
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
# CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_BIC=y

#
# IP: Virtual Server Configuration
#
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_NETLINK is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CONNTRACK_MARK is not set
# CONFIG_IP_NF_CONNTRACK_EVENTS is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
# CONFIG_IP_NF_NETBIOS_NS is not set
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
# CONFIG_IP_NF_PPTP is not set
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_DCCP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
# CONFIG_IP_NF_MATCH_STRING is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
# CONFIG_IP_NF_TARGET_NFQUEUE is not set
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
# CONFIG_IP_NF_TARGET_TTL is not set
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_ATM=y
# CONFIG_ATM_CLIP is not set
# CONFIG_ATM_LANE is not set
# CONFIG_ATM_BR2684 is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=m
CONFIG_LLC2=m
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_NET_CLS_IND is not set
# CONFIG_CLS_U32_MARK is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set

#
# Connector - unified userspace <-> kernelspace linker
#
# CONFIG_CONNECTOR is not set

#
# Memory Technology Devices (MTD)
#
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
CONFIG_MTD_CONCAT=m
CONFIG_MTD_PARTITIONS=y
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
# CONFIG_MTD_CMDLINE_PARTS is not set

#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_AMDSTD_RETRY=0
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m
# CONFIG_MTD_OBSOLETE_CHIPS is not set

#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
CONFIG_MTD_PHYSMAP_START=0x8000000
CONFIG_MTD_PHYSMAP_LEN=0x4000000
CONFIG_MTD_PHYSMAP_BANKWIDTH=2
CONFIG_MTD_PNC2000=m
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
# CONFIG_MTD_TS5500 is not set
CONFIG_MTD_SBC_GXX=m
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
CONFIG_MTD_NETtel=m
CONFIG_MTD_DILNETPC=m
CONFIG_MTD_DILNETPC_BOOTSIZE=0x80000
# CONFIG_MTD_L440GX is not set
CONFIG_MTD_PCI=m
# CONFIG_MTD_PLATRAM is not set

#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
# CONFIG_MTD_PMC551_BUGFIX is not set
# CONFIG_MTD_PMC551_DEBUG is not set
CONFIG_MTD_SLRAM=m
CONFIG_MTD_PHRAM=m
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLKMTD=m
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCECC=m
# CONFIG_MTD_DOCPROBE_ADVANCED is not set
CONFIG_MTD_DOCPROBE_ADDRESS=0

#
# NAND Flash Device Drivers
#
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set
# CONFIG_MTD_NAND_NANDSIM is not set

#
# Parallel port support
#
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
# CONFIG_PARPORT_SERIAL is not set
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_NOT_PC=y
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_PNPACPI=y

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_BLK_DEV_INITRD=y
CONFIG_LBD=y
# CONFIG_CDROM_PKTCDVD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDETAPE=m
CONFIG_BLK_DEV_IDEFLOPPY=m
CONFIG_BLK_DEV_IDESCSI=m
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=m
CONFIG_BLK_DEV_OPTI621=m
CONFIG_BLK_DEV_RZ1000=m
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
CONFIG_BLK_DEV_AEC62XX=m
CONFIG_BLK_DEV_ALI15X3=y
# CONFIG_WDC_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_ATIIXP=m
CONFIG_BLK_DEV_CMD64X=m
# CONFIG_BLK_DEV_TRIFLEX is not set
CONFIG_BLK_DEV_CY82C693=m
CONFIG_BLK_DEV_CS5520=m
CONFIG_BLK_DEV_CS5530=m
CONFIG_BLK_DEV_HPT34X=m
# CONFIG_HPT34X_AUTODMA is not set
CONFIG_BLK_DEV_HPT366=m
CONFIG_BLK_DEV_SC1200=m
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_IT821X is not set
CONFIG_BLK_DEV_NS87415=m
CONFIG_BLK_DEV_PDC202XX_OLD=m
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_PDC202XX_NEW=m
CONFIG_PDC202XX_FORCE=y
CONFIG_BLK_DEV_SVWKS=m
CONFIG_BLK_DEV_SIIMAGE=y
# CONFIG_BLK_DEV_SIS5513 is not set
CONFIG_BLK_DEV_SLC90E66=m
CONFIG_BLK_DEV_TRM290=m
CONFIG_BLK_DEV_VIA82CXXX=m
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
CONFIG_SCSI_3W_9XXX=y
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=y
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=8
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
CONFIG_SCSI_AIC79XX=y
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
CONFIG_AIC79XX_ENABLE_RD_STRM=y
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_MEGARAID_NEWGEN is not set
CONFIG_MEGARAID_LEGACY=y
CONFIG_MEGARAID_SAS=y
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_CPQFCTS is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA24XX is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
CONFIG_SCSI_DEBUG=m

#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
CONFIG_I2O=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_CONFIG=y
# CONFIG_I2O_CONFIG_OLD_IOCTL is not set
CONFIG_I2O_BUS=y
CONFIG_I2O_BLOCK=y
CONFIG_I2O_SCSI=y
CONFIG_I2O_PROC=y

#
# Network device support
#
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_EQUALIZER=m
# CONFIG_TUN is not set
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#
# CONFIG_PHYLIB is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set

#
# Tulip family network device support
#
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
# CONFIG_ULI526X is not set
# CONFIG_HP100 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=m
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=m
CONFIG_E1000_NAPI=y
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=m
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set

#
# ATM drivers
#
# CONFIG_ATM_TCP is not set
# CONFIG_ATM_LANAI is not set
# CONFIG_ATM_ENI is not set
# CONFIG_ATM_FIRESTREAM is not set
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
# CONFIG_ATM_IDT77252 is not set
# CONFIG_ATM_AMBASSADOR is not set
# CONFIG_ATM_HORIZON is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E_MAYBE is not set
# CONFIG_ATM_HE is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=m
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_EVBUG=m

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_LKKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_ACPI is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
# CONFIG_TIPAR is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
# CONFIG_IPMI_POWEROFF is not set

#
# Watchdog Cards
#
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_SC520_WDT=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
# CONFIG_IBMASR is not set
CONFIG_WAFER_WDT=m
# CONFIG_I6300ESB_WDT is not set
CONFIG_I8XX_TCO=m
CONFIG_SC1200_WDT=m
CONFIG_60XX_WDT=m
# CONFIG_SBC8360_WDT is not set
CONFIG_CPU5_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
# CONFIG_W83977F_WDT is not set
CONFIG_MACHZ_WDT=m

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
CONFIG_WDT_501_PCI=y

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_GEN_RTC=m
CONFIG_GEN_RTC_X=y
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_SONYPI=m

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=m
CONFIG_AGP_ALI=m
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=m
CONFIG_AGP_AMD64=m
CONFIG_AGP_INTEL=m
CONFIG_AGP_NVIDIA=m
CONFIG_AGP_SIS=m
CONFIG_AGP_SWORKS=m
CONFIG_AGP_VIA=m
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=m
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m

#
# TPM devices
#
# CONFIG_TCG_TPM is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
# CONFIG_I2C_AMD756_S4882 is not set
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_STUB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_ISA is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
# CONFIG_SENSORS_PCA9539 is not set
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
# CONFIG_SENSORS_ADM1026 is not set
CONFIG_SENSORS_ADM1031=m
# CONFIG_SENSORS_ADM9240 is not set
CONFIG_SENSORS_ASB100=m
# CONFIG_SENSORS_ATXP1 is not set
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
# CONFIG_SENSORS_FSCPOS is not set
CONFIG_SENSORS_GL518SM=m
# CONFIG_SENSORS_GL520SM is not set
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_LM63 is not set
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
# CONFIG_SENSORS_LM87 is not set
CONFIG_SENSORS_LM90=m
# CONFIG_SENSORS_LM92 is not set
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
# CONFIG_SENSORS_W83792D is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia Capabilities Port drivers
#

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m

#
# Video For Linux
#

#
# Video Adapters
#
# CONFIG_VIDEO_BT848 is not set
# CONFIG_VIDEO_BWQCAM is not set
# CONFIG_VIDEO_CQCAM is not set
# CONFIG_VIDEO_W9966 is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_TUNER_3036 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_ZR36120 is not set
# CONFIG_VIDEO_MEYE is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set

#
# Radio Adapters
#
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_MAESTRO is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
CONFIG_FB=y
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
CONFIG_FB_SOFT_CURSOR=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
# CONFIG_FB_VESA is not set
CONFIG_VIDEO_SELECT=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON_OLD is not set
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
# CONFIG_FB_ATY is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

#
# Logo configuration
#
# CONFIG_LOGO is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_GENERIC_DRIVER=y

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_AC97_BUS=m

#
# PCI devices
#
CONFIG_SND_ALI5451=m
CONFIG_SND_ATIIXP=m
# CONFIG_SND_ATIIXP_MODEM is not set
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS4281=m
CONFIG_SND_EMU10K1=m
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_CA0106 is not set
CONFIG_SND_KORG1212=m
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_HDSP=m
# CONFIG_SND_HDSPM is not set
CONFIG_SND_TRIDENT=m
CONFIG_SND_YMFPCI=m
# CONFIG_SND_AD1889 is not set
CONFIG_SND_ALS4000=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_VIA82XX=m
# CONFIG_SND_VIA82XX_MODEM is not set
CONFIG_SND_VX222=m
# CONFIG_SND_HDA_INTEL is not set

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_BANDWIDTH=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_ISP116X_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_OBSOLETE_OSS_USB_DRIVER is not set
# CONFIG_USB_BLUETOOTH_TTY is not set
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
# CONFIG_USB_STORAGE_USBAT is not set
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
# CONFIG_USB_STORAGE_ONETOUCH is not set

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y

#
# USB HID Boot Protocol drivers
#
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
CONFIG_USB_AIPTEK=m
CONFIG_USB_WACOM=m
# CONFIG_USB_ACECAD is not set
CONFIG_USB_KBTAB=m
CONFIG_USB_POWERMATE=m
CONFIG_USB_MTOUCH=m
# CONFIG_USB_ITMTOUCH is not set
CONFIG_USB_EGALAX=m
# CONFIG_USB_YEALINK is not set
CONFIG_USB_XPAD=m
CONFIG_USB_ATI_REMOTE=m
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set
CONFIG_USB_VICAM=m
CONFIG_USB_DSBR=m
CONFIG_USB_IBMCAM=m
CONFIG_USB_KONICAWC=m
CONFIG_USB_OV511=m
CONFIG_USB_SE401=m
CONFIG_USB_SN9C102=m
CONFIG_USB_STV680=m
CONFIG_USB_PWC=m

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_GL620A is not set
CONFIG_USB_NET_NET1080=m
# CONFIG_USB_NET_PLUSB is not set
# CONFIG_USB_NET_RNDIS_HOST is not set
# CONFIG_USB_NET_CDC_SUBSET is not set
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_MON=y

#
# USB port drivers
#
CONFIG_USB_USS720=m

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRPRIME is not set
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_WHITEHEAT is not set
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
# CONFIG_USB_SERIAL_KEYSPAN_MPR is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28X is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28XA is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28XB is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA18X is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19W is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19QW is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19QI is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA49W is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA49WLC is not set
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_HP4X is not set
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
# CONFIG_USB_SERIAL_TI is not set
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYTHERM=m
# CONFIG_USB_PHIDGETKIT is not set
CONFIG_USB_PHIDGETSERVO=m
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
CONFIG_USB_TEST=m

#
# USB DSL modem support
#
# CONFIG_USB_ATM is not set

#
# USB Gadget Support
#
CONFIG_USB_GADGET=m
# CONFIG_USB_GADGET_DEBUG_FILES is not set
CONFIG_USB_GADGET_SELECTED=y
CONFIG_USB_GADGET_NET2280=y
CONFIG_USB_NET2280=m
# CONFIG_USB_GADGET_PXA2XX is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_DUMMY_HCD is not set
CONFIG_USB_GADGET_DUALSPEED=y
CONFIG_USB_ZERO=m
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y
CONFIG_USB_GADGETFS=m
CONFIG_USB_FILE_STORAGE=m
# CONFIG_USB_FILE_STORAGE_TEST is not set
CONFIG_USB_G_SERIAL=m

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# SN Devices
#

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=y
CONFIG_XFS_EXPORT=y
CONFIG_XFS_QUOTA=m
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_MINIX_FS=m
CONFIG_ROMFS_FS=m
CONFIG_INOTIFY=y
CONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_RPCSEC_GSS_KRB5=y
# CONFIG_RPCSEC_GSS_SPKM3 is not set
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="cp437"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m

#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_EARLY_PRINTK is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_TGR192 is not set
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
# CONFIG_CRC16 is not set
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=m
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y