mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: mgross <mgross@unix-os.sc.intel.com>
To: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	lse-tech@lists.sourceforge.net
Cc: richard.a.griffiths@intel.com
Subject: ext3 performance bottleneck as the number of spindles gets large
Date: Wed, 19 Jun 2002 17:29:45 -0400	[thread overview]
Message-ID: <200206200022.g5K0MKP27994@unix-os.sc.intel.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

We've been doing some throughput comparisons and benchmarks of block I/O 
throughput for 8KB writes as the number of SCSI addapters and drives per 
adapter is increased.

The Linux platform is a dual processor 1.2GHz PIII, 2Gig or RAM, 2U box.
Similar results have been seen with both 2.4.16 and 2.4.18 base kernel, as 
well as one of those patched up O(1) 2.4.18 kernels out there.

The benchmark is Bonnie++.

What seems to be happening is the throughput for 8Kb sequential Write's with 
300MB files goes down with the number of spindles. We have negative scale WRT 
spindles per SCSI adapter, and very poor scaling per SCSI adapter.

(The other 2 processor + OS platform sees its throughput go up with adapters and 
spindles. )

Running this benchmark with lockmeter ends up pointing a big finger at BKL 
contention in: ext3_commit_write, ext3_dirty_inode, ext3_get_block_handle 
and, ext3_prepare_write (twice!).  Attached is the output from the worst 
case, 4 SCSI adapters with 6 drives per adapter.

Has anyone done any work looking into the I/O scaling of Linux / ext3 per 
spindle or per adapter?  We would like to compare notes.

I've only just started to look at the ext3 code but it seems to me that replacing the 
BKL with a per - ext3 file system lock could remove some of the contention thats 
getting measured.  What data are the BKL protecting in these ext3 functions?  Could a 
lock per FS approach work?

Thoughts? 
Comments?
Ideas?

--mgross



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SPINLOCKS         HOLD            WAIT
  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME

        3.7%  0.7us(  44ms)  7.8us(  44ms)(22.9%)  49644038 96.3%  3.7% 0.00%  *TOTAL*

 26.6% 71.2%   13us(  44ms)  8.0us(8076us)( 5.8%)    632107 28.8% 71.2%    0%    ext3_commit_write+0x38
  4.4% 30.3%  4.3us( 360us)   13us(7511us)( 2.1%)    316124 69.7% 30.3%    0%    ext3_dirty_inode+0x2c
 28.1%  7.9%   14us(1660us)  9.7us(6842us)(0.78%)    632239 92.1%  7.9%    0%    ext3_get_block_handle+0x8c
  1.2% 27.2%  0.6us( 240us)   11us(6604us)( 3.0%)    632107 72.8% 27.2%    0%    ext3_prepare_write+0x34
 0.26% 88.1%  0.1us(  74us)  9.6us(7026us)( 8.6%)    632107 11.9% 88.1%    0%    ext3_prepare_write+0xe0


[-- Attachment #2: lm_4x6_300MBw --]
[-- Type: text/plain, Size: 40755 bytes --]

Lockmeter statistics are now RESET
Lockmeter statistics are now ON
___________________________________________________________________________________________
System: Linux TSRLT2 2.4.18 #2 SMP Mon Jun 17 08:28:25 PDT 2002 i686
Total counts

All (2) CPUs

Start time: Mon Jun 17 11:12:36 2002
End   time: Mon Jun 17 11:13:07 2002
Delta Time: 30.99 sec.
Hash table slots in use:      314.
Global read lock slots in use: 884.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SPINLOCKS         HOLD            WAIT
  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME

        3.7%  0.7us(  44ms)  7.8us(  44ms)(22.9%)  49644038 96.3%  3.7% 0.00%  *TOTAL*

 0.00%    0%  1.6us( 3.4us)    0us                        3  100%    0%    0%  [0xdff2bf90]
 0.00%    0%  3.4us( 3.4us)    0us                        1  100%    0%    0%    complete+0x1c
 0.00%    0%  1.3us( 1.3us)    0us                        1  100%    0%    0%    wait_for_completion+0x18
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    wait_for_completion+0x98

 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%  [0xdff56694]
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    exec_mmap+0x8c

 0.00%    0%  1.2us( 2.0us)    0us                        7  100%    0%    0%  [0xf410c22c]
 0.00%    0%  0.9us( 1.5us)    0us                        5  100%    0%    0%    unmap_fixup+0x8c
 0.00%    0%  1.9us( 2.0us)    0us                        2  100%    0%    0%    unmap_fixup+0x134

 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%  [0xf683bf0c]
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    neigh_destroy+0x108

 0.00%    0%  2.5us( 3.9us)    0us                       15  100%    0%    0%  [0xf6e170d0]
 0.00%    0%  2.5us( 3.9us)    0us                       15  100%    0%    0%    dev_watchdog+0x14

 0.00%    0%  0.4us( 0.7us)    0us                        2  100%    0%    0%  [0xf703f504]
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    skb_recv_datagram+0x90
 0.00%    0%  0.7us( 0.7us)    0us                        1  100%    0%    0%    unix_dgram_sendmsg+0x35c

 0.37% 0.31%  2.5us(  12us)  3.9us(  10us)(0.00%)     45356 99.7% 0.31%    0%  allocator_request_lock
 0.07% 0.49%  1.0us( 7.8us)  3.9us(  10us)(0.00%)     22678 99.5% 0.49%    0%    scsi_free+0x1c
 0.30% 0.14%  4.1us(  12us)  3.7us( 7.0us)(0.00%)     22678 99.9% 0.14%    0%    scsi_malloc+0x48

 0.00%    0%  0.1us( 0.6us)    0us                       36  100%    0%    0%  arbitration_lock
 0.00%    0%  0.1us( 0.6us)    0us                       30  100%    0%    0%    deny_write_access+0xc
 0.00%    0%  0.2us( 0.3us)    0us                        6  100%    0%    0%    get_write_access+0xc

 0.00%    0%  0.7us( 2.8us)    0us                     1011  100%    0%    0%  bdev_lock
 0.00%    0%  0.7us( 2.8us)    0us                     1011  100%    0%    0%    bdget+0x34

 0.00%    0%  4.1us( 5.8us)    0us                       18  100%    0%    0%  call_lock
 0.00%    0%  4.1us( 5.8us)    0us                       18  100%    0%    0%    smp_call_function+0x58

 0.00%    0%  0.2us( 0.9us)    0us                        6  100%    0%    0%  cdev_lock
 0.00%    0%  0.2us( 0.9us)    0us                        6  100%    0%    0%    cdput+0x28

  2.9% 0.59%  0.7us( 6.5us)  1.0us( 3.6us)(0.01%)   1256678 99.4% 0.59%    0%  contig_page_data+0xa8
 0.72%  1.1%  0.4us( 5.8us)  1.0us( 3.4us)(0.01%)    628341 98.9%  1.1%    0%    __free_pages_ok+0xc8
  2.2% 0.06%  1.1us( 6.5us)  0.9us( 3.6us)(0.00%)    628337  100% 0.06%    0%    rmqueue+0x28

 0.14% 0.02%  0.1us(  84us)  4.4us(  47us)(0.00%)    325106  100% 0.02%    0%  dcache_lock
 0.00%    0%  0.1us( 0.6us)    0us                       20  100%    0%    0%    d_alloc+0x128
 0.00%    0%  0.1us( 0.1us)    0us                        2  100%    0%    0%    d_delete+0x10
 0.00%    0%  0.1us( 0.3us)    0us                       23  100%    0%    0%    d_instantiate+0x1c
 0.01% 0.02%  0.4us(  24us)  0.9us( 0.9us)(0.00%)      4375  100% 0.02%    0%    d_lookup+0x5c
 0.00%    0%  0.2us( 1.2us)    0us                       20  100%    0%    0%    d_rehash+0x40
 0.00%    0%  1.2us( 1.2us)    0us                        1  100%    0%    0%    do_readv_writev+0x28c
 0.00% 0.09%  0.1us( 3.1us)  1.2us( 1.2us)(0.00%)      1069  100% 0.09%    0%    dput+0x30
 0.00%    0%  2.0us( 3.4us)    0us                       10  100%    0%    0%    link_path_walk+0x2a8
 0.00%    0%  0.2us( 0.7us)    0us                        4  100%    0%    0%    notify_change+0xec
 0.00%    0%  1.1us( 1.9us)    0us                        4  100%    0%    0%    prune_dcache+0x14
 0.00% 0.87%  0.5us(  59us)  0.9us( 1.5us)(0.00%)      3231 99.1% 0.87%    0%    prune_dcache+0x138
 0.00%    0%  2.3us( 2.3us)    0us                        1  100%    0%    0%    sys_getcwd+0xc8
 0.00% 0.30%  0.2us( 1.4us)  0.6us( 0.6us)(0.00%)       334 99.7% 0.30%    0%    sys_read+0xac
 0.13% 0.01%  0.1us(  84us)  8.0us(  47us)(0.00%)    316012  100% 0.01%    0%    sys_write+0xac

 0.20% 0.10%  1.3us(  41us)  1.7us( 6.1us)(0.00%)     47856 99.9% 0.10%    0%  device_request_lock
 0.01% 0.15%  0.1us( 1.6us)  2.0us( 6.1us)(0.00%)     23928 99.8% 0.15%    0%    __scsi_release_command+0x14
 0.19% 0.05%  2.4us(  41us)  1.0us( 1.7us)(0.00%)     23928  100% 0.05%    0%    scsi_allocate_device+0x30

 0.00%    0%  0.2us( 2.5us)    0us                      642  100%    0%    0%  files_lock
 0.00%    0%  0.1us( 1.1us)    0us                      200  100%    0%    0%    file_move+0x18
 0.00%    0%  0.1us( 0.8us)    0us                      201  100%    0%    0%    fput+0x80
 0.00%    0%  0.3us( 2.5us)    0us                      203  100%    0%    0%    get_empty_filp+0xc
 0.00%    0%  0.7us( 1.4us)    0us                       37  100%    0%    0%    get_empty_filp+0xdc
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    put_filp+0x18

  2.9%  8.9%   32us(1727us)    0us                    27945 91.1%    0%  8.9%  global_bh_lock
  2.9%  8.9%   32us(1727us)    0us                    27945 91.1%    0%  8.9%    bh_action+0x18

 0.05%    0%  5.2us( 7.8us)    0us                     3099  100%    0%    0%  i8253_lock
 0.05%    0%  5.2us( 7.8us)    0us                     3099  100%    0%    0%    timer_interrupt+0x2c

 0.01%    0%  1.2us( 2.5us)    0us                     3099  100%    0%    0%  i8259A_lock
 0.01%    0%  1.2us( 2.5us)    0us                     3099  100%    0%    0%    timer_interrupt+0x90

 0.00%    0%  0.4us( 0.4us)    0us                        1  100%    0%    0%  inet_peer_unused_lock
 0.00%    0%  0.4us( 0.4us)    0us                        1  100%    0%    0%    cleanup_once+0x24

 0.00%    0%  0.1us( 1.2us)    0us                       34  100%    0%    0%  init_mm+0x2c
 0.00%    0%  0.9us( 1.2us)    0us                        2  100%    0%    0%    __vmalloc+0x70
 0.00%    0%  0.1us( 0.3us)    0us                       32  100%    0%    0%    __vmalloc+0x120

 0.00%    0%  0.5us( 295us)    0us                     2768  100%    0%    0%  inode_lock
 0.00%    0%  0.4us( 1.9us)    0us                      309  100%    0%    0%    __mark_inode_dirty+0x48
 0.00%    0%  0.8us( 1.0us)    0us                        4  100%    0%    0%    get_empty_inode+0x24
 0.00%    0%  0.7us( 1.3us)    0us                       12  100%    0%    0%    get_new_inode+0x34
 0.00%    0%  1.3us( 2.0us)    0us                       12  100%    0%    0%    iget4+0x3c
 0.00%    0%  0.5us( 0.5us)    0us                        2  100%    0%    0%    insert_inode_hash+0x44
 0.00%    0%  0.1us( 2.1us)    0us                     2138  100%    0%    0%    iput+0x68
 0.00%    0%  182us( 295us)    0us                        4  100%    0%    0%    prune_icache+0x1c
 0.00%    0%  5.8us( 8.0us)    0us                        6  100%    0%    0%    sync_unlocked_inodes+0x10
 0.00%    0%  1.1us(  42us)    0us                      281  100%    0%    0%    sync_unlocked_inodes+0x10c

  2.0%  2.6%  0.8us( 103us)  2.6us(  29us)(0.08%)    788392 97.4%  2.6%    0%  io_request_lock
 0.00%  6.5%  0.6us( 2.0us)  4.6us(  13us)(0.00%)       262 93.5%  6.5%    0%    __get_request_wait+0x90
  1.1%  2.0%  0.5us(  46us)  3.0us(  29us)(0.06%)    666006 98.0%  2.0%    0%    __make_request+0xc0
 0.19%  4.4%  2.4us(  36us)  2.8us(  22us)(0.00%)     23953 95.6%  4.4%    0%    ahc_linux_isr+0x2ec
 0.03%  9.6%  4.6us(  46us)  1.6us(  15us)(0.00%)      2319 90.4%  9.6%    0%    generic_unplug_device+0x10
 0.31%  8.9%  4.0us( 103us)  1.5us(  27us)(0.01%)     24068 91.1%  8.9%    0%    scsi_dispatch_cmd+0x11c
 0.02%  3.9%  0.3us( 1.9us)  2.3us(  20us)(0.00%)     23928 96.1%  3.9%    0%    scsi_finish_command+0x18
 0.17%  3.0%  2.2us(  36us)  2.2us(  15us)(0.00%)     23928 97.0%  3.0%    0%    scsi_queue_next_request+0x18
 0.16%  7.0%  2.0us(  37us)  1.4us(  11us)(0.00%)     23928 93.0%  7.0%    0%    scsi_request_fn+0x31c

 0.51%    0%  0.1us( 147us)    0us                  1300436  100%    0%    0%  jh_splice_lock
 0.27%    0%  0.1us(  59us)    0us                   665500  100%    0%    0%    __journal_remove_journal_head+0xe8
 0.24%    0%  0.1us( 147us)    0us                   634936  100%    0%    0%    journal_add_journal_head+0xd0

 12.8% 0.49%  0.1us(  15ms)  1.3us(  15ms)(0.33%)  33219940 99.5% 0.49%    0%  journal_datalist_lock
 0.00%    0%  2.3us( 2.3us)    0us                        1  100%    0%    0%    dispose_buffer+0x18
 0.00%  2.7%  1.1us( 7.8us)  0.7us( 1.0us)(0.00%)       828 97.3%  2.7%    0%    do_get_write_access+0x9c
  1.6% 0.25%  0.1us( 163us)  1.0us(  84us)(0.03%)   8220760 99.8% 0.25%    0%    do_get_write_access+0x204
  2.0% 0.40%  0.1us( 208us)  0.9us( 182us)(0.05%)   8855491 99.6% 0.40%    0%    journal_add_journal_head+0x10
 0.62% 0.49%  0.3us( 158us)  0.8us( 6.0us)(0.00%)    634936 99.5% 0.49%    0%    journal_add_journal_head+0x88
 0.01%    0%   25us(1934us)    0us                      179  100%    0%    0%    journal_commit_transaction+0x1bc
  3.0% 0.53% 2460us(  15ms)  1.0us( 1.2us)(0.00%)       377 99.5% 0.53%    0%    journal_commit_transaction+0x258
 0.08%    0%   23us( 369us)    0us                     1111  100%    0%    0%    journal_commit_transaction+0x3c0
 0.01% 0.73%  1.4us(  50us)  1.0us( 1.4us)(0.00%)      1786 99.3% 0.73%    0%    journal_commit_transaction+0xd5c
 0.00%  1.1%  0.2us( 0.8us)  1.1us( 1.3us)(0.00%)       179 98.9%  1.1%    0%    journal_commit_transaction+0xee4
 0.45% 0.44%  0.2us( 111us)  1.0us( 2.8us)(0.00%)    632107 99.6% 0.44%    0%    journal_dirty_data+0x54
  2.7% 0.28%  0.2us( 578us)  0.9us(  50us)(0.02%)   5376063 99.7% 0.28%    0%    journal_dirty_metadata+0x54
 0.01% 0.70%  0.4us(  19us)  1.0us( 2.9us)(0.00%)      5537 99.3% 0.70%    0%    journal_file_buffer+0x18
 0.00%    0%  2.2us(  20us)    0us                      619  100%    0%    0%    journal_get_create_access+0x130
 0.60%  9.4%  0.3us( 106us)  1.8us(  15ms)(0.18%)    632510 90.6%  9.4%    0%    journal_try_to_free_buffers+0x4c
 0.00% 0.36%  0.2us( 2.4us)  0.8us( 1.1us)(0.00%)      1965 99.6% 0.36%    0%    journal_unfile_buffer+0xc
  1.7% 0.28%  0.1us( 320us)  1.0us( 263us)(0.04%)   8855491 99.7% 0.28%    0%    journal_unlock_journal_head+0xc

 0.01%    0%   21us(  41us)    0us                      176  100%    0%    0%  kbd_controller_lock
 0.01%    0%   21us(  41us)    0us                      176  100%    0%    0%    keyboard_interrupt+0x14

 64.2% 46.6%  7.0us(  44ms)   10us(  13ms)(21.4%)   2845886 53.4% 46.6%    0%  kernel_flag
 0.00%    0%  1.3us( 1.3us)    0us                        1  100%    0%    0%    chrdev_open+0x4c
 0.00% 57.1%  0.4us( 0.5us)  8.6us(  20us)(0.00%)         7 42.9% 57.1%    0%    de_put+0x28
 0.00% 20.0%   89us( 142us)  4.2us( 4.2us)(0.00%)         5 80.0% 20.0%    0%    do_exit+0xd8
 26.6% 71.2%   13us(  44ms)  8.0us(8076us)( 5.8%)    632107 28.8% 71.2%    0%    ext3_commit_write+0x38
 0.00%    0%   43us(  43us)    0us                        1  100%    0%    0%    ext3_delete_inode+0x48
  4.4% 30.3%  4.3us( 360us)   13us(7511us)( 2.1%)    316124 69.7% 30.3%    0%    ext3_dirty_inode+0x2c
 0.00%    0%  2.2us( 2.2us)    0us                        1  100%    0%    0%    ext3_force_commit+0x38
 28.1%  7.9%   14us(1660us)  9.7us(6842us)(0.78%)    632239 92.1%  7.9%    0%    ext3_get_block_handle+0x8c
  1.2% 27.2%  0.6us( 240us)   11us(6604us)( 3.0%)    632107 72.8% 27.2%    0%    ext3_prepare_write+0x34
 0.26% 88.1%  0.1us(  74us)  9.6us(7026us)( 8.6%)    632107 11.9% 88.1%    0%    ext3_prepare_write+0xe0
 0.00%    0%  5.1us( 5.1us)    0us                        1  100%    0%    0%    get_chrfops+0x88
 0.00%  100%  0.8us( 0.8us)   33us(  33us)(0.00%)         1    0%  100%    0%    locks_remove_posix+0x3c
 0.00%    0%  137us( 221us)    0us                        2  100%    0%    0%    lookup_hash+0x7c
 0.00% 25.0%   17us(  49us)   27us(  27us)(0.00%)         4 75.0% 25.0%    0%    notify_change+0x50
 0.00% 50.0%   40us( 121us)  5.6us( 9.4us)(0.00%)        16 50.0% 50.0%    0%    real_lookup+0x64
  3.7% 58.4% 1007us(  15ms) 1062us(  13ms)( 1.1%)      1136 41.6% 58.4%    0%    schedule+0x508
 0.00% 83.3%  181us( 284us)   17us(  57us)(0.00%)         6 16.7% 83.3%    0%    sync_old_buffers+0x1c
 0.00%    0%  3.2us( 4.2us)    0us                        3  100%    0%    0%    sys_ioctl+0x4c
 0.00% 50.0%  1.3us( 2.1us)   33us( 105us)(0.00%)         8 50.0% 50.0%    0%    sys_llseek+0x88
 0.00%    0%  0.8us( 0.8us)    0us                        1  100%    0%    0%    sys_lseek+0x70
 0.00% 50.0%  7.5us( 9.1us)   13us(  13us)(0.00%)         2 50.0% 50.0%    0%    sys_sysctl+0x70
 0.00%    0%  115us( 172us)    0us                        2  100%    0%    0%    vfs_create+0x84
 0.00%    0%   13us(  13us)    0us                        1  100%    0%    0%    vfs_link+0xa4
 0.00%    0%   12us(  24us)    0us                        2  100%    0%    0%    vfs_readdir+0x68
 0.00%    0%   15us(  16us)    0us                        2  100%    0%    0%    vfs_unlink+0x108

 0.00%    0%  0.6us( 0.9us)    0us                        5  100%    0%    0%  lastpid_lock
 0.00%    0%  0.6us( 0.9us)    0us                        5  100%    0%    0%    get_pid+0x20

 0.00%    0%  0.5us( 1.3us)    0us                      176  100%    0%    0%  logbuf_lock
 0.00%    0%  0.5us( 1.3us)    0us                      176  100%    0%    0%    release_console_sem+0x1c

  7.5% 0.74%  0.9us(  44ms)   14us(  44ms)(0.46%)   2683880 99.3% 0.74%    0%  lru_list_lock
  1.2%  1.5%  7.1us( 499us)  2.5us( 134us)(0.00%)     50765 98.5%  1.5%    0%    balance_dirty+0x18
 0.06% 16.8%   24us( 121us)   12us(  91us)(0.00%)       792 83.2% 16.8%    0%    bdflush+0x98
 0.54%    0%   28ms(  44ms)    0us                        6  100%    0%    0%    bdflush+0xb8
 0.14% 0.42%  0.1us(  85us)  2.0us(  19us)(0.01%)    632101 99.6% 0.42%    0%    buffer_insert_inode_data_queue+0x10
 0.00%  100%  0.4us( 0.4us)   11us(  11us)(0.00%)         1    0%  100%    0%    fsync_inode_buffers+0x28
 0.00%    0%  0.7us( 0.7us)    0us                        1  100%    0%    0%    fsync_inode_data_buffers+0x28
 0.00%    0%  0.6us( 0.6us)    0us                        1  100%    0%    0%    fsync_inode_data_buffers+0xb4
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    fsync_inode_data_buffers+0x128
 0.00%  3.8%  0.1us( 0.7us)  1.4us( 3.5us)(0.00%)      1760 96.2%  3.8%    0%    inode_has_buffers+0x10
 0.00%  1.1%  0.1us( 0.7us)  1.1us( 1.9us)(0.00%)       877 98.9%  1.1%    0%    invalidate_inode_buffers+0x10
 0.19%    0% 9733us(  17ms)    0us                        6  100%    0%    0%    kupdate+0x98
 0.00%    0%  1.0us( 1.0us)    0us                        1  100%    0%    0%    osync_inode_buffers+0x14
 0.00%    0%  0.8us( 0.8us)    0us                        1  100%    0%    0%    osync_inode_data_buffers+0x14
  1.2% 0.29%  0.3us( 192us)   45us(  44ms)(0.29%)   1363983 99.7% 0.29%    0%    refile_buffer+0xc
 0.00%    0%  0.5us( 0.8us)    0us                        6  100%    0%    0%    sync_old_buffers+0x64
  4.2%  1.9%  2.1us( 384us)  7.7us(  29ms)(0.15%)    633578 98.1%  1.9%    0%    try_to_free_buffers+0x1c

 0.00%    0%  0.5us( 1.9us)    0us                       55  100%    0%    0%  mmlist_lock
 0.00%    0%  0.1us( 0.1us)    0us                        4  100%    0%    0%    copy_mm+0x120
 0.00%    0%  0.5us( 0.5us)    0us                        1  100%    0%    0%    exec_mmap+0x50
 0.00%    0%  0.4us( 0.6us)    0us                        5  100%    0%    0%    mmput+0x28
 0.00%    0%  0.6us( 1.9us)    0us                       45  100%    0%    0%    swap_out+0x50

 0.00%    0%  0.3us( 1.7us)    0us                      223  100%    0%    0%  page_uptodate_lock.0
 0.00%    0%  0.3us( 1.7us)    0us                      223  100%    0%    0%    end_buffer_io_async+0x38

  5.7% 0.86%  0.9us( 278us)  1.6us( 240us)(0.04%)   1901672 99.1% 0.86%    0%  pagecache_lock
 0.00%  2.5%  0.5us( 2.4us)  1.2us( 2.4us)(0.00%)      1028 97.5%  2.5%    0%    __find_get_page+0x18
  2.6% 0.35%  1.3us( 278us)  4.7us( 240us)(0.02%)    632107 99.7% 0.35%    0%    __find_lock_page+0xc
  1.6%  1.1%  0.8us( 196us)  1.0us( 158us)(0.01%)    632348 98.9%  1.1%    0%    add_to_page_cache_unique+0x18
 0.00% 0.29%  0.6us( 3.5us)  1.4us( 1.4us)(0.00%)       349 99.7% 0.29%    0%    do_generic_file_read+0x1a4
 0.00%    0%  1.1us( 1.5us)    0us                        2  100%    0%    0%    do_generic_file_read+0x370
 0.00%    0%  0.1us( 0.9us)    0us                      282  100%    0%    0%    filemap_fdatasync+0x20
 0.00% 0.35%  0.1us( 1.1us)  2.0us( 2.0us)(0.00%)       282 99.6% 0.35%    0%    filemap_fdatawait+0x14
 0.00% 0.69%  0.8us(  54us)  1.0us( 2.0us)(0.00%)      1011 99.3% 0.69%    0%    find_or_create_page+0x38
 0.00% 0.49%  0.7us( 2.4us)  1.0us( 1.4us)(0.00%)      1011 99.5% 0.49%    0%    find_or_create_page+0x78
 0.00% 0.68%  0.1us( 1.2us)  0.8us( 0.8us)(0.00%)       146 99.3% 0.68%    0%    page_cache_read+0x48
 0.00%    0%  0.3us( 0.3us)    0us                        1  100%    0%    0%    remove_inode_page+0x18
 0.00%  4.5%  0.1us( 0.9us)  1.4us( 1.5us)(0.00%)       111 95.5%  4.5%    0%    set_page_dirty+0x24
  1.5%  1.1%  0.7us( 163us)  1.1us(  69us)(0.01%)    632992 98.9%  1.1%    0%    shrink_cache+0x2c0
 0.00%    0%  0.9us( 0.9us)    0us                        1  100%    0%    0%    truncate_inode_pages+0x38
 0.00%    0%  0.2us( 0.2us)    0us                        1  100%    0%    0%    truncate_list_pages+0x158

  6.3% 20.8%  1.5us( 956us)  1.2us( 953us)(0.54%)   1310161 79.2% 20.8%    0%  pagemap_lru_lock
 0.00%  2.3%  0.7us(  71us)  1.0us( 2.2us)(0.00%)      1688 97.7%  2.3%    0%    activate_page+0xc
 0.69%  2.6%  0.3us( 166us)  1.5us( 254us)(0.04%)    634142 97.4%  2.6%    0%    lru_cache_add+0x1c
 0.00% 0.70%  0.5us( 135us)  1.0us( 1.2us)(0.00%)       714 99.3% 0.70%    0%    lru_cache_del+0xc
 0.03% 12.7%  0.4us(  30us)  1.7us(  79us)(0.01%)     19785 87.3% 12.7%    0%    refill_inactive+0x10
 0.11% 11.6%  1.8us(  71us)  2.3us( 122us)(0.01%)     19785 88.4% 11.6%    0%    shrink_cache+0x50
 0.00%  1.6%  1.0us(  26us)  1.5us( 1.9us)(0.00%)       385 98.4%  1.6%    0%    shrink_cache+0x194
 0.00%    0%  1.2us(  13us)    0us                       85  100%    0%    0%    shrink_cache+0x21c
  5.5% 39.8%  2.7us( 956us)  1.2us( 953us)(0.48%)    632981 60.2% 39.8%    0%    shrink_cache+0x290
 0.00% 18.0%  1.0us(  14us)  1.1us( 4.0us)(0.00%)       596 82.0% 18.0%    0%    shrink_cache+0x2b0

 0.45%  1.8%  3.8us(  21us)  3.3us(  11us)(0.00%)     36592 98.2%  1.8%    0%  runqueue_lock
 0.09%  4.6%  3.2us(  11us)  3.5us( 8.7us)(0.00%)      9263 95.4%  4.6%    0%    __wake_up+0x5c
 0.00%    0%  0.7us( 0.7us)    0us                        1  100%    0%    0%    complete+0x6c
 0.00%    0%  0.3us( 0.7us)    0us                        3  100%    0%    0%    deliver_signal+0x48
 0.00%    0%  3.0us( 9.8us)    0us                       20  100%    0%    0%    process_timeout+0x14
 0.00%    0%  3.6us( 3.6us)    0us                        1  100%    0%    0%    schedule_tail+0x58
 0.31% 0.35%  5.4us(  21us)  4.7us(  11us)(0.00%)     18215 99.7% 0.35%    0%    schedule+0xa0
 0.00%    0%  1.6us( 4.6us)    0us                       43  100%    0%    0%    schedule+0x264
 0.04%  2.2%  1.5us( 8.3us)  2.1us( 7.1us)(0.00%)      8325 97.8%  2.2%    0%    schedule+0x4c8
 0.00%    0%  0.8us( 8.4us)    0us                      721  100%    0%    0%    wake_up_process+0x14

 0.00%    0%  0.4us( 5.9us)    0us                      496  100%    0%    0%  sb_lock
 0.00%    0%  0.1us( 0.1us)    0us                      168  100%    0%    0%    drop_super+0x24
 0.00%    0%  0.5us( 4.4us)    0us                      174  100%    0%    0%    sync_supers+0x6c
 0.00%    0%  4.0us( 5.9us)    0us                        6  100%    0%    0%    sync_unlocked_inodes+0x18
 0.00%    0%  0.4us( 1.3us)    0us                      148  100%    0%    0%    sync_unlocked_inodes+0x18c

 0.03% 0.06%  0.1us( 1.9us)  1.2us( 2.1us)(0.00%)     68015  100% 0.06%    0%  scsi_bhqueue_lock
 0.01% 0.07%  0.1us( 1.2us)  1.3us( 2.1us)(0.00%)     43947  100% 0.07%    0%    scsi_bottom_half_handler+0x1c
 0.02% 0.04%  0.2us( 1.9us)  0.9us( 1.5us)(0.00%)     24068  100% 0.04%    0%    scsi_done+0x3c

 0.00%    0%  0.4us( 2.1us)    0us                     2366  100%    0%    0%  semaphore_lock
 0.00%    0%  0.4us( 1.9us)    0us                     1483  100%    0%    0%    __down+0x44
 0.00%    0%  0.5us( 2.1us)    0us                      715  100%    0%    0%    __down+0x78
 0.00%    0%  0.3us( 0.3us)    0us                      168  100%    0%    0%    __down_trylock+0x10

 0.00%    0%  0.1us( 6.3us)    0us                      416  100%    0%    0%  swap_info+0x8
 0.00%    0%  0.2us( 6.3us)    0us                      111  100%    0%    0%    get_swap_page+0x74
 0.00%    0%  0.1us( 1.3us)    0us                      123  100%    0%    0%    swap_duplicate+0x54
 0.00%    0%  0.1us( 0.8us)    0us                      182  100%    0%    0%    swap_info_get+0xb4

 0.00%    0%  0.4us( 9.0us)    0us                      294  100%    0%    0%  swaplock
 0.00%    0%  0.4us( 9.0us)    0us                      111  100%    0%    0%    get_swap_page+0x20
 0.00%    0%  1.7us( 1.7us)    0us                        1  100%    0%    0%    si_swapinfo+0x18
 0.00%    0%  0.4us( 2.3us)    0us                      182  100%    0%    0%    swap_info_get+0x88

 0.04% 0.07%  0.2us(  15us)  1.5us( 7.8us)(0.00%)     52142  100% 0.07%    0%  timerlist_lock
 0.01% 0.07%  0.2us( 2.9us)  1.3us( 3.1us)(0.00%)     24279  100% 0.07%    0%    add_timer+0x10
 0.01% 0.07%  0.1us( 1.4us)  1.6us( 7.8us)(0.00%)     24423  100% 0.07%    0%    del_timer+0x14
 0.00%    0%  0.3us( 0.7us)    0us                       27  100%    0%    0%    del_timer_sync+0x1c
 0.00%    0%  0.8us( 1.7us)    0us                      197  100%    0%    0%    mod_timer+0x18
 0.02%    0%  1.5us(  15us)    0us                     3099  100%    0%    0%    timer_bh+0xcc
 0.00%    0%  0.2us( 0.9us)    0us                      117  100%    0%    0%    timer_bh+0x254

 0.00% 0.02%  0.1us( 0.9us)  1.4us( 1.4us)(0.00%)      5566  100% 0.02%    0%  tqueue_lock
 0.00%    0%  0.1us( 0.8us)    0us                     2259  100%    0%    0%    __run_task_queue+0x14
 0.00%    0%  0.1us( 0.8us)    0us                     1067  100%    0%    0%    batch_entropy_store+0x7c
 0.00% 0.05%  0.1us( 0.6us)  1.4us( 1.4us)(0.00%)      2064  100% 0.05%    0%    generic_plug_device+0x34
 0.00%    0%  0.2us( 0.9us)    0us                      176  100%    0%    0%    schedule_task+0x28

 0.00%    0%  0.9us( 1.6us)    0us                        3  100%    0%    0%  uidhash_lock
 0.00%    0%  1.6us( 1.6us)    0us                        1  100%    0%    0%    alloc_uid+0x10
 0.00%    0%  0.1us( 0.1us)    0us                        1  100%    0%    0%    alloc_uid+0x94
 0.00%    0%  1.0us( 1.0us)    0us                        1  100%    0%    0%    free_uid+0x28

  1.5% 0.12%  0.4us( 192us)  2.0us( 147us)(0.00%)   1269889 99.9% 0.12%    0%  unused_list_lock
 0.37% 0.19%  0.2us( 192us)  2.0us( 142us)(0.00%)    635121 99.8% 0.19%    0%    get_unused_buffer_head+0x8
 0.00% 0.22%  0.1us( 1.3us)  1.0us( 2.4us)(0.00%)      1786 99.8% 0.22%    0%    put_unused_buffer_head+0xc
  1.2% 0.05%  0.6us( 157us)  2.0us( 147us)(0.00%)    632982  100% 0.05%    0%    try_to_free_buffers+0x54

 0.00%    0%  0.3us( 1.1us)    0us                        8  100%    0%    0%  __kmem_cache_shrink+0x18
 0.00%    0%  0.1us( 0.3us)    0us                      142  100%    0%    0%  __kmem_cache_shrink+0x48
 0.71% 0.01%  0.1us(  34us)  1.8us( 8.9us)(0.00%)   2045139  100% 0.01%    0%  __wake_up+0x24
 0.00%    0%  0.1us( 0.6us)    0us                     8216  100%    0%    0%  add_wait_queue+0x10
 0.00%  6.8%  0.1us( 1.3us)  2.9us( 8.4us)(0.00%)      1490 93.2%  6.8%    0%  add_wait_queue_exclusive+0x10
 0.50% 0.08%  4.3us( 124us)  2.8us(  10us)(0.00%)     35856  100% 0.08%    0%  ahc_linux_isr+0x24
 0.28% 0.21%  3.6us(  34us)  6.4us(  93us)(0.00%)     24068 99.8% 0.21%    0%  ahc_linux_queue+0x34
 0.00%    0%  0.7us( 1.6us)    0us                       24  100%    0%    0%  change_protection+0x34
 0.00%    0%   11us(  15us)    0us                        9  100%    0%    0%  clear_page_tables+0x1c
 0.00%    0%  0.1us( 0.4us)    0us                       63  100%    0%    0%  copy_mm+0x1e8
 0.00%    0%  2.8us(  48us)    0us                       84  100%    0%    0%  copy_mm+0x230
 0.00%    0%  3.9us(  48us)    0us                       84  100%    0%    0%  copy_page_range+0x100
 0.02%    0%  0.3us( 2.2us)    0us                    27234  100%    0%    0%  do_IRQ+0x40
 0.02%    0%  0.3us( 2.4us)    0us                    27234  100%    0%    0%  do_IRQ+0xc0
 0.00%    0%  0.9us( 5.2us)    0us                      617  100%    0%    0%  do_anonymous_page+0x5c
 0.00%    0%  0.4us( 0.9us)    0us                        5  100%    0%    0%  do_brk+0x1d4
 0.00%    0%  0.1us( 0.1us)    0us                        5  100%    0%    0%  do_exit+0x124
 0.00%    0%  0.1us( 0.3us)    0us                        5  100%    0%    0%  do_exit+0x84
 0.00%    0%  0.1us( 0.1us)    0us                        5  100%    0%    0%  do_exit+0xf4
 0.00%    0%  0.6us( 2.7us)    0us                       86  100%    0%    0%  do_mmap_pgoff+0x40c
 0.00%    0%  0.5us( 3.8us)    0us                      168  100%    0%    0%  do_mmap_pgoff+0x418
 0.00%    0%  0.1us( 0.8us)    0us                       38  100%    0%    0%  do_munmap+0x1b8
 0.00%    0%  0.6us( 5.4us)    0us                      114  100%    0%    0%  do_munmap+0xe0
 0.00%    0%  0.1us( 1.3us)    0us                     1025  100%    0%    0%  do_no_page+0xdc
 0.00%    0%  0.3us( 1.0us)    0us                       14  100%    0%    0%  do_page_fault+0xe0
 0.00%    0%  0.4us( 4.9us)    0us                      114  100%    0%    0%  do_sigaction+0x58
 0.00%    0%  0.6us( 1.5us)    0us                        9  100%    0%    0%  do_sigaction+0xd8
 0.00%    0%  2.6us( 6.2us)    0us                        7  100%    0%    0%  do_signal+0x54
 0.00%    0%  1.1us( 4.2us)    0us                      205  100%    0%    0%  do_wp_page+0x118
 0.00%    0%  0.1us( 0.1us)    0us                        9  100%    0%    0%  exit_mmap+0x18
 0.00%    0%  0.1us( 0.7us)    0us                      143  100%    0%    0%  exit_mmap+0x88
 0.00%    0%  1.7us( 2.8us)    0us                        5  100%    0%    0%  exit_sighand+0x18
 0.13% 0.02%  7.9us(  30us)  1.2us( 1.2us)(0.00%)      5232  100% 0.02%    0%  free_block+0x1c
 0.00%    0%  0.3us(  17us)    0us                     2025  100%    0%    0%  handle_mm_fault+0x34
 0.00%    0%  0.4us( 0.7us)    0us                        5  100%    0%    0%  handle_signal+0xb0
 0.00%    0%  1.7us( 3.3us)    0us                        5  100%    0%    0%  insert_vm_struct+0x60
 0.00%    0%  0.1us( 0.3us)    0us                      182  100%    0%    0%  interruptible_sleep_on+0x28
 0.00%    0%  0.1us( 0.2us)    0us                      182  100%    0%    0%  interruptible_sleep_on+0x54
 0.19% 0.02%  2.9us(  51us)  2.1us( 2.4us)(0.00%)     19844  100% 0.02%    0%  kmem_cache_alloc_batch+0x18
 0.00%    0%  0.1us( 1.3us)    0us                     6765  100%    0%    0%  kmem_cache_grow+0x1d4
 0.01% 0.01%  0.3us( 1.5us)  1.5us( 1.5us)(0.00%)      6765  100% 0.01%    0%  kmem_cache_grow+0x80
 0.00%    0%  0.1us( 0.6us)    0us                      160  100%    0%    0%  kmem_cache_reap+0x25c
 0.00% 0.07%  0.1us( 0.6us)   10us(  15us)(0.00%)      7063  100% 0.07%    0%  kmem_cache_reap+0x2c4
 0.72% 0.00%  1.2us( 327us)  3.0us( 6.9us)(0.00%)    194356  100% 0.00%    0%  kmem_cache_reap+0xa4
 0.00%    0%  1.0us( 2.0us)    0us                       24  100%    0%    0%  mprotect_fixup+0x2a8
 0.00%    0%  0.7us( 1.5us)    0us                       24  100%    0%    0%  mprotect_fixup+0x2b4
 0.00%    0%  5.0us(  41us)    0us                       27  100%    0%    0%  pte_alloc+0x88
 0.00%    0%  0.3us( 0.8us)    0us                        5  100%    0%    0%  put_dirty_page+0x3c
 0.00%    0%  0.1us( 0.4us)    0us                        5  100%    0%    0%  release_task+0x3c
 0.00%  2.9%  0.1us( 1.2us)  3.5us( 9.2us)(0.00%)      9706 97.1%  2.9%    0%  remove_wait_queue+0x10
 0.07%    0%  1.1us( 263us)    0us                    18189  100%    0%    0%  schedule+0x478
 0.00%    0%  1.7us( 7.4us)    0us                        5  100%    0%    0%  schedule_tail+0x20
 0.00%    0%   13us(  31us)    0us                        6  100%    0%    0%  send_sig_info+0x4c
 0.00%    0%  0.1us( 0.4us)    0us                       40  100%    0%    0%  sleep_on+0x28
 0.00%    0%  0.1us( 0.8us)    0us                       40  100%    0%    0%  sleep_on+0x54
 0.01%    0%   57us( 546us)    0us                       45  100%    0%    0%  swap_out+0xc8
 0.00%    0%  0.1us( 0.1us)    0us                        9  100%    0%    0%  sys_rt_sigprocmask+0x18c
 0.00%    0%  0.3us( 1.8us)    0us                       69  100%    0%    0%  sys_rt_sigprocmask+0x98
 0.00%    0%  0.4us( 0.7us)    0us                        5  100%    0%    0%  sys_sigreturn+0x84
 0.00%    0%  1.1us( 1.7us)    0us                        8  100%    0%    0%  unmap_fixup+0xa8
 0.00%    0%  0.6us( 1.0us)    0us                        8  100%    0%    0%  unmap_fixup+0xb8
 0.00%    0%  0.1us( 4.1us)    0us                      249  100%    0%    0%  vma_merge+0x54
 0.01%    0%  5.9us( 398us)    0us                      300  100%    0%    0%  zap_page_range+0x48

- - - - - - - - - - - -  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RWLOCK READS   HOLD    MAX  RDR BUSY PERIOD      WAIT
  UTIL  CON    MEAN   RDRS   MEAN(  MAX )   MEAN(  MAX )( %CPU)     TOTAL NOWAIT SPIN  NAME

       0.68%                               2.1us( 229us)(0.11%)   5063033 99.3% 0.68%  *TOTAL*

 0.00%    0%   3.9us     1  3.9us(  19us)    0us                        3  100%    0%  [0xe6f1b044]
          0%                                 0us                        3  100%    0%    copy_files+0x158

 0.00%    0%   0.2us     1  0.2us( 5.8us)    0us                        1  100%    0%  [0xf417a044]
          0%                                 0us                        1  100%    0%    do_fcntl+0x104

 0.00%    0%   1.2us     1  1.2us( 1.2us)    0us                        1  100%    0%  [0xf703f224]
          0%                                 0us                        1  100%    0%    unix_write_space+0x14

 0.00%    0%   1.3us     1  1.3us( 1.3us)    0us                        1  100%    0%  [0xf703f448]
          0%                                 0us                        1  100%    0%    unix_dgram_sendmsg+0x80

 0.00%    0%  10.1us     1   10us(  10us)    0us                        1  100%    0%  [0xf703f564]
          0%                                 0us                        1  100%    0%    sock_def_readable+0x14

 0.00%    0%   4.5us     1  4.5us( 4.5us)    0us                        1  100%    0%  [0xf703f788]
          0%                                 0us                        1  100%    0%    unix_dgram_sendmsg+0x21c

 0.00%    0%   0.2us     1  0.2us( 3.3us)    0us                        1  100%    0%  [0xf7658d24]
          0%                                 0us                        1  100%    0%    sys_getcwd+0x38

 0.00%    0%   0.7us     1  0.7us( 0.9us)    0us                        2  100%    0%  arp_tbl+0xc4
          0%                                 0us                        2  100%    0%    neigh_lookup+0x40

 0.00%    0%   0.9us     1  0.9us( 1.3us)    0us                        5  100%    0%  binfmt_lock
          0%                                 0us                        5  100%    0%    search_binary_handler+0x38

 0.00%    0%   1.3us     1  1.3us( 1.3us)    0us                        1  100%    0%  chrdevs_lock
          0%                                 0us                        1  100%    0%    get_chrfops+0x28

 0.00%    0%   3.4us     1  3.4us( 4.2us)    0us                        4  100%    0%  fib_hash_lock
          0%                                 0us                        4  100%    0%    fn_hash_lookup+0x10

  2.9% 0.72%   0.2us     2  0.2us( 288us)  2.1us( 229us)(0.11%)   4745142 99.3% 0.72%  hash_table_lock
       0.72%                               2.1us( 229us)(0.11%)   4745142 99.3% 0.72%    get_hash_table+0x60

 0.00%    0%   0.8us     1  0.8us( 1.4us)    0us                        4  100%    0%  inetdev_lock
          0%                                 0us                        2  100%    0%    arp_rcv+0x28
          0%                                 0us                        2  100%    0%    ip_route_input_slow+0x18

 0.01%    0%  28.0us     2   28us(  80us)    0us                       69  100%    0%  tasklist_lock
          0%                                 0us                        6  100%    0%    count_active_tasks+0xc
          0%                                 0us                        5  100%    0%    exit_notify+0x18
          0%                                 0us                       43  100%    0%    schedule+0x218
          0%                                 0us                        1  100%    0%    sys_setsid+0x10
          0%                                 0us                       14  100%    0%    sys_wait4+0x8c

 0.00%    0%   2.1us     1  2.1us( 3.2us)    0us                        3  100%    0%  udp_hash_lock
          0%                                 0us                        3  100%    0%    udp_v4_mcast_deliver+0x10

 0.00%    0%   0.6us     1  0.6us( 1.4us)    0us                       36  100%    0%  xtime_lock
          0%                                 0us                       36  100%    0%    do_gettimeofday+0x14

          0%                                 0us                        5  100%    0%  copy_files+0x100
          0%                                 0us                        5  100%    0%  do_fork+0x35c
          0%                                 0us                       14  100%    0%  do_select+0x24
          0%                                 0us                   316664  100%    0%  fget+0x1c
          0%                                 0us                        5  100%    0%  ip_route_input+0x88
          0%                                 0us                        5  100%    0%  net_rx_action+0x48
          0%                                 0us                        4  100%    0%  path_init+0x114
          0%                                 0us                     1056  100%    0%  path_init+0x30

- - - - - - - - - - -  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RWLOCK WRITES     HOLD           WAIT (ALL)           WAIT (WW) 
  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )( %CPU)   MEAN(  MAX )     TOTAL NOWAIT SPIN(  WW )  NAME

        4.9%  1.4us( 217us)  1.2us( 218us)(0.06%)  0.3us(  34us)    643483 95.1%  3.5%( 1.4%)  *TOTAL*

 0.00%    0%  0.6us( 1.3us)    0us                   0us                 6  100%    0%(   0%)  [0xe6f1b204]
 0.00%    0%  0.5us( 1.3us)    0us                   0us                 3  100%    0%(   0%)    copy_files+0x12c
 0.00%    0%  0.6us( 0.7us)    0us                   0us                 3  100%    0%(   0%)    expand_fd_array+0x88

 0.00%    0%  4.4us(  21us)    0us                   0us                 5  100%    0%(   0%)  [0xf417a044]
 0.00%    0%  0.1us( 0.1us)    0us                   0us                 2  100%    0%(   0%)    do_fcntl+0x140
 0.00%    0%  7.3us(  21us)    0us                   0us                 3  100%    0%(   0%)    sys_dup2+0x2c

 0.00%    0%  0.1us( 0.1us)    0us                   0us                 4  100%    0%(   0%)  [0xf50ce044]
 0.00%    0%  0.1us( 0.1us)    0us                   0us                 2  100%    0%(   0%)    do_pipe+0x174
 0.00%    0%  0.1us( 0.1us)    0us                   0us                 2  100%    0%(   0%)    do_pipe+0x1a4

 0.00%    0%  0.7us( 0.7us)    0us                   0us                 1  100%    0%(   0%)  [0xf7658d24]
 0.00%    0%  0.7us( 0.7us)    0us                   0us                 1  100%    0%(   0%)    sys_chdir+0x9c

 0.00%    0%  0.1us( 0.1us)    0us                   0us                 1  100%    0%(   0%)  [0xf7df24f4]
 0.00%    0%  0.1us( 0.1us)    0us                   0us                 1  100%    0%(   0%)    neigh_destroy+0x8c

 0.00%    0%  104us( 104us)    0us                   0us                 1  100%    0%(   0%)  arp_tbl+0xc4
 0.00%    0%  104us( 104us)    0us                   0us                 1  100%    0%(   0%)    neigh_periodic_timer__thr+0x20

 0.00%    0%  0.5us( 0.6us)    0us                   0us                 2  100%    0%(   0%)  dn_lock
 0.00%    0%  0.5us( 0.6us)    0us                   0us                 2  100%    0%(   0%)    fcntl_dirnotify+0x94

  2.9%  5.0%  1.4us( 217us)  1.2us( 218us)(0.06%)  0.3us(  34us)    634589 95.0%  3.6%( 1.4%)  hash_table_lock
 0.00% 0.59%  1.3us(  30us)   17us(  34us)(0.00%)  8.8us(  34us)      1011 99.4% 0.10%(0.49%)    hash_page_buffers+0x48
  2.9%  5.0%  1.4us( 217us)  1.2us( 218us)(0.06%)  0.2us( 4.9us)    633578 95.0%  3.6%( 1.4%)    try_to_free_buffers+0x28

 0.00%    0%  7.6us(  38us)    0us                   0us                15  100%    0%(   0%)  tasklist_lock
 0.00%    0%  2.2us( 2.7us)    0us                   0us                 5  100%    0%(   0%)    do_fork+0x530
 0.00%    0%   20us(  38us)    0us                   0us                 5  100%    0%(   0%)    exit_notify+0x1b0
 0.00%    0%  0.6us( 1.0us)    0us                   0us                 5  100%    0%(   0%)    release_task+0x7c

 0.00%    0%   20us(  40us)    0us                   0us                 4  100%    0%(   0%)  vmlist_lock
 0.00%    0%  4.4us( 5.2us)    0us                   0us                 2  100%    0%(   0%)    get_vm_area+0x3c
 0.00%    0%   35us(  40us)    0us                   0us                 2  100%    0%(   0%)    vfree+0x58

 0.12%    0%  5.9us(  19us)    0us                   0us              6198  100%    0%(   0%)  xtime_lock
 0.02%    0%  1.7us( 6.7us)    0us                   0us              3099  100%    0%(   0%)    timer_bh+0xc
 0.10%    0%   10us(  19us)    0us                   0us              3099  100%    0%(   0%)    timer_interrupt+0x10

 0.00%    0%  0.9us( 1.2us)    0us                   0us                 5  100%    0%(   0%)  flush_old_exec+0x22c
 0.00%    0%  0.2us( 3.3us)    0us                   0us               813  100%    0%(   0%)  get_unused_fd+0x24
 0.00%    0%  0.1us( 0.1us)    0us                   0us                 5  100%    0%(   0%)  load_elf_binary+0x190
 0.00%    0%  0.4us( 0.5us)    0us                   0us                 4  100%    0%(   0%)  neigh_periodic_timer__thr+0xa8
 0.00%    0%  0.1us( 4.1us)    0us                   0us               820  100%    0%(   0%)  rt_check_expire__thr+0x64
 0.00%    0%  0.2us( 2.6us)    0us                   0us               206  100%    0%(   0%)  sys_close+0x1c
 0.00%    0%  0.1us( 1.7us)    0us                   0us               188  100%    0%(   0%)  sys_open+0x60
 0.00%    0%  0.1us( 0.8us)    0us                   0us               616  100%    0%(   0%)  sys_open+0xa8
_________________________________________________________________________________________________________________________
Number of read locks found=16
Lockmeter statistics are now OFF

             reply	other threads:[~2002-06-20  0:23 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2002-06-19 21:29 mgross [this message]
2002-06-20  0:54 ` Andrew Morton
2002-06-20  4:09   ` [Lse-tech] " Dave Hansen
2002-06-20  6:03     ` Andreas Dilger
2002-06-20  6:53       ` Andrew Morton
2002-06-20  9:54   ` Stephen C. Tweedie
2002-06-20  1:55 ` Andrew Morton
2002-06-20  6:05   ` Jens Axboe
2002-06-20 15:26 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
2002-06-20 20:18 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
2002-06-20 18:08   ` mgross
2002-06-20 21:25     ` Andrew Morton
2002-06-20 21:50 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
2002-06-21  7:58 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
2002-06-21 18:46   ` mgross
2002-06-21 19:26     ` Chris Mason
2002-06-21 19:56     ` Andrew Morton
2002-06-23  4:02 ` Christopher E. Brown
2002-06-23  4:33   ` Andreas Dilger
2002-06-23  6:00     ` Christopher E. Brown

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=200206200022.g5K0MKP27994@unix-os.sc.intel.com \
    --to=mgross@unix-os.sc.intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lse-tech@lists.sourceforge.net \
    --cc=richard.a.griffiths@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®