server configuration: hardware: memory: 2-way: .5 Gb 4-way: 1 Gb 8-way: 1 Gb cpus: 2-way: Pentium II, 300 MHz [48]-way: Pentium III, 700 MHz NICs: 100 Mbps ethernet (2) software: distribution: Redhat 7.0 kernel: 2-way: 2.4.0-test10 patched with lockmeter1.4.5-2.4.0 [48]-way: 2.4.0 patched with lockmeter1.4.5-2.4.0, 2.4.0.MQ1-sched.rt database: postgresql-7.0.2-17 client: pgbench (distributed with postgresql) lockstat excerpts: 2way: SPINLOCKS HOLD WAIT UTIL CON MEAN ( MAX ) MEAN ( MAX ) TOTAL NOWAIT SPIN REJECT NAME 4.04% 1.22% 50us( 3344us) 5.2us( 2014us) 36515 36068 447 0 kernel_flag 0.01% 3.47% 46us( 427us) 17us( 2014us) 144 139 5 0 do_coredump+0x24 0.00% 0.00% 960us( 960us) 0us 1 1 0 0 do_exit+0x94 0.00% 4.00% 2.0us( 4.2us) 75us( 1876us) 25 24 1 0 ext2_discard_prealloc+0x24 0.03% 0.70% 11us( 1048us) 1.3us( 682us) 1144 1136 8 0 ext2_get_block+0x50 1.78% 0.79% 455us( 3344us) 0.8us( 759us) 1766 1752 14 0 ext2_sync_file+0x28 0.62% 0.84% 12us( 1289us) 2.5us( 1717us) 23353 23157 196 0 real_lookup+0x68 1.46% 1.29% 186us( 2980us) 5.4us( 1824us) 3553 3507 46 0 schedule+0x490 0.01% 0.00% 456us( 596us) 0us 9 9 0 0 sync_old_buffers+0x20 0.01% 1.83% 9.4us( 84us) 0.7us( 92us) 328 322 6 0 sys_fcntl64+0x44 0.00% 3.87% 8.0us( 329us) 6.7us( 1011us) 155 149 6 0 sys_ioctl+0x48 0.02% 2.79% 1.9us( 805us) 19us( 1986us) 5483 5330 153 0 sys_lseek+0x70 0.00% 0.00% 22us( 22us) 0us 1 1 0 0 sys_sysctl+0x50 0.01% 3.23% 17us( 84us) 0.5us( 25us) 155 150 5 0 tty_read+0xbc 0.02% 2.35% 39us( 110us) 0.2us( 11us) 213 208 5 0 tty_write+0x1dc 0.07% 1.09% 168us( 1442us) 0.7us( 116us) 184 182 2 0 vfs_readdir+0x70 0.00% 0.00% 31us( 31us) 0us 1 1 0 0 vfs_statfs+0x54 24.38% 23.93% 15us( 218us) 4.3us( 111us) 744475 566289 178186 0 runqueue_lock 0.06% 15.97% 4.5us( 26us) 2.6us( 67us) 5592 4699 893 0 __wake_up+0xdc 0.00% 10.27% 0.4us( 1.3us) 1.5us( 60us) 146 131 15 0 deliver_signal+0x58 1.16% 8.59% 1.5us( 27us) 2.3us( 111us) 360313 329373 30940 0 process_timeout+0x14 0.00% 0.00% 0.6us( 0.6us) 0us 1 1 0 0 release+0x28 23.15% 38.78% 28us( 218us) 6.2us( 108us) 376292 230381 145911 0 schedule+0xe0 0.01% 45.34% 3.7us( 24us) 16us( 82us) 686 375 311 0 schedule+0x458 0.00% 0.00% 2.8us( 70us) 0us 89 89 0 0 schedule+0x504 0.01% 8.55% 3.0us( 18us) 1.9us( 68us) 1356 1240 116 0 wake_up_process+0x14 0.11% 4.97% 12us( 1113us) 1.0us( 1540us) 4041 3840 201 0 sem_ids+0x24 0.00% 1.32% 7.1us( 88us) 0.1us( 11us) 303 299 4 0 semctl_main+0x4c 0.06% 3.85% 11us( 281us) 0.5us( 81us) 2392 2300 92 0 sys_semop+0xe8 0.04% 7.80% 15us( 1113us) 2.2us( 1540us) 1346 1241 105 0 sys_semop+0x3c8 2.31% 6.86% 0.9us( 15us) 0.2us( 13us) 1102822 1027206 75616 0 timerlist_lock 1.07% 4.75% 1.3us( 11us) 0.1us( 8.8us) 365451 348102 17349 0 add_timer+0x14 0.00% 1.91% 0.3us( 4.2us) 0.1us( 4.5us) 3935 3860 75 0 del_timer+0x14 0.32% 5.71% 0.4us( 7.2us) 0.2us( 13us) 362967 342246 20721 0 del_timer_sync+0x2c 0.02% 1.47% 1.8us( 9.1us) 0.0us( 6.2us) 3942 3884 58 0 mod_timer+0x18 0.02% 0.09% 2.3us( 15us) 0.0us( 2.8us) 4514 4510 4 0 timer_bh+0xd0 0.89% 10.33% 1.1us( 7.6us) 0.3us( 8.2us) 362013 324604 37409 0 timer_bh+0x26c 4way: SPINLOCKS HOLD WAIT UTIL CON MEAN ( MAX ) MEAN ( MAX ) TOTAL NOWAIT SPIN REJECT NAME 0.18% 33.57% 6.0us( 89us) 3.2us( 114us) 97322 64653 32669 0 sem_ids+0x24 0.01% 15.07% 2.0us( 69us) 0.9us( 44us) 10551 8961 1590 0 semctl_main+0x50 0.00% 0.00% 1.3us( 3.7us) 0us 248 248 0 0 sys_semget+0xd0 0.07% 23.57% 4.1us( 86us) 3.1us( 105us) 54350 41537 12813 0 sys_semop+0xf0 0.10% 56.77% 10us( 89us) 4.2us( 114us) 32173 13907 18266 0 sys_semop+0x35c 0.13% 10.71% 0.4us( 3.6us) 0.2us( 18us) 1147726 1024826 122900 0 timerlist_lock 0.06% 9.85% 0.6us( 3.0us) 0.2us( 14us) 361475 325856 35619 0 add_timer+0x10 0.00% 0.19% 0.1us( 1.2us) 0.0us( 6.2us) 45152 45068 84 0 del_timer+0x14 0.03% 11.15% 0.3us( 2.4us) 0.2us( 18us) 341333 303277 38056 0 del_timer_sync+0x1c 0.01% 0.44% 0.5us( 3.6us) 0.0us( 7.2us) 46186 45981 205 0 mod_timer+0x18 0.01% 0.01% 0.5us( 2.9us) 0.0us( 1.2us) 32429 32425 4 0 timer_bh+0xcc 0.03% 15.24% 0.3us( 2.0us) 0.3us( 10us) 321151 272219 48932 0 timer_bh+0x254 0.00% 7.03% 0.2us( 2.3us) 0.2us( 20us) 6882 6398 484 0 add_wait_queue_exclusive+0x10 0.00% 50.00% 0.1us( 0.1us) 3.2us( 6.4us) 2 1 1 0 inet_wait_for_connect+0x104 0.07% 7.48% 0.8us( 13us) 0.2us( 13us) 294222 272202 22020 0 process_timeout+0x24 0.02% 10.56% 0.5us( 4.6us) 0us 114853 102721 0 12132 reschedule_idle+0x3a4 0.04% 15.82% 1.2us( 5.7us) 0us 101053 85069 0 15984 schedule+0x5a8 0.00% 12.64% 2.3us( 12us) 0.3us( 11us) 2461 2150 311 0 schedule+0xb44 0.00% 16.00% 1.6us( 3.0us) 0.5us( 3.5us) 50 42 8 0 schedule+0xb80 0.00% 10.36% 0.4us( 13us) 1.4us( 20us) 251 225 26 0 tcp_close+0x30 0.00% 5.24% 0.1us( 1.4us) 1.1us( 39us) 248 235 13 0 tcp_setsockopt+0x98 8way: SPINLOCKS HOLD WAIT UTIL CON MEAN ( MAX ) MEAN ( MAX ) TOTAL NOWAIT SPIN REJECT NAME 1.15% 9.78% 7.6us( 363us) 2.1us( 862us) 1560956 1408297 152659 0 io_request_lock 0.00% 45.45% 0.5us( 5.3us) 13us( 250us) 58066 31677 26389 0 __get_request_wait+0x70 0.72% 12.06% 28us( 363us) 2.5us( 696us) 266880 234706 32174 0 __make_request+0xfc 0.01% 21.74% 0.2us( 7.0us) 3.9us( 862us) 241846 189278 52568 0 blk_get_queue+0x14 0.25% 4.19% 8.2us( 87us) 0.8us( 303us) 310337 297332 13005 0 do_aic7xxx_isr+0x20 0.02% 3.39% 2.7us( 35us) 0.3us( 233us) 63155 61015 2140 0 generic_unplug_device+0x10 0.04% 6.39% 2.4us( 34us) 2.2us( 314us) 155168 145259 9909 0 scsi_dispatch_cmd+0x12c 0.03% 6.09% 1.9us( 12us) 1.2us( 285us) 155168 145725 9443 0 scsi_old_done+0x614 0.07% 0.06% 4.9us( 65us) 0.0us( 177us) 155168 155080 88 0 scsi_queue_next_request+0x18 0.02% 4.47% 1.2us( 11us) 0.9us( 349us) 155168 148225 6943 0 scsi_request_fn+0x338 0.28% 36.38% 7.1us( 973us) 4.6us( 987us) 405038 257673 147365 0 sem_ids+0x24 0.01% 19.36% 2.2us( 42us) 2.3us( 149us) 56047 45198 10849 0 semctl_main+0x50 0.00% 0.00% 1.9us( 98us) 0us 992 992 0 0 sys_semget+0xd0 0.12% 23.36% 5.6us( 232us) 3.4us( 987us) 214063 164056 50007 0 sys_semop+0xf0 0.15% 64.59% 12us( 973us) 7.5us( 973us) 133936 47427 86509 0 sys_semop+0x35c 0.54% 14.58% 0.6us( 12us) 0.5us( 78us) 8829923 7542237 1287686 0 timerlist_lock 0.21% 11.40% 0.8us( 12us) 0.4us( 78us) 2856951 2531286 325665 0 add_timer+0x10 0.00% 0.82% 0.1us( 8.1us) 0.0us( 28us) 158853 157543 1310 0 del_timer+0x14 0.12% 15.81% 0.4us( 7.8us) 0.5us( 76us) 2787756 2347125 440631 0 del_timer_sync+0x1c 0.02% 1.19% 0.9us( 9.6us) 0.0us( 28us) 183111 180931 2180 0 mod_timer+0x18 0.02% 0.08% 1.6us( 9.1us) 0.0us( 5.5us) 102552 102466 86 0 timer_bh+0xcc 0.18% 18.89% 0.7us( 8.0us) 0.5us( 42us) 2740700 2222886 517814 0 timer_bh+0x254