Negative scalability by removal of lock_kernel()? (Was: Strange performance behavior of 2.4.0-test9)

[kumon@flab.fujitsu.co.jp]

Finally, I found:
Removal of lock_kernel in fs/fcntl.c causes the strange performance of
2.4.0-test9.


The removal causes following negative scalability on Apache-1.3.9:
	* 8-way performance dropped to 60% of 4-way performance.
	* Adding lock_kernel() gains 2.4x performance on 8-way.

This suggests some design malfunction exist in the fs-code.

The lock_kernel() is removed in test9, as shown in below, then the
strange behavior appeared.

linux-2.4.0-test8/fs/fcntl.c:
asmlinkage long sys_fcntl(unsigned int fd, unsigned int cmd, unsigned long arg)
{	
	struct file * filp;
	long err = -EBADF;

	filp = fget(fd);
	if (!filp)
		goto out;

-->	lock_kernel();
	err = do_fcntl(fd, cmd, arg, filp);
-->	unlock_kernel();

 	fput(filp);
out:
	return err;
}

Adding the lock_kernel()/unlock_kernel() to test9:fs/fcntl.c,
	The performance is restored,
	The number of task switch is reduced, and
	Positive scalability is observed.

The lock region may be narrowed to around call of posix_lock_file()
in fcntl_setlk() (fs/locks.c).

I usually prefer removal of kernel_lock, but at this time,
the removal severy struck the performance.

Please give me suggestions..


kumon@flab.fujitsu.co.jp writes:
 > kumon@flab.fujitsu.co.jp writes:
 >  > Rik van Riel writes:
 >  >  > On Wed, 25 Oct 2000 kumon@flab.fujitsu.co.jp wrote:
 >  >  > > I found very odd performance behavior of 2.4.0-test9 on a large SMP
 >  >  > > server, and I want some clues to investigate it.
 >  >  > > 
 >  >  > > 1) At the 8 cpu configuration, test9 shows extremely inferior
 >  >  > >    performance.
 >  >  > > 2) on test8, 8-cpu configuration shows about 2/3 performance of 4-cpu.
 >  >  >         ^^^^^ test9 ??
 > 
 > IMHO, the modification of file-system code causes the weird
 > performance.
 > 
 > Most of processes are slept at:
 > 	posix_lock_file()->locks_block_on()->interruptible_sleep_on_locks()
 > 
 > We revert two of test9 files (fs/fcntl.c fs/flock.c), to the previous
 > version, the performance problem disappeared and it becomes to the
 > same level as test8.
 > 
 > To narrow the problem, we measured performance of 3 configuration:
 > 1) test9 with test8 fs/fcntl.c, test8 fs/flock.c
 > 2) test9 with test8 fs/fcntl.c
 > 3) test9 with test8 fs/flock.c
 > 
 > Only 3) shows the problem, so the main problem reside in fcntl.c (not
 > in flock.c).
 > 
 > So it seems:
 > the web-server, apache-1.3.9 in the redhat-6.1, issues lots of fcntl
 > to the file and those fcntls collide each other, and the processes
 > are blocked.
 > 
 > 
 > What has happend to fcntl.c?
 > 
 > --
 > Computer Systems Laboratory, Fujitsu Labs.
 > kumon@flab.fujitsu.co.jp
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/