NFS: low read/stat performance on small files

NFS: low read/stat performance on small files

[Vivek Trivedi]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

Hi,
we are facing below 2 performance issue on NFS:

1. Read speed is low for small files
==========================

[ Log on NFS Client]

$ echo 3 > /proc/sys/vm/drop_caches
$ dd if=200KBfile.txt of=/dev/null
400+0 records in
400+0 records out
204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s


Read speed for 200KB file is 7.2 MB


[ Log on NFS Client]

$ echo 3 > /proc/sys/vm/drop_caches
$ dd if=100MBfile.txt of=/dev/null
204800+0 records in
204800+0 records out
104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s

Read speed for 100MB file is 10.7 MB

As you see read speed for 200KB file is only 7.2MB/sec while it is
10.7 MB/sec when we read 100MB file.
Why there is so much difference in read performance ?
Is there any way to achieve high read speed for small files ?


2. Read/stat for a directory tree is slow on NFS than local
==========================================

we have lot of *.jpg files in a directory. If we try to "stat" and
"read" from this directory,
 performannce is very slow on NFS Client compared to Local(NFS server)
"stat" and "read"


[ Log on Local (NFS Server) ]

$ echo 3 > /proc/sys/vm/drop_caches
$ ./stat_read_files_test ./lot_of_jpg_files/
 Time Taken : 9288 msec


[ Log on NFS Client]

$ echo 3 > /proc/sys/vm/drop_caches
$ ./stat_read_files_test ./lot_of_jpg_files/
 Time Taken : 19966 msec

As you see, on NFS client time taken is almost *double* than that of
local(NFS server)
We are using UDP with rsize,wsize=32k on 100MB ethernet link.
I am attaching read/stat testcase.

Is there any way to improve this performance ?


Thanks,
Vivek

[-- Attachment #2: traversepath.c --]
[-- Type: text/x-csrc, Size: 2188 bytes --]

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/time.h>

#define BUFFERSIZE 4096
char READBUFFER[BUFFERSIZE];

#include <dirent.h>
int TraversePath(char *path)
{
	struct dirent *d = NULL;
	DIR *dir = NULL; /* pointer to directory head*/
	char buf[255]={0}; /* buffer to store the complete file/dir name*/
	struct stat statbuf; /* to obtain the statistics of file/dir */
	int retval =0; /* to hold the return value*/
	int fd = 0;

	memset(&statbuf,0,sizeof(struct stat)); 

	retval = stat(path,&statbuf);

	/* if the stat returned success and path provided is of valid directory*/
	if(S_ISDIR(statbuf.st_mode) && (retval==0))
	{
		dir = opendir(path); /* open the directory*/	
		/* reads entry one by one*/
		while((d = readdir(dir)) != NULL)
		{
			if((strcmp(d->d_name,".")!=0) && (strcmp(d->d_name,"..")!=0))
			{
				sprintf(buf,"%s/%s",path,d->d_name);
				retval = stat(buf,&statbuf);
				if(retval == 0)
				{
					if(!S_ISDIR(statbuf.st_mode))
					{		
						/* This is file - read from this, Since read ahead 
						 * will itself bring 128KB, so we can just read 4KB
						 * to start with */
						fd = open(buf,O_RDONLY,(mode_t)777);
						if(fd) {
							read(fd, READBUFFER, BUFFERSIZE);
							close(fd);	
						}
					}
					else
					{
						/*
						   This is a directory, recursive search in it
						   once all files are read
						 */
						TraversePath(buf);
					}
				}
				else
				{
					perror("stat failed\n");
				}
			}
		}
	}
	else
	{
		perror("Failed");
	}	
	return retval;
}

int main(int argc, char **argv)	
{
	struct timeval rv;
	struct timeval rv1;

	int stat_time = 0;

	if(argc < 2) {
		printf("./TraversePath <path> \n");
		return 0;
	}	

	//Traverse the complete path inside timing unit
	gettimeofday(&rv, 0);
	TraversePath(argv[1]);
	gettimeofday(&rv1, 0);

	stat_time = (rv1.tv_sec * 1000 + rv1.tv_usec / 1000) - (rv.tv_sec * 1000 + rv.tv_usec / 1000);

	printf(" Traversed Path : %s \n", argv[1]);
	printf(" Time Taken : %d msec \n",stat_time);

	return 0;
}

Re: NFS: low read/stat performance on small files

[Jim Rees]

Vivek Trivedi wrote:

  204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
  Read speed for 200KB file is 7.2 MB

  104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
  Read speed for 100MB file is 10.7 MB
  
  As you see read speed for 200KB file is only 7.2MB/sec while it is
  10.7 MB/sec when we read 100MB file.
  Why there is so much difference in read performance ?
  Is there any way to achieve high read speed for small files ?

That seems excellent to me.  204800 bytes at 11213252 per sec would be 18.2
msec, so your per-file overhead is around 9 msec.  The disk latency alone
would normally be more than that.

Re: NFS: low read/stat performance on small files

[Myklebust, Trond]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1792 bytes --]

On Fri, 2012-03-23 at 07:49 -0400, Jim Rees wrote:
> Vivek Trivedi wrote:
> 
>   204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
>   Read speed for 200KB file is 7.2 MB
> 
>   104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
>   Read speed for 100MB file is 10.7 MB
>   
>   As you see read speed for 200KB file is only 7.2MB/sec while it is
>   10.7 MB/sec when we read 100MB file.
>   Why there is so much difference in read performance ?
>   Is there any way to achieve high read speed for small files ?
> 
> That seems excellent to me.  204800 bytes at 11213252 per sec would be 18.2
> msec, so your per-file overhead is around 9 msec.  The disk latency alone
> would normally be more than that.

...and the reason why the performance is worse for the 200K file
compared to the 100M one is easily explained.

When opening the file for reading, the client has a number of
synchronous RPC calls to make: it needs to look up the file, check
access permissions and possibly revalidate its cache. All these tasks
have to be done in series (you cannot do them in parallel), and so the
latency of each task is limited by the round-trip time to the server.

On the other hand, once you get to doing READs, the client can send a
bunch of readahead requests in parallel, thus ensuring that the server
can use all the bandwidth available to the TCP connection.

So your result is basically showing that for small files, the proportion
of (readahead) tasks that can be done in parallel is smaller. This is as
expected.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: NFS: low read/stat performance on small files

[Namjae Jeon]

2012/3/23 Myklebust, Trond <Trond.Myklebust@netapp.com>:
> On Fri, 2012-03-23 at 07:49 -0400, Jim Rees wrote:
>> Vivek Trivedi wrote:
>>
>>   204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
>>   Read speed for 200KB file is 7.2 MB
>>
>>   104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
>>   Read speed for 100MB file is 10.7 MB
>>
>>   As you see read speed for 200KB file is only 7.2MB/sec while it is
>>   10.7 MB/sec when we read 100MB file.
>>   Why there is so much difference in read performance ?
>>   Is there any way to achieve high read speed for small files ?
>>
>> That seems excellent to me.  204800 bytes at 11213252 per sec would be 18.2
>> msec, so your per-file overhead is around 9 msec.  The disk latency alone
>> would normally be more than that.
>
> ...and the reason why the performance is worse for the 200K file
> compared to the 100M one is easily explained.
>
> When opening the file for reading, the client has a number of
> synchronous RPC calls to make: it needs to look up the file, check
> access permissions and possibly revalidate its cache. All these tasks
> have to be done in series (you cannot do them in parallel), and so the
> latency of each task is limited by the round-trip time to the server.
>
> On the other hand, once you get to doing READs, the client can send a
> bunch of readahead requests in parallel, thus ensuring that the server
> can use all the bandwidth available to the TCP connection.
>
> So your result is basically showing that for small files, the proportion
> of (readahead) tasks that can be done in parallel is smaller. This is as
> expected.
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
>

Dear Trond.
I agree your answer.
Thanks a lot for your specific explaination.