Re: using splice/vmsplice to improve file receive performance

["saeed bishara" <saeed.bishara@gmail.com>]

On 12/21/06, saeed bishara <saeed.bishara@gmail.com> wrote:
> Hi,
> I'm trying to use the splice/vmsplice system calls to improve the
> samba server write throughput, but before touching the smbd, I started
> to improve the ttcp tool since it simple and has the same flow. I'm
> expecting to avoid the "copy_from_user" path when using those
> syscalls.
> so far, I couldn't make any improvement, actually the throughput get
> worst. the new receive flow looks like this (code also attached):
> 1. read tcp packet (64 pages) to page aligned buffer.
> 2. vmsplice the buffer to pipe with SPLICE_F_MOVE.
> 3. splice the pipe to the file, also with SPLICE_F_MOVE.
>
> the strace shows that the splice takes a lot of time. also when
> profiling the kernel, I found that the memcpy() called to often !!

I found that when doing free to the buffer after the vmsplice and
befaore the splice syscall, the page is really moved without any
memcpy, this means the flow of my application should be:
- malloc aligned buffer
- fill the buffer with the desired data
- vmsplice
- free the buffer
- call splice.

but I still don't get I improvements, and when profing the kernel I
see _clear_user_page() too often, I guess this function called to
clean the new buffers allocated by the user, for securty and privacy
reasons, but the overhead of this operation is expensive.

is there any way to prevent the kernel from cleaning new allocated buffers?
saeed