The memory usage of the network block device

The memory usage of the network block device

[chen, xiangping]

Hi, 

I am trying to use the network block device in the Linux kernel.

The nbd works good in light io load, but during intensive io load
testing, it seems that it fails to release the memory fast enough.
Eventually the driver blocks at tcp_sendmsg, and waits for the
memory that seems never come. A simple bonnie test to create
a file about the size of the host memory shows the problem.

Is there any way to free up memory more quickly? or why the
network memory is held without being released?

Thanks,

Xiangping

Re: The memory usage of the network block device

[Steven Whitehouse]

Hi,

I'm on holiday this week, so apologies in advance if my replies are not
as prompt as they would otherwise be....

Which kernel version are you using ? What are your settings in the
/proc/sys/net/ipv4/tcp_*mem files or did you just use the default
values ? Did you run the nbd server on the same machine as the client ?
How much memory do you have in the machine that you are testing with ?

Its a little while since I looked at nbd in detail, so if you can send me
enough info to reproduce your set up, then I'll try and have a go in the
next few days,

Steve.

> 
> Hi, 
> 
> I am trying to use the network block device in the Linux kernel.
> 
> The nbd works good in light io load, but during intensive io load
> testing, it seems that it fails to release the memory fast enough.
> Eventually the driver blocks at tcp_sendmsg, and waits for the
> memory that seems never come. A simple bonnie test to create
> a file about the size of the host memory shows the problem.
> 
> Is there any way to free up memory more quickly? or why the
> network memory is held without being released?
> 
> Thanks,
> 
> Xiangping
> 

Re: The memory usage of the network block device

[Steven Whitehouse]

Hi,

> 
> Hi,
> 
> I changed a setup to make the server response quicker, and the 
> client slower. Looks OK now. So the problem seems like too many
> pending tcp requests stucks the client system. Is there any good
> way to control the flow or pending requests of the system? I 
> tried to set some threshold in do_nbd_requests, only blocks the
> processes in _get_request_wait() which is not I wanted.
> 
> Thanks,
> 
> Xiangping
> 
> 

I did wonder about that whih was why I asked the question about the tcp_*mem
settings. I guess that if that is the case either you can increase the tcp_*mem
settings or make the maximum number of blocks merged into a single request
lower. Personally I'd go for the former solution, all you need to do is select
suitable numbers to set for these files (see 
linux/Documentation/networking/ip-sysctl.txt for their exact meaning). If
this does turn out to be the problem, then you'll need to make the config
changes on both server and client as they can both send maximum sized requests
(128 (MAX_SEGMENTS) sectors plus the size of the header).

To reduce the request size you can alter MAX_SEGMENTS in blkdev.h but this
isn't really a very good solution as it reduces the ability to merge I/O
for local disks.

Steve.

RE: The memory usage of the network block device

[chen, xiangping]

Hi,

I changed a setup to make the server response quicker, and the 
client slower. Looks OK now. So the problem seems like too many
pending tcp requests stucks the client system. Is there any good
way to control the flow or pending requests of the system? I 
tried to set some threshold in do_nbd_requests, only blocks the
processes in _get_request_wait() which is not I wanted.

Thanks,

Xiangping


-----Original Message-----
From: Steven Whitehouse [mailto:steve@gw.chygwyn.com]
Sent: Friday, November 16, 2001 5:16 AM
To: chen, xiangping
Cc: linux-kernel@vger.kernel.org
Subject: Re: The memory usage of the network block device


Hi,

I'm on holiday this week, so apologies in advance if my replies are not
as prompt as they would otherwise be....

Which kernel version are you using ? What are your settings in the
/proc/sys/net/ipv4/tcp_*mem files or did you just use the default
values ? Did you run the nbd server on the same machine as the client ?
How much memory do you have in the machine that you are testing with ?

Its a little while since I looked at nbd in detail, so if you can send me
enough info to reproduce your set up, then I'll try and have a go in the
next few days,

Steve.

> 
> Hi, 
> 
> I am trying to use the network block device in the Linux kernel.
> 
> The nbd works good in light io load, but during intensive io load
> testing, it seems that it fails to release the memory fast enough.
> Eventually the driver blocks at tcp_sendmsg, and waits for the
> memory that seems never come. A simple bonnie test to create
> a file about the size of the host memory shows the problem.
> 
> Is there any way to free up memory more quickly? or why the
> network memory is held without being released?
> 
> Thanks,
> 
> Xiangping
> 

RE: The memory usage of the network block device

[chen, xiangping]

Hi, Steve

Thanks for the quick reply. I have tried a couple of 2.4.x kernel 
version, got the same result. I did not tune the /proc/sys/net/ipv4/tcp*mem,
so it should be the default value. Which value you recommend? The memory
size in the driver machine is 256M, I run into problem when do a bonnie 
test of 200M file size. I run nbd on two separate machines, so it should
not be a localhost deadlock problem.


Thanks,

Xiangping

-----Original Message-----
From: Steven Whitehouse [mailto:steve@gw.chygwyn.com]
Sent: Friday, November 16, 2001 5:16 AM
To: chen, xiangping
Cc: linux-kernel@vger.kernel.org
Subject: Re: The memory usage of the network block device


Hi,

I'm on holiday this week, so apologies in advance if my replies are not
as prompt as they would otherwise be....

Which kernel version are you using ? What are your settings in the
/proc/sys/net/ipv4/tcp_*mem files or did you just use the default
values ? Did you run the nbd server on the same machine as the client ?
How much memory do you have in the machine that you are testing with ?

Its a little while since I looked at nbd in detail, so if you can send me
enough info to reproduce your set up, then I'll try and have a go in the
next few days,

Steve.

> 
> Hi, 
> 
> I am trying to use the network block device in the Linux kernel.
> 
> The nbd works good in light io load, but during intensive io load
> testing, it seems that it fails to release the memory fast enough.
> Eventually the driver blocks at tcp_sendmsg, and waits for the
> memory that seems never come. A simple bonnie test to create
> a file about the size of the host memory shows the problem.
> 
> Is there any way to free up memory more quickly? or why the
> network memory is held without being released?
> 
> Thanks,
> 
> Xiangping
>