Re: I/O performance test

[Roger Larsson <roger.larsson@skelleftea.mail.telia.com>]

On torsdagen den 17 oktober 2002 11.58, G.de-With wrote:
> Hello
> 
> Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> network we have used Gigabit ethernet. On our cluster we are running RH
> 7.2, with Linux 2.4.19.
> 
> These are some of the hardware specs:
> 
> - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> - 8 Gb ECC DDR
> - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK

Do you have a 72 GB HD per dual processors? Or only on one?
Or do the data have to pass through the ethernet for your tests?

> 
> The problem we have with our cluster is as follows. When running large
> scientific simulations the computer performance gradually goes down as
> the I/O activity is increasing. At some point the response of the
> computer is so poor that we have to kill the simulation. In a worst case
> when the simulation was running overnight the computer hangs and a reset
> of the computer is necessary. Nevertheless, even when we manage to kill
> the simulation in time the computer remains very slow and a reboot is
> necessary to regain full computer power.
> 
> My first suspicion was that the computer simply started swapping, but
> there is no swap space being used, instead free RAM memory is still
> apparent
> (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> and another 50% is in usage. In addition the cpu usage is very low as
> well.

Could still be that all DMA or normal zone memory is used up.
Use the magic sysrq to get zone info.
 Alt+Sys Rq+M
check in vt10 (Alt+F10 or Ctrl+Alt+F10) or with dmesg

> 
> To investigate the I/O performance I installed an I/O performance
> benchmark program called bonnie++. The first test I performed was a
> single bonnie++ -s 16096 instance.
> 
> # bonnie++ -s 16096
> 
> Unfortunately, as a result of running bonnie++ the computer started to
> slow down till it finally hang. All the symptoms I discover with
> bonnie++ are identical to what I experience when running our scientific
> calculations.

Fortunately!

You may have found a way for others to reproduce your problem :-)

> 
> In order to improve the I/O performance I have add some patches to the
> kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> bounce buffers. Unfortunately none of the patches let to any improvement
> in the I/O performance.
> 
> I don't think the machine should be behaving like this. I certainly
> expect some slowdowns with that much IO, but the computer should still
> be reasonably responsive, particularly because no system or user files
> that need to be accessed are on that channel of the SCSI controller.
> 
> As a sort of a desperate move I did two other test in addition which
> could be of use to the understanding of the problem:
> 
> - I removed 6Gb from the server and run the test bonnie++ -s 16096
> succesfully with 2Gb of RAM.

This points towards a highmem issue...

> - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> of RAM succesfully.

This is why I asked the first question.

My guess: Highmem issue with Gb Ethernet?

Brand and model of the Ethernet boards?


-- 
Roger Larsson
Skellefteå
Sweden