From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S263574AbUGBMCQ (ORCPT ); Fri, 2 Jul 2004 08:02:16 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S263457AbUGBMCQ (ORCPT ); Fri, 2 Jul 2004 08:02:16 -0400 Received: from www.logi-track.com ([213.239.193.212]:16272 "EHLO mail.logi-track.com") by vger.kernel.org with ESMTP id S263574AbUGBMCH convert rfc822-to-8bit (ORCPT ); Fri, 2 Jul 2004 08:02:07 -0400 Date: Fri, 2 Jul 2004 14:01:44 +0200 From: Markus Schaber To: Helge Hafting , Linux Kernel Mailing List Subject: Re: Block Device Caching Message-Id: <20040702140144.43088dc2@kingfisher.intern.logi-track.com> In-Reply-To: <40E543D7.9030303@hist.no> References: <20040630002014.4970b82d@kingfisher.intern.logi-track.com> <40E1FDEC.6020606@techsource.com> <40E279F4.4090708@hist.no> <20040630123828.7d48c6e6@kingfisher.intern.logi-track.com> <40E543D7.9030303@hist.no> Organization: logi-track ag, =?ISO-8859-15?Q?z=FCrich?= X-Mailer: Sylpheed version 0.9.11claws (GTK+ 1.2.10; i386-pc-linux-gnu) X-Face: Nx5T&>Nj$VrVPv}sC3IL&)TqHHOKCz/|)R$i"*r@w0{*I6w;UNU_hdl1J4NI_m{IMztq=>cmM}1gCLbAF+9\#CGkG8}Y{x%SuQ>1#t:;Z(|\qdd[i]HStki~#w1$TPF}:0w-7"S\Ev|_a$K wrote: > > [Caching problems] > >Any Ideas? > > > Not much. "bear" may have decided to drop cache in one case but not > in the other - that is valid although strange. Take a look at what > block size the device uses. A mounted device uses the fs blocksize, a > partition(or entire disk) may be set to some different blocksize. (I > believe it might be the classical 512 byte) Caching blocks > with a size different from the page size (usually 4k) has extra > overhead and could lead to extra memory pressure and different caching > decisions. > > So I recommend retesting bear with a good blocksize, such as 4k, on > the device. That also gives somewhat better performance in general > than 512-byte blocks. There is a syscall for changing the block size > of a device. > > (Block size may also be changed by the following hack: mount a fs > like ext2 on the device, then umount it. Of course you need to use > mkfs so this can't be done on a device that contains data. > "mount" sets the device block size to the size used by the fs. > After umount, the device is free to be opened directly again, but it > retains the new blocksize.) This sounds reasonable, and so I did some tests wr/t mounting and memory pressure: bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775776k used, 3177848k free, 458060k buffers Swap: 1048568k total, 0k used, 1048568k free, 31812k cached bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.374819 seconds, this is 80.809263 MBytes/Sec second run needed 12.004670 seconds, this is 83.300915 MBytes/Sec third run needed 11.898035 seconds, this is 84.047492 MBytes/Sec bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775648k used, 3177976k free, 457904k buffers Swap: 1048568k total, 0k used, 1048568k free, 31900k cached bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2 bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775720k used, 3177904k free, 457944k buffers Swap: 1048568k total, 0k used, 1048568k free, 31860k cached bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.199237 seconds, this is 81.972340 MBytes/Sec second run needed 12.028633 seconds, this is 83.134966 MBytes/Sec third run needed 12.454502 seconds, this is 80.292251 MBytes/Sec bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 1125432k used, 2828192k free, 808008k buffers Swap: 1048568k total, 0k used, 1048568k free, 31792k cached bear:~/readcachetest# ./readcache /testinmount/test.dump 1000 frist run needed 11.985626 seconds, this is 83.433272 MBytes/Sec second run needed 3.682361 seconds, this is 271.564901 MBytes/Sec third run needed 3.638563 seconds, this is 274.833774 MBytes/Sec bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 2150128k used, 1803496k free, 808736k buffers Swap: 1048568k total, 0k used, 1048568k free, 1055688k cached bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.418232 seconds, this is 80.526761 MBytes/Sec second run needed 12.381597 seconds, this is 80.765026 MBytes/Sec third run needed 12.185159 seconds, this is 82.067046 MBytes/Sec bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 2149232k used, 1804392k free, 807784k buffers Swap: 1048568k total, 0k used, 1048568k free, 1055756k cached bear:~/readcachetest# umount /testinmount/ bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775728k used, 3177896k free, 457872k buffers Swap: 1048568k total, 0k used, 1048568k free, 31728k cached bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.171772 seconds, this is 82.157306 MBytes/Sec second run needed 11.842943 seconds, this is 84.438471 MBytes/Sec third run needed 12.167059 seconds, this is 82.189131 MBytes/Sec bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775656k used, 3177968k free, 457876k buffers Swap: 1048568k total, 0k used, 1048568k free, 31860k cached So files from the mounted fs get cached, but not direct reads from the block device, although the lvm volume is mounted, and enough memory is available. I currently suspect this to be a problem with the cciss controller, see my other mail to the list... > You may also want to test with a smaller count. Reading directly from > pagecache should be noticeably faster than reading from the cache on > the raid controller, although either obviously is way faster than > reading from actual disks. bear:~/readcachetest# ./readcache /dev/daten/testing 100 frist run needed 1.202330 seconds, this is 83.171841 MBytes/Sec second run needed 0.349686 seconds, this is 285.970842 MBytes/Sec third run needed 0.346644 seconds, this is 288.480401 MBytes/Sec bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2 bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 1.205797 seconds, this is 82.932699 MBytes/Sec second run needed 0.367004 seconds, this is 272.476594 MBytes/Sec third run needed 0.362783 seconds, this is 275.646874 MBytes/Sec bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 0.363097 seconds, this is 275.408500 MBytes/Sec second run needed 0.363002 seconds, this is 275.480576 MBytes/Sec third run needed 0.362852 seconds, this is 275.594457 MBytes/Sec bear:~/readcachetest# umount /testinmount/ bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2 bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 1.189227 seconds, this is 84.088235 MBytes/Sec second run needed 0.367456 seconds, this is 272.141426 MBytes/Sec third run needed 0.363139 seconds, this is 275.376646 MBytes/Sec Hmm, I'm somehow irritated now, as 275 MB/Sec seems rather low for reads from the page cache on a 2.8Ghz Xeon machine (even my laptop gets about 400 MB/Sec with this, and Skate even reaches 3486.385664 MBytes/Sec.) No idea... Markus. -- markus schaber | dipl. informatiker logi-track ag | rennweg 14-16 | ch 8001 zürich phone +41-43-888 62 52 | fax +41-43-888 62 53 mailto:schabios@logi-track.com | www.logi-track.com