Re: Questions about Enterprise Storage with Linux

Re: Questions about Enterprise Storage with Linux

[Jesse Pollard]

james rich <james.rich@m.cc.utah.edu>:
> On Wed, 7 Mar 2001, Tom Sightler wrote:
> 
> > 2.  Does linux have any problems with large (500GB+) NFS exports, how about
> > large files over NFS?
> > 
> > 3.  What filesystem would be best for such large volumes?  We currently use
> > reirserfs on our internal system, but they generally have filesystems in the
> > 18-30GB ranges and we're talking about potentially 10-20x that.  Should we
> > look at JFS/XFS or others?
> 
> I think that for filesystems this size you definately want to look at XFS
> of JFS.  Maybe you will decide not to use them - but you should test them.
> 
> I am currently using XFS and it really works.  It currently has some
> issues when used with raid 1, but it is probably the most suited for what
> you want.  Exporting an XFS volume over NFS is no problem.  You can also
> use xfs_growfs to change the size of your XFS partition.  I haven't had
> any instability during all the time I've used XFS.

The biggest difficulty I had with XFS (not on linux as a server) had
more to do with NFS/XFS performance. The SGI clients worked fine while
the Linux clients were about 10-20% slower. This was a year ago so this
may not apply anymore. I haven't seen any Linux NFS benchmarks recently.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Questions about Enterprise Storage with Linux

[james rich]

On Thu, 8 Mar 2001, Jesse Pollard wrote:

> james rich <james.rich@m.cc.utah.edu>:
> > On Wed, 7 Mar 2001, Tom Sightler wrote:
> > 
> > > 2.  Does linux have any problems with large (500GB+) NFS exports, how about
> > > large files over NFS?
> > > 
> > > 3.  What filesystem would be best for such large volumes?  We currently use
> > > reirserfs on our internal system, but they generally have filesystems in the
> > > 18-30GB ranges and we're talking about potentially 10-20x that.  Should we
> > > look at JFS/XFS or others?
> > 
> > I think that for filesystems this size you definately want to look at XFS
> > of JFS.  Maybe you will decide not to use them - but you should test them.
> > 
> > I am currently using XFS and it really works.  It currently has some
> > issues when used with raid 1, but it is probably the most suited for what
> > you want.  Exporting an XFS volume over NFS is no problem.  You can also
> > use xfs_growfs to change the size of your XFS partition.  I haven't had
> > any instability during all the time I've used XFS.
> 
> The biggest difficulty I had with XFS (not on linux as a server) had
> more to do with NFS/XFS performance. The SGI clients worked fine while
> the Linux clients were about 10-20% slower. This was a year ago so this
> may not apply anymore. I haven't seen any Linux NFS benchmarks recently.

Recent changes in CVS appear to have resolved this issue.

James Rich
james.rich@m.cc.utah.edu

Re: Questions about Enterprise Storage with Linux

[Jauder Ho]

Well, I stand corrected. Look forward to trying it out.

--Jauder

On Fri, 9 Mar 2001, Tigran Aivazian wrote:

> Hi,
>
> On Wed, 7 Mar 2001, Jauder Ho wrote:
> > I am not sure what you intend this application for. If it is mission
> > critical in any way shape or form, I would still recommend using something
> > like Veritas (which unfortunately is not ported to Linux yet).
>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> What do you mean not ported? Of course it has been ported and everything
> works nicely and has been so for ages. Ok, it is true that you can't buy
> it just yet but should be able to sometime this year. By "it" I meant the
> volume manager, vxfs journalling filesystem etc...
>
> Regards,
> Tigran
>
>
>
>

Re: Questions about Enterprise Storage with Linux

[Tigran Aivazian]

Hi,

On Wed, 7 Mar 2001, Jauder Ho wrote:
> I am not sure what you intend this application for. If it is mission
> critical in any way shape or form, I would still recommend using something
> like Veritas (which unfortunately is not ported to Linux yet).
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What do you mean not ported? Of course it has been ported and everything
works nicely and has been so for ages. Ok, it is true that you can't buy
it just yet but should be able to sometime this year. By "it" I meant the
volume manager, vxfs journalling filesystem etc...

Regards,
Tigran

Re: Questions about Enterprise Storage with Linux

[Thomas Davis]

Tom Sightler wrote:
> 
> Hi All,
> 
> I'm seeking information in regards to a large Linux implementation we are
> planning.  We have been evaluating many storage options and I've come up
> with some questions that I have been unable to answer as far as Linux
> capabilities in regards to storage.
> 
> We are looking at storage systems that provide approximately 1TB of capacity
> for now and can scale to 10+TB in the future.  We will almost certainly use
> a storage system that provides both fiber channel connectivity as well as
> NFS connectivity.
> 
> The questions that have been asked are as follows (assume 2.4.x kernels):
> 
> 1.  What is the largest block device that linux currently supports?  i.e.
> Can I create a single 1TB volume on my storage device and expect linux to
> see it and be able to format it?
> 

Yes.

[root@pdsfdv10 data]# df .
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/rza3            1046274600 889731608 146074448  86% /export/data

> 2.  Does linux have any problems with large (500GB+) NFS exports, how about
> large files over NFS?
> 

No.
[root@pdsflx002 pdsfdv10]# df .
Filesystem           1k-blocks      Used Available Use% Mounted on
pdsfdv10.nersc.gov:/export/data
                     1046274600 889731608 146074448  86% /auto/pdsfdv10

(same filesystem, via NFS)

files > 2gb need LFS support in ia32 environments.

> 3.  What filesystem would be best for such large volumes?  We currently use
> reirserfs on our internal system, but they generally have filesystems in the
> 18-30GB ranges and we're talking about potentially 10-20x that.  Should we
> look at JFS/XFS or others?
> 

ext2 works fine, you just have to wait about 3 hrs to FSCK a crashed
filesystem; ext3 also works fine.  Get a 2.2.18, apply the ext3 fs
patches, bang, your done.

reiserfs won't work via NFS, without kernel patches.

-- 
------------------------+--------------------------------------------------
Thomas Davis		| ASG Cluster guy
tadavis@lbl.gov		| 
(510) 486-4524		| "80 nodes and chugging Captain!"

Re: Questions about Enterprise Storage with Linux

[james rich]

On Wed, 7 Mar 2001, Tom Sightler wrote:

> 2.  Does linux have any problems with large (500GB+) NFS exports, how about
> large files over NFS?
> 
> 3.  What filesystem would be best for such large volumes?  We currently use
> reirserfs on our internal system, but they generally have filesystems in the
> 18-30GB ranges and we're talking about potentially 10-20x that.  Should we
> look at JFS/XFS or others?

I think that for filesystems this size you definately want to look at XFS
of JFS.  Maybe you will decide not to use them - but you should test them.

I am currently using XFS and it really works.  It currently has some
issues when used with raid 1, but it is probably the most suited for what
you want.  Exporting an XFS volume over NFS is no problem.  You can also
use xfs_growfs to change the size of your XFS partition.  I haven't had
any instability during all the time I've used XFS.

James Rich
james.rich@m.cc.utah.edu

Re: Questions about Enterprise Storage with Linux

[Tim Moore]

Tom Sightler wrote:
> ...
> For example if we purchase a NetApp Filer, or EMC Celerra with 1TB of
> storage, and elect to export that entire amount as a single NFS mount, and
> then use that storage to allow several Linux boxes to share 100GB
> (admittedly temporary) files, will Linux handle that, at least in theory?

Linux/NFS works very well with Filers.  I did a lot of throughput
testing at Netapp circa 2.2.10-12 with Gigabit Ethernet (AceNIC).  Why
would you need to put 1TB on a single mount point?

Filers are also blessed by Oracle and can take care of the volume
management and backup issues.  The principle advantage is avalibility
(balanced against cost of course).  If you do talk to Netapp, ask for
someone that has linux/Filer experience.

rgds,
tim.

--

Re: Questions about Enterprise Storage with Linux

[Jauder Ho]

On Wed, 7 Mar 2001, Tom Sightler wrote:

> The questions that have been asked are as follows (assume 2.4.x kernels):
>
> 1.  What is the largest block device that linux currently supports?  i.e.
> Can I create a single 1TB volume on my storage device and expect linux to
> see it and be able to format it?

http://www.suse.de/~aj/linux_lfs.html should clear up

> 2.  Does linux have any problems with large (500GB+) NFS exports, how about
> large files over NFS?

See above

> 3.  What filesystem would be best for such large volumes?  We currently use
> reirserfs on our internal system, but they generally have filesystems in the
> 18-30GB ranges and we're talking about potentially 10-20x that.  Should we
> look at JFS/XFS or others?

I am not sure what you intend this application for. If it is mission
critical in any way shape or form, I would still recommend using something
like Veritas (which unfortunately is not ported to Linux yet). I have
heard good things about XFS although I have not had the chance to use it
yet.

> 4.  We're seriously considering using LVM for volume management.  Does it
> have size limits per volume or other limitations that we should be aware of?
>
> I'm sure these answers are out there, but I haven't been able to find
> definitive answers (it seems everyone has a different answer to each
> question).  Any assistance in pointing me to the correct information would
> be greatly appreciated.

Have fun and do post your findings back to l-k. I would be interested in
hearing what you end up with.

--Jauder

Re: Questions about Enterprise Storage with Linux

[Tom Sightler]

> >1.  What is the largest block device that linux currently supports?  i.e.
> >Can I create a single 1TB volume on my storage device and expect linux to
> >see it and be able to format it?
>
> Checkout the GFS project for really large filesystems with a high
capability
> of "fail safe" configuration.
>
> The block/file limits are more determined by the size of the hosts.
Alpha/Sparc
> based systems use 64 bit operations, Intel/AMD use 32 bit. It also depends
> on usage of the sign bit in the drivers. Most 32bit systems are limited
> to 1 TB (depending on the driver of course - some allow for 2 TB).

Yes, I should have clarified, we will be using the Intel platform.

A large portion of the fiber channel storage (as much as 300GB in year one)
will be dedicated to an Oracle 8i database, I'm not sure if GFS is truly
suited for this server, although we have considered using LVM (perhaps even
RAWIO on LVM although I see reports of problems with that) to ease some
storage management issues (we've been testing LVM with reiserfs and are very
impressed with the ease of adding space and growing the filesystem live).

> >2.  Does linux have any problems with large (500GB+) NFS exports, how
about
> >large files over NFS?
>
> I can't really say - you might clarify by what you count as large ( just >
2G
> should be fine for any current kernel), not sure if "large" means
25-100GB.

For example if we purchase a NetApp Filer, or EMC Celerra with 1TB of
storage, and elect to export that entire amount as a single NFS mount, and
then use that storage to allow several Linux boxes to share 100GB
(admittedly temporary) files, will Linux handle that, at least in theory?
Basically, I don't know what the file size limits are on NFS (on any system,
not neccesarily just Linux) and was hoping someone could tell me.

And actually, we have looked at GFS as an alternative for this function,
biggest disadvantage is cost of connecting 7 machines via fiber channel (we
already have GigE from the network connectivity), but it does look like an
excellent option which we may very well go with.

And some of these sizes I'm asking about are just seeking the limits, I'll
personally be amazed if we actually approach within half of these sizes any
time soon, but then again, data always manages to grow to fill whatever
storage you have available so I guess you never know, right?

> >3.  What filesystem would be best for such large volumes?  We currently
use
> >reirserfs on our internal system, but they generally have filesystems in
the
> >18-30GB ranges and we're talking about potentially 10-20x that.  Should
we
> >look at JFS/XFS or others?
>
> The GFS project already has tested 2TB in fiber channel array(s) with full
> multi-host connectivity (shared filesystems rather than NFS).
>
> The big advantage with GFS is that redundant servers can be available
> by having two or more NFS servers attached to the same GFS filesystem.

Yep, as I mentioned above, we may very well use GFS here.  It certainly has
some advantages that we really like, but we need to justify the additional
host of HBA's and another fiber channel switch (saying it won't work over
NFS would be enough to justify it).  We also had some concerns of stability
with GFS, not that we know of any issues, but we don't really have an easy
way test it in our current environment, and we haven't ordered the new
equipment yet.

> >4.  We're seriously considering using LVM for volume management.  Does it
> >have size limits per volume or other limitations that we should be aware
of?
>
> GFS may serve better. It is a full shared filesystem with RAID target
disks
> (these are smart controllers) and incudes journaling.

The LVM issue was largely for the managent of the volumes dedicated to the
Oracle database (basically a replacement for Veritas Volume Manager that
Oracle on Solaris folks seem to love).  Once again, we don't think GFS is
really suited for Oracle (other opinions accepted).  I was interrupted a few
too many times when composing the first email, I should have been more
detailed about exactly what we were doing, what we've looked at, etc.

Thanks,
Tom

Re: Questions about Enterprise Storage with Linux

[Jesse Pollard]

On Wed, 07 Mar 2001, Tom Sightler wrote:
>Hi All,
>
>I'm seeking information in regards to a large Linux implementation we are
>planning.  We have been evaluating many storage options and I've come up
>with some questions that I have been unable to answer as far as Linux
>capabilities in regards to storage.
>
>We are looking at storage systems that provide approximately 1TB of capacity
>for now and can scale to 10+TB in the future.  We will almost certainly use
>a storage system that provides both fiber channel connectivity as well as
>NFS connectivity.
>
>The questions that have been asked are as follows (assume 2.4.x kernels):
>
>1.  What is the largest block device that linux currently supports?  i.e.
>Can I create a single 1TB volume on my storage device and expect linux to
>see it and be able to format it?

Checkout the GFS project for really large filesystems with a high capability
of "fail safe" configuration.

The block/file limits are more determined by the size of the hosts. Alpha/Sparc
based systems use 64 bit operations, Intel/AMD use 32 bit. It also depends
on usage of the sign bit in the drivers. Most 32bit systems are limited
to 1 TB (depending on the driver of course - some allow for 2 TB).

>2.  Does linux have any problems with large (500GB+) NFS exports, how about
>large files over NFS?

I can't really say - you might clarify by what you count as large ( just > 2G
should be fine for any current kernel), not sure if "large" means 25-100GB.

>3.  What filesystem would be best for such large volumes?  We currently use
>reirserfs on our internal system, but they generally have filesystems in the
>18-30GB ranges and we're talking about potentially 10-20x that.  Should we
>look at JFS/XFS or others?

The GFS project already has tested 2TB in fiber channel array(s) with full
multi-host connectivity (shared filesystems rather than NFS). See:

http://www.sistina.com/gfs/

for details. It is not currently included in Linux distribution. The
current GFS version is 4.0 and works in kernel 2.2.18 and higher.

The big advantage with GFS is that redundant servers can be available
by having two or more NFS servers attached to the same GFS filesystem.

>4.  We're seriously considering using LVM for volume management.  Does it
>have size limits per volume or other limitations that we should be aware of?

GFS may serve better. It is a full shared filesystem with RAID target disks
(these are smart controllers) and incudes journaling.

>I'm sure these answers are out there, but I haven't been able to find
>definitive answers (it seems everyone has a different answer to each
>question).  Any assistance in pointing me to the correct information would
>be greatly appreciated.

I haven't had direct expierence with GFS, but it looks very impressive in
the documentation.

-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.

Re: Questions about Enterprise Storage with Linux

[Marcelo Tosatti]

On Wed, 7 Mar 2001, Jesse Pollard wrote:

> On Wed, 07 Mar 2001, Tom Sightler wrote:
> >Hi All,
> >
> >I'm seeking information in regards to a large Linux implementation we are
> >planning.  We have been evaluating many storage options and I've come up
> >with some questions that I have been unable to answer as far as Linux
> >capabilities in regards to storage.
> >
> >We are looking at storage systems that provide approximately 1TB of capacity
> >for now and can scale to 10+TB in the future.  We will almost certainly use
> >a storage system that provides both fiber channel connectivity as well as
> >NFS connectivity.
> >
> >The questions that have been asked are as follows (assume 2.4.x kernels):
> >
> >1.  What is the largest block device that linux currently supports?  i.e.
> >Can I create a single 1TB volume on my storage device and expect linux to
> >see it and be able to format it?
> 
> Checkout the GFS project for really large filesystems with a high capability
> of "fail safe" configuration.
> 
> The block/file limits are more determined by the size of the hosts. Alpha/Sparc
> based systems use 64 bit operations, Intel/AMD use 32 bit. It also depends
> on usage of the sign bit in the drivers. Most 32bit systems are limited
> to 1 TB (depending on the driver of course - some allow for 2 TB).

Even on 64-bit architectures the hard upper limit is 2TB. (32-bit block
numbers)

Questions about Enterprise Storage with Linux

[Tom Sightler]

Hi All,

I'm seeking information in regards to a large Linux implementation we are
planning.  We have been evaluating many storage options and I've come up
with some questions that I have been unable to answer as far as Linux
capabilities in regards to storage.

We are looking at storage systems that provide approximately 1TB of capacity
for now and can scale to 10+TB in the future.  We will almost certainly use
a storage system that provides both fiber channel connectivity as well as
NFS connectivity.

The questions that have been asked are as follows (assume 2.4.x kernels):

1.  What is the largest block device that linux currently supports?  i.e.
Can I create a single 1TB volume on my storage device and expect linux to
see it and be able to format it?

2.  Does linux have any problems with large (500GB+) NFS exports, how about
large files over NFS?

3.  What filesystem would be best for such large volumes?  We currently use
reirserfs on our internal system, but they generally have filesystems in the
18-30GB ranges and we're talking about potentially 10-20x that.  Should we
look at JFS/XFS or others?

4.  We're seriously considering using LVM for volume management.  Does it
have size limits per volume or other limitations that we should be aware of?

I'm sure these answers are out there, but I haven't been able to find
definitive answers (it seems everyone has a different answer to each
question).  Any assistance in pointing me to the correct information would
be greatly appreciated.

Thanks,
Tom