Re: IDE/ATAPI in 2.5

Re: IDE/ATAPI in 2.5

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4394 bytes --]

>From zaitcev@redhat.com Sun Jul 14 17:05:31 2002

>>>I will violently oppose anything that implies that the IDE layer uses the
>>>SCSI layer normally.  No way, Jose. I'm all for scrapping, but the thing
>>>that should be scrapped is ide-scsi.
>> 
>> Nobody who has a technical backgroupd and knows what to do wuld ever
>> make such a proposal.
>> 
>> Instead, there needs to be one or more SCSI HA driver as part of the
>> SCSI stack. This driver also needs to deal with plain ATA in order
>> to be able to coordinate access.
>> 
>> Jörg

>Such driver would only work with ATAPI devices. Joerg, does not
>seem to realize that the vast majority of IDE devices do not
>support ATAPI at all. As a rule of thumb, winchesters do not,
>CD-ROMs and such do, and tapes do too. To make a pseudo-HA
>driver which speaks plain IDE and plugs into SCSI subsistem
>would saddle the IDE with SCSI limitations, and add one more
>layer for no benefit whatsoever.

You did not seem to read my previous postings. It is simple to have
a second upper level interface for the IDE HBS driver that connects to
e.g. the ATA HD driver interface. If you like to have 100% backwards
compatibility you could even put the current non-standard part from 
ide-cd on top of this 2nd type of upper level code. Make sure, that 
the ATAPI/SCSI part is probed first, to allow to connect the SCSI 
drivers first and lock the ATA type interface against being used by
drivers like ide-cd in case that a SCSI type of high level driver did
feel responsible to deal with a specific drive.

Of curse this will only work if there is some kind of accepted 
development leadership that makes a concept, works on data structures and 
finally instructs the maintainers of old code to find a way to convert
their current driver to the new model. This new model usually would
cause parts of the code that exists more than once to be removed from
all the old drivers that have been considered to represent 'the wrong way'
in the tree of the evolution of the kernel.

Stop looking at your own belly and have a look at what other OS do.
It seems to be one of the main problems in the Linux development.


People usually only look at Linux and inside Linux, they only look at 
their part of the kernel. Doing it this way, they define their own 
interfaces and don't look enough to the left and to the right.

To make it not look as if I am only bashing current IDE/SCSI code, let
me give an example from the filesystem part.

Have a look at the file 

ftp://ftp.fokus.gmd.de/pub/unix/star/testscript/rock.tar.bz2

It is a tar archive that contains 207,440 empty files and one directory 
using the names of the files in the 'rock' subdirectory from the 
freedb project. This is a snapshot taken on May 30th 2002.

If you extract this archive on a machine running Solaris or FreeBSD,
it takes less than 7 minutes.

A pentium 800 running Solaris 9 results in:

# star -xp -time < rock.tar.bz2
star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
star: Total time 405.988sec (255 kBytes/sec)
6:46.051r 12.920u 63.420s 18% 0M 0+0k 0st 0+0io 0pf+0w

You see during the 6:43, the machine is 82% idle.


A Pentium 1200 running Linux-2.5.25 (ext3) results in:

# star -xp -time < rock.tar.bz2
star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
star: Total time 3190.483sec (32 kBytes/sec)
53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w

You see, during the 53:20, the machine is only 7% idle!

It wasted 2900 seconds of CPU time on Linux. Let me guess: this was done
inside the function strcmp().

There are ~ 5 different filesystems on Linux, but none if the projects seem
to care about the code outside the FS low level code. I suspect, that
this is not any better if you use reiserfs.

Solaris and FreeBSD put all the effort into one filesystem trying to make
it as good as possible. In Linux, it seems that nobody prooved the overall
concept of the kernel.

Jörg

 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
       schilling@fokus.gmd.de		(work) chars I am J"org Schilling
 URL:  http://www.fokus.gmd.de/usr/schilling   ftp://ftp.fokus.gmd.de/pub/unix

Re: IDE/ATAPI in 2.5

[Tomas Szepe]

> A Pentium 1200 running Linux-2.5.25 (ext3) results in:
> 
> # star -xp -time < rock.tar.bz2
> star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> star: Total time 3190.483sec (32 kBytes/sec)
> 53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w
> 
> You see, during the 53:20, the machine is only 7% idle!


A Pentium 1200, eh?
More like Pentium 120 or star just doesn't cut it.

--

Athlon 1GHz:

kala@nibbler:/tmp/1$ time tar xjf rock.tar.bz2
real    3m19.703s
user    0m9.870s
sys     0m24.840s

According to top, the system was ~90% idle during the extraction.
Linux 2.4.19-rc1-ac3, reiserfs 3.6.

T.

PS. Solaris is over 60% slower than Linux 2.2/2.4 in common fs
operations on my SMP SPARCstation 10.

Re: IDE/ATAPI in 2.5

[Willy TARREAU]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1324 bytes --]

On Sun, Jul 14, 2002 at 08:11:33PM +0200, Joerg Schilling wrote:
> You did not seem to read my previous postings. It is simple to have

Jörg,

could you please stop starting all your mails like this one and
insulting every people who don't obviously understand what you
nearly didn't explain in 30 mails ? You're whinning about people
who don't take you seriously, but you don't try to be constructive.
Don't forget that this is a mailing list and that there are lots
of people who take the threads long after they have begun, and
please forgive them for not having read your 5000 previous lines.

Since you claim to have very clear ideas about what you'd like to
see in the kernel, please post a clear and complete proposal here
(with another subject, BTW since most people won't read it else).

This way, you'll be able to argue on technical points with some
competent and clever people, but you won't if you continue to
measure your personal experience to that of others. You say you
don't want to talk to them because, unlike you, they don't know
what a kernel is, but this is non sense. They may have a different
conception than you, that's all.

>From your claimed experience, you seem to be 45, but from your
attitude, 8. I think reality is between both, but I hope the
former.

Thanks for your understanding,
Willy

Re: IDE/ATAPI in 2.5

[Tomas Szepe]

> > A Pentium 1200 running Linux-2.5.25 (ext3) results in:
> > 
> > # star -xp -time < rock.tar.bz2
> > star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> > star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> > star: Total time 3190.483sec (32 kBytes/sec)
> > 53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w
> > 
> > You see, during the 53:20, the machine is only 7% idle!
> 
> A Pentium 1200, eh?
> More like Pentium 120 or star just doesn't cut it.

Now I'm actually pretty sure you meant 386DX/33!

I don't know whom you're trying to fool, but even my P2/233
can get the work done in under 5 minutes:

kala@hubert:/tmp$ time tar xjf rock.tar.bz2

real    4m50.598s
user    0m36.700s
sys     1m51.860s

Linux 2.4.19-pre10-ac2, reiserfs 3.6.

T.

Re: IDE/ATAPI in 2.5

[Sean Neakums]

commence  Tomas Szepe  quotation:
> > > A Pentium 1200 running Linux-2.5.25 (ext3) results in:
> > > 
> > > # star -xp -time < rock.tar.bz2
> > > star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> > > star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> > > star: Total time 3190.483sec (32 kBytes/sec)
> > > 53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w
> > > 
> > > You see, during the 53:20, the machine is only 7% idle!
> > 
> > A Pentium 1200, eh?
> > More like Pentium 120 or star just doesn't cut it.
> 
> Now I'm actually pretty sure you meant 386DX/33!
> 
> I don't know whom you're trying to fool, but even my P2/233
> can get the work done in under 5 minutes:
> 
> kala@hubert:/tmp$ time tar xjf rock.tar.bz2
> 
> real    4m50.598s
> user    0m36.700s
> sys     1m51.860s
> 
> Linux 2.4.19-pre10-ac2, reiserfs 3.6.

Aha, but reiserfs's directories are indexed, are they not?  Whereas
ext3's directories are flat and require a linear search for lookups
and modifications.  This may be what Joerg's example highlights.

-- 
 /                          |
[|] Sean Neakums            |  Questions are a burden to others;
[|] <sneakums@zork.net>     |      answers a prison for oneself.
 \                          |

Re: IDE/ATAPI in 2.5

[Tomas Szepe]

> > > > A Pentium 1200 running Linux-2.5.25 (ext3) results in:
> > > > 
> > > > # star -xp -time < rock.tar.bz2
> > > > star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> > > > star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> > > > star: Total time 3190.483sec (32 kBytes/sec)
> > > > 53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w
> > > > 
> > > > You see, during the 53:20, the machine is only 7% idle!
> > > 
> > > A Pentium 1200, eh?
> > > More like Pentium 120 or star just doesn't cut it.
> > 
> > Now I'm actually pretty sure you meant 386DX/33!
> > 
> > I don't know whom you're trying to fool, but even my P2/233
> > can get the work done in under 5 minutes:
> > 
> > kala@hubert:/tmp$ time tar xjf rock.tar.bz2
> > 
> > real    4m50.598s
> > user    0m36.700s
> > sys     1m51.860s
> > 
> > Linux 2.4.19-pre10-ac2, reiserfs 3.6.
> 
> Aha, but reiserfs's directories are indexed, are they not?  Whereas
> ext3's directories are flat and require a linear search for lookups
> and modifications.  This may be what Joerg's example highlights.

Unfortunately, I don't have an ext3 partition handy to perform a quick
test, but the following Joerg's accusation has been invalidated nonetheless:

> Solaris and FreeBSD put all the effort into one filesystem trying to make
> it as good as possible. In Linux, it seems that nobody prooved the overall
> concept of the kernel.

If that's the case, how come a Linux-powered Pentium 233 can do in
5 minutes for what a Solaris-powered Pentium 800 needs 7 minutes?

I believe Joerg owes a big big apology to Al Viro here.

T.

Re: IDE/ATAPI in 2.5

[Andrew Morton]

Joerg Schilling wrote:
> 
> ...
> ftp://ftp.fokus.gmd.de/pub/unix/star/testscript/rock.tar.bz2
> 
> It is a tar archive that contains 207,440 empty files and one directory
> using the names of the files in the 'rock' subdirectory from the
> freedb project. This is a snapshot taken on May 30th 2002.
> 
> If you extract this archive on a machine running Solaris or FreeBSD,
> it takes less than 7 minutes.

With ext3 and the htree directory indexing, a 500MHz PIII does
it in 57 seconds.

tar xjf ~/rock.tar.bz2  19.63s user 29.42s system 84% cpu 57.879 total

> A pentium 800 running Solaris 9 results in:
> 
> # star -xp -time < rock.tar.bz2
> star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> star: Total time 405.988sec (255 kBytes/sec)
> 6:46.051r 12.920u 63.420s 18% 0M 0+0k 0st 0+0io 0pf+0w
> 
> You see during the 6:43, the machine is 82% idle.
> 
> A Pentium 1200 running Linux-2.5.25 (ext3) results in:
> 
> # star -xp -time < rock.tar.bz2
> star: WARNING: Archive is bzip2 compressed, trying to use the -bz option.
> star: 10372 blocks + 1536 bytes (total of 106210816 bytes = 103721.50k).
> star: Total time 3190.483sec (32 kBytes/sec)
> 53:10.490r 12.299u 2970.099s 93% 0M 0+0k 0st 0+0io 4411pf+0w
> 
> You see, during the 53:20, the machine is only 7% idle!
> 
> It wasted 2900 seconds of CPU time on Linux. Let me guess: this was done
> inside the function strcmp().

Nope. ext3 and ext2 directories use the traditional first-fit
search-from-start for directories.  So adding 200k files to
a single directory is pathological.

> There are ~ 5 different filesystems on Linux, but none if the projects seem
> to care about the code outside the FS low level code. I suspect, that
> this is not any better if you use reiserfs.
> 
> Solaris and FreeBSD put all the effort into one filesystem trying to make
> it as good as possible. In Linux, it seems that nobody prooved the overall
> concept of the kernel.

Apply http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.25/ext3-htree.patch
to your 2.5.25 tree, mount with `-o index' and enjoy watching ext3 eat
Solaris and FreeBSD's lunch.

htree isn't quite ready yet and development seems a little moribund.
It'd be nice to get it finished off.

-

Re: IDE/ATAPI in 2.5

[Andrew Morton]

Steffen Persvold wrote:
> 
> On Sun, 14 Jul 2002, Andrew Morton wrote:
> 
> > > Solaris and FreeBSD put all the effort into one filesystem trying to make
> > > it as good as possible. In Linux, it seems that nobody prooved the overall
> > > concept of the kernel.
> >
> > Apply http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.25/ext3-htree.patch
> > to your 2.5.25 tree, mount with `-o index' and enjoy watching ext3 eat
> > Solaris and FreeBSD's lunch.
> >
> > htree isn't quite ready yet and development seems a little moribund.
> > It'd be nice to get it finished off.
> >
> 
> Hi Andrew,
> 
> Will the patch ever be backported to 2.4 ?
> 

That's a 2.5 forward-port, actually.  Christopher, Daniel, Stephen
and co are working against 2.4.

It mostly works, but seems to still need a bit of debugging of corner
cases and general hardening up.

-

2.5 ext3 + htree (was: IDE/ATAPI in 2.5)

[Matthias Andree]

On Sun, 14 Jul 2002, Andrew Morton wrote:

> > It wasted 2900 seconds of CPU time on Linux. Let me guess: this was done
> > inside the function strcmp().
> 
> Nope. ext3 and ext2 directories use the traditional first-fit
> search-from-start for directories.  So adding 200k files to
> a single directory is pathological.
> 
> > There are ~ 5 different filesystems on Linux, but none if the projects seem
> > to care about the code outside the FS low level code. I suspect, that
> > this is not any better if you use reiserfs.
> > 
> > Solaris and FreeBSD put all the effort into one filesystem trying to make
> > it as good as possible. In Linux, it seems that nobody prooved the overall
> > concept of the kernel.
> 
> Apply http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.25/ext3-htree.patch
> to your 2.5.25 tree, mount with `-o index' and enjoy watching ext3 eat
> Solaris and FreeBSD's lunch.

I didn't benchmark, but how much lunch is left? UFS_DIRHASH was
introduced into FreeBSD by Ian Dowse a year ago (released in FreeBSD
4.4), and activated in FreeBSD 4.5's generic table. In case you missed
that, FreeBSD 4.6 is out since about four weeks.

options         UFS_DIRHASH             #Improve performance on big directories

http://www.freebsd.org/releases/4.4R/relnotes-i386.html#AEN197
http://www.freebsd.org/releases/4.5R/relnotes-i386.html#AEN250
http://www.cnri.dit.ie/Downloads/Malone_2001_bsdcon.pdf

The latter document by David Malone (November 2001) claims "MH 33 k
files create: 70s to 2.5s, pack: 240s to 2.5s, rm: 4.7s to 2s."

So might be ext3fs comes just in time for dessert, or is htree so much
faster than UFS_DIRHASH?

-- 
Matthias Andree