Re: aio

Re: aio

[Mikael Pettersson]

On Wed, 19 Dec 2001 19:21:36 -0500, Benjamin LaHaise wrote:
>People are throwing money at the problem.  We're now at a point that in 
>order to provide the interested people with something they can use, we 
>need some kind of way to protect their applications against calling an 
>unsuspecting new mmap syscall instead of the aio syscall specified in 
>the kernel they compiled against.

One option is to use a mechanism where the kernel selects the aio
syscall number from any of the available numbers, and publishes it
in some easily accessible location. User-space then needs to grab it
from that location at app/lib init time before doing any actual work.
(This of course goes away when/if you do get an official syscall number.)

This is at least how I intend to do the "ioctl on device" to "real
syscall" transition for my x86 performance counters driver.

/Mikael

Re: Scheduler ( was: Just a second ) ...

[Daniel Phillips]

On December 18, 2001 08:04 pm, Alan Cox wrote:
> > I'm not subscribed to any, thank you very much. I read them through a news
> > gateway, which gives me access to the common ones.
> > 
> > And if the discussion wasn't on the common ones, then it wasn't an open
> > discussion.
> 
> If the discussion was on the l/k list then most kernel developers arent
> going to read it because tey dont have time to wade through all the crap
> that doesnt matter to them.

Hi Alan,

It's AIO we're talking about, right?  AIO is interesting to quite a few 
people.  I'd read the thread.  I'd also read any background material that Ben 
would be so kind as to supply.

--
Daniel

aio

[Ben LaHaise]

On Wed, Dec 19, 2001 at 09:01:59AM -0800, Linus Torvalds wrote:
> 
> On Wed, 19 Dec 2001, Daniel Phillips wrote:
> >
> > It's AIO we're talking about, right?  AIO is interesting to quite a few
> > people.  I'd read the thread.  I'd also read any background material that Ben
> > would be so kind as to supply.
> 
> Case closed.
> 
> Dan didn't even _know_ of the patches.

He doesn't read l-k apparently.

> Ben: end of discussion. I will _not_ apply any patches for aio if they
> aren't openly discussed. We're not microsoft, and we're not Sun. We're
> "Open Source", not "cram things down peoples throat and spring new
> features on them as a fait accompli".

Discuss them then to your heart's content.  I've posted announcements to 
both l-k and linux-aio which are both on marc.theaimsgroup.com if you're 
too lazy to get your IS to add a new list yo the internal news gateway.

> The ghost of "binary compatibility" is not an issue - if Ben or anytbody
> else finds a flaw with the design, it's a hell of a lot better to have
> that flaw fixed _before_ it's part of my kernel rather than afterwards.

Thanks for the useful feedback on the userland interface then.  Evidently 
nobody cares within the community about improving functionality on a 
reasonable timescale.  If this doesn't change soon, Linux is doomed.

		-ben

Re: aio

[Dan Kegel]

Ben LaHaise wrote:
> > Ben: end of discussion. I will _not_ apply any patches for aio if they
> > aren't openly discussed. We're not microsoft, and we're not Sun. We're
> > "Open Source", not "cram things down peoples throat and spring new
> > features on them as a fait accompli".
> 
> Discuss them then to your heart's content.  I've posted announcements to
> both l-k and linux-aio which are both on marc.theaimsgroup.com ...

Ben, I think maybe we need to get people excited about your patches,
and build up a user base, before putting them in the mainline kernel.
The volume on the linux-aio list has been pretty light, and the
visibility of the patches has been pretty low.

I know I volunteered to write some doc for your aio, and haven't delivered;
thus I'm contributing to the problem.  Mea culpa.  But there are some
small things that could be done.  A freshmeat.net entry for the project,
for instance.  Shall I create one, or would you rather do it?
A home page for linux-aio would be great, too.
- Dan

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 11:29:15AM -0800, Dan Kegel wrote:
> I know I volunteered to write some doc for your aio, and haven't delivered;
> thus I'm contributing to the problem.  Mea culpa.  But there are some
> small things that could be done.  A freshmeat.net entry for the project,
> for instance.  Shall I create one, or would you rather do it?
> A home page for linux-aio would be great, too.

I've started writing some web pages, and will gratiously accept additional 
docs/text/questions.  Freshmeat will get an entry right after I send this 
out.

		-ben
-- 
Fish.

Re: aio

[Daniel Phillips]

On December 19, 2001 07:57 pm, Ben LaHaise wrote:
> On Wed, Dec 19, 2001 at 09:01:59AM -0800, Linus Torvalds wrote:
> > 
> > On Wed, 19 Dec 2001, Daniel Phillips wrote:
> > >
> > > It's AIO we're talking about, right?  AIO is interesting to quite a few
> > > people.  I'd read the thread.  I'd also read any background material 
> > > that Ben would be so kind as to supply.
> > 
> > Case closed.
> > 
> > Dan didn't even _know_ of the patches.
> 
> He doesn't read l-k apparently.

Dan Kegel put it succinctly:

   http://marc.theaimsgroup.com/?l=linux-aio&m=100879005201064&w=2

Your original patch is here, and I do remember the post at the time:

   http://marc.theaimsgroup.com/?l=linux-kernel&m=98114243104171&w=2

This post provides *zero* context.  Ever since, I've been expecting to see 
some explanation of what the goals are, what the design principles are, what 
the historical context is, etc. etc., and that hasn't happened.

I've got a fairly recent version of the patch too, it's a little too long to 
just sit down and read, to reverse-engineer the above information.  What's 
missing here is some kind of writeup like Suparna did for Jens' bio patch 
(hint, hint).  There's no reason why every single person who might be 
interested should have to take the time to reverse-engineer the patch without 
context.

As Linus points out, the active discussion hasn't happened yet.

--
Daniel

Re: aio

[Davide Libenzi]

On Wed, 19 Dec 2001, Ben LaHaise wrote:

> Thanks for the useful feedback on the userland interface then.  Evidently
> nobody cares within the community about improving functionality on a
> reasonable timescale.  If this doesn't change soon, Linux is doomed.

Ben, maybe it's true, nobody cares :( This could be either bad or good.
On one side it could be good because this means that everyone is happy
with the kernel performance level and this could be due the fact that real
world loads does not put their applications under stress. It could be bad
because it's possible that exist applications that are currently under
stress ( yes ), but their developers do not understand that by using
different interfaces they can improve their software ( or they simply do
not understand that the application is under stress ). Or maybe application
developers are not in lk. Or maybe they're not willing to rewrite/experiment
new APIs. On one side i understand that you can have an intrinsic attitude
to push/defend your patch, while one the other side i can agree with the
Linus point to have some kind of broad discussion/adoption about it.
But if applications developers are not in this list there won't be a broad
discussion and if the patch does not go inside the mainstream kernel
"external" applications developers are not going to use it. The Linus
point could be: "why do i have to merge a new api that has had a so cold
discussion/adoption inside lk ?".
Yes egg-chicken draws the picture very well.



- Davide

Re: aio

[Pete Zaitcev]

> > > > It's AIO we're talking about, right?  AIO is interesting to quite a few
> > > > people.  I'd read the thread.  I'd also read any background material 
> > > > that Ben would be so kind as to supply.
> > > 
> > > Case closed.
> > > 
> > > Dan didn't even _know_ of the patches.

> I've got a fairly recent version of the patch too, it's a little too long to 
> just sit down and read, to reverse-engineer the above information.

Heh, I agree, in a way. I did that once, did not find any major
objections and documented about 20 small things like functions
that have extra arguments which are never used, etc. Ben saw it
and said "I know about all that, never mind".
Perhaps I should have had posted it somewhere?

-- Pete

Re: aio

[David S. Miller]

   From: Ben LaHaise <bcrl@redhat.com>
   Date: Wed, 19 Dec 2001 13:57:08 -0500
   
   Thanks for the useful feedback on the userland interface then.  Evidently 
   nobody cares within the community about improving functionality on a 
   reasonable timescale.  If this doesn't change soon, Linux is doomed.

Maybe it's because the majority of people don't care nor would ever
need to use AIO.  Are you willing to accept this possibly? :-) Linux
is anything but doomed, because you will notice that the things that
actually matter for most people are in fact improved and worked on
within a reasonable timescale.

Only very specialized applications can even benefit from AIO.  This
doesn't make it useless, but it does decrease the amount of interest
(and priority) anyone in the community will have in working on it.

Now, if these few and far between people who are actually interested
in AIO are willing to throw money at the problem to get it worked on,
that is how the "reasonable timescale" will be arrived at.  And if
they aren't willing to toss money at the problem, how important can it
really be to them? :-)

Maybe, just maybe, most people simply do not care one iota about AIO.

Linux caters to the general concerns not the nooks and cranies, that
is why it is anything but doomed.

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 04:13:59PM -0800, David S. Miller wrote:
> Now, if these few and far between people who are actually interested
> in AIO are willing to throw money at the problem to get it worked on,
> that is how the "reasonable timescale" will be arrived at.  And if
> they aren't willing to toss money at the problem, how important can it
> really be to them? :-)

People are throwing money at the problem.  We're now at a point that in 
order to provide the interested people with something they can use, we 
need some kind of way to protect their applications against calling an 
unsuspecting new mmap syscall instead of the aio syscall specified in 
the kernel they compiled against.

> Maybe, just maybe, most people simply do not care one iota about AIO.
> 
> Linux caters to the general concerns not the nooks and cranies, that
> is why it is anything but doomed.

What I'm saying is that for more people to play with it, it needs to be 
more widely available.  The set of developers that read linux-kernel and 
linux-aio aren't giving much feedback.  I do not expect the code to go 
into 2.5 at this point in time.  All I need is a set of syscall numbers 
that aren't going to change should this implementation stand up to the 
test of time.

		-ben
-- 
Fish.

Re: aio

[Andrew Morton]

Benjamin LaHaise wrote:
> 
> All I need is a set of syscall numbers that aren't going to change
> should this implementation stand up to the test of time.

The aio_* functions are part of POSIX and SUS, so merely reserving
system call numbers for them does not seems a completely dumb
thing to do, IMO.

-

Re: aio

[H. Peter Anvin]

Followup to:  <3C213270.966DABFE@zip.com.au>
By author:    Andrew Morton <akpm@zip.com.au>
In newsgroup: linux.dev.kernel
> 
> The aio_* functions are part of POSIX and SUS, so merely reserving
> system call numbers for them does not seems a completely dumb
> thing to do, IMO.
> 

Yes, it is, unless you already have a design for how to map the aio_*
library functions onto system calls.

	-hpa


-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: aio

[Davide Libenzi]

On Wed, 19 Dec 2001, Benjamin LaHaise wrote:

> What I'm saying is that for more people to play with it, it needs to be
> more widely available.  The set of developers that read linux-kernel and
> linux-aio aren't giving much feedback.  I do not expect the code to go
> into 2.5 at this point in time.  All I need is a set of syscall numbers
> that aren't going to change should this implementation stand up to the
> test of time.

It would be nice to have a cooperation between glibc and the kernel to
have syscalls mapped by name, not by number.
With name->number resolved by crtbegin.o reading a public kernel table
or accessing a fixed-ID kernel map function and filling a map.
So if internally ( at the application ) sys_getpid has index 0, the
sysmap[0] will be filled with the id retrieved inside the kernel by
looking up "sys_getpid".
Eat too spicy today ?




- Davide

Re: aio

[Bill Huey]

On Wed, Dec 19, 2001 at 04:13:59PM -0800, David S. Miller wrote:
> Maybe it's because the majority of people don't care nor would ever
> need to use AIO.  Are you willing to accept this possibly? :-) Linux
> is anything but doomed, because you will notice that the things that
> actually matter for most people are in fact improved and worked on
> within a reasonable timescale.
> 
> Only very specialized applications can even benefit from AIO.  This
> doesn't make it useless, but it does decrease the amount of interest
> (and priority) anyone in the community will have in working on it.

Folks doing serious server side Java and runtime internals would
definitely be able to use this stuff, namely me. It'll remove the
abuse of threading used to deal with large IO systems when NIO comes out
for 1.4. And as a JVM engineer for the FreeBSD community I'm drooling
over stuff like that.

> Now, if these few and far between people who are actually interested
> in AIO are willing to throw money at the problem to get it worked on,
> that is how the "reasonable timescale" will be arrived at.  And if
> they aren't willing to toss money at the problem, how important can it
> really be to them? :-)

Like the Java folks ? few and far between ? What you're saying it just
plain outdated and from a previous generation of thinking that has
become irrelevant as the community has grown.

> Maybe, just maybe, most people simply do not care one iota about AIO.
> 
> Linux caters to the general concerns not the nooks and cranies, that
> is why it is anything but doomed.

Again, Linux collectively has outgrown that thinking and the scope of
what the previous generation of engineers can responsible for, which is
why it's important for folks like Ben should be encourage to take it to
the next level.

bill

Re: aio

[David S. Miller]

   From: Bill Huey <billh@tierra.ucsd.edu>
   Date: Wed, 19 Dec 2001 17:16:31 -0800
   
   Like the Java folks ? few and far between ?

Precisely, in fact.  Anyone who can say that Java is going to be
relevant in a few years time, with a straight face, is only kidding
themselves.

Java is not something to justify a new kernel feature, that is for
certain.

Re: aio

[Bill Huey]

On Wed, Dec 19, 2001 at 05:20:46PM -0800, David S. Miller wrote:
> Precisely, in fact.  Anyone who can say that Java is going to be
> relevant in a few years time, with a straight face, is only kidding
> themselves.

Oh give me coke shooting, Steeley Dan, late 70s bitter kernel
programmer break...

> Java is not something to justify a new kernel feature, that is for
> certain.

Java is here now and used extensively on server side applications.
Simply dismissing it doesn't invalidate the claim that I made before
about how this mentality is outdated.

The economic inertia of Java driven server applications should have
enough force that it is justifyable to RedHat and other commerical
organizations to support it regardless of what your current view is
on this topic.

Even within the BSD/OS group at BSDi/WindRiver, (/me former BSD/OS
engineer) some kind of dedicated async IO system inside kernel was
talked about as highly desireable and possibly a more direct way
of dealing with VM page/async IO event issues that don't map
conceptually to a scheduler context cleanly.

AIO is good, plain and simple.

bill

Re: aio

[Cameron Simpson]

On Wed, Dec 19, 2001 at 05:20:46PM -0800, David S. Miller <davem@redhat.com> wrote:
|    From: Bill Huey <billh@tierra.ucsd.edu>
|    Like the Java folks ? few and far between ?
| Precisely, in fact.  Anyone who can say that Java is going to be
| relevant in a few years time, with a straight face, is only kidding
| themselves.

Maybe. I'm good at that.

| Java is not something to justify a new kernel feature, that is for
| certain.

Of itself, maybe. (Though an attitude like yours is a core reason Java is
spreading as slowly as it is - much like Linux desktops...)

However, heavily threaded apps regardless of language are hardly likely
to disappear; threads are the natural way to write many many things. And
if the kernel implements threads as on Linux, then the scheduler will
become much more important to good performance.
-- 
Cameron Simpson, DoD#743        cs@zip.com.au    http://www.zip.com.au/~cs/

Questions are a burden to others,
	Answers, a prison for oneself.

Re: aio

[David S. Miller]

   From: Bill Huey <billh@tierra.ucsd.edu>
   Date: Wed, 19 Dec 2001 18:26:28 -0800
   
   The economic inertia of Java driven server applications should have
   enough force that it is justifyable to RedHat and other commerical
   organizations to support it regardless of what your current view is
   on this topic.

So they'll get paid to implement and support it, and that is precisely
what is happening right now.  And the whole point I'm trying to make
is that that is where it's realm is right now.

If AIO was so relevant+sexy we'd be having threads of discussion about
the AIO implementation instead of threads about how relevant it is or
is not for the general populace.  Wouldn't you concur?  :-)

The people doing Java server applets are such a small fraction of the
Linux user community.

Re: aio

[John Heil]

On Wed, 19 Dec 2001, David S. Miller wrote:

> Date: Wed, 19 Dec 2001 18:45:27 -0800 (PST)
> From: "David S. Miller" <davem@redhat.com>
> To: billh@tierra.ucsd.edu
> Cc: bcrl@redhat.com, torvalds@transmeta.com, linux-kernel@vger.kernel.org,
>     linux-aio@kvack.org
> Subject: Re: aio
> 
>    From: Bill Huey <billh@tierra.ucsd.edu>
>    Date: Wed, 19 Dec 2001 18:26:28 -0800
>    
>    The economic inertia of Java driven server applications should have
>    enough force that it is justifyable to RedHat and other commerical
>    organizations to support it regardless of what your current view is
>    on this topic.
> 
> So they'll get paid to implement and support it, and that is precisely
> what is happening right now.  And the whole point I'm trying to make
> is that that is where it's realm is right now.
> 
> If AIO was so relevant+sexy we'd be having threads of discussion about
> the AIO implementation instead of threads about how relevant it is or
> is not for the general populace.  Wouldn't you concur?  :-)
> 
> The people doing Java server applets are such a small fraction of the
> Linux user community.

True for now, but if we want to expand linux into the enterprise and the
desktop to a greater degree, then we need to support the Java community to
draw them and their management in, rather than delaying beneficial 
features until their number on lkml reaches critical mass for a design
discussion.


-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes
1-714-774-6952
johnhscs@sc-software.com
http://www.sc-software.com
-----------------------------------------------------------------

Re: aio

[David S. Miller]

   From: John Heil <kerndev@sc-software.com>
   Date: Wed, 19 Dec 2001 18:57:34 +0000 (   )
   
   True for now, but if we want to expand linux into the enterprise and the
   desktop to a greater degree, then we need to support the Java community to
   draw them and their management in, rather than delaying beneficial 
   features until their number on lkml reaches critical mass for a design
   discussion.

Firstly, you say this as if server java applets do not function at all
or with acceptable performance today.  That is not true for the vast
majority of cases.

If java server applet performance in all cases is dependent upon AIO
(it is not), that would be pretty sad.  But it wouldn't be the first
time I've heard crap like that.  There is propaganda out there telling
people that 64-bit address spaces are needed for good java
performance.  Guess where that came from?  (hint: they invented java
and are in the buisness of selling 64-bit RISC processors)

Re: aio

[John Heil]

On Wed, 19 Dec 2001, David S. Miller wrote:

> Date: Wed, 19 Dec 2001 19:06:29 -0800 (PST)
> From: "David S. Miller" <davem@redhat.com>
> To: kerndev@sc-software.com
> Cc: billh@tierra.ucsd.edu, bcrl@redhat.com, torvalds@transmeta.com,
>     linux-kernel@vger.kernel.org, linux-aio@kvack.org
> Subject: Re: aio
> 
>    From: John Heil <kerndev@sc-software.com>
>    Date: Wed, 19 Dec 2001 18:57:34 +0000 (   )
>    
>    True for now, but if we want to expand linux into the enterprise and the
>    desktop to a greater degree, then we need to support the Java community to
>    draw them and their management in, rather than delaying beneficial 
>    features until their number on lkml reaches critical mass for a design
>    discussion.
> 
> Firstly, you say this as if server java applets do not function at all
> or with acceptable performance today.  That is not true for the vast
> majority of cases.
> 
> If java server applet performance in all cases is dependent upon AIO
> (it is not), that would be pretty sad.  But it wouldn't be the first
> time I've heard crap like that.  There is propaganda out there telling
> people that 64-bit address spaces are needed for good java
> performance.  Guess where that came from?  (hint: they invented java
> and are in the buisness of selling 64-bit RISC processors)
> 

Agree. However, put your business hat for a minute. We want increased
market share for linux and a lot of us, you included, live by it. 
If aio, the proposed implementation or some other, can provide an
adequate performance boost for Java (yet to be seen), that at least 
allows the marketing folks one more argument to draw users to linux. 
Do think the trade mags etc don't watch what we do? A demonstrable
advantage in Java performance is marketable and beneficial to all.
   

-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes
1-714-774-6952
johnhscs@sc-software.com
http://www.sc-software.com
-----------------------------------------------------------------

Re: aio

[Bill Huey]

On Wed, Dec 19, 2001 at 07:06:29PM -0800, David S. Miller wrote:
> Firstly, you say this as if server java applets do not function at all
> or with acceptable performance today.  That is not true for the vast
> majority of cases.
> 
> If java server applet performance in all cases is dependent upon AIO
> (it is not), that would be pretty sad.  But it wouldn't be the first

Java is pretty incomplete in this area, which should be addressed to a
great degree in the new NIO API.

The core JVM isn't dependent on this stuff per se for performance, but
it is critical to server side programs that have to deal with highly
scalable IO systems, largely number of FDs, that go beyond the current
expressiveness of select()/poll().

This is all standard fare in *any* kind of high performance networking
application where some kind of high performance kernel/userspace event
delivery system is needed, kqueue() principally.

> time I've heard crap like that.  There is propaganda out there telling
> people that 64-bit address spaces are needed for good java
> performance.  Guess where that came from?  (hint: they invented java
> and are in the buisness of selling 64-bit RISC processors)

What ? oh god. HotSpot is a pretty amazing compiler and it performs well.
Swing does well now, but the lingering issue in Java is the shear size
of it and possibly GC issues. It pretty clear that it's going to get
larger, which is fine since memory is cheap.

bill

Re: aio

[Martin Dalecki]

Bill Huey wrote:

>On Wed, Dec 19, 2001 at 07:06:29PM -0800, David S. Miller wrote:
>
>>Firstly, you say this as if server java applets do not function at all
>>or with acceptable performance today.  That is not true for the vast
>>majority of cases.
>>
>>If java server applet performance in all cases is dependent upon AIO
>>(it is not), that would be pretty sad.  But it wouldn't be the first
>>
>
>Java is pretty incomplete in this area, which should be addressed to a
>great degree in the new NIO API.
>
>The core JVM isn't dependent on this stuff per se for performance, but
>it is critical to server side programs that have to deal with highly
>scalable IO systems, largely number of FDs, that go beyond the current
>expressiveness of select()/poll().
>
>This is all standard fare in *any* kind of high performance networking
>application where some kind of high performance kernel/userspace event
>delivery system is needed, kqueue() principally.
>
>>time I've heard crap like that.  There is propaganda out there telling
>>people that 64-bit address spaces are needed for good java
>>performance.  Guess where that came from?  (hint: they invented java
>>and are in the buisness of selling 64-bit RISC processors)
>>
>
>What ? oh god. HotSpot is a pretty amazing compiler and it performs well.
>Swing does well now, but the lingering issue in Java is the shear size
>of it and possibly GC issues. It pretty clear that it's going to get
>larger, which is fine since memory is cheap.
>
I remind you: ORACLE 9i is requiring half a gig as a minimum just due to the
use of the CRAPPY PIECE OF SHIT written in the Java, called, you guess 
it: Just the
bloody damn Installer. Java is really condemned just due to the fact 
that both terms: speed
and memmory usage are both allways only *relative* to other systems.

And yes GC's have only one problem - they try to give a general solution 
for problems
which can be easly prooven to be mathmematically insolvable. The 
resulting undeterministic
behaviour of applications is indeed the thing which is hurting most.

Re: aio

[David S. Miller]

   From: John Heil <kerndev@sc-software.com>
   Date: Wed, 19 Dec 2001 19:30:13 +0000 (   )
   
   Agree. However, put your business hat for a minute. We want increased
   market share for linux and a lot of us, you included, live by it. 

Oh my buisness hat is certainly on, which is why I keep talking about
the people who need this "paying for implementation and support of AIO
for Linux". :-)

Make no mistake, I do agree with your points though in general.

But those things are not dependent upon "standard Linus Linux" having
AIO first, this is what vendors do for differentiation by shipping
feature X in their kernel before others.

Re: aio

[Bill Huey]

On Wed, Dec 19, 2001 at 06:45:27PM -0800, David S. Miller wrote:
> So they'll get paid to implement and support it, and that is precisely
> what is happening right now.  And the whole point I'm trying to make
> is that that is where it's realm is right now.
> 
> If AIO was so relevant+sexy we'd be having threads of discussion about
> the AIO implementation instead of threads about how relevant it is or
> is not for the general populace.  Wouldn't you concur?  :-)

I attribute the lack of technical discussion to the least common denomiator
culture of the Linux community and not the merits of the actual technical
system itself. That's what linux-aio@ is for...

And using lkml as a AIO forum is probably outside of the scope of this list
and group.

> The people doing Java server applets are such a small fraction of the
> Linux user community.

Yeah, but the overall Unix community probably has something different to say
about that, certainly. Even in BSD/OS, this JVM project I've been working on is
recognized as one of the most important systems second (probably) only to the
kernel itself. And, IMO, they have a more balanced view of this language
system and the value of it economically as a money making platform instead of
showing off to their peers. It's a greatly anticipated project in all of the
BSDs.

That's my semi-obnoxious take on it. ;-)

bill

Re: aio

[David S. Miller]

   From: Bill Huey <billh@tierra.ucsd.edu>
   Date: Wed, 19 Dec 2001 19:07:16 -0800
   
   And using lkml as a AIO forum is probably outside of the scope of this list
   and group.

This whole thread exists because Linus wants public general and
technical discussion on lkml of new features to happen before he
considers putting them into the tree, and the fact that they are not
in the tree because he isn't seeing such enthusiastic discussions
happening at all.

I don't think AIO, because of it's non-trivial impact to the tree, is
at all outside the scope of this list.  This is in fact the place
where major stuff like AIO is meant to be discussed, not some special
list where only "AIO people" hang out, of course people on that list
will be enthusiastic about AIO!

Frankly, on your other comments, I don't give a rats ass what BSD/OS
people are doing about, nor how highly they rate, Java.  That is
neither here nor there.  Java is going to be dead in a few years, and
let's just agree to disagree about this particular point ok?

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 07:13:54PM -0800, David S. Miller wrote:
> I don't think AIO, because of it's non-trivial impact to the tree, is
> at all outside the scope of this list.  This is in fact the place
> where major stuff like AIO is meant to be discussed, not some special
> list where only "AIO people" hang out, of course people on that list
> will be enthusiastic about AIO!

Well maybe yourself and others should make some comments about it then.

> Frankly, on your other comments, I don't give a rats ass what BSD/OS
> people are doing about, nor how highly they rate, Java.  That is
> neither here nor there.  Java is going to be dead in a few years, and
> let's just agree to disagree about this particular point ok?

Who cares about Java?  What about high performance LDAP servers or tux-like 
userspace performance?  How about faster select and poll?  An X server that 
doesn't have to make a syscall to find out that more data has arrived?  What 
about nbd or iscsi servers that are in userspace and have all the benefits 
that their kernel side counterparts do?

		-ben
-- 
Fish.

Re: aio

[Alan Cox]

> Who cares about Java?  What about high performance LDAP servers or tux-like 
> userspace performance?  How about faster select and poll?  An X server that 

select/poll is a win - and Java recently discovered poll/select semantics 8)

Re: aio

[Benjamin LaHaise]

On Fri, Dec 21, 2001 at 05:24:33PM +0000, Alan Cox wrote:
> select/poll is a win - and Java recently discovered poll/select semantics 8)

Anything is a win over Java's threading model.

		-ben
-- 
Killer Attack Fish.

Re: aio

[Bill Huey]

On Fri, Dec 21, 2001 at 12:16:45PM -0500, Benjamin LaHaise wrote:
> On Fri, Dec 21, 2001 at 05:24:33PM +0000, Alan Cox wrote:
> > select/poll is a win - and Java recently discovered poll/select semantics 8)
> 
> Anything is a win over Java's threading model.
> 
> 		-ben

Yeah, it's just another abstraction layer that lives on top of native threading
model, so you don't have to worry about stuff like spinlock contention since
it's been pushed down into the native threading implementation. It doesn't really
add a tremendous amount of overhead given how delegates all of that to the
native OS threading model.

Also, t would be nice to have some regular way of doing read-write lock without
having to implement it in Java language itself, but it's not too critical since
folks don't really push or use the JVM in that way just yet. It's certainly
important in certain high contention systems in the kernel.

bill

Re: aio

[David S. Miller]

   From: Benjamin LaHaise <bcrl@redhat.com>
   Date: Wed, 19 Dec 2001 22:47:17 -0500

   Well maybe yourself and others should make some comments about it then.
   
Because, like I keep saying, it is totally uninteresting for most of
us.

   Who cares about Java?

The people telling me on this list how important AIO is for Linux :-)

   What about high performance LDAP servers or tux-like 
   userspace performance?

People have done "faster than TUX" userspace web service with the
current kernel, that is without AIO.  There is no reason you can't
do a fast LDAP server with the current kernel either, any such claim
is simply rubbish.  Why do we need AIO again?

   How about faster select and poll?

You don't need faster select and poll as demonstrated by the
userspace "faster than TUX" example above.

   An X server that doesn't have to make a syscall to find out that
   more data has arrived?

Who really needs this kind of performance improvement?  Like anyone
really cares if their window gets the keyboard focus or a pixel over a
AF_UNIX socket a few nanoseconds faster.  How many people do you think
believe they have unacceptable X performance right now and that
select()/poll() syscalls overhead is the cause?  Please get real.

People who want graphics performance are not pushing their data
through X over a filedescriptor, they are either using direct
rendering in the app itself (ala OpenGL) or they are using shared
memory for the bulk of the data (ala Xshm or Xv extensions).

   What about nbd or iscsi servers that are in userspace and have all
   the benefits  that their kernel side counterparts do?

I do not buy this claim that it is not possible the achieve the
desired performance using existing facilities.

The only example of AIO benefitting performance I see right now are
databases.

Franks a lot,
David S. Miller
davem@redhat.com

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 09:39:10PM -0800, David S. Miller wrote:
>    How about faster select and poll?
> 
> You don't need faster select and poll as demonstrated by the
> userspace "faster than TUX" example above.

Step back for a moment.  I know of phttpd and zeus.  They both have 
a serious problem: they fall down when the load on the system exceeds 
the capabilities of the cpu.  If you'd bother to take a look at the 
aio api I'm proposing, it has less overhead under heavy load as events 
get coalesced.  Even then, the overhead under light load is less than 
signals or select or poll.

		-ben
-- 
Fish.

Re: aio

[David S. Miller]

   From: Benjamin LaHaise <bcrl@redhat.com>
   Date: Thu, 20 Dec 2001 00:58:03 -0500
   
   Step back for a moment.  I know of phttpd and zeus.  They both have 
   a serious problem: they fall down when the load on the system exceeds 
   the capabilities of the cpu.  If you'd bother to take a look at the 
   aio api I'm proposing, it has less overhead under heavy load as events 
   get coalesced.  Even then, the overhead under light load is less than 
   signals or select or poll.

No I'm not talking about phttpd nor zeus, I'm talking about the guy
who did the hacks where he'd put the http headers + content into a
seperate file and just sendfile() that to the client.

I forget what his hacks were named, but there certainly was a longish
thread on this list about it about 1 year ago if memory serves.

Re: aio

[Mike Castle]

On Wed, Dec 19, 2001 at 10:00:40PM -0800, David S. Miller wrote:
> No I'm not talking about phttpd nor zeus, I'm talking about the guy
> who did the hacks where he'd put the http headers + content into a
> seperate file and just sendfile() that to the client.
> 
> I forget what his hacks were named, but there certainly was a longish
> thread on this list about it about 1 year ago if memory serves.


Would that be Fabio Riccardi's X15 stuff?

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: aio

[Robert Love]

On Thu, 2001-12-20 at 01:46, Mike Castle wrote:
> On Wed, Dec 19, 2001 at 10:00:40PM -0800, David S. Miller wrote:
> > No I'm not talking about phttpd nor zeus, I'm talking about the guy
> > who did the hacks where he'd put the http headers + content into a
> > seperate file and just sendfile() that to the client.
> > 
> > I forget what his hacks were named, but there certainly was a longish
> > thread on this list about it about 1 year ago if memory serves.
> 
> Would that be Fabio Riccardi's X15 stuff?

Yes.  I was about to reply to this effect.

X15 was a userspace httpd that operated using the Tux-designed
constructs -- sendfile and such.  IIRC, Ingo actually pointed out some
things Fabio did were non-RFC (sending the static headers may of been
one of them, since the timestamp was wrong) and Fabio made a lot of
changes.  X15 seemed promising, especially since it trumpeted that Linux
"worked" without sticking things in kernel-space, but I don't remember
if we ever saw source (let alone a free license)?

	Robert Love

Re: aio

[Mike Castle]

On Thu, Dec 20, 2001 at 01:55:25AM -0500, Robert Love wrote:
> changes.  X15 seemed promising, especially since it trumpeted that Linux
> "worked" without sticking things in kernel-space, but I don't remember
> if we ever saw source (let alone a free license)?

We did, but licensing was personal use only.  It's all in the archives for
those curious.  Though following the URLs Fabio posted just got me to
login screen.

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: aio

[David S. Miller]

   From: Mike Castle <dalgoda@ix.netcom.com>
   Date: Wed, 19 Dec 2001 22:46:52 -0800
   
   Would that be Fabio Riccardi's X15 stuff?

Yes, that sounds like the one.

Re: aio

[Daniel Phillips]

On December 20, 2001 06:39 am, David S. Miller wrote:
>    From: Benjamin LaHaise <bcrl@redhat.com>
>    Date: Wed, 19 Dec 2001 22:47:17 -0500
>    An X server that doesn't have to make a syscall to find out that
>    more data has arrived?
> 
> Who really needs this kind of performance improvement?  Like anyone
> really cares if their window gets the keyboard focus or a pixel over a
> AF_UNIX socket a few nanoseconds faster.  How many people do you think
> believe they have unacceptable X performance right now and that
> select()/poll() syscalls overhead is the cause?  Please get real.

I care, I always like faster graphics.

> People who want graphics performance are not pushing their data
> through X over a filedescriptor, they are either using direct
> rendering in the app itself (ala OpenGL) or they are using shared
> memory for the bulk of the data (ala Xshm or Xv extensions).

You're probably overgeneralizing.  Actually, I run games on my server and 
display the graphics on my laptop.  It works.  I'd be happy if it was faster.

I don't see right off how AIO would make that happen though.  Ben, could you 
please enlighten me, what would be the mechanism?  Are other OSes doing X 
with AIO?

--
Daniel

Re: aio

[William Lee Irwin III]

On Thu, Dec 20, 2001 at 11:44:05AM +0100, Ingo Molnar wrote:
> we need a sane interface that covers *all* sorts of IO, not just sockets.
> I used to have exactly the same optinion as you have now, but now i'd like
> to have a common async IO interface that will cover network IO, block IO
> [or graphics IO, or whatever comes up]. We should have something saner and
> more explicit than a side-branch of fcntl() handling the socket fasync
> code.

I second this wholeheartedly. And I believe there are still more
motivations for providing asynchronous interfaces for all I/O in
the realm of assisting the userland:

(1) It would simplify the ways applications have and the kernel
	overhead of responding to user input while I/O is in progress.

(2) It would provide a more efficient way to do M:N threading than
	watchdogs and nonblocking poll/select in itimers.


Cheers,
Bill

Re: aio

[Dan Kegel]

Ingo Molnar wrote:

> it's not a fair comparison. The system was set up to not exhibit any async
> IO load. So a pure, atomic sendfile() outperformed TUX slightly, where TUX
> did something slightly more complex (and more RFC-conform as well - see
> Date: caching in X12 for example). Not something i'd call a proof - this
> simply works around the async IO interface. (which RT-signal driven,
> fasync-helped async IO interface, as phttpd has proven, is not only hard
> to program and is unrobust, it also performs *very badly*.)

Proper wrapper code can make them (almost) easy to program with.
See http://www.kegel.com/dkftpbench/doc/Poller_sigio.html for an example
of a wrapper that automatically handles the fallback to poll() on overflow.
Using this wrapper I wrote ftp clients and servers which use a thin wrapper
api that lets the user choose from select, poll, /dev/poll, kqueue/kevent, and RT signals
at runtime.

That said, I think that using the RT signal queue is just plain the
wrong way to go, and I can't wait for better approaches to make it
into the standard kernel someday.

- Dan

Re: aio

[Davide Libenzi]

On Thu, 20 Dec 2001, Dan Kegel wrote:

> Ingo Molnar wrote:
>
> > it's not a fair comparison. The system was set up to not exhibit any async
> > IO load. So a pure, atomic sendfile() outperformed TUX slightly, where TUX
> > did something slightly more complex (and more RFC-conform as well - see
> > Date: caching in X12 for example). Not something i'd call a proof - this
> > simply works around the async IO interface. (which RT-signal driven,
> > fasync-helped async IO interface, as phttpd has proven, is not only hard
> > to program and is unrobust, it also performs *very badly*.)
>
> Proper wrapper code can make them (almost) easy to program with.
> See http://www.kegel.com/dkftpbench/doc/Poller_sigio.html for an example
> of a wrapper that automatically handles the fallback to poll() on overflow.
> Using this wrapper I wrote ftp clients and servers which use a thin wrapper
> api that lets the user choose from select, poll, /dev/poll, kqueue/kevent, and RT signals
> at runtime.

Hey, you forgot /dev/epoll, the fastest one :)




- Davide

Re: aio

[Lincoln Dale]

At 08:32 AM 20/12/2001 -0800, Dan Kegel wrote:
>Proper wrapper code can make them (almost) easy to program with.
>See http://www.kegel.com/dkftpbench/doc/Poller_sigio.html for an example
>of a wrapper that automatically handles the fallback to poll() on overflow.
>Using this wrapper I wrote ftp clients and servers which use a thin wrapper
>api that lets the user choose from select, poll, /dev/poll, kqueue/kevent, 
>and RT signals
>at runtime.

SIGIO sucks in the real-world for a few reasons right now, most of them 
unrelated to 'sigio' itself:

  1. SIGIO uses signals.
         look at how signals are handled on multiprocessor (SMP) boxes.
         can you say "cache ping-pong", not to mention the locking and 
task_struct
         loop lookups?
         every signal to user-space results in 512-bytes of memory-copy 
from kernel-to-user-space.

  2. SIGIO is very heavy
         userspace only gets back one-event-per-system-call, thus you end 
up with tens-of-
         thousands of user<->kernel transitions per second eating up 
valuable cpu resources.
         there is neither: (a) aggregation of SIGIO events on a per-socket 
basis, nor
         (b) aggregating multiple SIGIO events from multiple sockets onto a 
single system call.

  3. enabling SIGIO is racy at socket-accept
         multiple system calls are required to accept a connection on a 
socket and then
         enable SIGIO on it.  packets can arrive in the meantime.
         one can workaround this with a poll() but its bad.

  4. in practical terms, SIGIO-based I/O isn't very good at expressing a 
"no POLL_OUT" signal.

  5. SIGIO is only a _notification_ mechanism.  it does NOTHING for 
zero-copy-i/o from/to-disk
    from/to-userspace from/to-network.


cheers,

lincoln.

Re: aio

[Linus Torvalds]

On Thu, 20 Dec 2001, Lincoln Dale wrote:
>
> SIGIO sucks in the real-world for a few reasons right now, most of them
> unrelated to 'sigio' itself:

Well, there _is_ one big one, which definitely is fundamentally related to
sigio itself:

sigio is an asynchronous event programming model.

And let's face it, asynchronous programming models suck. They inherently
require that you handle various race conditions etc, and have extra
locking.

Note that "asynchronous programming model" is not the same as
"asynchronous IO completion". The former implies a threaded user space,
the latter implies threaded kernel IO.

And let's face it - threading is _hard_ to get right. People just don't
think well about asynchronous events.

It's much easier to have a synchronous interface to the asynchronous IO,
ie one where you do not have to worry about events happening "at the same
time".

SIGIO just isn't very nice. It's useful for some event notification (ie if
you don't actually _do_ anything in the signal handler), but let's be
honest: it's an extremely heavy notifier. Something synchronous like
"poll" or "select" will beat it just about every time (yes, they don't
scale well, but neither does SIGIO).

		Linus

Re: aio

[Gerold Jury]

sigtimedwait and sigwaitinfo in combination with SIGIO prevents the 
asynchronous event problem and works very well for me.

Gerold

On Thursday 20 December 2001 22:59, Linus Torvalds wrote:
> It's much easier to have a synchronous interface to the asynchronous IO,
> ie one where you do not have to worry about events happening "at the same
> time".
>

Re: aio

[Lincoln Dale]

At 01:59 PM 20/12/2001 -0800, Linus Torvalds wrote:

>On Thu, 20 Dec 2001, Lincoln Dale wrote:
> >
> > SIGIO sucks in the real-world for a few reasons right now, most of them
> > unrelated to 'sigio' itself:
>
>Well, there _is_ one big one, which definitely is fundamentally related to
>sigio itself:
>
>sigio is an asynchronous event programming model.
>
>And let's face it, asynchronous programming models suck. They inherently
>require that you handle various race conditions etc, and have extra
>locking.

actually, i disagree with your assertion that "asyncronous programming 
models suck".

for MANY applications, it doesn't matter.  the equivalent to async is to do 
either:
  - thread-per-connection or process-per-connection (ala apache, sendmail, 
inetd-type services, ...)
  - a system that blocks -- handles one-connection-at-a-time

the only time async actually starts to matter is if you start to stress the 
precipitous performance characteristics associated with thousands of 
concurrent tasks in a thread/process-per-connection model.  (limited 
processor L2 cache size, multiple tasks sharing the same cache-lines 
(suboptimal cache colouring), scheduler overhead, wasted 
stack-space-per-thread/process, ..).

if you care about that level of performance, then you generally move to an 
async model.
moving to an async model doesn't have to be hard -- people generally start 
with their own pseudo scheduler and go from there.
"harder" than non-async: yes.  but "hard": no.

>SIGIO just isn't very nice. It's useful for some event notification (ie if
>you don't actually _do_ anything in the signal handler), but let's be
>honest: it's an extremely heavy notifier. Something synchronous like
>"poll" or "select" will beat it just about every time (yes, they don't
>scale well, but neither does SIGIO).

actually, my experience (circa 12 months ago) was that they were roughly equal.
poll()'s performance dropped off significantly at a few thousand FDs 
whereas sigio's latency just went up.
but it was somewhat trivial to _make_ poll() go faster by being intelligent 
about what fd's to poll.  simple logic of "if a FD didn't have anything 
active, don't poll for it on the next poll() loop" didn't increase the 
latency in servicing that FD by any noticable amount but basically triples 
the # of FDs one could handle.


cheers,

lincoln.
NB. sounds like you're making a case for the current trend in Java Virtual 
Machines insistance on "lots of processes" being a good thing. <grin, duck, 
run>

Re: aio

[Luigi Genoni]

On Wed, 19 Dec 2001, David S. Miller wrote:

>    From: Bill Huey <billh@tierra.ucsd.edu>
>    Date: Wed, 19 Dec 2001 19:07:16 -0800
>
>    And using lkml as a AIO forum is probably outside of the scope of this list
>    and group.
>
> This whole thread exists because Linus wants public general and
> technical discussion on lkml of new features to happen before he
> considers putting them into the tree, and the fact that they are not
> in the tree because he isn't seeing such enthusiastic discussions
> happening at all.
>
YES, and he is right doing so.

> I don't think AIO, because of it's non-trivial impact to the tree, is
> at all outside the scope of this list.  This is in fact the place
> where major stuff like AIO is meant to be discussed, not some special
> list where only "AIO people" hang out, of course people on that list
> will be enthusiastic about AIO!
agreed
>
> Frankly, on your other comments, I don't give a rats ass what BSD/OS
> people are doing about, nor how highly they rate, Java.  That is
> neither here nor there.  Java is going to be dead in a few years, and
> let's just agree to disagree about this particular point ok?
mmhh, java will not be death untill a lot of commecial software will use
it for graphical interfaces.
Infact it is simpler and cheaper for them to use java, and so we have to
deal with this bad future, where a dead language will be keept alive by
software houses.
That said, should we care about this? In my opinion, NO. and why we
should? when there are no good technical reasons, political reasons should
please disappear.

Luigi

Re: aio

[Henning Schmiedehausen]

"David S. Miller" <davem@redhat.com> writes:

>neither here nor there.  Java is going to be dead in a few years, and
>let's just agree to disagree about this particular point ok?

Care to point out why? Because of Sun or because of C#?

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: aio

[M. Edward (Ed) Borasky]

On Thu, 20 Dec 2001, Henning Schmiedehausen wrote:

> "David S. Miller" <davem@redhat.com> writes:
>
> >neither here nor there.  Java is going to be dead in a few years, and
> >let's just agree to disagree about this particular point ok?
>
> Care to point out why? Because of Sun or because of C#?

Because MSFT is bigger than SUNW :-) As that great American philosopher,
Damon Runyon, once said, "The race is not always to the swift, nor the
battle to the strong -- but that's the way to bet!"
--
M. Edward Borasky

znmeb@borasky-research.net
http://www.borasky-research.net

If I had 40 billion dollars for every software monopoly that sells an
unwieldy and hazardously complex development environment and is run by
an arrogant college dropout with delusions of grandeur who treats his
employees like serfs while he is acclaimed as a man of compelling
vision, I'd be a wealthy man.

Re: aio

[David S. Miller]

   From: henning@forge.intermeta.de (Henning Schmiedehausen)
   Date: Thu, 20 Dec 2001 17:26:05 +0000 (UTC)

   Care to point out why? Because of Sun or because of C#?

That's a circular question, because C# exists due to Sun's mistakes
with handling Java.  So my answer is "both".

Re: aio

[Pete Zaitcev]

> I attribute the lack of technical discussion to the least common denomiator
> culture of the Linux community and not the merits of the actual technical
> system itself. That's what linux-aio@ is for...
> 
> And using lkml as a AIO forum is probably outside of the scope of this list
> and group.

Bill, who is going to read linux-aio? There are many splinter
lists. Just about a week ago Dave Gilbert hissed at me for
not posting to linux-scsi. OK, I admit, the USB cabal made me
to subscribe to linux-usb-devel - only because the subsystem
was so out of whack that I was spending all my time trying to fix
it and dealing with broken sourceforge listserver did not make
it much worse. I can make exception for Ben, out of pure respect.
But then what? Those lists proliferate like cockroaches, every day!
I wish I could subscribe to linux-aio, linux-scsi, linux-nfs,
linux-networking, linux-afs, linux-sound, an OpenGFS list,
and "open" AFS list, linux-s390, linux-on-vaio, linux-usb-user,
linux-infi-devel, Hotplug, and perhaps more.

-- Pete

Re: aio

[Dan Kegel]

"David S. Miller" wrote:
> If AIO was so relevant+sexy we'd be having threads of discussion about
> the AIO implementation instead of threads about how relevant it is or
> is not for the general populace.  Wouldn't you concur?  :-)
> 
> The people doing Java server applets are such a small fraction of the
> Linux user community.

People writing code for NT/Win2K/WinXP are being channelled into
using AIO because that's the way to do things there (NT doesn't
really support nonblocking I/O).  Thus another valid economic
reason AIO is important is to make it easier to port code from NT.
I have received requests from NT folks for things like aio_recvfrom()
(and have passed them on to Ben), so I'm not just guessing here.

As should be clear from my c10k page, I love nonblocking I/O,
but I firmly believe that some form of AIO is vital.

- Dan

Re: aio

[Gerold Jury]

On Thursday 20 December 2001 17:16, Dan Kegel wrote:
> "David S. Miller" wrote:
> > If AIO was so relevant+sexy we'd be having threads of discussion about
> > the AIO implementation instead of threads about how relevant it is or
> > is not for the general populace.  Wouldn't you concur?  :-)
> >
> > The people doing Java server applets are such a small fraction of the
> > Linux user community.
>
> reason AIO is important is to make it easier to port code from NT.
>
> but I firmly believe that some form of AIO is vital.
>
> - Dan

>From the aio-0.3.1/README
section Current State

  IPv4 TCP and UDP (rx only) sockets.

It is simply too early for sexy discussions. For me, the most appealing part 
of AIO is the socket handling. It seems a little bit broken in the current 
glibc emulation/implementation.
Recv and send operations are ordered when used on the same socket handle.
Thus a recv must be finished before a subsequent send will happen.
Good idea for files, bad for sockets.

SGI's implementation kaio, which works perfect for me, is widespread ignored 
and sufferes from the unreserved syscall problem like Ben's aio. I am sure 
there is a reason for ignoring SGI-kaio, i just do not remember.

With the current state of the different implementations it is difficult to 
have sex about or use them.
But i would really like tooooooooooooooooooo.

Gerold

-
The one-sig-perfd patch did not get much attention either.
No one seems to use sockets these days.

Re: aio

[Ingo Molnar]

On Fri, 21 Dec 2001, Gerold Jury wrote:

> It is simply too early for sexy discussions. For me, the most
> appealing part of AIO is the socket handling. It seems a little bit
> broken in the current glibc emulation/implementation. Recv and send
> operations are ordered when used on the same socket handle. Thus a
> recv must be finished before a subsequent send will happen. Good idea
> for files, bad for sockets.

is this a fundamental limitation expressed in the interface, or just an
implementational limitation? On sockets this is indeed a big problem, HTTP
pipelining wants completely separate receive/send queues.

	Ingo

Re: aio

[Gerold Jury]

On Friday 21 December 2001 14:48, Ingo Molnar wrote:
> On Fri, 21 Dec 2001, Gerold Jury wrote:
> > It is simply too early for sexy discussions. For me, the most
> > appealing part of AIO is the socket handling. It seems a little bit
> > broken in the current glibc emulation/implementation. Recv and send
> > operations are ordered when used on the same socket handle. Thus a
> > recv must be finished before a subsequent send will happen. Good idea
> > for files, bad for sockets.
>
> is this a fundamental limitation expressed in the interface, or just an
> implementational limitation? On sockets this is indeed a big problem, HTTP
> pipelining wants completely separate receive/send queues.
>
> 	Ingo
>

That is a very good question.

The Single UNIX ® Specification, Version 2 has the following to say.

If _POSIX_SYNCHRONIZED_IO is defined and synchronised I/O is enabled on the 
file associated with aiocbp->aio_fildes, the behaviour of this function is 
according to the definitions of synchronised I/O data integrity completion 
and synchronised I/O file integrity completion.

Maybe a was a little bit too fast in blaming glibc. I will go and look for 
more documentation about disabling synchronised I/O on a socket.

Dup()licating the socket handle is an easy workaround, but now i am 
convinced, a little bit man page digging will be lots of fun.

I hope the efforts of Benjamin LaHaise receive more attention and as soon as 
i know more about disabling synchronised I/O on sockets i will send an other 
email.

Gerold

--
I love AIO

Re: aio

[Gerold Jury]

> On Friday 21 December 2001 14:48, Ingo Molnar wrote:
> > is this a fundamental limitation expressed in the interface, or just an
> > implementational limitation? On sockets this is indeed a big problem,
> > HTTP pipelining wants completely separate receive/send queues.
> >
> > 	Ingo
>

I got the _POSIX_SYNCHRONIZED_IO completely wrong.
It has nothing to do with the ordering of the aio read/write requests.

Aio_read and aio_write work with absolute positions inside the file size.
The order of requests is unspecified in SUSV V2.
SUSV V2 does neither prevent nor force the desired behaviour.

It seems like it is up to the implementation how to deal with the request 
order.
As mentioned earlier, SGI-kaio does the right thing with the same interface.

I want to add, that a combination of sigwaitinfo / sigtimedwait and aio is a 
very efficient way to deal with sockets. The accept may be handled with real 
time signals as well by using fcntl F_SETSIG and F_SETFL FASYNC.


Gerold

Re: aio

[Henning Schmiedehausen]

"David S. Miller" <davem@redhat.com> writes:

>The people doing Java server applets are such a small fraction of the
>Linux user community.

The people doing Java Servlets are maybe a small fraction of the Linux
Kernel Hackers community. Not the user community. They simply do not
show up here because they don't care for Linux 2.5.0-rc1-prepatched.

Short head count: Who here is also a regular reader on Apache Jakarta
lists?

Kernel hacking and Java most of the times doesn't mix. And Java folks
are completly ambivalent to their OS: If Linux doesn't deliver,
well. Windows, Solaris and BSD do. That's what Java is all about. 

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: aio

[David S. Miller]

   From: Cameron Simpson <cs@zip.com.au>
   Date: Thu, 20 Dec 2001 13:37:05 +1100
   
   (Though an attitude like yours is a core reason Java is
   spreading as slowly as it is - much like Linux desktops...)
   
It's actually Sun's fault more than anyone else's.

   However, heavily threaded apps regardless of language are hardly likely
   to disappear; threads are the natural way to write many many things. And
   if the kernel implements threads as on Linux, then the scheduler will
   become much more important to good performance.

We are not talking about the scheduler, we are talking about
AIO.

Re: aio

[Cameron Simpson]

On Wed, Dec 19, 2001 at 06:47:18PM -0800, David S. Miller <davem@redhat.com> wrote:
|    (Though an attitude like yours is a core reason Java is
|    spreading as slowly as it is - much like Linux desktops...)
| It's actually Sun's fault more than anyone else's.

Debatable. But fortunately off topic.

|    However, heavily threaded apps regardless of language are hardly likely
|    to disappear; threads are the natural way to write many many things. And
|    if the kernel implements threads as on Linux, then the scheduler will
|    become much more important to good performance.
| We are not talking about the scheduler, we are talking about
| AIO.

It was in the same thread - I must have ignored the detail switch. Ignore
me in turn. But while I'm here, tell me why async I/O is important
to Java and not to anything else, which still seems the thrust of your
remarks.
-- 
Cameron Simpson, DoD#743        cs@zip.com.au    http://www.zip.com.au/~cs/

Always code as if the guy who ends up maintaining your code will be a violent
psychopath who knows where you live.
	- Martin Golding, DoD #0236, martin@plaza.ds.adp.com

Re: aio

[David S. Miller]

   From: Cameron Simpson <cs@zip.com.au>
   Date: Thu, 20 Dec 2001 13:52:21 +1100
   
   tell me why async I/O is important
   to Java and not to anything else, which still seems the thrust of
   your remarks.

Not precisely my thrust, which is that AIO is not important to any
significant population of Linux users, it is "nook and cranny" in
scope.  And that those "nook and cranny" folks who really find it
important can get paid implementation+support of AIO.

Re: aio

[Linus Torvalds]

On Wed, 19 Dec 2001, David S. Miller wrote:
>
> Not precisely my thrust, which is that AIO is not important to any
> significant population of Linux users, it is "nook and cranny" in
> scope.  And that those "nook and cranny" folks who really find it
> important can get paid implementation+support of AIO.

I disagree - we can probably make the aio by Ben quite important. Done
right, it becomes a very natural way of doing event handling, and it could
very well be rather useful for many things that use select loops right
now.

So I actually like the thing as it stands now. What I don't like is how
it's been handled, with people inside Oracle etc working with it, but
_not_ people on the kernel mailing list. I don't worry about the code
nearly as much as I worry about people starting to clique together.

		Linus

Re: aio

[David S. Miller]

   From: Linus Torvalds <torvalds@transmeta.com>
   Date: Wed, 19 Dec 2001 21:47:18 -0800 (PST)
   
   it could very well be rather useful for many things that use select
   loops right now.

Then let us agree to disagree. :-) I think it's potential advantages,
and how many things really "require it" for better performance, is
being blown out of proportion.

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 09:57:30PM -0800, David S. Miller wrote:
> Then let us agree to disagree. :-) I think it's potential advantages,
> and how many things really "require it" for better performance, is
> being blown out of proportion.

Show me how to make a single process server that can handle 100000 or more 
open tcp sockets that doesn't collapse under load.  I can do it with aio; 
can you do it without?

		-ben
-- 
Fish.

Re: aio

[David S. Miller]

   From: Benjamin LaHaise <bcrl@redhat.com>
   Date: Thu, 20 Dec 2001 00:59:28 -0500
   
   Show me how to make a single process server that can handle 100000 or more 
   open tcp sockets that doesn't collapse under load.  I can do it with aio; 
   can you do it without?

Why are you limiting me to a single process? :-)  Can I have at least
1 per cpu possibly? :-)))

Re: aio

[Benjamin LaHaise]

On Wed, Dec 19, 2001 at 10:02:47PM -0800, David S. Miller wrote:
> Why are you limiting me to a single process? :-)  Can I have at least
> 1 per cpu possibly? :-)))

1 process.  1 cpu machine.  1 gige card.  As much ram as you want.  No 
syscalls.  Must exhibit a load curve similar to:

	y
	|  ...............
	| .
	|.
	+----------------x

Where x == requests per second sent to the machine and y is the number 
of resposes per second sent out of the machine.  Hint: read the phttpd 
and /dev/poll papers for an idea of the breakdown that happens for larger 
values of x (make the cpu slower to cause the interesting points to move 
lower).  For a third dimension to the graph, make the number of total 
connections the z axis.

		-ben
-- 
Fish.

Re: aio

[Linus Torvalds]

Could we get back on track, and possibly discuss the patches themselves,
ok? We want _constructive_ criticism of the interfaces.

I think it's clear that many people do want to have aio support. At least
as far as I'm concerned, that's not the reason I want to have public
discussion. I want to make sure that the interfaces are good for aio
users, and that the design isn't stupid.

If somebody can point to a better way of doing aio, and giving good
arguments for that, more power to him. But let's not go down the path of
"_I_ don't like aio, so _you_ must be stupid".

		Linus

Re: aio

[Suparna Bhattacharya]

On Wed, Dec 19, 2001 at 10:09:40PM -0800, Linus Torvalds wrote:
> Could we get back on track, and possibly discuss the patches themselves,
> ok? We want _constructive_ criticism of the interfaces.
> 
> I think it's clear that many people do want to have aio support. At least

Yes, to add to the lot, even though you probably don't need any more proof :)
we have at least 3 different products requiring this. Both for scalable 
communications aio (large no. of connections) as well as file/disk aio.  
In fact, one of the things that I've been worried about once 2.5 opened up
and the bio changes started flowing in (much as I was delighted to see
Jens' stuff finally getting integrated), was whether this would mean a 
longer timeframe before we can hope to see aio in a distribution (which is 
the question which I have to respond to our product groups about). 

> as far as I'm concerned, that's not the reason I want to have public
> discussion. I want to make sure that the interfaces are good for aio
> users, and that the design isn't stupid.
> 

My feeling about this is that we shouldn't necessarily need to
have the entire aio code in perfect shape and bring it in all in one shot.
The thing I like about the design is that it is quite possible to split
this up into smaller core patches and bring them in slowly. And I agree
with Ingo that we should be able to start stabilizing the basic internal
mechanisms or foundations on which aio is built before we freeze the 
external interfaces. In fact, these two things could happen parallely.
So I was hoping that an evolutionary approach would work. In the current 
design, the aio path is handled separately from the normal i/o paths, so the 
impact on existing interfaces is less. It mainly affects aio users. So it 
ought to be possible to integrate it in a way that doesn't hurt regular
operations, and it should be easier to change it once it is in without
breaking too many things.

Existing aio users on other platforms that I have come across seem to use 
either POSIX aio (for file /disk aio) or completion port style interfaces 
(mainly for communications aio), both of which seem to be possible with 
Ben's implementation. One has to explicitly associate each i/o with the 
completion queue (ctx), rather than associate an fd as a whole with it so 
that all completion events on that fd come to the ctx. That should be OK. 
Besides with async poll support we can have per-file readiness notification 
as well. I was hoping for the async poll piece being available early to 
exercise the top half or event handling side of aio, so we have scalable 
select/poll support, so was focussing on that part to start with.

Your point about some critical discussion of the interfaces and the design 
is well-taken. We have had a few discussions on the aio mailing and more on 
irc about some aspects, but not quite a thorough out and out analysis of 
pros and cons of the whole design. I just started writing some points here, 
but then realized that it is going to take much longer, so decided to do
that while working with Ben on the documentation, and discuss more after 
that.

Regards
Suparna

Re: aio

[David S. Miller]

   From: Benjamin LaHaise <bcrl@redhat.com>
   Date: Thu, 20 Dec 2001 01:07:42 -0500
   
   1 process.  1 cpu machine.  1 gige card.  As much ram as you want.  No 
   syscalls.  Must exhibit a load curve similar to:
   
   	y
   	|  ...............
   	| .
   	|.
   	+----------------x
   
   Where x == requests per second sent to the machine and y is the number 
   of resposes per second sent out of the machine.  Hint: read the phttpd 
   and /dev/poll papers for an idea of the breakdown that happens for larger 
   values of x (make the cpu slower to cause the interesting points to move 
   lower).  For a third dimension to the graph, make the number of total 
   connections the z axis.

Ok, TUX can do it.  Now list for me some server that really matters
other than web and ftp?  If you say databases, then I agree with you
but I will also reiterate how the people who need that level of
database performance is "nook and cranny".

I think there is nothing wrong with doing a TUX module for situations
where 1) the server is important for enough people and 2) scaling to
the levels you are talking about is a real issue for that service.

Re: aio

[Linus Torvalds]

On Wed, 19 Dec 2001, David S. Miller wrote:
>
> Ok, TUX can do it.  Now list for me some server that really matters
> other than web and ftp?

Now now, that's unfair. We should be able to do it in user space.

I think the question you _should_ be lobbying at Ben and the other aio
people is how the aio stuff could do zero-copy from disk cache to the
network, ie do the things that Tux does internally where it does
nonblocking reads from disk ad then sends them out non-blocking to the
network without havign to copy the data _or_ have to use extremely
expensive TLB mapping tricks to get at it..

Ie tie the "sendfile" and "aio" threads together, and ask Ben if we can do
aio-sendfile and have thousands of asynchronous sendfiles going on at the
same time, like Tux can do. And if not, then why not? Missing or bad
interfaces?

Ben? Doing user-space IO is all well and good, but that extra copy and TLB
stuff kills you. Tell us how to do it ;)

		Linus

Re: aio

[Ingo Molnar]

On Wed, 19 Dec 2001, Linus Torvalds wrote:

> I think the question you _should_ be lobbying at Ben and the other aio
> people is how the aio stuff could do zero-copy from disk cache to the
> network, ie do the things that Tux does internally where it does
> nonblocking reads from disk ad then sends them out non-blocking to the
> network without havign to copy the data _or_ have to use extremely
> expensive TLB mapping tricks to get at it..

months ago i already offered Ben to port TUX to the aio interfaces once
they are available in the kernel. Unfortunately right now i cant afford
maintaining two separate TUX trees - so it's a chicken and egg thing in
this context.

But once aio is available, i *will* do it, because one of Ben's goals
fully state-machine-driven async block IO, which i'd like to use (and
test, and finetune, and improve) very much. (right now TUX does async
block IO via helper kernel threads. Async net-io is fully IRQ-driven.)
I'd also like to prove that our aio interfaces are capable.

there are two possibilities i can think of:

1) lets get Ben's patch in but do *not* export the syscalls, yet.

2) find some nice way of doing 'experimental syscalls', which are not
   guaranteed to stay that way. (Perhaps this is a naive proposition,
   often there is nothing more permanent than temporary solutions.)
   Something like reserving 'temporary' syscalls at the end of the syscall
   space, which would be frequently moved/removed/renamed just to keep
   folks from relying on it. No interface is guaranteed. Perhaps some
   technical solution can be find to make these syscalls truly temporary.

i'm sure people will get excited about (ie. use) aio once it's in the
kernel. Ben is very good at coding, perhaps not as good at PR, but should
such level of PR really be a natural part of Linux development?

> Ie tie the "sendfile" and "aio" threads together, and ask Ben if we
> can do aio-sendfile and have thousands of asynchronous sendfiles going
> on at the same time, like Tux can do. And if not, then why not?
> Missing or bad interfaces?

i'd love to find out. *If* it's guaranteed that some sort of sane aio will
always be available from the point on it's introduced into the kernel then
i'll switch TUX to it. (it will change TUX upside down, this is why i
cannot maintain two separate TUX trees.) TUX doesnt need stable
interfaces. While TUX might not be as important, usage-wise, it's
certainly a good playing ground for such things.

	Ingo

Re: aio

[Robert Love]

On Thu, 2001-12-20 at 05:18, Ingo Molnar wrote:

> there are two possibilities i can think of:
> 
> 1) lets get Ben's patch in but do *not* export the syscalls, yet.

This is an excellent way to give aio the testing and exposure Linus
wants without getting into the commitment / syscall mess.

Stick aio in the kernel, play with it via Tux, etc.  The really
interested can add temporary syscalls.  aio (which I like, btw) will get
testing and in time, once proven, we can add the syscalls.

Comments?

	Robert Love

Re: aio

[Cameron Simpson]

On Thu, Dec 20, 2001 at 01:20:55PM -0500, Robert Love <rml@tech9.net> wrote:
| On Thu, 2001-12-20 at 05:18, Ingo Molnar wrote:
| > there are two possibilities i can think of:
| > 1) lets get Ben's patch in but do *not* export the syscalls, yet.
| 
| This is an excellent way to give aio the testing and exposure Linus
| wants without getting into the commitment / syscall mess.
| Stick aio in the kernel, play with it via Tux, etc.  The really
| interested can add temporary syscalls.  aio (which I like, btw) will get
| testing and in time, once proven, we can add the syscalls.
| Comments?

Only that it would be hard for user space people to try it - does Ben's
patch (with hypothetical syscalls) present the POSIX async interfaces out
of the box? If not, testing with in-kernel things is sufficient. But
if it does then it becomes more reasonable to transiently define some
syscall numbers (high up, in some defined as "testing and like shifting
sands" range) so user space can test the interface.

Thought: is there a meta-syscall in the kernel API for calling other syscalls?
You could have such a beast taking negative numbers for experimental calls...
-- 
Cameron Simpson, DoD#743        cs@zip.com.au    http://www.zip.com.au/~cs/

Sometimes the only solution is to find a new problem.

Re: aio

[Benjamin LaHaise]

On Fri, Dec 21, 2001 at 09:30:27AM +1100, Cameron Simpson wrote:
> Only that it would be hard for user space people to try it - does Ben's
> patch (with hypothetical syscalls) present the POSIX async interfaces out
> of the box?

No.  POSIX aio does not have any concept of a completion queue.  Completion 
in POSIX aio comes via a thread callback, signal delivery or polling, all 
of which are horrendously inefficient.

> If not, testing with in-kernel things is sufficient. But
> if it does then it becomes more reasonable to transiently define some
> syscall numbers (high up, in some defined as "testing and like shifting
> sands" range) so user space can test the interface.

Maybe.  The unfortunate aspect to this is that you can't tell if a number 
matches the name you expect it to be, and invariably people end up running 
the wrong code on the wrong kernel.  Or vendors start shipping patches to 
enable these new syscalls....

> Thought: is there a meta-syscall in the kernel API for calling other 
> syscalls?  You could have such a beast taking negative numbers for 
> experimental calls...

I'm working on something.  Stay tuned.

		-ben
-- 
Fish.

Re: aio

[Pete Zaitcev]

>[...]
> However, heavily threaded apps regardless of language are hardly likely
> to disappear; threads are the natural way to write many many things. And
> if the kernel implements threads as on Linux, then the scheduler will
> become much more important to good performance.

Cameron seems to be arguing with DaveM, but subconsciously he
only supports DaveM's point about AIO: Java cannot make use
of AIO, so that's one (large or small, important or unimportant)
group of applications down from the count.

Just trying to keep on topic :)

-- Pete

Re: aio

[Cameron Simpson]

On Thu, Dec 20, 2001 at 12:07:21AM -0500, Pete Zaitcev <zaitcev@redhat.com> wrote:
| >[...]
| > However, heavily threaded apps regardless of language are hardly likely
| > to disappear; threads are the natural way to write many many things. And
| > if the kernel implements threads as on Linux, then the scheduler will
| > become much more important to good performance.
| 
| Cameron seems to be arguing with DaveM,

About the wrong things, but no matter.

| but subconsciously he
| only supports DaveM's point about AIO: Java cannot make use
| of AIO, so that's one (large or small, important or unimportant)
| group of applications down from the count.

You're sure? Java _authors_ can't make use of it, but Java _implementors_
probably have good reason to want it ...

| Just trying to keep on topic :)

Whatever for?
--
Cameron Simpson, DoD#743        cs@zip.com.au    http://www.zip.com.au/~cs/

Reaching consensus in a group often is confused with finding the right
answer.	- Norman Maier

Re: aio

[Alan Cox]

> Precisely, in fact.  Anyone who can say that Java is going to be
> relevant in a few years time, with a straight face, is only kidding
> themselves.

Oh it'll be very relevant. Its leaking into all sorts of embedded uses, from
Digital TV to smartcards. Its still useless for serious high end work an
likely to stay so.

> Java is not something to justify a new kernel feature, that is for
> certain.

There we agree. Things like the current asynch/thread mess in java are
partly poor design of language and greatly stupid design of JVM.

Re: aio

[Bill Huey]

On Fri, Dec 21, 2001 at 05:28:36PM +0000, Alan Cox wrote:
> > Precisely, in fact.  Anyone who can say that Java is going to be
> > relevant in a few years time, with a straight face, is only kidding
> > themselves.
> 
> Oh it'll be very relevant. Its leaking into all sorts of embedded uses, from
> Digital TV to smartcards. Its still useless for serious high end work an
> likely to stay so.
> 
> > Java is not something to justify a new kernel feature, that is for
> > certain.
> 
> There we agree. Things like the current asynch/thread mess in java are
> partly poor design of language and greatly stupid design of JVM.

It's not the fault of the JVM runtime nor the the language per se since
both are excellent. The blame should instead be placed on the political
process within Sun, which has created a lag in getting a decent IO event
model/system available in the form of an API.

This newer system is suppose to be able to scale to tens of thousands of
FDs and be able to handle heavy duty server side stuff in a more graceful
manner. It's a reasonable system from what I saw, but the implementation
of it is highly OS dependent and will be subject to those environmental
constraints. Couple this and the HotSpot compiler (supposeablly competitive
with gcc's -O3 from benchmarks) and it should be high useable for a broad
range of of server side work when intelligently engineered.  

bill

Re: aio

[Dan Kegel]

Bill Huey wrote:
> > There we agree. Things like the current asynch/thread mess in java are
> > partly poor design of language and greatly stupid design of JVM.
> 
> It's not the fault of the JVM runtime nor the the language per se since
> both are excellent. The blame should instead be placed on the political
> process within Sun, which has created a lag in getting a decent IO event
> model/system available in the form of an API.
> 
> This newer system is suppose to be able to scale to tens of thousands of
> FDs and be able to handle heavy duty server side stuff in a more graceful
> manner. It's a reasonable system from what I saw, but the implementation
> of it is highly OS dependent and will be subject to those environmental
> constraints. Couple this and the HotSpot compiler (supposeablly competitive
> with gcc's -O3 from benchmarks) and it should be high useable for a broad
> range of of server side work when intelligently engineered.

I served on JSR-51, the expert group that helped design the new I/O
model.  (The design was Sun's, but we had quite a bit of input.)
For network I/O, there's a Selector object which is essentially
a nice OO wrapper around the /dev/poll or kqueue/kevent abstraction.
Selector does have a distinctly Unixy feel to it, but it can probably
be implemented well on top of any reasonable OS; I'm quite sure
it can be expressed fairly well in terms of Windows NT's asych I/O
or Linux's rt signal stuff.  

(I suspect the initial Linux implementations will just use poll(),
but that's something the Blackdown team can fix.  And heck, it
ought to be easy to implement it on top of all the nifty poll
replacements and choose between them at jvm startup time without
any noticable overhead.)

- Dan

p.s. Davide, I didn't forget /dev/epoll, I just haven't had time to
post Poller_devepoll yet!

Re: aio

[Davide Libenzi]

On Sat, 22 Dec 2001, Dan Kegel wrote:

> p.s. Davide, I didn't forget /dev/epoll, I just haven't had time to
> post Poller_devepoll yet!

Yep, i just started feeling angry about this :)




- Davide