Re: [take22 0/4] kevent: Generic event handling mechanism.

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Jonathan Lemon]

In article <local.mail.linux-kernel/20061103084240.GB1184@2ka.mipt.ru>,
Evgeniy Polyakov  <johnpol@2ka.mipt.ru> wrote:
>On Thu, Nov 02, 2006 at 11:40:43AM -0800, Nate Diller
>(nate.diller@gmail.com) wrote:
>> Are you saying that the *only* reason we choose not to be
>> source-compatible with BSD is the 32 bit userland on 64 bit arch
>> problem?  I've followed every thread that gmail 'kqueue' search
>
>I.e. do you want that generic event handling mechanism would not work on
>x86_64? I doubt you do.
>
>> returns, which thread are you referring to?  Nicholas Miell, in "The
>> Proposed Linux kevent API" thread, seems to think that there are no
>> advantages over kqueue to justify the incompatibility, an argument you
>> made no effort to refute.  I've also read the Kevent wiki at
>> linux-net.osdl.org, but it too is lacking in any direct comparisons
>> (even theoretical, let alone benchmarks) of the flexibility,
>> performance, etc. between the two.
>> 
>> I'm not arguing that you've done a bad design, I'm asking you to brag
>> about the things you improved on vs. kqueue.  Your emphasis on
>> unifying all the different event types into one interface is really
>> cool, fill me in on why that can't be effectively done with the kqueue
>> compatability and I also will advocate for kevent inclusion.
>
>kqueue just can not be used as is in Linux (_maybe_ *bsd has different
>types, not those which I found in /usr/include in my FC5 and Debian
>distro). It will not work on x86_64 for example. Some kind of a pointer
>or unsigned long in structures which are transferred between kernelspace
>and userspace is so much questionable, than it is much better even do
>not see there... (if I would not have so political correctness, I would
>describe it in a much different words actually).
>So, kqueue API and structures can not be usd in Linux.

Let me be a little blunt here: that is just so much bullshit.

Yes, I understand the problem that 32-bit userspace on a 64-bit kernel has.
Mea culpa - I didn't forsee this years ago, and none of my many reviewers
caught it either.  It was designed for 32/32 and 64/64, not 32/64.

However, this is trivially fixed by adding a union to the structure, as
pointed out earlier on this list.  Code would still be source compatible
with any kqueue apps, which is what counts.  Even NetBSD and FreeBSD have
differing definitions of the kq constants, and nobody notices.

I really have no stake in this matter, so if you want to go invent a 
better mousetrap, more power to you.  But don't claim that "kqueue can
not be used on Linux"; this just makes you look foolish - I have code
running on x86_64 that trivially disproves your statement.
-- 
Jonathan

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

On Fri 2006-11-03 10:30:12, Jonathan Lemon wrote:
> In article <local.mail.linux-kernel/20061103084240.GB1184@2ka.mipt.ru>,
> Evgeniy Polyakov  <johnpol@2ka.mipt.ru> wrote:
> >On Thu, Nov 02, 2006 at 11:40:43AM -0800, Nate Diller
> >kqueue just can not be used as is in Linux (_maybe_ *bsd has different
> >types, not those which I found in /usr/include in my FC5 and Debian
> >distro). It will not work on x86_64 for example. Some kind of a pointer
> >or unsigned long in structures which are transferred between kernelspace
> >and userspace is so much questionable, than it is much better even do
> >not see there... (if I would not have so political correctness, I would
> >describe it in a much different words actually).
> >So, kqueue API and structures can not be usd in Linux.
> 
> Let me be a little blunt here: that is just so much bullshit.
> 
> Yes, I understand the problem that 32-bit userspace on a 64-bit kernel has.
> Mea culpa - I didn't forsee this years ago, and none of my many reviewers
> caught it either.  It was designed for 32/32 and 64/64, not 32/64.
> 
> However, this is trivially fixed by adding a union to the structure, as
> pointed out earlier on this list.  Code would still be source compatible
> with any kqueue apps, which is what counts.  Even NetBSD and FreeBSD have
> differing definitions of the kq constants, and nobody notices.

It has been show in this thread that kevent is too different to kqueue
as is... but what are the advantages of kevent? Perhaps we should use
kqueue on Linux, too (even if it means one more rewrite for you...?)

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Sun, Nov 05, 2006 at 09:47:41PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> It has been show in this thread that kevent is too different to kqueue
> as is... but what are the advantages of kevent? Perhaps we should use
> kqueue on Linux, too (even if it means one more rewrite for you...?)

Should we use *BSD VMM system when we have superiour Linux one?

P.S. Do not drop Cc: list.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

On Mon 2006-11-06 13:13:29, Evgeniy Polyakov wrote:
> On Sun, Nov 05, 2006 at 09:47:41PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> > It has been show in this thread that kevent is too different to kqueue
> > as is... but what are the advantages of kevent? Perhaps we should use
> > kqueue on Linux, too (even if it means one more rewrite for you...?)
> 
> Should we use *BSD VMM system when we have superiour Linux one?

Very different question; VMM system is not something that has userland
API.

Can you explain why kevent is better than kqueue?

> P.S. Do not drop Cc: list.

It was not me who dropped cc list.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Mon, Nov 06, 2006 at 11:16:33AM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> On Mon 2006-11-06 13:13:29, Evgeniy Polyakov wrote:
> > On Sun, Nov 05, 2006 at 09:47:41PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> > > It has been show in this thread that kevent is too different to kqueue
> > > as is... but what are the advantages of kevent? Perhaps we should use
> > > kqueue on Linux, too (even if it means one more rewrite for you...?)
> > 
> > Should we use *BSD VMM system when we have superiour Linux one?
> 
> Very different question; VMM system is not something that has userland
> API.

So what? We still create new things, which work better than old ones
even if it requires 'to reinvent the wheel'.

It was shown too many times already why kqueue api can not be used in
Linux.

Btw, if you want someone to rewrite something, you can start with mmaped
based malloc for example. Why don't you want to do it - although API is
the same, but underlying logic is different.

> Can you explain why kevent is better than kqueue?

According to my tests kevent is noticebly faster.
It is already too big flag that old system should not be used.
And half of my previous mail to you shows why kevent is better/different
from kqueue.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

> > Can you explain why kevent is better than kqueue?
> 
> According to my tests kevent is noticebly faster.
> It is already too big flag that old system should not be used.
> And half of my previous mail to you shows why kevent is better/different
> from kqueue.

You shown why it is _different_. How much faster is "noticebly
faster"?

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Mon, Nov 06, 2006 at 01:58:18PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> Hi!
> 
> > > Can you explain why kevent is better than kqueue?
> > 
> > According to my tests kevent is noticebly faster.
> > It is already too big flag that old system should not be used.
> > And half of my previous mail to you shows why kevent is better/different
> > from kqueue.
> 
> You shown why it is _different_. How much faster is "noticebly
> faster"?

It is different on purpose, don't you think?
If I will put all benchmark results in all mails, no one will read even
half of it.

Here is conlusion section on kevent homepage where FreeBSD kqueue is
compared with kevent (different NIC than recent Linux kevent tests, 
but there are links to old kevent benchamrks there):

"After various sysctls have been changed (sysctl -a output is available
here) things become slightly better (btw, default FreeBSD installation
does not allow such tests at all due to default network parameters), but
number of "connection reset" errors is still very high.
FreeBSD drops too many connections due to either misconfiguration or
lack of resources.

According to FreeBSD and Linux comparison, in Linux number of connection
errors is much smaller than in FreeBSD with comparable or bigger
requests rate."

Briefly saying, FreeBSD kqueue behaves like Linux epoll, sometimes
better (with small request rate), sometimes worse (with 3k simultaneous
connections rate), and the latter was shown to behave worse than kevent.


Actually, Pavel, I do not understand your point. Why do you want to use
*BSD subsystem even if it is impossible to have the same API? You want
me to rewrite kevent so it would look like kqueue, but you did not know
how it looks like, likely you did not know it's API (it uses switches of
commands which are too much frowned upon in Linux kernel), you did not
know what features kevent provides and what is present and what does not
exist in kqueue.
So please point me to the magic Bodhi way which can enlighten me to think 
that completely different system, which works with completely different
OS with completely different API, ABI and kernel internals, should be 
ported to Linux instead of creation new and superior system?
When I become as luminous as you I will go and create new sendfile()
system call which will have the same parameters as BSD. Or not, I will ask
you to do it (actually not, why should we create something new, when
there is BSD system which already has everything we want?).

> 								Pavel

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

Kevent pipe benchmark kevent_pipe kernel kevent part:

epoll (edge-triggered):   248408 events/sec
kevent (edge-triggered):  269282 events/sec
Busy reading loop:        269519 events/sec

Kevent is definitely a winner with extremely small overhead.

I will add kevent_pipe into next kevent release which will be available
soon.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Tue, Nov 07, 2006 at 12:18:43PM +0300, Evgeniy Polyakov (johnpol@2ka.mipt.ru) wrote:
> On Mon, Nov 06, 2006 at 10:17:37PM +0100, Eric Dumazet (dada1@cosmosbay.com) wrote:
> > Evgeniy Polyakov a écrit :
> > >
> > >If there would exist sockets support, then I could patch it to work with
> > >kevents.
> > >
> > 
> > OK I post here my last version of epoll_bench.
> 
> My results with AF_INET are inlined.
> Hardware 2.4 Ghz Xeon w(1 CPU, HT enabled) with 1 GB of RAM.
> 
> [root@pcix event]# ./epoll_bench -n 2000 -i
> 2000 handles setup
> 49758 evts/sec 1.56177 samples per call
> 38999 evts/sec 95 ctxt/sec 2.77247 samples per call
> 54042 evts/sec 130 ctxt/sec 4.19909 samples per call
> 60155 evts/sec 188 ctxt/sec 5.38024 samples per call
> 59588 evts/sec 178 ctxt/sec 6.38112 samples per call
> 60023 evts/sec 188 ctxt/sec 7.19564 samples per call
> 59694 evts/sec 186 ctxt/sec 7.93067 samples per call
> 60182 evts/sec 190 ctxt/sec 8.52397 samples per call
> 59750 evts/sec 182 ctxt/sec 9.08015 samples per call
> 60158 evts/sec 192 ctxt/sec 9.53548 samples per call
> 59739 evts/sec 188 ctxt/sec 9.97013 samples per call
> 60054 evts/sec 216 ctxt/sec 10.32 samples per call
> 59820 evts/sec 206 ctxt/sec 10.6641 samples per call
> 60095 evts/sec 218 ctxt/sec 10.9289 samples per call
> 59376 evts/sec 158 ctxt/sec 11.3231 samples per call
> Avg: 57428 evts/sec
> [root@pcix event]# ./kevent_bench -n2000 -i
> 2000 handles setup
> 57960 evts/sec 0.276702 samples per call
> 59802 evts/sec 75 ctxt/sec 0.462737 samples per call
> 59864 evts/sec 71 ctxt/sec 0.623457 samples per call
> 59651 evts/sec 72 ctxt/sec 0.721579 samples per call
> 59504 evts/sec 84 ctxt/sec 0.804311 samples per call
> 61019 evts/sec 72 ctxt/sec 0.904817 samples per call
> 59846 evts/sec 72 ctxt/sec 0.949439 samples per call
> 60550 evts/sec 74 ctxt/sec 1.00416 samples per call
> 59421 evts/sec 66 ctxt/sec 1.04133 samples per call
> 60334 evts/sec 75 ctxt/sec 1.06845 samples per call
> 60000 evts/sec 67 ctxt/sec 1.09594 samples per call
> 59429 evts/sec 74 ctxt/sec 1.11404 samples per call
> 60508 evts/sec 77 ctxt/sec 1.14482 samples per call
> 59530 evts/sec 66 ctxt/sec 1.15454 samples per call
> 59506 evts/sec 73 ctxt/sec 1.17937 samples per call
> Avg: 59794 evts/sec
> [root@pcix event]# ./kevent_bench -n2000 -i -f
> 2000 handles setup
> 82893 evts/sec
> 88624 evts/sec 390 ctxt/sec
> 88751 evts/sec 475 ctxt/sec
> 88784 evts/sec 488 ctxt/sec
> 88918 evts/sec 458 ctxt/sec
> 88866 evts/sec 504 ctxt/sec
> 88950 evts/sec 458 ctxt/sec
> 88883 evts/sec 472 ctxt/sec
> 88915 evts/sec 404 ctxt/sec
> 88836 evts/sec 368 ctxt/sec
> 89065 evts/sec 442 ctxt/sec
> 88859 evts/sec 398 ctxt/sec
> 89070 evts/sec 446 ctxt/sec
> 88809 evts/sec 428 ctxt/sec
> 89012 evts/sec 542 ctxt/sec
> Avg: 88482 evts/sec
> 
> epoll: 57428
> kevent: 59794
> max: 88482
> 
> BUT!
> Kevent does not support analogue for EPOLLET, i.e. the case when the
> same event is used, instead kevent must modify existing one (i.e. behave
> exactly like epoll without EPOLLET), so modified epoll_bench to work
> without EPOLLET like kevent.
> epoll with EPOLLET shows upto 71k events/sec.
> 
> Lack of such feature is a minus for kevent indeed.
> I will add it into todo list behind (implemented) new ring buffer
> implementation and (implemented) wake-up-one-thread flag implementation.
> Hopefully I will include it into next kevent release soon, but do not 
> expect it today/tomorrow, there some unrelated to hacking problems.

Here is edge-triggered behavior of kevent:
[root@pcix event]# ./kevent_bench -n2000 -i
2000 handles setup
67057 evts/sec 1.18746 samples per call
79239 evts/sec 68 ctxt/sec 1.30531 samples per call
78877 evts/sec 140 ctxt/sec 1.34172 samples per call
79017 evts/sec 82 ctxt/sec 1.35835 samples per call
78957 evts/sec 115 ctxt/sec 1.36885 samples per call
79084 evts/sec 70 ctxt/sec 1.37419 samples per call
79083 evts/sec 98 ctxt/sec 1.38 samples per call
79083 evts/sec 72 ctxt/sec 1.38194 samples per call
79025 evts/sec 111 ctxt/sec 1.38426 samples per call
79139 evts/sec 78 ctxt/sec 1.38554 samples per call
79055 evts/sec 112 ctxt/sec 1.38701 samples per call
79118 evts/sec 72 ctxt/sec 1.39 samples per call
79040 evts/sec 94 ctxt/sec 1.39108 samples per call
79098 evts/sec 81 ctxt/sec 1.39136 samples per call
79104 evts/sec 90 ctxt/sec 1.39269 samples per call
Avg: 78265 evts/sec

So, kevent is faster than epoll.
It was proven using three independent benchmarks (mine
evserver_kevent.c, Johann Borck's own web server and Eric's epoll_bench).

I plan to release new version with all additional goodies today.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Jeff Garzik]

David Miller wrote:
> From: Pavel Machek <pavel@ucw.cz>
> Date: Fri, 3 Nov 2006 09:57:12 +0100
> 
>> Not sure what you are smoking, but "there's unsigned long in *bsd
>> version, lets rewrite it from scratch" sounds like very bad idea. What
>> about fixing that one bit you don't like?
> 
> I disagree, it's more like since we have to be structure incompatible
> anyways, let's design something superior if we can.

Definitely agreed.

	Jeff

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Jeff Garzik]

Nate Diller wrote:
> Indesiciveness has certainly been an issue here, but I remember akpm
> and Ulrich both giving concrete suggestions.  I was particularly
> interested in Andrew's request to explain and justify the differences
> between kevent and BSD's kqueue interface.  Was there a discussion
> that I missed?  I am very interested to see your work on this
> mechanism merged, because you've clearly emphasized performance and
> shown impressive results.  But it seems like we lose out on a lot by
> throwing out all the applications that already use kqueue.


kqueue looks pretty nice, the filter/note models in particular.  I don't 
see anything about ring buffers though.

I also wonder about the asynchronous event side (send), not just the 
event reception side.

	Jeff

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Mon, Nov 06, 2006 at 10:17:37PM +0100, Eric Dumazet (dada1@cosmosbay.com) wrote:
> Evgeniy Polyakov a écrit :
> >
> >If there would exist sockets support, then I could patch it to work with
> >kevents.
> >
> 
> OK I post here my last version of epoll_bench.

My results with AF_INET are inlined.
Hardware 2.4 Ghz Xeon w(1 CPU, HT enabled) with 1 GB of RAM.

[root@pcix event]# ./epoll_bench -n 2000 -i
2000 handles setup
49758 evts/sec 1.56177 samples per call
38999 evts/sec 95 ctxt/sec 2.77247 samples per call
54042 evts/sec 130 ctxt/sec 4.19909 samples per call
60155 evts/sec 188 ctxt/sec 5.38024 samples per call
59588 evts/sec 178 ctxt/sec 6.38112 samples per call
60023 evts/sec 188 ctxt/sec 7.19564 samples per call
59694 evts/sec 186 ctxt/sec 7.93067 samples per call
60182 evts/sec 190 ctxt/sec 8.52397 samples per call
59750 evts/sec 182 ctxt/sec 9.08015 samples per call
60158 evts/sec 192 ctxt/sec 9.53548 samples per call
59739 evts/sec 188 ctxt/sec 9.97013 samples per call
60054 evts/sec 216 ctxt/sec 10.32 samples per call
59820 evts/sec 206 ctxt/sec 10.6641 samples per call
60095 evts/sec 218 ctxt/sec 10.9289 samples per call
59376 evts/sec 158 ctxt/sec 11.3231 samples per call
Avg: 57428 evts/sec
[root@pcix event]# ./kevent_bench -n2000 -i
2000 handles setup
57960 evts/sec 0.276702 samples per call
59802 evts/sec 75 ctxt/sec 0.462737 samples per call
59864 evts/sec 71 ctxt/sec 0.623457 samples per call
59651 evts/sec 72 ctxt/sec 0.721579 samples per call
59504 evts/sec 84 ctxt/sec 0.804311 samples per call
61019 evts/sec 72 ctxt/sec 0.904817 samples per call
59846 evts/sec 72 ctxt/sec 0.949439 samples per call
60550 evts/sec 74 ctxt/sec 1.00416 samples per call
59421 evts/sec 66 ctxt/sec 1.04133 samples per call
60334 evts/sec 75 ctxt/sec 1.06845 samples per call
60000 evts/sec 67 ctxt/sec 1.09594 samples per call
59429 evts/sec 74 ctxt/sec 1.11404 samples per call
60508 evts/sec 77 ctxt/sec 1.14482 samples per call
59530 evts/sec 66 ctxt/sec 1.15454 samples per call
59506 evts/sec 73 ctxt/sec 1.17937 samples per call
Avg: 59794 evts/sec
[root@pcix event]# ./kevent_bench -n2000 -i -f
2000 handles setup
82893 evts/sec
88624 evts/sec 390 ctxt/sec
88751 evts/sec 475 ctxt/sec
88784 evts/sec 488 ctxt/sec
88918 evts/sec 458 ctxt/sec
88866 evts/sec 504 ctxt/sec
88950 evts/sec 458 ctxt/sec
88883 evts/sec 472 ctxt/sec
88915 evts/sec 404 ctxt/sec
88836 evts/sec 368 ctxt/sec
89065 evts/sec 442 ctxt/sec
88859 evts/sec 398 ctxt/sec
89070 evts/sec 446 ctxt/sec
88809 evts/sec 428 ctxt/sec
89012 evts/sec 542 ctxt/sec
Avg: 88482 evts/sec

epoll: 57428
kevent: 59794
max: 88482

BUT!
Kevent does not support analogue for EPOLLET, i.e. the case when the
same event is used, instead kevent must modify existing one (i.e. behave
exactly like epoll without EPOLLET), so modified epoll_bench to work
without EPOLLET like kevent.
epoll with EPOLLET shows upto 71k events/sec.

Lack of such feature is a minus for kevent indeed.
I will add it into todo list behind (implemented) new ring buffer
implementation and (implemented) wake-up-one-thread flag implementation.
Hopefully I will include it into next kevent release soon, but do not 
expect it today/tomorrow, there some unrelated to hacking problems.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Mon, Nov 06, 2006 at 10:17:37PM +0100, Eric Dumazet (dada1@cosmosbay.com) wrote:
> AF_INET
> # ./epoll_bench -n 2000 -i
> 2000 handles setup
> 69210 evts/sec 2.97224 samples per call
> 59436 evts/sec 12876 ctxt/sec 5.48675 samples per call
> 60722 evts/sec 12093 ctxt/sec 8.03185 samples per call
> 60583 evts/sec 14582 ctxt/sec 10.5644 samples per call
> 58192 evts/sec 12066 ctxt/sec 12.999 samples per call
> 54291 evts/sec 10613 ctxt/sec 15.2398 samples per call
> 47978 evts/sec 10942 ctxt/sec 17.2222 samples per call
> 59009 evts/sec 13692 ctxt/sec 19.6426 samples per call
> 58248 evts/sec 15099 ctxt/sec 22.0306 samples per call
> 58708 evts/sec 15118 ctxt/sec 24.4497 samples per call
> 58613 evts/sec 14608 ctxt/sec 26.816 samples per call
> 58490 evts/sec 13593 ctxt/sec 29.1708 samples per call
> 59108 evts/sec 15078 ctxt/sec 31.5557 samples per call
> 59636 evts/sec 15053 ctxt/sec 33.9292 samples per call
> 59355 evts/sec 15531 ctxt/sec 36.2914 samples per call
> Avg: 58771 evts/sec
> 
> The last sample shows that epoll overhead is very small indeed, since 
> disabling it doesnt boost AF_INET perf at all.
> AF_INET + no epoll
> # ./epoll_bench -n 2000 -i -f
> 2000 handles setup
> 79939 evts/sec
> 78468 evts/sec 9989 ctxt/sec
> 73153 evts/sec 10207 ctxt/sec
> 73668 evts/sec 10163 ctxt/sec
> 73667 evts/sec 20084 ctxt/sec
> 74106 evts/sec 10068 ctxt/sec
> 73442 evts/sec 10119 ctxt/sec
> 74220 evts/sec 10122 ctxt/sec
> 74367 evts/sec 10097 ctxt/sec
> 64402 evts/sec 47873 ctxt/sec
> 53555 evts/sec 58733 ctxt/sec
> 46000 evts/sec 48984 ctxt/sec
> 67052 evts/sec 21006 ctxt/sec
> 68460 evts/sec 12344 ctxt/sec
> 67629 evts/sec 10655 ctxt/sec
> Avg: 69475 evts/sec

Without epoll number of events/sec is about 18% more - 58k vs 69k.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Eric Dumazet]

[-- Attachment #1: Type: text/plain, Size: 7573 bytes --]

Evgeniy Polyakov a écrit :
> 
> If there would exist sockets support, then I could patch it to work with
> kevents.
> 

OK I post here my last version of epoll_bench.

It works with pipes (default),
or AF_UNIX socketpair() (option -u),
or AF_INET sockets (on loopback device), (option -i)
Only one machine involved (so no real ethernet trafic, and a limit on max 
number of AF_INET sockets since I use one listener 'only')

Option -f ask to bypass epoll.

On a dual opteron 246 machine (2GHZ cpu, 1MB of cache on each cpu, but 
somewhat busy 2.6.18 )

Perf for 2000 concurrent streams is :

259643 evts/sec for pipes
170188 evts/sec for AF_UNIX sockets (-u)
58771 evts/sec for AF_INET sockets (-i)
69475 evts/sec for AF_INET and no epoll gathering at all. (-i -f)

I believe difference between AF_INET sockets and other streams come from 
synchronous/asynchronous wakeups :
I added counters of context switches per second and also number of events 
handled per epoll_wait() call, and we can see that in AF_INET case, the 
consumer is awaken more often. That means lower latency, but less bandwidth alas.

Detailed Results :
pipe
# ./epoll_bench -n 2000
2000 handles setup
255320 evts/sec 362.074 samples per call
254054 evts/sec 10473 ctxt/sec 381.569 samples per call
249868 evts/sec 9155 ctxt/sec 407.461 samples per call
181010 evts/sec 22656 ctxt/sec 420.36 samples per call
233368 evts/sec 8565 ctxt/sec 348.773 samples per call
284682 evts/sec 11114 ctxt/sec 299.987 samples per call
292485 evts/sec 10235 ctxt/sec 279.042 samples per call
279194 evts/sec 10760 ctxt/sec 267.694 samples per call
267917 evts/sec 12035 ctxt/sec 264.106 samples per call
291450 evts/sec 11024 ctxt/sec 247.028 samples per call
266837 evts/sec 11732 ctxt/sec 241.915 samples per call
272762 evts/sec 11492 ctxt/sec 247.629 samples per call
253756 evts/sec 11011 ctxt/sec 253.395 samples per call
251250 evts/sec 9912 ctxt/sec 259.88 samples per call
260706 evts/sec 10754 ctxt/sec 265.079 samples per call
Avg: 259643 evts/sec

  AF_UNIX
# ./epoll_bench -n 2000 -u
2000 handles setup
264827 evts/sec 6.01538 samples per call
259241 evts/sec 15682 ctxt/sec 5.70332 samples per call
262266 evts/sec 17072 ctxt/sec 5.64829 samples per call
262730 evts/sec 16744 ctxt/sec 5.43087 samples per call
253212 evts/sec 17343 ctxt/sec 5.14736 samples per call
255219 evts/sec 17579 ctxt/sec 5.0197 samples per call
166655 evts/sec 13090 ctxt/sec 5.27575 samples per call
111348 evts/sec 10127 ctxt/sec 5.61362 samples per call
104812 evts/sec 9476 ctxt/sec 5.93361 samples per call
95897 evts/sec 8876 ctxt/sec 6.22481 samples per call
97096 evts/sec 9372 ctxt/sec 6.51874 samples per call
113808 evts/sec 11142 ctxt/sec 6.86422 samples per call
102509 evts/sec 10035 ctxt/sec 7.17618 samples per call
100318 evts/sec 9731 ctxt/sec 7.47926 samples per call
102893 evts/sec 9458 ctxt/sec 7.78841 samples per call
Avg: 170188 evts/sec

AF_INET
# ./epoll_bench -n 2000 -i
2000 handles setup
69210 evts/sec 2.97224 samples per call
59436 evts/sec 12876 ctxt/sec 5.48675 samples per call
60722 evts/sec 12093 ctxt/sec 8.03185 samples per call
60583 evts/sec 14582 ctxt/sec 10.5644 samples per call
58192 evts/sec 12066 ctxt/sec 12.999 samples per call
54291 evts/sec 10613 ctxt/sec 15.2398 samples per call
47978 evts/sec 10942 ctxt/sec 17.2222 samples per call
59009 evts/sec 13692 ctxt/sec 19.6426 samples per call
58248 evts/sec 15099 ctxt/sec 22.0306 samples per call
58708 evts/sec 15118 ctxt/sec 24.4497 samples per call
58613 evts/sec 14608 ctxt/sec 26.816 samples per call
58490 evts/sec 13593 ctxt/sec 29.1708 samples per call
59108 evts/sec 15078 ctxt/sec 31.5557 samples per call
59636 evts/sec 15053 ctxt/sec 33.9292 samples per call
59355 evts/sec 15531 ctxt/sec 36.2914 samples per call
Avg: 58771 evts/sec

The last sample shows that epoll overhead is very small indeed, since 
disabling it doesnt boost AF_INET perf at all.
AF_INET + no epoll
# ./epoll_bench -n 2000 -i -f
2000 handles setup
79939 evts/sec
78468 evts/sec 9989 ctxt/sec
73153 evts/sec 10207 ctxt/sec
73668 evts/sec 10163 ctxt/sec
73667 evts/sec 20084 ctxt/sec
74106 evts/sec 10068 ctxt/sec
73442 evts/sec 10119 ctxt/sec
74220 evts/sec 10122 ctxt/sec
74367 evts/sec 10097 ctxt/sec
64402 evts/sec 47873 ctxt/sec
53555 evts/sec 58733 ctxt/sec
46000 evts/sec 48984 ctxt/sec
67052 evts/sec 21006 ctxt/sec
68460 evts/sec 12344 ctxt/sec
67629 evts/sec 10655 ctxt/sec
Avg: 69475 evts/sec

I add here oprofile results for the AF_INET (with epoll) test

CPU: AMD64 processors, speed 1992.3 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit 
mask of 0x00 (No unit mask) count 50000
samples  %        symbol name
1127210   9.1969  tcp_sendmsg
692516    5.6502  fget_light
598653    4.8844  lock_sock
575396    4.6946  __tcp_push_pending_frames
364699    2.9756  tcp_ack
364352    2.9727  tcp_v4_rcv
356383    2.9077  ipt_do_table
324388    2.6467  do_sync_write
257869    2.1039  wait_on_retry_sync_kiocb
255977    2.0885  inet_sk_rebuild_header
255171    2.0819  tcp_recvmsg
249554    2.0361  copy_user_generic_c
232551    1.8974  tcp_transmit_skb
215471    1.7580  release_sock
208563    1.7017  tcp_window_allows
194983    1.5909  kfree
186842    1.5244  system_call
180074    1.4692  kmem_cache_free
160799    1.3120  ep_poll_callback
159235    1.2992  update_send_head
134291    1.0957  sys_epoll_wait
133670    1.0906  ip_queue_xmit
132829    1.0837  ret_from_sys_call
129348    1.0553  __mod_timer
129258    1.0546  sys_write
117884    0.9618  tcp_rcv_established
115181    0.9398  tcp_poll
102805    0.8388  memcpy
99017     0.8079  skb_clone
91125     0.7435  vfs_write
87087     0.7105  __kfree_skb
75387     0.6151  tcp_mss_to_mtu
72483     0.5914  init_or_fini
72207     0.5891  do_sync_read
72054     0.5879  tcp_ioctl
70555     0.5757  local_bh_enable_ip
70001     0.5711  tg3_start_xmit_dma_bug
69914     0.5704  ip_local_deliver
69002     0.5630  tcp_v4_do_rcv
68681     0.5604  dev_queue_xmit
68411     0.5582  do_ip_getsockopt
68235     0.5567  skb_copy_datagram_iovec
66489     0.5425  local_bh_enable

oprofile results for the pipe case :
(where epoll is not noise)

CPU: AMD64 processors, speed 1992.3 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit 
mask of 0x00 (No unit mask) count 50000
samples  %        symbol name
1346203  12.2441  ep_poll_callback
1220770  11.1033  pipe_writev
1020377   9.2806  sys_epoll_wait
991913    9.0218  pipe_readv
779611    7.0908  fget_light
638929    5.8113  __wake_up
625332    5.6876  current_fs_time
486427    4.4242  __mark_inode_dirty
385763    3.5086  __write_lock_failed
217402    1.9773  system_call
175292    1.5943  sys_write
153698    1.3979  __wake_up_common
153242    1.3938  bad_pipe_w
143597    1.3061  generic_pipe_buf_map
140814    1.2807  pipe_poll
130028    1.1826  ret_from_sys_call
122930    1.1181  do_pipe
122359    1.1129  copy_user_generic_c
107443    0.9772  file_update_time
106037    0.9644  sysret_check
101256    0.9210  sys_read
99176     0.9020  iov_fault_in_pages_read
96823     0.8806  generic_pipe_buf_unmap
96675     0.8793  vfs_write
64635     0.5879  rw_verify_area
62997     0.5730  pipe_ioctl
60983     0.5547  tg3_start_xmit_dma_bug
59624     0.5423  get_task_comm
49573     0.4509  tg3_poll
46041     0.4188  schedule
44321     0.4031  vfs_read
35962     0.3271  eventpoll_release_file
30267     0.2753  tg3_write_flush_reg32
29395     0.2674  ipt_do_table
27683     0.2518  page_to_pfn
27492     0.2500  touch_atime
24921     0.2267  memcpy


Eric

[-- Attachment #2: epoll_bench.c --]
[-- Type: text/plain, Size: 5645 bytes --]

/*
 * How to stress epoll
 *
 * This program uses many pipes|sockets and two threads.
 * First we open as many pipes|sockets we can. (see ulimit -n)
 * Then we create a worker thread.
 * The worker thread will send bytes to random streams.
 * The main thread uses epoll to collect ready events and clear them, reading streams.
 * Each second, a number of collected events is printed on stderr
 * After one minute, program prints an average value and stops.
 *
 * Usage : epoll_bench [-f] [-{u|i}] [-n X]
 *   -f : No epoll loop, just feed streams in a cyclic manner
 *   -u : Use AF_UNIX sockets (instead of pipes)
 *   -i : Use AF_INET sockets
 */
#include <pthread.h>
#include <stdlib.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/epoll.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/socket.h>
# include <netinet/in.h>
#include <fcntl.h>
#include <sys/ioctl.h>

int nbhandles = 1024;
int time_test = 15;
unsigned long nbhandled;
unsigned long epw_samples;
unsigned long epw_samples_cnt;

struct pipefd {
	int fd[2];
} *tab;

int epoll_fd;
int fflag;
int afunix;
int afinet;

static int alloc_streams()
{
	int i;
	int listen_sock;
	struct sockaddr_in me, to;
	socklen_t namelen;
    int on = 1;
	int off = 0;

	if (!fflag) {
		epoll_fd = epoll_create(nbhandles);
		if (epoll_fd == -1) {
			perror("epoll_create");
			return -1;
		}
	}
	tab = malloc(sizeof(struct pipefd) * nbhandles);
	if (tab == NULL) {
		perror("malloc");
		return -1;
	}
	if (afinet) {
		listen_sock = socket(AF_INET, SOCK_STREAM, 0);
		if (listen_sock == -1) {
			perror("socket");
			return -1;
		}
		if (listen(listen_sock, 256) == -1) {
			perror("listen");
			return -1;
		}
		namelen = sizeof(me);
		getsockname(listen_sock, (struct sockaddr *)&me, &namelen);
	}
	for (i = 0 ; i < nbhandles ; i++) {
		if (afinet) {
			tab[i].fd[0] = socket(AF_INET, SOCK_STREAM, 0);
			if (tab[i].fd[0] == -1)
				break;
			to = me;
			ioctl(tab[i].fd[0], FIONBIO, &on);
			if (connect(tab[i].fd[0], (struct sockaddr *)&to, sizeof(to)) != -1 || errno != EINPROGRESS)
				break;
			tab[i].fd[1] = accept(listen_sock, (struct sockaddr *)&to, &namelen);
			if (tab[i].fd[1] == -1)
				break;
			ioctl(tab[i].fd[0], FIONBIO, &off);
		}
		else if (afunix) {
			if (socketpair(AF_UNIX, SOCK_STREAM, 0, tab[i].fd) == -1)
				break;
		} else {
			if (pipe(tab[i].fd) == -1)
				break;
		}
		if (!fflag) {
			struct epoll_event ev;
			ev.events = EPOLLIN | EPOLLET;
			ev.data.u64 = (uint64_t)i;
			epoll_ctl(epoll_fd, EPOLL_CTL_ADD, tab[i].fd[0], &ev);
		}
	}
	nbhandles = i;
	printf("%d handles setup\n", nbhandles);
	return 0;
}

int sample_proc_stat(long *ctxt)
{
	int fd = open("/proc/stat", 0);
	char buffer[4096+1], *p;
	int lu;
	*ctxt = 0;
	if (fd == -1) {
		perror("/proc/stat");
		return -1;
	}
	lu = read(fd, buffer, sizeof(buffer));
	close(fd);
	if (lu < 10)
		return -1;
	buffer[lu] = 0;
	p = strstr(buffer, "ctxt");
	if (p)
		*ctxt = atol(p + 4);
	return 0;
}


static void timer_func()
{
	char buffer[128];
	size_t len;
	static unsigned long old;
	static unsigned long oldctxt=0;
	unsigned long ctxt;
	unsigned long delta = nbhandled - old;
	static int alarm_events = 0;

	old = nbhandled;
	len = sprintf(buffer, "%lu evts/sec", delta);
	sample_proc_stat(&ctxt);
	delta = ctxt - oldctxt;
	if (delta && oldctxt)
		len += sprintf(buffer + len, " %lu ctxt/sec", delta);
	oldctxt = ctxt;
	if (epw_samples)
		len += sprintf(buffer + len, " %g samples per call", (double)epw_samples_cnt/(double)epw_samples);
	buffer[len++] = '\n';
	write(2, buffer, len);
	if (++alarm_events >= time_test) {
		delta = nbhandled/alarm_events;
		len = sprintf(buffer, "Avg: %lu evts/sec\n", delta);
		write(2, buffer, len);
		exit(0);
	}
}

static void timer_setup()
{
	struct itimerval it;
	struct sigaction sg;

	memset(&sg, 0, sizeof(sg));
	sg.sa_handler = timer_func;
	sigaction(SIGALRM, &sg, 0);
	it.it_interval.tv_sec = 1;
	it.it_interval.tv_usec = 0;
	it.it_value.tv_sec = 1;
	it.it_value.tv_usec = 0;
	if (setitimer(ITIMER_REAL, &it, 0))
		perror("setitimer");
}

static void * worker_thread_func(void *arg)
{
	int fd = -1;
	char c = 1;
	int cnt = 0;
	nice(10);
	for (;;) {
		if (fflag)
			fd = (fd + 1) % nbhandles;
		else
			fd = rand() % nbhandles;
		write(tab[fd].fd[1], &c, 1);
		if (++cnt >= nbhandles) {
			cnt = 0 ;
			pthread_yield(); /* relax :) */
			}
	}
}

void usage(int code)
{
	fprintf(stderr, "Usage : epoll_bench [-n num] [-{u|i}] [-f] [-t duration] [-l limit] [-e maxepoll]\n");
	exit(code);
}

int main(int argc, char *argv[])
{
	char buff[1024];
	pthread_t tid;
	int c, fd;
	int limit = 1000;
	int max_epoll = 1024;

	while ((c = getopt(argc, argv, "fuin:l:e:t:")) != EOF) {
		if (c == 'n') nbhandles = atoi(optarg);
		else if (c == 'f') fflag++;
		else if (c == 'l') limit = atoi(optarg);
		else if (c == 'e') max_epoll = atoi(optarg);
		else if (c == 't') time_test = atoi(optarg);
		else if (c == 'u') afunix++;
		else if (c == 'i') afinet++;
		else usage(1);
	}
	alloc_streams();
	pthread_create(&tid, NULL, worker_thread_func, (void *)0);
	timer_setup();

	if (fflag) {
		for (fd = 0;;fd = (fd + 1) % nbhandles) {
			if (read(tab[fd].fd[0], buff, 1024) > 0)
				nbhandled++;
			}
		}
	else {
		struct epoll_event *events;
		events = malloc(sizeof(struct epoll_event) * max_epoll) ;
		for (;;) {
			int nb = epoll_wait(epoll_fd, events, max_epoll, -1);
			int i;
			epw_samples++;
			epw_samples_cnt += nb;
			for (i = 0 ; i < nb ; i++) {
				fd = tab[events[i].data.u64].fd[0];
				if (read(fd, buff, 1024) > 0)
					nbhandled++;
			}
			if (nb < limit)
				pthread_yield();
		}
	}
}

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Sun, Nov 05, 2006 at 12:19:33PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> Hi!
> 
> On Fri 2006-11-03 12:13:02, Evgeniy Polyakov wrote:
> > On Fri, Nov 03, 2006 at 09:57:12AM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> > > > So, kqueue API and structures can not be usd in Linux.
> > > 
> > > Not sure what you are smoking, but "there's unsigned long in *bsd
> > > version, lets rewrite it from scratch" sounds like very bad idea. What
> > > about fixing that one bit you don't like?
> > 
> > It is not about what I dislike, but about what is broken or not.
> > Putting u64 instead of a long or some kind of that _is_ incompatible
> > already, so why should we even use it?
> 
> Well.. u64 vs unsigned long *is* binary incompatible, but it is
> similar enough that it is going to be compatible at source level, or
> maybe userland app will need *minor* ifdefs... That's better than two
> completely different versions...
> 
> > And, btw, what we are talking about? Is it about the whole kevent
> > compared to kqueue in kernelspace, or just about what structure is being
> > transferred between kernelspace and userspace?
> > I'm sure, it was some kind of a joke to 'not rewrite *bsd from scratch
> > and use kqueue in Linux kernel as is'.
> 
> No, it is probably not possible to take code from BSD kernel and "just
> port it". But keeping same/similar userland interface would be nice.

It is not only probably, but not even unlikely - it is impossible to get
FreeBSD kqueue code and port it - that port will be completely different
system.
It is impossible to have the same event structure, one should create
#if defined kqueue
fill all members of the structure
#else if defined kevent
fill different members name, since Linux does not even have some types
#endif

*BSD kevent (structure transferred between userspace and kernelspace)
struct kevent {
	uintptr_t ident;	     /* identifier for this event */
	short     filter;	     /* filter for event */
	u_short   flags;	     /* action flags for kqueue */
	u_int     fflags;	     /* filter flag value */
	intptr_t  data;	     		/* filter data value */
	void      *udata;		/* opaque user data identifier */
};

You must fill all fields differently due to above.
Just an example: Linux kevent has extended ID field which is grouped
into type.event, kqueue has different pointer indent and short filter.

Linux kevent does not have filters, but instead it has generic storages
of events which can be processed in any way origin of the storage wants
(this for example allows to create aio_sendfile() (which is dropped from
patchset currently) which no other system in the wild has).

There are too many differences. It is just different systems.
If both can be described by sentence "system which handles events", it
does not mean that they are the same and can use the structures or even
have similar design. 

Kevent is not kqueue in any way (although there are certain
similarities), so they can not share anything.

> 										Pavel
> -- 
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

On Fri 2006-11-03 12:13:02, Evgeniy Polyakov wrote:
> On Fri, Nov 03, 2006 at 09:57:12AM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> > > So, kqueue API and structures can not be usd in Linux.
> > 
> > Not sure what you are smoking, but "there's unsigned long in *bsd
> > version, lets rewrite it from scratch" sounds like very bad idea. What
> > about fixing that one bit you don't like?
> 
> It is not about what I dislike, but about what is broken or not.
> Putting u64 instead of a long or some kind of that _is_ incompatible
> already, so why should we even use it?

Well.. u64 vs unsigned long *is* binary incompatible, but it is
similar enough that it is going to be compatible at source level, or
maybe userland app will need *minor* ifdefs... That's better than two
completely different versions...

> And, btw, what we are talking about? Is it about the whole kevent
> compared to kqueue in kernelspace, or just about what structure is being
> transferred between kernelspace and userspace?
> I'm sure, it was some kind of a joke to 'not rewrite *bsd from scratch
> and use kqueue in Linux kernel as is'.

No, it is probably not possible to take code from BSD kernel and "just
port it". But keeping same/similar userland interface would be nice.
										Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Fri, Nov 03, 2006 at 07:49:16PM +0100, Oleg Verych (olecom@flower.upol.cz) wrote:
> [ Please do not answer just to answer, cc list is big, no one from       ]
> [ The Big Jury seems to care. (well, Jonathan does, but he wasn't in cc) ]
> 
> Friendly, Oleg.

Just in case some misunderstanding happend: I do not want to insult
anyone who is against kevent, I just do not understand cases, when
people require me to do something to convince them in rude manner.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Fri, Nov 03, 2006 at 07:49:16PM +0100, Oleg Verych (olecom@flower.upol.cz) wrote:
> > applications can be found on project's homepage.
> > There is a link to archive there, where you can find plenty of sources.
> 
> But no single makefile. Or what CC and options do not mater really?
> You can easily find in your server's apache logs, my visit of that
> archive in the day of my message (today i just confirmed my assertions):
> browser lynx, host flower.upol.cz.

If you can not compile that sources, than you should not use kevent for
a while. Definitely.
Options are pretty simple: -W -Wall -I$(path_to_kernel_tree)/include

> > You likely do not know, but it is a bit risky business to patch all
> > existing applications to show that approach is correct, if
> > implementation is not completed.
> 
> Fortunately to me, `lighthttpd' is real-life *and* in the benchmark
> area also. Just see that site how much there was measured: different OSes,
> special tunning. *That* is i'm talking about. Epoll _wrapper_ there,
> is 3461 byte long, your answer to _me_ 2580. People are bringing you a
> test bed, with all set up ready to use; need less code, go on, comment
> needless out!

So what?
People bring me tons of various stuff, and I prefer to use my own for
tests. If _you_ need it, _you_ can always patch any sources you like.

> > You likely do not know, but after I first time announced kevents in
> > February I changed interfaces 4 times - and it is just interfaces, not
> > including numerous features added/removed by developer's requests.
> 
> I think that called open source, linux kernel case.

You missed the point - I'm not going to patch tons of existing
applications when I'm asked to change an interface once per month.

When all requested features are implemented I definitely with patch some
popular web-server to show how kevent is used.

> > > There were some comments about laking much of such programs, answers were
> > > "was in prev. e-mail", "need to update them", something like that.
> > > "Trivial web server" sources url, mentioned in benchmark isn't pointed
> > > in patch advertisement. If it was, should i actually try that new
> > > *trivial* wheel?
> > 
> > Answer is trivial - there is archive where one can find a source code
> > (filenames are posted regulary). Should I create a rpm? For what glibc
> > version?
> 
> Hmm. Let me answer on that "dup" with stuff from LKML archive. That
> will reveal, that my guesses were told by The Big Jury to you already:
> 
> [^0] Message-ID: 44CA66D8.3010404@oracle.com
> [^1] Message-ID: 20060818104120.GA20816@infradead.org,
>      Message-ID: 20060816133014.GB32499@infradead.org
> 
> more than 10 takes ago.

And? Please provide a link to archive.

> > > Saying that, i want to give you some short examples, i know.
> > > *Linux kernel <-> userspace*:
> > > o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;
> > 
> > iproute documentation was way too bad when Alexey presented it first 
> > time :)
> 
> As example, after have read some books on TCP/IP and Ethernet, internal
> help of `ip' was all i needed to know.

:)) i.e. it is ok for you to 'read some books on TCP/IP and Ethernet' to
understand how utility works, and it is not ok to determine how to
compile my sources? Do not compile my sources.

> > Btw, show me splice() 'shiny' application? Does lighttpd use it?
> > Or move_pages().
> 
> You know who proposed that, and you know how many (few) releases ago.

And why lighttpd still do not use it?
You should start to blame authors of the splice() for that.
You will not? Then I can not consider your words in my direction as
serious.

> > > To make a little hint to you, Evgeniy, why don't you find a little
> > > animal in the open source zoo to implement little interface to
> > > proposed kernel subsystem and then show it to The Big Jury (not me),
> > > we have here? And i can not see, how you've managed to implement
> > > something like that having almost nothing on the test basket.
> > > Very *suspicious* ch.
> > 
> > There are always people who do not like something, what can I do with
> 
> I didn't think, that my message was offensive. Also i didn't even say,
> that you have not bothered feed your code to "scripts/Lindent".

You do not use kevent, why do you care about indent of the userspace
tools?

> []
> > I created trivial web servers, which send single static page and use
> > various event handling schemes, and I test new subsystem with new tools,
> > when tests are completed and all requested features are implemented it
> > is time to work on different more complex users.
> 
> Please, see [^0],
> 
> > So let's at least complete what we have right now, so no developer's
> > efforts could be wasted writing empty chars in various places.
> 
> and [^1].
> 
> [ Please do not answer just to answer, cc list is big, no one from       ]
> [ The Big Jury seems to care. (well, Jonathan does, but he wasn't in cc) ]

This thread is just to answer for the sake of answers - there is
completely no sense in it.
You blame me that I did not create some benchmarks you like, but I do not
care about it. I created usefull patch and test is in the way I like,
because it is much more productive, than spending a lot of time
detemining how different sources work with appropriate loads.
When there will be strong requirement to perform additional tests, I
will do them.

> Friendly, Oleg.
> ____

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Oleg Verych]

On Wed, Nov 01, 2006 at 09:57:46PM +0300, Evgeniy Polyakov wrote:
> On Wed, Nov 01, 2006 at 06:20:43PM +0000, Oleg Verych (olecom@flower.upol.cz) wrote:
[] 
> > Where's real-life application to do configure && make && make install?
> 
> Your real life or mine as developer?
> I fortunately do not know anything about your real life, but my real life

To do not further shift conversation in no technical way, think of my
sentence as question *and* as definition.

> applications can be found on project's homepage.
> There is a link to archive there, where you can find plenty of sources.

But no single makefile. Or what CC and options do not mater really?
You can easily find in your server's apache logs, my visit of that
archive in the day of my message (today i just confirmed my assertions):
browser lynx, host flower.upol.cz.

> You likely do not know, but it is a bit risky business to patch all
> existing applications to show that approach is correct, if
> implementation is not completed.

Fortunately to me, `lighthttpd' is real-life *and* in the benchmark
area also. Just see that site how much there was measured: different OSes,
special tunning. *That* is i'm talking about. Epoll _wrapper_ there,
is 3461 byte long, your answer to _me_ 2580. People are bringing you a
test bed, with all set up ready to use; need less code, go on, comment
needless out!

> You likely do not know, but after I first time announced kevents in
> February I changed interfaces 4 times - and it is just interfaces, not
> including numerous features added/removed by developer's requests.

I think that called open source, linux kernel case.

> > There were some comments about laking much of such programs, answers were
> > "was in prev. e-mail", "need to update them", something like that.
> > "Trivial web server" sources url, mentioned in benchmark isn't pointed
> > in patch advertisement. If it was, should i actually try that new
> > *trivial* wheel?
> 
> Answer is trivial - there is archive where one can find a source code
> (filenames are posted regulary). Should I create a rpm? For what glibc
> version?

Hmm. Let me answer on that "dup" with stuff from LKML archive. That
will reveal, that my guesses were told by The Big Jury to you already:

[^0] Message-ID: 44CA66D8.3010404@oracle.com
[^1] Message-ID: 20060818104120.GA20816@infradead.org,
     Message-ID: 20060816133014.GB32499@infradead.org

more than 10 takes ago.

> > Saying that, i want to give you some short examples, i know.
> > *Linux kernel <-> userspace*:
> > o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;
> 
> iproute documentation was way too bad when Alexey presented it first 
> time :)

As example, after have read some books on TCP/IP and Ethernet, internal
help of `ip' was all i needed to know.

> Btw, show me splice() 'shiny' application? Does lighttpd use it?
> Or move_pages().

You know who proposed that, and you know how many (few) releases ago.
 
> > To make a little hint to you, Evgeniy, why don't you find a little
> > animal in the open source zoo to implement little interface to
> > proposed kernel subsystem and then show it to The Big Jury (not me),
> > we have here? And i can not see, how you've managed to implement
> > something like that having almost nothing on the test basket.
> > Very *suspicious* ch.
> 
> There are always people who do not like something, what can I do with

I didn't think, that my message was offensive. Also i didn't even say,
that you have not bothered feed your code to "scripts/Lindent".

[]
> I created trivial web servers, which send single static page and use
> various event handling schemes, and I test new subsystem with new tools,
> when tests are completed and all requested features are implemented it
> is time to work on different more complex users.

Please, see [^0],

> So let's at least complete what we have right now, so no developer's
> efforts could be wasted writing empty chars in various places.

and [^1].

[ Please do not answer just to answer, cc list is big, no one from       ]
[ The Big Jury seems to care. (well, Jonathan does, but he wasn't in cc) ]

Friendly, Oleg.
____

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Fri, Nov 03, 2006 at 10:42:04AM +0800, zhou drangon (drangon.mail@gmail.com) wrote:
> As for the VFS system, when we introduce the AIO machinism, we add aio_read,
> aio_write, etc... to file ops, and then we make the read, write op to
> call aio_read,
> aio_write, so that we only remain one implement in kernel.
> Can we do event machinism the same way?
> when kevent is robust enough, can we implement epoll/select/io_submit etc...
> base on kevent ??
> In this way, we can simplified the kernel, and epoll can gain
> improvement from kevent.

There is AIO implementaion on top of kevent, although it was confirmed
that it has a good design, except minor API layering changes, it was
postponed for a while.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Fri, Nov 03, 2006 at 09:57:12AM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> > So, kqueue API and structures can not be usd in Linux.
> 
> Not sure what you are smoking, but "there's unsigned long in *bsd
> version, lets rewrite it from scratch" sounds like very bad idea. What
> about fixing that one bit you don't like?

It is not about what I dislike, but about what is broken or not.
Putting u64 instead of a long or some kind of that _is_ incompatible
already, so why should we even use it?
And, btw, what we are talking about? Is it about the whole kevent
compared to kqueue in kernelspace, or just about what structure is being
transferred between kernelspace and userspace?
I'm sure, it was some kind of a joke to 'not rewrite *bsd from scratch
and use kqueue in Linux kernel as is'.

> 								Pavel
> -- 
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[David Miller]

From: Pavel Machek <pavel@ucw.cz>
Date: Fri, 3 Nov 2006 09:57:12 +0100

> Not sure what you are smoking, but "there's unsigned long in *bsd
> version, lets rewrite it from scratch" sounds like very bad idea. What
> about fixing that one bit you don't like?

I disagree, it's more like since we have to be structure incompatible
anyways, let's design something superior if we can.

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

> > returns, which thread are you referring to?  Nicholas Miell, in "The
> > Proposed Linux kevent API" thread, seems to think that there are no
> > advantages over kqueue to justify the incompatibility, an argument you
> > made no effort to refute.  I've also read the Kevent wiki at
> > linux-net.osdl.org, but it too is lacking in any direct comparisons
> > (even theoretical, let alone benchmarks) of the flexibility,
> > performance, etc. between the two.
> > 
> > I'm not arguing that you've done a bad design, I'm asking you to brag
> > about the things you improved on vs. kqueue.  Your emphasis on
> > unifying all the different event types into one interface is really
> > cool, fill me in on why that can't be effectively done with the kqueue
> > compatability and I also will advocate for kevent inclusion.
> 
> kqueue just can not be used as is in Linux (_maybe_ *bsd has different
> types, not those which I found in /usr/include in my FC5 and Debian
> distro). It will not work on x86_64 for example. Some kind of a pointer
> or unsigned long in structures which are transferred between kernelspace
> and userspace is so much questionable, than it is much better even do
> not see there... (if I would not have so political correctness, I would
> describe it in a much different words actually).
> So, kqueue API and structures can not be usd in Linux.

Not sure what you are smoking, but "there's unsigned long in *bsd
version, lets rewrite it from scratch" sounds like very bad idea. What
about fixing that one bit you don't like?
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Thu, Nov 02, 2006 at 11:40:43AM -0800, Nate Diller (nate.diller@gmail.com) wrote:
> Are you saying that the *only* reason we choose not to be
> source-compatible with BSD is the 32 bit userland on 64 bit arch
> problem?  I've followed every thread that gmail 'kqueue' search

I.e. do you want that generic event handling mechanism would not work on
x86_64? I doubt you do.

> returns, which thread are you referring to?  Nicholas Miell, in "The
> Proposed Linux kevent API" thread, seems to think that there are no
> advantages over kqueue to justify the incompatibility, an argument you
> made no effort to refute.  I've also read the Kevent wiki at
> linux-net.osdl.org, but it too is lacking in any direct comparisons
> (even theoretical, let alone benchmarks) of the flexibility,
> performance, etc. between the two.
> 
> I'm not arguing that you've done a bad design, I'm asking you to brag
> about the things you improved on vs. kqueue.  Your emphasis on
> unifying all the different event types into one interface is really
> cool, fill me in on why that can't be effectively done with the kqueue
> compatability and I also will advocate for kevent inclusion.

kqueue just can not be used as is in Linux (_maybe_ *bsd has different
types, not those which I found in /usr/include in my FC5 and Debian
distro). It will not work on x86_64 for example. Some kind of a pointer
or unsigned long in structures which are transferred between kernelspace
and userspace is so much questionable, than it is much better even do
not see there... (if I would not have so political correctness, I would
describe it in a much different words actually).
So, kqueue API and structures can not be usd in Linux.

> NATE

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[zhou drangon]

2006/11/2, Eric Dumazet <dada1@cosmosbay.com>:
> zhou drangon a écrit :
> > performance is great, and we are exciting at the result.
> >
> > I want to know why there can be so much improvement, can we improve
> > epoll too ?
>
> Why did you remove most of CC addresses but lkml ?
> Dont do that please...
I seldom reply to the mailing list, Sorry for this.
>
> Good question :)
>
> Hum, I think I can look into epoll and see how it can be improved (if necessary)
>
I have an other question.
As for the VFS system, when we introduce the AIO machinism, we add aio_read,
aio_write, etc... to file ops, and then we make the read, write op to
call aio_read,
aio_write, so that we only remain one implement in kernel.
Can we do event machinism the same way?
when kevent is robust enough, can we implement epoll/select/io_submit etc...
base on kevent ??
In this way, we can simplified the kernel, and epoll can gain
improvement from kevent.

> This is not to say we dont need kevent ! Please Evgeniy continue your work !
Yes! We are expecting for you greate work.

I create an userland event-driven framework for my application.
but I have to use multiple thread to receive event, epoll to wait most event,
and io_getevent to wait disk AIO event, I hope we can get a universal
event machinism
to make the code elegance.
>
> Just to remind you that according to
> http://www.xmailserver.org/linux-patches/nio-improve.html David Libenzi had to
> wait 18 months before epoll being officialy added into kernel.
>
> At that time, many applications were using epoll, and we were patching our
> kernels for that.
>
>
> I cooked a very simple program (attached in this mail), using pipes and epoll,
> and got 250.000 events received per second on an otherwise lightly loaded
> machine (dual opteron 246 , 2GHz, 1MB cache per cpu) with 10.000 pipes (20.000
> handles)
>
> It could be nice to add support for other event providers in this program
> (AF_INET & AF_UNIX sockets for example), and also add support for kevent, so
> that we really can compare epoll/kevent without a complex setup.
> I should extend the program to also add/remove sources during lifetime, not
> only insert at setup time.
>
> # gcc -O2 -o epoll_pipe_bench epoll_pipe_bench.c -lpthread
> # ulimit -n 1000000
> # epoll_pipe_bench -n 10000
> ^C after a while...
>
> oprofile results say that ep_poll_callback() and sys_epoll_wait() use 20% of
> cpu time.
> Even if we gain a two factor in cpu time or cache usage, we wont eliminate
> other costs...
>
> oprofile results gave :
>
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
> mask of 0x00 (No unit mask) count 50000
> samples  %        symbol name
> 2015420  11.1309  ep_poll_callback
> 1867431  10.3136  pipe_writev
> 1791872   9.8963  sys_epoll_wait
> 1357297   7.4962  fget_light
> 1277515   7.0556  pipe_readv
> 998447    5.5143  current_fs_time
> 801597    4.4271  __mark_inode_dirty
> 755268    4.1713  __wake_up
> 587065    3.2423  __write_lock_failed
> 582931    3.2195  system_call
> 297132    1.6410  iov_fault_in_pages_read
> 296136    1.6355  sys_write
> 290106    1.6022  __wake_up_common
> 270692    1.4950  bad_pipe_w
> 261516    1.4443  do_pipe
> 257208    1.4205  tg3_start_xmit_dma_bug
> 254917    1.4079  pipe_poll
> 252925    1.3969  copy_user_generic_c
> 234212    1.2935  generic_pipe_buf_map
> 228659    1.2629  ret_from_sys_call
> 212541    1.1738  sysret_check
> 166529    0.9197  sys_read
> 160038    0.8839  vfs_write
> 151091    0.8345  pipe_ioctl
> 136301    0.7528  file_update_time
> 107173    0.5919  tg3_poll
> 77846     0.4299  ipt_do_table
> 75081     0.4147  schedule
> 73059     0.4035  vfs_read
> 69787     0.3854  get_task_comm
> 63923     0.3530  memcpy
> 60019     0.3315  touch_atime
> 57490     0.3175  eventpoll_release_file
> 56152     0.3101  tg3_write_flush_reg32
> 54468     0.3008  rw_verify_area
> 47833     0.2642  generic_pipe_buf_unmap
> 47777     0.2639  __switch_to
> 44106     0.2436  bad_pipe_r
> 41824     0.2310  proc_nr_files
> 41319     0.2282  pipe_iov_copy_from_user
>
>
> Eric
>
>
>
> /*
>  * How to stress epoll
>  *
>  * This program uses many pipes and two threads.
>  * First we open as many pipes we can. (see ulimit -n)
>  * Then we create a worker thread.
>  * The worker thread will send bytes to random pipes.
>  * The main thread uses epoll to collect ready pipes and read them.
>  * Each second, a number of collected bytes is printed on stderr
>  *
>  * Usage : epoll_bench [-n X]
>  */
> #include <pthread.h>
> #include <stdlib.h>
> #include <errno.h>
> #include <stdio.h>
> #include <string.h>
> #include <sys/epoll.h>
> #include <signal.h>
> #include <unistd.h>
> #include <sys/time.h>
>
> int nbpipes = 1024;
>
> struct pipefd {
>         int fd[2];
> } *tab;
>
> int epoll_fd;
>
> static int alloc_pipes()
> {
>         int i;
>
>         epoll_fd = epoll_create(nbpipes);
>         if (epoll_fd == -1) {
>                 perror("epoll_create");
>                 return -1;
>         }
>         tab = malloc(sizeof(struct pipefd) * nbpipes);
>         if (tab ==NULL) {
>                 perror("malloc");
>                 return -1;
>         }
>         for (i = 0 ; i < nbpipes ; i++) {
>                         struct epoll_event ev;
>                 if (pipe(tab[i].fd) == -1)
>                         break;
>                 ev.events = EPOLLIN | EPOLLOUT | EPOLLHUP | EPOLLPRI | EPOLLET;
>                 ev.data.u64 = (uint64_t)i;
>                 epoll_ctl(epoll_fd, EPOLL_CTL_ADD, tab[i].fd[0], &ev);
>         }
>         nbpipes = i;
>         printf("%d pipes setup\n", nbpipes);
>         return 0;
> }
>
>
> unsigned long nbhandled;
> static void timer_func()
> {
>         char buffer[32];
>         size_t len;
>         static unsigned long old;
>         unsigned long delta = nbhandled - old;
>         old = nbhandled;
>         len = sprintf(buffer, "%lu\n", delta);
>         write(2, buffer, len);
> }
>
> static void timer_setup()
> {
>         struct itimerval it;
>         struct sigaction sg;
>
>         memset(&sg, 0, sizeof(sg));
>         sg.sa_handler = timer_func;
>         sigaction(SIGALRM, &sg, 0);
>         it.it_interval.tv_sec = 1;
>         it.it_interval.tv_usec = 0;
>         it.it_value.tv_sec = 1;
>         it.it_value.tv_usec = 0;
>         if (setitimer(ITIMER_REAL, &it, 0))
>                 perror("setitimer");
> }
>
> static void * worker_thread_func(void *arg)
> {
>         int fd;
>         char c = 1;
>         for (;;) {
>                 fd = rand() % nbpipes;
>                 write(tab[fd].fd[1], &c, 1);
>         }
> }
>
>
> int main(int argc, char *argv[])
> {
>         char buff[1024];
>         pthread_t tid;
>         int c;
>
>         while ((c = getopt(argc, argv, "n:")) != EOF) {
>                 if (c == 'n') nbpipes = atoi(optarg);
>         }
>         alloc_pipes();
>         pthread_create(&tid, NULL, worker_thread_func, (void *)0);
>         timer_setup();
>
>         for (;;) {
>                 struct epoll_event events[128];
>                 int nb = epoll_wait(epoll_fd, events, 128, 10000);
>                 int i, fd;
>                 for (i = 0 ; i < nb ; i++) {
>                         fd = tab[events[i].data.u64].fd[0];
>                         if (read(fd, buff, 1024) > 0)
>                                 nbhandled++;
>                 }
>         }
> }
>
>
>

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Nate Diller]

On 11/1/06, Evgeniy Polyakov <johnpol@2ka.mipt.ru> wrote:
> On Wed, Nov 01, 2006 at 06:12:41PM -0800, Nate Diller (nate.diller@gmail.com) wrote:
> > Indesiciveness has certainly been an issue here, but I remember akpm
> > and Ulrich both giving concrete suggestions.  I was particularly
> > interested in Andrew's request to explain and justify the differences
> > between kevent and BSD's kqueue interface.  Was there a discussion
> > that I missed?  I am very interested to see your work on this
> > mechanism merged, because you've clearly emphasized performance and
> > shown impressive results.  But it seems like we lose out on a lot by
> > throwing out all the applications that already use kqueue.
>
> It looks you missed that discussion - freebsd kqueue has fields in the
> kevent structure which have diffent sizes in 32 and 64 bit environments.

Are you saying that the *only* reason we choose not to be
source-compatible with BSD is the 32 bit userland on 64 bit arch
problem?  I've followed every thread that gmail 'kqueue' search
returns, which thread are you referring to?  Nicholas Miell, in "The
Proposed Linux kevent API" thread, seems to think that there are no
advantages over kqueue to justify the incompatibility, an argument you
made no effort to refute.  I've also read the Kevent wiki at
linux-net.osdl.org, but it too is lacking in any direct comparisons
(even theoretical, let alone benchmarks) of the flexibility,
performance, etc. between the two.

I'm not arguing that you've done a bad design, I'm asking you to brag
about the things you improved on vs. kqueue.  Your emphasis on
unifying all the different event types into one interface is really
cool, fill me in on why that can't be effectively done with the kqueue
compatability and I also will advocate for kevent inclusion.

NATE

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Eric Dumazet]

On Thursday 02 November 2006 09:46, Evgeniy Polyakov wrote:

> By poll design we have to setup following data:
> poll_table_struct, which contains a callback
> that callback will be called in each
> sys_poll()->drivers_poll()->poll_wait(),
> callback will allocate new private structure, which must have
> wait_queue_t (it's callback will be invoked each time wake_up() is
> called for given wait_queue_head), which should be linked to the given
> wait_queue_head.

In epoll case, the setup is done at epoll_ctl() time, not for each event 
received by the consumer like poll()/select()

As for wake_up() overhead, I feel it's necessary if we want to wakeup a 
consumer (tell him some events are availlable)

I suspect benchmark results might depend for a large part on some 'features' 
of the scheduler, or the number of events provider and CPU cache size, not on 
inherent limitations of epoll.

I changed a litle bit epoll_pipe_bench and we can see effects of scheduling on 
the overall rate of events per second.

I can now receive 350.000 events per second, instead of 230.000

Tests done on my laptop : a UP machine (Intel(R) Pentium(R) M processor 
1.60GHz) (Dell latitude D610)

# ./epoll_pipe_bench -n 1000 -l 1
1000 pipes setup
223065 evts/sec 1.00563 samples per call
228508 evts/sec 1.00277 samples per call
229076 evts/sec 1.00184 samples per call
227860 evts/sec 1.00191 samples per call
229498 evts/sec 1.00153 samples per call
228027 evts/sec 1.00136 samples per call
229465 evts/sec 1.00122 samples per call
227845 evts/sec 1.00132 samples per call
227456 evts/sec 1.00118 samples per call
228355 evts/sec 1.00113 samples per call

# ./epoll_pipe_bench -n 1000 -l 10
1000 pipes setup
328599 evts/sec 314.75 samples per call
344947 evts/sec 314.447 samples per call
342844 evts/sec 314.185 samples per call
345486 evts/sec 314.013 samples per call
345144 evts/sec 314.079 samples per call
344270 evts/sec 313.989 samples per call
344249 evts/sec 314.004 samples per call
320577 evts/sec 326.967 samples per call
313990 evts/sec 343.926 samples per call
313578 evts/sec 359.034 samples per call

In oprofile, It's not obvious epoll is responsible for cpu costs...

CPU: PIII, speed 1600 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit 
mask of 0x00 (No unit mask) count 50000
samples  %        symbol name
206966    7.9385  sysenter_past_esp
158337    6.0733  _spin_lock_irqsave
143339    5.4980  pipe_writev
137434    5.2715  fget_light
132293    5.0743  _spin_unlock_irqrestore
109593    4.2036  pipe_readv
102669    3.9380  ep_poll_callback
99801     3.8280  dnotify_parent
99388     3.8122  sys_epoll_wait
98710     3.7862  vfs_write
94560     3.6270  _write_lock_irqsave
91774     3.5201  current_fs_time
90541     3.4728  pipe_poll
89153     3.4196  _spin_lock
85150     3.2661  __wake_up
83213     3.1918  __wake_up_common
72913     2.7967  try_to_wake_up
67996     2.6081  rw_verify_area
64243     2.4641  file_update_time
54211     2.0793  vfs_read


But but but ! If I add more pipes, results are reversed, because CPU cache is 
not large enough : It's better to deliver events as fast as possible to 
consumer to keep hot caches (but only on UP machine, and because my test prog 
is threaded. If it was using two process, context switch would be more 
expensive)


# ./epoll_pipe_bench -n 10000 -l 1
10000 pipes setup
171444 evts/sec 1 samples per call
174556 evts/sec 1 samples per call
173976 evts/sec 1 samples per call
174715 evts/sec 1.00003 samples per call
173215 evts/sec 1.00478 samples per call
174930 evts/sec 1.00397 samples per call

# ./epoll_pipe_bench -n 10000 -l 10
10000 pipes setup
149701 evts/sec 759.904 samples per call
153476 evts/sec 767.537 samples per call
149217 evts/sec 767.585 samples per call
152396 evts/sec 763.624 samples per call
153517 evts/sec 762.215 samples per call
148026 evts/sec 763.434 samples per call

CPU: PIII, speed 1600 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit 
mask of 0x00 (No unit mask) count 50000
samples  %        symbol name
404924   12.4647  pipe_poll
263399    8.1081  pipe_writev
259230    7.9798  rw_verify_area
246368    7.5839  fget_light
242360    7.4605  sys_epoll_wait
174913    5.3843  kmap_atomic
168623    5.1907  __wake_up_common
147661    4.5454  ep_poll_callback
137003    4.2173  pipe_readv
128799    3.9648  _spin_lock_irqsave
100015    3.0787  sysenter_past_esp
74031     2.2789  __copy_to_user_ll
72857     2.2427  file_update_time
69523     2.1401  vfs_write
69444     2.1377  mutex_lock


I also added a -f flag, to bypass epoll completely and measure the 
pipe/read/write overhead/performance.

# ./epoll_pipe_bench -n 10000 -f
10000 pipes setup
300230 evts/sec
309770 evts/sec
264426 evts/sec
265842 evts/sec
265551 evts/sec
266814 evts/sec
266551 evts/sec
264415 evts/sec

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Thu, Nov 02, 2006 at 09:18:55AM +0100, Eric Dumazet (dada1@cosmosbay.com) wrote:
> Evgeniy Polyakov a écrit :
> >pipes will work with kevent's poll mechanisms only, so there will not be
> >any performance gain at all since it is essentially the same as epoll
> >design with waiting and rescheduling (all my measurements with 
> >epoll vs. kevent_poll always showed the same rates), pipes require the same
> >notifications as sockets for maximum perfomance.
> >I've put it into todo list.
> 
> Evgeniy I think this part is *important*. I think most readers of lkml are 
> not aware of exact mechanisms used in epoll, kevent poll, and 'kevent'
> 
> I dont understand why epoll is bad for you, since for me, 
> ep_poll_callback() is fast enough, even if we can make it touch less cache 
> lines if reoredering 'struct epitem' correctly. My epoll_pipe_bench doesnt 
> change the rescheduling rate of the test machine.
> 
> Could you in your home page add some doc that clearly show the path taken 
> for those 3 mechanisms and different events sources (At least sockets)

It is.

"It [kevent] supports socket notifications (accept, sending and receiving),
network AIO (aio_send(), aio_recv() and aio_sendfile()), inode
notifications (create/remove), generic poll()/select() notifications and
timer notifications."
In each patch I give a short description and socket notification patch

By poll design we have to setup following data:
poll_table_struct, which contains a callback
that callback will be called in each
sys_poll()->drivers_poll()->poll_wait(),
callback will allocate new private structure, which must have
wait_queue_t (it's callback will be invoked each time wake_up() is
called for given wait_queue_head), which should be linked to the given
wait_queue_head.

Kevent has different approach: so called origins (files, inodes,
sockets and so on) have a queues of userspace requests, for example
socket origin can only have a queue which will contain one of the
following events ($type.$event): socket.send, socket.recv,
socket.accept. So when new data has arrived, appropriate event is marked
as ready and moved into ready queue (very short operations) and
requested thread is awakened, which can then get ready events and
requeue them back (or remove, depending on flags). There are no
allocations in kevent_get_events() (epoll_wait() does not have it too),
no potentially long lists of wait_queue linked to the same 
wait_queue_head_t, which is traversed each time we call wake_up(),
it has much smaller memory footprint compared to epoll (there is only
one kevent compared to epitem and eppoll_entry).

> Eric

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Eric Dumazet]

Evgeniy Polyakov a écrit :
> pipes will work with kevent's poll mechanisms only, so there will not be
> any performance gain at all since it is essentially the same as epoll
> design with waiting and rescheduling (all my measurements with 
> epoll vs. kevent_poll always showed the same rates), pipes require the same
> notifications as sockets for maximum perfomance.
> I've put it into todo list.

Evgeniy I think this part is *important*. I think most readers of lkml are not 
aware of exact mechanisms used in epoll, kevent poll, and 'kevent'

I dont understand why epoll is bad for you, since for me, ep_poll_callback() 
is fast enough, even if we can make it touch less cache lines if reoredering 
'struct epitem' correctly. My epoll_pipe_bench doesnt change the rescheduling 
rate of the test machine.

Could you in your home page add some doc that clearly show the path taken for 
those 3 mechanisms and different events sources (At least sockets)

Eric

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Thu, Nov 02, 2006 at 08:46:41AM +0100, Eric Dumazet (dada1@cosmosbay.com) wrote:
> zhou drangon a écrit :
> >performance is great, and we are exciting at the result.
> >
> >I want to know why there can be so much improvement, can we improve 
> >epoll too ?
> 
> Why did you remove most of CC addresses but lkml ?
> Dont do that please...

Sure, since for example I'm not subscribed (fortunately) to lkml, and I
think you want me to answer the question too...

> Good question :)
> 
> Hum, I think I can look into epoll and see how it can be improved (if 
> necessary)

epoll can not be improved, since the whole polling is designed to have
several layers of dereferencing, kevent simplifies that chain noticebly.

> This is not to say we dont need kevent ! Please Evgeniy continue your work !

I will :)

> Just to remind you that according to 
> http://www.xmailserver.org/linux-patches/nio-improve.html David Libenzi had 
> to wait 18 months before epoll being officialy added into kernel.

kevent exists for about 10 month. We have plenty of time :)

> At that time, many applications were using epoll, and we were patching our 
> kernels for that.
> 
> 
> I cooked a very simple program (attached in this mail), using pipes and 
> epoll, and got 250.000 events received per second on an otherwise lightly 
> loaded machine (dual opteron 246 , 2GHz, 1MB cache per cpu) with 10.000 
> pipes (20.000 handles)

pipes will work with kevent's poll mechanisms only, so there will not be
any performance gain at all since it is essentially the same as epoll
design with waiting and rescheduling (all my measurements with 
epoll vs. kevent_poll always showed the same rates), pipes require the same
notifications as sockets for maximum perfomance.
I've put it into todo list.

> It could be nice to add support for other event providers in this program 
> (AF_INET & AF_UNIX sockets for example), and also add support for kevent, 
> so that we really can compare epoll/kevent without a complex setup.
> I should extend the program to also add/remove sources during lifetime, not 
> only insert at setup time.

If there would exist sockets support, then I could patch it to work with
kevents.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Eric Dumazet]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

zhou drangon a écrit :
> performance is great, and we are exciting at the result.
> 
> I want to know why there can be so much improvement, can we improve 
> epoll too ?

Why did you remove most of CC addresses but lkml ?
Dont do that please...

Good question :)

Hum, I think I can look into epoll and see how it can be improved (if necessary)

This is not to say we dont need kevent ! Please Evgeniy continue your work !

Just to remind you that according to 
http://www.xmailserver.org/linux-patches/nio-improve.html David Libenzi had to 
wait 18 months before epoll being officialy added into kernel.

At that time, many applications were using epoll, and we were patching our 
kernels for that.


I cooked a very simple program (attached in this mail), using pipes and epoll, 
and got 250.000 events received per second on an otherwise lightly loaded 
machine (dual opteron 246 , 2GHz, 1MB cache per cpu) with 10.000 pipes (20.000 
handles)

It could be nice to add support for other event providers in this program 
(AF_INET & AF_UNIX sockets for example), and also add support for kevent, so 
that we really can compare epoll/kevent without a complex setup.
I should extend the program to also add/remove sources during lifetime, not 
only insert at setup time.

# gcc -O2 -o epoll_pipe_bench epoll_pipe_bench.c -lpthread
# ulimit -n 1000000
# epoll_pipe_bench -n 10000
^C after a while...

oprofile results say that ep_poll_callback() and sys_epoll_wait() use 20% of 
cpu time.
Even if we gain a two factor in cpu time or cache usage, we wont eliminate 
other costs...

oprofile results gave :

Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit 
mask of 0x00 (No unit mask) count 50000
samples  %        symbol name
2015420  11.1309  ep_poll_callback
1867431  10.3136  pipe_writev
1791872   9.8963  sys_epoll_wait
1357297   7.4962  fget_light
1277515   7.0556  pipe_readv
998447    5.5143  current_fs_time
801597    4.4271  __mark_inode_dirty
755268    4.1713  __wake_up
587065    3.2423  __write_lock_failed
582931    3.2195  system_call
297132    1.6410  iov_fault_in_pages_read
296136    1.6355  sys_write
290106    1.6022  __wake_up_common
270692    1.4950  bad_pipe_w
261516    1.4443  do_pipe
257208    1.4205  tg3_start_xmit_dma_bug
254917    1.4079  pipe_poll
252925    1.3969  copy_user_generic_c
234212    1.2935  generic_pipe_buf_map
228659    1.2629  ret_from_sys_call
212541    1.1738  sysret_check
166529    0.9197  sys_read
160038    0.8839  vfs_write
151091    0.8345  pipe_ioctl
136301    0.7528  file_update_time
107173    0.5919  tg3_poll
77846     0.4299  ipt_do_table
75081     0.4147  schedule
73059     0.4035  vfs_read
69787     0.3854  get_task_comm
63923     0.3530  memcpy
60019     0.3315  touch_atime
57490     0.3175  eventpoll_release_file
56152     0.3101  tg3_write_flush_reg32
54468     0.3008  rw_verify_area
47833     0.2642  generic_pipe_buf_unmap
47777     0.2639  __switch_to
44106     0.2436  bad_pipe_r
41824     0.2310  proc_nr_files
41319     0.2282  pipe_iov_copy_from_user


Eric


[-- Attachment #2: epoll_pipe_bench.c --]
[-- Type: text/plain, Size: 2424 bytes --]

/*
 * How to stress epoll
 *
 * This program uses many pipes and two threads.
 * First we open as many pipes we can. (see ulimit -n)
 * Then we create a worker thread.
 * The worker thread will send bytes to random pipes.
 * The main thread uses epoll to collect ready pipes and read them.
 * Each second, a number of collected bytes is printed on stderr
 *
 * Usage : epoll_bench [-n X]
 */
#include <pthread.h>
#include <stdlib.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/epoll.h>
#include <signal.h>
#include <unistd.h>
#include <sys/time.h>

int nbpipes = 1024;

struct pipefd {
	int fd[2];
} *tab;

int epoll_fd;

static int alloc_pipes()
{
	int i;

	epoll_fd = epoll_create(nbpipes);
	if (epoll_fd == -1) {
		perror("epoll_create");
		return -1;
	}
	tab = malloc(sizeof(struct pipefd) * nbpipes);
	if (tab ==NULL) {
		perror("malloc");
		return -1;
	}
	for (i = 0 ; i < nbpipes ; i++) {
			struct epoll_event ev;
		if (pipe(tab[i].fd) == -1)
			break;
		ev.events = EPOLLIN | EPOLLOUT | EPOLLHUP | EPOLLPRI | EPOLLET;
		ev.data.u64 = (uint64_t)i;
		epoll_ctl(epoll_fd, EPOLL_CTL_ADD, tab[i].fd[0], &ev);
	}
	nbpipes = i;
	printf("%d pipes setup\n", nbpipes);
	return 0;
}


unsigned long nbhandled;
static void timer_func()
{
	char buffer[32];
	size_t len;
	static unsigned long old;
	unsigned long delta = nbhandled - old;
	old = nbhandled;
	len = sprintf(buffer, "%lu\n", delta);
	write(2, buffer, len);
}

static void timer_setup()
{
	struct itimerval it;
	struct sigaction sg;

	memset(&sg, 0, sizeof(sg));
	sg.sa_handler = timer_func;
	sigaction(SIGALRM, &sg, 0);
	it.it_interval.tv_sec = 1;
	it.it_interval.tv_usec = 0;
	it.it_value.tv_sec = 1;
	it.it_value.tv_usec = 0;
	if (setitimer(ITIMER_REAL, &it, 0))
		perror("setitimer");
}

static void * worker_thread_func(void *arg)
{
	int fd;
	char c = 1;
	for (;;) {
		fd = rand() % nbpipes;
		write(tab[fd].fd[1], &c, 1);
	}
}


int main(int argc, char *argv[])
{
	char buff[1024];
	pthread_t tid;
	int c;

	while ((c = getopt(argc, argv, "n:")) != EOF) {
		if (c == 'n') nbpipes = atoi(optarg);
	}
	alloc_pipes();
	pthread_create(&tid, NULL, worker_thread_func, (void *)0);
	timer_setup();

	for (;;) {
		struct epoll_event events[128];
		int nb = epoll_wait(epoll_fd, events, 128, 10000);
		int i, fd;
		for (i = 0 ; i < nb ; i++) {
			fd = tab[events[i].data.u64].fd[0];
			if (read(fd, buff, 1024) > 0)
				nbhandled++;
		}
	}
}

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Wed, Nov 01, 2006 at 06:12:41PM -0800, Nate Diller (nate.diller@gmail.com) wrote:
> Indesiciveness has certainly been an issue here, but I remember akpm
> and Ulrich both giving concrete suggestions.  I was particularly
> interested in Andrew's request to explain and justify the differences
> between kevent and BSD's kqueue interface.  Was there a discussion
> that I missed?  I am very interested to see your work on this
> mechanism merged, because you've clearly emphasized performance and
> shown impressive results.  But it seems like we lose out on a lot by
> throwing out all the applications that already use kqueue.

It looks you missed that discussion - freebsd kqueue has fields in the 
kevent structure which have diffent sizes in 32 and 64 bit environments.

> NATE

-- 
	Evgeniy Polyakov

re: [take22 0/4] kevent: Generic event handling mechanism.

[zhou drangon]

performance is great, and we are exciting at the result.

I want to know why there can be so much improvement, can we improve epoll too ?

for Kevent, I am more interesting at a universal event machanism.
In one interface, we can wait for timer event, socket event, and disk AIO event,
this can make the userland application easier to handle multiple event.


2006/11/2, Nate Diller <nate.diller@gmail.com>:
> On 11/1/06, Evgeniy Polyakov <johnpol@2ka.mipt.ru> wrote:
> > On Wed, Nov 01, 2006 at 06:20:43PM +0000, Oleg Verych (olecom@flower.upol.cz) wrote:
> > >
> > > Hallo, Evgeniy Polyakov.
> >
> > Hello, Oleg.
> >
> > > On 2006-11-01, you wrote:
> > > []
> > > >> Quantifying "how much more scalable" would be nice, as would be some
> > > >> example where it is useful. ("It makes my webserver twice as fast on
> > > >> monster 64-cpu box").
> > > >
> > > > Trivial kevent web-server can handle 3960+ req/sec on Xeon 2.4Ghz with
> > > [...]
> > >
> > > Seriously. I'm seeing that patches also. New, shiny, always ready "for
> > > inclusion". But considering kernel (linux in this case) as not thing
> > > for itself, i want to ask following question.
> > >
> > > Where's real-life application to do configure && make && make install?
> >
> > Your real life or mine as developer?
> > I fortunately do not know anything about your real life, but my real life
> > applications can be found on project's homepage.
> > There is a link to archive there, where you can find plenty of sources.
> > You likely do not know, but it is a bit risky business to patch all
> > existing applications to show that approach is correct, if
> > implementation is not completed.
> > You likely do not know, but after I first time announced kevents in
> > February I changed interfaces 4 times - and it is just interfaces, not
> > including numerous features added/removed by developer's requests.
> >
> > > There were some comments about laking much of such programs, answers were
> > > "was in prev. e-mail", "need to update them", something like that.
> > > "Trivial web server" sources url, mentioned in benchmark isn't pointed
> > > in patch advertisement. If it was, should i actually try that new
> > > *trivial* wheel?
> >
> > Answer is trivial - there is archive where one can find a source code
> > (filenames are posted regulary). Should I create a rpm? For what glibc
> > version?
> >
> > > Saying that, i want to give you some short examples, i know.
> > > *Linux kernel <-> userspace*:
> > > o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;
> >
> > iproute documentation was way too bad when Alexey presented it first
> > time :)
> >
> > > o Maxim Krasnyansky tun net driver <-> vtun daemon application;
> > >
> > > *Glibc with mister Drepper* has huge set of tests, please search for
> > > `tst*' files in the sources.
> >
> > Btw, show me splice() 'shiny' application? Does lighttpd use it?
> > Or move_pages().
> >
> > > To make a little hint to you, Evgeniy, why don't you find a little
> > > animal in the open source zoo to implement little interface to
> > > proposed kernel subsystem and then show it to The Big Jury (not me),
> > > we have here? And i can not see, how you've managed to implement
> > > something like that having almost nothing on the test basket.
> > > Very *suspicious* ch.
> >
> > There are always people who do not like something, what can I do with
> > it? I present the code, we discuss it, I ask for inclusion (since it is
> > the only way to get feedback), something requires changes, it is changed
> > and so on - it is development process.
> > I created 'little animal in the open source zoo' by myself to show how
> > simple kevents are.
> >
> > > One, that comes in mind is lighthttpd <http://www.lighttpd.net/>.
> > > It had sub-interface for event systems like select,poll,epoll, when i
> > > checked its sources last time. And it is mature, btw.
> >
> > As I already told several times, I changed only interfaces 4 times
> > already, since no one seems to know what we really want and how
> > interface should look like.
>
> Indesiciveness has certainly been an issue here, but I remember akpm
> and Ulrich both giving concrete suggestions.  I was particularly
> interested in Andrew's request to explain and justify the differences
> between kevent and BSD's kqueue interface.  Was there a discussion
> that I missed?  I am very interested to see your work on this
> mechanism merged, because you've clearly emphasized performance and
> shown impressive results.  But it seems like we lose out on a lot by
> throwing out all the applications that already use kqueue.
>
> NATE
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Nate Diller]

On 11/1/06, Evgeniy Polyakov <johnpol@2ka.mipt.ru> wrote:
> On Wed, Nov 01, 2006 at 06:20:43PM +0000, Oleg Verych (olecom@flower.upol.cz) wrote:
> >
> > Hallo, Evgeniy Polyakov.
>
> Hello, Oleg.
>
> > On 2006-11-01, you wrote:
> > []
> > >> Quantifying "how much more scalable" would be nice, as would be some
> > >> example where it is useful. ("It makes my webserver twice as fast on
> > >> monster 64-cpu box").
> > >
> > > Trivial kevent web-server can handle 3960+ req/sec on Xeon 2.4Ghz with
> > [...]
> >
> > Seriously. I'm seeing that patches also. New, shiny, always ready "for
> > inclusion". But considering kernel (linux in this case) as not thing
> > for itself, i want to ask following question.
> >
> > Where's real-life application to do configure && make && make install?
>
> Your real life or mine as developer?
> I fortunately do not know anything about your real life, but my real life
> applications can be found on project's homepage.
> There is a link to archive there, where you can find plenty of sources.
> You likely do not know, but it is a bit risky business to patch all
> existing applications to show that approach is correct, if
> implementation is not completed.
> You likely do not know, but after I first time announced kevents in
> February I changed interfaces 4 times - and it is just interfaces, not
> including numerous features added/removed by developer's requests.
>
> > There were some comments about laking much of such programs, answers were
> > "was in prev. e-mail", "need to update them", something like that.
> > "Trivial web server" sources url, mentioned in benchmark isn't pointed
> > in patch advertisement. If it was, should i actually try that new
> > *trivial* wheel?
>
> Answer is trivial - there is archive where one can find a source code
> (filenames are posted regulary). Should I create a rpm? For what glibc
> version?
>
> > Saying that, i want to give you some short examples, i know.
> > *Linux kernel <-> userspace*:
> > o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;
>
> iproute documentation was way too bad when Alexey presented it first
> time :)
>
> > o Maxim Krasnyansky tun net driver <-> vtun daemon application;
> >
> > *Glibc with mister Drepper* has huge set of tests, please search for
> > `tst*' files in the sources.
>
> Btw, show me splice() 'shiny' application? Does lighttpd use it?
> Or move_pages().
>
> > To make a little hint to you, Evgeniy, why don't you find a little
> > animal in the open source zoo to implement little interface to
> > proposed kernel subsystem and then show it to The Big Jury (not me),
> > we have here? And i can not see, how you've managed to implement
> > something like that having almost nothing on the test basket.
> > Very *suspicious* ch.
>
> There are always people who do not like something, what can I do with
> it? I present the code, we discuss it, I ask for inclusion (since it is
> the only way to get feedback), something requires changes, it is changed
> and so on - it is development process.
> I created 'little animal in the open source zoo' by myself to show how
> simple kevents are.
>
> > One, that comes in mind is lighthttpd <http://www.lighttpd.net/>.
> > It had sub-interface for event systems like select,poll,epoll, when i
> > checked its sources last time. And it is mature, btw.
>
> As I already told several times, I changed only interfaces 4 times
> already, since no one seems to know what we really want and how
> interface should look like.

Indesiciveness has certainly been an issue here, but I remember akpm
and Ulrich both giving concrete suggestions.  I was particularly
interested in Andrew's request to explain and justify the differences
between kevent and BSD's kqueue interface.  Was there a discussion
that I missed?  I am very interested to see your work on this
mechanism merged, because you've clearly emphasized performance and
shown impressive results.  But it seems like we lose out on a lot by
throwing out all the applications that already use kqueue.

NATE

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Wed, Nov 01, 2006 at 06:20:43PM +0000, Oleg Verych (olecom@flower.upol.cz) wrote:
> 
> Hallo, Evgeniy Polyakov.

Hello, Oleg.

> On 2006-11-01, you wrote:
> []
> >> Quantifying "how much more scalable" would be nice, as would be some
> >> example where it is useful. ("It makes my webserver twice as fast on
> >> monster 64-cpu box").
> >
> > Trivial kevent web-server can handle 3960+ req/sec on Xeon 2.4Ghz with
> [...]
> 
> Seriously. I'm seeing that patches also. New, shiny, always ready "for
> inclusion". But considering kernel (linux in this case) as not thing
> for itself, i want to ask following question.
> 
> Where's real-life application to do configure && make && make install?

Your real life or mine as developer?
I fortunately do not know anything about your real life, but my real life
applications can be found on project's homepage.
There is a link to archive there, where you can find plenty of sources.
You likely do not know, but it is a bit risky business to patch all
existing applications to show that approach is correct, if
implementation is not completed.
You likely do not know, but after I first time announced kevents in
February I changed interfaces 4 times - and it is just interfaces, not
including numerous features added/removed by developer's requests.

> There were some comments about laking much of such programs, answers were
> "was in prev. e-mail", "need to update them", something like that.
> "Trivial web server" sources url, mentioned in benchmark isn't pointed
> in patch advertisement. If it was, should i actually try that new
> *trivial* wheel?

Answer is trivial - there is archive where one can find a source code
(filenames are posted regulary). Should I create a rpm? For what glibc
version?

> Saying that, i want to give you some short examples, i know.
> *Linux kernel <-> userspace*:
> o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;

iproute documentation was way too bad when Alexey presented it first 
time :)

> o Maxim Krasnyansky tun net driver <-> vtun daemon application;
>
> *Glibc with mister Drepper* has huge set of tests, please search for
> `tst*' files in the sources.

Btw, show me splice() 'shiny' application? Does lighttpd use it?
Or move_pages().

> To make a little hint to you, Evgeniy, why don't you find a little
> animal in the open source zoo to implement little interface to
> proposed kernel subsystem and then show it to The Big Jury (not me),
> we have here? And i can not see, how you've managed to implement
> something like that having almost nothing on the test basket.
> Very *suspicious* ch.

There are always people who do not like something, what can I do with
it? I present the code, we discuss it, I ask for inclusion (since it is
the only way to get feedback), something requires changes, it is changed
and so on - it is development process.
I created 'little animal in the open source zoo' by myself to show how
simple kevents are.

> One, that comes in mind is lighthttpd <http://www.lighttpd.net/>.
> It had sub-interface for event systems like select,poll,epoll, when i
> checked its sources last time. And it is mature, btw.

As I already told several times, I changed only interfaces 4 times
already, since no one seems to know what we really want and how
interface should look like. You suggest to patch lighttpd? Well, it is
doable, but then I will be asked to change apache and nginx. And then
someone will suggest to change order of parameters. Will you help me
rewrite userspace? No, you will not. You asks for something without
providing anything back (not getting into account code, but discussion,
ideas, testing time, nothing), and you do it in ultimate manner.
Btw, kevent also support AIO notifications - do you suggest to patch
reactor/proactor for tests?
It supports network AIO - do you suggest to write support for that into
apache?
What about timers? It is possible to rewrite all POSIX timers users to
usem instead.
There is feature request for userspace events and singal delivery - what
to do with that?

I created trivial web servers, which send single static page and use
various event handling schemes, and I test new subsystem with new tools,
when tests are completed and all requested features are implemented it
is time to work on different more complex users.

So let's at least complete what we have right now, so no developer's
efforts could be wasted writing empty chars in various places.

> Cheers.
> 
> [ -*- OT -*-                                                           ]
> [ I wouldn't write all this, unless saw your opinion about the         ]
> [ reportbug (part of the Debian Bug Tracking System) this week.        ]
> [ While i'm nobody here, imho, the first thing about good programmer   ]
> [ must be, that he is excellent user.                                  ]
> ____

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Oleg Verych]

Hallo, Evgeniy Polyakov.

On 2006-11-01, you wrote:
[]
>> Quantifying "how much more scalable" would be nice, as would be some
>> example where it is useful. ("It makes my webserver twice as fast on
>> monster 64-cpu box").
>
> Trivial kevent web-server can handle 3960+ req/sec on Xeon 2.4Ghz with
[...]

Seriously. I'm seeing that patches also. New, shiny, always ready "for
inclusion". But considering kernel (linux in this case) as not thing
for itself, i want to ask following question.

Where's real-life application to do configure && make && make install?

There were some comments about laking much of such programs, answers were
"was in prev. e-mail", "need to update them", something like that.
"Trivial web server" sources url, mentioned in benchmark isn't pointed
in patch advertisement. If it was, should i actually try that new
*trivial* wheel?

Saying that, i want to give you some short examples, i know.
*Linux kernel <-> userspace*:
o Alexey Kuznetsov  networking     <-> (excellent) iproute set of utilities;
o Maxim Krasnyansky tun net driver <-> vtun daemon application;

*Glibc with mister Drepper* has huge set of tests, please search for
`tst*' files in the sources.

To make a little hint to you, Evgeniy, why don't you find a little
animal in the open source zoo to implement little interface to
proposed kernel subsystem and then show it to The Big Jury (not me),
we have here? And i can not see, how you've managed to implement
something like that having almost nothing on the test basket.
Very *suspicious* ch.

One, that comes in mind is lighthttpd <http://www.lighttpd.net/>.
It had sub-interface for event systems like select,poll,epoll, when i
checked its sources last time. And it is mature, btw.

Cheers.

[ -*- OT -*-                                                           ]
[ I wouldn't write all this, unless saw your opinion about the         ]
[ reportbug (part of the Debian Bug Tracking System) this week.        ]
[ While i'm nobody here, imho, the first thing about good programmer   ]
[ must be, that he is excellent user.                                  ]
____

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Wed, Nov 01, 2006 at 05:05:51PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> Hi!

Hi Pavel.

> > Kevent is a generic subsytem which allows to handle event notifications.
> > It supports both level and edge triggered events. It is similar to
> > poll/epoll in some cases, but it is more scalable, it is faster and
> > allows to work with essentially eny kind of events.
> 
> Quantifying "how much more scalable" would be nice, as would be some
> example where it is useful. ("It makes my webserver twice as fast on
> monster 64-cpu box").

Trivial kevent web-server can handle 3960+ req/sec on Xeon 2.4Ghz with
1Gb RAM, epoll based - 2200-2500 req/sec.
100 Mbit wire is filled almost 100% (10582.7 KB/s of data without
TCP and below headers).
More benchmarks created by me and Johann Borck can be found on project's 
homepage as long as all my sources used in tests.

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[James Morris]

On Wed, 1 Nov 2006, Pavel Machek wrote:

> Hi!
> 
> > Generic event handling mechanism.
> > 
> > Consider for inclusion.
> > 
> > Changes from 'take21' patchset:
> 
> We are not interrested in how many times you spammed us, nor we want
> to know what was wrong in previous versions. It would be nice to have
> short summary of what this is good for, instead.

I'm interested in knowing which version the patches belong to and what has 
changed (geez, it's rare enough that someone actually bothers to do this 
with an updated patchset, and to complain about it?)



- James
-- 
James Morris
<jmorris@namei.org>

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

> > > Generic event handling mechanism.
> > > 
> > > Consider for inclusion.
> > > 
> > > Changes from 'take21' patchset:
> > 
> > We are not interrested in how many times you spammed us, nor we want
> > to know what was wrong in previous versions. It would be nice to have
> > short summary of what this is good for, instead.
> 
> Let me guess, short explaination in subsequent emails is not
> enough...

Yes.

> Kevent is a generic subsytem which allows to handle event notifications.
> It supports both level and edge triggered events. It is similar to
> poll/epoll in some cases, but it is more scalable, it is faster and
> allows to work with essentially eny kind of events.

Quantifying "how much more scalable" would be nice, as would be some
example where it is useful. ("It makes my webserver twice as fast on
monster 64-cpu box").

> Events are provided into kernel through control syscall and can be read
> back through mmaped ring or syscall.
> Kevent update (i.e. readiness switching) happens directly from internals
> of the appropriate state machine of the underlying subsytem (like
> network, filesystem, timer or any other).
> 
> I will put that text into introduction message.

Thanks.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

On Wed, Nov 01, 2006 at 02:06:14PM +0100, Pavel Machek (pavel@ucw.cz) wrote:
> Hi!
> 
> > Generic event handling mechanism.
> > 
> > Consider for inclusion.
> > 
> > Changes from 'take21' patchset:
> 
> We are not interrested in how many times you spammed us, nor we want
> to know what was wrong in previous versions. It would be nice to have
> short summary of what this is good for, instead.

Let me guess, short explaination in subsequent emails is not enough...
If changelog will be removed, then how people will detect what happend 
after previous release?

Kevent is a generic subsytem which allows to handle event notifications.
It supports both level and edge triggered events. It is similar to
poll/epoll in some cases, but it is more scalable, it is faster and
allows to work with essentially eny kind of events.

Events are provided into kernel through control syscall and can be read
back through mmaped ring or syscall.
Kevent update (i.e. readiness switching) happens directly from internals
of the appropriate state machine of the underlying subsytem (like
network, filesystem, timer or any other).

I will put that text into introduction message.

> 								Pavel
> -- 
> Thanks, Sharp!

-- 
	Evgeniy Polyakov

Re: [take22 0/4] kevent: Generic event handling mechanism.

[Pavel Machek]

Hi!

> Generic event handling mechanism.
> 
> Consider for inclusion.
> 
> Changes from 'take21' patchset:

We are not interrested in how many times you spammed us, nor we want
to know what was wrong in previous versions. It would be nice to have
short summary of what this is good for, instead.

								Pavel
-- 
Thanks, Sharp!

[take22 0/4] kevent: Generic event handling mechanism.

[Evgeniy Polyakov]

Generic event handling mechanism.

Consider for inclusion.

Changes from 'take21' patchset:
 * minor cleanups (different return values, removed unneded variables, whitespaces and so on)
 * fixed bug in kevent removal in case when kevent being removed
   is the same as overflow_kevent (spotted by Eric Dumazet)

Changes from 'take20' patchset:
 * new ring buffer implementation
 * removed artificial limit on possible number of kevents
With this release and fixed userspace web server it was possible to 
achive 3960+ req/s with client connection rate of 4000 con/s
over 100 Mbit lan, data IO over network was about 10582.7 KB/s, which
is too close to wire speed if we get into account headers and the like.

Changes from 'take19' patchset:
 * use __init instead of __devinit
 * removed 'default N' from config for user statistic
 * removed kevent_user_fini() since kevent can not be unloaded
 * use KERN_INFO for statistic output

Changes from 'take18' patchset:
 * use __init instead of __devinit
 * removed 'default N' from config for user statistic
 * removed kevent_user_fini() since kevent can not be unloaded
 * use KERN_INFO for statistic output

Changes from 'take17' patchset:
 * Use RB tree instead of hash table. 
	At least for a web sever, frequency of addition/deletion of new kevent 
	is comparable with number of search access, i.e. most of the time events 
	are added, accesed only couple of times and then removed, so it justifies 
	RB tree usage over AVL tree, since the latter does have much slower deletion 
	time (max O(log(N)) compared to 3 ops), 
	although faster search time (1.44*O(log(N)) vs. 2*O(log(N))). 
	So for kevents I use RB tree for now and later, when my AVL tree implementation 
	is ready, it will be possible to compare them.
 * Changed readiness check for socket notifications.

With both above changes it is possible to achieve more than 3380 req/second compared to 2200, 
sometimes 2500 req/second for epoll() for trivial web-server and httperf client on the same
hardware.
It is possible that above kevent limit is due to maximum allowed kevents in a time limit, which is
4096 events.

Changes from 'take16' patchset:
 * misc cleanups (__read_mostly, const ...)
 * created special macro which is used for mmap size (number of pages) calculation
 * export kevent_socket_notify(), since it is used in network protocols which can be 
	built as modules (IPv6 for example)

Changes from 'take15' patchset:
 * converted kevent_timer to high-resolution timers, this forces timer API update at
	http://linux-net.osdl.org/index.php/Kevent
 * use struct ukevent* instead of void * in syscalls (documentation has been updated)
 * added warning in kevent_add_ukevent() if ring has broken index (for testing)

Changes from 'take14' patchset:
 * added kevent_wait()
    This syscall waits until either timeout expires or at least one event
    becomes ready. It also commits that @num events from @start are processed
    by userspace and thus can be be removed or rearmed (depending on it's flags).
    It can be used for commit events read by userspace through mmap interface.
    Example userspace code (evtest.c) can be found on project's homepage.
 * added socket notifications (send/recv/accept)

Changes from 'take13' patchset:
 * do not get lock aroung user data check in __kevent_search()
 * fail early if there were no registered callbacks for given type of kevent
 * trailing whitespace cleanup

Changes from 'take12' patchset:
 * remove non-chardev interface for initialization
 * use pointer to kevent_mring instead of unsigned longs
 * use aligned 64bit type in raw user data (can be used by high-res timer if needed)
 * simplified enqueue/dequeue callbacks and kevent initialization
 * use nanoseconds for timeout
 * put number of milliseconds into timer's return data
 * move some definitions into user-visible header
 * removed filenames from comments

Changes from 'take11' patchset:
 * include missing headers into patchset
 * some trivial code cleanups (use goto instead of if/else games and so on)
 * some whitespace cleanups
 * check for ready_callback() callback before main loop which should save us some ticks

Changes from 'take10' patchset:
 * removed non-existent prototypes
 * added helper function for kevent_registered_callbacks
 * fixed 80 lines comments issues
 * added shared between userspace and kernelspace header instead of embedd them in one
 * core restructuring to remove forward declarations
 * s o m e w h i t e s p a c e c o d y n g s t y l e c l e a n u p
 * use vm_insert_page() instead of remap_pfn_range()

Changes from 'take9' patchset:
 * fixed ->nopage method

Changes from 'take8' patchset:
 * fixed mmap release bug
 * use module_init() instead of late_initcall()
 * use better structures for timer notifications

Changes from 'take7' patchset:
 * new mmap interface (not tested, waiting for other changes to be acked)
	- use nopage() method to dynamically substitue pages
	- allocate new page for events only when new added kevent requres it
	- do not use ugly index dereferencing, use structure instead
	- reduced amount of data in the ring (id and flags), 
		maximum 12 pages on x86 per kevent fd

Changes from 'take6' patchset:
 * a lot of comments!
 * do not use list poisoning for detection of the fact, that entry is in the list
 * return number of ready kevents even if copy*user() fails
 * strict check for number of kevents in syscall
 * use ARRAY_SIZE for array size calculation
 * changed superblock magic number
 * use SLAB_PANIC instead of direct panic() call
 * changed -E* return values
 * a lot of small cleanups and indent fixes

Changes from 'take5' patchset:
 * removed compilation warnings about unused wariables when lockdep is not turned on
 * do not use internal socket structures, use appropriate (exported) wrappers instead
 * removed default 1 second timeout
 * removed AIO stuff from patchset

Changes from 'take4' patchset:
 * use miscdevice instead of chardevice
 * comments fixes

Changes from 'take3' patchset:
 * removed serializing mutex from kevent_user_wait()
 * moved storage list processing to RCU
 * removed lockdep screaming - all storage locks are initialized in the same function, so it was
learned 
	to differentiate between various cases
 * remove kevent from storage if is marked as broken after callback
 * fixed a typo in mmaped buffer implementation which would end up in wrong index calcualtion 

Changes from 'take2' patchset:
 * split kevent_finish_user() to locked and unlocked variants
 * do not use KEVENT_STAT ifdefs, use inline functions instead
 * use array of callbacks of each type instead of each kevent callback initialization
 * changed name of ukevent guarding lock
 * use only one kevent lock in kevent_user for all hash buckets instead of per-bucket locks
 * do not use kevent_user_ctl structure instead provide needed arguments as syscall parameters
 * various indent cleanups
 * added optimisation, which is aimed to help when a lot of kevents are being copied from
userspace
 * mapped buffer (initial) implementation (no userspace yet)

Changes from 'take1' patchset:
 - rebased against 2.6.18-git tree
 - removed ioctl controlling
 - added new syscall kevent_get_events(int fd, unsigned int min_nr, unsigned int max_nr,
			unsigned int timeout, void __user *buf, unsigned flags)
 - use old syscall kevent_ctl for creation/removing, modification and initial kevent 
	initialization
 - use mutuxes instead of semaphores
 - added file descriptor check and return error if provided descriptor does not match
	kevent file operations
 - various indent fixes
 - removed aio_sendfile() declarations.

Thank you.

Signed-off-by: Evgeniy Polyakov <johnpol@2ka.mipt.ru>