accept()ing socket connections with level triggered epoll

accept()ing socket connections with level triggered epoll

[kyle]

Hello,

I think I may have found a bug in Linux's implementation of epoll.  My
program creates a server socket that listens for incoming SOCK_STREAM
connections.  It uses epoll to wait for notification of a new connection
(and also to handle the client sockets).  While the client sockets use edge
triggered epoll, for performance reasons, the server socket uses level
triggered epoll.

I have found that when I open connections to my program very quickly, it is
sometimes possible to call accept more than once before reaching the point
where no more connections are available and EAGAIN is returned.  If I return
to epoll_wait without accepting all of the available connections, I should
immediately be notified that a read is still available on the server socket,
since I am using level triggered epoll for that descriptor (at least that is
my understanding of how all of this is supposed to work ;).  However, epoll
does not make this notification.  Even if the program accepts further
incoming connections, the missed connection is never accepted, and
eventually times out on the client side.

Kernel version is 2.6.9.  I can provide test code if needed.

Thanks,
Kyle Cronan
<kyle@pbx.org>

Re: accept()ing socket connections with level triggered epoll

[Heikki Orsila]

On Wed, Apr 26, 2006 at 02:55:57PM -0600, kyle@pbx.org wrote:
> Kernel version is 2.6.9.  I can provide test code if needed.

Does it happen with the latest kernel? And if so, please provide a test
code.

-- 
Heikki Orsila                   Barbie's law:
heikki.orsila@iki.fi            "Math is hard, let's go shopping!"
http://www.iki.fi/shd

Re: accept()ing socket connections with level triggered epoll

[Davide Libenzi]

On Wed, 26 Apr 2006, kyle@pbx.org wrote:

> I think I may have found a bug in Linux's implementation of epoll.  My
> program creates a server socket that listens for incoming SOCK_STREAM
> connections.  It uses epoll to wait for notification of a new connection
> (and also to handle the client sockets).  While the client sockets use edge
> triggered epoll, for performance reasons, the server socket uses level
> triggered epoll.
>
> I have found that when I open connections to my program very quickly, it is
> sometimes possible to call accept more than once before reaching the point
> where no more connections are available and EAGAIN is returned.  If I return
> to epoll_wait without accepting all of the available connections, I should
> immediately be notified that a read is still available on the server socket,
> since I am using level triggered epoll for that descriptor (at least that is
> my understanding of how all of this is supposed to work ;).  However, epoll
> does not make this notification.  Even if the program accepts further
> incoming connections, the missed connection is never accepted, and
> eventually times out on the client side.
>
> Kernel version is 2.6.9.  I can provide test code if needed.

Correct, if it's LT you have to get the event because before returning 
from epoll_wait(), the event is automatically re-armed if f_op->poll() 
returns it. Can you post the *minimal* test code for this case?



- Davide

Re: accept()ing socket connections with level triggered epoll

[Willy Tarreau]

Hello,

On Wed, Apr 26, 2006 at 02:55:57PM -0600, kyle@pbx.org wrote:
> Hello,
> 
> I think I may have found a bug in Linux's implementation of epoll.  My
> program creates a server socket that listens for incoming SOCK_STREAM
> connections.  It uses epoll to wait for notification of a new connection
> (and also to handle the client sockets).  While the client sockets use edge
> triggered epoll, for performance reasons, the server socket uses level
> triggered epoll.
> 
> I have found that when I open connections to my program very quickly, it is
> sometimes possible to call accept more than once before reaching the point
> where no more connections are available and EAGAIN is returned.  If I return
> to epoll_wait without accepting all of the available connections, I should
> immediately be notified that a read is still available on the server socket,
> since I am using level triggered epoll for that descriptor (at least that is
> my understanding of how all of this is supposed to work ;).  However, epoll
> does not make this notification.  Even if the program accepts further
> incoming connections, the missed connection is never accepted, and
> eventually times out on the client side.

I find this very strange because if your program accepts other connections,
I don't see how it could "select" some connections and ignore others. The
accept() call returns the next connection(s) in the listen queue. Stupid
question : are you sure that you don't miss anything in the loop around
accept() ? eg: reinitialise one error code or anything which could prevent
accept() from being further called after you have successfully done several
accept() at once ? I'm personnally using epoll in level triggered mode
in haproxy, which often does multiple accept() per call on very high loads
(>10k sessions/s), and although I've encountered difficult beginnings with
epoll, it's rock solid now.

> Kernel version is 2.6.9.  I can provide test code if needed.

I would suggest trying 2.6.16 first to see if it may be related to a bug
which has been fixed since then, and otherwise, some test code would help.

> Thanks,
> Kyle Cronan
> <kyle@pbx.org>

Cheers,
Willy

Re: accept()ing socket connections with level triggered epoll

[kyle]

On Wed, Apr 26, 2006 at 03:14:16PM -0700, Davide Libenzi wrote:
> 
> Correct, if it's LT you have to get the event because before returning from 
> epoll_wait(), the event is automatically re-armed if f_op->poll() returns it. 
> Can you post the *minimal* test code for this case?
> 
> - Davide
> 

I tried reducing the code I have to the minimum necessary to demonstrate the
problem.  It went away, I'm afraid.  Since I'm already aware of a workaround
(call accept in a loop until you get EAGAIN), I guess I'll just forget about
it.  Unfortunately I can't post the full code, not that you'd want to dig
through all of it anyway.

Thanks to everyone that responded.

Kyle
<kyle@pbx.org>

Re: accept()ing socket connections with level triggered epoll

[Willy Tarreau]

On Wed, Apr 26, 2006 at 06:05:20PM -0600, kyle@pbx.org wrote:
> On Wed, Apr 26, 2006 at 03:14:16PM -0700, Davide Libenzi wrote:
> > 
> > Correct, if it's LT you have to get the event because before returning from 
> > epoll_wait(), the event is automatically re-armed if f_op->poll() returns it. 
> > Can you post the *minimal* test code for this case?
> > 
> > - Davide
> > 
> 
> I tried reducing the code I have to the minimum necessary to demonstrate the
> problem.  It went away, I'm afraid.  Since I'm already aware of a workaround
> (call accept in a loop until you get EAGAIN), I guess I'll just forget about
> it.  Unfortunately I can't post the full code, not that you'd want to dig
> through all of it anyway.

Sorry, I misunderstood you the first time. I thought that it was *when*
your accept looped that you encountered the problem. If you need performance,
I *really* encourage you to loop on accept() as much as you can. Missing an
accept() and reading EAGAIN is cheap, while looping through all your event
loop is usually more expensive. In haproxy, I had performance problems 5
years ago, I could not get above 1500-2000 sessions/s because I was doing
one accept at a time. After putting a small "while" loop around, it
immediately jumped over 10000.

Regards,
Willy