Using libev with pthreads and one listening socket
pierre-yves.kerembellec at dailymotion.com
Wed Mar 30 12:30:10 CEST 2011
> On Wed, Mar 30, 2011 at 03:27:04PM +1100, Arlen Cuss <celtic at sairyx.org> wrote:
>> I enjoyed reading your (many) comments in the documentation talking
>> about the same :-P, particularly "a frankenpoll" re: epoll.
> It's a direct consequence of my many frustrated workarounds and insightful
> moments ("ugh").
>> This will be deployed on FreeBSD, so the performance characteristics may
>> end up varying anyway.
> It would be interesting how though - kqueeu "suffers" from the same
> inefficiency as epoll in this respect, but there is a big difference:
> everything is done by passing a big changelist into a single syscall,
> while for epoll, each change is one extra syscall.
> This difference can make it work fine.
>> This seems to be the case. Right now I'm still employing the (possibly
>> criminal) method of having all threads listen on the fd - it saves on
>> the syscalls -- maybe I don't need to worry about that anyway -- but
>> it's also a simpler design.
> That it is indeed, and you only need to be fast enough after all.
> Whether it's "criminal" or not mostly depends on what the kernel does or not.
> If your threads are all very busy with other stuff, then you either get no
> wakeup at all (because the kernel checks *again* to see if there are any new
> conenctions) or you get a spurious wakeup, but also some work to do.
> So under enough load, doing it in all threads might just work out.
You may also consider the model where a main process (or thread) will open the listening sockets and
perform the accept() calls (in a dedicated ev_loop), and then distribute the accepted connections to a
pool of processes or threads, in a round-robin or more sophisticated way.
If you're using a pool of worker processes, it's easy to pass an opened fd from the master using socketpair()
and sendmsg(). If you're found of threads, it's quite easy to "inject" notifications in workers's ev_loop() via
I've used both approaches in different projects successfully and this effectively eliminates the thundering
herd effect on accept(). You may search this list or Google for more information on the two approaches.
More information about the libev