Using libev with pthreads and one listening socket

Marc Lehmann schmorp at schmorp.de
Mon Mar 14 11:31:45 CET 2011


On Mon, Mar 14, 2011 at 11:47:47AM +1100, Arlen Cuss <celtic at sairyx.org> wrote:
> The problem I'm having is with choosing a method to actually get clients
> on the different threads. I'm using one event loop per thread, and so
> far the best I've been able to come up with is to "round-robin" the
> ev_io watcher on the listening socket itself [2]:

(the leader/follower-pattern)

> limit -- but if not, then my guess is that it's the
> ev_io_stop/ev_io_start, and maybe the ev_asyncs (though they'll be
> unavoidable no matter how I do the round-robin, I think), that's causing
> it to be slower.

If you use linux and epoll, then yes, every start and stop will likely lead
to a syscall in your situation.

and ev_async will add another two syscalls.

and on multiple cores, this might become even slower.

> to other threads? I can imagine I could do that by maintaining a
> list/array of waiting fds in an array for each thread, appending to that
> (via mutex), then triggering the ev_async to tell the thread to pick it
> up. I'm not sure at this stage if that would be slower, but it's
> probably worth testing.

It depends on many factors, so, yes, it is worth testing, especially with
a real-world implementation doing some actual work.

Linux unfortunately has no good event interface - select/poll work nicely
but don't scale, epoll does scale but is otherwise beyond brokenness, and
has a high overhead.

Tough situation :)

However, with luck, your performance issue will evaporate once your proxy
does actual work, as the extra syscall overhead will then reduce.

On Mon, Mar 14, 2011 at 06:14:00PM +1100, Arlen Cuss <celtic at sairyx.org> wrote:
> I've partly resolved my own complain; I've actually started instead to
> watch the listener fd on *all* loops at once. It works on Linux at
> least, though I've yet to try it on my other target (BSD). I'm trying
> for portability, so if it doesn't cut it, I'll have to look elsewhere.

It should *work* everywhere, but performance is another issue. My guess is
that you get one wakeup in the lucky case of an idle thread accepting the
connection quickly enough, and an all-wakeup when all your loops are busy.

Can't imagine that will help with performance, but under very high load,
this basically degrades to polling once per loop, which is probably much
more efficient than any leader/follower scheme.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\



More information about the libev mailing list