Using libev with pthreads and one listening socket

Arlen Cuss celtic at sairyx.org
Wed Mar 30 06:23:41 CEST 2011


Hi Maarten,

You make a very good point. I suppose I was wanting to avoid having to
use mutexes (muticies? ;-)) and ev_async to be constantly stopping the
other threads to hand them work.

It seems to me that the threads *do* seem to get scheduled across the
cores (i.e. it's not a requirement that they're child processes), though
this almost certainly varies from OS [ver] to OS [ver] (tested on Linux
2.6.35). Of course, I'm assuming it's going on all 4 cores judging by
its CPU being ~350% at full speed, likewise the system load approaches
4.0.

Thanks for your insights!

Arlen

On Mon, 2011-03-14 at 09:30 +0100, maarten wrote:
> Hi,
> 
> it seems very strange to listen on one socket with multiple threads
> and/or multiple loops.  To me it feels more natural to only listen once
> and then possibly hand it off to multiple children or threads.  Since
> you got multiple instances using 1 resource, locking comes to mind as
> well.  I don't know the effect this has on sockets, but it seems ugly :)
> 
> I also get the impression that you expect that 4 threads (in 1 process)
> will be evenly distributed on 4 cores, I thought that would only happen
> with child processes. (Not 100% on this though)
> 
> I probably raised more questions than I answered,
> hope I helped anyway.
> 
> regards,
> Maarten
> 
> 
> 
> On Mon, 2011-03-14 at 18:14 +1100, Arlen Cuss wrote:
> > I've partly resolved my own complain; I've actually started instead to
> > watch the listener fd on *all* loops at once. It works on Linux at
> > least, though I've yet to try it on my other target (BSD). I'm trying
> > for portability, so if it doesn't cut it, I'll have to look elsewhere.
> > 
> > I consider this only a partial solution, however, since it appears - at
> > least on Linux - that when a connection is received, about half the time
> > it'll end up triggering one random watcher (ideal), but the other half
> > of the time, all watchers. Of course, it's a matter of whichever thread
> > gets to the trigger first as to who gets the fd (which I don't mind),
> > but it's nicer to be able to avoid the excess computation of the other 3
> > (on a 4-core) threads all trying to accept().
> > 
> > Any suggestions would be warmly appreciated.
> > 
> > Cheers,
> > Arlen
> > 
> > On Mon, 2011-03-14 at 11:47 +1100, Arlen Cuss wrote:
> > > Hi all,
> > > 
> > > I apologise if this question's been asked before -- I've had a Google
> > > around, and haven't found a satisfactory answer (yet).
> > > 
> > > I'm writing a HTTP proxy using libev (code here: [1]), and I'm now
> > > trying to adapt it to run with an arbitrary number of threads, ideally
> > > so that it would be able to use all the available cores/CPUs on a
> > > system.
> > > 
> > > The problem I'm having is with choosing a method to actually get clients
> > > on the different threads. I'm using one event loop per thread, and so
> > > far the best I've been able to come up with is to "round-robin" the
> > > ev_io watcher on the listening socket itself [2]:
> > > 
> > >     if (!terminating && threads > 1) {
> > > 	++current_thread;
> > > 	if (current_thread == threads)
> > > 	    current_thread = 0;
> > > 
> > > 	/* Stop w (listener_watcher) on this loop, and signal the next
> > > 	 * loop to pick it up. */
> > > 	ev_io_stop(EV_A_ w);
> > > 	ev_async_send(thread_loops[current_thread],
> > > 		&thread_asyncs[current_thread]);
> > >     }
> > > 
> > > When the watcher for the listening socket is triggered, we accept()
> > > until there's nothing left to accept, then stop the watcher (in this
> > > loop and thread), and trigger the ev_async for the next thread, which
> > > then uses the same watcher, on the new loop [3]:
> > > 
> > > static void async_handler(EV_P_ ev_async *w, int revents) {
> > > [...]
> > > 	ev_io_start(EV_A_ &listener_watcher);
> > > [...]
> > > }
> > > 
> > > This works, but it actually ends up being *slower* than using a single
> > > thread for the test cases I've come up with so far. It's probably
> > > because, right now, there's not an awful lot of computation (it's not
> > > proxying yet) happening for the clients, so it's possibly an artificial
> > > limit -- but if not, then my guess is that it's the
> > > ev_io_stop/ev_io_start, and maybe the ev_asyncs (though they'll be
> > > unavoidable no matter how I do the round-robin, I think), that's causing
> > > it to be slower.
> > > 
> > > If anyone has any other thoughts for how to divvy up the clients between
> > > threads, I'd be most interested in hearing them. It may involve leaving
> > > the listener's watcher on one thread only, then feeding the accepted fds
> > > to other threads? I can imagine I could do that by maintaining a
> > > list/array of waiting fds in an array for each thread, appending to that
> > > (via mutex), then triggering the ev_async to tell the thread to pick it
> > > up. I'm not sure at this stage if that would be slower, but it's
> > > probably worth testing.
> > > 
> > > Cheers, and thanks to Marc for a great library and most amusing
> > > documentation.
> > > 
> > > Arlen
> > > 
> > > [1] https://github.com/celtic/gadogado2
> > > [2] https://github.com/celtic/gadogado2/blob/master/src/listener.c#L216
> > > [3] https://github.com/celtic/gadogado2/blob/master/src/main.c#L38
> > > _______________________________________________
> > > libev mailing list
> > > libev at lists.schmorp.de
> > > http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
> > 
> > 
> > _______________________________________________
> > libev mailing list
> > libev at lists.schmorp.de
> > http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev





More information about the libev mailing list