multi-threading

Marc Lehmann schmorp at schmorp.de
Tue Jan 29 14:31:21 CET 2008


On Mon, Jan 28, 2008 at 12:07:34PM -0800, Eric Brown <yogieric.list at gmail.com> wrote:
> > In fact, I could envisage that this functionality (signaling a loop
> > asynchronously) could become part of libev proper if there is demand, as
> > libev already has to handle asynchronous signals at leats in one loop
> > and therefore needs the mechanism already (and that way it could e.g. be
> > optimised by using an eventfd or other os-specific means).
> 
> That occurred to me as well.

Yeah, but only later in your mail :)

> I don't like the idea of using SO_REUSEPORT as it just doesn't seem that
> elegant.

I am curious on what you want to achieve with so_reuseport. I don't think it
does what you think it does (besides being highly unportable as well :).

> > It's usually better only to use the pipe as wakeup mechanism, pass only
> > a dummy byte and use regular locking.
> 
> Interesting. Though I'd then have to lock twice - once to add the connection
> to some data structure in the first thread, and once to retrieve the
> connection from the data structure. But it certainly is easier than dealing
> with making sure I read N bytes, etc.

unless you have a lot of contention, most locking is done completely in
userspace, and thus fast. The goal would be to avoid pipe read/writes and
only use it as a fast wake-up mechanism.

However, you should read up on the pipe semantics - if your packets are
small enough (<= 512 bytes) the pipe reads/writes are atomic, so you never
need to care about half reads or half writes. Could be wrong, me memory is
fuzzy, but I think this is in poisx for a long time, and also traditional
behaviour.

Might not work on windows, but pipes don't work there anyways.

> One might even be able to not pass
> > anything, depending on how the other side(s) work.
> 
> I'm not sure I follow.

You might get away with passing only a single byte for >1 connections,
if contention is high, as you only need to write to the pipe if you know
the pipe is empty (libev does that for signals, by usign a global flag to
check wether signal handling is already about to handle, getting the race
right is easy).

(the more I think about it, the more I think we want an async intra-loop
signaling mechanism).

> My targets shifted from bsd to linux now, but I was trying to stay away from
> a pipe in case I ever wanted to move back to kqueue. But, alas, as you say,
> it is hard to stay portable.

pipes generally work for kqueue on !darwin, if you use a current kernel :)

> Lots of connections with keep alive firing requests every so often. Each
> request requires some amount of CPU. Single threaded without processing, I
> handle 35,000 request-responses/second. With processing, it drops to about
> 5000 request-responses/second.

Then new-conenction processing should indeed not be that much of a problem.
You could even do totally weird stuff like running an accept in a loop
in round-robin fashion, e.g. on int(seconds) + 0 == thread 0, int(seconds) +
0.1 == thread 1 etc.. or something like that.

That woudl have a constant overhead of 10 syscalls/s, but would get rid of
any communication overhead.

(thats usually best if you do have quite a lot of connections, but requires
tuning).

if conenctions are rare enough, the using-a-pipe trick might work best
because you don't care about overhead much there and just want to avoid
extra processing in each thread.

> Maybe a thread-pool is the way to go. I'd just thought since I (1) have
> keep-alive and (2) the thread-pool has to re-engage the I/O thread to send
> the response, it would be more logical to have one loop / thread than to use
> a thread-pool.

Yes, indeed. The single i/o thread would be a bottleneck. Having, say, an
acceptor thread that sends down the new fd to some low-conenction-count
thread with it sown i/o loop sounds quite good as a solution.

> I may just have to try it different ways. (I'm just not sure I have time
> before my wife delivers and I have to stop playing around with this stuff.)

Good luck :)

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      pcg at goof.com
      -=====/_/_//_/\_,_/ /_/\_\



More information about the libev mailing list