[eio] groups? multiple libev loops?

Marc Lehmann schmorp at schmorp.de
Fri Oct 17 16:11:13 CEST 2008


On Fri, Oct 17, 2008 at 03:19:36PM +0200, ryah dahl <ry at tinyclouds.org> wrote:
> 1. Are groups used to execute several requests sequentially in one thread?

No, they are just a method to organise higher-level functions AND/OR to
conserve resources.

For example, in IO::AIO (my only eio user), there are so-called
"composite" requests, sich as aio_load, which loads a file into memory:

   my $grp = aio_group $cb;

   add $grp aio_open $path, O_RDONLY, 0, sub {
      my $fh = shift;

      add $grp aio_read $fh, 0, (-s $fh), $$data, 0, sub {
         $grp->result ($_[0]);
      };
   };

   return $grp

It creates a group simply to e able to return a *single* request to the
caller, even though it executes multiple ones. It's a simple but effective
synchronisation mechanism "make a request that finishes when these other
requests have finished".

The other usage is using a feeder, to limit the number of parallel
requests, for example, in IO::AIO, the aio_scandir function uses this:

   my $statgrp = aio_group sub {
      $grp->result (\@dirs, \@nondirs);
   };

   # never more than $maxreq parallel requests
   limit $statgrp $maxreq;
   feed $statgrp sub {
      # stat a file
      return unless @$entries;
      my $entry = pop @$entries;

      aioreq_pri $pri;
      add $statgrp aio_stat "$path/$entry/.", sub {
         ...

In here, the group also signals completion, but there is a feeder on it,
and libeio will try to keep $maxreq requests in flight.

Neither use affects ordering strongly.


> 2. Do you have a suggestion for how to use eio with multiple ev loops?

No good ones.

> By doing the pipe trick, like in your example, then adding ev_io
> watchers on the read end of the pipe, one can poll.

Hmm, that's not completely clear to me, multiple I/O watchers? that could
work, but you have to make sure only one thread of control calls into
eio_poll.

(This migth not be neecssary - it might be that it is perfetcly fine to
poll from multiple threads, but I haven't checked this or tried to make it
work).

> The only problem is that suppose you have the following situation. two
> ev_loops one of which has sent a bunch of eio requests - and then
> stops looping (perhaps it was executed with ev_loop(loop,
> EV_ONESHOT)). when the other loop is executed - all of the first
> loop's eio requests get handled despite the fact that the other loop
> is frozen for a while. although i have no evidence of this actually
> being a problem, it seems it could be problematic.

How could this be problematic? Since you have no control over which loop
handles the results (there is only one result queue), you do have to
handle this in any case, in which acse it shouldn't be problematic? :)

> Do I have this right? Is there a different way to use eio with
> multiple ev loops?

I can only imagine variations on the above theme: for example, you could
have one (multiplexer) thread exclusively poll for results, store the
ev_async watcher to wake up in each request, as well as your own queueing
mechanism, so that thread would enqueue the result and wake up the
corresponding loop.

The principal problem is that there is only one result queue.

This could be lifted by creating a result queue abstraction each with its
own wake-up callback and result queue, but none of that exists in libeio
at the moment.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      pcg at goof.com
      -=====/_/_//_/\_,_/ /_/\_\



More information about the libev mailing list