announcement: AnyEvent::Fork{,::RPC,::Pool}

[Marc Lehmann]

On Wed, Apr 24, 2013 at 03:25:15PM -0700, Lee Aylward <laylward at gmail.com> wrote:
> I switched a few projects from AnyEvent::Worker to AnyEvent::Fork::Pool, and things seem to be working well so far!

Thats quite reassuring, thank you for trying it out!

> One difference I noticed is error handling. WIth AE::Worker,
> your callback still gets called if there was an error, but with
> $@ set. I didn't see any mention of similar functionality for
> AE::Fork::RPC. Perhaps it exists and I missed it.

I don't know if it's in the cpan version yet, but the current docs state
that throwing exceptions is not supported in any way, i.e. you are screwed
when an exception is thrown (and will likely lose unrelated jobs or
worse).

If your code throws exceptions, you are currently expected to put an eval
around it, and somehow signal the error to the caller yourself.

I have no plans to change this right now - you'd have to signal the error
by passing some extra agruments to the callback ($@ might be set already),
or you'd have to clear $@ on each call, both of which I dislike.

I think there are only two clean solutions: use condvars (which can signal an
exception cleanly by re-throwing it on ->recv), or use threads (inversion of
control allows re-throwing the exception at the point where it happens).

Neither of these appeals to me for this kind of "simple" module. So
as long as one can build this on top, I think it's up to the user of
these modules, all of which are meant as building blocks for custom
implementations (but of course, AnyEvent::Fork::Pool least so).

I.e. it should be possible to create your own My::Pool::run function that
wraps the actual function call in an eval and returns it's own $pool
function.

Or, in your case, as an "end user" of the pool module, you could change
your worker to look like:

   sub my::worker {
      my @res = (1, eval {
         ... actual code
      });

      $@ ? (0, "$@") : @res
   }

And then the first callback argument will be 0 ("failure") or 1 ("success").

Or whatever variation of that that suits you best.

Another reason why not to handle exceptions is that AnyEvent does it
neither (so it's consistent :), and that it would be impossible to
implement with the async backend (as even if the exception could be
caught, there would be no way to associate it with the actual job that
caused it).

> Additionally, a feature I think would be useful for AE::Fork::Pool is
> something like "max_requests". It would kill a worker after it handled
> a certain number of requests. Useful when using leaky libraries like
> ImageMagick.

That should be trivial to implement for a change:

   my $count;

   sub worker::function {
      $count++ > 1000
         and AnyEvent::Fork::Pool::retire ();

      ...
   }

The advantage of this way is that you are not limited by some hardcoded
algorithm (number of requests, which isn't what you want in your example
anyway, but is a good approximation), but can use anything to decide, for
example, looking at actual memory usage and so on.

The disadvantage is that it isn't exact - "retire" just asks the parent to
shut down the worker as soon as possible, but it will still have to handle
any extra requests that are thrown at it, but I can't come up with a case
where this is a real issue (and could be worked around by setting "load"
to 1).

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\