announcement: AnyEvent::Fork{,::RPC,::Pool}

[Lee Aylward]

(resending this to the list)

Hi Marc,

I switched a few projects from AnyEvent::Worker to AnyEvent::Fork::Pool, and things seem to be working well so far!

One difference I noticed is error handling. WIth AE::Worker, your callback still gets called if there was an error, but with $@ set. I didn't see any mention of similar functionality for AE::Fork::RPC. Perhaps it exists and I missed it.

http://search.cpan.org/dist/AnyEvent-Worker/lib/AnyEvent/Worker.pm#SYNOPSIS

Additionally, a feature I think would be useful for AE::Fork::Pool is something like "max_requests". It would kill a worker after it handled a certain number of requests. Useful when using leaky libraries like ImageMagick.

-- 
Lee

On Apr 21, 2013, at 3:19 PM, Marc Lehmann <schmorp at schmorp.de> wrote:

> I mentioned it here and there, but now that the code is on CPAN, I'd like
> to announce it here.
> 
> There are a lot of modules on CPAN that create pools of worker processes,
> but all that I have seen have a fatal flaw: they use fork at runtime (and
> from perl) and do processing in the child. fork doesn't work safely in
> programs using pthreads, which is almost every program I write these days,
> and forking from a running program suffers form other hazards, such as
> having ltos of event watchers for things that only make sense in one of
> the process copies.
> 
> So here's my take on the problem:
> 
> AnyEvent::Fork (http://software.schmorp.de/pkg/AnyEvent-Fork)
> 
>   This module execs a new perl interpreter in a safe way, avoiding
>   both the fork-in-perl problem and fork-copies-everything problems.
> 
>   It is also surprisingly fast - despite the overhead, creating a process
>   via AnyEvent::Fork is often cheaper than using fork+exec, as fork has
>   to make a (virtual) copy of your whole process, and with AnyEvent::Fork
>   you can usualy make your copies from smaller processes that only
>   contain the code that you really need.
> 
>   Small example:
> 
>    AnyEvent::Fork
>       ->new
>       ->require ("MyModule")
>       ->run ("MyModule::server", my $cv = AE::cv);
> 
>    my $fh = $cv->recv;
> 
> The module doesn't, however, allow you to talk with the new processes, it
> only allows you to create them and run your code. To talk with them, there
> is:
> 
> AnyEvent::Fork::RPC (http://software.schmorp.de/pkg/AnyEvent-Fork-RPC)
> 
>   This module takes a process created with AnyEvent::Fork, and adds
>   simple RPC functionality, by running a simple server and giving you
>   a code reference that you can use to call a function in the other
>   process. It has a synchronous and an asynchronous backend (the async
>   backend uses AnyEvent and can handle multiple RPC calls in parallel).
>   Supports any serialiser you want.
> 
>   Small example:
> 
>    my $rpc = AnyEvent::Fork
>       ->new
>       ->require ("MyModule")
>       ->AnyEvent::Fork::RPC::run (
>          "MyModule::server",
>       );
> 
>   $rpc->(1, 2, 3, sub {
>       print "MyModule::server returned @_\n";
>   });
> 
> It doesn't, however, create some pool of processes for you or anything like
> that, which is why there is also:
> 
> AnyEvent::Fork::Pool (http://software.schmorp.de/pkg/AnyEvent-Fork-Pool)
> 
>   This uses a process created via AnyEvent::Fork as a template process
>   to create multiple worker processes. The pool of processes resizes
>   dynamically according to load and allows of variety of configurations.
> 
>   Small example:
> 
>    my $pool = AnyEvent::Fork
>       ->new
>       ->require ("MyWorker")
>       ->AnyEvent::Fork::Pool::run (
>            "MyWorker::run",
>            max => 4,
>         );
> 
>    for (1..10) {
>       $pool->(doit => $_, sub {
>          print "MyWorker::run returned @_\n";
>       });
>    }
> 
> Right now, AnyEvent::Fork seems to look pretty stable, and could use some
> real-world testing. AnyEvent::Fork::RPC is more like early beta, and
> AnyEvent::Fork::Pool is more like alpha quality at this point. The API
> might change if I find a bug that requires changing it, which is not at
> all unlikely at the moment.
> 
> The modules are also quite small (sloccount says all three together are
> 604 loc), and since they are completely seperate distributions on CPAN,
> you only pay for what you actually use.
> 
> Even windows us supported, to the extent possible (cygwin doesn't support
> any kind of handle-open or fd passing, and strawberry/activeperl of course
> are not getting better with time, and require the usual silly workarounds
> from your side).
> 
> My idea is not that everybody should go and start using AnyEvent::Fork::Pool -
> you are invited to do your own pools using AnyEvent::Fork::RPC. Or even
> ignore AnyEvent::Fork::RPC, and just use AnyEvent::Fork to create processes
> and do whatever you want with them.
> 
> It would however, be nice if more modules would use AnyEvent::Fork instead
> of some home-grown fork-and-hack-around-or-ignore-problems approach. Mostly
> new modules, as the design of AnyEvent::Fork makes it unsuitable as drop-in
> replacement for existing process pool managers.
> 
> However, modules such as AnyEvent::DBI could easily switch to
> AnyEvent::Fork, or AnyEvent::Fork::RPC, as they don't expose the
> implementation. And the CVS version of Gtk2::CV already uses
> AnyEvent::Fork, got a lot faster, and wastes a lot less memory.
> 
> In any case, as always, I release my code in the hope that is is useful to
> somebody, and can be safely ignored by the rest :)
> 
> -- 
>                The choice of a       Deliantra, the free code+content MORPG
>      -----==-     _GNU_              http://www.deliantra.net
>      ----==-- _       generation
>      ---==---(_)__  __ ____  __      Marc Lehmann
>      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
>      -=====/_/_//_/\_,_/ /_/\_\
> 
> _______________________________________________
> anyevent mailing list
> anyevent at lists.schmorp.de
> http://lists.schmorp.de/mailman/listinfo/anyevent
>