some questions about AnyEvent::Fork

[Marc Lehmann]

On Fri, Apr 19, 2013 at 04:46:50AM +0000, Dietmar Maurer <dietmar at proxmox.com> wrote:
> >From what I see AnyEvent::Fork would solve the problem, but I do not really want to
> allocate a 'TEMPLATE' process.

You can avoid any template process by calling AnyEvent::Fork->new_exec,
but that starts a new perl interpreter every time, and requires it to
compile all modules you need.

The advantage of having a template process is that almost all memory will
be shared.

> In our case we know that we can always fork the 'master', so there is no need to allocate
> an additional Template.

If you can always fork the master (no event loop, no pthreads, nor
anything that could interfere), then... you can always fork the master.

The problem is that you want to fork the children. Forking a process
actively using an event loop and using it in both processes is, of course,
one of the challenges that AnyEvent::Fork wants to help with.

> Do you think we can modify AnyEvent::Fork to work without requiring a TEMPLATE?

If you only ever call new_exec instead of new, then you will always get
exactly one new process. If your tasks are long-running and the memory
doesn't matter, it might be a viable solution.

Now, if you often have a task running, then little would be wrong with a
template process - you can fork it quickly, and apart from the process
slot, it will share most of its data with the child (on normal systems).

You could save on the number of template processes by, for example,
arranging for your master to have one template, and fork tasks from
it. That will complicate things as you have to communicate from
the workers to your master.  I might think about a way how to pass
AnyEvent::Fork objects over a socket somehow, which make that easier.

In any cace, these are roughly your alternatives:

- use normal fork, and somehow deal with the fallout, by e..g not using
  AnyEvent in the workers, or not using AnyEvent in the tasks (and neither
  IO::AIO and so on).
- use new_exec, higher startup time, but no extra processes, but you
  have to serialise things (AnyEvent::Fork::RPC can help).
- use template processes somehow.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\