AE Fork Pool reaping problem

[Michael Wright]

On Tue, Mar 21, 2017 at 10:41 AM, Marc Lehmann <schmorp at schmorp.de> wrote:
>
> Hi!
>
> On Wed, Mar 15, 2017 at 11:50:03AM +0000, Michael Wright <mjw at methodanalysis.com> wrote:
> > I've been using AnyEvent::Fork::Pool for a while now in a Mojolicious
> > server environment, and it works well, but I recently introduced retirement
> > of worker processes after a certain number of jobs, and discovered this
> > results in zombies.
> >
> > Not every retirement does, some are clearly reaped fine, and the ratio
> > of defunct processes to retirements does vary quite a bit (no clue
> > why / according to what), but there are a lot.
> >
> > After requesting retirement the child receives and completes remaining
> > jobs, andthe worker return value of the last job, is received fine by the
> > parent, a new worker is spawned, it's just that the old process is still
> > there, some of the time.
> >
> > Defunct process never result from the worker pool spinning down
> > processes because it is idle, only ever when workers request to be retired.
> >
> > Any ideas on what might be the problem?
>
> What is the parent process of the zombie (use ps to find out)? If it's
> one of the processes forked by AnyEvent::Fork, then you have to find out
> why the kernel doesn't reap them - AnyEvent::Fork sets SIGCHLD to IGNORE,
> which means automatic reaping of zombies, but if you take over the process
> itself (using ->run), then SIGCHLD is restored to the previous value, and
> your code would be responsible for reaping children for example. Likewise
> if e.g. SIGCHLD is blocked.
>
> As long as the parent is one of AnyEvent::Fork's template processes or the
> template process has exited, there is no way you should get zombies unless
> your own code somehow enables it.


Hi, thank you for your reply, I've just been revisiting this issue and trying
to create a minimal example, and I've come up with this 'defuncter.pl':

    #!/usr/bin/perl

    use strict;
    use warnings;

    use EV;
    use AnyEvent;
    use AnyEvent::Fork;
    use AnyEvent::Fork::Pool;

    #sleep 2;

    my $worker_pool_template = AnyEvent::Fork->new->require('Worker');

    my $pool = $worker_pool_template->AnyEvent::Fork::Pool::run(
            'Worker::work',
            'init'  => 'Worker::init',
            'max'   => 100,
            'idle'  => 10,
            'load'  => 1,
            'start' => 2,
            'stop'  => 1,
            );

    $pool->('bar', sub { print "child $_[0] parent $$ callback\n"; });

    my $w = AnyEvent->condvar;
    $w->recv;

And this 'lib/Worker.pm':

    package Worker;

    use strict;
    use warnings;

    use Mojo::UserAgent;

    sub init {
            print "child $$ init\n";
    }

    sub work {
            print "child $$ working\n";
            #sleep 2;
            print "child $$ done\n";
            return $$;
    }

    1;

Running it like so:

    $ perl -Ilib ./defuncter.pl
    child 16065 init
    child 16065 working
    child 16065 done
    child 16065 parent 16062 callback
    child 16066 init
    child 16067 init
    child 16068 init
    child 16069 init
    child 16070 init
    child 16071 init
    child 16072 init
    child 16073 init
    child 16074 init
    child 16075 init

It generates one defunct process:

    $ ps auxw|grep -E 'Worker|defuncter'
    mjw      16150  0.2  0.1  36544  6916 pts/5    S+   13:17   0:00
perl -Ilib ./defuncter.pl
    mjw      16153  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16154  0.0  0.0      0     0 pts/5    Z+   13:17   0:00
[Worker::work of] <defunct>
    mjw      16155  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16156  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16157  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16158  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16159  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16160  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16161  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16162  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16175  0.0  0.5  90588 23660 pts/5    S+   13:17   0:00
Worker::work of 16150
    mjw      16198  0.0  0.0  12728  2292 pts/3    S+   13:17   0:00
grep -E Worker|defuncter
    $

If I remove the "use Mojo::UserAgent;", the problem does not present.
If I uncomment either of the sleep statements, the problem does not present.

Perhaps in light of this it's more a Mojolicious issue and this isn't the right
place for the discussion any more, but I thought I'd see what you thought.