AnyEvent->loop?

[Steve Freegard]

Hi Marc,

On 16 Feb 2011, at 02:01, Marc Lehmann wrote:

> On Wed, Feb 16, 2011 at 12:16:27AM +0000, Steve Freegard <steve.freegard at fsl.com> wrote:
>> I have one API related question though; when AnyEvent->condvar->wait
>> is used for mainloop emulation so it's not possible to use another
>> condvar elsewhere without receiving the 'AnyEvent::CondVar:
>> recursive blocking wait detected at ..' error.
> 
> It is, as long as it is not blocking, or you use Coro threads (and wait in
> a separate thread).
> 

I haven't played with Coro at all - yet.

> Note also that ->wait is an unsupported and undocumented method, you
> should use the documented ->recv instead.
> 

Ok

>> I realised that I should really be calling the actual event loop
>> providers event loop instead in my main program and then my CondVar's
>> worked as expected.
> 
> One would argue that they work as expected already, see below.
> 
>> Looking at the actual code and not the documentation - it would seem
>> that I can simply call:
>> 
>> AnyEvent->loop;
> 
> While that might work in some cases, it does not work with all backends.
> 
>> Any reason why this isn't documented?  I've tried it and it works great.
> 
> Well, it certainly doesn't work great when the method isn't available in a
> backend, so "works great" is of limited value.
> 
> The reason it is not documented is that AnyEvent is primarily meant for
> module authors and not standalone programs (where you can usually just use
> whatever event loop you want directlry).
> 
> This is not entirely true anymore - AnyEvent apparently has become popular
> for (some) standalone programs, but still I couldn't bring it over me to make
> it so easy to block.
> 

Ok - I'm working on some patches for qpsmtpd (http://smtpd.develooper.com) as someone posted an AnyEvent personality of qpsmtpd to the mailing list.  So this is another standalone program use for AnyEvent.  

I've been adding some missing features to that code and attempting to convert some of the plugins to AnyEvent so they can be merged into a future qpsmtpd release for others to use.


> Now, your concrete issue seems to be different: you want to use
> AE::cv->recv as a main loop replacement and than run into the problem of
> not being able to block via a condvar in a callback.
> 
> While you can work aroudn this protection by running an event loop, note that
> you already have a semantic bug and likely an actual bug - the semantic bug
> is that you are not supposed to block the process in some callback, i.e. by
> using a blocking wait on a condvar in a callback, your program stopped beign
> event-based.
> 
> The likely actual bug is that you *probably* don't protect against recursion,
> i.e. what keeps your blocking callback from being invoked again recursively?
> Or what happens when there is another blocking wait call somewhere else
> (maybe not even in your code?) - what keeps that from recursing?
> 
> So using AE::cv->recv as "mainloop" protects you from these problems.


Here is the call sequence; a non-blocking socket is created, bound and listen is called on it prior to dropping privileges to a non-root user.  Plugins are initialised and then accept_loop() is called which does this:

sub accept_loop { 
    my $w = AE::io $SERVER, 0, sub {
        while ($SERVER && (my $peer = accept my $fh, $SERVER)) {
             fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside world does not
    
             my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
             my $qp = Qpsmtpd::AnyEvent->new($fh, format_address($host), $service);
             $qp->process_line("Connect");
          } 
    };
    
    # Mainloop emulation
    AnyEvent->condvar->recv;
    exit;
}

Each connection is handled by the read callback on the socket.  Each SMTP command is dispatched and calls 'hooks' that can be registered from plugins.  There should be no recursion at all as far as I can tell.

What I did that showed up the issue with using AnyEvent->condvar->recv as the mainloop was to covert a plugin called 'tls' which implements the STARTTLS SMTP extension.  The 'main' part of this plugin does this:

sub _convert_to_ssl_event {
    my ($self) = @_;
    my $cv = AnyEvent->condvar;
    $self->qp->{handle}->on_starttls(sub {
        my($hdl, $success, $errormsg) = @_;
        $cv->send($success, $errormsg);
    });
    $self->qp->{handle}->starttls('accept', $self->ssl_context);
    my ($success,  $errormsg) = $cv->recv;
    ($success) ? $self->qp->connection->notes('tls_enabled', 1) : $@ = $errormsg;
    return $success;
}   

I was basically using the condvar as a merge point and had it in my mind was that this is like a 'yield back to the event loop until the result is returned'.

The only place the a condvar is used is here and in the accept_loop() as this is the first plugin I've attempted to convert.  Hence why I was surprised at the 'recursive blocking wait detected at ..' and had put it down to not calling the real event loop instead of the mainloop emulation.   

The above code works fine if I call EV::loop instead of AnyEvent->condvar->recv in accept_loop.  So do I need to use Coro in this case? (an example would be helpful if so).

I don't understand this as well as I thought I did so apologies for the newbie questions and thanks for your patience.

Thanks,
Steve.