Workaround for using synchronous modules in AnyEvent?

[Mike Schilli]

On Fri, 5 Apr 2013, Marc Lehmann wrote:

>> Does anyone have any solution, or should I just start on writing .xs
>> code for a ZooKeeper client that utilizes ZooKeeper's single-threaded
>> library for a new and truly asyncronous AnyEvent::ZooKeeper module?
> Or do that.

Thanks much for your reponse, it really helped clarify the issues.

So, I've dabbled around with this a bit and I think the best solution
is to write AnyEvent::ZooKeeper. Zookeeper offers an asynchronous C API,
now I'm working on integrating it with an event loop that I can then
use with AnyEvent.

I've started with some .xs code using the ZooKeeper client async C API, 
which offers calls like

     int zoo_aget_children(zhandle_t *zh, const char *path, int watch,
         strings_completion_t dc, const void *data)

which will return immediatly and run the completion callback (dc) 
with the found data later when the ZooKeeper server's response has arrived.
(It's not easy to maintain the callbacks, but I'm probably going to do
what Tk does and tie their refs to data structures in Perl land, not C,
but I digress).

So far so good. However, ZooKeeper client API calls aren't using a traditional
event loop, but offer two API calls to keep the system ticking:

     zookeeper_interest(zh, &fd, &interest, &tv);
     (https://github.com/apache/zookeeper/blob/trunk/src/c/src/cli.c#L741)

does some preliminary work and returns a number of file descriptors 
to watch for incoming data. Then their client waits with select:

     rc = select(fd+1, &rfds, &wfds, &efds, &tv);
     (https://github.com/apache/zookeeper/blob/trunk/src/c/src/cli.c#L757)

and finally this API call processes the result (and calls the callback):

     zookeeper_process(zh, events);
     https://github.com/apache/zookeeper/blob/trunk/src/c/src/cli.c#L795

The respective implementations of these functions are here:

     https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L1878

And here:

     https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L2560

So I guess what needs to be done to make this work with AnyEvent is: Call
zookeeper_interest, get the file descriptors, then tell the event loop that
AnyEvent's currently running that we want to watch for these file descriptors
and call a callback if something happens. Is this possible in a somewhat
event loop agnostic way?

Or would it be smarter to run Coro's select as you pointed out in your
previous response?

Thanks for any help ...

-- 
-- Mike

anyevent at perlmeister.com

> On Fri, Apr 05, 2013 at 09:10:49AM -0700, Mike Schilli
> <anyevent at perlmeister.com> wrote:
>> I'm using AnyEvent somewhat uncleanly with a library that makes
>> synchronous socket calls. Crazy, I know!
>
> Well, most modules on CPAN are like that, we sometimes don't want to
> reinvent the wheel completely, and you can't even blame the authors. So
> the crazyness is quite the norm :)
>
>> So, I was looking for a quick solution to fix that and stumbled across
>> Coro, which offers threading, but alas, I think it blocks on I/O.
>
> While there might be "quick" solutions if you are the god of knowledge,
> humans need to invest more.
>
> Coro *can* help, but there is no guaranteed recipe.
>
> For example, a surprising number of "blocking" modules use IO::Select, or
> select to actually wait for a response. I have never understood why, but
> if Net::ZooKeeper does, you can try to override its select function with
> the one from Coro::Select (see the manpage, which has an example for
> Net::DBus::Reactor).
>
> This doesn't make the module thread-safe, but most modules are, and the
> remaining ones often are when you only use one outstanding request at a
> time.
>
> Another option it to see if the module uses IO::Socket::INET or so, and
> somehow "patching" it to use Coro::Socket instead, which is reasonably
> compatible to IO::Socket to often work.
>
> The last resort would be a process pool. Luckily enough, I am just writing
> one, which will appear as AnyEvent::Fork on CPAN in the next few days
> (or rather, a relatively stable one that has seen a bit of testing, I am
> mostly waiting for its two support modules to appear working).
>
> That allows you to safely spawn one or more processes doing the zookeeper
> requests, and is essentially the method used by AnyEvent::DBI, which I plan
> to convert to AnyEvent::Fork.
>
> Any such process pool solution isn't quick either - you have to worry about
> serialiasing your data, and AnyEvent::Fork, no matter how much I praise it,
> only gives you a lame old socket at the moment. I plan to put another module
> on top that gives nice request/response-based workers, but I ain't there yet.
>
> (You could so sth. similar with perl's built-in process emulation,
> ithreads, but you have to worry about serialisation there as well, and
> yous itll would need to use some sockets for notification).
>
> At least you can see that others suffer from similar problems as you :)
>
> Ah, CPAN finally is done, let me check Net::ZooKeeper.
>
> Holy shit, it uses XS, and threads, and wow, so heavy duty. Ah right, now I
> remember about zookeper.
>
> No, there is no quick way for this module, you either need to live with
> it...
>

>
> I would have some ideas to enable XS modules to "give up" the perl
> interpreter in the same way as python can do (which would allow
> net::zookeeper to put release_perl() and acquire_perl() calls around it's
> pthread_cond_timedwait's to let other threads run), but I don't think it
> would gather up enough momentum.
>
> Now, to get into the relam of crazy ideas - couldn't you set the timeout
> to 0 when calling e..g wait, and just use a timer and check regularly,
> say, after 0.01, 0.02, 0.04, 0.08s and so on? Assuming ->wait is the
> problem of course, and assuming you normally get a response quickly.
>
> Polling like this isn't working well when you really poll for events
> though.
>
> You could ask the zookeeper maintainer to provide the option of writing
> something to a file descriptor of your choice on events, or maybe the option
> of giving you the file descriptor it uses internally (if any).
>
> In the first case, you could provide a pipe for it to write, and you to
> read in any anyevent watcher, giving you a good idea on when to ask for
> new events.
>
> In the second case, you might want to wait for the file descriptor to become
> ready when there are no events yet.
>
> Both modifications would be relatively small, and might make it possible
> to write a slim AnyEvent::ZooKeeper module that just interfaces with the
> existing one, much like AnyEvent::AIO does for example.
>
> In any case, maybe you got some ideas by these comments and the comments
> by other people, and good luck. If you find a solution or write
> AnyEvent::ZooKeeper, feel free to tell the list :)
>
>