Workaround for using synchronous modules in AnyEvent?

[Marc Lehmann]

On Fri, Apr 05, 2013 at 09:10:49AM -0700, Mike Schilli <anyevent at perlmeister.com> wrote:
> I'm using AnyEvent somewhat uncleanly with a library that makes
> synchronous socket calls. Crazy, I know!

Well, most modules on CPAN are like that, we sometimes don't want to
reinvent the wheel completely, and you can't even blame the authors. So
the crazyness is quite the norm :)

> So, I was looking for a quick solution to fix that and stumbled across
> Coro, which offers threading, but alas, I think it blocks on I/O.

While there might be "quick" solutions if you are the god of knowledge,
humans need to invest more.

Coro *can* help, but there is no guaranteed recipe.

For example, a surprising number of "blocking" modules use IO::Select, or
select to actually wait for a response. I have never understood why, but
if Net::ZooKeeper does, you can try to override its select function with
the one from Coro::Select (see the manpage, which has an example for
Net::DBus::Reactor).

This doesn't make the module thread-safe, but most modules are, and the
remaining ones often are when you only use one outstanding request at a
time.

Another option it to see if the module uses IO::Socket::INET or so, and
somehow "patching" it to use Coro::Socket instead, which is reasonably
compatible to IO::Socket to often work.

The last resort would be a process pool. Luckily enough, I am just writing
one, which will appear as AnyEvent::Fork on CPAN in the next few days
(or rather, a relatively stable one that has seen a bit of testing, I am
mostly waiting for its two support modules to appear working).

That allows you to safely spawn one or more processes doing the zookeeper
requests, and is essentially the method used by AnyEvent::DBI, which I plan
to convert to AnyEvent::Fork.

Any such process pool solution isn't quick either - you have to worry about
serialiasing your data, and AnyEvent::Fork, no matter how much I praise it,
only gives you a lame old socket at the moment. I plan to put another module
on top that gives nice request/response-based workers, but I ain't there yet.

(You could so sth. similar with perl's built-in process emulation,
ithreads, but you have to worry about serialisation there as well, and
yous itll would need to use some sockets for notification).

At least you can see that others suffer from similar problems as you :)

Ah, CPAN finally is done, let me check Net::ZooKeeper.

Holy shit, it uses XS, and threads, and wow, so heavy duty. Ah right, now I
remember about zookeper.

No, there is no quick way for this module, you either need to live with
it...

> Does anyone have any solution, or should I just start on writing .xs
> code for a ZooKeeper client that utilizes ZooKeeper's single-threaded
> library for a new and truly asyncronous AnyEvent::ZooKeeper module?

Or do that.

I would have some ideas to enable XS modules to "give up" the perl
interpreter in the same way as python can do (which would allow
net::zookeeper to put release_perl() and acquire_perl() calls around it's
pthread_cond_timedwait's to let other threads run), but I don't think it
would gather up enough momentum.

Now, to get into the relam of crazy ideas - couldn't you set the timeout
to 0 when calling e..g wait, and just use a timer and check regularly,
say, after 0.01, 0.02, 0.04, 0.08s and so on? Assuming ->wait is the
problem of course, and assuming you normally get a response quickly.

Polling like this isn't working well when you really poll for events
though.

You could ask the zookeeper maintainer to provide the option of writing
something to a file descriptor of your choice on events, or maybe the option
of giving you the file descriptor it uses internally (if any).

In the first case, you could provide a pipe for it to write, and you to
read in any anyevent watcher, giving you a good idea on when to ask for
new events.

In the second case, you might want to wait for the file descriptor to become
ready when there are no events yet.

Both modifications would be relatively small, and might make it possible
to write a slim AnyEvent::ZooKeeper module that just interfaces with the
existing one, much like AnyEvent::AIO does for example.

In any case, maybe you got some ideas by these comments and the comments
by other people, and good luck. If you find a solution or write
AnyEvent::ZooKeeper, feel free to tell the list :)

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\