Async iterators

[Clinton Gormley]

On Mon, 2012-08-06 at 19:17 +0200, Marc Lehmann wrote:
>    my $processthem; $processthem = sub {
>       my ($s, @results) = @_;
> 
>       if (@results) {
>          for (@results) {
>             ...
>          }
> 
>          # if we want more:
>          $s->next (100, $processthem);

Doesn't this end up with a lot of recursion? Given that we could be
processing billions of records, the stack could become very large.


>    my $s = new ElasticSearch::ScrolledSearch
>                   ...,
>                   cb => sub {
>                      my ($s, $result) = @_;
> 
>                      $s->next;
>                   };
> 
>    # now you have the problem of storing $s somewhere for the duration of the
>    # search.

> Variations would be to pass one or more results to the callback, or having
> the callback return a boolean to cancel or continue the search (or a
> number of further results to return).

... or providing a method which can be called to cancel further pulls.

Again, I'm not sure how to trigger the "pull new results if needed"
without ending up with massive recursion.

> make sure existing users can still use the old way. one reason for this is
> that breaking compatibility is bad (AnyEvent has taken great pains to keep
> the api compatible over it's lifetime). another is that event-based apis are
> definitely more complicated, cumbersome and write-intensive then simple
> blocking apis might be. I cannot judge if this will be the case for your
> current API, but maybe it is. At the very least, people might not be able to
> get their mind around a callback api and this closure stuff.

Agreed

> 
> so having a simple blockign api is useful on it's own.
> 
> you might even consider having an extra module such as
> ElasticSearch::AnyEvent (or AnyEvent::ElasticSearch) and then base your
> existing module on that one (they could also be packaged together of
> course).

Yes, that's what I was thinking

thanks again Marc

clint

>