Capturing the output of Multiple parallel http requests

[Ian Stuart]

On 28/01/11 13:40, Marc Lehmann wrote:
> On Fri, Jan 28, 2011 at 12:30:13PM +0000, Ian Stuart<Ian.Stuart at ed.ac.uk>  wrote:
>> I can fire off multiple requests, and I can see the response from
>> each of those queries.... but I can't get my head round how to
>> return those responses back to my core code.
>
> The question is, of course, where your core code is :)
Indeed... my core code is this code, extended to aquire the range or web 
hosts to contact via a complex set of Application queries.

>
>>          'User::Agent'         =>  'OA-RJ Broker v0.2',
>
> (and this is a ^^ bug).
Too much perl! should be 'User-Agent' :chuckle:

>> How do I run a bunch of queries, and then capture each response?
>
> If I understand you correctly, then you don't care whether you get the
> responses when they arrive, but you want to process them in a non-event-based
> fashion, basically:
>
> phase 1: fire off requests
> phase 2: wait for responses
> phase 3: process responses
Pretty much


> You can do something like that (with phase 2 and 3 combined) by using a
> condvar for each request:
>
>     sub transfer($) {
>        ...
>        my $cv = AnyEvent->condvar;
>
>        http_request POST =>  ...,
>           ...,
>           $cv;
>
>        $cv
>     }
>
>     for my $cv (map transfer $url, @my_urls) {
---^ should be "(map transfer $_, @my_list_of_params)"
>        my ($body, $hdr) = $cv->recv;
>        ...
>     }
>
> Basically, transfer returns an objetc that you cna query for results - if
> results are already there, then ->recv will just reutnr them, otherwise it
> will wait for them.
>
> The for loop then calls transfer for each of @my_urls, and then asks the
> condvars it returns in order for results.
>
> That way you get the results in the same order that you started the
> requests, regardless of how they actually arrive form the network.
Excellent!!!

Working!

OK, so to test my understanding:

There are two parts to this system
A) there is a function that makes an http_request, and returns the data 
from that call when ask... and when there is data there to return
B) there is a loop that sets up a number of [parallel] function calls, 
and then waits for them to complete.

*Part A*
1) I pass in a set of parameters, which I process to create a URL
2) I set up a condition variable (which exists for that instance of the 
loop)
3) I call http_request with the appropriate set of parameters, the last 
of which is the scalar that the result of the request will end up in 
(where "result" in this case means a reference to a list : [$body, 
$header] )
4) the subroutine returns the scalar that is the condition variable, 
which is the thing that [will, when the event finishes] contain the 
return from the http_request.

*Part B*
1) The "map" fires off a function call for each item in the list
2) The foreach loop takes the scalar that [will, when the event 
finishes] contain the return data from the function call and prods it to 
to see if its ready to actually return data.
3a) If there is data to return, it process it & loops back to the next 
item in the foreach loop
3b) If there is no data, the foreach loop blocks until such time as data 
can be returned.

*The Advantage*
Like a traditional linear system, there is a delay until the first query 
returns data, however there is then a gain as each other query has also 
had time to execute, thus is likely to be ready to return data immediately.

Would that be right?

Would you like (a slight variation of, and fully working version of) my 
code as an example for AnyEvent::HTTP (or one of the other documents)?

-- 

Ian Stuart.
Developer: Open Access Repository Junction and OpenDepot.org
Bibliographics and Multimedia Service Delivery team,
EDINA,
The University of Edinburgh.

http://edina.ac.uk/

This email was sent via the University of Edinburgh.

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.