Capturing the output of Multiple parallel http requests

[Ian Stuart]

I'm trying to make http post-requests to a number of hosts.
I can do this sequentially, however that solution does not scale, so I 
am re-implementing in a parallel mode... using AnyEvent::HTTP
(In the environment I will use this, the number of POST requests is 
variable, hence the use of a loop rather than any attempt to just set up 
a sequential just as in the "finger" example in AnyEvent::Intro)

I can fire off multiple requests, and I can see the response from each 
of those queries.... but I can't get my head round how to return those 
responses back to my core code.

My references for this are the documentation for AnyEvent, 
AnyEvent::HTTP and AnyEvent::Intro - the info is, I think, there... I 
just can't piece it all together to make it work

My code:
------- start code ------
#!/path/to/bin/perl -w

use strict;
use Event;
use AnyEvent::HTTP;
use AnyEvent::Strict;
use MIME::Base64;

use Data::Dumper;

my @targets = qw(host1 host2 host3 host4 host4);

my ($host, $username, $password, $realm, $sword_url, $collection) = ();

my $endpoints = {};

# code deleted for making actual connections, but they look like this:
#$endpoints->{'nost_n'}                 = {};
#$endpoints->{'host_n'}->{'host'}       = 'the.host.name';
#$endpoints->{'host_n'}->{'collection'} = '/path/to/location';
#$endpoints->{'host_n'}->{'username'}   = 'l-user';
#$endpoints->{'host_n'}->{'password'}   = 'pa$$wrd';

# file to send via POST:
my $zipfile = "broker_deposit.zip";    # A DSpace/Mets zip file

# An AnyEvent method to send a zip file to a sword endpoint
#
sub transfer($)
{
     my ($params) = @_;

     my ($file, $host, $collection, $username, $password, $protocol);

     # delete on a hash element returns undef if that key does not exist,
     # therefor that scalar is set to undef if the parameter is not
     # passed into the method
     $file       = delete $params->{file};
     $host       = delete $params->{host};
     $protocol   = delete $params->{protocol};
     $collection = delete $params->{collection};
     $username   = delete $params->{username};
     $password   = delete $params->{password};

     # Load the file. If there is no file, then return from the event
     # with nothing
     my $archive = "";
     open(FILE, $file)
       or return "Unable to create Transfer package";
     binmode FILE;
     while (my $l = <FILE>) { $archive .= $l; }
     close FILE;

     # Tell SWORD to process the contents of the zip file as the new OA-RJ
     # type
     my $auth = "Basic ".MIME::Base64::encode("$username:$password", '');

     my %headers = (
         'X-Packaging'         => 'http://opendepot.org/broker/1.0',
         'X-No-Op'             => 'false',
         'X-Verbose'           => 'false',
         'Content-Disposition' => "filename=$file",
         'Content-Type'        => 'application/zip',
         'User::Agent'         => 'OA-RJ Broker v0.2',
         'Authorization'       => $auth,

                   );

     my $myReturn;

     # this is a long call, intersperced with comments!
     $myReturn = http_request(

         # the main request we want to make
         "POST", "http://${host}${collection}",

         # the body of the http_request
         body => $archive,

         # the headers for that request
         headers => \%headers,

         # The routine called on Error or on completion
         sub {
           my ($body, $header) = @_;

             my $text = "";
             if ($header->{Status} =~ /^2\d{2}/)
             {
               $text = $body;
             } else
             {
               $text = "Error: ".$header->{Status}.": 
".$header->{Reason}."\n";
             }
             print "body: $text\n";
             return $text
         }
     );
     print "returning: ".Dumper ($myReturn);

     return $myReturn;

} ## end sub transfer($$)

my %responses;
my $cv = AnyEvent->condvar;

foreach my $target (@targets)
{

     my %params = (
            protocol   => 'http',
            host       => delete $endpoints->{$target}->{'host'},
            collection => delete $endpoints->{$target}->{'collection'},
            username   => delete $endpoints->{$target}->{'username'},
            password   => delete $endpoints->{$target}->{'password'},
            file       => $zipfile,
            );

     $responses{$target} = transfer( \%params,);

} ## end foreach my $target (@targets...)

for my $response ( keys %responses ){
   print "$response ",  $cv->recv, "\n";
}
$cv = undef;
print Dumper(\@responses);

exit;
---------- end code --------------

The script runs, with the following output:

----- start output ------
body: Error: 401: Unauthorized

body: Error: 401: Unauthorized

body: Error: 415: Unsupported Media Type

body: <?xml version="1.0" encoding="UTF-8"?><atom:entry 
xmlns:atom="http://www.w3.org/2005/Atom" 
xmlns:sword="http://purl.org/net/sword/">
// <!-- snip -->
</atom:entry>
body: <?xml version="1.0" encoding="UTF-8"?><atom:entry 
xmlns:atom="http://www.w3.org/2005/Atom" 
xmlns:sword="http://purl.org/net/sword/">
// <!-- snip -->
</atom:entry>
----- end output -----

This shows me that I'm getting correct responses for each http_request, 
however I'm just not passing them back.

I'm sure I have the right general idea, but the details are eluding me :(
How do I run a bunch of queries, and then capture each response?

-- 

Ian Stuart.
Developer: Open Access Repository Junction and OpenDepot.org
Bibliographics and Multimedia Service Delivery team,
EDINA,
The University of Edinburgh.

http://edina.ac.uk/

This email was sent via the University of Edinburgh.

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.