Anyevent-MP cluster starting

Fri May 2 14:29:17 CEST 2014

>> Things go wrong when client and server are started before seed.
>> Chatting does not happen at all
> you can specify the "other" node as additional seeds to get things going.
Good point, thank tou
>
> in general, aemp regularly tries to connect to the seeds when it needs to,
> typically every 15 seconds. did you wait that long after starting the seed
> node?
Yes, I did try. I have turned logging on. This is chat server's output 
(same for chat client) when seed is not yet started:

<cut>
2014-05-02 15:37:08.000000 +0400 debug AnyEvent::Util: Using Guard 
module to implement guards.
2014-05-02 15:37:08.000000 +0400 info AnyEvent: Autoloaded model 
'AnyEvent::Impl::EV', using it.
2014-05-02 15:37:08.892250 +0400 debug AnyEvent::MP::Kernel: node 
lamoz/SPRvjO6dG9 starting up.
2014-05-02 15:37:08.900890 +0400 trace AnyEvent::MP::Kernel: running 
post config hooks and init.
2014-05-02 15:37:08.901004 +0400 debug AnyEvent::MP::Kernel: node 
listens on [10.15.8.4:34150].
2014-05-02 15:37:08.901350 +0400 trace AnyEvent::MP::Kernel: trying 
connect to seed node 127.0.0.1:9999.
2014-05-02 15:37:08.902066 +0400 trace AnyEvent::MP::Kernel: starting 
search for master node.
2014-05-02 15:37:08.902270 +0400 trace AnyEvent::MP::Kernel: starting 
services.
server ready.
2014-05-02 15:37:08.903789 +0400 trace AnyEvent::MP::Transport: 
127.0.0.1:9999 disconnected - В соединении отказано.
</cut>

"В соединении отказано" means "Connection refused". I wait for minutes 
to no avail. Nothing seems to happen, sniffer keeps silent. It seems 
that seed reconnection works only after first successful connection.
>> (and I get no visible indication of errors).
> it's not exactly an error condition.
>
> the standard way out is to have a timeout for everything - cannot find the
> chat server in n seconds? thats an error. cannot find any services you expect
> in n seconds? that's an error.
Oh I got it, that's fun! ))
>
> unfortunately, which way is best depends on the problem, and aemp doesn't
> have many high level directives to help you (such as monitor instances
> that manage timeouts and restart/reconnect).
I saw keepalive service in the source code (Kernel.pm line 570). It is 
just not public (yet?).
>
>> Now the question is: what is the recommended way of starting
>> the cluster of MP-nodes?
> Obviously, things go most smooth (=quick) when at least one seednode is up
> and accessible at all times, and avodiing network splits is also useful.
>
> However, even when you start the seednode last, things should still work -
> you just need some patience to wait for the seednode connection retry in
> the aemp kernel.
That gonna be not true. I've attached all sources (seed, server and 
client) to avoid misunderstanding. Server and client are from aemp with 
just logging and shebang line changed.
> The good thing is, your expectations aren't wrong :)
Glad to hear it!
>
> So, if you don't get a connect within, say, 20s after starting the seed, it
> would be time to use tcpdump or strace to see if the node actually tries to
> connect to the seednode again, and if not, check the config and if all works
> out, there must be a bug in aemp.
Please check scripts I've attached when you have spare time. It seems 
that I am not doing something against your words, but code bahaves 
differently.
>
> Feel free to report further questions and your findings here - if there is a
> bug in seednode management, it shouldn't be that hard to fix. If you want to
> look for yourself, the code is in AnyEvent::MP::Global:
>
> set_seeds stores the configured seeds and starts the $SEED_WATCHER that
> calls more_seeding regularly. You could have a warn in both functions to
> see when they are called, and if they are called.
>
I am not ready to dive that deep. I am missing something here about 
AEMP::Global. My understanding of "Global Nodes" is that they are some 
nodes that participate in distributed database by running some "Global 
Service". This service starts when node does something to that 
distributed DB. Say, node sets some key on that DB. Key is stored in 
local shard of the DB. Then Global Service is started to merge that 
changes into other nodes vision of the DB. ...and I'm lost. Every node 
do something with DB to announce itself to the cluster. Then every node 
is Global according to my understanding. Is it? Pod says that only seeds 
should be global.
-------------- next part --------------
#!/usr/bin/env perl

BEGIN {
    $ENV{AE_MP_TRACE} = 1;
    $ENV{PERL_ANYEVENT_VERBOSE} = 9;
};

# Usage: ./chat_client nickname optional-servernode
# implement a chat client using "bridge-head" methodology.

use common::sense;
use AnyEvent::MP 2.0;

my $nick = shift;

configure
    seeds => [ '127.0.0.1:9999' ];

$| = 1;

my $port = port;

my ($client, $server);

sub server_connect {
   my $db_mon; $db_mon = db_mon eg_chat_server2 => sub {
      return unless %{ $_[0] };
      undef $db_mon;

      print "\rconnecting...\n";

      $client = port { print "\r  \r at _\n> " };
      mon $client, sub {
         print "\rdisconnected @_\n";
         &server_connect;
      };

      $server = spawn [keys %{ $_[0] }]->[0], "::client_connect", $client, $nick;
      mon $server, $client;
   };
}

server_connect;

my $w = AE::io 0, 0, sub {
   chomp (my $line = <STDIN>);
   print "> ";
   snd $server, $line
     if $server;
};

print "> ";
AE::cv->recv;

-------------- next part --------------
#!/usr/bin/env perl

# Usage: ./chat_server
# implement a chat client using "bridge-head" methodology.

BEGIN {
    $ENV{AE_MP_TRACE} = 1;
    $ENV{PERL_ANYEVENT_VERBOSE} = 9;
};

use common::sense;
use AnyEvent::MP;

configure
    seeds => [ '127.0.0.1:9999' ];

db_set eg_chat_server2 => $NODE;

my %clients;

sub msg {
   print "relaying: $_[0]\n";
   snd $_, $_[0]
      for values %clients;
}

sub client_connect {
   my ($client, $nick) = @_;

   mon $client;
   mon $client, psub {
      delete $clients{$client};
      msg "$nick (quits, @_)";
   };

   $clients{$client} = $client;

   msg "$nick (joins)";

   rcv $SELF, sub { msg "$nick: $_[0]" };
}

warn "server ready.\n";

AE::cv->recv;

-------------- next part --------------
#!/usr/bin/env perl

use strict;
use warnings;

BEGIN {
    $ENV{AE_MP_TRACE} = 1;
    $ENV{PERL_ANYEVENT_VERBOSE} = 9;
};

use AnyEvent;
use AnyEvent::MP 2.0;

warn 'Laggy start';
sleep(5);

configure
    binds => [ '127.0.0.1:9999' ];
my $w = AE::timer 0, 1, sub { warn "still working"; };
AnyEvent->condvar->recv;