Anyevent-MP cluster starting

[Marc Lehmann]

On Wed, Apr 30, 2014 at 02:55:57PM +0400, "Konstantin A. Pustovalov" <lamoz at adriver.ru> wrote:
> I'm trying to get involved into AEMP. Now talking about v2.0 from CVS.

Cool (especially the 2.0 part :)

> Things go wrong when client and server are started before seed.
> Chatting does not happen at all

you can specify the "other" node as additional seeds to get things going.

in general, aemp regularly tries to connect to the seeds when it needs to,
typically every 15 seconds. did you wait that long after starting the seed
node?

> (and I get no visible indication of errors).

it's not exactly an error condition.

the standard way out is to have a timeout for everything - cannot find the
chat server in n seconds? thats an error. cannot find any services you expect
in n seconds? that's an error.

and so on.

the example chat client simply try to connect to the chat server every
second, without ever giving up.

that's a valid (and simple) way to handle this, but having an absolute
timeout on things is more appropriate for when you want diagonstics in
this case.

unfortunately, which way is best depends on the problem, and aemp doesn't
have many high level directives to help you (such as monitor instances
that manage timeouts and restart/reconnect).

> Now the question is: what is the recommended way of starting
> the cluster of MP-nodes?

Obviously, things go most smooth (=quick) when at least one seednode is up
and accessible at all times, and avodiing network splits is also useful.

However, even when you start the seednode last, things should still work -
you just need some patience to wait for the seednode connection retry in
the aemp kernel.

> How I do expect it to work (probably horribly wrong). I start seeds
> and non-seeds in nearly one moment, it deviates a little depending on
> compilation time and ssh/operator/supervisor lag. Non-seed nodes just
> trying to reconnect to seed ones in case of first unsuccessful
> attempt.

This is what should happen. If your client never connects, even when waiting
for, say, 20 seconds, then things are deifnitely not going right.

> somehow in bundle and let them find each other. So seeds can start
> after non-seeds, but within some window. Application sets timeout
> (say, 30sec)  and just fails if timeout fires before expected
> connectivity is met (chat client and server see one seed).

Yes, that's sound. The only open variable here is the value of "30
seconds", and that depends on the internal monitoring timeout, which is
currently 15 seconds (querying/chagning that is currently not documented).

Your value of 30 seconds is quite reasonable. If you have only one seed,
it should try to connect to it every 15 seconds (wit more than one seed it
simply picks a seed at random).

> So how would you guys propose me to start aemp cluster? Why is my
> expectations are wrong?

The good thing is, your expectations aren't wrong :)

So, if you don't get a connect within, say, 20s after starting the seed, it
would be time to use tcpdump or strace to see if the node actually tries to
connect to the seednode again, and if not, check the config and if all works
out, there must be a bug in aemp.

Feel free to report further questions and your findings here - if there is a
bug in seednode management, it shouldn't be that hard to fix. If you want to
look for yourself, the code is in AnyEvent::MP::Global:

set_seeds stores the configured seeds and starts the $SEED_WATCHER that
calls more_seeding regularly. You could have a warn in both functions to
see when they are called, and if they are called.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\