1146447 - formalise Consul bootstrap process

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Daniel Maher [:phrawzty]]

We need to formalise (and automate) the Consul bootstrap process.  This includes:

* Determining the external and internal addresses of the Consul nodes.
* Logging into said nodes and joining the Server cluster nodes together.
* Populating Consul with some basic information (maybe).
* Updating the S3-based Hiera with the internal addresses of the Consul Servers.

That final point is important: new roles being spun up by ASGs will have no foreknowledge of Consul, but they will know how to query the S3-based Hiera, and from that we can trigger a consul -join.

Comment 1

[Daniel Maher [:phrawzty]]

(In reply to Daniel Maher [:phrawzty] from comment #0)
> * Populating Consul with some basic information (maybe).

:rhelmer, could your bulk_load.sh be used for this ?

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Daniel Maher [:phrawzty] from comment #1)
> (In reply to Daniel Maher [:phrawzty] from comment #0)
> > * Populating Consul with some basic information (maybe).
> 
> :rhelmer, could your bulk_load.sh be used for this ?

Sure, if all you want to do is load a config file full of key/value pairs into a similarly-named prefix, it'll do the job. AFAICT consul has no built-in bulk loading support right now, just GET and PUT of individual keys.

If your needs are more complex, it'd be easy to whip something up in Python using the requests library to work with the Consul HTTP API.

Comment 3

[Robert Helmer [:rhelmer]]

This actually *almost* works now - we use an ELB for consul (the hostname lives in Hiera S3) and consul nodes attempt to auto-join it.

I tried bringing up a prod consul cluster and the only problem is that it joined the consul cluster before restarting in server mode, so it left the cluster.

We just need a dependency here and it should be automatic.

Comment 4

[Robert Helmer [:rhelmer]]

The process is basically "terraform apply" consul, and use the bulk_load.sh

I'd like to continue to improve this - maybe switch to consulate for this, but good enough for now.