formalise Consul bootstrap process

RESOLVED FIXED

Status

Socorro
Infra
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: phrawzty, Assigned: rhelmer)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
We need to formalise (and automate) the Consul bootstrap process.  This includes:

* Determining the external and internal addresses of the Consul nodes.
* Logging into said nodes and joining the Server cluster nodes together.
* Populating Consul with some basic information (maybe).
* Updating the S3-based Hiera with the internal addresses of the Consul Servers.

That final point is important: new roles being spun up by ASGs will have no foreknowledge of Consul, but they will know how to query the S3-based Hiera, and from that we can trigger a consul -join.
(Reporter)

Updated

3 years ago
Depends on: 1146449
(Reporter)

Updated

3 years ago
Depends on: 1146537
(Reporter)

Comment 1

3 years ago
(In reply to Daniel Maher [:phrawzty] from comment #0)
> * Populating Consul with some basic information (maybe).

:rhelmer, could your bulk_load.sh be used for this ?
Flags: needinfo?(rhelmer)
(Assignee)

Comment 2

3 years ago
(In reply to Daniel Maher [:phrawzty] from comment #1)
> (In reply to Daniel Maher [:phrawzty] from comment #0)
> > * Populating Consul with some basic information (maybe).
> 
> :rhelmer, could your bulk_load.sh be used for this ?

Sure, if all you want to do is load a config file full of key/value pairs into a similarly-named prefix, it'll do the job. AFAICT consul has no built-in bulk loading support right now, just GET and PUT of individual keys.

If your needs are more complex, it'd be easy to whip something up in Python using the requests library to work with the Consul HTTP API.
Flags: needinfo?(rhelmer)
(Assignee)

Comment 3

3 years ago
This actually *almost* works now - we use an ELB for consul (the hostname lives in Hiera S3) and consul nodes attempt to auto-join it.

I tried bringing up a prod consul cluster and the only problem is that it joined the consul cluster before restarting in server mode, so it left the cluster.

We just need a dependency here and it should be automatic.
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
(Assignee)

Updated

3 years ago
Blocks: 1118288
No longer blocks: 1118306
(Assignee)

Comment 4

3 years ago
The process is basically "terraform apply" consul, and use the bulk_load.sh

I'd like to continue to improve this - maybe switch to consulate for this, but good enough for now.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.