1501347 - Redeploy ec2-manager

Status: RESOLVED WONTFIX

(Taskcluster :: Services, enhancement)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Add this to taskcluster-terraform and deploy it in dev and staging clusters.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Let's wait a little bit and see if bug 1487736 makes any progress, and perhaps use that library in ec2-manager so that we can do the same kind of postgres deployment for ec2-manager as for queue and other services.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

So, we want tc-terraform to remain agnostic to where the Kubernetes cluster is deployed.  That means it does not deploy anything specific to GCP (with exceptions for worker-manager and the object service, where it will be optional).  This is important to allow someone deploying TC to pick the clouds they want to use, rather than being locked into our choices.

So the options we have are:
 * deploy postgres in the ec2-manager pod with a Kubernetes storage object attached; or
 * deploy Google Cloud SQL in tc-mozilla-terraform and pass the credentials to tc-terraform

The second option would require that tc-mozilla-terraform has the advantage that it is a managed DB solution.  It has these disadvantages:
 * only supports Postgres 9.6 at the moment (10 and 11 add lots of support for partitioning, which we are likely to want)
 * puts database on a public IP (private IPs are in beta)
 * would require tc-mozilla-terraform to pass different config for each service using postgres

I'm leaning toward the deploy-in-the-pod option.  John, what do you think?

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #2)
> So, we want tc-terraform to remain agnostic to where the Kubernetes cluster
> is deployed.  That means it does not deploy anything specific to GCP (with
> exceptions for worker-manager and the object service, where it will be
> optional).  This is important to allow someone deploying TC to pick the
> clouds they want to use, rather than being locked into our choices.
> 
> So the options we have are:
>  * deploy postgres in the ec2-manager pod with a Kubernetes storage object
> attached; or

Would that storage object be local disk with syncing for backup, or would it be running a postgres database out of a cloud object directly?

>  * deploy Google Cloud SQL in tc-mozilla-terraform and pass the credentials
> to tc-terraform
> 
> The second option would require that tc-mozilla-terraform has the advantage
> that it is a managed DB solution.  It has these disadvantages:
>  * only supports Postgres 9.6 at the moment (10 and 11 add lots of support
> for partitioning, which we are likely to want)

ec2-manager does not use partitioning and 9.6 satisfies the requirements for production.  It also has a pretty small (in sql terms) dataset.  In fact, we're currently running production on 9.6:

$ heroku pg:psql --remote production
--> Connecting to postgresql-<snip>
psql (10.6, server 9.6.3)

We probably will want 10 and 11 in future, but we don't need to do it right now.

>  * puts database on a public IP (private IPs are in beta)
>  * would require tc-mozilla-terraform to pass different config for each
> service using postgres

config being username, password and db name?

> I'm leaning toward the deploy-in-the-pod option.  John, what do you think?

In general, the hosted postgres option is what I prefer, but it's not a strong preference in this case.  The main reason is that Cloud SQL is managed and likely has a lot better tooling than us loading up a postgres instance in our kubernetes deployment.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I believe GKE provisions storage using something like EBS.  I don't know how reliable that is, but certainly more reliable than instance storage.  We would still be responsible for backups (as we are now with Azure).

ec2-manager is a light user, but it's queue that I'm thinking forward to.  It would be a shame to get to postgresql-ifying the queue and have to start over with a new postgres hosting solution (and probably take another downtime to switch to it).

I'll continue to think on it.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

We discussed this a bit in today's meeting, and it seems that the answers to these questions are still unclear, but one thing that is clear is that ec2-manager itself will not need to be redeployed.