1465217 - Mission Control production service

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement, P1)

Description

[William Lachance (:wlach)]

Since the mission control dev server is running well, it would be good to produce a production service that can exist at a more stable/human readable URL and runs off a "known good" branch of the code (instead of autodeploying from master).

Roughly I think the dockerflow/circleci setup we have for dev is fine. The only changes I would make would be:

1. Put it at a less weird url (proposal: missioncontrol.telemetry.mozilla.org)
2. Have it deploy from a branch called "production" instead of "master" (this may involve making circle autobuild that branch)
3. Make the database a t2.medium instead of a micro (I would also like to make this change for the dev server).

I would like to keep the dev instance around for continuous deployment of master. Testing changes in a safe environment should save everyone headaches.

:whd and I talked about this a bit on irc, and I think the only thing he wanted changed before we do this is to reduce the load on athena so we don't double the load on this service-- this should be relatively easy for me to do (https://github.com/mozilla/missioncontrol/issues/258) 

:whd, is this something you could do Monday next week, assuming I fix the athena issue above?

Comment 1

[Wesley Dawson [:whd]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #0)

> 1. Put it at a less weird url (proposal: missioncontrol.telemetry.mozilla.org)

This works for me, though I do recall there was some discussion about making it have a less... epic sounding name. Regardless, *.tmo is the current recommended production domain.

> 2. Have it deploy from a branch called "production" instead of "master" (this may involve making circle autobuild that branch)

For dockerflow we usually use a tag on master for production builds, so the deploy process for prod would be (1) tag master with a new version and (2) let me know so I can deploy that tag to production. The missioncontrol jenkins jobs are already provisioned to work this way. We generally don't automate (2) out of caution, but we could.

> 3. Make the database a t2.medium instead of a micro (I would also like to make this change for the dev server).

I'll make sure prod is configured to use this instance type. As for dev, this can be done at your convenience from the rds console (db identifier is ds5nypdyi4rzi). If you prefer I can do it instead in which case we should schedule that.

> :whd, is this something you could do Monday next week, assuming I fix the
> athena issue above?

Yes.

Comment 2

[William Lachance (:wlach)]

(In reply to Wesley Dawson [:whd] from comment #1)
> (In reply to William Lachance (:wlach) (use needinfo!) from comment #0)
> 
> > 1. Put it at a less weird url (proposal: missioncontrol.telemetry.mozilla.org)
> 
> This works for me, though I do recall there was some discussion about making
> it have a less... epic sounding name. Regardless, *.tmo is the current
> recommended production domain.

I suspect the ship has already sailed on this one, I think we should just keep it as-is. Renaming the project at this point is probably just going to confuse people.

> > 2. Have it deploy from a branch called "production" instead of "master" (this may involve making circle autobuild that branch)
> 
> For dockerflow we usually use a tag on master for production builds, so the
> deploy process for prod would be (1) tag master with a new version and (2)
> let me know so I can deploy that tag to production. The missioncontrol
> jenkins jobs are already provisioned to work this way. We generally don't
> automate (2) out of caution, but we could.

Fine by me, any preferred format for the tags?

I'll tag whatever's deployed and stable on Monday with whatever we decide on.

Comment 3

[Wesley Dawson [:whd]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #2)
> Fine by me, any preferred format for the tags?

We don't have a standard so you can use whatever you would like. Absent a standard I would recommend the existing version string [0] prefixed with a "v" e.g. "v1.0".

> I'll tag whatever's deployed and stable on Monday with whatever we decide on.

Sounds good to me.

[0] https://github.com/mozilla/missioncontrol/blob/master/setup.py#L7

Comment 4

[William Lachance (:wlach)]

Created tag v1.0 from current master: https://github.com/mozilla/missioncontrol/releases/tag/v1.0

Comment 5

[Wesley Dawson [:whd]]

:wlach has asked that I copy over the existing dev database from a snapshot (to avoid backfilling historical data), which is going to require updating the CFN templates to conditionally accept a bootstrap snapshot in addition to the standard options. I expect that this will still be finished today, but it's a bit more work than previously estimated.

Comment 6

[Wesley Dawson [:whd]]

This has been deployed and should be available at https://missioncontrol.telemetry.mozilla.org/. There was a SNAFU with the docker image build process that :wlach corrected, and the RDS provisioning took the usual few hours and involved re-reseting some passwords, but otherwise things went smoothly. Also thanks to :digi for the quick turnaround time on the IT DNS request, since I forgot that I don't have the requisite access to do that myself anymore.

https://github.com/mozilla-services/cloudops-deployment/pull/2085

Comment 7

[Wesley Dawson [:whd]]

NI :robotblake for review of https://github.com/mozilla-services/cloudops-deployment/pull/2085

Comment 8

[Blake Imsland [:robotblake]]

Reviewed.