1311634 - ATMO V2: Port to dockerflow

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Jannis Leidel [:jezdez]]

In parallel to the Heroku set up it was decided to run the production environment in dockerflow.

Comment 1

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:circle > mozilla:master

Comment 2

[Blake Imsland [:robotblake]]

I added several more commits to the PR to address concerns about running multiple processes and fix a build error in circle. Review / approval of the PR is currently blocking any movement forward on deployment to our production environment.

Comment 3

[Jannis Leidel [:jezdez]]

:robotblake, I've merged the PR earlier after review and some modifications for our test setup (it uses docker now for running the tests as well instead of using Travis CI's/CircleCI's own Python. That should give us closer dev/prod parity.

Comment 4

[Jannis Leidel [:jezdez]]

Blake, regarding bug 1311288 can you clarify which Sentry project to use for the various setups?

The plan as I understand it when Dockerflow is running is to have three environments:

1. atmo-dev on Heroku
2. atmo-stage on Dockerflow
3. atmo-prod on Dockerflow

Currently we have atmo-staging and atmo-prod on Heroku, so my plan for moving to it is: 

- create a new app called atmo-dev on Heroku using the atmo-dev Sentry credentials
- retire atmo-staing and atmo-prod on Heroku
- create a atmo-prod Sentry project
- ask you to add the atmo-stage and atmo-prod Sentry credentials as environment variables SENTRY_DSN to the Dockerflow

Does that sound right for you?

What else is needed to make Dockerflow a reality?

Should we use Git tags to do releases (as the circle.yml is configured to do that)?

Are stage deployments automatic?

What are the steps to do deployments from stage to prod?

Comment 5

[Blake Imsland [:robotblake]]

That all sounds correct.

I have a staging deploy working but I need to double-check a couple things before I give the :+1:.

I'd prefer to deploy from tags.

Stage deployments will be automatic (again, ideally from tags).

Promotion to prod is currently triggered by someone clicking a button in the Jenkins pipeline overview for ATMO (this can be done by me or made available to :jezdez and :mdoglio).

Comment 6

[Jannis Leidel [:jezdez]]

(In reply to Blake Imsland [:robotblake] from comment #5)
> That all sounds correct.
> 
> I have a staging deploy working but I need to double-check a couple things
> before I give the :+1:.

Awesome, please keep us in the loop about this so we can plan the roll out and termination of the Heroku instances next week accordingly.

> I'd prefer to deploy from tags.
> 
> Stage deployments will be automatic (again, ideally from tags).
> 
> Promotion to prod is currently triggered by someone clicking a button in the
> Jenkins pipeline overview for ATMO (this can be done by me or made available
> to :jezdez and :mdoglio).

Okay that makes sense, that means our dev workflow will be:

Dev:
- work on code, push changes to GitHub on master branch, which auto-deployed to Heroku if CI passes
- then we manually test things on Heroku (optionally), and amend the code if needed

Stage:
- tag a release
- push to GitHub
- have CircleCI auto-push the built Docker image to Docker hub
- stage is auto-deployed using the tagged Docker image

Prod:
- press a button in Jenkins to promote stage to prod once we're comfortable with state of stage

That should give us enough flexibility to review changes before they hit stage/prod environments. Obviously each of those steps require the automatic testing via Circle CI to pass.

For the record, https://mana.mozilla.org/wiki/display/SVCOPS/Jenkins has the info about the Jenkins instance that is only available via VPN.

Comment 7

[Jannis Leidel [:jezdez]]

Blake, can you clarify how things look like with stage and prod on Dockerflow? When can we expect the deploy to work?

The atmo-staging.herokuapp.com instance reports to https://sentry.prod.mozaws.net/operations/atmo-dev/ now and I've enabled client side (JS) reporting as well.

One thing I noted is that https://sentry.prod.mozaws.net/operations/atmo-stage/ is currently receiving errors, but we haven't configured it anywhere. Can you look into which environment is pushing error there and turn it off? Could this be a case of a dangling worker containers on deploy?

Comment 8

[Blake Imsland [:robotblake]]

I had it working in stage last week but ran into some issues with the assets missing from the Docker container. I've got the documentation mostly together and a PR to push to the repository, but other than that things are very close to being ready to go. The environment that was pushing to "atmo-stage" in Sentry was an old version of the app that I had manually deployed to stage for testing purposes, it's been terminated now.

I'd like to give you and :mdoglio a chance to look over the documentation I'll be sending along this evening and then maybe have a short meeting in case there are any questions, maybe this coming Wednesday (11/02/16) if that would work for the both of you.

Comment 9

[Jannis Leidel [:jezdez]]

I'm looking forward to take a look at the changes needed, thanks!

Yep, a meeting on Wed sounds good to me.

Comment 10

[Jannis Leidel [:jezdez]]

Blake, I've made the changes to the deploy repo here: https://github.com/mozilla-services/cloudops-deployment/pull/237

Please review it.

Also, since you said secrets are merged differently and aren't stored in the cloudops-deployment repo that I have access to, here is the list of MUST HAVE environment variables for ATMO to work:

AWS_ACCESS_KEY_ID
AWS_DEFAULT_REGION
AWS_SECRET_ACCESS_KEY
* DATABASE_URL
* DJANGO_ALLOWED_HOSTS
* DJANGO_CONFIGURATION
* DJANGO_DEBUG
DJANGO_SECRET_KEY
* DJANGO_SITE_URL
NEW_RELIC_APP_NAME
NEW_RELIC_CONFIG_FILE = /app/newrelic.ini
NEW_RELIC_ENVIRONMENT
NEW_RELIC_LICENSE_KEY
NEW_RELIC_LOG = stdout
* REDIS_URL
SENTRY_DSN
SENTRY_PUBLIC_DSN

I've marked [*] the variables that have been set in the cloudops-deployment repo.
Please double check if all the other variable have been set in the secret place that is merged into during deployment.

Note that the SECRET_KEY environment variable has been renamed to DJANGO_SECRET_KEY, and needs to be adjusted in the secret store. The other Django specific env vars have been prefixed with "DJANGO_" as well.

Comment 11

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug1311634 > mozilla:master

Comment 12

[Jannis Leidel [:jezdez]]

Closing as that's done.