1464107 - write a script for doing migrations in post-deploy

Status: RESOLVED FIXED

(Socorro :: Infra, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We run alembic and Django migrations after deploys as a post-migration step. Currently, that code is in the cloudops repo.

This bug covers writing a script that runs the migrations and have it live in the Socorro report, then fixing the deploys to use that script.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We have this in the cloudops repo:

  socorro::exec { 'socorro_upload_telemetry_schema':
    command => "${docker} /app/socorro-cmd upload_telemetry_schema",
  }

  socorro::exec { 'socorro_migrate_django':
    command => "${docker} /app/webapp-django/manage.py migrate --no-input",
    onlyif  => "${docker} /app/webapp-django/manage.py showmigrations | grep -qF '[ ]'",
  }

  socorro::exec { 'socorro_migrate_alembic':
    command => "${docker} alembic -c docker/config/alembic.ini upgrade head",
    unless  => "${docker} alembic -c docker/config/alembic.ini current | grep -qF '(head)'",
  }


Both migration systems are idempotent and won't do anything if there's nothing to do, so running them every time is fine. I'll write a script to do that now.

Comment 2

[Brian Pitts]

Nice idea!

Running these commands in a python script with access to your code and settings would let you report errors in them to sentry, in case you prefer that to datadog events.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Oh, that's an interesting idea. That's probably easier than datadog events.

This blog post covers sending things to Sentry via bash:

https://blog.sentry.io/2017/11/28/sentry-bash

That seems pretty straight-forward.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4461: fix bug 1464107: add script to run migrations

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Here's the docs for sentry-cli:

https://docs.sentry.io/learn/cli/send-event/

and:

https://docs.sentry.io/learn/cli/send-event/#bash-hook

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/c04df654f46c040cd6faedf6cafc63a604aec23c
fix bug 1464107: add script to run migrations

This codifies all the migration stuff in a single script which we'll run
as a post-deploy script.

This adds curl and gawk to the Ubuntu setup. We need curl for installing
sentry-cli and we need gawk because it's the version of awk that sentry-cli
uses.

This also adds the sentry-cli to the crontabber container. This enables
us to send errors in bash scripts to Sentry.

https://github.com/mozilla-services/socorro/commit/2519cf9d88c80620374f0f20ed4ce7ede26c8579
Merge pull request #4461 from willkg/1464107-migration-script

fix bug 1464107: add script to run migrations

Comment 7

[Brian Pitts]

willkg, are we ready for me to change the relevant puppet code to

  socorro::exec { 'socorro_upload_telemetry_schema':
    command => "${docker} /app/socorro-cmd upload_telemetry_schema",
  }

  socorro::exec { 'socorro_run_migrations':
    command => "${docker} /app/scripts/run_migrations.sh",
  }

?

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Brian: I was thinking we'd wait for this to get to -stage, then test it out there, then we could adjust the puppet script.

To test it on stage, we'd want to do the following:

1. Run:

   ${docker} /app/scripts/run_migrations.sh

   * Does it kick up any errors? It shouldn't.
   * Does it show that migration state is fine? It should.
   * Does it show a datestamp? It should.

2. Edit the script changing "alembic" to "lembic" (which doesn't exist) and run the script.

   * Does it kick up an error? It should.
   * Does it send an error to Sentry? It should.

Can you do that when you have a moment today?

Comment 9

[Brian Pitts]

Makes sense to me! I'll plan to do it today.

Comment 10

[Brian Pitts]

Output for 1:
RAVEN_DSN defined--enabling sentry.
Mon Jun  4 16:46:43 UTC 2018
DatadogMetrics configured: localhost:8125 None
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, eventlog, sessions, status, symbols, tokens, waffle
Running migrations:
  No migrations to apply.
Creating Socorro permissions:
Creating Socorro groups:
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.


Output for 2:
RAVEN_DSN defined--enabling sentry.
Mon Jun  4 16:44:47 UTC 2018
scripts/run_migrations.sh: line 26: python4: command not found

and it's in sentry at https://sentry.prod.mozaws.net/operations/socorro-new-stage/issues/4374649/

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

That's awesome! Seems good to go to me.

Comment 12

[Brian Pitts]

Puppet side was merged in https://github.com/mozilla-services/cloudops-deployment/pull/2080 so this should be good to close.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Yay us! Marking as fixed!