1280496 - Unblock the next Treeherder prod deploy

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Production was last deployed from master 10 days ago, and is currently 57 commits behind master:
https://github.com/mozilla/treeherder/compare/production...master

To unblock the next deploy, we need to:

1) Check exactly what migrations will be run, and whether any of them need hand-holding (I believe some will). In addition, the indexes on `failureline` were changed twice -- perhaps if we make the changes manually we'll avoid Django doing a CREATE-DROP-CREATE?
https://github.com/mozilla/treeherder/compare/production...master#diff-5ee69346b60806ec7256b084782fe668

2) Re-enable auto-classifcation and the store-error-lines task, which will involve:

  (a) checking we're happy with the perf, now that a few tweaks have been made
  (b) Setting `AUTOCLASSIFY_JOBS` on stage+prod back to 1
  (c) Reverting the prod hotfix (https://github.com/mozilla/treeherder/compare/hotfix-autoclassify) - which will happen on the next deploy
  (d) clearing the several million task backlog that's built up whilst those tasks were disabled (on VPN see http://treeherder-rabbitmq2.private.scl3.mozilla.com:15672/#/queues)

3) Perform the standard checks against stage to make sure nothing else has broken

Comment 1

[James Graham [:jgraham]]

Only two model migrations are missing; these are the failure_line index ones; I think running only the second is approximately the right thing to do, but it might be a sensible idea to just do the CREATEs with a unique name before dropping the existing index.

jgraham@luna:~/develop/treeherder$ git diff --name-status origin/production.. treeherder/model/migrations/ 
A       treeherder/model/migrations/0027_failure_line_idx_signature_test.py
A       treeherder/model/migrations/0028_test_match_created_index.py

Comment 2

[Ed Morley [:emorley]]

To clear the backlog, I was thinking I could deploy this:
https://github.com/mozilla/treeherder/compare/hotfix-autoclassify...hotfix-autoclassify2
(another commit on top of the existing hotfix)

And then we can clear the queues without them building back up, ready for the actual deploy.

James, does the change look ok to you?

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #2)
> To clear the backlog, I was thinking I could deploy this:
> https://github.com/mozilla/treeherder/compare/hotfix-autoclassify...hotfix-
> autoclassify2
> (another commit on top of the existing hotfix)

Done.

And queues cleared on prod.

Comment 4

[Ed Morley [:emorley]]

<jgraham> db migration done on production
<emorley> great - ty
<emorley> do you need me to do the --fake? And for both the perf model and the treeherder model changes or just the latter?
<emorley> could you update the bug?
<jgraham> I just did the treeherder model changes, and if you don't mind doing the fake that would work; I forgot to prod the db

Comment 5

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #0)
> 2) Re-enable auto-classifcation and the store-error-lines task, which will
> involve:
> 
>   (a) checking we're happy with the perf, now that a few tweaks have been
> made

Done.

>   (b) Setting `AUTOCLASSIFY_JOBS` on stage+prod back to 1

Will leave this to bug 1278451 and deps.

>   (c) Reverting the prod hotfix
> (https://github.com/mozilla/treeherder/compare/hotfix-autoclassify) - which
> will happen on the next deploy

Done (have deployed)

>   (d) clearing the several million task backlog that's built up whilst those
> tasks were disabled (on VPN see
> http://treeherder-rabbitmq2.private.scl3.mozilla.com:15672/#/queues)

Done a few days ago.

> 3) Perform the standard checks against stage to make sure nothing else has
> broken

Done.

Migration --fake run:

[[e-mail]3 treeherder-service]$ sudo git fetch --quiet origin production
[[e-mail]3 treeherder-service]$ sudo git reset --hard FETCH_HEAD
HEAD is now at 5d4b9ec Bug 1244989 - Move the 'add profile' and 'add exclusion' buttons to the top of the sheriff panel (#1598) r=camd
[[e-mail]3 treeherder-service]$ sudo find . -type f -name "*.pyc" -delete
[[e-mail]3 treeherder-service]$ source ../treeherder-env.sh
[[e-mail]3 treeherder-service]$ ../venv/bin/python manage.py migrate model --fake
Operations to perform:
  Apply all migrations: model
Running migrations:
  Rendering model states... DONE
  Applying model.0027_failure_line_idx_signature_test... FAKED
  Applying model.0028_test_match_created_index... FAKED
The following content types are stale and need to be deleted:

    model | device
    model | machinenote

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes
[[e-mail]3 treeherder-service]$ ../venv/bin/python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: log_parser, django_browserid, corsheaders, autoclassify, staticfiles, messages, rest_framework_swagger, runserver_nostatic, webapp, rest_framework, embed, hawkrest, etl
  Apply all migrations: perf, sessions, admin, sites, auth, contenttypes, credentials, model
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying perf.0017_performancebugtemplate... OK
[[e-mail]3 treeherder-service]$ ths
[[e-mail]3 treeherder-service]$ source ../treeherder-env.sh
[[e-mail]3 treeherder-service]$ ../venv/bin/python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: log_parser, django_browserid, corsheaders, autoclassify, staticfiles, messages, rest_framework_swagger, runserver_nostatic, webapp, rest_framework, embed, hawkrest, etl
  Apply all migrations: perf, sessions, admin, sites, auth, contenttypes, credentials, model
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  No migrations to apply.