Unblock the next Treeherder prod deploy

RESOLVED FIXED

Status

Tree Management
Treeherder: Infrastructure
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

Details

Production was last deployed from master 10 days ago, and is currently 57 commits behind master:
https://github.com/mozilla/treeherder/compare/production...master

To unblock the next deploy, we need to:

1) Check exactly what migrations will be run, and whether any of them need hand-holding (I believe some will). In addition, the indexes on `failureline` were changed twice -- perhaps if we make the changes manually we'll avoid Django doing a CREATE-DROP-CREATE?
https://github.com/mozilla/treeherder/compare/production...master#diff-5ee69346b60806ec7256b084782fe668

2) Re-enable auto-classifcation and the store-error-lines task, which will involve:

  (a) checking we're happy with the perf, now that a few tweaks have been made
  (b) Setting `AUTOCLASSIFY_JOBS` on stage+prod back to 1
  (c) Reverting the prod hotfix (https://github.com/mozilla/treeherder/compare/hotfix-autoclassify) - which will happen on the next deploy
  (d) clearing the several million task backlog that's built up whilst those tasks were disabled (on VPN see http://treeherder-rabbitmq2.private.scl3.mozilla.com:15672/#/queues)

3) Perform the standard checks against stage to make sure nothing else has broken
Depends on: 1278451
Only two model migrations are missing; these are the failure_line index ones; I think running only the second is approximately the right thing to do, but it might be a sensible idea to just do the CREATEs with a unique name before dropping the existing index.

jgraham@luna:~/develop/treeherder$ git diff --name-status origin/production.. treeherder/model/migrations/ 
A       treeherder/model/migrations/0027_failure_line_idx_signature_test.py
A       treeherder/model/migrations/0028_test_match_created_index.py
To clear the backlog, I was thinking I could deploy this:
https://github.com/mozilla/treeherder/compare/hotfix-autoclassify...hotfix-autoclassify2
(another commit on top of the existing hotfix)

And then we can clear the queues without them building back up, ready for the actual deploy.

James, does the change look ok to you?
(In reply to Ed Morley [:emorley] from comment #2)
> To clear the backlog, I was thinking I could deploy this:
> https://github.com/mozilla/treeherder/compare/hotfix-autoclassify...hotfix-
> autoclassify2
> (another commit on top of the existing hotfix)

Done.

And queues cleared on prod.

Updated

2 years ago
Blocks: 1090643
<jgraham> db migration done on production
<emorley> great - ty
<emorley> do you need me to do the --fake? And for both the perf model and the treeherder model changes or just the latter?
<emorley> could you update the bug?
<jgraham> I just did the treeherder model changes, and if you don't mind doing the fake that would work; I forgot to prod the db
(In reply to Ed Morley [:emorley] from comment #0)
> 2) Re-enable auto-classifcation and the store-error-lines task, which will
> involve:
> 
>   (a) checking we're happy with the perf, now that a few tweaks have been
> made

Done.

>   (b) Setting `AUTOCLASSIFY_JOBS` on stage+prod back to 1

Will leave this to bug 1278451 and deps.

>   (c) Reverting the prod hotfix
> (https://github.com/mozilla/treeherder/compare/hotfix-autoclassify) - which
> will happen on the next deploy

Done (have deployed)

>   (d) clearing the several million task backlog that's built up whilst those
> tasks were disabled (on VPN see
> http://treeherder-rabbitmq2.private.scl3.mozilla.com:15672/#/queues)

Done a few days ago.

> 3) Perform the standard checks against stage to make sure nothing else has
> broken

Done.

Migration --fake run:

[emorley@treeherderadm.private.scl3 treeherder-service]$ sudo git fetch --quiet origin production
[emorley@treeherderadm.private.scl3 treeherder-service]$ sudo git reset --hard FETCH_HEAD
HEAD is now at 5d4b9ec Bug 1244989 - Move the 'add profile' and 'add exclusion' buttons to the top of the sheriff panel (#1598) r=camd
[emorley@treeherderadm.private.scl3 treeherder-service]$ sudo find . -type f -name "*.pyc" -delete
[emorley@treeherderadm.private.scl3 treeherder-service]$ source ../treeherder-env.sh
[emorley@treeherderadm.private.scl3 treeherder-service]$ ../venv/bin/python manage.py migrate model --fake
Operations to perform:
  Apply all migrations: model
Running migrations:
  Rendering model states... DONE
  Applying model.0027_failure_line_idx_signature_test... FAKED
  Applying model.0028_test_match_created_index... FAKED
The following content types are stale and need to be deleted:

    model | device
    model | machinenote

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes
[emorley@treeherderadm.private.scl3 treeherder-service]$ ../venv/bin/python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: log_parser, django_browserid, corsheaders, autoclassify, staticfiles, messages, rest_framework_swagger, runserver_nostatic, webapp, rest_framework, embed, hawkrest, etl
  Apply all migrations: perf, sessions, admin, sites, auth, contenttypes, credentials, model
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying perf.0017_performancebugtemplate... OK
[emorley@treeherderadm.private.scl3 treeherder-service]$ ths
[emorley@treeherderadm.private.scl3 treeherder-service]$ source ../treeherder-env.sh
[emorley@treeherderadm.private.scl3 treeherder-service]$ ../venv/bin/python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: log_parser, django_browserid, corsheaders, autoclassify, staticfiles, messages, rest_framework_swagger, runserver_nostatic, webapp, rest_framework, embed, hawkrest, etl
  Apply all migrations: perf, sessions, admin, sites, auth, contenttypes, credentials, model
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  No migrations to apply.
Assignee: nobody → emorley
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.