1266062 - Move job classifications to a Django model

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[William Lachance (:wlach)]

The job note tables need to be migrated away from datasource and moved into the central database for us to fix bug . I'm hoping this will also make the classification code easier to extend and maintain.

Comment 1

[William Lachance (:wlach)]

We should add this after bug 1265037 is fixed, so we have a fixed reference point to the per-project jobs table (until we migrate that into the master db as well).

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1266062 > mozilla:master

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8760502 [details] [review]
[treeherder] wlach:1266062 > mozilla:master

Hey James, do you want to take a look at this and let me know what you think? One potentially controversial change is that I removed the "type" field from bug job map. We identify the "type" of a classification/bug job map based on whether the user is set (if it isn't => it is autoclassified).

I don't think it makes sense to land this change until after Q2 is over, so no rush.

Comment 4

[James Graham [:jgraham]]

Comment on attachment 8760502 [details] [review]
[treeherder] wlach:1266062 > mozilla:master

I didn't look in huge amounts of detail, but what I saw looks great.

Comment 5

[William Lachance (:wlach)]

Comment on attachment 8760502 [details] [review]
[treeherder] wlach:1266062 > mozilla:master

Addressed comments and split out ORM additions into a seperate commit.

Comment 6

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/db2ccb44f4a41c34ad18dae5f384b59a782c1225
Bug 1266062 - Add bug job map, job note ORM models + migrations

https://github.com/mozilla/treeherder/commit/c44c2582950b57104f9de6bc9050cca1bc9d870e
Bug 1266062 - Move bug job map ingestion / queries to Django ORM

https://github.com/mozilla/treeherder/commit/5335bf41631aa7c7e2480be245b08f947d69c73a
Bug 1266062 - Move job note ingestion / queries to Django ORM

https://github.com/mozilla/treeherder/commit/e4797b92ea17559f7f280e2a174a36583216ada2
Merge pull request #1568 from wlach/1266062

Bug 1266062 - Move bug job map and job notes to ORM from datasource

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/9a710210cf850305172baaa746ee0158c9b14c99
Bug 1266062 - Handle non-ascii note text when migrating legacy job notes

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0898f8633b200365c6ee6a4fda382e670a8a4b5a
Bug 1266062 - Fix orangefactor submission of bugjobmap

Comment 9

[William Lachance (:wlach)]

This is now in production!

Comment 10

[Ed Morley [:emorley]]

I don't suppose you could also run the management command against the Heroku stage instance? (Plus the prototype, though you'll need to wait for James to rebase the custom branch onto master, since it's not on auto-deploy of master)

If it helps, these aliases speed up running Heroku commands against each app:

alias thh="HEROKU_APP=treeherder-heroku heroku"
alias ths="HEROKU_APP=treeherder-stage heroku"
alias thp="HEROKU_APP=treeherder-prod heroku"

Then eg:

ths run ./manage.py ...

(Heroku prod doesn't have a DB yet, so no need to run against that).

Thanks :-)

Comment 11

[William Lachance (:wlach)]

(In reply to Ed Morley [:emorley] from comment #10)
> I don't suppose you could also run the management command against the Heroku
> stage instance? (Plus the prototype, though you'll need to wait for James to
> rebase the custom branch onto master, since it's not on auto-deploy of
> master)
> 
> If it helps, these aliases speed up running Heroku commands against each app:
> 
> alias thh="HEROKU_APP=treeherder-heroku heroku"
> alias ths="HEROKU_APP=treeherder-stage heroku"
> alias thp="HEROKU_APP=treeherder-prod heroku"
> 
> Then eg:
> 
> ths run ./manage.py ...
> 
> (Heroku prod doesn't have a DB yet, so no need to run against that).
> 
> Thanks :-)

Done on stage. Not sure if it's important on the prototype, unless we have manually classified data there that we want to preserve?

Comment 12

[Ed Morley [:emorley]]

Thank you. Agreed, let's skip the prototype.