1273231 - Store job log url and parse status in main db using ORM

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[William Lachance (:wlach)]

Currently it's in a seperate db using datasource, but like other things this should be moved to the main db. It makes sense to do this now, while we're actively working on things related to it (e.g. autoclassification, see bug 1233164)

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1273231 > mozilla:master

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8753013 [details] [review]
[treeherder] wlach:1273231 > mozilla:master

Hey Ed, I was wondering if you could have a quick look at this and let me know me know what you think. Some of this is general clean up which I might move to a separate patch set.

The main unanswered question is how to apply this change without disrupting the log ingestion process. I'm thinking maybe:

1. Change the log parser task to get a url instead of a json blob describing the log object (keeping the existing code, which can actually handle this case just fine)
2. Write a small migration script to populate the new job_log table that I created based on the existing data, run it.
3. Temporarily turn off log parsing
4. Switch over to the new log reference storage code so new jobs get stored in the job_log table
5. Migrate any jobs that were ingested between 2 and 4.
6. Turn log parsing back on

If this makes sense, what's the best way of turning log parsing off temporarily?

Comment 3

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #2)
> The main unanswered question is how to apply this change without disrupting
> the log ingestion process. I'm thinking maybe:
> ...
> 
> If this makes sense, what's the best way of turning log parsing off
> temporarily?

Sorry I didn't get a chance to look at this today, will do so tomorrow.

I think the best way to turn off log parsing would be to modify the queues listed in the scripts in the bin directory, that way there just isn't a worker consuming the log parsing queues (which would be just growing in size in the meantime). This is probably preferable to disabling the celery process on the log processor nodes themselves, since I don't have a huge amount of faith that some puppet change will go overlooked, causing puppet to stomp all over it anyway.

Comment 4

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1273231-pre > mozilla:master

Comment 5

[William Lachance (:wlach)]

Comment on attachment 8755896 [details] [review]
[treeherder] wlach:1273231-pre > mozilla:master

Here's some preparatory work which should make the actual patch easier to understand after I've prepared it.

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8755896 [details] [review]
[treeherder] wlach:1273231-pre > mozilla:master

r+ with the parse endpoint left as-is until the UI no longer uses it

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/377d4e71a42510a27e597a593b623d3250173f53
Bug 1273231 - Pass just URL to log parsing task, not full object

Necessary for a subsequent simplification. We'll need to deploy
this change before others on an existing environment.

https://github.com/mozilla/treeherder/commit/eaf242ad03f94ce9ff9e65bb2f8d691309756604
Bug 1273231 - Schedule tasks with just job log url, not object

We already allowed this, but confusingly had the option of passing a
full reference to the database object. Let's clean this up.

https://github.com/mozilla/treeherder/commit/3547055561e0305aa8d1ff2de121470d4166428a
Bug 1273231 - Remove ability to set job log parse status via a web api

They were never used externally and just complicate things. If this is
ever needed we can always provide an improved interface for people
to use.

https://github.com/mozilla/treeherder/commit/44c55f04ac2d292f522667fa09796f1c2c24990a
Bug 1273231 - Bump treeherder client version to 3.0.0

The removal of the parse status methods is technically a breaking
change, even though nothing depends on them.

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8753013 [details] [review]
[treeherder] wlach:1273231 > mozilla:master

Ok, I think this is ready for review now. I actually *did* wind up taking out the method to manually reparse a log, because it turned out that functionality didn't even work for the last year, and no one noticed. :)

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8753013 [details] [review]
[treeherder] wlach:1273231 > mozilla:master

(In reply to William Lachance (:wlach) from comment #8)
> I actually *did* wind up taking
> out the method to manually reparse a log, because it turned out that
> functionality didn't even work for the last year, and no one noticed. :)

Awesome, good spot :-)

Comment 10

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/073f0bc7999b0f0127b4ef4b3459d5500d3eebbf
Bug 1273231 - Remove support for manually triggering a job parse

This was meant to reduce wait times for job parse status to be updated,
but it was accidentally broken over a year ago and no one noticed (we
started requiring hawk credentials, which were not available in the UI).
Let's just remove it. See bug 1076769 for details on the functionality
this was meant to provide.

https://github.com/mozilla/treeherder/commit/ce52f5375ffbcd03679b1752e9495de3e7404246
Bug 1273231 - Add centralized job log model + migration script

https://github.com/mozilla/treeherder/commit/0e9d516f8a785ddcdaa7b4f704eb64e618a3f461
Bug 1273231 - Store job log url/name/parse information in central database

https://github.com/mozilla/treeherder/commit/d61ab5eadf8caf17c2f6314093e1a7ca090c2b05
Merge pull request #1496 from wlach/1273231

Bug 1273231 - Store job log url/name/parse information in central database

Comment 11

[William Lachance (:wlach)]

Migration and deployment complete on both stage and prod