Move job info (detail) objects to master database

RESOLVED FIXED

Status

Tree Management
Treeherder
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

Details

Attachments

(3 attachments)

To faciliate fixing bug 1218537 (ingesting multiple job artifacts) while also hopefully helping get rid of datasource (bug 1178641)

If we store these objects seperately in their own table it will be easier to update them (as well as fetch them!)
Created attachment 8749394 [details] [review]
[treeherder] wlach:1270629 > mozilla:master
Comment on attachment 8749394 [details] [review]
[treeherder] wlach:1270629 > mozilla:master

So this is based on the PR I earlier posted to bug 1218537. I have split it into two commits: one that just adds the model to django along with a migration script, and another that modifies the ingestion code. My thinking is that we can then deploy this in two stages:

(1) Add the model, convert all existing artifacts
(2) Deploy frontend code, flip the switch to ingest into job details instead of saving job info artifacts, migrate any stragglers between the time (1) finished and this step happened.

I've done a bunch of local testing, converting all the job info artifacts currently on m-c stage. Everything looks pretty good!
Attachment #8749394 - Flags: review?(cdawson)
Comment on attachment 8749394 [details] [review]
[treeherder] wlach:1270629 > mozilla:master

I made a few comments, but nothing terribly serious.  Feel free to make the changes or not, as you see fit.
Attachment #8749394 - Flags: review?(cdawson) → review+

Comment 4

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d0ea27feb88a44907a4dd2843f6bd1c50a549360
Bug 1270629 - Add initial parts of jobdetail model/API

Ingestion not turned on (yet), pending initial migration of data

https://github.com/mozilla/treeherder/commit/af9c208792c76c22b0558c1a7c2fc4517f88ef78
Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint

This is simpler and removes one more type of artifact from the per-project
databases.

https://github.com/mozilla/treeherder/commit/238fc614c5b57b29b5d69772d1f08de149bcee42
Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion

We'll continue ingesting these artifacts in parallel with the jobdetail
objects, to make sure we can revert cleanly if something breaks.

Comment 5

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/7cb214306da114eaed39456772f1b6a8a6040c50
Revert "Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion"

This reverts commit 238fc614c5b57b29b5d69772d1f08de149bcee42.

https://github.com/mozilla/treeherder/commit/5e8e6d12a3c28f632da237c6758feaed9e9fe03d
Revert "Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint"

This reverts commit af9c208792c76c22b0558c1a7c2fc4517f88ef78.

https://github.com/mozilla/treeherder/commit/635467a77dd91d7b56db5a6fd3eaae3f2d76cf4e
Revert "Bug 1270629 - Add initial parts of jobdetail model/API"

This reverts commit d0ea27feb88a44907a4dd2843f6bd1c50a549360.
Created attachment 8750512 [details] [review]
[treeherder] wlach:1270629-2 > mozilla:master

Comment 7

2 years ago
It is not clear to me what changes are required to submit the job info/job details information. Will the old TreeherderJob add_artifact('Job Info', 'json', {'job_details': job_details}) continue to work? What is the preferred method going forward?
(In reply to Bob Clary [:bc:] from comment #7)
> It is not clear to me what changes are required to submit the job info/job
> details information. Will the old TreeherderJob add_artifact('Job Info',
> 'json', {'job_details': job_details}) continue to work? What is the
> preferred method going forward?

Yes, that will still work (in fact it will still be the only way to submit this information). Eventually we might want to change the artifact format to enable better error checking via jsonschema, but that will happen in a separate bug.

Comment 9

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/645c32fc0daeb04d10c62deb169c039c893e514a
Bug 1270629 - Add initial parts of jobdetail model/API

Ingestion not turned on (yet), pending initial migration of data

https://github.com/mozilla/treeherder/commit/82221633b838bd58224c437876eebbb160a437e8
Bug 1270629 - Replace Job Info artifacts with a "jobdetail" endpoint

This is simpler and removes one more type of artifact from the per-project
databases.

https://github.com/mozilla/treeherder/commit/aa0ceb60285b7f1d20aba85103f73fd6d89fed3d
Bug 1270629 - Temporarily revert changes to remove job info artifact ingestion

We'll continue ingesting these artifacts in parallel with the jobdetail
objects, to make sure we can revert cleanly if something breaks.

https://github.com/mozilla/treeherder/commit/2ca0d29aa1e6c79db6ef096e80d657cd4565b84c
Merge pull request #1475 from wlach/1270629-2

Bug 1270629 - Add a jobdetail table + endpoint, replacing jobinfo artifacts
Running in production now. Leaving open until I remove the transition/migration code.
James, are you still using the job info artifacts are can we stop storing those?
Flags: needinfo?(james)
Still using them, but I will fix my script when they disappear.
Flags: needinfo?(james)
Created attachment 8757422 [details] [review]
[treeherder] wlach:1270629-3 > mozilla:master

Comment 15

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/477b464dd96eef494ff4942d8fca37ab6fc369b0
Bug 1270629 - Remove migration code for job info -> detail change

https://github.com/mozilla/treeherder/commit/e2c92101df5df4486267ac2302f7c72df1bcd1d7
Merge pull request #1529 from wlach/1270629-3

Bug 1270629 - Remove migration code for job info -> detail change
Ok, migration script and legacy job info artifact generation removed. I think we're done here.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Just to clarify - this bug moved a proportion of the rows in the job_artifacts table to a new unified job details table, right? And looking at the now removed migrations script, it seems like the old rows were left in the job_artifacts table (ie copied not moved). So presumably will be covered by the standard cycle_data job as the 4 months window passes? Is this correct?

Also, are we ok with the new Heroku stage not having back history for these?

(Replication of SCL3 prod -> new Heroku RDS stage stopped running on 11th May, and the now removed migrations script hasn't been run there.)
Correct on all counts. I don't think we care about the job info artifacts on any stage environment (I only ran it on our internal stage for the purposes of verifying that the migration process worked.
Sounds good to me, thank you :-)
Depends on: 1278597
You need to log in before you can comment on or make changes to this bug.