Closed Bug 1265037 Opened 6 years ago Closed 6 years ago

Add an intermediate "job" table/model to the master db

Categories

(Tree Management :: Treeherder, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wlach, Assigned: wlach)

References

Details

Attachments

(2 files)

To facilitate moving more project-specific artifacts to the master database, it would be useful to create a table/model corresponding to the job. This way we could track guid, id, in one place, and look it up easily. Eventually, this table could be fleshed out into a full-fledged replacement for the project-specific jobs table.
Comment on attachment 8741901 [details] [review]
[treeherder] wlach:1265037 > mozilla:master

Hi James, I think you'd be a good person to take a look at this. This isn't very useful by itself, but I think it should make the eventual migration away from datasource easier, while we incrementally migrate away from it with things like bug 1218537.
Attachment #8741901 - Flags: review?(james)
Blocks: 1265503
Blocks: 1266062
Attachment #8741901 - Flags: review?(james) → review+
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/104787282d0db965ea948342ce1f2926936f3d27
Bug 1265037 - Create an intermediate jobs model/table

Not really useful in-and-of-itself, but should be helpful by acting
as a centralized place to relate and map job information from what
is stored in the main treeherder database to what's in the per-project
databases. Eventually, we can extend this into a full-fledged jobs model.

https://github.com/mozilla/treeherder/commit/081d86bd7b699af7128841c266939464cdd795c9
Bug 1265037 - Temporarily disable creation of intermediate job entries

We will wait to turn this on until the initial migration is complete to
ensure jobs are more-or-less in chronological order

https://github.com/mozilla/treeherder/commit/4a49dfe6938a43980c24c7d8ecb116918e0d44a3
Merge pull request #1411 from wlach/1265037

Bug 1265037 - Create an intermediate jobs model/table
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/f744b91de1b82a35c018c64ca09b17abfbd3fbe8
Revert "Bug 1265037 - Temporarily disable creation of intermediate job entries"

This reverts commit 081d86bd7b699af7128841c266939464cdd795c9.

https://github.com/mozilla/treeherder/commit/43ff9980498ff1d179bbff1e30aa83cac35ed226
Bug 1265037 - Adapt migration script to assume most jobs migrated

https://github.com/mozilla/treeherder/commit/ceaa8d1b629b0ef4767cb5a7eb1f2ec1e208ac23
Merge pull request #1425 from wlach/1265037-2

Bug 1265037 - Enable dynamic intermediate job ingestion
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a36773fcf4135291d50605370e3650e2df415b6c
Bug 1265037 - Fix edge case in migration script

If ingestion was ongoing during the migration, it was possible for things
to error out since operations were batched inside a transaction. Since
it should be rare to need to insert new items when this script is being
run, just skip the transaction.
The migration was horrible and required hand-hacking, but I believe this is now done.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Depends on: 1272532
Depends on: 1274191
You need to log in before you can comment on or make changes to this bug.