1199291 - Create result set on the fly for jobs ingested where the RS does not exist yet.

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Cameron Dawson [:camd]]

We can simplify our job ingestion if a job includes the most basic revision info with it.  

1. If we see a job, and don't have a resultset for it, then we create the skeleton resultset (possibly with 'onhold' status so it won't show in the UI)
2. We can ingest the job against that resultset
3. Schedule fetching revisions and other information for the resultset (author, push_timestamp, etc)

This kind of relies on us being able to change our push_timestamp policy to just use the latest revision rather than a hash of the first 200 revisions+branch values.

Thoug, for buildbot, if we have the ``pushID`` when we schedule the backfill passing in the pushID and the new resultset.id.  When we get the values back from json-pushes, we can find the resultset and fill it in, including the revision_hash.

Comment 1

[Ed Morley [:emorley]]

I'm not sure I understand the difference between this and bug 1199511, could you adjust the summary a bit for both and explain in the bug? :-)

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:resultset-on-the-fly > mozilla:master

Comment 4

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:tmp-rs-on-fly > mozilla:master

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:resultset-on-the-fly > mozilla:master

Comment 6

[Cameron Dawson [:camd]]

Comment on attachment 8731919 [details] [review]
[treeherder] mozilla:resultset-on-the-fly > mozilla:master

Thanks for making yourself available to review this, Will.  :)
Ed: Adding your for feedback in case you get a few minutes to check it out.

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8731919 [details] [review]
[treeherder] mozilla:resultset-on-the-fly > mozilla:master

This looks good, there's some minor modifications I'd like to see before this goes in but nothing major.

Comment 8

[Ed Morley [:emorley]]

Comment on attachment 8731919 [details] [review]
[treeherder] mozilla:resultset-on-the-fly > mozilla:master

With Will's comments addressed, looks good! :-)

Comment 9

[William Lachance (:wlach)]

Comment on attachment 8731919 [details] [review]
[treeherder] mozilla:resultset-on-the-fly > mozilla:master

Looks good now! Please rebase and squash before landing. :)

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/41522044fdb51208a7abd564f71625dfae86ae56
Bug 1199291 - Create result set on the fly for jobs ingested where the RS does not exist yet