1077136 - Fetch missing resultsets during job ingestion if we encounter a job from an unknown result set

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Cameron Dawson [:camd]]

This will be a separate task that gets populated if we look in pending, running of builds4hr and find jobs for repos we support that have not yet ingested their push/resultset.  This will catch if we missed some resultsets due to an infrastructure issue.

Comment 2

[Ed Morley [:emorley]]

camd's PR:
https://github.com/mozilla/treeherder-service/pull/237

Comment 3

[Cameron Dawson [:camd]]

Attachment: pull request

Comment 4

[Mauro Doglio [:mdoglio]]

Comment on attachment 8501313 [details] [review]
pull request

Tests are failing at the moment, r? me when you fix it

Comment 5

[Cameron Dawson [:camd]]

Attachment: l10n_revision example

Comment 6

[Cameron Dawson [:camd]]

Running into some odd data coming out of builds4hr/pending/running.  This happens more (seemingly) in mozilla-central, but other repos as well.  As an example, a job in the l10n_revision example isn't in the mozilla-central mercurial.  Because it's in the locale repo.  So, the revision property and the branch name in the object don't relate to each other.  

Not clear if the branch property should be mozilla-central-l10n or equivalent.

Comment 7

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #6)
> Running into some odd data coming out of builds4hr/pending/running.  This
> happens more (seemingly) in mozilla-central, but other repos as well.  As an
> example, a job in the l10n_revision example isn't in the mozilla-central
> mercurial.  Because it's in the locale repo.  So, the revision property and
> the branch name in the object don't relate to each other.  
> 
> Not clear if the branch property should be mozilla-central-l10n or
> equivalent.

If this is the case we saw yesterday on IRC, then we can skip these jobs and can file a releng bug to fix up builds-4hr

Comment 8

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/27b0b19c3416175034fcca82186ab70dde46ee3a
Bug 1077136 - new async task to catch missing resultsets

https://github.com/mozilla/treeherder-service/commit/49d228a62e7893c2d31eb6cc0adff0657603eb60
Bug 1077136 - new cleanup_tasks file to avoid circular imports

https://github.com/mozilla/treeherder-service/commit/4cad93d0a76f53e948eb2986055b26432d845380
Bug 1077136 - test for new missing resultset task

https://github.com/mozilla/treeherder-service/commit/3a1ae7df291ecf5d8a8628960f1e769cc0344b94
Bug 1077136 - fix urlencode params for changesets

https://github.com/mozilla/treeherder-service/commit/9cf09d8372f69e1999f457fc1cfa646944b53f8e
bug 1077136 - test passing for ingestion

https://github.com/mozilla/treeherder-service/commit/89639f96ae2f4d73713dc8ccc26489d13da2deb3
Bug 1077136 - fix builds4hr tests and improve tasks per feedback

https://github.com/mozilla/treeherder-service/commit/1d686f2d238bbfb56c48954555ee9364f4af42db
Bug 1077136 - checks before schedulig async task

https://github.com/mozilla/treeherder-service/commit/ad723005d62d8dbb1b9b2e95a1aeab252715dffa
Bug 1077136 - Use set instead of list for missing_revisions to prevent dups

https://github.com/mozilla/treeherder-service/commit/3127f53e19e435a99a708f4e57ec0664eb086f61
Bug 1077136 - removing dead socket.io code

https://github.com/mozilla/treeherder-service/commit/e1810cf419433220d52f72ea9701362566f2093f
Bug 1077136 - better logging

https://github.com/mozilla/treeherder-service/commit/06728b98356f325bdd5b41313e55404dd2414e83
Bug 1077136 - handle pushes missing from json-pushes gracefully

https://github.com/mozilla/treeherder-service/commit/914ef7c7f152bbcc47fe127ba3d74688004f3258
Bug 1077136 - fix resultset with missing revisions on ingestion

also fix broken tests

https://github.com/mozilla/treeherder-service/commit/08c04ae79ab0b4d4eac8b68e1657bae758209125
bug 1077136 - ingest bad pushes as 'onhold' so we don't keep trying to ingest them

https://github.com/mozilla/treeherder-service/commit/184ac0aff96b10902c0114d0686e1801629dd39a
bug 1077136 - fix doc bug and merge from master

https://github.com/mozilla/treeherder-service/commit/5fa23c2aa0c18dbf034f5d0edbf933dcba0b3329
bug 1077136 - implement mauro's review suggestions

https://github.com/mozilla/treeherder-service/commit/9e414b8a61d805da7cee70feca8ec782dbfd71fa
bug 1077136 - added test and fixed when RS didn't get set to 'onhold'

https://github.com/mozilla/treeherder-service/commit/b1adcd4a55c06fbc980c94f566b1d5576f5f08ec
Merge pull request #246 from mozilla/queue-missing-resultsets2

Bug 1077136 - Queue missing resultsets2

Comment 9

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #6)
> Running into some odd data coming out of builds4hr/pending/running.  This
> happens more (seemingly) in mozilla-central, but other repos as well.  As an
> example, a job in the l10n_revision example isn't in the mozilla-central
> mercurial.  Because it's in the locale repo.  So, the revision property and
> the branch name in the object don't relate to each other.  
> 
> Not clear if the branch property should be mozilla-central-l10n or
> equivalent.

Filed bug 1085649 to analyse these cases and figure out whether treeherder or buildbot* needs changing.