1069467 - Need fallback import from daily jobs archive if builds-4hr is down

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Ed Morley [:emorley]]

TBPL has two method of importing data:
1) For keeping up to date: builds4hr [1]
2) For initial setup (or if something went wrong with #2): using N days worth of the archived daily job results [2]

If builds-4hr has a problem, or should some extra backfilling (beyond the last 4 hours) be required say in a local Vagrant instance, someone can manually run #2 specifying N.

Currently treeherder only supports #1 [3], so if there was ever a problem with builds4hr [4] - we wouldn't have an easy way to fix the missing data.

TBPL's implementation is at:
https://hg.mozilla.org/webtools/tbpl/file/default/dataimport/import-buildbot-data.py#l297

From my understanding of the files in buildjson & also inspection of the TBPL implementation, the daily and 4hr formats are the same, so this should be pretty straight-forwards to add to:

https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildapi.py


[1] https://secure.pub.build.mozilla.org/builddata/buildjson/builds-4hr.js.gz
[2] eg https://secure.pub.build.mozilla.org/builddata/buildjson/builds-2014-09-18.js.gz
[3] https://github.com/mozilla/treeherder-service/blob/f4d64e72cf174233c0d39eb6f83649a6d6654a58/treeherder/settings/base.py#L223
[4] https://bugzilla.mozilla.org/showdependencytree.cgi?id=926246&maxdepth=1&hide_resolved=0

Comment 1

[Ed Morley [:emorley]]

This would also help us recover from situations like those had recently, where try was missing 12+ hours worth of pushes and we had no way to re-import.

Comment 2

[Ed Morley [:emorley]]

P2 given this would help with bug 1151806.

Comment 3

[Ed Morley [:emorley]]

Now that bug 1096878 is wontfix, it's less likely we'd need this.
We also haven't really missed having this in the last year, so let's just wontfix.
I'd rather not invest any more time than necessary into buildbot support.