1079867 - Handle/report missing job attributes (eg log_url)

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P4)

Description

[Ed Morley [:emorley]]

Due to bug 942616, some job attributes are missing from builds-4hr.

TBPL had special handling for these which was pretty hacky:
https://hg.mozilla.org/webtools/tbpl/file/5b782df152cb/dataimport/import-buildbot-data.py#l119

Treeherder does not have such handling, which is why there has been an increase in the number of "log not available"s for jobs.

Now the real solution here is to fix the pulse data, and make builds-4hr go away (bug 1031238), but in lieu of that I think we should:
1) Put pressure on bug 942616
2) Report these missing attributes somewhere (new relic?)
3) Possibly add the TBPL-style hacky handling for missing log urls, to reduce the number of jobs that don't display properly in treeherder.

Comment 1

[Ed Morley [:emorley]]

We do have this already:
https://github.com/mozilla/treeherder-service/blob/b05c507b46c9965f8a953d67e7d82a7d0d2f036e/treeherder/etl/buildapi.py#L520

Comment 2

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #0)
> Treeherder does not have such handling, which is why there has been an
> increase in the number of "log not available"s for jobs.

https://github.com/mozilla/treeherder-service/blob/b05c507b46c9965f8a953d67e7d82a7d0d2f036e/treeherder/etl/buildapi.py#L153

Comment 3

[Ed Morley [:emorley]]

There is very little Treeherder can do if data is bad/missing in builds-4hr.
With the move to TC this will go away.