1097080 - Analyse the difference in properties between Pulse and builds-{pending,running,4hr}

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder: Data Ingestion, defect, P5)

Description

[Ed Morley [:emorley]]

When Treeherder was being created, the idea was to use Pulse as a data source instead of builds-{pending,running,4hr}.

There are apparently properties missing from Pulse that we need, which is why Treeherder went with polling builds-{pending,running,4hr}.

Sadly what exactly was missing was never recorded, so this bug is to analyse the differences between builds-{pending,running,4hr} and Pulse, to figure out what release engineering bugs need to be filed to make Pulse suitable for Treeherder's consumption (bug 1031238).

Comment 1

[Ed Morley [:emorley]]

https://secure.pub.build.mozilla.org/builddata/buildjson/builds-pending.js
https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js
https://secure.pub.build.mozilla.org/builddata/buildjson/builds-4hr.js.gz

https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildapi.py
https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py

Presume this is our original attempt at Pulse ingestion:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/pulse.py

Comment 2

[Chris AtLee [:catlee]]

So I took a quick look at this today since I was curious.

The two things that stood out were:
- we don't get request.$buildername.submitted events for everything, and the events we do get are missing the request id.
- the build.X.started events don't include the build or request id either

The combination of these makes it impossible to tie the results from the final .log_uploaded to the initial pending/running task.

Worthwhile to fix at this point?

Comment 3

[Ed Morley [:emorley]]

I think switching to Pulse is unlikely at this point before buildbot goes away :-)