1162964 - Reduce duplication in buildapi.py to make it easier to maintain/debug

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Ed Morley [:emorley]]

Broken out of bug 1162947.

Comment 1

[Ed Morley [:emorley]]

Attachment: Reduce duplication in buildapi.py & remove builds-4hr analyser

With this PR, buildapi.py goes from 1000 lines to 500 :-)

I've split it up into a few commits to make the review easier (and to reduce risk, given the fragility of the builds-* etl).

Aside from the switch to iteritems() for some of the outer loops (pending was using it, running wasn't), this should be a no-op.

Comment 2

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/6e05cfae03fea38c0dd1631ec07cb8debe10150e
Bug 1162964 - Factor out the buildapi pending/running source name

This reduces the differences between RunningTransformerMixin and
PendingTransformerMixin, so in a later commit we can combine the two,
and switch to passing in the name of the source ("pending"/"running")
from the callers.

https://github.com/mozilla/treeherder/commit/fc7631ffa2c6216e090f797098e4b7cc664024b4
Bug 1162964 - Rename '{pending,running}_job' to 'job'

https://github.com/mozilla/treeherder/commit/1a5f3d343aa3694c3f3dd2465767109c4ba32bbc
Bug 1162964 - Factor out pending/running job dict differences

In a later commit we'll put these inside conditionals based on 'source',
so we can use the same method for both pending and running.

https://github.com/mozilla/treeherder/commit/d4d2bd4b8b962462ace39d95e244acd79f2bd10a
Bug 1162964 - Use iteritems() for RunningTransformerMixin too

Previously we were using it for the outer PendingTransformerMixin loop,
but not RunningTransformerMixin's.

With this and prior changes, the two methods are now virtually identical
and can be more safely combined.

https://github.com/mozilla/treeherder/commit/f17ea5b4238d7ab6190252ba7450f27b95d76a07
Bug 1162964 - Combine {Running,Pending}TransformerMixin

The two methods were identical apart from the artifacts_request_id,
request_id and start_timestamp properties, which are now conditional on
the value of 'source'.

https://github.com/mozilla/treeherder/commit/cd91887966266fbb6432cc72ee34297eb53557de
Bug 1162964 - Remove the builds-4hr analyser

Since we no longer use it & it's likely out of date anyway.