1178227 - [Meta] Simplify the Treeherder data model

Status: NEW

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

We have a whole bunch of complexity to support use-cases we thought might occur, but haven't yet. Let's remove things we aren't using, since even if a use-case does present itself in the future, we may wish to implement differently when it actually comes to it.

Comment 1

[Ed Morley [:emorley]]

Some more ideas for later:
* Remove pending_eta/running_eta from the jobs table (IMO we should have one reference data lookup for this in the UI and then use that to generate ETAs for each job dynamically in the UI)
* Sort out machine platform vs build platform (given bug 1056928) and decide if we really need both
* do something with option_collation
* remove the "tier" field from the job table (IMO it doesn't belong there; it's a visibility layer thing, not a job property)
* Checking field sizes
* Checking indexes (both missing and unnecessary)

Comment 2

[Ed Morley [:emorley]]

* Remove the 'type' field from the result_set table
* Rename result_set to "pushes"?
* Replace revision_hash with 'revision' (since I don't think we're going to need to support the multi-repo use case any more, now that most things use a manifest to pin revisions)

Comment 3

[Ed Morley [:emorley]]

* Consolidate the concepts of "project" and "repository"
* Actually start using the "product" field on the jobs table, or else remove it

Comment 4

[Ed Morley [:emorley]]

* Remove the result_set aggregate_id field since it's not used anywhere, and the only reference to what it does is: "A id to use for aggregating result_sets. This is primarily used for supporting a github like pull request work flow but could also be used for any other type of grouping."

Comment 5

[Ed Morley [:emorley]]

* machine_note table

Comment 6

[Cameron Dawson [:camd]]

after the objectstore is removed in bug 1140349:

1. delete the objectstore databases.  
2. remove ``contenttype`` field from datasources table

Comment 7

[Cameron Dawson [:camd]]

In looking at the data format for our pulse ingestion, I'm scrutinizing each field.  Some of these stood out as perhaps not used or useful.  Need a 50 cent word here...  vestigial?  :)

It seems we don't store:
build_url
machine VM status

We do store "product" (as opposed to "project") but we don't appear to use it in the UI.  Do we need/want this?

Perhaps these are less about "data model" and more about data ingestion.

Comment 8

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #7)
> It seems we don't store:
> build_url
> machine VM status

Let's get rid of them both. Also from the object here:
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py#L981 (and related)

> We do store "product" (as opposed to "project") but we don't appear to use
> it in the UI.  Do we need/want this?

So I think the reason for this was so we could differentiate between products (eg: "Firefox", "Firefox for Android", "B2G", ...) and projects (iirc that's the name we use for repos? eg "mozilla-central", "mozilla-inbound", ...).

The complication is that a "project" could have several "products" built from it (eg just look at the different rows on mozilla-central). Plus at the time we did the spec for Treeherder, we were thinking some builds might use multiple repos, so have two ways to slice the data (eg Thunderbird builds, where we could slice it by mozilla-central repo pushes or by comm-central pushes).

I think there is a use case for having "product" around - which is say the B2G team wanting to run queries against all B2G jobs across multiple repositories. So really product would be a grouping of job types - thereby reducing the overloading of "os_platform" - see bug 1060769.

I think we maybe need to decide our plan for this long term?

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #8)
> I think we maybe need to decide our plan for this long term?

And by that I mean us take a look at bug 1060769 comment 1 and see if that or something similar makes sense :-)

Comment 10

[Ed Morley [:emorley]]

Remaining candidates:
* Remove running_eta from the jobs table (and look up dynamically)
* Sort out machine platform vs build platform (given bug 1056928) and decide if we really need both
* Do something with option_collection
* Remove the "tier" field from the job table (IMO it doesn't belong there; it's a visibility layer thing, not a job property)
* Remove the "type" field from the result_set table
* Consolidate the concepts of "project" and "repository"
* Actually start using the "product" field on the jobs table, or else remove it

+ checking field sizes and checking indexes (both missing and unnecessary)