1183537 - Stop outputting the 'builders', 'slaves' & 'masters' keys in builds-4hr

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

builds-4hr has the following top level keys:
* builders
* builds
* masters
* slaves
* starttime
* endtime

Treeherder only uses "builds" - and given the name of the file is "builds-4hr" not "all the things", I'm not sure the rest belongs in there.

If there isn't anything else obvious using it, let's remove the 'builders', 'slaves' & 'masters' keys - to reduce load on the DB, speed up builds-4hr generation and also the time it takes for Treeherder to process it.

These keys account for ~55,000 lines of builds-4hr and also the majority of the runtime of the queries:

{'masters': 0.13559746742248535, 'builders': 4.3739259243011475, 'builds': 0.280078649520874, 'slaves': 0.01535797119140625}

Chris, do you know of anything using these keys?

Comment 1

[Ed Morley [:emorley]]

Attachment: Don't output the 'builders', 'slaves' & 'masters' keys

Comment 2

[Chris AtLee [:catlee]]

You rely on the buildername/slavename properties rather than builder_id / slave_id?

Comment 3

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #2)
> You rely on the buildername/slavename properties rather than builder_id /
> slave_id?

Yeah

https://github.com/mozilla/treeherder/blob/3f34e5da134909b3eb12769e4dad525e59b429db/treeherder/etl/buildapi.py#L211
https://github.com/mozilla/treeherder/blob/3f34e5da134909b3eb12769e4dad525e59b429db/treeherder/etl/buildapi.py#L238

We don't even enumerate anything other than the 'builds' key:
https://github.com/mozilla/treeherder/blob/3f34e5da134909b3eb12769e4dad525e59b429db/treeherder/etl/buildapi.py#L132

Comment 4

[Chris AtLee [:catlee]]

Armen, do you rely on those?

Comment 5

[Alice Scarpa [:adusca]]

In mozci, the helper function that fetches the data from builds-4hr just returns the contents of the key "builds" and discards everything else [1]. Every function that uses builds-4hr data gets is data from this helper function, so the keys 'builders', 'slaves' & 'masters' are ignored by mozci.

[1] https://github.com/armenzg/mozilla_ci_tools/blob/master/mozci/sources/buildjson.py#L47

Comment 6

[Armen [:armenzg]]

What Alice says.

Please let us know when this is live to see if we need to land anything for mozci.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8633358 [details] [diff] [review]
Don't output the 'builders', 'slaves' & 'masters' keys

Ok, let's give it a shot.

I think we should advertise this change on the relevant newsgroups, with instructions on how to get the now-removed data.

Comment 8

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #7)
> I think we should advertise this change on the relevant newsgroups, with
> instructions on how to get the now-removed data.

Posted https://groups.google.com/forum/#!topic/mozilla.release.engineering/2h3l_JuDATg

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8633358 [details] [diff] [review]
Don't output the 'builders', 'slaves' & 'masters' keys

https://hg.mozilla.org/build/buildapi/rev/f0870af2a51f

Comment 10

[Ed Morley [:emorley]]

Would someone mind deploying this for me? :-)

From what I can tell, the builds-4hr script uses the files from /data/releng/src/buildapi on relengwebadm.

Comment 11

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #10)
> Would someone mind deploying this for me? :-)
> 
> From what I can tell, the builds-4hr script uses the files from
> /data/releng/src/buildapi on relengwebadm.

Comment 12

[Chris AtLee [:catlee]]

I've just deployed this.

Comment 13

[Ed Morley [:emorley]]

Thank you; builds-4hr is reflecting the changes :-)