1262538 - 55.04% build times (linux64) regression on push f5090987b7c6 (Tue Mar 22 2016)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

there is a build times change as a result of push f5090987b7c6 on inbound:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=fcb7f6fbe3ba&tochange=f5090987b7c6

I assume it is related to that change, but it could be elsewhere, you can see the pattern of change here:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bfx-team,1475a04eb1cb14461eaf5e439a6fd435d0f78184,0%5D&series=%5Bmozilla-inbound,1475a04eb1cb14461eaf5e439a6fd435d0f78184,1%5D&series=%5Bmozilla-central,1475a04eb1cb14461eaf5e439a6fd435d0f78184,0%5D&zoom=1458484586503.9841,1458654712021.9124,368.0297397769518,3669.1449814126395&selected=%5Bmozilla-inbound,1475a04eb1cb14461eaf5e439a6fd435d0f78184,28877,24282464%5D

not sure if we are really interested in fixing this, but we should be aware of it :)

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Interestingly, if I overlay the build times on central there's an improvement when that merges. It's almost like we're doing more complete builds on inbound, but those builds are faster, which we also see in the more complete builds on central.

Comment 2

[Chris Manchester (limited bugmail, email directly)]

Actually, looking more closely at the graph, the new norm is established around the time of bug 1251713, so there may be a submission confusion (like bug 1261081 at play here).

Comment 3

[Gregory Szorc [:gps]]

IIRC we have sccache enabled on inbound but not central. So inbound build times should be shorter than central's.

Comment 4

[Mike Hommey [:glandium]]

Clicking on a few of the "job" links on the points with high values, they were all taskcluster jobs.

We don't use sccache on taskcluster, btw (bug 1207573)

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Chris Manchester (:chmanchester) from comment #2)
> Actually, looking more closely at the graph, the new norm is established
> around the time of bug 1251713, so there may be a submission confusion (like
> bug 1261081 at play here).

Sounds like this was just us mixing Taskcluster (without sccache at the time) and Buildbot build data. We fixed that in bug 1315041.