1403456 - 187.77% build times (android-4-0-armv7-api16) regression on push efff4f307675a2226d4ef5867d64b6499bde39e3 (Thu Sep 21 2017)

Status: RESOLVED INVALID

(Firefox for Android Graveyard :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=efff4f307675a2226d4ef5867d64b6499bde39e3

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

188%  build times summary android-4-0-armv7-api16 opt taskcluster-c4.2xlarge     648.73 -> 1,866.84


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=9614

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[Ionuț Goldan [:igoldan]]

This build time regression on Beta appeared after a merge with the integration branches. Do you know why this is happening?

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

I have no idea...maybe Nathan has an idea?

Comment 3

[Nathan Froyd [:froydnj]]

The graph for this metric over the last 30 days (even though the dropdown box says 90, for whatever reason):

https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-beta,f86559dd9363fbf8589b335f02840056823515a1,1%5D&selected=%5Bmozilla-beta,f86559dd9363fbf8589b335f02840056823515a1%5D

Looks pretty strongly bimodal, whereas after this push, we're no longer bimodal.  So perhaps something actually fixed the bimodalness we were seeing?  Nothing really jumps out at me in the pushlog that would address bimodalness, though...

Comment 4

[Valentina Claudia Ona]

Hi Ionut,
All versions are affected, right?

Comment 5

[Ionuț Goldan [:igoldan]]

If you refer to Android versions, then only Android 4.0 build processes are affected.
But if you mean Firefox trees, then only Firefox Beta 57 experiences this issue.

This can be observed better from here:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=mozilla-inbound,1544599,1,2&series=mozilla-beta,1544696,1,2&series=mozilla-release,1554120,1,2

mozilla-inbound is for Fx58
mozilla-beta is for Fx57
mozilla-release is for Fx56

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I do see what :froydnj mentions as valid- this was bi-modal, now we end up with a single reported value instead of toggling between two- this makes it more deterministic and this isn't a regression for half the builds :)

Comment 7

[(not currently active) Ted Mielczarek]

I looked at the graph from comment 3, and clicking on one of the low data points and one of the high data points, and then clicking through to "job" showed me:
low: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=a6fb3b978941f76b727eafee21ea3d440299648d&selectedJob=132341639&group_state=expanded
high: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=efff4f307675a2226d4ef5867d64b6499bde39e3&selectedJob=133054669&group_state=expanded

The low job here is a "findbugs" job, the high one is a build. Looking at:
https://hg.mozilla.org/mozilla-central/annotate/20d57b9c4183973af4af5e078dff2aec0b74f928/taskcluster/ci/build/android-stuff.yml#l225

suggests that this got fixed in bug 1383973 to properly separate the data from these different job types.