1334534 - Nightly builds being lumped together with integration builds in perfherder

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[William Lachance (:wlach)]

Android nightly builds have different characteristics than integration builds (e.g. they have multiple locales) but are being lumped in with the integration builds by perfherder for the apk size measurement:

https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-aurora,7601f68563dc6dad10bc630a38bb80e7b78fe5b1,1,2%5D

(it's hard to select items currently because of bug 1334527, please ignore that)

We should add "nightly" to extra_options in the log for these builds to distinguish them. I.e.

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 17957868}, {"name": "classes.dex", "value": 6861832}, {"name": "omni.ja", "value": 10430384}], "name": "installer size", "value": 38131527, "alertThreshold": 0.25}, ...

should be:

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 17957868}, {"name": "classes.dex", "value": 6861832}, {"name": "omni.ja", "value": 10430384}], "name": "installer size", "value": 38131527, "alertThreshold": 0.25, "extraOptions": ["nightly"]}, ...

(we may also want to flag these builds differently for build times)

Chris, Callek suggested on irc that you'd be a good person to fix this. If you're not, hopefully you know someone who would be...

Some more irc discussion here: http://logs.glob.uno/?c=mozilla%23mobile#c615321

Comment 1

[Chris Manchester (limited bugmail, email directly)]

For some reason I thought when reviewing bug 1328472 I thought "android-api-15-nightly" would end up in the platform name somewhere, and we wouldn't have this problem. Anyhow, I'll take a look.

Comment 2

[Chris Manchester (limited bugmail, email directly)]

We're having this issue for desktop builds as well, it's probably adding some noise there, too.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1334534 - Distinguish between nightly and opt builds in perfherder metrics.

Review commit: https://reviewboard.mozilla.org/r/107836/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107836/

Comment 4

[Chris Manchester (limited bugmail, email directly)]

This appears effective in https://treeherder.mozilla.org/#/jobs?repo=try&revision=b3433f9ee8e4a7b823d19399ef8444b208819ab4

Comment 5

[William Lachance (:wlach)]

Comment on attachment 8831273 [details]
Bug 1334534 - Distinguish between nightly and opt builds in perfherder metrics.

https://reviewboard.mozilla.org/r/107836/#review109042

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5d50db2b42ee
Distinguish between nightly and opt builds in perfherder metrics. r=wlach

Comment 7

[Chris Manchester (limited bugmail, email directly)]

As Will pointed out on irc, this is only working for the step durations, not for the installer size. I'm testing a new patch out on try now.

Comment 8

[Chris Manchester (limited bugmail, email directly)]

This is looking better: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d56deaada5222cdf66b626c2cc8e7a323db1127c

Comment 9

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1334534 - Distinguish nightlies for all build metrics, not just build times.

Review commit: https://reviewboard.mozilla.org/r/107856/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107856/

Comment 10

[William Lachance (:wlach)]

(In reply to Chris Manchester (:chmanchester) from comment #8)
> This is looking better:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=d56deaada5222cdf66b626c2cc8e7a323db1127c

If you look at this on stage (which has my patch for bug applied) you can see that `nightly` is now being set for all series. yay!

https://treeherder.allizom.org/#/jobs?repo=try&revision=d56deaada5222cdf66b626c2cc8e7a323db1127c

Comment 11

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/5d50db2b42ee