Closed Bug 1334534 Opened 3 years ago Closed 3 years ago

Nightly builds being lumped together with integration builds in perfherder

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox54 fixed)

RESOLVED FIXED
mozilla54
Tracking Status
firefox54 --- fixed

People

(Reporter: wlach, Assigned: chmanchester)

References

Details

Attachments

(2 files)

Android nightly builds have different characteristics than integration builds (e.g. they have multiple locales) but are being lumped in with the integration builds by perfherder for the apk size measurement:

https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-aurora,7601f68563dc6dad10bc630a38bb80e7b78fe5b1,1,2%5D

(it's hard to select items currently because of bug 1334527, please ignore that)

We should add "nightly" to extra_options in the log for these builds to distinguish them. I.e.

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 17957868}, {"name": "classes.dex", "value": 6861832}, {"name": "omni.ja", "value": 10430384}], "name": "installer size", "value": 38131527, "alertThreshold": 0.25}, ...

should be:

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 17957868}, {"name": "classes.dex", "value": 6861832}, {"name": "omni.ja", "value": 10430384}], "name": "installer size", "value": 38131527, "alertThreshold": 0.25, "extraOptions": ["nightly"]}, ...

(we may also want to flag these builds differently for build times)

Chris, Callek suggested on irc that you'd be a good person to fix this. If you're not, hopefully you know someone who would be...

Some more irc discussion here: http://logs.glob.uno/?c=mozilla%23mobile#c615321
Flags: needinfo?(cmanchester)
For some reason I thought when reviewing bug 1328472 I thought "android-api-15-nightly" would end up in the platform name somewhere, and we wouldn't have this problem. Anyhow, I'll take a look.
Assignee: nobody → cmanchester
Blocks: 1328472
Flags: needinfo?(cmanchester)
We're having this issue for desktop builds as well, it's probably adding some noise there, too.
Summary: Android nightly builds being lumped together with integration builds in perfherder → Nightly builds being lumped together with integration builds in perfherder
Comment on attachment 8831273 [details]
Bug 1334534 - Distinguish between nightly and opt builds in perfherder metrics.

https://reviewboard.mozilla.org/r/107836/#review109042
Attachment #8831273 - Flags: review?(wlachance) → review+
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5d50db2b42ee
Distinguish between nightly and opt builds in perfherder metrics. r=wlach
As Will pointed out on irc, this is only working for the step durations, not for the installer size. I'm testing a new patch out on try now.
Blocks: 1334667
Attachment #8831299 - Flags: review?(wlachance)
(In reply to Chris Manchester (:chmanchester) from comment #8)
> This is looking better:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=d56deaada5222cdf66b626c2cc8e7a323db1127c

If you look at this on stage (which has my patch for bug applied) you can see that `nightly` is now being set for all series. yay!

https://treeherder.allizom.org/#/jobs?repo=try&revision=d56deaada5222cdf66b626c2cc8e7a323db1127c
https://hg.mozilla.org/mozilla-central/rev/5d50db2b42ee
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 54
Component: Build Config → General
Product: Firefox → Firefox Build System
Target Milestone: Firefox 54 → mozilla54
You need to log in before you can comment on or make changes to this bug.