Reduce threshold for perfherder alerts on APK size changes

RESOLVED FIXED in Firefox 48

Status

defect
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: Margaret, Assigned: gbrown)

Tracking

Trunk
mozilla48

Firefox Tracking Flags

(firefox48 fixed)

Details

Attachments

(1 attachment)

Reporter

Description

3 years ago
Right now perfherder is configured to alert on performance regressions/improvements of 2% (I think) or more, but this creates a threshold that's too high for APK size changes.

Chatting with wlach, it sounds like we can change this to alert on smaller changes.

gbrown, can you help make this happen?
Flags: needinfo?(gbrown)
The magic incantation to make this work is to add alertThreshold to the PERFHERDER_DATA string. If you want something like 0.25%, you would do something like:

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 25478543}, {"name": "classes.dex", "value": 5193188}, {"name": "omni.ja", "value": 6037744}], "name": "installer size", "value": 38296509, alertThreshold: 0.25}, ]}

Everything is documented here: https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json
Assignee

Comment 2

3 years ago
The idea here is that the Fennec APK size is about 40 MB, 0.25% = 100K, which seems large enough to be actionable.

This change affects all platforms. A 0.25% change may not be as important to desktop...but then again, maybe it should be? Let's try this and dial it back -- perhaps for different platforms -- if it causes too many alerts, or non-actionable alerts.

:jmaher - Thought I'd check in with you since you may be triaging these -- any objections?
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Attachment #8743013 - Flags: review?(wlachance)
Attachment #8743013 - Flags: feedback?(jmaher)
Comment on attachment 8743013 [details] [diff] [review]
use alert threshold of 0.25% for "installer size"

Review of attachment 8743013 [details] [diff] [review]:
-----------------------------------------------------------------

great stuff!
Attachment #8743013 - Flags: feedback?(jmaher) → feedback+
Comment on attachment 8743013 [details] [diff] [review]
use alert threshold of 0.25% for "installer size"

Review of attachment 8743013 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!
Attachment #8743013 - Flags: review?(wlachance) → review+

Comment 6

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/43534e7fd750
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.