Closed Bug 1265861 Opened 4 years ago Closed 4 years ago

Reduce threshold for perfherder alerts on APK size changes

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: Margaret, Assigned: gbrown)

Details

Attachments

(1 file)

Right now perfherder is configured to alert on performance regressions/improvements of 2% (I think) or more, but this creates a threshold that's too high for APK size changes.

Chatting with wlach, it sounds like we can change this to alert on smaller changes.

gbrown, can you help make this happen?
Flags: needinfo?(gbrown)
The magic incantation to make this work is to add alertThreshold to the PERFHERDER_DATA string. If you want something like 0.25%, you would do something like:

PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 25478543}, {"name": "classes.dex", "value": 5193188}, {"name": "omni.ja", "value": 6037744}], "name": "installer size", "value": 38296509, alertThreshold: 0.25}, ]}

Everything is documented here: https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json
The idea here is that the Fennec APK size is about 40 MB, 0.25% = 100K, which seems large enough to be actionable.

This change affects all platforms. A 0.25% change may not be as important to desktop...but then again, maybe it should be? Let's try this and dial it back -- perhaps for different platforms -- if it causes too many alerts, or non-actionable alerts.

:jmaher - Thought I'd check in with you since you may be triaging these -- any objections?
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Attachment #8743013 - Flags: review?(wlachance)
Attachment #8743013 - Flags: feedback?(jmaher)
Comment on attachment 8743013 [details] [diff] [review]
use alert threshold of 0.25% for "installer size"

Review of attachment 8743013 [details] [diff] [review]:
-----------------------------------------------------------------

great stuff!
Attachment #8743013 - Flags: feedback?(jmaher) → feedback+
Comment on attachment 8743013 [details] [diff] [review]
use alert threshold of 0.25% for "installer size"

Review of attachment 8743013 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!
Attachment #8743013 - Flags: review?(wlachance) → review+
https://hg.mozilla.org/mozilla-central/rev/43534e7fd750
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.