new bloom test on osx is generating too many alerts

RESOLVED FIXED in Firefox 55

Status

Testing
Talos
RESOLVED FIXED
7 months ago
7 months ago

People

(Reporter: jmaher, Assigned: jmaher)

Tracking

Trunk
mozilla55
Points:
---

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(1 attachment)

(Assignee)

Description

7 months ago
I am concerned about the new bloom test, specifically with the alerts we are seeing from OSX.

here is the data for all the platforms:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bautoland,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&series=%5Bautoland,7595a193a0efa0d6b73e058f912a913862eaa9a1,1,1%5D&series=%5Bautoland,297a614b54fa9f991fb2cff58ea4db9cbc7b1bd2,1,1%5D

if you mute (uncheck) the series, you can see the win7 and win8 are acting normal.  Linux just got started yesterday, so we don't have much data.

Now look at osx:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bautoland,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D

8 alerts in the last week (4 regressions, 4 improvements).  These are not related to code landing and backing out.  

I feel this test is too sensitive on OSX- I would like to find a way to reduce the alerts we see here so we do not randomize developers.

a few options:
1) increase the alert threshold to 5% (not sure how to do this for osx specifically, we can probably figure it out)
2) do not run the test on OSX
3) realize that OSX is problematic and put resources towards investigating these
4) adjust the test

honestly options 1 or 2 are the most realistic.
(Assignee)

Comment 1

7 months ago
:bholley, do you have thoughts here or ideas of who might have interest in helping figure out what is going on and what to do?
Flags: needinfo?(bobbyholley)
Bumping the threshold to 5% on all the perf reftests should be fine. The swings we're looking for with these tests are larger than that, and I don't want to waste anybody's time here.
Flags: needinfo?(bobbyholley)
(Assignee)

Comment 3

7 months ago
Created attachment 8861514 [details] [diff] [review]
5% alert threshold for bloom tests
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8861514 - Flags: review?(rwood)

Updated

7 months ago
Attachment #8861514 - Flags: review?(rwood) → review+

Comment 4

7 months ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4da0e269a156
new bloom test on osx is generating too many alerts. r=rwood
https://hg.mozilla.org/mozilla-central/rev/4da0e269a156
Status: ASSIGNED → RESOLVED
Last Resolved: 7 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.