here is a graph showing the regressions on linux32, linux64, win7, win8, osx 10.6, and 10.8 isn't on there because of 1 really high data point which makes the graphs hard to navigate: http://graphs.mozilla.org/graph.html#tests=%5B%5B232,131,31%5D,%5B232,131,35%5D,%5B232,131,25%5D,%5B232,131,33%5D,%5B232,63,21%5D%5D&sel=1414467732433.1672,1418258207349.5552&displayrange=90&datatype=running So I don't really see alerts to match up for this. Linux: Oct 22: +10 Nov 3: +20 Nov 10: -10 Nov 13: +6 Nov 19: +4 Nov 25: +10 Nov 28: -4 Linux64: Oct 22: +10 Nov 3: +20 Nov 10: -10 Nov 13: +6 -- didn't see the Nov 25 + 28 issues Win7: Oct 17: +10 Oct 22: +10 Nov 3: +30 Win8: Nov 3: +25 Osx 10.6: Oct 22: +10 Nov 3: +40 Nov 13: -40 Nov 17: +30 (slow climb from 17th-23rd) what is in common here is: Oct 22 Nov 3 Nov 13 (probably a backout of Nov 3) Looking in more detail, here is what looks to be the oct 22 culprit: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=5a443ae5cf22 Nov 3rd seems to be related to: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=d6cfdeca8a4c I believe the Nov 3rd one is accurate as it was landed and backed out on Nov 3rd and then landed again on Nov 4th.
this is known as it is on aurora as well.
Since dzAlerts sends an email when it finds a sustained regression. If the regression happens on the same changeset along multiple branches and architectures no more alerts are sent out, they are silently added to the database. Which that in mind, it seems dzAlerts did alert all improvments: > Oct 22: +10 Multiple alerts sent out between 18th and 24th, as both stanford-crypto-pbkdf2 and json-parse-financial cause improvements/regressions Branch=Mozilla-Inbound Platform=mac (OS X 10.8) Suite=kraken Test Name=stanford-crypto-pbkdf2 Date/Time=2014-10-22 15:21:30 Before=101.4 After=109.6 Diff=8.1 > Nov 3: Multiple alerts sent out as the exact push was unclear: Sent Nov 5th, 3:14am Branch=Mozilla-Inbound-Non-PGO Platform=win (6.2.9200) Suite=kraken Test Name=stanford-crypto-ccm Date/Time=2014-11-04 10:16:24 Before=108.1 After=118.1 Diff=9.9 > Nov 13: +6 Mail sent Nov 17 11:41, cause by audio-beat-detection Branch=Firefox Platform=mac (OS X 10.8) Suite=kraken Test Name=audio-beat-detection Date/Time=2014-11-14 22:48:07 Before=132.6 After=143.3 Diff=11.0 > Nov 19: +4 Sent Nov 21 07:36 Branch Firefox Mozilla-Inbound Platform win (6.2.9200) Suite kraken Test Name json-parse-financial Date/Time 2014-11-19 18:25 Before 68.43 After 75.71 Diff 7.3 > Nov 25: +10 Sent Nov 26 9:42 Branch Firefox Mozilla-Inbound-Non-PGO Platform linux (Ubuntu 12.04) Suite kraken Test Name json-parse-financial Date/Time 2014-11-25 12:44 Before 81.88 After 85.88 Diff 4 What's concerning is the lack of alert for regressions: > Nov 10: -10 > Nov 28: -4 Which makes me think that there is something filtering them. I will run these tests through development and see what happens
Found the problem: Tests were defaulting to lower==better. I changed kraken to be the opposite.
oh awesome! well done.
(In reply to Kyle Lahnakoski [:ekyle] from comment #3) > Found the problem: Tests were defaulting to lower==better. I changed > kraken to be the opposite. Kraken uses the time in ms, so lower==better should be correct... Or did we change the Kraken harness/score for Talos?
I always have to look this up- kraken: lower = better the reverse tests (higher = better) in talos are: dromaeo_dom dromaeo_css v8_7 canvasmark
Doh! I switched kraken back to better==lower. At least now it is explicit, and I added a reference to this bug in the settings
this is now on mozilla-beta.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.