3.5-5.5% kraken regression on inbound (v.36) 4 different times with no alerts

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
4 years ago
3 years ago

People

(Reporter: jmaher, Unassigned)

Tracking

({perf, regression})

Trunk
perf, regression
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [talos_regression])

(Reporter)

Description

4 years ago
here is a graph showing the regressions on linux32, linux64, win7, win8, osx 10.6, and 10.8 isn't on there because of 1 really high data point which makes the graphs hard to navigate:
http://graphs.mozilla.org/graph.html#tests=%5B%5B232,131,31%5D,%5B232,131,35%5D,%5B232,131,25%5D,%5B232,131,33%5D,%5B232,63,21%5D%5D&sel=1414467732433.1672,1418258207349.5552&displayrange=90&datatype=running

So I don't really see alerts to match up for this.

Linux:
Oct 22: +10
Nov 3:  +20
Nov 10: -10
Nov 13: +6
Nov 19: +4
Nov 25: +10
Nov 28: -4

Linux64:
Oct 22: +10
Nov 3:  +20
Nov 10: -10
Nov 13: +6
-- didn't see the Nov 25 + 28 issues

Win7:
Oct 17: +10
Oct 22: +10
Nov 3:  +30

Win8:
Nov 3:  +25

Osx 10.6:
Oct 22: +10
Nov 3:  +40
Nov 13: -40
Nov 17: +30 (slow climb from 17th-23rd)

what is in common here is:
Oct 22
Nov 3
Nov 13 (probably a backout of Nov 3)


Looking in more detail, here is what looks to be the oct 22 culprit:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=5a443ae5cf22

Nov 3rd seems to be related to:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=d6cfdeca8a4c

I believe the Nov 3rd one is accurate as it was landed and backed out on Nov 3rd and then landed again on Nov 4th.
(Reporter)

Comment 1

4 years ago
this is known as it is on aurora as well.




Since dzAlerts sends an email when it finds a sustained regression.  If the regression happens on the same changeset along multiple branches and architectures no more alerts are sent out, they are silently added to the database.  Which that in mind, it seems dzAlerts did alert all improvments:

> Oct 22: +10 Multiple alerts sent out between 18th and 24th, as both stanford-crypto-pbkdf2 and json-parse-financial cause improvements/regressions
Branch=Mozilla-Inbound
Platform=mac (OS X 10.8) 	
Suite=kraken 
Test Name=stanford-crypto-pbkdf2 		
Date/Time=2014-10-22 15:21:30 	
Before=101.4
After=109.6	
Diff=8.1


> Nov 3:  Multiple alerts sent out as the exact push was unclear: Sent Nov 5th, 3:14am
Branch=Mozilla-Inbound-Non-PGO
Platform=win (6.2.9200)
Suite=kraken
Test Name=stanford-crypto-ccm 	
Date/Time=2014-11-04 10:16:24
Before=108.1 
After=118.1
Diff=9.9


> Nov 13: +6  Mail sent Nov 17 11:41, cause by audio-beat-detection
Branch=Firefox
Platform=mac (OS X 10.8) 
Suite=kraken
Test Name=audio-beat-detection
Date/Time=2014-11-14 22:48:07
Before=132.6 
After=143.3 
Diff=11.0


> Nov 19: +4  Sent Nov 21 07:36
Branch	Firefox Mozilla-Inbound
Platform win (6.2.9200)
Suite	kraken
Test Name	json-parse-financial
Date/Time	2014-11-19 18:25
Before	68.43
After	75.71
Diff	7.3


> Nov 25: +10  Sent Nov 26 9:42
Branch	Firefox Mozilla-Inbound-Non-PGO
Platform	linux (Ubuntu 12.04)
Suite	kraken
Test Name	json-parse-financial
Date/Time	2014-11-25 12:44
Before	81.88
After	85.88
Diff	4


What's concerning is the lack of alert for regressions:  

> Nov 10: -10
> Nov 28: -4

Which makes me think that there is something filtering them.  I will run these tests through development and see what happens
Found the problem:  Tests were defaulting to lower==better.  I changed kraken to be the opposite.
(Reporter)

Comment 4

4 years ago
oh awesome!  well done.
(In reply to Kyle Lahnakoski [:ekyle] from comment #3)
> Found the problem:  Tests were defaulting to lower==better.  I changed
> kraken to be the opposite.

Kraken uses the time in ms, so lower==better should be correct... Or did we change the Kraken harness/score for Talos?
(Reporter)

Comment 6

4 years ago
I always have to look this up-
kraken: lower = better

the reverse tests (higher = better) in talos are:
dromaeo_dom
dromaeo_css
v8_7
canvasmark
Doh!  I switched kraken back to better==lower.  

At least now it is explicit, and I added a reference to this bug in the settings
(Reporter)

Comment 8

4 years ago
this is now on mozilla-beta.
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.