1204633 - ensure perfherder alerts are at parity with graph server alerts

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in bug 1150152 we have been working to get valid data and alerts from perfherder.  We are almost there and now we need to validate the data one last time.  We expect to see test and counter alerts.

Comment 1

[William Lachance (:wlach)]

Attachment: alerts from last week as of september 14th

Output of running this on stage:

./manage.py test_analyze_perf --project mozilla-inbound --project fx-team --project mozilla-central --project mozilla-aurora --project mozilla-beta --project b2g-inbound

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

looking at this alert in the attachment:
fx-team,windows7-32,2134698ffee49a2f33235c3640553083cf715b8a,tps summary e10s opt,4968,1441725005,47.8075,145.869460195,7.04229393366,5f5ffdb6dba3

this shows that on fx-team revision 5f5ffdb6dba3 we have a win7 tps e10s regression of 145.87.

Lets look at the graph:
graph server:
http://graphs.mozilla.org/graph.html#tests=[[329,64,47],[329,132,47]]&sel=1441623752535.163,1441768370560.914,8.57142857142857,91.42857142857143&displayrange=30&datatype=geo

perfherder:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[fx-team,2134698ffee49a2f33235c3640553083cf715b8a,1]&series=[fx-team,ebf7e378c7d25068b99e8924e1d89e454e684b4c,1]


both graphs show the same thing!  This is great.  The inconsistency is that we have an alert for non-pgo when it should be pgo (at least for that value).  Perfherder doesn't have a pgo alert, only a non-pgo alert.

Graph server has both pgo and non pgo alerts:
"Fx-Team-Non-PGO","WINNT 6.1 (ix) (e10s)","Talos Page Switch","2015-09-09 06:05:34","-221%","20cc75c647f1d3f8b4bc594b989f4cd0e73c9949"
"Fx-Team","WINNT 6.1 (ix) (e10s)","Talos Page Switch","2015-09-10 20:37:03","-161%","5f5ffdb6dba376015615bac236f5cba08f544edf"

my script failed to find this as a match since the pgo regression is not as amplified as the non-pgo regression.  


Also, graph server has 300+ alerts in the last week, many for tps, and there is only one tps alert in the entire attachment.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-central,6764e9b1ce6291a6d9683810b584f1f3536362e6,1]&highlightedRevisions=dd2a1d737a64

and;
http://graphs.mozilla.org/graph.html#tests=[[323,1,43]]&sel=none&displayrange=30&datatype=geo

I am looking for an alert on tp5o scroll e10s linux* from perherder on sept 9th from rev dd2a1d737a64, but I can't even see it on the graph.  Do we not have data prior to september 9th?  That would be really odd.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

oh, silly me, opt is new on september 9th, but pgo has been around for a while:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-central,ec9a6eab017cc5e0fb8b9d024b4b9788af627d86,1]&series=[mozilla-central,a184c3ca3ee4bd9f7db02216f5127915ef285ba9,1]&highlightedRevisions=dd2a1d737a64

I don't see alerts for this.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: alerts from graph server

Comment 6

[William Lachance (:wlach)]

Attachment: alerts-sept-17-2015-last-three-weeks.csv

Alerts from the last week, should be a superset of what graphserver is producing:

./manage.py test_analyze_perf --project mozilla-inbound --project fx-team --project mozilla-central --project mozilla-aurora --project mozilla-beta --project b2g-inbound --time-interval 1814400 > ~/alerts-sept-17-2015-last-three-weeks.csv

Comment 7

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #6)
> Created attachment 8662522 [details]
> alerts-sept-17-2015-last-three-weeks.csv
> 
> Alerts from the last week, should be a superset of what graphserver is
> producing:

Sorry, that should say "last three weeks".

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

ok, I mark this project as done- looking over the latest bits from wlach, we are generating the proper alerts and matching up with graphserver quite well.