1021951 - iframe.svg spewed alerts as a regression spread through branches

Status: RESOLVED WONTFIX

(Datazilla Graveyard :: Metrics, defect)

Description

[Kyle Lahnakoski [:ekyle]]

about 2014-06-03 13:51:45 iframe.svg regressed (http://people.mozilla.org/~klahnakoski/talos/DZ-ShowPage.html#sampleMax=2014-06-07+00%3A00%3A00&sampleMin=2014-06-04+00%3A00%3A00&test=iframe.svg&branch=Mozilla-Inbound&bits=x86_64&platform=x86_64&os=win) as the regression spread through the branches (including the virtual PGO/non-PGO branches) more alerts were raised.

The problem is lack of changeset<->pushlog mappings in the database currently used.  this makes the alert system miss many of the changesets in a push.

There is also the issue that the window size was bigger than the first regression range (http://people.mozilla.org/~klahnakoski/talos/DZ-ShowPage.html#sampleMax=2014-06-07+00%3A00%3A00&sampleMin=2014-06-01+00%3A00%3A00&test=iframe.svg&branch=Mozilla-Inbound&bits=x86_64&platform=x86_64&os=linux ) which doubled the problem.

Comment 1

[Kyle Lahnakoski [:ekyle]]

There is a "preview anomaly" which I believe (without evidence) is being caused by improper ordering used by using only pushdate:

The *Preview Anomaly* can be seen before many perf test regressions (or improvements).  The performance results will stay at a normal measure for quite some time (A), jump up to the regressed measure for one or more revisions (B), then drop down to the normal measure (C) for a few revisions, and finally "start" a sustained regression (D).  

"start" is quoted because the "preview" (B) is not long enough to be considered a sustained regression.  It is feels as though B and C are reversed.  


Example:
http://people.mozilla.org/~klahnakoski/talos/Alert-Talos.html#sampleMin=2014-10-01&platform=x86_64&sampleMax=2014-10-31&branch=Fx-Team&suite=tp5o_scroll.dailymotion.com&os=mac.OS+X+10.8

Comment 2

[Kyle Lahnakoski [:ekyle]]

dzAlerts is a dead project