Missing talos alert for very large improvement to trobopan



6 years ago
3 years ago


(Reporter: gbrown, Unassigned)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: [SfN])



6 years ago
The patch for bug 802400 made a substantial improvement to trobopan, reducing its metric from ~550000 to ~19000. I reviewed the dev-tree-management alerts between 

dev-tree-management Digest, Vol 47, Issue 143
dev-tree-management Digest, Vol 47, Issue 131

and could find no mention of the change. It seems to me that we occasionally miss alerts for both improvements and regressions...I do not know why.

(The same patch produced modest improvements to tcheck2 and tcheck3, which were reported in dev-tree-management Digest, Vol 47, Issue 141).
Whiteboard: [SfN]
OS: Mac OS X → Android
Hardware: x86 → ARM
this would be the magic graph server script

Comment 2

6 years ago
Here's a more subtle example: 

Android Ts regressed on m-i from about 3100 to about 4800 on November 30, 2012. The same regression is seen on m-c on December 1. I cannot find any dev-tree-management reports for these regressions.

However, what appears to be the same regression on mozilla-aurora on January 8 was reported in dev-tree-management Digest, Vol 49, Issue 49:

Message: 4
Date: Wed, 09 Jan 2013 13:41:20 -0000
From: nobody@cruncher.build.mozilla.org
To: dev-tree-management@lists.mozilla.org
Subject: <Regression> Mozilla-Aurora - Ts - Android 2.2 (Native) -
Content-Type: text/plain; charset="us-ascii"

Regression: Mozilla-Aurora - Ts - Android 2.2 (Native) - 54.3% increase
    Previous: avg 3145.322 stddev 356.677 of 30 runs up to revision 1903a40f342d
    New     : avg 4852.908 stddev 298.683 of 5 runs since revision a459eaaea5ae
    Change  : +1707.586 (54.3% / z=4.787)
    Graph   : http://mzl.la/13gF1KU

Changeset range: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=1903a40f342d&tochange=a459eaaea5ae

I say "more subtle example" because in this case there is a lot of noise: Ts went from a range of about 3000-3200 to a range of about 3000-5500.
we miss alerts on graph server for a variety of reasons, some are bugs in graph server, some are suppression code in the algorithm which seems to cause us to miss other alerts on other platforms as well.

We are moving to autophone and the regression detection will be much different.
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.