Missing talos alert for very large improvement to trobopan

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
6 years ago
3 years ago

People

(Reporter: gbrown, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [SfN])

(Reporter)

Description

6 years ago
The patch for bug 802400 made a substantial improvement to trobopan, reducing its metric from ~550000 to ~19000. I reviewed the dev-tree-management alerts between 

dev-tree-management Digest, Vol 47, Issue 143
...
dev-tree-management Digest, Vol 47, Issue 131

and could find no mention of the change. It seems to me that we occasionally miss alerts for both improvements and regressions...I do not know why.

(The same patch produced modest improvements to tcheck2 and tcheck3, which were reported in dev-tree-management Digest, Vol 47, Issue 141).

Updated

6 years ago
Whiteboard: [SfN]

Updated

6 years ago
OS: Mac OS X → Android
Hardware: x86 → ARM
this would be the magic graph server script
(Reporter)

Comment 2

6 years ago
Here's a more subtle example: 

Android Ts regressed on m-i from about 3100 to about 4800 on November 30, 2012. The same regression is seen on m-c on December 1. I cannot find any dev-tree-management reports for these regressions.

However, what appears to be the same regression on mozilla-aurora on January 8 was reported in dev-tree-management Digest, Vol 49, Issue 49:

Message: 4
Date: Wed, 09 Jan 2013 13:41:20 -0000
From: nobody@cruncher.build.mozilla.org
To: dev-tree-management@lists.mozilla.org
Subject: <Regression> Mozilla-Aurora - Ts - Android 2.2 (Native) -
        54.3%
Message-ID:
        <20130109134120.532631042ED@cruncher.srv.releng.scl3.mozilla.com>
Content-Type: text/plain; charset="us-ascii"

Regression: Mozilla-Aurora - Ts - Android 2.2 (Native) - 54.3% increase
-----------------------------------------------------------------------
    Previous: avg 3145.322 stddev 356.677 of 30 runs up to revision 1903a40f342d
    New     : avg 4852.908 stddev 298.683 of 5 runs since revision a459eaaea5ae
    Change  : +1707.586 (54.3% / z=4.787)
    Graph   : http://mzl.la/13gF1KU

Changeset range: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=1903a40f342d&tochange=a459eaaea5ae


I say "more subtle example" because in this case there is a lot of noise: Ts went from a range of about 3000-3200 to a range of about 3000-5500.
we miss alerts on graph server for a variety of reasons, some are bugs in graph server, some are suppression code in the algorithm which seems to cause us to miss other alerts on other platforms as well.

We are moving to autophone and the regression detection will be much different.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.