678266 - No email to mozilla.dev.tree-management for Android talos tp4m regression

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Matt Brubeck (:mbrubeck)]

There was a 20% regression in tp4m earlier this week (bug 678261) on mozilla-inbound and mozilla-central, but there was no notification sent to the dev-tree-management list.  We'd like to figure out why, and ensure that we get notified about future regressions.

(I'm not sure if this should be filed under Releng, or Testing/Talos...)

Comment 1

[Chris AtLee [:catlee]]

tp4m == "Tp4 Mobile" on the "Android 2.2" platform, according to graph server?

Comment 2

[Chris AtLee [:catlee]]

Ah ha, I think this got disabled when we made the switch from Tp4 to Tp5 on desktop.

Comment 3

[Chris AtLee [:catlee]]

Is this what you were looking for?

http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/ce58137b46ea6012

Comment 4

[Matt Brubeck (:mbrubeck)]

(In reply to Chris AtLee [:catlee] from comment #3)
> Is this what you were looking for?
> 
> http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/
> thread/ce58137b46ea6012

Yes, that looks good.  Thanks!

Comment 5

[Matt Brubeck (:mbrubeck)]

We've had a couple more tp4m improvements and regressions this week with no email for mozilla-inbound.  (We did get an email for mozilla-central.)  Does this need to be kicked again?  I'll open a new bug if it continues...

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Chris is the expert on this system, but he's out for a week so I took a quick look. In the log for the watcher it looks like it's paying attention to Android on Inbound.

Can you point to some regressions it has missed ? IIRC the algorithm ignores single data points, since that's likely to be a slave-related problem, eg these two

Tp4 Mobile NoChrome (m-c):
http://graphs-new.mozilla.org/graph.html#tests=[[84,11,20]]&sel=none&displayrange=7&datatype=running

Tp4 Mobile NoChrome (m-i):
http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=none&displayrange=7&datatype=running

Comment 7

[Matt Brubeck (:mbrubeck)]

There was an improvement of about 6% near the midpoint of this graph, on both inbound and mozilla-central:

http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=1313577712084.623,1313632809047.9739&displayrange=7&datatype=running

There was an email for the improvement on mozilla-central, but not on inbound.

Perhaps some of the extreme outliers in the recent mozilla-inbound data have just thrown off the statistics.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

I talked to catlee and he said the analyser looks for regressions/improvements in the last 4 or 5 data points, based on the history of the 20 data points prior to that. So changes which dip and recover in that 5 point window aren't reported. FWIW, I can't see a permanent 6% improvement on that graph, but perhaps that's just the choice of y-scale.