There was a 20% regression in tp4m earlier this week (bug 678261) on mozilla-inbound and mozilla-central, but there was no notification sent to the dev-tree-management list. We'd like to figure out why, and ensure that we get notified about future regressions. (I'm not sure if this should be filed under Releng, or Testing/Talos...)
tp4m == "Tp4 Mobile" on the "Android 2.2" platform, according to graph server?
Ah ha, I think this got disabled when we made the switch from Tp4 to Tp5 on desktop.
Is this what you were looking for? http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/ce58137b46ea6012
(In reply to Chris AtLee [:catlee] from comment #3) > Is this what you were looking for? > > http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/ > thread/ce58137b46ea6012 Yes, that looks good. Thanks!
We've had a couple more tp4m improvements and regressions this week with no email for mozilla-inbound. (We did get an email for mozilla-central.) Does this need to be kicked again? I'll open a new bug if it continues...
Chris is the expert on this system, but he's out for a week so I took a quick look. In the log for the watcher it looks like it's paying attention to Android on Inbound. Can you point to some regressions it has missed ? IIRC the algorithm ignores single data points, since that's likely to be a slave-related problem, eg these two Tp4 Mobile NoChrome (m-c): http://graphs-new.mozilla.org/graph.html#tests=[[84,11,20]]&sel=none&displayrange=7&datatype=running Tp4 Mobile NoChrome (m-i): http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=none&displayrange=7&datatype=running
There was an improvement of about 6% near the midpoint of this graph, on both inbound and mozilla-central: http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=1313577712084.623,1313632809047.9739&displayrange=7&datatype=running There was an email for the improvement on mozilla-central, but not on inbound. Perhaps some of the extreme outliers in the recent mozilla-inbound data have just thrown off the statistics.
I talked to catlee and he said the analyser looks for regressions/improvements in the last 4 or 5 data points, based on the history of the 20 data points prior to that. So changes which dip and recover in that 5 point window aren't reported. FWIW, I can't see a permanent 6% improvement on that graph, but perhaps that's just the choice of y-scale.