Closed Bug 678266 Opened 9 years ago Closed 9 years ago

No email to for Android talos tp4m regression


(Release Engineering :: General, defect, P2)



(Not tracked)



(Reporter: mbrubeck, Assigned: catlee)



There was a 20% regression in tp4m earlier this week (bug 678261) on mozilla-inbound and mozilla-central, but there was no notification sent to the dev-tree-management list.  We'd like to figure out why, and ensure that we get notified about future regressions.

(I'm not sure if this should be filed under Releng, or Testing/Talos...)
Assignee: nobody → catlee
Priority: -- → P2
tp4m == "Tp4 Mobile" on the "Android 2.2" platform, according to graph server?
Ah ha, I think this got disabled when we made the switch from Tp4 to Tp5 on desktop.
(In reply to Chris AtLee [:catlee] from comment #3)
> Is this what you were looking for?
> thread/ce58137b46ea6012

Yes, that looks good.  Thanks!
Closed: 9 years ago
Resolution: --- → FIXED
We've had a couple more tp4m improvements and regressions this week with no email for mozilla-inbound.  (We did get an email for mozilla-central.)  Does this need to be kicked again?  I'll open a new bug if it continues...
Chris is the expert on this system, but he's out for a week so I took a quick look. In the log for the watcher it looks like it's paying attention to Android on Inbound.

Can you point to some regressions it has missed ? IIRC the algorithm ignores single data points, since that's likely to be a slave-related problem, eg these two

Tp4 Mobile NoChrome (m-c):[[84,11,20]]&sel=none&displayrange=7&datatype=running

Tp4 Mobile NoChrome (m-i):[[84,63,20]]&sel=none&displayrange=7&datatype=running
There was an improvement of about 6% near the midpoint of this graph, on both inbound and mozilla-central:[[84,63,20]]&sel=1313577712084.623,1313632809047.9739&displayrange=7&datatype=running

There was an email for the improvement on mozilla-central, but not on inbound.

Perhaps some of the extreme outliers in the recent mozilla-inbound data have just thrown off the statistics.
I talked to catlee and he said the analyser looks for regressions/improvements in the last 4 or 5 data points, based on the history of the 20 data points prior to that. So changes which dip and recover in that 5 point window aren't reported. FWIW, I can't see a permanent 6% improvement on that graph, but perhaps that's just the choice of y-scale.
Product: → Release Engineering
You need to log in before you can comment on or make changes to this bug.