If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

No email to mozilla.dev.tree-management for Android talos tp4m regression

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: mbrubeck, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
There was a 20% regression in tp4m earlier this week (bug 678261) on mozilla-inbound and mozilla-central, but there was no notification sent to the dev-tree-management list.  We'd like to figure out why, and ensure that we get notified about future regressions.

(I'm not sure if this should be filed under Releng, or Testing/Talos...)
(Assignee)

Updated

6 years ago
Assignee: nobody → catlee
Priority: -- → P2
(Assignee)

Comment 1

6 years ago
tp4m == "Tp4 Mobile" on the "Android 2.2" platform, according to graph server?
(Assignee)

Comment 2

6 years ago
Ah ha, I think this got disabled when we made the switch from Tp4 to Tp5 on desktop.
(Assignee)

Comment 3

6 years ago
Is this what you were looking for?

http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/ce58137b46ea6012
(Reporter)

Comment 4

6 years ago
(In reply to Chris AtLee [:catlee] from comment #3)
> Is this what you were looking for?
> 
> http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/
> thread/ce58137b46ea6012

Yes, that looks good.  Thanks!
(Assignee)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 5

6 years ago
We've had a couple more tp4m improvements and regressions this week with no email for mozilla-inbound.  (We did get an email for mozilla-central.)  Does this need to be kicked again?  I'll open a new bug if it continues...
Chris is the expert on this system, but he's out for a week so I took a quick look. In the log for the watcher it looks like it's paying attention to Android on Inbound.

Can you point to some regressions it has missed ? IIRC the algorithm ignores single data points, since that's likely to be a slave-related problem, eg these two

Tp4 Mobile NoChrome (m-c):
http://graphs-new.mozilla.org/graph.html#tests=[[84,11,20]]&sel=none&displayrange=7&datatype=running

Tp4 Mobile NoChrome (m-i):
http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=none&displayrange=7&datatype=running
(Reporter)

Comment 7

6 years ago
There was an improvement of about 6% near the midpoint of this graph, on both inbound and mozilla-central:

http://graphs-new.mozilla.org/graph.html#tests=[[84,63,20]]&sel=1313577712084.623,1313632809047.9739&displayrange=7&datatype=running

There was an email for the improvement on mozilla-central, but not on inbound.

Perhaps some of the extreme outliers in the recent mozilla-inbound data have just thrown off the statistics.
I talked to catlee and he said the analyser looks for regressions/improvements in the last 4 or 5 data points, based on the history of the 20 data points prior to that. So changes which dip and recover in that 5 point window aren't reported. FWIW, I can't see a permanent 6% improvement on that graph, but perhaps that's just the choice of y-scale.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.